Email marketing often appears as the top-performing channel in last-click attribution because it reaches existing customers who are already likely to buy. True incremental measurement, using holdout groups and marketing mix modelling, typically shows a lower but still significant contribution that reflects what email genuinely adds above organic purchase intent.
Email consistently appears near the top of attribution reports for most e-commerce and subscription businesses. Some of that high reported ROI is genuine. Some of it is a measurement artefact. When you send a promotional email to someone who was already planning to buy, and they click it before purchasing, email gets the attribution credit for a sale it did not actually cause.
The last-click problem in email attribution
Email open rates and click-through rates measure engagement with the email itself. Conversion rates in email platform dashboards measure how many people clicked the email and then purchased within the attribution window, typically 24 hours to 7 days. Neither metric tells you whether those purchases would have happened without the email.
Loyal customers, who buy repeatedly regardless of whether they receive an email, inflate email's attributed revenue significantly. A customer who buys every six weeks and receives an email every two weeks will almost always have clicked a recent email before their latest purchase, even if the email had nothing to do with their decision.
Holdout groups: measuring true incrementality
A holdout group is a segment of your email list that is excluded from one or more email sends while the rest of the list receives them normally. By comparing the purchase behaviour of the holdout group against the send group over the same period, you measure how much incremental revenue the emails generated.
The holdout group needs to be randomly assigned and large enough to detect a meaningful difference. A holdout of 10 percent of your list is a common starting point. If you have a list of 100,000 subscribers, 10,000 are held out and 90,000 receive the campaign. Compare purchase rates, average order value, and total revenue per user between the two groups during and after the send.
- Assign holdout segments randomly rather than by customer tier or engagement level. Segmentation bias will corrupt your results.
- Hold the same group out for multiple sends to build statistical confidence over time rather than running a one-off test.
- Measure revenue per subscriber, not total revenue, so that group size differences do not distort the comparison.
- Include unsubscribes and bounces in both groups consistently so the comparison remains clean.
- Run holdouts for promotional campaigns and for automated flows (welcome series, abandoned cart) separately, because they have very different incrementality profiles.
Incrementality by email type
Abandoned cart emails and back-in-stock alerts typically show high incrementality because they reach people who signalled intent but did not convert. Broadcast promotional emails to the full list, and re-engagement campaigns, tend to show lower incrementality because a larger proportion of recipients would have purchased anyway.
Understanding incrementality at the email type level, rather than for email as a whole, helps you allocate effort and investment more accurately. Your abandoned cart automation may genuinely recover revenue that would have been lost. Your weekly promotional blast may be accelerating purchases that would have happened the following week anyway.
Marketing mix modelling and email
In an MMM, email spend is usually low and the send volume is the relevant variable. The model uses variation in email send frequency and volume over time to estimate contribution to sales. Email MMM is often less precise than social or TV modelling because email contacts are not random: you are sending to a self-selected list with known purchase history, which makes it harder to separate the email effect from the underlying customer intent.
Despite this limitation, MMM can help you understand whether increasing email frequency beyond a certain point adds revenue or simply redistributes it across different send dates. This frequency-response relationship is one of the most practically useful outputs of email modelling.
Revenue contribution versus customer lifetime value
A more sophisticated frame for email ROI is its contribution to customer retention and lifetime value (LTV), rather than just short-term revenue. Customers who engage with email have higher retention rates and longer LTV, but it is difficult to establish whether email causes that retention or whether more loyal customers are simply more likely to engage with email. Cohort analysis, comparing the retention of customers who have been in email programmes for different durations against those who have not, can help answer this question.
How do I run a holdout test without losing revenue?
The revenue risk from a holdout is real but usually small. A 10 percent holdout that reduces email-attributed revenue by 15 percent on that group during the test period typically represents a short-term cost of less than two percent of total email revenue. Weigh that against the insight gained from understanding true incrementality, which usually enables decisions that more than recoup the cost. Start with a single promotional campaign rather than holding out of automated flows, to limit the initial revenue exposure.
Does sending frequency affect incrementality?
Yes, significantly. Higher send frequency tends to reduce incrementality per email because you are reaching customers more often than their purchase intent rises. You are also training higher unsubscribe rates, which erodes your future reachable audience. The optimal send frequency for incrementality, not just open rates, is lower than most brands currently send. Holdout experiments that test different frequency levels for different segments can identify your specific optimal cadence.
How should I handle cross-channel attribution where email and paid social both touched the same customer?
This is where single-channel attribution completely breaks down. A customer who received an email and also saw a retargeting ad before purchasing generated one sale, not two. Last-click gives that sale to whichever channel they clicked last. Multi-touch models share the credit, but the shares are arbitrary. Marketing mix modelling is the right tool for cross-channel allocation because it estimates each channel's marginal contribution to total revenue, rather than assigning individual conversions to single channels.