Incrementality testing measures the true causal impact of your marketing by comparing what happens when people see your ads against what happens when they do not. It is the closest thing marketing has to a scientific experiment.
Every marketer wants to know the same thing: did that campaign actually work, or did those customers buy regardless? Incrementality testing is the most direct way to answer that question.
The principle is simple. You split your potential audience into two groups. One group sees your marketing. The other group does not. Then you compare the results. Whatever difference you see between the groups is the incremental effect of your marketing.
The clinical trial analogy
Think about how pharmaceutical companies test whether a new drug works. They give the drug to one group of patients and a placebo to another. The difference in outcomes between the two groups tells them how effective the drug actually is.
Incrementality testing applies the same logic to marketing. Your ad is the drug. The holdout group, the people who do not see your ad, is the control group. The difference in conversion rates or revenue between the two groups is your incremental lift.
Types of incrementality tests
Geo experiments
Geo experiments split your audience by geography. You run your campaign in some regions but not others, then compare the sales performance between them. This works well for campaigns that are hard to turn off for specific individuals, like TV or out-of-home advertising.
Holdout tests
Holdout tests randomly exclude a percentage of your target audience from seeing your campaign. If 90% of your audience sees your retargeting ads and 10% do not, you can compare the purchase rate between the two groups to measure how much the ads contributed.
Ghost ads or matched markets
Some platforms allow you to show a placeholder ad to a control group so that both groups have the same browsing experience, with only the actual campaign ad differing. This removes some potential bias from the comparison.
What makes a good incrementality test?
- Random assignment: the people or regions in each group should be randomly allocated, not selected by some other factor
- Large enough groups: small experiments have high statistical uncertainty and can produce misleading results
- Long enough duration: most consumer purchases involve consideration periods, so short tests understate the effect
- A clean holdout: if your control group is exposed to other campaigns running at the same time, the comparison is contaminated
What does incrementality testing tell you that MMM does not?
Marketing Mix Modelling estimates incremental contribution using historical patterns. It is very useful for understanding your full media mix over time. But it is an estimate based on correlations, not a controlled experiment.
A well-designed incrementality test gives you a more direct causal answer for the specific campaign being tested. The two methods work best together: MMM gives you the big picture, and incrementality experiments validate and calibrate the model.
When incrementality test results are fed back into an MMM as calibration data, the model becomes significantly more accurate. This is one of the key advances in modern marketing measurement.
When should you run an incrementality test?
- When a channel's platform-reported ROAS seems implausibly high
- Before significantly scaling up spend in a new channel
- When you are about to make a major budget reallocation decision
- To validate what your MMM is telling you about a specific channel
- When leadership is questioning whether a channel is really working
The honest limitation
Incrementality tests are not free. There is a cost to withholding marketing from your control group, because some of those people might have bought if they had seen the campaign. For most businesses this is a worthwhile investment, but it does need to be factored into the experiment design.
There is also a lag effect: the results of today's brand campaign may not show up in purchases for weeks or months. Tests designed to measure short-term direct response will understate the true impact of brand-building activity.
How long should an incrementality test run?
Most incrementality tests need to run for at least four weeks to account for variation and consideration periods. For higher-consideration purchases, six to eight weeks is better.
Can small brands run incrementality tests?
Yes, but the sample size requirements mean you need enough volume to detect a statistically meaningful difference between groups. Brands with very small audiences may find geo experiments easier to run than audience-level holdouts.
What is a good incremental lift result?
It depends heavily on the channel and campaign type. Branded search campaigns typically show low incremental lift. Upper-funnel brand campaigns often show higher lift but over longer time periods. The goal is not a specific number but an honest understanding of what you are actually driving.