Use a geo holdout when you need a clean test that is hard for platforms to contaminate. Use an audience holdout when geo splits are impractical and you trust the platform to enforce the exclusion properly.
Both methods aim to answer the same question: how much revenue does my advertising cause? But they work very differently, and each has situations where it performs better.
How audience holdouts work
An audience holdout randomly assigns a portion of your target users to a control group. These users are excluded from seeing your ads for the duration of the test. At the end, you compare conversion rates between the exposed and unexposed groups.
Facebook's Conversion Lift tool and Google's Brand Lift studies both use this approach. The platform manages the split and reports the results. The major limitation is that you are trusting the platform both to run the test and to report on its own performance.
How geo holdouts work
A geo holdout suppresses advertising in a set of matched geographic regions while keeping it live in other comparable regions. You measure the difference in business outcomes, such as store visits, website revenue, or total sales, between the two groups.
Because suppression is done at the campaign or channel level, not at the user level, you have direct control over what is and is not running. The platform cannot easily override or contaminate the holdout.
When to choose a geo holdout
- When you want a platform-independent test that the channel being measured does not control
- When the channel you are testing has weak or unreliable audience-level suppression capabilities
- When you have strong offline sales data that can be matched to geographic regions
- When you are testing TV, radio, out-of-home, or other channels where audience holdouts are technically impossible
When to choose an audience holdout
Audience holdouts work best for digital channels where platforms have reliable suppression tools and where you have enough users across your audience to create statistically balanced groups.
If your business is concentrated in a small number of geographic markets, splitting by region may leave you with groups that are too small and too variable to produce reliable results. An audience-level split across the whole market avoids this problem.
Geo holdouts tend to produce more conservative, and usually more accurate, lift estimates than platform-run audience holdouts. If the two methods give you very different answers, the geo test is usually closer to the truth.
The hybrid approach
The most rigorous approach is to run a geo holdout as your primary test and use an audience holdout on the same platform as a secondary check. If both methods produce similar lift estimates, you have strong evidence. If they diverge significantly, investigate why before acting on either.
Running both is more expensive in time and resource, but it is the best way to catch systematic bias in either method.
Does a geo holdout work for e-commerce if I sell nationally?
Yes. You suppress advertising in a subset of regions and measure whether online purchases from those regions fall relative to your test regions. You need clean data linking purchases to geographic location, which most e-commerce platforms provide as standard.
Can geo holdouts be contaminated?
The main contamination risk is spillover: people in a holdout region visiting a treatment region and seeing ads there, or word-of-mouth from treated customers influencing holdout customers. This is most common in small, tightly connected geographic areas. Larger geographic holdouts, such as entire regions or states, suffer less from spillover.
What if I only have a few geographic markets?
With only three to five markets available, geo holdouts are very difficult to make statistically rigorous. The variance between individual markets is too high. In this situation, an audience-level holdout or a longer-duration time-based comparison is usually more practical.