Most incrementality tests should run for three to six weeks. Short tests are dominated by noise. Very long tests risk market conditions shifting mid-experiment, which makes your comparison group less valid.
Duration is one of the most common sources of bad incrementality results. A two-week test during a seasonal spike looks very different from a two-week test during a normal period. A twelve-week test runs the risk of a competitor promotion, a pricing change, or a news event distorting the results.
Why three weeks is a common minimum
Weekly sales cycles create natural variation in every business. Monday and Saturday behave differently. The first week of the month and the last week often behave differently too. Running a test for at least three weeks means you capture multiple full weekly cycles and average out these patterns.
Tests shorter than two weeks often cannot distinguish a real advertising effect from a normal Monday-to-Monday swing. The result looks real but is driven by timing, not causality.
How your purchase cycle affects duration
If your customers typically take two weeks from first awareness to purchase, a three-week test gives them one week of settling time after the test concludes before you analyse results. That may not be enough.
For products with longer consideration cycles, such as financial services, home improvement, or B2B software, you should extend the test to at least six to eight weeks and allow additional time after the test ends before pulling the data.
Things that shorten useful test duration
- High conversion volume: more conversions per day mean you reach statistical significance faster
- Large expected lift: a campaign expected to drive 20 percent lift is detectable much faster than one expected to drive 5 percent lift
- Low baseline variance: if your sales are very stable week on week, smaller samples can reliably detect a signal
Things that lengthen required test duration
Low conversion volume, small expected lift, and high natural sales variance all force you to run longer. A brand generating 200 conversions per month needs far more weeks of data than one generating 20,000.
If you calculate that you need 20 weeks to reach adequate power, that is a sign the test may not be appropriate for your current business size. Consider an MMM approach instead.
Do not stop a test early because the results are looking good. Peeking at results before the planned end date inflates your false positive rate. Decide on the duration before you start and commit to it.
Avoiding seasonal contamination
Schedule tests away from known seasonal peaks unless the test is specifically designed to measure performance during those peaks. Running a test that straddles Black Friday will produce results that are hard to separate from seasonal demand.
If you must test during a promotional period, ensure both your test and holdout groups are equally exposed to other promotional activity. Any difference in promotional treatment between groups will contaminate the result.
Should I include a pre-test period in my analysis?
Yes. Analysing a pre-test period of equivalent length helps you verify that your test and holdout groups were behaving similarly before the experiment started. It also provides a baseline to normalise the comparison and identify any pre-existing differences between groups.
What happens if I need to stop the test early?
If you must stop early due to a business reason, treat the results with extreme caution. Depending on how early you stopped, the data may be underpowered. Report the confidence intervals, not just the point estimate, and note that the test was not completed as designed.
Can I run multiple tests at the same time?
Running concurrent tests in overlapping audiences or regions creates interference that is very hard to separate. If you need to test multiple channels simultaneously, design the tests to use entirely separate, non-overlapping regions or audiences, and be aware that contamination is still a risk.