Validating a marketing mix model means checking that it accurately reflects how your marketing actually works, not just that it fits the historical data. Good validation combines statistical tests, business sense-checks, and comparison against independent experiments.
A model can fit historical data well and still be wrong. It might be attributing sales to the wrong channels, or getting the direction right for the wrong reasons. Validation is the process of stress-testing the model before you use it to make budget decisions.
Statistical fit: necessary but not sufficient
The first check is model fit. R-squared (the proportion of sales variation the model explains) should typically be above 0.8 for a well-specified model. Mean absolute percentage error (MAPE), which measures how far off the model's week-by-week predictions are from actual sales, should ideally be below 10 percent.
These statistics tell you the model fits the past. They do not tell you whether it will perform well on future data or whether the channel coefficients are correctly attributed. You need additional checks for that.
Out-of-sample testing
Out-of-sample testing holds back a portion of your data (typically the most recent 3 to 6 months) and checks how accurately the model predicts sales in that period. If the model was built on 2 years of data but only tested on the same 2 years, it is easy to make it look accurate by overfitting (which means tailoring the model so closely to the training data that it loses predictive power on new data).
- Hold out the most recent 10 to 20 percent of your data for testing
- Build the model on the remaining data without touching the held-out period
- Compare model predictions to actual sales in the held-out period
- A MAPE below 15 percent in the held-out period is generally acceptable
Business sense-checks
Review each channel's coefficient against your expectations. If the model says email marketing is your highest-ROI channel by a large margin, but your email list has 2,000 subscribers and generates very little revenue, something is wrong. Counterintuitive results should trigger investigation, not acceptance.
The baseline sense-check is one of the most important. Ask the model: what would sales be in a period with no advertising? If that figure is implausibly high or low given what you know about your brand, challenge the modellers before accepting any of the channel ROI numbers.
Calibration against experiments
The gold standard for validation is comparing model-estimated channel contributions against results from controlled experiments. A geo holdout test (where you pause a channel in some regions but not others) gives you an independent read on that channel's contribution. If the model says TV drives 15 percent of sales but a geo test shows 22 percent, you have a calibration problem to investigate.
Most brands cannot run geo tests for every channel simultaneously. Prioritise experiments for your highest-spend channels, as these have the greatest impact on budget decisions.
Stability across model specifications
A reliable result should be robust across slightly different model specifications. If the TV ROI estimate changes dramatically when you adjust the adstock decay rate by a small amount, that instability suggests the data is not strong enough to pin down TV's effect precisely. Your modeller should run a sensitivity analysis (testing how results change under different assumptions) and show you the results.
Who should validate the model: us or the modelling partner?
Both. The modelling partner runs the statistical validation. Your team runs the business sense-checks. Ideally, an independent third party reviews the methodology. Asking your modelling partner to validate their own work is a conflict of interest you should be aware of.
What should we do if the model fails validation?
Identify the specific failure. If it is a poor fit, more or better data may be needed. If it is a business sense issue, investigate whether a control variable is missing. If experimental results disagree significantly with model estimates, recalibrate before using the model for budget decisions.
How long does validation take?
Statistical validation is usually completed as part of the model build. Business sense-checks and comparison against experimental results can add one to two weeks, depending on whether experiments are available and whether any calibration iterations are needed.