Causal inference lab · A/B test vs observational
A/B test vs observational data: why do they disagree?
A randomised A/B test and an observational comparison can give very different answers for the same campaign. In a test, chance decides who receives the campaign, so the two groups are alike. In observational data someone chose the recipients, usually the customers most likely to buy anyway, and that choice is counted as campaign effect. This demo measures one synthetic campaign both ways.
These demos are illustrative. Every customer, account, region and order is synthetic, generated and analysed in your browser, and every number shown is computed from that synthetic data. They show general principles of causal analysis, not the results of any real campaign, product or project.
Nothing is uploaded. No customer data is used.
Experiments / selection bias
One campaign. Two measurements.
What this shows: when a campaign is sent to the customers most likely to buy, comparing recipients with everyone else reports a lift several times the true one. A coin-flip test on the same customers does not.
Set the targeting
At 0% the campaign reaches customers at random in both measurements. Higher settings send it to customers who were already more likely to buy. Try a true lift of zero.
Estimated effect on purchase rate
50,000 synthetic customersWho received the campaign in the observational data
Amber: observational targetingBlue dashed: randomised test, 50% in every band
Synthetic data · illustrative · not results from a real business
Campaign, promotion and feature reports built on recipients versus non-recipients, or adopters versus non-adopters. The better the targeting, the larger the apparent lift, whether or not the campaign changed anyone’s behaviour.
What this experiment models
50,000 synthetic customers with a prior propensity score; the chance of buying without the campaign is a logistic function of it. The campaign adds the chosen lift to every customer’s purchase probability. The randomised test assigns the campaign by coin flip. The observational version sends it with a probability that rises with prior propensity; the slider sets how steeply. The interval is a normal-approximation 95% interval for a difference of two proportions. The adjusted estimate compares recipients and non-recipients within ten equal-sized propensity bands and averages by band size, skipping any band with fewer than 30 customers on either side. Fixed seed: the same settings always give the same numbers.
The principle
How it works
What the demo shows
50,000 synthetic customers each have a prior propensity to buy; without the campaign, 22.9% of them purchase. The campaign adds 3 percentage points to every customer’s purchase probability. In this sample that gives a true effect of +3.0 pts, known exactly because the simulation holds each customer’s outcome with and without the campaign.
The same customers are measured twice. In the randomised test a coin flip decides who receives the campaign, and the estimate is +2.8 pts, with a 95% interval from +2.1 pts to +3.6 pts. In the observational version the campaign goes preferentially to customers with high prior propensity: 93% of the top tenth receive it and 7% of the bottom tenth. Recipients buy at 33.5% and non-recipients at 15.2%, an apparent effect of +18.3 pts, about 6.1 times the true one.
The difference is selection bias. Set the true lift to zero and the observational comparison still reports a large positive effect, while the randomised test reports roughly none. Comparing customers within ten bands of prior propensity brings the observational estimate down to +3.6 pts. That works here because the one variable that drove targeting is recorded, and every band contains both recipients and non-recipients.
Why it matters for a decision
A report built on customers who received the campaign against customers who did not rewards targeting, not persuasion. A campaign aimed at likely buyers always looks effective, and budget then moves towards customers who would have bought anyway. A holdout costs a little reach and tells you what the campaign added.
What to check in your own data
- Who decided who got the campaign, and on what basis? A propensity model, a segment rule and a sales team’s judgement are each a source of selection.
- Can you hold out a random group? Even a small randomised holdout gives an unbiased benchmark for the observational numbers.
- If you must use observational data, is every driver of targeting recorded, and is there overlap? Where a band of customers is almost always or almost never targeted there is nothing to compare; the demo leaves such bands out of the adjusted estimate.
- Does the estimate pass a placebo test? Measure the effect on an outcome the campaign could not have changed, such as purchases in the month before it.
For how experiments and observational designs are chosen in a project, see causal inference consulting; for background, the causal inference guide.
FAQ
Common questions
Why does observational data overestimate campaign lift?
Because campaigns are usually sent to customers who were more likely to buy in the first place. The comparison with non-recipients then mixes the campaign’s effect with the difference that already existed between the groups. That difference is selection bias.
Can I measure incrementality without an A/B test?
Sometimes. If the factors that determined targeting are recorded, and both targeted and untargeted customers exist at each level of them, methods such as stratification, matching or propensity weighting can remove the bias. The result rests on the assumption that nothing unrecorded drove targeting, which should be stated and stress-tested.
How large should a holdout group be?
It depends on the baseline rate and the smallest effect worth detecting. In this demo, with about 25,000 customers in each arm and a baseline purchase rate near 22.9%, the 95% interval reaches about 0.8 percentage points either side of the estimate. A smaller holdout or a smaller effect needs a power calculation before launch.
Is a randomised test always the better measurement?
For the average effect of the action, it is the most direct estimate. It can still mislead if it is stopped early on a good day, if the two groups influence each other, or if the customers in the test differ from the ones you will act on.
Causal inference lab
More demos
All five demos on one page: the causal inference labCausal inference consulting →Causal inference guide →