Causal inference lab · Difference-in-differences
Difference-in-differences: what if parallel trends fails?
Difference-in-differences estimates the effect of a launch by comparing the change in the launch region with the change in a region that did not get it. It rests on one assumption: without the launch, both would have moved in parallel. This demo lets you break that assumption and watch the estimate absorb the difference in trends as if it were launch effect.
These demos are illustrative. Every customer, account, region and order is synthetic, generated and analysed in your browser, and every number shown is computed from that synthetic data. They show general principles of causal analysis, not the results of any real campaign, product or project.
Nothing is uploaded. No customer data is used.
Quasi-experiments / difference-in-differences
No control group. One assumption.
What this shows: difference-in-differences recovers a launch effect when the two regions were moving in parallel, and reports a wrong effect, with no warning of its own, when they were not.
Change the regions
At zero extra trend the regions move in parallel apart from noise. Any other value breaks the parallel-trends assumption. Try a true effect of zero.
Weekly orders in two regions
36 weeks · launch after week 24Blue: launch regionAmber: comparison regionWhite dashed: launch region without the launch, as the method assumes
Synthetic data · illustrative · not results from a real business
Launches, price changes and policy changes rolled out to one region, store group or customer tier without a randomised control. Regions are often chosen for a launch because they are growing, and that growth is then reported as the launch’s effect.
What this experiment models
Two synthetic regions, 36 weeks of orders, launch in one region after week 24. Both share a trend of +0.5 orders a week and independent week-to-week noise (standard deviation 3). The launch adds a constant number of orders a week. The extra trend applies to the launch region from week 1. Difference-in-differences is the change in the launch region’s mean (weeks 25–36 minus weeks 1–24) minus the same change in the comparison region. The placebo applies the same calculation to weeks 13–24 against weeks 1–12. The gap trend is the least-squares slope of the difference between the regions over weeks 1–24. The dashed line is the comparison region shifted by the average pre-launch gap. Fixed seed.
The principle
How it works
What the demo shows
Two synthetic regions report weekly orders for 36 weeks. A feature launches in one of them after week 24 and, by default, adds 12 orders a week. Both regions share an upward trend of half an order a week, plus week-to-week noise.
A before-and-after comparison in the launch region gives +20.5 orders a week, because it counts the shared trend as launch effect. Difference-in-differences subtracts the change in the comparison region over the same weeks and gives +11.1, against a true effect of +12.0. The remaining −0.9 is noise: the estimate averages only 12 post-launch weeks.
The extra-trend slider gives the launch region growth of its own from week 1. The comparison region then no longer shows what would have happened without the launch, and the estimate moves away from the truth in the direction of the extra trend. Set the launch effect to zero and the method still reports an effect.
Two diagnostics use only pre-launch weeks. The placebo estimate pretends the launch happened after week 12 and applies the same calculation to weeks 1 to 24; with parallel trends it should be near zero, and at the default it is −2.0. The gap trend is the weekly change in the difference between the regions before launch, −0.10 orders a week at the default. Both move away from zero when the trends diverge.
Why it matters for a decision
Difference-in-differences is the standard tool when a change reaches one region or group and nothing was randomised. It is easy to compute, which is also its risk: the arithmetic always returns a number, and the number carries no sign of whether the comparison was fair. A launch judged a success on a diverging trend gets rolled out everywhere.
What to check in your own data
- Plot both series for as many pre-launch periods as you have. Parallel lines before launch do not prove they would have stayed parallel; diverging lines show the assumption had already failed.
- Run a placebo on pre-launch dates, and on outcomes the launch could not have affected.
- Ask what else changed at the same time in one region only: a competitor, a campaign, a price.
- Report the estimate with its uncertainty. With few periods or few regions, noise alone can be as large as the effect.
- If the trends differ, consider a comparison built for the purpose, such as synthetic control or region-specific trends, and state what each assumes.
For how these designs are chosen and checked in a project, see causal inference consulting; for background, the causal inference guide.
FAQ
Common questions
What is the parallel trends assumption?
It is the assumption that, without the intervention, the treated group’s outcome would have changed by the same amount as the comparison group’s. The groups may differ in level. They must not differ in trend.
Can you test parallel trends?
Only partly. You can check whether the trends were parallel before the intervention, with plots, placebo dates and pre-period trend estimates. Whether they would have stayed parallel afterwards cannot be observed, so the assumption also needs an argument from knowledge of the business.
What happens if parallel trends is violated?
The estimate equals the true effect plus the difference in trends that builds up between the pre-launch and post-launch periods. In the demo the error grows in proportion to the extra trend you add.
How is difference-in-differences different from a before-and-after comparison?
A before-and-after comparison attributes every change over time to the intervention. Difference-in-differences removes the changes shared with a comparison group, such as seasonality or market growth. In the demo the before-and-after figure is +20.5 orders a week and the difference-in-differences estimate is +11.1, against a true effect of +12.0.
Causal inference lab
More demos
All five demos on one page: the causal inference labCausal inference consulting →Causal inference guide →