Causal inference lab · Simpson’s paradox
Simpson’s paradox: why does the effect reverse?
Simpson’s paradox is a comparison that points one way in the whole dataset and the other way inside every subgroup. It appears when a hidden variable, a confounder, drives both who gets an action and the outcome. In this demo a success-team call raises renewal in both customer segments, yet called accounts renew less often overall, because calls go mostly to accounts that were already at risk.
These demos are illustrative. Every customer, account, region and order is synthetic, generated and analysed in your browser, and every number shown is computed from that synthetic data. They show general principles of causal analysis, not the results of any real campaign, product or project.
Nothing is uploaded. No customer data is used.
Confounding / Simpson’s paradox
Helps in every segment. Looks harmful overall.
What this shows: a call that raises renewal for at-risk accounts and for healthy accounts can still show a lower renewal rate among called accounts, when at-risk accounts are the ones that get called.
Compare the accounts
At 0% every account is equally likely to be called. At 100%, 90% of at-risk accounts and 10% of healthy accounts are called. The call’s true effect stays the same throughout.
Renewal rate: called vs not called
20,000 synthetic accountsBlue: calledWhite: not called
Synthetic data · illustrative · not results from a real business
Any action aimed at the cases that need it most: retention calls, discounts for price-sensitive customers, escalations for difficult tickets, maintenance for the oldest machines. A dashboard that compares treated with untreated then measures who was chosen as much as what the action did.
What this experiment models
20,000 synthetic accounts; half are at risk. Without a call, 80% of healthy and 40% of at-risk accounts renew; a call adds 8 percentage points in both segments. The slider sets how strongly calls depend on the segment: at 0% every account has a 50% chance of a call, at 100% it is 90% for at-risk and 10% for healthy accounts. The adjusted figure is the within-segment difference averaged over the two segments by size. Each account keeps the same random draws at every slider position, and the data is fixed by a seed, so the same settings always give the same numbers.
The principle
How it works
What the demo shows
The demo generates 20,000 synthetic accounts. Half are at risk of leaving, and that is the hidden variable. In the simulation a call from the success team raises the chance of renewal by 8 percentage points in both segments, so the true effect is +8.0 pts. At-risk accounts renew far less often than healthy ones, called or not.
At the default setting 80% of at-risk accounts and 20% of healthy accounts get a call. Compare all accounts together and the called group renews at 56.1% against 71.9% for the rest: a difference of −15.8 pts, which reads as the call doing harm. Compare within each segment and the sign reverses: +7.6 pts among at-risk accounts and +7.1 pts among healthy ones. Averaging the two segments by size gives +7.3 pts, close to the true +8.0 pts; the remaining gap is sampling noise.
Neither calculation is wrong. The pooled comparison answers a different question: 79% of called accounts are at risk, against 20% of the others, so it mostly compares at-risk accounts with healthy ones. Move the slider to 0% and calls no longer depend on the segment; the pooled difference returns to roughly the true effect.
Why it matters for a decision
The pooled number is what a dashboard shows by default: renewal for accounts that were contacted against accounts that were not. Read at face value, it says to stop calling. The same pattern appears whenever an action is aimed at the cases that need it most, and the action then looks useless or harmful because of who receives it.
What to check in your own data
- How was the action assigned? Write down the rule or the judgement people used. Any input to that decision that also affects the outcome is a candidate confounder.
- Is that variable recorded? If it is, compare within its levels and see whether the estimate moves. If it lived only in someone’s head, no adjustment can recover it.
- Does the mix differ between the groups? A table of treated and untreated units by segment often shows the problem before any modelling.
- Can part of the action be randomised? Holding out a random share of eligible accounts gives a comparison that no hidden variable can distort.
Stratifying is not always right. Adjusting for something the action itself changes removes part of the effect. The causal structure, not the data, says which variables to adjust for. For how this is handled in a project, see causal inference consulting; for background, the causal inference guide.
FAQ
Common questions
What is Simpson’s paradox in simple terms?
It is a reversal: a trend that holds in every subgroup of the data points the other way when the subgroups are combined. It arises when the groups being compared contain different mixes of subgroups, and the subgroups have different baseline outcomes.
Which is correct, the pooled result or the stratified result?
It depends on the causal structure. When the subgroup variable influences both the action and the outcome, as account risk does in this demo, the within-subgroup comparison reflects the action’s effect. When the variable is a consequence of the action, adjusting for it is the mistake.
What is a confounder?
A confounder is a variable that affects both whether a unit receives an action and the outcome being measured. It creates an association between action and outcome that the action did not cause.
How do I detect confounding in my own data?
Data alone cannot prove its absence. Start from how the action was assigned, list what influenced that decision, check whether those factors differ between the treated and untreated groups, and see whether the estimate changes when you adjust for them. A randomised holdout is the one check that does not depend on having measured everything.
Causal inference lab
More demos
All five demos on one page: the causal inference labCausal inference consulting →Causal inference guide →