Experiments first
When you can randomize who gets the action, do. A/B tests and randomized holdouts give the most reliable estimate. We design the comparison, sample size and analysis plan before launch.
Prediction & Causal Analysis · Causal inference
Did it work?
How much, and for whom?
Causal analysis for pricing, retention, marketing and operations decisions. We estimate what an action changed, compared with what would have happened without it, and state the assumptions behind each estimate.
A specialist practice of MLAIA Data ScienceLed by Dr. Yochai Edlitz · Ph.D., Weizmann Institute
Prediction vs cause
Prediction says what will happen.
Causal analysis says what changes it.
A model can predict sales well and still be wrong about what a price rise will do. Causal questions need a comparison: the outcome with the action against the outcome without it. Only one of them is ever observed.
Our causal inference guide introduces the ideas, and the ladder of causation shows how the questions differ. For a full textbook treatment, see Hernán and Robins, Causal Inference: What If.
Quasi-experiments
When you can’t randomize, a rollout by region, date or threshold can still give a fair comparison. Difference-in-differences, synthetic control and regression discontinuity use that structure.
Adjusting historical data
With only historical data, we adjust for the factors that drove both the decision and the outcome. That works only if the important ones are measured, so we test how sensitive the answer is.
Methods
The design comes first.
The estimator follows.
Each method rests on an assumption. We choose the one your data and rollout can support, and say what it assumes.
Experiment design
Randomization unit, sample size and power, guardrail metrics and an analysis plan fixed in advance. We check balance, avoid stopping early on a lucky result, and use pre-period data to reduce noise where it helps.
Difference-in-differences
Compare the change in a treated group with the change in a comparison group. It assumes both would have moved in parallel without the action. We check trends before the change. See Cunningham’s Causal Inference: The Mixtape.
Synthetic control
For one treated market or region, build a weighted mix of untreated ones that tracks it before the change. The gap afterwards estimates the effect (Abadie, Diamond & Hainmueller, 2010).
Propensity methods
Model the chance that each unit got the action, then match or weight so groups are comparable on measured factors (Rosenbaum & Rubin, 1983). We check overlap and balance, and often pair it with an outcome model.
Uplift modeling
Estimate how the effect varies across customers, to target an action where it helps. Meta-learners build this on standard ML models (Künzel et al., 2019). We evaluate with uplift or Qini curves on randomized data.
Assumptions & sensitivity
Every estimate rests on assumptions the data can’t fully check. We draw them as a causal diagram, test what we can, and report how strong a hidden confounder would need to be to change the answer (Cinelli & Hazlett; E-values).
Methods we useA/B tests & holdoutsDifference-in-differencesSynthetic controlPropensity weighting & matchingUplift meta-learnersSensitivity analysis
What we measure
An effect, a range and the assumptions.
Reported together.
A causal result is more than one number. We report these with every estimate.
Effect with uncertainty
The estimated change in the outcome, in business units, with an interval. A wide interval is a result too: it tells you whether to collect more data before deciding.
Effect for whom
Average effects by segment, and uplift curves where targeting is the goal. We say where the data are too thin to tell.
Checks on the design
Trends before the change, balance and overlap for propensity methods, and placebo tests on dates or groups that weren’t treated.
Robustness
Sensitivity to hidden confounding and to reasonable changes in the analysis. If the answer flips easily, we say so.
Not every causal question can be answered from the data you have. The first step tells you whether yours can, and what experiment or extra data would close the gap.
How an engagement runs
From a business question to a measured effect.
In four steps.
Each step ends with evidence you can check before the next one starts.
Frame the question
We define the action, the outcome, the comparison and the decision the answer will inform. We draw the causal assumptions with your team.
Choose the design
We check the data and how the action was assigned, then pick an experiment or a quasi-experimental design it can support.
Estimate & stress-test
We estimate the effect with uncertainty, run placebo and sensitivity checks, and show results by segment where the data allow.
Decide & keep measuring
We turn the result into a decision rule, such as who gets an offer, and keep a holdout so the effect stays measured.
Before we start
Good questions. Straight answers.
Can’t we just compare customers who got the offer with those who didn’t?
Usually not. Customers are chosen for offers for a reason, and that reason often affects the outcome too. The difference mixes the offer’s effect with who was chosen. A randomized holdout, or a design that accounts for how offers were assigned, separates them.
We can’t run an A/B test. Can you still estimate the effect?
Often, if the action rolled out in a way that gives a fair comparison, such as by region or date. Difference-in-differences and synthetic control use that structure. The estimate rests on stronger assumptions than an experiment, and we state them.
What is uplift modeling?
Uplift modeling estimates how much an action changes the outcome for different customers, rather than who is likely to have the outcome. It’s used to target offers where they help. It works best when trained and evaluated on randomized data.
Isn’t a feature-importance chart enough?
No. Feature importance and SHAP values describe what a predictive model uses. They don’t show what happens if you change that factor. Causal claims need a design that compares outcomes with and without the action.
What do you need to start?
The decision you want to make, how the action was assigned in the past, and a description of the outcome data. Please don’t send confidential data.
Prediction & Causal Analysis · Causal Inference
Book a 30-minute
forecasting review.
Bring one decision whose effect you want to measure. In 30 minutes we’ll cover:
- What your data can realistically support
- Which design fits how the action was assigned
- Whether an experiment would give a clearer answer
Please don’t include confidential datasets, credentials or patient information.