Uplift Modeling: Estimating Who Responds to an Intervention

Uplift modeling estimates the incremental effect of an action on each individual: the difference between their probability of an outcome if treated and their probability if not treated. It answers "whose behaviour will this offer change?", which is a different question from "who is likely to buy?" or "who is likely to churn?". It is trained on data from a randomized experiment, is built with meta-learners or tree-based methods, and is evaluated with Qini and uplift curves because the individual effect is never observed directly.

The same quantity is called the conditional average treatment effect (CATE) in the causal inference literature. It sits on the second rung of the ladder of causality: it concerns the effect of an intervention, conditional on a customer's characteristics.

How is uplift modeling different from response modeling?

A response model estimates P(Y = 1 | X) among people who received a campaign, and targets those with the highest score. An uplift model estimates P(Y = 1 | X, treated) − P(Y = 1 | X, not treated). The two can disagree sharply. Loyal customers have high response scores and would have bought anyway, so spending on them produces little additional revenue. Targeting by response score tends to select exactly that group. The underlying difficulty is what Holland called the fundamental problem of causal inference: each person is observed either treated or untreated, never both, so there is no individual-level label for uplift to train on.

What are the four customer types in uplift modeling?

Uplift modeling is usually explained with a two-by-two classification of how a person would behave with and without the treatment:

Sleeping dogs, also called the do-not-disturb group, are the reason uplift can be negative: a retention call can remind a customer to cancel. Nobody can be assigned to a type with certainty, because only one of the two columns is ever observed for a given person. An uplift model estimates, for people with similar characteristics, the balance of these types.

What are the two-model and meta-learner approaches?

Meta-learners are recipes that turn ordinary supervised learners (gradient-boosted trees, regularized regression, neural networks) into estimators of treatment effects. The S-, T- and X-learner names follow the terminology of Künzel and colleagues.

• Two-model approach (T-learner). Fit one outcome model on the treated group and another on the control group; the predicted uplift is the difference between their predictions. It is simple and uses any learner. Its weakness is that each model is tuned to predict the outcome, not the difference, so two individually reasonable models can produce a noisy difference, especially when the effect is small relative to the baseline rate.
• S-learner. Fit a single model on all the data with the treatment indicator as one more feature; predict each person twice, with the indicator set to treated and to untreated, and subtract. It shares information between groups. Its weakness is that regularization can shrink the role of the treatment indicator, biasing estimated effects toward zero.
• X-learner. Stage one fits the two outcome models as in the T-learner. Stage two imputes an individual effect for each person: for treated people, the observed outcome minus the control model's prediction; for control people, the treated model's prediction minus the observed outcome. Two further models are fitted to these imputed effects and combined with a weight, typically the propensity score. It is designed for cases where one group is much larger than the other.

Other established methods include the transformed outcome (class transformation) approach, which recodes the target so that a single classifier estimates uplift; uplift trees, whose split criterion maximizes the difference in treatment effect between branches; causal forests, associated with Wager and Athey; and the R-learner and doubly robust learners, which combine outcome and propensity models. No method is best on all data, and the choice should be made by evaluation on held-out randomized data.

How are uplift models evaluated with Qini and uplift curves?

Standard metrics such as AUC or log-loss compare predictions with observed labels, and uplift has no observed label. Evaluation therefore works at the level of groups, using a held-out part of the randomized experiment:

• Uplift by decile. Sort the holdout by predicted uplift, split it into bins, and in each bin compare the outcome rate of treated and control individuals. A good model shows the largest observed difference in the top bin and a steady decline, possibly below zero, toward the bottom.
• Uplift curve. Plot the cumulative incremental outcome against the fraction of the population targeted, in descending order of predicted uplift. Random targeting gives a straight line from the origin to the overall effect.
• Qini curve. A variant introduced by Radcliffe that plots cumulative incremental positive outcomes, scaling the control group's count to the size of the treated group at each point. The Qini coefficient is the area between the model's curve and the random-targeting line.

These curves are estimated from differences between two samples and are noisy. Report them with confidence intervals, for example from bootstrap resampling, and compare models on the same holdout.

What data does uplift modeling need?

The reliable basis is a randomized holdout: within the population eligible for the action, some individuals are assigned to receive it and a randomly chosen control group is withheld. The requirements are:

• Random assignment across the whole eligible population, including people the current rules would never target, so that the model can learn where the effect is low or negative.
• Features measured before treatment. Anything recorded after assignment can be affected by the treatment and must be excluded.
• The same outcome definition and observation window in both groups, and an outcome that reflects the business goal.
• Enough volume. Estimating how an effect varies between individuals needs much more data than detecting an average effect.
• A continuing control group after deployment, so that the model can be monitored and retrained on unbiased data.

Uplift can be estimated from observational data with propensity adjustment, but the result then depends on the assumption that all drivers of past targeting were measured. Past campaigns were normally targeted by rules or scores, which is the situation in which that assumption is weakest. See the causal inference overview for the design options.

What are the common failure modes of uplift modeling?

• Training on targeted data. If treatment followed a rule, the model learns the rule's selection effect and reports it as uplift.
• Weak signal. Individual effects are small compared with outcome noise, so models overfit easily and rankings are unstable between samples.
• Evaluating with the wrong metric. Selecting a model by AUC on the outcome rewards finding sure things, not persuadables.
• Post-treatment leakage. Features that are consequences of treatment produce uplift estimates that cannot be reproduced at decision time.
• Outcome and cost mismatch. Uplift in clicks or redemptions is different from uplift in margin. An offer that costs more than the incremental value it creates is unprofitable even with positive uplift.
• Non-compliance and interference. Not everyone assigned to treatment receives it, and one customer's treatment can affect another's outcome; both change what the estimate means.

Where is uplift modeling used?

Typical applications are retention offers, cross-sell and promotion targeting, credit-line and pricing actions, and outreach programmes with limited capacity. In churn prediction, uplift is the step that moves from ranking customers by risk to ranking them by how much an intervention helps. The same estimation problem appears in medicine as heterogeneity of treatment effect, discussed in causal inference in healthcare and medical AI.

Frequently asked questions

What is uplift modeling?

Uplift modeling estimates, for each individual, the difference between the probability of an outcome if they receive a treatment and the probability if they do not. It is used to target an action, such as a retention offer, at the people whose behaviour it is most likely to change.

How is uplift modeling different from churn or propensity modeling?

A churn or propensity model predicts how likely an outcome is. An uplift model predicts how much an action changes that likelihood. The customers most likely to buy or to churn are often not the ones most affected by an offer, so the two models can rank customers very differently.

What data do you need for uplift modeling?

You need data from a randomized experiment in which some eligible individuals received the treatment and a randomly chosen holdout did not, with features recorded before treatment and the outcome measured the same way in both groups. Because individual effects are small and noisy, the experiment usually has to be considerably larger than one sized only to detect an average effect.

What is the difference between the S-learner, T-learner and X-learner?

The S-learner fits a single outcome model with the treatment indicator as a feature. The T-learner, also called the two-model approach, fits separate outcome models for the treated and control groups and subtracts their predictions. The X-learner adds a second stage that imputes individual effects using the opposite group's model and then models those imputed effects, which helps when the two groups differ greatly in size.

What is a Qini curve?

A Qini curve plots the cumulative incremental outcomes obtained when individuals are targeted in descending order of predicted uplift, against the fraction of the population targeted. A model that ranks well rises above the straight line for random targeting, and the area between the two is summarized as the Qini coefficient.

Related in this wiki: Causal Inference in Business and Data Science, The Ladder of Causality and Causal Inference in Healthcare and Medical AI. For how MLAIA applies these methods, see causal inference consulting and uplift modeling. Try it: the interactive uplift modelling demo.

MLAIA consults on causal inference and experiment design for business decisions — from A/B testing programs to uplift models. See our BI & predictive analytics expertise or contact us. Related reading: Transforming Sales with AI and Expertise Without Boundaries.