A model that predicts well can still be wrong about what to do

Rich Caruana and colleagues opened their 2015 paper on interpretable models for health care with a cautionary tale from a project to predict which pneumonia patients were likely to die, so that those at low risk might be treated as outpatients. One of the models, built from rules, produced a striking one: patients with a history of asthma had a lower risk of dying from pneumonia than the general population. A clinician reading that rule would be baffled, because asthma makes pneumonia more dangerous.

The rule reflected the data faithfully. Asthmatic patients with pneumonia were usually admitted directly to the intensive care unit, and the aggressive treatment they received worked so well that their outcomes were better than average. The model had learned a true pattern about the past. If it had been used to decide who could go home, it would have sent asthmatics out of the hospital first, removing the very care that made them look safe. The neural network that learned the same pattern was reportedly accurate on held-out data and was not deployed.

The story is a compact lesson in a distinction that business teams keep missing. Prediction answers the question of what will happen, given how things have been run so far. A decision asks what will happen if we do something different. The first can be answered from historical data, if the world stays the same. The second needs a claim about cause, and historical data answers it only under conditions that are rarely met. A model can be excellent at the first and misleading about the second, as the asthma rule shows, and the usual accuracy metrics will not say which situation you are in.

Simpson's paradox is the mechanism

The classic demonstration involves a comparison of two treatments for kidney stones, reported by C. R. Charig and colleagues in 1986 and reanalyzed by Steven Julious and Mark Mullee in 1994. Open surgery had a success rate of 78% (273 of 350 patients). Percutaneous nephrolithotomy, a less invasive procedure, had a success rate of 83% (289 of 350). On the face of it, the less invasive procedure is better.

	Small stones	Large stones	All patients
Open surgery	93% (81 of 87)	73% (192 of 263)	78% (273 of 350)
Percutaneous nephrolithotomy	87% (234 of 270)	69% (55 of 80)	83% (289 of 350)

Within each stone size, open surgery did better. The reversal arises because stone size drove both the choice of treatment and the outcome: surgeons assigned open surgery to the larger, harder cases, and large stones have lower success rates under any treatment. The overall comparison mixes a treatment effect with a difference in case mix. Whether the aggregate or the stratified figures are the right ones depends on the causal story, which the numbers alone do not tell you. Here stone size is a confounder that precedes the choice of treatment, so the stratified comparison is the one to trust.

The paradox matters outside medicine because every business action has the same structure. Customers who receive a discount, a call from a sales representative or a retention offer are not a random sample. Someone, or something, chose them, usually on the basis of the same variables that predict the outcome. A comparison between treated and untreated customers measures that selection along with whatever the treatment did.

The ladder of causal questions

Judea Pearl's picture, in "The Book of Why," arranges questions on a ladder of three rungs. On the first, seeing, one asks what a variable tells us about another: customers who open emails buy more. On the second, doing, one asks what happens if we intervene: will customers buy more if we send them more emails? The third, imagining, concerns counterfactuals about a particular case. Machine learning as practiced mostly lives on the first rung, where supervised learning is excellent. Most business decisions sit on the second.

The rung matters because data collected without intervention cannot, in general, answer second-rung questions. No amount of additional rows or model capacity changes that, and no hyperparameter will repair it. The identification has to come from the design, either an experiment or an argument about what drove the assignment of treatment.

What the churn model is actually for

Take a subscription business that builds a model to predict which customers will cancel in the next month, and uses it to send the highest-risk customers a retention offer. The model might be very accurate, with an AUC of 0.90. The business question is a different one: for which customers does the offer change the outcome? Marketing analysts speak of four groups. Persuadables stay because of the offer and would have left without it. Sure things stay either way, and the offer is wasted on them. Lost causes leave either way. Sleeping dogs are the ones who would have stayed, and leave because the offer or the contact reminded them to cancel.

A churn-probability ranking says nothing about which group a customer is in, and the customers at the very top of the ranking are disproportionately lost causes. An illustration, with made-up but plausible numbers, shows the cost. Each offer costs $15, and a retained customer is worth $300. Segment A has a 50% churn rate without the offer, and the offer reduces it by 2 points. Segment B has a 20% churn rate, and the offer reduces it by 8 points. The expected value of an offer is the lift times $300, minus $15. For segment A that is 0.02 × 300 − 15 = −$9 per customer. For segment B it is 0.08 × 300 − 15 = +$9. A campaign aimed at the highest predicted risk goes after segment A, and across 10,000 customers it loses $90,000. A campaign aimed at the highest predicted lift goes after segment B, and over 10,000 customers earns $90,000. The decision of whom to treat is worth $180,000, and the churn model cannot make it.

Observational adjustment is not enough

A natural response is to adjust for everything observable and call the difference a treatment effect. The evidence on how well that works is sobering, and one of the cleanest tests comes from advertising. Brett Gordon, Florian Zettelmeyer and colleagues compared the results of randomized experiments at Facebook with what common observational methods would have concluded from the same data. They used 15 experiments covering about 500 million user-experiment observations and 1.6 billion ad impressions. Even with extensive demographic and behavioral variables, observational methods often failed to reproduce the experimental result. In the working-paper summary of the study, matching and regression approaches overstated the effect of advertising, and in half of the studies the estimated lift in purchases was off by a factor of three.

The explanation is selection on what the analyst cannot see. Ads are shown to people who are already inclined to buy, and those inclinations are not fully recorded in any dataset. Adjusting for the variables you have cannot remove the part you lack. The result applies well beyond advertising to any business where treatment is targeted, which describes most business interventions.

There is a second obstacle that a larger dataset does not remove, and it is built by the company's own policy. If every customer above a score threshold has always received the offer, and no one below it has, then no high-scoring customer in the history went untreated. Causal methods need overlap, meaning customers with similar characteristics in both groups, and a deterministic targeting rule destroys it. A model can still be fitted and will produce a number for each customer, but that number rests on extrapolation, with nothing in the data to check it against. The cheapest cure is a little randomness in the rule, such as treating a small random share of customers below the threshold and withholding from a small random share above it, which gives the future analysis something to compare.

Building the decision model correctly

Treat every model-driven action as an intervention and evaluate it as one.

The first step is to keep a randomized control group. Hold out a small share of eligible customers, say 5% to 10%, who receive no model-driven treatment regardless of their score. Their outcomes are the counterfactual against which the program's effect is measured. Without them, the best that can be said is how treated customers fared, which says nothing about what the treatment did.

The second is to randomize treatment within score bands, at least for a period, so the data contain customers of the same predicted risk who were and were not treated. That data allows an uplift model to be estimated. Several methods exist: fitting separate models for treated and control groups and subtracting their predictions, the causal forests of Stefan Wager and Susan Athey (2018), which estimate heterogeneous effects with valid confidence intervals, and the double machine learning approach of Victor Chernozhukov and colleagues (2018), which uses flexible models for nuisance quantities while preserving statistical guarantees for the effect.

A caution about the arithmetic of heterogeneity. Estimating an average effect is far cheaper than estimating the effect in each segment, and the targeting decision depends on the second. Suppose churn is 20% without the offer and 12% with it in some segment, an effect of 8 points. With 2,000 customers per arm, the standard error of the difference is about 1.2 points, and the 95% interval runs from roughly 5.7 to 10.3 points. With 500 per arm, the standard error is about 2.3 points and the interval runs from 3.5 to 12.5. A segment ranking built from the second estimate will confuse many segments with one another. This is why uplift programs usually need several months of randomized data before the targeting rule is trusted, and why the rule should be rechecked against a fresh holdout.

The third is to plan the experiment's size, because the effects are small and noise is large. Detecting a rise in conversion from 5% to 6% with 80% power at the 5% level needs about 8,100 customers in each arm. Detecting a rise from 5% to 5.5% needs about 31,000 per arm, and detecting 2.0% to 2.2% needs more than 80,000. The smaller the lift that justifies the program, the larger the experiment. A team that skips this arithmetic can run an experiment that is too small to tell a success from a failure, and report the noise as a finding.

The fourth is to use quasi-experiments when randomization is not possible. A staggered rollout across regions allows a difference-in-differences comparison. A score cutoff, below which customers are offered something and above which they are not, supports a regression discontinuity design. A supply glitch that randomly delayed the treatment for some customers can serve as an instrument. Each of these requires an argument for why it approximates randomization, and the argument should be written down and reviewed.

When prediction is exactly what you need

None of this makes predictive models less useful. When nobody intervenes on the outcome, a prediction is the right tool. Forecasting demand for capacity planning, estimating claims for reserves, projecting cash flow and pricing the risk in a portfolio are all cases where the model describes what will happen and the business adapts around it. A prediction is also enough when the action is known to work and the only question is where to deploy limited capacity, as when a fraud team has more alerts than analysts and the intervention, blocking a transaction, is not in dispute. The error to avoid is narrower: using a ranking of who is likely to experience an outcome as if it ranked who would benefit from an intervention. The first is a statement about the past, and the second is a claim about cause.

Where this shows up with language models

The same distinction operates in AI-assisted workflows. A model that predicts which support tickets will escalate does not tell you whether routing them to a senior agent reduces escalations. A system that recommends which sales leads to call does not show whether calling them changes anything. A copilot that proposes actions to employees changes the data it will later be trained on, which makes the observational record a record of the system's own influence. The remedy is the same: a holdout group, randomization in the rollout, and a comparison of outcomes between treated and untreated cases.

Start with the decision

Before building any model, write one sentence naming the action, the people to whom it will be applied, and the outcome it is meant to change. Then write a second naming the comparison group: the people who will not receive the action, and how they will be chosen. If the second sentence cannot be written, the project is a prediction project, and its output should be used to forecast, not to assign treatment. A model that cannot be tied to a defensible comparison group will go on producing numbers that are accurate and unhelpful, which was the situation of the pneumonia model that looked at asthma and saw safety.

Sources

1. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N. (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD 2015.
2. Charig, C. R., Webb, D. R., Payne, S. R., Wickham, J. E. A. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. British Medical Journal 292.
3. Julious, S. A., Mullee, M. A. (1994). Confounding and Simpson's paradox. British Medical Journal 309.
4. Pearl, J., Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
5. Gordon, B. R., Zettelmeyer, F., Bhargava, N., Chapsky, D. (2019). A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook. Marketing Science 38(2).
6. Wager, S., Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association 113(523).
7. Chernozhukov, V., Chetverikov, D., Demirer, M., et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21(1).