Data leakage is how a weak model earns a strong score
I generated a dataset of 100 rows and 2,000 columns, every value drawn at random from a bell curve, and assigned each row a label of 0 or 1 by coin flip. There is nothing to learn in this data. No model, however clever, can beat chance on it, and the honest accuracy is 50%. Then I did what many pipelines do. I looked at all 100 rows, picked the 20 columns that correlated best with the label, and ran five-fold cross-validation with a simple nearest-centroid classifier on those 20 columns. Across 12 repetitions with different random seeds, the average accuracy was 86.5%, with a range from 83% to 90%.
I repeated the experiment with a single change. The choice of the 20 columns was made inside each cross-validation fold, using only that fold's training rows. The average accuracy fell to 47.5%, with a range from 37% to 58%, which is what coin flips look like. The model did not change, and neither did the data or the metric. The only difference was whether the test rows had been allowed to influence which features the model saw. Trevor Hastie, Robert Tibshirani and Jerome Friedman made the same point in "The Elements of Statistical Learning" with a similar setup, in a section titled "The Wrong and Right Way to Do Cross-Validation." Decades on, the mistake is still common enough to fill a literature.
Why does noise score 86%? With 100 rows and 2,000 candidate columns, chance alone guarantees some columns that line up with the labels. The correlation between a random column and a random label across 100 rows has a standard deviation of about 0.1, and the best of 2,000 such columns typically sits more than three standard deviations out, around 0.35 in absolute value. Each of the 20 columns selected this way separates the two classes a little in the whole dataset, and a classifier that combines 20 of them finds a pattern that is real in this particular sample and absent from the world. The cross-validation then tests on rows that took part in choosing the columns, so the pattern shows up in every fold. The selection step was part of the model, and it was trained on the test rows.
This essay is about that family of errors, usually called leakage: information from the data that the model will not have at prediction time finding its way into training or evaluation. It matters to a business more than to an academic because the cost is not a retracted paper. It is a model that was approved on a number that was never real.
Hundreds of papers carried it
In 2022 Sayash Kapoor and Arvind Narayanan surveyed the literature on machine-learning-based science and found errors of this kind in 17 fields, affecting 329 papers according to their preprint. They organized the failures into a taxonomy of eight types of leakage, ranging from textbook mistakes to open research problems, and noted that in some cases the errors led to wildly overoptimistic conclusions.
Their civil-war prediction case study is the most instructive. A set of papers claimed that complex machine-learning models predicted the outbreak of civil wars far better than logistic regression, a decades-old technique. Kapoor and Narayanan attempted to reproduce the claims and found that all of the papers making this claim failed to reproduce because of data leakage. Once the leakage was fixed, the complex models did not perform substantively better than logistic regression. The headline result of an entire strand of work had come from the pipeline and not from the model.
Nobody in this story needed bad intentions. The leakage lived in details that seem harmless when you write them: where an imputation was fitted, how a split was made, which version of a table was used. People who care a great deal about getting it right still make these mistakes, because a high score feels like confirmation and nothing in the usual workflow raises an alarm.
The four kinds you will meet
Business data produces leakage in recurring patterns. The table lists four, with the test that exposes each.
| Kind | How it happens | A test that exposes it |
|---|---|---|
| Preprocessing leakage | Scaling, imputation, feature selection or resampling fitted on all rows before splitting, as in the experiment above | Fit every step inside the training fold only, and compare scores |
| Target leakage | A feature recorded after, or because of, the outcome: a cancellation reason field in a churn model, a collections flag in a credit model | For each feature, ask when its value became known relative to the prediction moment |
| Group leakage | The same customer, patient, device or household appears in both training and test, so the model memorizes identities | Split by group, never by row, and check the overlap |
| Temporal leakage | A random split on time-ordered data, or features rebuilt with information from after the prediction date | Train on the past, test on the future, and use point-in-time feature snapshots |
Target leakage is the most dangerous, because it can be nearly perfect. A churn model with a field that records "account closure initiated" will score an AUC near 1 and be useless, since by the time that field is populated the customer has already left. A good rule of thumb is that on a hard business problem, an AUC above 0.95 is a reason to look for a leak before it is a reason to celebrate.
Group leakage is the sneakiest. If a dataset has ten visits per patient and rows are split at random, each patient's visits land in both halves, and a model that recognizes a patient by their idiosyncrasies in the lab values scores well on the test visits. In a deployed system it meets new patients, and the skill it learned does not transfer. The same applies to customers with many orders, to machines with many sensor readings and to documents that were chunked into many pieces.
Temporal leakage hides in feature engineering. A "customer lifetime value" column exported from the warehouse today contains the future of every historical customer. Training on it to predict last year's churn gives a model access to what happened after the prediction date. The fix is point-in-time correctness: every feature has to be reconstructed as it would have looked on the day of the prediction. Feature stores exist to enforce that, and plain tables rarely do.
When the problem is a different world
Not every failed model leaked. Some were evaluated honestly in one place and deployed in another. In 2021 a team at the University of Michigan published an external validation of the Epic Sepsis Model, a proprietary tool widely implemented in US hospitals, on 38,455 hospitalizations. They reported an area under the curve of 0.63, a sensitivity of 33%, a positive predictive value of 12%, and alerts on 18% of all hospitalized patients. The model missed 67% of the patients who developed sepsis, and among cases where clinicians had not yet recognized the condition, it identified only 7%. The study appeared in JAMA Internal Medicine.
The vendor disputed how the study defined sepsis, arguing that a definition based on billing codes was not the clinically meaningful one, and that is a legitimate point of contention. It illustrates the second problem with a headline number. A label is a definition, and a model scored against one definition during development may be judged against another in the hospital, the bank or the warehouse where it is used. The sensible reading of the episode is that an internal validation number describes performance under the developer's data and definitions, and a purchaser needs to know both before relying on it.
A shortcut the model can see
Some failures come from confounders that a model learns instead of the thing you wanted. John Zech and colleagues tested in 2018 whether a deep network trained to detect pneumonia on chest radiographs would work at hospitals other than the one that supplied its training data. They used 158,323 images from three institutions. A model trained on pooled data from the NIH Clinical Center and Mount Sinai reached an internal AUC of 0.931. On images from Indiana University, it scored 0.815. The explanation was visible in the data. Pneumonia prevalence was 34.2% at Mount Sinai and about 1% at the other two sites, so merely guessing the hospital gave an AUC of 0.861 on the pooled data. And the network could tell which hospital an image came from, with more than 99.9% accuracy at the two sites where the authors tested it, though they could not fully establish which cues it used. Part of what it learned was where the picture was taken.
For a company, the lesson is to ask what else in the data predicts the label. A fraud model can learn which branch office handles the claims. A support-ticket classifier can learn which agent wrote the ticket. Neither is wrong on the test set, and both fail the first time the business changes who sits where.
How to catch it before the boardroom does
A few habits find most of this early, and they cost very little compared with a bad deployment.
Run the label-shuffling test. Take the entire pipeline, from raw data to final metric, and rerun it with the labels randomly permuted. The score must collapse to chance. If it does not, something in the pipeline is seeing information it should not, which is exactly what the experiment at the top of this essay detects. The test needs no domain knowledge and takes minutes, and it is the single cheapest check I know.
Count the experiments. Gavin Cawley and Nicola Talbot showed in 2010 that when hyperparameters are tuned by optimizing a cross-validation score, the same score is a biased estimate of performance, and the degradation from overfitting the selection criterion can be surprisingly large, often comparable to the differences between learning algorithms. A team that tries 200 configurations and reports the best score has used the test data 200 times. Nested cross-validation, or a holdout reserved for the final model alone, removes the bias.
Keep a final holdout that nobody touches until the end. Every time a team looks at a test set and adjusts the model, the test set leaks a little. A holdout that is used once, after all decisions are frozen, gives an honest estimate. Keep it in a place that the modeling team cannot read without a ticket.
Make time and groups explicit in the split. Write down, in the project documentation, the unit of prediction (a customer, an order, a patient-visit) and the moment of prediction, and construct the split from those two sentences. Whoever reviews the model should be able to check the split against the sentences.
Audit features for timing. For each column, record the system that produces it and the moment it becomes available. Features without a defensible answer are removed until they have one.
For research-style projects, use a structured checklist. In 2024 Kapoor, Narayanan and 17 colleagues published REFORMS in Science Advances, a consensus checklist of 32 items in eight sections, for researchers, referees and journals. Many of its items are questions a model review board could adopt without change.
The same error in language-model evaluations
Leakage also appears in evaluations of language models, in a form that is easy to recognize now. When test items are present in the training data, scores measure memory. The Scale AI study that wrote fresh problems matching a popular math benchmark found some model families scoring up to 8% lower on the new problems. A subtler version occurs in retrieval-augmented systems: if the evaluation questions are generated by a language model from the same documents that the system retrieves, and the same model family answers them, the test is partly measuring the model's agreement with itself.
The remedy has the same shape as before. Keep some test items private. Write at least part of the evaluation by hand, from real user questions. And ask of every impressive number what the model was allowed to see.
The organizational fix
The difficult part is incentive. A person who builds a model wants it to work, and a strong score confirms that. Reviewing one's own evaluation is a weak control. Separate the roles where possible, so that someone who did not build the pipeline writes the split and runs the shuffle test. Treat a result that looks too good as an incident to be investigated. The model that tells you what you hoped to hear is the one to check hardest.
Sources
- Hastie, T., Tibshirani, R., Friedman, J. (2009). The Elements of Statistical Learning, 2nd edition, section 7.10.2, The Wrong and Right Way to Do Cross-Validation. Springer.
- Kapoor, S., Narayanan, A. (2022). Leakage and the Reproducibility Crisis in ML-based Science. arXiv:2207.07048; published in Patterns (2023).
- Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine 181(8).
- Kapoor, S., Cantrell, E. M., Peng, K., et al. (2024). REFORMS: Consensus-based Recommendations for Machine-learning-based Science. Science Advances 10(18), eadk3452.
- Zhang, H., Da, J., Lee, D., et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. NeurIPS 2024 Datasets and Benchmarks. arXiv:2405.00332.
- Simulation described at the start of this essay: 100 rows, 2,000 noise columns, random labels, 20 features selected by class-mean separation, nearest-centroid classifier, five-fold cross-validation, 12 random seeds (October 2026).
- Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., Oermann, E. K. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine 15(11).
- Cawley, G. C., Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research 11.