Accuracy is the last thing a production model tells you

In mid-March 2020, Instacart's system for predicting whether a given product was available at a given store started getting worse. The metric the company used to track how often items were found at the store dropped from 93% to 61%, according to Fortune's reporting in June of that year. Nothing in the model had changed. Shoppers' habits had: demand surged, store inventories swung from hour to hour, and the patterns learned from earlier months no longer described the present. Instacart's engineers retrained, shifting from several weeks of shopper-reported "found" and "not found" data to roughly ten days of it, since recent behavior better matched the new reality, and they began re-scoring the model every hour instead of every three.

The episode makes a general point. A model is a frozen description of a period of the world, and it is deployed into a world that keeps moving. The distinguishing feature of the Instacart case is that the team could tell. Its metric depended on shoppers reporting, within the hour, whether an item was there. Most models do not get an answer that fast. A credit-scoring model can take a year to learn whether a loan defaulted. A churn model learns who left only after the quiet period has passed. A fraud model learns from chargebacks that can take months to arrive. For these, the accuracy dashboard is a lagging indicator by construction, and a team that waits for it is steering by the wake.

My claim in this essay is that monitoring has to be built in layers matched to what can be observed at each delay, and that the design of the alerts matters as much as the choice of metric. Accuracy is the last layer to report, which makes it the least useful for catching trouble early and the most useful for judging how bad it was.

Three kinds of change

The word drift covers several different phenomena. João Gama and colleagues, in their 2014 survey of concept drift adaptation, set out the vocabulary that the field still uses, and the distinctions help decide what to monitor.

Covariate shift is a change in the distribution of the inputs while the relationship between inputs and outcomes stays the same. A lending model sees a new mix of applicants because a marketing campaign targeted a different segment. The model's logic may still be right, but it is being applied to regions of the input space it saw rarely. Label shift, also called prior shift, is a change in how common the outcomes are: the default rate rises in a recession even if each type of applicant behaves as before. Concept drift is a change in the relationship itself. The same input now implies a different outcome. During 2020 a furlough meant something different about a worker's income stability than it had in 2019, and a model that treated employment history as a stable signal was reading a changed language.

The three have different remedies. Covariate shift is detected from the inputs alone, without labels. Label shift can often be detected from the distribution of the model's predictions. Concept drift is usually visible only when outcomes arrive. A monitoring design that watches only inputs misses concept drift, and one that waits for outcomes finds all three late.

Layer one: data integrity

The cheapest failures to cause are also the cheapest to detect. An upstream team renames a column, changes a unit from reais to centavos, starts sending nulls where it used to send zeros, or stops sending a feed altogether. A model downstream continues to produce predictions, which look plausible only because the dashboard has no way to know they are garbage.

The checks in this layer are simple and should run on every batch or on a sample of live traffic: that the schema matches, that null rates stay within their historical range, that numeric values fall within expected bounds, that categorical fields contain only known values, that the volume of records is close to normal, and that the data is fresh. These checks need no labels and no statistics beyond ranges, and they can alert within minutes. They are also the closest analogue to what Sculley and colleagues, in the 2015 paper on hidden technical debt, called testing the invariants of an input pipeline.

Layer two: drift, measured by size and not by significance

The second layer compares the distribution of inputs, and of the model's predictions, against a reference period such as the training data. The temptation is to run a statistical test on every feature and alert on small p-values. At scale, that goes wrong in two ways.

The first is that large samples make trivial differences significant. Consider a feature that has shifted by one hundredth of a standard deviation, an amount no business would care about. With a million rows in each of the reference and current samples, the two-sample Kolmogorov-Smirnov statistic for this shift is about 0.004. The critical value at the 5% level for samples of that size is about 0.0019, so the test rejects with confidence. A test of significance answers whether a difference exists, and with enough data the answer is always yes. What a monitor needs is how large the difference is.

The second problem is multiplicity. A model with 200 input features tested daily at the 5% level will raise about ten false alarms per check on perfectly stable data, since 200 × 0.05 = 10. A team that receives ten alerts a day learns to ignore all of them.

The remedy is an effect-size metric and thresholds set from experience. The Population Stability Index, long used in credit scoring, divides a feature into bins and sums, over bins, the difference between the current share and the reference share times the logarithm of their ratio. Take a feature binned into five income bands, with reference shares of 20%, 30%, 25%, 15% and 10%. If production shows 14%, 26%, 27%, 19% and 14%, the PSI is about 0.05. If production shows 8%, 20%, 27%, 25% and 20%, the PSI is about 0.27. The conventional reading in credit risk is that values below 0.1 are stable, values between 0.1 and 0.25 deserve attention, and values above 0.25 indicate a significant shift. Those thresholds are an industry convention and not a statistical law, and each team should calibrate its own by looking at what past values preceded real problems.

For models with many features, per-feature tests are a noisy way to detect that something has changed. Stephan Rabanser, Stephan Günnemann and Zachary Lipton compared detection methods in "Failing Loudly" (NeurIPS 2019) and found that a two-sample test applied after reducing dimensionality with a pre-trained classifier performed best across the shifts they studied, and that domain-discriminating approaches were useful for characterizing a shift and judging whether it was harmful. In practice this means monitoring a few summary quantities, such as the distribution of the model's own scores or of its internal representations, in addition to individual columns.

The prediction distribution is the most useful single series. Plot the share of predictions above the decision threshold, per day. A model that approved 31% of applications last month and approves 22% today has changed behavior, whatever the cause, and the person on call needs to know even before anyone can say whether the change is good or bad.

Layer three: outcomes, and what to do while waiting

Labels eventually arrive, and when they do, performance should be tracked by cohort as well as overall. A model whose aggregate accuracy is stable can be deteriorating on a segment that is growing. Calibration deserves its own panel: if the model says 20% and 12% of those cases occur, the number is no longer a probability, and every threshold built on it is off.

How many labels does it take to notice a decline? Suppose the model's reference accuracy is 90%. With 100 labeled cases in a monitoring window, the standard error is 3 points, and a one-sided test at the 5% level with 80% power can detect only a drop of about 7.5 points or more. With 400 labels the detectable drop is about 3.7 points, and with 1,000 it is about 2.4. Comparing two groups of labels against each other, as when checking a challenger against a champion, takes more: detecting a fall from 90% to 87% needs roughly 1,800 labeled cases in each group. A weekly sample of a hundred reviewed cases is worth having, since it catches collapses and gives people a feel for the failures, but it cannot replace the proxies and the drift layers for the smaller changes that add up.

While labels are delayed, there are ways to get early signals from outcomes. Short-horizon proxies help, such as first-payment default as an early indicator for a twelve-month default model, or a ticket reopened within a week for a support classifier. Sampling and labeling by hand gives a direct estimate: a reviewer who labels one hundred random predictions a week produces a time series of accuracy with a wide but honest interval. Shadow deployment, where a new model runs on live traffic without acting, collects comparisons before any change is exposed to customers.

Alerts need severity and an owner

Not every signal should wake someone. A workable scheme has three levels. A page, reserved for conditions where the model is probably producing wrong outputs right now: a data-integrity failure, a prediction distribution that has jumped well outside its range, an action limit reached. A ticket, for slower moves that someone should look at within days: a PSI creeping past its attention threshold, a calibration gap that has opened, a cohort whose outcomes have worsened. And a weekly report, for trends with no immediate action, kept so that gradual change can be seen over months. Each alert names an owner, a person and not a team alias, because an alert that belongs to everyone is read by nobody.

It helps to review the alerts themselves every quarter. List the ones that fired, mark which led to an action, and retire or retune the ones that never did. An alerting system that is never pruned decays into noise.

What to do when an alert fires

A runbook written in advance saves the first hour. The first step is to check data integrity, since pipeline faults are a common cause of apparent drift. The second is to slice: which features, which segments, which time window? The third is to decide among four responses. Ignore it, if the shift is small and harmless. Roll back or fall back to a simpler rule if the model is clearly failing and the cost of errors is high. Retrain on recent data if the world has changed and recent labels exist. Or escalate to a human review, if the change is something the model was never meant to handle.

Retraining deserves caution because it is the response people reach for first and the one that can hide a problem. If a model's own decisions influence the labels it learns from, as in lending or fraud where only approved applications generate outcomes, then retraining on recent data reinforces the model's earlier choices. Instacart's choice of a shorter training window reflects a trade that every team faces: fresher data adapts faster but is noisier, and the right window depends on how quickly the world is moving and how many labels arrive per day. Automatic retraining without a validation gate is a way to ship a worse model on a schedule. A sound pipeline retrains, evaluates the candidate against the current model on a recent holdout, and promotes it only if it wins.

Sculley and colleagues recommended action limits, caps on what a model can do in the world, such as a ceiling on approvals per hour. They complement monitoring by bounding the damage in the interval between a change and its detection.

A one-page spec for each model

Each production model needs a short monitoring document with an owner's name on it. It should list the five to ten data-integrity checks and their thresholds; the two or three drift metrics, including the prediction distribution, with the thresholds that page someone; how long labels take to arrive and which proxies or manual samples stand in for them; the retraining trigger and the validation gate; and the rollback procedure, including who may invoke it. A model that cannot fill out that page is not ready for production, and a model that already has been running without one can have it written this week, starting with the model whose errors cost the most.

Sources

1. Vanian, J. (9 June 2020). Fortune article on Instacart and the effect of the pandemic on its machine-learning models. fortune.com/2020/06/09/instacart-coronavirus-artificial-intelligence.
2. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys 46(4).
3. Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015.
4. Rabanser, S., Günnemann, S., Lipton, Z. C. (2019). Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS 2019. arXiv:1810.11953.
5. Instacart Engineering (2020). How Instacart's item availability evolved over the pandemic. tech.instacart.com.
6. Smirnov, N. (1948). Table for estimating the goodness of fit of empirical distributions. Annals of Mathematical Statistics 19(2).