Neither tool nor junior hire: manage AI output as a production lot with a defect rate

In the Harvard and Boston Consulting Group experiment released as a working paper in 2023, 758 consultants were split between two experiments, and within each they were randomly assigned to work without AI, with GPT-4, or with GPT-4 plus a short prompt-engineering overview. In the experiment with a business case built so that careless use of the model would lead to the wrong conclusion, which 373 of them did, the control group reached the correct recommendation about 84.5% of the time. The two GPT-4 groups got it right about 70% (GPT-4 alone) and 60% (GPT-4 plus the overview) of the time. The same paper reports that graders scored the AI-assisted recommendations higher on quality than the controls' recommendations, and they did so among the wrong answers as well as the right ones.

One reading is that AI can make experts worse. I draw a narrower lesson as a manager. How an output looks and whether it is correct are two separate measurements, and only one of them can be taken by skimming. Work that arrives fluent, formatted and confident passes most informal reviews, so an organization with no measurement of correctness is in effect measuring polish.

Two mental models are available to a manager facing this kind of work. In the first, the model is a tool, like a spreadsheet formula, and a wrong answer is a bug that gets reported and fixed. In the second, it is a junior employee, and the right response is the supervision one would give a new hire: read the work, correct it, and let a record of good results build trust. My argument is that both models fail at the same point. A better one comes from manufacturing. Treat each batch of AI output as a production lot with a defect rate that can be estimated, set the depth of review by the cost of an escaped error, and measure the reviewers as carefully as the model, since the people checking the work are the part of the process that degrades without anyone noticing.

The tool model fails because the defect rate is real and changes with the task

A formula returns the same answer every time, and when it is wrong there is a cause that someone can find and remove. Model output has a different character. A wrong answer is a draw from a distribution, and the question that matters is how often the draws are wrong for this kind of work.

The best-documented case is legal research. Magesh and colleagues at Stanford ran a preregistered evaluation of commercial legal research tools whose vendors had marketed retrieval as a way to avoid hallucination. They found that Lexis+ AI and two Thomson Reuters products each produced hallucinated content between 17% and 33% of the time, less often than GPT-4 alone but far from zero, and they judged the vendors' claims to be overstated. A rate of that size is not a defect that a ticket can close. It is a property of the process, like the scrap rate of a machine, and it gets the management a scrap rate gets: someone measures it and decides what happens to the parts that come out wrong.

The rate also moves with the task, and the surface of the work does not show the movement. In the BCG study, 18 tasks selected to be within the model's reach saw consultants with AI complete 12.2% more tasks on average, 25.1% faster, with quality rated more than 40% higher. On the task outside the frontier, consultants with the tool were on average 19 percentage points less likely to be right than those without it. A different half of the sample did each kind of task, so this compares groups and not the same people across two tasks. The authors call the pattern a jagged frontier: tasks that look similar in difficulty can sit on opposite sides of it. The evidence has limits. It used GPT-4 in 2023, and the outside-frontier result comes from a single case that the authors designed so that the model would struggle without extensive guidance, so I would not carry the 19 points into a plan. What transfers is the structure, a defect rate that differs by task and that the user cannot see from the output.

Self-assessment does not fill the gap. METR's 2025 randomized trial with 16 experienced open-source developers found that tasks took 19% longer when AI was allowed, while the developers believed it had made them about 20% faster. METR has since said that the result is out of date for current tools. The gap between what the developers felt and what the clock showed is the part that survives, and it is the reason a defect rate has to come from a count.

The junior-employee model fails because the model's errors look like its successes

The second model is more flattering, and it appeals because it supplies a ready management routine. A new hire's work is read, corrected and returned, and over months the manager learns which tasks to trust. Three features of that routine do not carry over.

A junior's draft carries signals. It is uneven, it comes with questions, and the effort shows. A manager uses those cues to decide where to look. Model output has a uniform surface. The BCG study's graders, two sets of them (consultants who had not taken part and business-school students), scored recommendations on a 1-10 rubric, and the AI-assisted group scored higher than the control group even among the subset of wrong answers. If the surface carries no information about correctness, a careful reading with no sample to calibrate against mostly produces reassurance.

A correction given to a junior changes next week's work. A correction given to a model changes nothing unless someone alters the prompt, the retrieval index, the evaluation set or the model itself. Feedback has to be routed into the process, which can be done, but the person who does it is the process owner. The reviewer's red pen only fixes one item.

Volume is the third difference. A junior produces a few items a day, and a model can produce a thousand drafts in a week. Supervising each item as one would a new hire's is a review-everything policy, and the arithmetic below shows that it can cost more than it saves on low-stakes work while leaving the high-stakes work under-checked. The junior model also builds trust from a run of good results, and a long run of good results is the setting in which the reviewers described below start to slip.

A weekly lot can be sampled, and the sample has to be read with its operating curve

Manufacturing met this problem nearly a century ago. Dodge and Romig at Bell Telephone Laboratories published a method of sampling inspection in 1929 for deciding whether a lot of product should be accepted on the evidence of a sample. The attribute sampling standards that followed, such as ISO 2859-1 and its military-standard ancestor MIL-STD-105, draw on that tradition (this lineage is from secondary accounts). NIST's engineering handbook describes the central tool: an operating characteristic curve, which plots the probability of accepting a lot against its true percentage of defects, together with two risks. The producer's risk is that a good lot is rejected, and the consumer's risk is that a bad lot is accepted.

A made-up example makes the calculation concrete. Suppose a team receives 1,000 low-stakes AI drafts each week, and a reviewer costs $3 per item. It wants to accept lots with a 2% defect rate at least 90% of the time and to accept lots with an 8% rate no more than 10% of the time. A binomial search gives a plan of 82 randomly chosen items with acceptance if three or fewer defects are found, a weekly sampling cost of $246. The table below gives the curve, and it also shows what happens when the reviewer is imperfect: its third column assumes, as an illustration, that the reviewer catches 70% of the defects that are present.

True defect rate in the lot	Chance the lot is accepted, perfect reviewer	Chance the lot is accepted, reviewer catches 70%
2%	92%	97%
4%	58%	80%
6%	27%	55%
8%	10%	32%
10%	3%	17%

A lot with an 8% true defect rate, which the plan was designed to reject nine times in ten, gets through nearly a third of the time when the reviewer misses 30% of the defects. The sampling plan is only as good as the instrument that counts defects, and the instrument is a person. The value of 70% is an assumption. Beck and colleagues, whose experiment I discuss below, saw participants correct 82% of one kind of planted error and 31% of another, so a single catch rate for all defects does not exist.

The sample also limits what can be claimed. If 299 randomly drawn items show no defects, the data support a defect rate under 1% at 95% confidence, provided the reviewer is perfect. With a reviewer who catches 70%, the same claim needs 427 clean items. A team that reports "we checked twenty and found nothing" has shown only that the rate is under about 14% at 95% confidence, and that again assumes a perfect reviewer.

Review depth should follow the cost of an escaped error, and a break-even rule prices it

Sampling estimates the rate. A second decision concerns what to do with each item, and a simple rule covers it. If a review costs c, the defect rate is p, a reviewer catches a share d of defects, and an escaped defect costs L, then reviewing an item pays when p times d times L exceeds c. With the illustrative figures of 4% defects, d of 0.7 and $3 per review, the break-even cost of an error is $107. Items whose errors would cost less than that are not worth reading one by one at this defect rate.

Consider three streams in a week, all invented. Internal drafts number 1,000 and an escaped error costs $20 in rework. Customer-facing messages number 250 and an error costs $400. Contractual or regulatory items number 50 and an error costs $10,000. Applying the rule gives a value per review of $0.56 for the first stream, $11.20 for the second and $280 for the third.

Stream	Items per week	Cost of an escaped error	Policy	Expected weekly cost
Internal drafts	1,000	$20	Sample 82, no item review	$1,046
Customer-facing	250	$400	Review every item once	$1,950
Contractual	50	$10,000	Two independent reviewers	$2,100

The tiered policy costs $5,096 a week in review spend plus expected losses. Reviewing nothing costs $24,800 in expected losses, and reading every item once costs $11,340. About 52 defective items arrive in the week, and the tiered policy lets 43 of them through, 40 of them in the $20 stream. That is the design working as intended, and it is hard to defend in a meeting, because the review-everything policy feels safer even though it spends $3,000 reading items whose errors cost $800 in total.

Two assumptions deserve scrutiny. The two reviewers in the third stream are treated as independent, which gives a combined catch rate of 91%. Wolfe used the same arithmetic to show how double screening could cut a 30% miss rate to 9%, and he stated the condition: independence. Two people reading the same fluent output are likely to be fooled together. The second assumption is the catch rate itself. If d falls from 0.7 to 0.5, the same policy costs $9,096, and more than half of that is the third stream. The estimate of L is the other soft input, since it has to include rework, customer harm and legal exposure, and it deserves an owner in finance.

Reviewers degrade in predictable ways, so the review step needs its own measurement

Lisanne Bainbridge's 1983 paper "Ironies of Automation" argued that automating a process leaves people the tasks that designers could not automate, and that the person left to monitor is poorly placed to do it. She cited vigilance research to say that effective attention to a source where very little happens cannot be held much beyond half an hour. She also noted that skills decay without use, and that if a computer makes decisions faster and on more dimensions than a person could, the person can only judge whether the decisions look acceptable. Her setting was process plants, and a reviewer reading a finished document after the fact is not a plant operator watching a dial. The mechanism of a rare error among many correct items still carries over.

Parasuraman and Manzey reviewed the empirical work in 2010. Complacency, they concluded, appears when a person has several tasks and the automated one competes for attention. Automation bias produces omission and commission errors when decision aids are imperfect, and both affect novices and experts and are not removed by simple practice or instructions. Nothing in the review exempts people who check AI output.

The rarity effect has a measured size in a different field. Wolfe and colleagues reported in Nature in 2005 that people searching for a target missed about 7% of them when targets appeared half the time and about 30% when they appeared 1% of the time. Wolfe repeated the numbers on NPR that June and added that nobody knew whether the effect held in airport or medical screening. If the effect carries over to AI review, a better model makes defects rarer, and rarer defects are the ones a reviewer is more likely to miss.

Direct tests with AI-like advice exist. In a 2023 Radiology study by Dratsch and colleagues, 27 radiologists read 50 mammograms with BI-RADS suggestions presented as AI output, and the suggestion was wrong for 12 of the 40 cases in the main set. Inexperienced readers were right under 20% of the time on those cases against almost 80% when the suggestion was correct, and readers with more than 15 years of experience fell from 82% to 45.5%. The sample is small, and the RSNA release describes the suggestions as purportedly from AI, so the study tests how people react to advice more than how they react to a particular product. In an experiment reported as an arXiv preprint in 2025 and in Harvard Data Science Review in 2026, Beck, Eckman, Kern and Kreuter had 2,784 participants check scripted "AI" suggestions for values in corporate emissions tables, with four of the seven later suggestions wrong. Spelling-type errors were corrected about 82% of the time and errors needing an understanding of the annotation rules about 31% of the time. Requiring a correction whenever a participant flagged an error reduced corrections, and a bonus for the top 10% of performers had no consistent effect. Skeptics of AI caught more errors than enthusiasts.

Skill may erode as well. A Lancet Gastroenterology and Hepatology study of four Polish centres compared colonoscopies done without AI in the three months before and after AI detection tools arrived, and the adenoma detection rate fell from 28.4% to 22.4%. The design is observational with no concurrent control, so the authors' reading, that over-reliance reduced attention, is a hypothesis. Anthropic's January 2026 study of 52 mostly junior programmers learning an unfamiliar library found a quiz average of 50% for those with an AI assistant against 67% for those coding by hand, with the widest gap on debugging questions, the skill reviewers need.

Countermeasures follow from the same literature. Measure the catch rate by slipping known defects into the review queue, which is what the airport screening machine Wolfe described does by projecting fictional threats into bags. Make flagging an error cheap. Rotate reviewers and keep some hands-on work in their week, which echoes Bainbridge's suggestion of hands-on control for part of each shift. Bainbridge also warned against raising the real failure rate of a system to keep operators alert, because they would then distrust it, and I treat a seeded test item as a different thing: it audits the reviewer without changing what the customer receives. That distinction is my judgment, and Wolfe noted a cost, that screeners trained on known threats may look for those and miss a new one.

The strongest objection is that the defect rate will not hold still

Sampling theory assumes a lot of similar items drawn from a stable process. LLM work is neither. Models are updated, prompts are edited, retrieval indexes change and input mix drifts. The jagged frontier means that p for contract summaries bears no relation to p for refund calculations. A defect rate estimated in March may describe nothing in April, and a policy built on it would be a false comfort. Deming's third point in Out of the Crisis tells managers to cease depending on inspection to achieve quality, and sampling is inspection.

I think the objection is right about the hazard and wrong about the conclusion. Shewhart's control chart exists for the question of whether a process is stable. He proposed limits three standard deviations from the mean, chosen, according to Wheeler's account, to balance the cost of reacting to noise against the cost of missing a real change. Applied here, the chart plots each week's sampled defect fraction. In the example, with an observed rate of 2.8% (a true 4% seen by a 70% reviewer) and 82 items a week, the upper limit is about 8.3%, so seven or more defects in a sample is a signal. The chart is also a weak instrument. If the true rate doubled to 8% (5.6% as the same reviewer would observe it), a single week's sample would breach the limit only about 18% of the time. The remedies are to read runs of weeks together, enlarge the sample where stakes are high, stratify by task type so that one stream cannot hide another, and restart the baseline whenever the model, the prompt or the retrieval changes.

Deming's point also applies in a way that sharpens the practice. Inspection data are a by-product that should feed upstream changes: better prompts, tighter retrieval, a stronger evaluation set. If the weekly count of defects never falls and nobody has changed anything upstream, the team has built an expensive way of watching. The aim is for inspection to shrink as the process improves, and the defect log is how one knows what to fix first.

A protocol small enough for one owner to start next week

The pieces fit into a routine that a single owner can begin with one stream of output.

1. List the streams of AI output and agree with finance a figure for the cost of one escaped error in each.
2. Draw a random sample from each stream every week, stratified by task type, reviewed blind to the model's confidence, and log defects by kind.
3. Seed a small share of known defects, synthetic or from past incidents, into reviewers' queues and record each reviewer's catch rate.
4. Plot the weekly defect fraction on a control chart, with the baseline reset after any change to the model, prompt or retrieval.
5. Each month recompute the break-even error cost from the measured defect rate and catch rate, and move streams between review tiers.

Nothing here settles what happens when the model gets very good. At a defect rate of one in a thousand, an extrapolation from Wolfe's 1% condition says a human reviewer would miss a large share of what remains, and Bainbridge's argument says the skill needed to catch it would have eroded. I did not find a field measurement of reviewer performance at that prevalence for AI output, and the colonoscopy result is observational, from four centres and six months of data. The seeded-defect measurement in step 3, run for a year in an ordinary company, would provide the first numbers.

Sources

1. Dell'Acqua, F., McFowland, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., Lakhani, K. R. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. SSRN 4573321 (working paper dated 22 September 2023); later published in Organization Science (2025), doi:10.1287/orsc.2025.21838. Figures on 758 consultants (385 and 373 in the two experiments), the 12.2%, 25.1% and 40% results, the 84.5%, 60% and 70% correctness rates, the 19-point average gap and the recommendation-quality scores are from a copy of the working paper that I read in full; I did not open the journal version.
2. Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., Ho, D. E. (2024; published 2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362; Journal of Empirical Legal Studies 22(2), 216-242 (2025), doi:10.1111/jels.12413 (citation details from secondary listings; the 17% to 33% range is from the arXiv abstract).
3. Bainbridge, L. (1983). Ironies of Automation. Automatica 19(6), 775-779. doi:10.1016/0005-1098(83)90046-8. Passages checked against a reproduction at complexcognition.co.uk and a summary at ferd.ca; I did not see the journal PDF.
4. Parasuraman, R., Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3), 381-410. doi:10.1177/0018720810376055 (abstract read; full text not available to me).
5. Wolfe, J. M., Horowitz, T. S., Kenner, N. M. (2005). Rare items often missed in visual searches. Nature 435, 439-440. The 7% and 30% miss rates are in the paper (author manuscript in PubMed Central, PMC4224304). The double-screening arithmetic, the remarks on screening, the fictional threats and the training caveat are from Wolfe's interview with Ira Flatow, "Finding a Needle...", NPR Talk of the Nation/Science Friday, 3 June 2005 (npr.org/transcripts/4679042).
6. Dratsch, T., et al. (2023). Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology, 2 May 2023; figures from the RSNA news release (rsna.org/news/2023/may/ai-bias-may-impair-accuracy), which I read; the paper itself I did not open.
7. Beck, J., Eckman, S., Kern, C., Kreuter, F. (2025; 2026). Bias in the Loop: How Humans Evaluate AI-Generated Suggestions. arXiv:2509.08514 (v1, figures on error types); Harvard Data Science Review 8(2), 2026 (listing seen on a secondary page; I did not open the journal version, and the figures are from the arXiv v1 full text).
8. Budzyń, K., et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology and Hepatology 10(10), 896-903. doi:10.1016/S2468-1253(25)00133-5.
9. Anthropic (29 January 2026). How AI assistance impacts the formation of coding skills. anthropic.com/research/AI-assistance-coding-skills.
10. METR (10 July 2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ (the page notes a February 2026 follow-up and that the 2025 results are out of date).
11. Dodge, H. F., Romig, H. G. (1929). A Method of Sampling Inspection. Bell System Technical Journal 8(4), October 1929. ISO 2859-1: Sampling procedures for inspection by attributes, Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection.
12. NIST/SEMATECH e-Handbook of Statistical Methods, section on lot acceptance sampling and operating characteristic curves. itl.nist.gov/div898/handbook/pmc/section2/pmc22.htm.
13. Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product. Van Nostrand. Rationale for three-sigma limits as described in Wheeler, D. J., Why Three-Sigma Limits? Quality Digest (qualitydigest.com/aug/wheeler.html); I did not read Shewhart's book.
14. Deming, W. E. (1982). Out of the Crisis. MIT Center for Advanced Engineering Study; point 3 as given in secondary lists of the fourteen points (for example deming.org, which cites pages 23-24), not read in the book.
15. Illustrative quality-control arithmetic: weekly lot of 1,000 drafts, $3 per review, sampling plan of 82 items accepting up to 3 defects, reviewer catch rates of 70% and 50%, defect rate of 4%, three streams of 1,000, 250 and 50 items with escaped-error costs of $20, $400 and $10,000, two independent reviewers, control-chart limits for an observed rate of 2.8%. All values are made up for the example and were computed with Python.