When a model solves your take-home, redesign the work sample instead of policing it

On 21 January 2026 Tristan Hume, a lead on Anthropic's performance optimization team, published an account of a hiring test that kept losing to the company's own models. He had designed it in November 2023, when Anthropic lacked performance engineers for its new scale and its standard pipeline could not evaluate all the candidates that outreach was bringing in. About 1,000 candidates took it over the next year and a half, and it helped hire most of the current team. The task was to optimize code for a simulated accelerator, and the instructions explicitly let candidates use AI. By May 2025, Hume writes, more than half of the candidates would have done better by delegating the task to Claude Code entirely, and a pre-release Claude Opus 4 beat almost all humans inside the four-hour limit. A pre-release Opus 4.5, run for two hours, matched the best human result in that window, and that human had made heavy use of an earlier Claude model with steering.

The usual responses to a story like this are to hunt for cheaters, ban the tools, or bring everyone onsite. I think all three aim at the wrong target. A take-home is a work sample, a slice of the job handed to the candidate, and its only claim to usefulness is that performance on the slice predicts performance on the job. Once a model can do the slice, it no longer resembles the work that remains for the human, and closer supervision of the candidate cannot restore the resemblance.

My argument has three parts. Personnel-selection research supports work samples and structured scoring in principle, with smaller numbers than the older textbooks give. The practical repair is to sample the work a person still has to do beside a model (scoping the problem, reviewing and debugging generated code, explaining the trade-offs) and to score it against a written rubric. The literature also says nothing about AI-era formats, so every redesign, including the one below, is a hypothesis for an employer to check against its own hires.

The validity case for work samples is real, and older than the problem

A common reference point in hiring advice is Schmidt and Hunter's 1998 review in Psychological Bulletin, which put the mean validity of work sample tests at .54 and of structured interviews at .51, with general cognitive ability at .51 and unstructured interviews at .38. Validity here is the correlation between the selection score and later job performance.

Sackett, Zhang, Berry and Lievens revisited those estimates in the Journal of Applied Psychology in 2022. Their argument is technical: the usual corrections for range restriction, applied through assumed distributions, overcorrected, and most of the validities had been overstated. In their revision, structured interviews fall from .51 to .42 and remain the top-ranked procedure. Job knowledge tests come out at .40, empirically keyed biodata at .38, work samples at .33, cognitive ability at .31, and unstructured interviews at .19. The abstract puts the typical reduction at .10 to .20 points. Work samples lose .21, and the authors trace it to a switch in the underlying evidence. The 1998 figure came from an older narrative review, and the replacement is a 2005 meta-analysis by Roth, Bobko and McFarland that covered 54 studies, 53 of them concurrent, with a mean observed validity of .26 that rises to .33 after correcting for unreliable performance ratings.

Two details of the revision bear on the take-home question. First, the authors point out that the strongest predictors in their revision (structured interviews, job knowledge tests, keyed biodata and work samples) are all job-specific, and they connect that to an old distinction between measures that are signs of a trait and measures that are samples of the behavior. A sample works because it resembles the thing it predicts. Second, they note that work samples and knowledge tests presume candidates who already have training or experience, which is the usual situation for an engineering hire and a reason the format suits it.

The structured interview's .42 comes with a residual standard deviation of .19, which means the validity in some settings sits near .18. Ranked by the lower bound of the 80 percent credibility interval in their Table 3, work samples place fourth (.21) and structured interviews fifth (.18). A work sample needs careful scoring and a local check on whether it predicts among your own hires.

Anthropic's redesign traded realism for novelty, and Hume says so

The Anthropic post is the most detailed public account I found of a work sample decaying in front of its owner. The first version gave candidates four hours. When Claude 3.7 Sonnet and Opus 4 began to match or beat most of them, Hume used Opus 4 to find where it struggled and built a second version around that point. He removed the multicore part, which the model had already solved and which slowed the development loop, added depth to the machine, and cut the time to two hours, partly because scheduling overhead had been causing multi-week delays in the pipeline. Opus 4.5 then reached the passing threshold in under an hour, stalled at what it took to be a memory bandwidth limit, found a trick when Hume gave it the target cycle count, and ended the two hours level with the best human.

Hume considered two obvious fixes and rejected both. He worried about the enforcement challenges of a ban, and he wanted candidates to distinguish themselves in a setting with AI, as they would on the job. Raising the bar to something like beating Claude Code makes the candidate a spectator, because the model works faster than a person can follow. He tried a second optimization problem, a data transposition task. On a patched version Opus 4.5 made progress without finding the most efficient solution, then solved it when run with extended thinking, and his conclusion was that the problem was too well covered by training data. A further attempt used puzzles in the style of the Zachtronics games, with a tiny unusual instruction set, no visualization tools and scoring by instruction count, so that building your own tooling becomes part of the assessment. Opus 4.5 failed one medium-hard puzzle. Early scores correlated with candidates' past work, an observation and not a validation.

He also states the cost. The original worked because it resembled real work, he writes, and the replacement works because it simulates novel work, and he says realism may be a luxury no longer affordable. A novelty-based test stays ahead of a model only until the next model, and it moves the sample away from the job, which is the reverse of what the validity literature rewards. Most employers do not hire for work that exotic, and for them the better direction is the opposite one: keep the task realistic and change what is being sampled.

The sample has to move to the work that remains

If a model writes the first draft of the code, a person on the job still decides what to build, judges whether the draft is right, finds out why it fails, and explains the choice to colleagues who must live with it. A redesigned sample should draw out exactly those skills. There is some direct evidence that they are also the ones that suffer when people delegate. In a randomized trial by Judy Hanwen Shen and Alex Tamkin at Anthropic, published as a preprint in January 2026, 52 mostly junior engineers learned an unfamiliar Python library, Trio, with or without an AI assistant, then took a quiz. The assisted group averaged about 50 percent and the hand-coding group about 67 percent, a gap with an effect size of 0.74. Debugging questions showed the largest difference. Assisted participants finished roughly two minutes faster, which was not statistically significant. Participants who asked conceptual questions or requested explanations along with code kept their scores up, while the three lowest-scoring patterns, which leaned on the assistant for generation or debugging, averaged below 40 percent.

Anthropic's write-up lists a relatively small sample, a comprehension quiz taken shortly after the task, no agentic tool in the study, and an association between interaction style and score that does not establish causation. The study also covers a single library. It does not show that a candidate's comprehension predicts job performance. It does show that comprehension varied widely between participants using the same tool, and a skill that varies that much is at least worth trying to measure.

Large employers have started to move their formats in this direction, though what I can cite is announcements and not outcomes. Meta began rolling out an AI-enabled coding round in October 2025, according to an interview-preparation site that quotes an internal Meta message. It replaces one of the two coding rounds, runs 60 minutes in an environment with a built-in assistant, and the evaluation covers the same four competencies as a traditional round: problem solving, code quality, checking the work, and communication. A report syndicated on 14 May 2026, based on Business Insider's reading of an internal Google document, described a pilot due to start in the second half of 2026 for junior and mid-level engineers in select US areas: a code comprehension round in which candidates read, debug and optimize an existing codebase using Gemini, assessed on prompting, output validation and debugging. A Google spokesperson confirmed the Gemini detail. I found no published validity or outcome data for either format as of October 2026, and the Google account rests on one internal document as reported by one outlet.

One exercise, rebuilt: a reconciliation task with a review and a live change

Here is a concrete case, with the numbers made up for illustration. A common take-home for a backend or data role asks the candidate to reconcile a bank export against an internal payments table: match records, flag the unmatched ones, handle partial payments, duplicate references and the timezone boundary at month end. The old format gave three to four hours and graded the code. I would expect a current model to produce a plausible solution in minutes, which leaves the artifact with little signal about the person.

The rebuilt version uses the same domain and has four parts, taking about three hours of candidate time, close to the original.

1. Scoping, 20 minutes, no code: the candidate receives the brief and a ten-row sample, and writes the questions they would put to the finance owner, the assumptions they would adopt if no answer came back, and what they would leave out of the first release.
2. Build, 90 minutes, any tools allowed including a model: the candidate submits the code with its tests, plus the exported session log or commit history.
3. Review, 45 minutes: the candidate receives a pull request that a current model produced beforehand and that we edited to hold a known set of defects, and writes a review that lists the defects found, ranks them, names a test that would catch each, and fixes one.
4. Live conversation, 40 minutes on video, with the same script for every candidate: walk through two decisions in your own submission, defend one against an alternative, then react to a requirement we change on the call, such as payments that settle across two days.

The defects in part three should come from evidence. Run the exercise through the strongest model available and keep the defect types that the model introduces itself or misses when asked to review without hints, for example floating-point sums on currency, local time used where UTC was needed, and a test that asserts the buggy behavior. This borrows Hume's method of starting from where the model struggles, applied to the review and not to the build. I have not run this exercise on candidates, and the claim that these particular defects trip current models is a proposal to test.

A current model can do parts one to three largely unaided, so their value lies in what they show about how the candidate directed the tool, what the candidate accepted and what the candidate caught. Part four is the link between the artifacts and the person, because its questions concern the candidate's own code and a requirement nobody has seen.

Scoring is where the redesign earns its validity

A better sample scored by impression would waste the best-supported result in the literature, that structured scoring outperforms unstructured judgment. Google's published guidance on structured interviewing describes the components: the same vetted questions for every candidate, detailed notes on answers, standardized rubrics that show what good and poor answers look like, and interviewer training and calibration. Its rubric grades responses on four levels (outstanding, solid, borderline, poor), with documented examples at each level for each attribute a question tests. Google reports its own findings that structured interviews predict performance better than unstructured ones, along with a saving of about 40 minutes per interview from pre-made materials and rejected candidates who were 35 percent happier after a structured interview, but the page gives no sample sizes or method for these, so I treat them as claims.

Applied to the exercise, the rubric below scores five dimensions on four levels, 4 for outstanding to 1 for poor. The weights are my own illustrative choices and should be set by the hiring team before the first candidate is seen.

Dimension	Evidence	Weight	Level 4 looks like	Level 1 looks like
Scoping	Part 1	20%	Questions that would change the design, with a defensible cut line	Restates the brief, asks nothing
Verification	Parts 2 and 3	30%	Tests that fail on wrong behavior, most planted defects found and ranked	Tests that mirror the code, defects missed
Diagnosis	Part 4, changed requirement	20%	Locates where the design breaks and reasons to a fix	Cannot say what the change affects
Trade-offs	Part 4, own decisions	20%	Names the alternative and what it would cost, citing own code	Cannot defend a choice without rereading it
Direction of the tool	Session log	10%	Constraints stated, wrong output rejected with a reason	First answer accepted, no checks

Two raters score each candidate independently from written evidence before they talk, and a gap of more than one level on any dimension is settled by looking at the evidence and not by averaging. The team calibrates on two anonymized practice submissions, and the hire threshold is fixed in advance. Everyone uses the same assistant in parts two and three, as the Meta and Google formats do with their provided tools.

A worked example, with invented candidates, shows why the weights matter. Candidate A scores 3, 3, 3, 4 and 3 on the five rows in order, for a weighted total of 0.6 + 0.9 + 0.6 + 0.8 + 0.3 = 3.2. Candidate B writes the stronger scoping memo and the stronger code and review, scoring 4 and 4 on the first two rows, but falters on the call and in the session log, scoring 1, 1 and 2 on the rest, for 0.8 + 1.2 + 0.2 + 0.2 + 0.2 = 2.6. With a threshold of 2.75 set in advance, A advances and B does not. I would expect B to have ranked higher under the old format, where only the code was graded. By my own estimate each candidate costs about two hours per rater for reading and the call, so a team with a large applicant pool should put a cheaper filter ahead of this stage.

The strongest case for in-person only, and why it falls short

The best version of the opposing view goes like this. Any remote format has an unobservable third party. Interview Coder, a tool built to feed AI answers to candidates during interviews without the interviewer noticing, was priced at $60 a month according to a CNBC report in March 2025, and its founder, Chungin Lee, argued that an interview premised on having no AI made little sense. If a hidden assistant cannot be ruled out, no remote score can be interpreted, and the only controlled setting is a room with the devices removed. Google's leadership was reported to be considering it. In a February 2025 internal town hall, reported by CNBC, Sundar Pichai said it was worth thinking about some fraction of interviews being in person (his stated reason was that it helps candidates understand the culture, and CNBC placed the remark inside a story about cheating), and the vice president of recruiting, Brian Ong, said virtual interviews were about two weeks faster but that more work was needed to integrate AI into the process.

I accept the premise about assurance and still reject the conclusion, for three reasons. The first is that a room establishes who is answering and says nothing about whether the task resembles the job. A closed-book exercise samples closed-book engineering, which I think describes few engineering jobs now. The same Google, according to the May 2026 reporting, is piloting a round where candidates use an approved assistant and are scored on validating its output. Meta's internal message, as quoted by the preparation site, described its AI-enabled round as more representative of the developer environment and less open to cheating through language models. Whether those formats predict anything is unknown, but their designers appear to doubt that the room alone is the answer.

Second, cost falls on candidates. Travel, time away from a current job and visa constraints are barriers that scale with distance from the office, so a required onsite round narrows the pool toward people who are nearby or can afford the trip. I have no source that quantifies this, and it is my judgment. The one piece of selection research on interview mode that I found cuts the other way. A 2016 meta-analysis by Blacksmith, Willford and Behrend pooled 12 studies (13 samples, 1,557 people) and found that technology-mediated interviews were rated lower than face-to-face ones, with an effect size of d = -.41. A university press release on the work says the articles dated from 2000 to 2007 and that video interviews fared worst, a detail I could not check in the paper itself, and video quality has changed since. The authors warned that using different modes for different candidates invites fairness problems. That is a fair point in favor of in-person, and it argues for one mode applied to everyone at a given stage.

Third, the two approaches combine. A redesigned exercise ends in a live part that could just as well take place in a room, with the same script and the same rubric. For finalists, where the stakes justify the cost to the candidate, running part four in person is a reasonable use of the format. What I would avoid is treating the room as the whole assessment, and treating a video call as fully safe. Questions anchored in the candidate's own submission and a requirement changed on the spot reduce the value of a hidden assistant. They do not remove it, and a team that cannot accept that residual risk should move the call to a room and price in the candidate's cost.

I found no validity data for AI-permitted work samples

The evidence under all of this is thin. The work sample estimate rests on 54 studies, 53 of them concurrent, in which current employees were tested and not applicants who were later followed up. Every number in the Sackett revision predates models that can pass a take-home, and they are averages across jobs and settings. I found no published criterion validity for an AI-permitted work sample, and nothing for the Meta or Google formats beyond announcements. The early correlation Hume describes for his new puzzles is an observation from a handful of people.

An employer cannot easily fill that gap alone. Take a team that makes 20 hires with a given format. A true correlation of .30 between its scores and later performance would usually fail to be distinguishable from zero in that sample, since the smallest correlation that clears conventional significance at n = 20 is about .44, and a two-sided test at the .05 level would detect a true .30 only about one time in four. The criterion also limits what can be seen: Sackett and colleagues assume a reliability of .60 for supervisor ratings of job performance in their main estimates, and report a lower .52 (a figure from earlier work on interrater reliability) as a sensitivity check. Under the lower value, even a perfect predictor would show an observed correlation near .72 against a typical rating, and near .77 under the higher one. Small teams should therefore expect noisy local evidence and lean on the structure that the large studies support, which is job-relevant content, the same script for everyone and anchored scoring.

Sackett and colleagues note that work samples, job knowledge tests and cognitive ability tests all show substantial mean differences between Black and White applicants, while structured interviews show smaller ones, and they advise combining predictors. A rubric-scored live part is partly a structured interview, so the redesign may help, but that should be checked on real score distributions and not assumed. And tool parity matters, because a format that rewards whoever pays for the better assistant partly measures income.

A next step that costs little: before changing anything, run the current exercise through the strongest model available and score its output with the existing rubric. If it clears the hire bar, the exercise has already stopped measuring what you hoped. Whatever format replaces it, store the component scores of every candidate beside their review ratings at six and twelve months, so that someone eventually has the data to estimate what the format predicts.

Sources

1. Hume, T. (21 January 2026). Designing AI-resistant technical evaluations. Anthropic Engineering. https://www.anthropic.com/engineering/AI-resistant-technical-evaluations
2. Sackett, P. R., Zhang, C., Berry, C. M., Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068. https://doi.org/10.1037/apl0000994
3. Schmidt, F. L., Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274. Figures as reported in item 2.
4. Roth, P. L., Bobko, P., McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating and integrating some classic literature. Personnel Psychology, 58(4), 1009-1037. Figures as reported in item 2.
5. Google re:Work. A guide to structured interviewing for better hiring practices. https://rework.withgoogle.com/intl/en/guides/a-guide-to-structured-interviewing-for-better-hiring-practices
6. Shen, J. H., Tamkin, A. (2026). How AI impacts skill formation. arXiv:2601.20245 (submitted 28 January 2026); and Anthropic (29 January 2026), How AI assistance impacts the formation of coding skills. https://www.anthropic.com/research/AI-assistance-coding-skills
7. Elias, J., CNBC (9 March 2025). Meet the 21-year-old helping coders use AI to cheat in Google and other tech job interviews; report on Google's internal town hall on in-person interviews and on Interview Coder, as syndicated by NBC New York. https://www.nbcnewyork.com/news/business/money-report/meet-the-21-year-old-helping-coders-use-ai-to-cheat-in-google-and-other-tech-job-interviews/6178911/
8. Moneywise article by Chase Kell, syndicated on Yahoo Finance (14 May 2026), summarizing Business Insider's report on Google's AI-assisted coding interview pilot (secondhand: based on an internal Google document). https://finance.yahoo.com/sectors/technology/articles/asking-kid-math-test-without-151500986.html
9. Hello Interview. Meta's AI-enabled coding round (secondhand: candidate reports and an internal Meta message as quoted by the site). https://www.hellointerview.com/blog/meta-ai-enabled-coding
10. Blacksmith, N., Willford, J. C., Behrend, T. S. (2016). Technology in the employment interview: A meta-analysis and future research agenda. Personnel Assessment and Decisions, 2(1). https://scholarworks.bgsu.edu/pad/vol2/iss1/2 (abstract read; the 2000-2007 date range and the video ranking come from a George Washington University press release, ScienceDaily, 25 July 2016, https://www.sciencedaily.com/releases/2016/07/160725111425.htm)
11. Illustrative exercise and arithmetic: the reconciliation task, the four-part timings, the rubric weights, the candidate scores (3, 3, 3, 4, 3 and 4, 4, 1, 1, 2), the 2.75 threshold, the two-hours-per-rater estimate and the 20-hire example are made up by the author for this essay. The critical correlation of about .44 at n = 20, the roughly 25 percent power for a true .30 (Fisher z approximation) and the .72 and .77 ceilings (square roots of .52 and .60) are standard statistical calculations.