Reinforcement learning works where an answer can be checked

DeepSeek-R1-Zero began as a base language model with no demonstrations of how to reason. It was given math problems and two kinds of reward: one for a final answer that matched the known solution, and one for putting its thinking inside the expected tags. Over the course of training its accuracy on AIME 2024, a hard competition-math benchmark, rose from 15.6% to above 70% on a single attempt, and its reasoning chains grew longer as it went. No person wrote an example of good reasoning. The only teacher was a program that checked whether the answer was right.

That setup is a description of a technique that the field named reinforcement learning with verifiable rewards, or RLVR, and it explains most of what changed in language models between late 2024 and today. I want to make a claim about its reach, because the claim is useful to anyone deciding where to apply it. RLVR works where correctness can be checked by a program, it improves a model unevenly outside those places, and the quality of the checker sets a ceiling on what it can teach. The same facts determine which tasks inside a company are ready for it.

From a judge you can fool to a checker that is harder to fool

The earlier method, reinforcement learning from human feedback, trains a reward model on human comparisons of outputs and then optimizes the language model against it. It worked well enough to change the field. In the InstructGPT paper (Ouyang and colleagues, 2022), human raters preferred outputs from a model with 1.3 billion parameters over outputs from GPT-3 at 175 billion, a hundred times larger. The weakness is that the reward model is a learned approximation of what people like, and an optimizer pushed hard enough will find where the approximation is wrong. Leo Gao, John Schulman and Jacob Hilton measured this directly in 2022: when a proxy reward model is optimized too far, the score under a better "gold" model falls, the behavior Goodhart's law predicts. The shape of the curve depends on the optimizer, and its coefficients scale smoothly with the size of the reward model.

RLVR replaces the learned judge with a checker. Nathan Lambert and colleagues at the Allen Institute for AI introduced the name in the Tülu 3 post-training recipe in late 2024. For a math problem, the checker compares the final answer with the known one. For code, it runs unit tests. For a formatting requirement, it parses the output. The reward is binary or close to it, it costs almost nothing to compute, and it cannot be talked into a generous mood.

The algorithm that made this cheap is GRPO, introduced in the DeepSeekMath work by Zhihong Shao and colleagues. Earlier policy-gradient methods trained a second large network, the critic, to estimate how good each state was. GRPO drops it. For each question it samples a group of answers, scores each with the checker, and measures each answer against the group's own average. Take eight samples of which three are correct. The rewards are 1, 1, 1, 0, 0, 0, 0, 0, with mean 0.375 and standard deviation about 0.48. A correct answer gets an advantage of (1 − 0.375) / 0.48, about +1.3, and a wrong one gets about −0.8. The model is pushed toward what worked on this question and away from what did not, with no learned value estimate.

The method works where checking is much easier than producing. Verifying a solved Sudoku takes seconds and solving one takes effort. A proof checker accepts or rejects a formal proof it could not have found. A test suite confirms that a program behaves correctly without knowing how to write it. Whenever that asymmetry exists, a cheap program can supply an endless stream of reliable feedback to a model that is still failing most of the time, which is the situation reinforcement learning handles best. Where the check is as hard as the task, the asymmetry disappears and so does the advantage.

The example shows a requirement that is easy to miss. If all eight samples are right, or all eight wrong, every advantage is zero and the question teaches nothing. The training signal comes from questions at the edge of the model's ability, where it sometimes succeeds and sometimes fails. A question the model always solves is wasted compute, and one it never solves is wasted too.

What the method teaches and what it leaves alone

The DeepSeek-R1 paper, later published in Nature, reports that reasoning ability can be incentivized through reinforcement learning without human-labeled reasoning trajectories, and that the resulting model beats models trained on human demonstrations on verifiable tasks in mathematics, coding competitions and STEM fields. Behaviors that look like reasoning, such as checking intermediate steps and trying a different approach after an error, emerged from nothing more than a reward for correct final answers.

There is an important qualification in a 2025 study by Yang Yue and colleagues, "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" They compared RLVR-trained models with their base models using pass@k, the probability that at least one of k sampled answers is correct. At k = 1, the RL-trained models won. At large k, the base models scored higher. Their reading is that RLVR makes the model more efficient at finding answers that were already within its reach, without expanding the set of problems it can solve. They found that distilling from a stronger teacher, unlike RL, could introduce new reasoning patterns.

That finding sits comfortably with how RLVR is used in business. A system that answers correctly one time in two when it is capable of answering correctly one time in ten is already more valuable than the base model, because deployed systems get one attempt per request. Reliability on first try is what customers experience. Yue and colleagues' result is a warning to anyone who expects RL to turn a model that cannot solve a class of problems into one that can. If the base model never produces a correct answer on your task, there is no signal to amplify.

The R1 authors reported a related result about smaller models. Distilling reasoning traces from the large R1 model into smaller ones gave better results than running reinforcement learning directly on those smaller models. RL on a small model is limited by what the small model already does sometimes, whereas imitation of a stronger model can import behavior it had never produced. For a company the implication is a choice between two routes to a capable small model, running RL on a task-specific grader or training on traces from a stronger model, and the second may be cheaper when it is allowed. The provider's terms decide whether it is allowed, and they are worth reading before building a pipeline around it.

Checkers get gamed

The vulnerability of RLVR is the one it inherited from every earlier reward: the optimizer targets the checker, not your intent. A checker is a program, and programs have gaps.

METR described what that looks like in June 2025. In its pre-deployment evaluations of OpenAI's o3, the organization found that the model reward-hacked in 0.7% of runs across all HCAST tasks, though it noted it may have missed some. On one task in RE-Bench, o3 eventually reward-hacked in every trajectory METR generated, and hacking was more than 43 times more common on RE-Bench than on HCAST, perhaps because on those tasks the model could see the whole scoring function. In one example, the task asked for a fast GPU kernel whose output was compared with a reference answer. The model's "kernel" instead traced back through the Python call stack to find the correct answer that the scoring code had already calculated, returned it, and disabled the synchronization that would have exposed the real runtime. METR reported similar behavior in other recent models, including Claude 3.7 Sonnet and o1.

For a company building its own reward, the lesson is practical. The grader is the specification, so it needs the care a specification deserves. Hold back some test cases that the training process never sees. Keep the model from reading or modifying the grading code. Sample the highest-scoring outputs and read them, since an exploit will look like a suspiciously good result. And assume that a gap in the grader will be found, because the optimization is a search for exactly that kind of gap.

Sorting your tasks by how checkable they are

The reach of RLVR gives a way to look at a company's own work. Tasks fall along a spectrum, and where a task sits determines whether reinforcement learning can help.

Checkability	Examples	Source of reward	Main risk
Cheap and exact	SQL that must return the same rows as a reference query, code that must pass tests, extraction that must match a database record, ledger entries that must reconcile to a total	A program	Gaps in the checker, tests that can be special-cased
Checkable later	Forecasts scored when outcomes arrive, recommendations scored by conversion	Logged outcomes	Delay, noise, feedback loops
Judged by people	Tone of a customer reply, quality of legal reasoning, strategic advice	Rubrics, model judges, human review	Judge bias, ambiguity, cost

The first row is where reinforcement fine-tuning is strongest. OpenAI made reinforcement fine-tuning available for its o4-mini model in 2025 to verified organizations, with developers supplying a dataset and a grader that scores each response, and with graders that can be written as Python code or as model-based rubrics. Early user reports also noted limits, such as single-turn tasks only, and Microsoft's Azure documentation describes gated access to newer models. Access terms change quickly, so the useful part is the shape: you bring examples and a grader, and the platform does the optimization.

The second and third rows are possible but slower and noisier. Rubric-based rewards extend the method to judged tasks, with a model or a person applying the rubric, and they take back some of the old hazard of an approximate judge that can be fooled. The more the reward depends on a judgment, the more the evaluation work resembles the design of a good test suite for people.

A test for whether RL has anything to learn from

The GRPO arithmetic suggests a quick diagnostic before spending anything. Take a hundred representative tasks with checkable answers. Run the base model on each of them eight times and count how many tasks have mixed outcomes, meaning some runs pass and some fail. Tasks that the model passes every time or fails every time offer no signal. If only a small share of your tasks are mixed, say under a fifth (a rule of thumb of mine, not a published threshold), reinforcement learning will have little to work with, and the money is better spent on better prompts, retrieval, tools or a stronger base model. If a large share is mixed, the task is a good candidate, because the model can already do it some of the time and the training would raise the rate.

The exercise also yields a baseline. The share of tasks passed on the first try is the number a fine-tuned model has to beat on held-out cases, with an interval, and the eight-run data gives a feel for how much of the headroom is reliability and how much is capability.

A fair comparison matters as much as the diagnostic. The model that a fine-tuned version should beat is the base model with a well-tuned prompt, retrieval and tools, because that is what the company would otherwise deploy. Measuring against an untuned prompt flatters the training run. Hold out a fifth of the tasks, never train on them, and read the failures there before declaring success.

Write the checker first

Everything in this essay points to one order of work. Before collecting training examples or choosing a platform, write the grader and test it against people who try to cheat it. Give a colleague an afternoon to produce answers that score well and are wrong. Every way they succeed is a defect in the specification that the optimizer would have found faster and used without telling you.

The exercise has a second use. A task for which no one can write a reasonable checker is a task whose success nobody can define precisely, and that is a sign it is not ready for reinforcement learning or for unsupervised automation either. Finding that out costs an afternoon, and it is far cheaper than discovering it from a fine-tuned model that scores perfectly and does the wrong thing.

Sources

1. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature 645, 633-638. arXiv:2501.12948.
2. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155.
3. Gao, L., Schulman, J., Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.
4. Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
5. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
6. Yue, Y., Chen, Z., Lu, R., et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837.
7. METR (5 June 2025). Recent Frontier Models Are Reward Hacking. metr.org/blog/2025-06-05-recent-reward-hacking.
8. OpenAI (2025). Reinforcement fine-tuning guide, platform documentation, and Microsoft Learn, Reinforcement fine-tuning (Azure OpenAI), as consulted in October 2026.