A confidence score is only useful if it means what it says

A lender approves loans when the model's estimated probability of default is below a threshold. Suppose a good loan earns $200 and a default costs $2,000. The break-even probability is 200 divided by 2,200, about 9.1%. A loan is worth making if the true chance of default is under that figure and not worth making if it is above. The lender's model is excellent at ranking, with an AUC near 0.85, and for a particular group of applicants it reports a 6% chance of default. The model is wrong about the level. The true rate for that group is 12%.

On the model's belief, each loan to the group has an expected value of 0.94 × 200 − 0.06 × 2,000, which is +$68. At the true rate it is 0.88 × 200 − 0.12 × 2,000, which is −$64. The lender will approve these loans, because the number says to, and across 10,000 of them it will lose $640,000 while believing it earned $680,000. Nothing was wrong with the model's ability to tell better applicants from worse ones. The harm came from the model's report of how sure it was.

Ranking quality and calibration are separate properties, and most evaluation reports measure only the first. AUC is blind to calibration: stretch every score by a monotone transformation and the ranking, and therefore the AUC, does not change, while every probability becomes wrong. For any decision that uses the number itself, in a threshold, an expected value, a price or a handoff rule, calibration is the property that determines whether the decision is sound. This essay explains what it is, why modern models tend to lack it, what happens with language models, and what to do.

What calibration means and how it is measured

A model is calibrated if, among all the cases where it says 70%, about 70% turn out to be positive. The idea is operational. If the weather service says there is a 30% chance of rain on many days, it should rain on about 30% of them. A reliability diagram checks this by sorting predictions into bins by predicted probability and plotting, for each bin, the observed frequency of the outcome. A calibrated model produces points along the diagonal. A common summary is the expected calibration error, the average gap between predicted confidence and observed accuracy across bins, weighted by how many predictions fall in each.

The Brier score, introduced by Glenn Brier in 1950 for weather forecasts, is the mean squared difference between the predicted probability and the outcome. It belongs to a family called proper scoring rules, which have the property that a forecaster minimizes the expected score by reporting honest beliefs. Log loss is another. Accuracy is not a proper scoring rule, and it is also blind to the probabilities. A model that says 51% and a model that says 99% earn the same accuracy when both are right, and very different rewards from a proper rule when one is wrong.

Why do modern models miss? Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Weinberger studied the question in 2017 and reported that modern neural networks, unlike those of a decade earlier, are poorly calibrated, with depth, width, weight decay and batch normalization all influencing the outcome. The typical failure is overconfidence: the network assigns 95% to predictions that are right 85% of the time. The cause is partly mechanical. Training with cross-entropy keeps rewarding larger margins on training examples, which pushes confidence up even after the model stops improving on held-out data. The same paper showed that a one-parameter fix, temperature scaling, which divides the model's logits by a constant fitted on a validation set, repairs most of the problem on the datasets they tested.

Where it matters, and where it does not

Calibration matters whenever the output feeds an arithmetic that treats it as a probability. Pricing risk is the plainest case: an insurer's premium for a policy depends on the estimated claim probability. Expected-value decisions, like the loan above, depend on it. So do cost-sensitive thresholds, where the cutoff is set from the ratio of error costs. Models whose outputs are combined also need it, since multiplying or averaging probabilities from separate models assumes each means what it says. And calibration decides whether a human can use the number: a clinician or underwriter who has learned that "90%" from the system is right about nine times in ten will use it differently from one who has learned it is right six times in ten.

It matters less where only the order counts. A recommender that shows the top ten items needs a good ranking and does not care whether the scores are probabilities. A retrieval system that passes the best twenty passages to a language model cares about order and little else. Where in doubt, the test is whether any downstream step uses the magnitude of the score. If so, it should be calibrated, and the calibration should be measured and not assumed.

What happens with language models

Language models complicate the picture. Saurav Kadavath and colleagues at Anthropic reported in 2022, in a paper titled "Language Models (Mostly) Know What They Know," that larger models are well calibrated on diverse multiple-choice and true-or-false questions when the questions are presented in a suitable format. They also showed that a model can be asked to estimate the probability that its own proposed answer is correct, which they called P(True), and to predict whether it knows the answer before attempting it, P(IK), with encouraging results and with weaker calibration on tasks unlike those it was trained on.

OpenAI's GPT-4 technical report added a caution. It states that the pre-trained model is highly calibrated on a multiple-choice benchmark, meaning its stated confidence tracks how often it is correct, and that the post-training process reduces calibration. The caption of the corresponding figure says that post-training hurts calibration significantly. The step that makes a model pleasant and helpful appears to make its confidence less honest.

A reason for this was set out in 2025 by Adam Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang in "Why Language Models Hallucinate." They argue that errors originate in pretraining as natural statistical pressures, and that training and evaluation then reward guessing over acknowledging uncertainty. Most benchmarks score a wrong answer and a "don't know" the same, which makes guessing the better strategy, and models optimized against such scores become good test-takers. Their proposed remedy is to change how existing benchmarks are scored, so that confident errors are penalized and abstentions are not.

For a company using these models, the practical consequences are direct. Do not read the token probabilities or the verbalized "I am 90% sure" of a chat model as probabilities until you have tested them on your own task. Collect a few hundred cases with known answers, record the confidence the model gives, and plot the reliability diagram. Where confidence is poorly calibrated, a mapping fitted on those cases can repair it, and agreement across several samples of the same question is often a more honest signal than the model's own statement.

Fixing it

There are two families of remedy, and they do different jobs.

Recalibration takes the model as it is and learns a mapping from its scores to probabilities, using held-out data from the same distribution the model will face. Temperature scaling, Platt scaling and isotonic regression are the standard tools, in rising order of flexibility and of data requirement. They are cheap, and they do not change the ranking. The mechanics of temperature scaling fit in two lines. A binary classifier produces a logit, and the probability is the logistic function of it. A logit of 3.0 gives 95.3%. Divide the logit by a fitted temperature of 1.5 and it becomes 2.0, which gives 88.1%. The ordering of cases does not change, because the transformation is monotone, and the stated confidence comes down to what the validation data justifies.

A common mistake lies upstream of all this. Teams that deal with rare events often oversample the positives or reweight the classes so the model sees a balanced training set. The model then learns probabilities for a world where half the cases are positive. If the true rate is 1%, an output of 50% means roughly 1% in production, since the odds must be multiplied by the ratio of the true prior odds, 0.01 to 0.99, to the training prior odds, 1 to 1. A pipeline that resamples and does not correct afterwards reports probabilities that are off by an order of magnitude, and everything built on them inherits the error.

One limitation applies to every method here. The calibration set must resemble production, and a mapping fitted to last year's data can fail on this year's, which is why calibration belongs on the monitoring dashboard next to drift.

Conformal prediction takes a different route and gives a guarantee in place of a probability. In Anastasios Angelopoulos and Stephen Bates's introduction, it builds sets or intervals that contain the true outcome with a user-specified probability, say 90%, and the guarantee is explicit, non-asymptotic and distribution-free, holding without assumptions about the model or the distribution. It can wrap any trained model, a neural network included. For regression, the output is an interval around the forecast whose width reflects the model's real uncertainty. For classification, it is a set of plausible labels, often one and sometimes several, and a large set is a direct signal that the model does not know. The guarantee rests on a condition called exchangeability, roughly that the calibration data and the new cases come from the same process, so it weakens when the world drifts.

Calibration also has a governance side. The scholars Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan showed in 2016 that when the base rates of an outcome differ between groups, a risk score cannot in general be calibrated within each group and also have equal error rates across groups, except in degenerate cases. The result forces a choice that no technique removes. A lender or a court deciding which property to prioritize is making a policy decision, and the people accountable for the system should make it knowingly, with the numbers on the table. Reporting calibration by group is the first step, since it shows which of the two properties the system currently has.

Designing for abstention

Calibrated confidence makes a policy possible that uncalibrated scores cannot support: let the model act when it is sure and send the rest to a person. Take an illustrative classifier with overall accuracy of 88.5%, whose predictions fall into four confidence bands. In the lowest, 10% of cases have confidence from 0.5 to 0.7 and accuracy of 60%. In the next, 30% of cases have confidence from 0.7 to 0.9 and accuracy of 82%. Then 30% fall between 0.9 and 0.97 with accuracy 94%, and the final 30% above 0.97 reach 99%. If the model automatically handles only the top two bands, which are 60% of the volume, accuracy on the automated cases is (0.3 × 0.94 + 0.3 × 0.99) / 0.6, or 96.5%, and the remaining 40% go to reviewers.

That trade replaces one accuracy number with a curve: for each level of automation, the error rate you accept. The business can pick the point on the curve that matches the cost of errors and the cost of review. None of it works without calibration, since the bands only mean what they say if the confidence in them is true. And it needs to be rechecked as the data shifts, because a band that held at 94% last quarter may hold at 88% this quarter.

Put the diagram in the review

The simplest institutional change is to make the reliability diagram a standard exhibit in every model review. Ask for it by segment as well as overall, since a model can be calibrated on average and badly off for one group of customers, as in the loan example. Ask for its confidence intervals, because a bin with forty cases says little: if the observed rate in the bin is 70%, the 95% interval runs about 14 points either way. With 200 cases in the bin it shrinks to about 6 points, and with 1,000 to about 3. That arithmetic sets how much labeled data a calibration check needs, and it is why a diagram built from a small validation set can look either reassuring or alarming by chance. Ask for the Brier score or log loss next to the AUC. And ask what the downstream process does with the number, so the team can say whether calibration matters for the use at hand.

An honest model is one whose confidence can be audited against outcomes. The audit takes a day, and it is the only way to find out whether a 70% from the system is worth acting on.

Sources

1. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1).
2. Gneiting, T., Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477).
3. Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017. arXiv:1706.04599.
4. Kadavath, S., Conerly, T., Askell, A., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
5. OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774, section on limitations and Figure 8.
6. Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664.
7. Angelopoulos, A. N., Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511.
8. Kleinberg, J., Mullainathan, S., Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. ITCS 2017. arXiv:1609.05807.