Give the model another chance

Paper 3 · Pødenphant Lund (2026) · Read on Zenodo

The paper is titled Friction-Guided Inference: Calibrating Correction Strategies and Abstention from Logprob Signals.

This is where it all began.I wanted to see whether a language model could get better by doubting. The open model Qwen3-235B answers 41.1 per cent of SimpleQA questions correctly. Ask it to think again in the way that suits that particular model, and it rises to 51.8 per cent, past both GPT-4o at 38.0 and GPT-4.1 at 40.0. No retraining, no external checker. The lift comes from giving the model another chance under the right conditions.

The model often does know

A language model that answers wrongly is often not missing knowledge. The correct answer frequently sits among its top candidates with a good share of probability, but a slightly more probable wrong answer wins. The paper calls this the commitment gap: retrieval succeeds, commitment fails.

That difference decides what can be done about it. A commitment gap can be closed by asking the model again under different conditions, so it samples afresh and sometimes lands on the correct answer. A genuine knowledge gap, where the correct answer carries almost no probability, cannot.

What lifts, and what does not

The most important finding in the paper went against expectation. The signal in the model’s output can tell when the model is in doubt. The obvious idea was to let the signal choose the answer: trust the first answer when the signal is low, the new one when it is high. That was tested on three cells, and it did worse than simply always trusting the new answer. Minus 1.7, minus 1.6 and minus 6.1 percentage points.

So the lift comes from the strategy, not from the signal. The signal still has a role, but a different one: during calibration it shows where in the task the friction sits, and which parts no strategy reaches. Which strategy works is decided by counting how many answers it rescues. And when the model should decline to answer is decided by a different measure, the entropy.

Who does what in the pipeline. Three different measures each have their own job, and the last question is one none of them can answer.

The signal

The signal is called competing routes: how many candidates for the next word carry a real share of probability at each position in the answer. It comes free from any API that returns the probabilities.

An example. The model answers “What is the capital of France?”, and the top candidates are “Paris” at 0.96, “paris” at 0.02 and “The” at 0.01. Only one is really in it. A reasoning step in a hard task can look like this instead: “yes” 0.32, “no” 0.28, “maybe” 0.21, “it” 0.14. Now four are in it, and the race is open.

The signal picks up when the model is uncertain. As an error detector it sits at 0.64 to 0.77 AUROC on the cells where anything works at all. It does not say which answer is right.

The strategy has to suit the model

There is no single strategy that works everywhere. Step-by-step reasoning is best for Qwen2.5 on mathematics. For LiquidAI’s model it is verification, and for Qwen3 on MMLU-Pro it is narrowing the question down. Challenging the model’s answer hurts on SimpleQA and on Qwen3’s MMLU-Pro, makes no difference for LiquidAI, and helps only on Cogito.

So the pipeline has to be calibrated for each model and each kind of task. That takes a set of 50 to 200 questions with known answers and about two hours of calls to the model. The weights are not touched. It is closer to a pre-deployment health check than to training a model.

Pre-mortem works, but not for the reason you would think

Pre-mortem is the technique from human decision-making in which you imagine the project has already failed and ask why. It is one of the strategies in the suite, and it lifts on some tasks.

But the gain sits in a particular place. It is largest where a rushed prompt has made the model decline questions it could in fact have answered. There, pre-mortem brings back the knowledge the rush had shut away. Where the model already thinks things through, the gain is small. And no simple signal in the probabilities could pick out in advance which questions a strategy would rescue. The extra call is therefore best triggered afterwards, when the answers disagree, rather than sorted in advance.

The results

Four cells, measured on questions not used for calibration. All four are statistically significant, and the mean is 11.8 percentage points:

Qwen2.5-7B on MATH-500. From 45.8 to 66.5 per cent, up 20.8 points.
Qwen3-235B on SimpleQA. From 41.1 to 51.8 per cent, up 10.6, and past GPT-4o and GPT-4.1.
Qwen3-235B on MMLU-Pro STEM. From 55.2 to 62.9 per cent, up 7.7.
LiquidAI LFM2 on MMLU-Pro STEM. From 33.8 to 41.9 per cent, up 8.1.

The grey bar is the model alone, the coloured bar is with the pipeline. The dashed lines at SimpleQA mark GPT-4o at 38.0 and GPT-4.1 at 40.0.

If the model is also allowed to decline the questions where entropy is highest, the share of correct answers among those it does answer rises by 9.0 to 15.7 percentage points at 20 per cent abstention. That is the well-known reject option, going back to Chow in 1970, and the paper uses it rather than proposing it. With both combined, the runs gave 12 to 21 percentage points, but those runs ranked abstention by the route signal, and entropy is slightly better. So that figure stands as a deployment note, not a headline result.

The friction ceiling

There is a limit no method built on these probabilities can get past. If the model is confident and wrong, there is no doubt to pick up. The signal measures what it costs to choose between routes, not which route is the right one.

The paper shows the limit with a fifth, harder cell: GPT-oss-20B on GPQA Diamond, 198 PhD-level science questions. There, none of the three signals tested picks up the errors. That cell is included as the negative case that shows how far the method can reach.

What it does not claim

The signal transfers without changes across the architectures tested: an ordinary transformer, a mixture-of-experts model, and LiquidAI’s network, which is not built on attention. That is not a claim that it works on all models. The mixture-of-experts model is still a transformer, and the LiquidAI result is one model on one task with a low starting accuracy. Whether it holds more broadly is an open question.

The pipeline has to be calibrated for each model and each kind of task, and calibration needs a set of questions with known answers. The cost depends on provider and model version.