Fine-tuning presses the doubt out

Paper 2B · Pødenphant Lund (2026) · Read on Zenodo

I study language models to understand how they hold on to knowledge.Fine-tuned language models give more confident wrong answers than RAG-based ones on exactly the same knowledge. The reason is in the training itself. Every gradient step strengthens the route that won and presses the alternatives a little further down; do it thousands of times and they are gone. It can be read straight off the model's output, token by token. The surprising consequence: in this experiment, fine-tuning left the model worse at applying knowledge than if it had never been given it at all.

What is this about?

There are two standard ways to give a language model new knowledge. You can keep it in the prompt. That is in-context learning (ICL); RAG (retrieval-augmented generation) is the most widely used practical form of that approach. Or you can train it into the model's weights, usually through a method called LoRA. That is fine-tuning (FT). The common view is that RAG and fine-tuning are two options along a cost axis: RAG/ICL is more expensive to run (you carry the retrieved context along), FT is more expensive to build (you have to actually train), but you end up with "the same knowledge" either way.

You do not end up with the same knowledge. You end up with knowledge held two different ways, and the difference shows up sharply in the model's behaviour. These are not two implementations of the same memory. They are two different ways of holding it.

The mechanism in one paragraph

Every time you take a gradient step on a language model (every backward pass under the usual loss) you strengthen the route that produced the "right" answer and press the alternatives down. Do it once and the effect is small. Do it thousands of times and the alternatives are pressed below the noise floor. They are effectively gone from the distribution the model can reach. ICL does not do this. ICL runs the prompt forward through the model and the model produces an answer. The distribution over candidate answers is still there; it is simply computed on the way through.

The compression appeared on every training set we tried, so it is a structural consequence of how training works rather than of anything particular in the data. But it does not arrive at the same rate regardless. Vary the wording during training and it measurably slows down, which comes back below.

What gets measured

The finding rests on one number, read token by token. At the first token of the answer, you look at the distance between the model's best and second-best candidate. If they sit close together, a race is still running. If they are far apart, one answer has won so cleanly that nothing else is within reach. That number, measured in nats, is what grows under training.

It is not new that fine-tuned models hallucinate more. It is documented behaviourally by Gekhman and colleagues and at the activation level by Shen and colleagues. What is new here is the reading at the output layer itself, plus three things: a dose-response from 5 to 100 epochs that keeps rising even after training accuracy has flattened; a compute-matched control that shows what the data alone does; and the bridge to a finding previously tied to RLHF, which turns out to hold for plain LoRA fine-tuning on innocuous facts too.

What the experiments showed

The experiments ran on an invented 47-fact knowledge base called "Zorbetik", invented so the model could not already know any of it, on Qwen2.5 base models at 3B and 7B parameters, with LoRA budgets from 5 to 100 epochs.

ICL wins on retrieval by 16–28 percentage points. The model that has just been told a fact answers better than the model that has been trained on it.
FT makes application worse. Not just worse than ICL, but worse than no context at all. Training the knowledge in degraded the model's ability to use that knowledge on related questions.
The race closes as training accumulates. The distance to the second-best answer goes from 5.46 nats under ICL to 17.85 after 30 epochs and 20.60 after 100. Under the compute-matched control, run across three random seeds, it goes from 14.82 at about 1,410 gradient steps to 17.69 at about 7,050. It is the amount of training that drives it.
Entropy goes to zero. Under every FT regime tested, uncertainty at the first answer token falls from 0.32 under ICL to essentially nothing.
Variation in the training data protects. Train the same fact in several wordings and the race stays open longer. At matched compute the distance is 4.5 nats narrower than raw training at 1,410 steps, and 2.3 nats narrower at 7,050. Retrieval accuracy holds flat at 86–88 per cent throughout, so what moves is the calibration, not whether the model can answer.
The format decides whether anything installs at all. If the answer can be copied from what stands in the training example, nothing installs: zero per cent recall on new questions. Hide the answer so the model has to produce it itself, and it installs almost completely, 97.5 to 100 per cent. That holds whether the user turn is phrased as a question or as a statement. Copyability is what decides it.
What is trained in gets defended, but is not immune. A fabricated fact trained in to perfect recall, 40 out of 40, is still overridden by a contradicting passage in context on 40 per cent of items. And the items it loses are not the ones where its own answer stood weakest.

ICL holds the distance at 5.46 nats. On the raw axis it grows with the amount of training, from 14.82 at about 1,410 gradient steps to 17.69 at about 7,050. At both levels the varied training sits lower than the raw, that is, less compressed: a 4.5-nat gap at the lower level and 2.3 at the higher. Entropy at the first answer token falls from 0.32 under ICL to near zero under every FT regime.

One correction is worth carrying, because it went the other way from what was expected. An early single-seed run appeared to show that varied training compressed more than raw, 21.12 against 17.85 nats. That comparison did not hold: the varied run took about five times as many gradient steps, so it measured the amount of training rather than the variation. The compute-matched control across three seeds reverses the direction, and that is where the numbers above come from.

Why this matters

Confident hallucination

FT-trained models hallucinate confidently because the alternative answers have been compressed out of reach. The "I am not sure" signal that would normally flag a wrong answer is gone, not because the model is sure, but because the substrate signal that carries the uncertainty has been pressed below the noise floor. The model is not lying about its confidence; the part of its computation that would have given it second thoughts has gone quiet.

FT-only agentic systems cannot represent uncertainty

A system built on a fine-tuned model cannot reliably tell its operator "I do not know" or "I am only 60 per cent sure". The signal that would have carried that information has been compressed. This is structural, not a fault in any particular training run.

RAG and fine-tuning, at the substrate level

The debate between RAG and fine-tuning gains one more dimension. RAG operates in ICL mode: retrieved documents are pulled into the prompt, the model evaluates them forward, calibration is preserved. Fine-tuning compresses it as a structural consequence. The choice between them is therefore architectural as well, not only a matter of deployment cost. This sits alongside the architectural argument the RAG literature raised itself; it does not replace it.

Long-context agents inherit the calibration for free

Claude Code, Cursor and any multi-turn conversation operate in ICL mode by default: every turn re-evaluates the full context with no weight update. They inherit the calibration property automatically. That is why a long agentic session can feel more honest than a fine-tuned chatbot on the same knowledge. The signal is still intact.

Hybrid architectures, and what they cannot count on

The obvious design is to put the stable, cheap knowledge in the weights and use the context for the part of the reasoning that has to keep the doubt open. The paper proposes a concrete composition of that kind. But the last experiment sets a bound on it: a fact trained in to perfect recall is still overridden by a contradicting passage in context two times out of five. A hybrid design that counts on retrieved context reliably correcting a fine-tuned error is counting on too much. What is trained in gets defended, not surrendered, and not held either.

What this does not claim

Fine-tuning was tested both as LoRA and in full-parameter mode, and both trace the same monotonic compression. They differ in rate, but there is a caveat there: the two runs use different learning rates per standard practice, and no sweep was run. The difference in rate could therefore be partly an effect of the learning rate rather than of the number of free parameters. Single random seed in the first experiments, three in the compute-matched controls, 42 training runs in total. Two model sizes, 3B and 7B, less than one order of magnitude. One invented domain. At the longest training budgets one seed fell clearly outside the others, so single measurements deep into training should be read more cautiously than the short ones. The contradicting-context experiments are 40 items per probe, one seed, one model family, and on one of them the exact split depends on who adjudicates the answers. The direction does not.