Per-Token Confidence Trajectory in LoRA Fine-Tuning: Substrate-Output Evidence for Cumulative Calibration Collapse

TL;DR

Why do fine-tuned language models hallucinate more confidently than their in-context-learning (or retrieval-augmented-generation, RAG) counterparts on the same knowledge? The standard framing treats in-context learning (ICL) and fine-tuning (FT) as alternatives along a deployment-cost axis. They are not interchangeable. The substrate-level distinction this paper establishes also offers a principled resolution of the RAG-vs-fine-tuning debate: RAG operates in ICL-mode and preserves calibration; FT compresses it as a structural consequence.

Mechanism. Each backward pass under cross-entropy loss amplifies the winning route asymmetrically and presses alternatives below the noise floor; the depth of this compression scales with how many gradient passes the substrate has absorbed. ICL preserves the model's calibrated distribution over candidate answers because no weight update has compressed it. FT compresses it as a structural consequence of cumulative gradient pressure — on any training data tested, though not at a rate independent of it: at matched compute, paraphrase-augmentation measurably slows the compression. The mechanism generalises a previously RLHF-specific (reinforcement learning from human feedback) finding (Paper 1 §3) to all weight-update training, including plain LoRA fine-tuning on innocuous factual data.

The observable. The measurement is the per-token top-2 logit margin at the answer position: the gap in nats between the best and second-best candidate token. As training presses the alternatives down, the margin widens, so the compression is directly readable at the substrate-output layer. Prior work documents the fine-tuning-induced hallucination phenomenon at two other layers — behaviourally (Gekhman et al. 2024) and at the activation layer (Shen et al. 2025). What this paper adds is the output-layer observable together with a dose-response across cumulative gradient passes, a compute-matched data-variation control, and the bridge from the RLHF-specific finding to plain LoRA fine-tuning.

Scope. The empirical claims are restricted to the silicon substrate, and specifically to fine-tuning of Qwen2.5 base models on a controlled invented domain. The cross-substrate extension — whether the same architectural distinction holds for biological memory systems — is developed in Paper 6 and is deliberately out of scope here.

Empirical findings

Three experiments on a 47-fact invented knowledge domain (Zorbetik), Qwen2.5 base models at 3B and 7B scales, LoRA fine-tuning budgets from 5 to 100 epochs plus a paraphrase-augmented variant:

• Cloze retrieval gap: ICL outperforms LoRA-FT by 16–28 percentage points across capacity scales
• Application degradation: FT-trained models actively degrade application accuracy below the no-context baseline
• Margin trajectory: M_pos0 climbs from 5.46 nats (ICL) to 17.85 (raw FT, 30 epochs) to 20.60 (100 epochs); under the three-seed compute-matched control, from 14.82 at ~1,410 gradient steps to 17.69 at ~7,050 — monotone with cumulative gradient passes. Definition: CR_pos0 = log(p_top1 / p_top2) at the first answer token; large positive values indicate that the top candidate dominates by many natural-log units, i.e. the distribution has collapsed to a single route. ICL values around 5 indicate the model still holds plural alternatives; FT values above 17 indicate effectively no alternative considered.
• Entropy collapse: position-0 entropy drops from 0.32 (ICL) to ≈0.00 under every FT regime tested, including the paraphrase-augmented one that slows the margin's growth
• Data variation is protective: in the three-seed compute-matched 2×2, paraphrase-augmented FT holds the margin narrower than raw FT at every compute level — 10.30 vs 14.82 at ~1,410 steps (4.5 nats), 15.37 vs 17.69 at ~7,050 (2.3 nats). Cloze accuracy holds flat at 86–88% across all four cells, so the margin–accuracy decoupling survives
• Installation is gated by copyability, not voice: in a voice × copyability 2×2 replicated on Qwen2.5-7B and -3B at matched gradient passes, any format that withholds the answer from the training input installs the facts (97.5–100% held-out recall) while any format that makes the answer copyable installs nothing (0%), whether the user turn is interrogative or declarative
• A defended prior, not an immune one: a fabricated fact fine-tuned to perfect recall (40/40) is still overridden by a contradicting in-context passage on 40% of items, and the items it loses are not the ones with the narrower answer-token margin (n=40, single seed, one model family)

Full per-condition results

Condition	Size	Cloze acc.	Application acc.	log(CRpos0)	Hpos0
No-context baseline	3B / 7B	~3% / ~5%	~12% / ~18%	—	—
ICL (facts in prompt)	3B	~62%	~28%	5.46	0.32
ICL (facts in prompt)	7B	~74%	~41%	5.46	0.32
LoRA-FT 5 epochs	3B	~28%	~6%	~10	~0.05
LoRA-FT 30 epochs	3B	~38%	~5%	17.85	≈0.00
LoRA-FT 30 epochs	7B	~58%	~9%	17.85	≈0.00
Paraphrase-aug FT 30ep (single seed, not compute-matched)	7B	~62%	~14%	21.12	≈0.00

Green: ICL preserves calibration (high entropy, moderate margin). Red: FT regimes collapse the distribution (entropy → 0, margin rises monotonically with gradient passes). Application accuracy on FT is worse than no-context at both sizes: the application-degradation result. The paraphrase row runs ~5× the cumulative gradient steps of the raw 30ep row beside it, so its position reflects compute rather than paraphrasing; the three-seed compute-matched comparison reverses the ordering, and it is that comparison the paraphrase claim rests on.

Applied consequences

• Confident hallucination: FT-trained models hallucinate confidently because alternative routes have been compressed beyond reach — the uncertainty signal that would flag them is gone
• Agentic uncertainty: agentic systems cannot reliably represent uncertainty when built on FT-only substrates; the substrate-level signal that would carry it has been compressed away
• RAG vs FT at the substrate level: RAG operates in ICL-mode and preserves calibration; FT compresses it as a structural consequence
• Long-context agentic conversations (Claude Code, Cursor, multi-turn agents): inherit ICL's calibration properties for free, since each turn re-evaluates the full context with no weight update
• Hybrid ICL+FT architectures: weight-resident knowledge for cheap retrieval, in-context reasoning where calibration must survive, at bounded context-window cost — with the bound set by the conflict result above, since a hybrid that counts on retrieved context reliably correcting a fine-tuned error is counting on too much

Caveats (scope conditions)

FT tested in both LoRA and full-parameter modes (3B + 7B); the two modes differ in rate of collapse, but they also use different learning rates per standard practice and no LR sweep was run, so that rate difference is not cleanly isolated from the LR confound. Single random seed in the single-budget runs; three seeds in the compute-matched and full-parameter ablations (42 training runs). Two model sizes span less than one order of magnitude. One invented domain (Zorbetik). One extended-depth seed outlier is flagged in the paper, so single measurements at high training depth warrant more caution than short budgets. The single-seed comparison in which paraphrase-FT appears to compress further than raw-FT carries a gradient-step confound; the multi-seed compute-matched control reverses the direction, and it is that comparison the paraphrase claim rests on. The conflict probes are n=40 per probe, single seed, one model family, and on one of them the exact split is adjudicator-dependent while the direction is not.