A Per-Token Measurement Standard for Route-Competition in Language Models
Pødenphant Lund, T. (2026) · Preprint · v5 live on Zenodo
A per-token entropy readout of the output distribution, and the conditions under which it is trustworthy.
The formal account of what bounded choice costs is mature: evidence accumulates to a threshold (Ratcliff 1978), the accumulation is derivable from an information-cost constraint (Hébert & Woodford 2017; Jung et al. 2019), and the threshold moves with urgency. This paper adds no axiom to that account and claims no new mechanism. It adds an instrument. In those models every quantity is inferred from an outcome — a choice, a latency, an error rate — and the competition that produced it is reconstructed by fitting a model to that scalar. An autoregressive substrate exposes the competition at every step of the accumulation, once per token, without fitting anything.
| DOI (concept) | 10.5281/zenodo.20012654 |
| DOI (this version) | 10.5281/zenodo.22071301 (v5) |
| Status | v5 live on Zenodo, 2026-08-23 |
| Length | ~12.8k words main text; the remainder in Supplement S3 within the paper |
| Author | Tomas Pødenphant Lund [ORCID] |
Note on this version
The substrate-universal mechanism claim of earlier versions is withdrawn on grounds of priority, not of error. The formal account it named already exists — sequential-sampling models, their information-cost formulations, and resource-rational analysis — and the paper now stands on that work and credits it. The cross-substrate reading is retained as motivation only. Several claims are downgraded in status; the measurements are unchanged.
The position, stated once
The paper claims no substrate-universal mechanism. What it offers is narrower: a substrate-specific instrument for investigating substrate-general principles of evidence accumulation that were established elsewhere and by other people. Empirical support is a small number of contrasts in language-model substrates that meet the standard below; extension to other substrates is an agenda, not a result, and nothing in the paper depends on it.
Whose formalism
Landauer’s principle sets the thermodynamic floor under any computation that reduces informational entropy. Ortega and Braun (2013) give the thermodynamic formulation of decision cost. Hébert and Woodford (2017) derive sequential evidence accumulation from an information-cost constraint — the bridge between accumulation-to-threshold and an information-cost account, built eight years before this paper. Jung, Kim, Matějka and Sims (2019) obtain, from a Shannon-capacity constraint, the discrete and persistent commitment this paper measures. The measurement is the contribution; the formalism is theirs.
The quantity
Friction is used throughout as a local name for one measured quantity: the Shannon entropy in nats of the next-token distribution at position t, computed over the returned top-k after renormalising to sum to one. It is a label with a formula and a unit attached, not a theoretical primitive and not a proposed mechanism. Where a result instead uses the competing-routes ratio or count, that is stated at the result.
The measurement standard
Four constraints follow from the readout being a thresholded count over a returned top-k list, and they determine which claims the measure can carry. They are stated with the methods rather than among the limitations.
- The readout class is not ours. Counting candidates above a probability floor is the operation the decoding literature performs when it truncates a distribution; CR differs only in being read rather than sampled from. Holtzman et al. (2020) locate unreliability below the nucleus edge and Fu et al. (2026) locate instability within the 0.1–0.9 band. The cutoff sits where the two meet, which is why the constraints below are properties of the design rather than accidents of a run.
- Levels are run-relative; contrasts must be demonstrated. Serving the same model the same prompts through different providers returned mean CRs of 1.24, 1.68 and 2.17 — a spread larger than most effects the paper reports — at the same nominal weight precision. A contrast is portable once it has survived a threshold sweep, a threshold-free surrogate, or a second serving; otherwise it is provisional.
- Entropy is the primary quantitative readout. A count over five candidates takes six values, and on windows of three tokens or fewer its median standard deviation is zero while the entropy of the same distribution still varies. The count is a step function at a threshold of 0.10, where returned probabilities are least stable, and it carries no identity with an information cost. Entropy is continuous, threshold-free, and defined on the same returned distribution at no additional cost. Where the two disagree, entropy is the measurement.
- No absolute-level gate. An earlier version proposed screening on absolute mean CR. That is an absolute comparison on a run-relative quantity, and applied to this corpus it admitted measurements carrying no variance at all. What it was meant to prevent is checked directly instead, by window length and by whether entropy varies across the response.
Each constraint was forced by a measurement rather than chosen: a provider comparison, a counting error in the paper’s own implementation, a corpus-wide check of window lengths, and a change of threshold convention. The audit is Supplement S2.
The ledger
Every empirical signature is sorted by that standard. Two meet it; one is withdrawn outright; the rest are reported with their direction as the claim and their magnitude as provisional.
| Signature | Standing |
|---|---|
| Decision-framing elevation at onset | Directional, replicated across two families and two stacks |
| Temporal-distance contrast | Cross-served on three servings, and no longer a null |
| Cross-serving agreement | Cross-served — an instrument property, not a theory signature |
| Friction ceiling | Single-provider, surrogate-backed; not cross-served |
| RLHF dispersion compression | Conjecture — direction only; magnitude withdrawn |
| Commit-position gaps | Single-serving, not cross-served |
| Trailing-task dropout under load | Single-serving; not on the standard’s quantity |
| Hysteresis Index | Single-serving, count-based, window with median SD zero; direction only |
| Parse-versus-generate sign | Withdrawn — arms served at different quantisations with unequal n |
The two contrasts that meet the standard
- Decision-framing elevation at onset. Moving a prompt from a knowledge framing to a decision framing raises first-five-token entropy. On 30 content-matched pairs, +0.273 nats, t = 6.564, df = 29 (Cogito-671B); +0.318 nats, t = 6.91, n = 25 on Llama-3.1-8B-Instruct through a different serving stack, with the matched base arm null (−0.017 nats, t = −0.70). The effect is localised to the onset window; over the full response it is less than half. The manipulation varies four properties at once — it also introduces a temporal horizon, lengthens the prompt by a factor of 2.5 in median word count, and introduces the second person in 30 of 30 pairs against 0 of 30 — so what is measured is elevation on decision-framed prompts as a bundle. The crossed design that would separate them is specified in Supplement S2.
- Behaviour-versus-friction dissociation under temporal framing. On MMLU-Pro (n = 200, Qwen3-235B) across three servings, accuracy rises monotonically with temporal distance by 11.0, 5.5 and 5.0 percentage points, while the entropy contrast is 0.005–0.02 nats. Read over a matched first-200-token window the gap excludes zero on two of three servings, with the sign agreeing in all three and in eight of nine pairwise contrasts. The servings do not agree on which contrast carries the effect, and no single coherent horizon effect is claimed. A fourth arm holding speech act constant does not resolve the design into one factor. The reading is route-selection rather than a graded temporal horizon; what is withdrawn from earlier versions is the stronger claim that the friction contrast is zero.
The friction ceiling
The readout does not separate a correct commitment from an incorrect one. Confident-right (n = 48) gives mean entropy 0.334 nats [0.312, 0.358] against confident-wrong (n = 52) at 0.346 [0.327, 0.365] — a difference of −0.012 nats with overlapping intervals. Uncertain-right and uncertain-wrong are likewise indistinguishable (0.415 against 0.394). The profile-shape claim is measured rather than asserted: the confident cells decline at indistinguishable rates (−0.167 [−0.208, −0.125] and −0.180 [−0.229, −0.128]) while the uncertain cells’ slopes cover zero.
Friction measures the cost of a commitment, not the correctness of the committed outcome, and that is a ceiling on any selector built on this readout. It rests on one provider and the first 30 tokens, corroborated by two threshold-free surrogates but not cross-served. The correct answer is present in the top-k distribution in roughly 80 % of confident-wrong cases, passed at lower friction than baseline rather than committed to: retrieval succeeds, commit fails.
Scope and non-claims
- No claim about biological substrates. The readout is defined for one substrate; any homology is a companion agenda.
- No cross-substrate gradient. Friction as defined here has been measured only in language-model substrates.
- No claim that CR is superior to standard uncertainty measures. On labelled data it is at best on par with Shannon entropy and the top-1/top-2 margin as an error predictor, and weaker than both on a larger free-form benchmark. It is used because it is the most direct read of the race-model primitive and free to compute, not because it out-predicts them.
- No controlled RLHF dose-response. Reactance is observed to scale with RLHF intensity across different models; isolating RLHF from architecture, scale and training-data confounds requires a within-model fine-tuning experiment that has not been run.
- Direct-answer generation only. The formalism is defined for models that produce a response as one continuous token sequence. It does not cover explicit reasoning models, whose APIs return per-token logprobs for the concatenation of reasoning and answer phases without routing flags.
Companion papers
- Paper 0 (BFT) — the framework this paper is read within. The relation was written BFT ⊂ FT; that wider claim is what is withdrawn here.
- Paper 3 (Friction-Guided Inference) — applies the readout to inference-time pipeline selection.
- Paper 2B — what cumulative gradient pressure does to the same distribution.
- Paper 21 — the readout used as a measurement model for a contested metacognitive phenomenon.