Reading creativity from the inside

Paper 31 · And why temperature is a dead end · Read on Zenodo

Creativity is not something a language model has. It is a property of how it runs the race between the possible next words. Every time the model writes a word there is a small contest between candidates: some routes are wide motorways (the obvious word), others are narrow paths (the surprising one). And that contest can be read from the inside, in the model's own probabilities, at no extra cost. That gives an answer to a question many people ask the wrong way round: why doesn't turning up the temperature make a model creative?

What others have already shown

Internal per-word measures already exist in the neighbouring literature, and that should be said up front. Yang, Li and Holtzman showed in 2025, with their "branching factor", that alignment (the training that makes a model helpful) concentrates the distribution, and most at the first words; Mohammadi found lower token entropy in aligned models in 2024. So the opening here is not "nobody has read from the inside". It is that nobody has read it as an instrument for creativity, held against judged novelty times value, and used to predict how varied the output will be.

Finding 1: creativity is an inverted U, and temperature does not reach it

Push the model toward the new, with prompts that suppress the obvious answer or inject distant material (what I call changing the landscape), and novelty rises steadily. But value and coherence fall off a cliff once you push too far. So what makes an answer usable, novelty times value, peaks in the middle: too narrow is banal (the obvious always wins), too broad is gibberish (a route too distant to hang together wins). It is the same inverted U as elsewhere in the theory, the optimum in the middle. And it says something rather nice about freedom: usable freedom is being free within your context, not free from it. Context is not the enemy of creativity; it is the constraint that turns novelty into value.

The sharp point: turning up the temperature does not do this. Temperature does not raise novelty. It merely wrecks value and coherence and goes straight to gibberish, without ever passing through the creative region. Geometrically: the landscape lever moves along the curve, temperature goes off it. That is the precise reason "turn up the temperature" never makes a model creative.

There was also a hint that the optimal breadth depends on model size (small models need pushing further, large ones peak early), but that part is shaky: it depends on which judge is scoring, so it stands only as a hint.

Finding 2: the internal fingerprint of mode collapse, graded and not universal

In two of three model families (Qwen, Mistral) the aligned model has less competition between routes exactly at the moment of choice, the first words, than its raw base sibling. The phenomenon itself, that alignment concentrates and most at the start, was established by Yang and colleagues. The contribution here is replication with an independent instrument plus a refinement: it is graded and recipe-dependent, not universal.

The third family (Llama-3.1) reverses it: its aligned model is the least collapsed of them all, broadest at the moment of choice and most varied in output. That is a genuine counterexample to the picture in which alignment always sharpens. The satisfying part is that across the three families, how much alignment narrows the routes tracks how much it collapses the actual output diversity. So the measure correctly reports how mode-collapsed a model is; Llama-3.1 is simply barely collapsed at all. That third family was exactly what a reviewer demanded be run, and it reversed the prediction and made the finding stronger.

On the U-curve this puts a heavily aligned model on the narrow left arm. The lever is to reopen the routes at the moment of choice, not to touch the temperature. It also gives a testable prediction: it should be possible to fine-tune a base model to be more creative, not by simply making it broader (which tips it into gibberish), but by preserving the breadth at the moment of choice while installing the coherence and steerability a base model typically lacks. That experiment has not been run.

Finding 3: how varied the output will be, from a single pass

You can predict how varied a model's output will be from the competition at the moment of choice in one single generation, where existing methods need twenty samples or a special prompt. That finding actually got stronger along the way. Yang and colleagues looked for the same correlation themselves and found none. The reviewers demanded that the explanation for the difference be tested rather than asserted, so I ran the ablation on my own data, and it disproved my first explanation: the correlation survives their own lexical measure perfectly well. The real explanation is range restriction: my design manipulates the breadth, so the predictor has real variance, whereas theirs had no breadth manipulation at all. And the one model where the correlation vanishes for me is precisely the one with the most compressed range. A positive finding where prior work reported a null, now with the tested explanation instead of the convenient one.

How certain is it

It was stress-tested with four independent judges. The U-shape held under three of four; the fourth rewards gibberish and dissented on one model, and that is stated openly in the paper. The size story turned out to be judge-dependent and was toned down accordingly. The first review round returned major revision, whose main demand was to run the third model family, and that was exactly what reversed the prediction and forced the graded rewrite above. The final round came back as accept, with both reviewers saying the reversal had strengthened the paper.

Where it points

The deeper point is that choosing a route which deviates from the statistically most likely one is the same event behind two things we normally study separately: creativity, where the value of the deviation is novelty, and error correction, where a confidently wrong default is overridden by a correct alternative and the value is correctness. Both require a live non-default competitor at the decision point. The hard, unsolved case is the confidently wrong one: a flat signal with a wrong commitment, where there is no visible competitor to open. One caveat carries the rest: a mechanism is a different thing from a finished tool, which would need calibration per model and per dataset.

Read the paper

The full article is freely available on Zenodo (concept DOI 10.5281/zenodo.20840284):

Read on Zenodo → · Technical version · Dansk version

Related on this site:

• Paper 1 (Friction Theory): the framework of competing routes that this measure reads.
• Using AI well: the practical end, how to get a model to think in a new direction.