Routes, not modalities

Paper 26 · Why an image doesn't always help · Read on Zenodo

There is a widespread rule of thumb: add a picture and people learn better. You see it everywhere, in teaching, in slides, in e-learning, and it holds often enough to become a truth nobody questions any more. But it also gets used in the one case it was never tested on: the picture that has nothing to do with the text. The question here is what happens when the extra image does not match what the words say. The answer points to something more general than pictures and text, and it can be measured directly as friction in a language model.

The core idea: it is the traces, not "picture vs. text"

The usual explanation credits the crossing of two sensory channels, that the combination of picture and text is itself the magic. The idea here is different: the active ingredient is not the modality but how many traces point at the same thing.

Think of it as a small race inside the system: each input, a word or an image, sets a trace running toward an answer. If the picture and the text point at the same thing, they stack up and strengthen the memory. But if the picture points nowhere, it does not vanish for free. The system still has to run a small race to establish that it was noise, and that costs. Length by itself starts races; irrelevant length is pure cost. A modality is just one reliable supplier of traces, not the magic in itself.

What the idea does not claim

Several things have long been known and are not mine. That you get better at recognising images by training on many variants of the same image is the foundation of all image recognition (it is "many traces to the same image", trivial). That picture plus text can beat text alone is the starting point of the whole field. And that irrelevant material costs, that a language model is disrupted by irrelevant text and an irrelevant image makes a vision-language model worse, has already been shown by others. The negative phenomenon itself is not my discovery.

The same goes for the race idea. Sensory research has, since the 1960s and 80s, formalised exactly this question: do extra channels help because they race (the fastest wins), or because they sum? The psychologist Jeff Miller gave a standard test for it in 1982, the "race model inequality", and the gain from an extra, congruent channel has its own name: redundancy gain. I inherited that formalism, I did not invent it, and I say so plainly in the paper. The contribution is the explanation and what governs it, not the phenomenon.

What I actually find

• An image that merely repeats the text does not help. When the model already has the name in the text, a matching picture adds almost nothing. It held across three different setups and two very different model types; the arrow even pointed slightly the wrong way. In the language of sensory research: there is no redundancy gain, in exactly the place where humans reliably get one.
• A decorative, irrelevant image costs. Hang a foreign image on a question the text can already answer, and it becomes measurably harder for the model to retrieve the answer. That is the race it has to run to reject the noise. It is the mechanical account of why a lifeless clipart image feels dead: friction with no gain.
• Recognising a symbol is not the same as being able to use it. The model can learn "this symbol means substance X" almost perfectly, and still not be able to use the symbol to retrieve X's property. Learning a connection and being able to draw on it are two different things.
• More traces also give better retention. Facts learned through many different traces seem to resist being overwritten when the model is later trained on something else entirely. The direction is the same across several experiments, but the effect is small, so it is not a finished finding yet.

What governs the cost? Size

I dug into the decorative image to find out why it costs. Two things came out: the cost grows with how many "chunks" the irrelevant image takes up (a pure crowding effect, the same happens with an equally long stretch of irrelevant text), plus a small extra term specific to images.

But the decisive part is that how much it costs depends on the size of the model. The large model looks at the irrelevant image just as much as the small one, yet does not pay. Run the same model type in two sizes and the cost collapses about tenfold from the small to the large. The large model simply has the bandwidth to run the rejection race for free and still solve the task; the small one does not. That fits the whole series' through-line: friction is capacity-relative. The race runs on every model, but only the one under resource pressure pays measurably. (The failed explanation, about image tokenisation, is reported as tested and rejected, not swept under the rug.)

A further sharpening: it is the structure, not the image

On a harder task, finding a path through a web of connections, something turned up that sharpens the whole point. The model can barely answer when the information is given as running text in random order, but it can when the same information is laid out together in a structure, either a bulleted list or a diagram. The active ingredient is the co-location, that the things sit together, not whether it is an image. And here the order flips relative to humans: for us a diagram gives a free overview, but for the model the structured text is as good as or better than the diagram, and the diagram is even the most expensive (the model has to "read" the image, and it gets worse the larger the web). It is an early, indicative finding, but it points the same way: it is the coincidence of traces, not the modality.

Why it is interesting

The first two findings are exactly the patterns known from human learning psychology: that repeated, redundant material does not help (the redundancy effect), that having to watch two things at once subtracts rather than adds (the split-attention effect, from John Sweller and colleagues), and that decorative images that do not belong lower learning (what Shannon Harp and Richard Mayer called the "seductive details" effect). I do not test humans. I ask which of these human patterns reappear in a non-human system, as measurable friction. That places the paper in the same family as the other "same pattern across substrates" papers, the social reactions in Paper 23 and the visual illusions in Paper 24. Others have recently tried to carry learning-psychology concepts over to language models too; what is new here is that I give the mechanism (the race) instead of just an analogy, measure it as friction, and find the two things others do not have: that the cost is size-dependent, and the retention.

Where it stands

The groundwork is done and the mechanism is settled: the clipart cost is a capacity-gated rejection-race cost, it collapses as the model grows, and the earlier image-tokenisation explanation is tested and rejected. The retention pattern points consistently the same way across five setups. It is still small-scale (few model sizes, few facts), and I say so plainly. The paper has been through external critique, and it is a hypothesis with a real pattern behind it, not a settled case.

Read the paper

The full article is freely available on Zenodo (concept DOI 10.5281/zenodo.20679145):

Read on Zenodo → · Technical version · Dansk version

Related on this site:

• Paper 16 — The Physics of Learning: why information-dumping teaches no one anything, the same mechanics from the other end.
• Paper 1 (Friction Theory): the framework of competing routes under bounded resources.