“It isn't knowledge. It's just information.”

You have heard it. A language model does not convey knowledge. It just makes something up from probabilities, text which we then assign a meaning to. Ask it again and it may answer differently. It may say something nobody has ever meant. So: information, not knowledge. A computer scientist I argue with on LinkedIn says it often, and he is not alone.

What is the criterion?

Take the argument seriously. What is supposed to make something knowledge? Not the content. The same sentence can be knowledge in one mouth and information in another. It is reliability: information that can be reliably reproduced is knowledge. Good criterion. It just fails in two places.

First place: the same holds for you

Start with yourself. How many animals of each kind did Moses take on the ark? Two, most people say. A moment later: it was Noah. The feeling of knowing was there the whole time, and it was exactly like when you actually know. The feeling does not discriminate. You can be confident and wrong, and it feels like being confident and right.

It is the same in the model. When it is confident and right, and when it is confident and wrong, it looks the same from inside. The numbers are equally high. The difference is whether there exists a counter-answer in the model that could have objected. If it does not exist yet, the first answer wins without resistance. So the model is most confident exactly where it can do least. That is the Dunning-Kruger effect, in the machine.

So if reliable content were what made something knowledge, most of your knowledge would not be knowledge either. The criterion hits the human as hard as the machine. It cannot be what separates them.

Second place: what can be measured is confidence

The answer can change, that is true. I have measured it. Same question twice, all randomness off, and the verdict changes in about one case in six.

But it does not happen at random. A language model writes one word at a time and computes the probability of every possible next word. Those numbers can be seen. Where one answer is far ahead, it answers the same every time, including when you rephrase. Where two answers sit close, it switches. In one of my runs the disagreement was zero on the easy items. Every switch sat where the gap between the two best answers was small.

Be precise about what is being measured. It is not whether the answer is true. It is how confident the model is, and so whether the answer can be reproduced. Stable and wrong exists, as above. But reproduction was what the criterion asked for, and it can be read off in advance, answer by answer. We can reliably say when the model will say the same thing again. And we can see where.

So what is knowledge?

What makes the difference for you happens over time. You become reliable by being wrong and finding out. The missing counter-answer gets built because you stood in the situation, answered wrongly and took it with you. That is what slowly makes your confidence track the truth.

My proposal: knowledge is information plus a system that has used the information over time and kept how it went. In my theory it is called storing your experience forward. What you used today stands in you tomorrow.

And then you can see exactly where the model stands. It has done that, during training. The weights are its experience of being wrong and corrected, and there its confidence roughly tracks the truth. But it stops the day it is put into use. It learns nothing between two conversations. It can remember the conversation, if you store it for it, and then it says "as we discussed yesterday" and holds a stable persona. But none of that is in the weights. Swap the transcript and it is someone else. That is memory, not experience.

So the argument is right about everything the model meets after training. It is information with the signature of knowledge and without its grounds. It knows what it has been wrong about. And it has not been wrong about anything since it was trained.

And it can be built

Here the argument slides from a description into a claim it has no cover for. That the models, as they are now, rarely know anything in that sense is fair. That a language model cannot know anything is something else. What is missing, we already run for it, just from outside: train a model further and you write experience into its weights. Run that on the model's own conversations and it stores forward.

I have done it in the small. Train a model on stories where rewards far out more often fail to arrive, and it weights the future less by itself, with nobody telling it to. Teach it things that are expensive to learn, and it keeps away from them, with nothing dangerous mentioned. Experience, stored forward, becomes direction.

But it cannot be done too fast, and it cannot be done too hard. Write something into the weights in one go and it compresses the alternatives, and the model becomes cocksure, also when it is wrong. The new has to be mixed with the old, a little at a time. That is exactly what your brain does. It does not write the day's experience in straight away. It does it at night, mixed with everything you already knew. Learning takes time, in the machine and in you, and for the same reason. It is not a wall. It is a step that has to be run at the right pace.

An aside on "something nobody has meant"

Notice that the argument has an ironic streak. Here it is held against the model that it can say something nobody has ever meant. At the other end of the debate the opposite is held against it: that it cannot come up with anything new, because everything is assembled from what already existed. Both cannot be true, and in fact neither is an argument. Everything humans make is also assembled from what already existed. That is almost the definition of creativity: something new that still connects to the known, because otherwise nobody can use it. What that looks like from the inside has its own page. And "it just makes something up from probabilities" has one too.

Where the line falls

The argument is right about the models as they are. It is wrong about what could be. And it puts the line in the wrong place. The line does not run between information and knowledge, as if those were two kinds of content. It runs between a system that learns from using its knowledge and one that does not. You do. The model does during training and then stops. And that is a setting, not a condition.

Where to next: “AI is only trained on words.” · The less you know, the surer you get · Does AI have an inner experience? · all topics