~ $ cd projects/next-word && cat README.md
The next word
A language model does one thing: given the words so far, it says what may come next, and how likely. Everything else is how well it says it. This one says it as badly as possible, so that the whole of it can be seen. It reads a text, counts which word followed which, and throws dice weighted by the counts.
Press next word and it throws the dice once. Press a word it offers and you are the dice. Press write and watch it go.
Things to try
- Count it yourself. In the eight short sentences, the is followed by cat three times, dog three times and car twice. Those are the chances it offers, and there is nothing else inside.
- Turn the temperature down to 0 and it always says the likeliest thing, and soon goes round in a circle. Turn it up to 2 and the unlikely words get their turn. It is the same dial the real ones have, doing the same sum.
- Let it read this website, and look two or three words back instead of one. One word back it babbles; three words back it starts quoting the pages it read, because so long a context has usually been seen only once. More memory needs more text: that trade is the whole history of the field.
- Start from a word it never saw. It takes the nearest word it knows, by counting letters. A real model avoids the problem by cutting words into pieces.
What it is not
A real model does not keep a table. It would need a row for every run of thousands of words ever written, and nearly every row would be empty. It squeezes the table into a network that answers for contexts nobody has seen, because they resemble ones somebody has — which is where everything interesting, and everything that goes wrong, comes from. But the question put to it is this one, and the answer is used this way: a list of chances, a dial, and dice.
Where it comes from
The first version is from February 2026, made to be shown in a talk rather than explained: the original, with its Catalan texts. This one is written again for this site, with nothing underneath it.