Live demo · NanoScale-LM
Everything on this page is genuine output from the trained checkpoint; every token, probability, byte count and surprisal score. Each panel shows what the model produced and explains what it is doing to produce it.
The model predicts a probability for every one of its 16,384 tokens, given everything before. To write the next word it samples from that distribution, appends the result, and repeats: one token at a time, each one conditioned on all of the previous ones.
Temperature reshapes the distribution before sampling. Divide every logit by
T: below 1 the peaks get sharper and the model plays safe; above 1 it
flattens and the model takes risks. At T = 0.2 you will see repetitive,
confident, slightly dull text; at T = 1.0 you will see invention and
occasional nonsense.
Seed fixes the random draws. Same prompt, same temperature, same seed → byte-identical output, every time. That is what makes any of the measurements on this project reproducible.
This model has 40.4M parameters and was trained for 3.2 hours on a laptop GPU, on children's stories. It writes children's stories. Ask it about anything else and it will politely continue writing children's stories.
Step through a greedy continuation. At each step you see the six most likely next tokens and their probabilities: the real distribution, not a summary.
This is the entire computation a language model performs. Everything else: attention, the feed-forward blocks, sixty-four million multiply-accumulates per token , exists to produce this one list of numbers.
Watch how the confidence changes. After "Once upon a" the model is nearly
certain the next token is " time": the phrase is fixed. A step later, when
it has to choose what the story is about, probability spreads across many
plausible options and no single one dominates.
That spread is the model's uncertainty, and it is measurable. Averaged over a corpus it is cross-entropy: the number every training curve plots, and the number that decides how well the model can compress text in panel 3.
Shannon proved that a symbol the model expects with probability p can be
written in −log₂(p) bits. So a model that predicts text well
is a compressor: feed its probabilities to an arithmetic coder and you get a
real, smaller file back.
These byte counts are measured, not estimated. Every one survived an encode → decode round trip that returned the original text byte-for-byte.
Now switch between the three samples. On children's stories the model needs
about 0.6 bits per byte, roughly four times better than xz.
On encyclopedia prose it needs six times more. On source code it does worse than
storing the raw bytes.
That collapse is the whole point rather than a flaw. The model is not a general compressor; it is a compressor for one distribution, and it buys its advantage there by being useless everywhere else. Which is exactly the trade you want when the thing you are archiving is a billion log lines that all look alike.
Each line scored by how many bits the model needs to encode it. No labels, no rules , just "how unlike my training data is this?". Sorted most-surprising first.
Darker = more surprising. Hover a token for its exact cost in bits.
This is the same number as panel 3, not summed. Compression asks "how many bits for the whole passage"; anomaly detection asks "which parts cost the most". One forward pass answers both.
The threshold is the 95th percentile of ordinary in-domain traffic, so it is calibrated by the data itself rather than hand-tuned. Anything above it gets flagged.
The interesting row is not the gibberish. Look at "Tom picked up the quantum entanglement and put it in his pocket": perfectly grammatical, right register, right characters. It scores high because one noun phrase is impossible in this world. That is semantic detection, not spell-checking.
For a model trained on one system's logs, this is a definition of "anomalous log line" that needs no labels, no regex library and no maintenance; it just needs the model to have read enough normal traffic.