Catch the next token.: sources & model

Open a genuinely trained miniature Transformer. Change token tiles, inspect real attention and vector calculations, sample a continuation, and test the limits of a confident prediction.

What supports the explanation?

What this model assumes

1. Original seven-token symbolic task, one reduced Transformer block and at most five input positions. No general English, external memory, tool use or remote inference.
2. No encoder, cross-attention, multiple heads, normalization, rotary positions, dropout, quantization or tied input/output matrix.
3. All 18 sequences are training data. The training-task score cannot distinguish memorization from a reusable selection rule or establish broad generalization.
4. Attention rows mix value vectors. The selected connection-removal operation is artificial and does not claim a comprehensive causal interpretation.
5. The default trained distribution is often very sharp; a temperature change does not promise a visibly different token on every sample.
6. The recorded training curve is not a live training animation. Context edits and sampling leave the checkpoint fixed.
7. Token/position/vector shapes and the exploded stack are original mathematical visualizations, not physical wires or brain anatomy.
8. Source reading, complete numerical comparisons and gradient checks do not certify learner understanding or browser/device/export behavior.
9. A deliberately reduced trained Transformer: This original decoder has one block, one head, width 8, feed-forward width 16, 695 parameter slots, learned positions, separate input/output matrices and no normalization or dropout. It is a symbolic color-selection model, not an English chatbot or a replica of a modern large model.
10. Exact vocabulary and tokenization: IDs 0–6 mean <bos>, <eos>, red, blue, green, first and last. The editor canonicalizes whitespace and rejects unknown labels. An ID is a row selector; its numeric magnitude is not its meaning or probability. The checkpoint and vocabulary must stay matched.
11. A separate subword arithmetic card: The optional red redder exercise begins with characters and retains the space. Four chosen merges produce red, space, red, der. This demonstrates that tokens need not equal words. Its hand-constructed vocabulary and ranks are not used by the neural checkpoint.
12. Queries, keys and values: For row vectors x, the model forms q=xWq, k=xWk and v=xWv. Each allowed attention score is q·k/√8. A stable softmax subtracts the maximum allowed score before exponentiating. Future positions have score −∞ and weight exactly zero.
13. Two different softmax operations: Attention softmax distributes mass over allowed context positions. Output softmax distributes mass over seven candidate next-token IDs. A 0.9 attention weight on first does not mean a 0.9 probability of emitting first.
14. The entire downstream path matters: Attention’s weighted value sum passes through Wo and adds to x. A position-wise ReLU network and a second residual produce z. Output logits are zU+b. The exploded stack shows actual vectors at one selected position; cell height is a declared compressed visual scale, not a physical dimension.
15. A worked value coordinate: For <bos> red blue first, the final attention weights are approximately 0.000858485, 0.003125844, 0.096708236 and 0.899307436. Multiplying them by value-coordinate-0 entries and summing gives approximately −1.957841407. Full-precision values drive the visualization.
16. A defined artificial intervention: Connection removal sets one allowed attention entry to zero, renormalizes the remaining row, and recomputes downstream layers with the original queries, keys, values and parameters. Removing the only remaining allowed connection is undefined. Replacing a token changes its vectors and normalization, so it tests a different intervention.
17. Attention is evidence, not a complete causal account: Published work debates which explanatory claims attention supports and how to test them. A heatmap alone, or one ablation, does not establish a complete mechanism. This workbench shows measured intermediates and specific changed calculations without inventing a natural-language inner monologue.
18. Causal prediction and shifted targets: All 18 six-token sequences supply five input–next-target pairs. At input position i, the current token is known, so attention to itself is allowed. Inputs at positions later than i cannot influence its logits. Showing a full training worksheet does not grant the model access to those future answers.
19. Teacher-forced training and autoregressive inference: Training uses the ground-truth preceding tokens while computing 90 next-token targets per full batch. Generation instead appends its selected output before the next forward pass. Ordinary inference neither computes a gradient update nor changes the checkpoint.
20. Temperature operates after logits: For positive temperature τ, output probabilities are proportional to exp((logit−maximum)/τ). Changing τ leaves current logits and attention fixed. Greedy choice is a separate operation, with exact ties resolved by lowest token ID. A different sampled token can change later context and therefore later attention.
21. Reproducible categorical draws: The educational 32-bit LCG advances state once for each random draw, then u=state/2³². Choose the first cumulative probability strictly greater than u. Seed 49 begins with 0.2550580417737365 and 0.7230514015536755. It is a classroom generator, not cryptography.
22. A sampling calculation you can do on paper: For the separate logits [ln 6, ln 3, 0], temperature 1 gives [0.6,0.3,0.1]. Temperature 0.5 gives [36/46,9/46,1/46]. This card is intentionally distinct from the trained model’s often much sharper distributions.
23. All training data are disclosed: The corpus enumerates three first colors, three second colors and two queries: 18 sequences. All are used in training, without a held-out split. The 18/18 answer-position result is a training-task check, not evidence of generalization or a reasoning benchmark.
24. Why loss has a nonzero floor: Earlier identical prefixes genuinely admit several next targets: three first colors, three second colors and two queries. The empirical conditional loss floor is (ln 3+ln 3+ln 2)/5≈0.578074352 nats per target. Near-perfect answer selection does not make mean loss over all five targets zero.
25. Recorded training, fixed inference: The downloadable original code initializes from NumPy PCG64 seed 49 and performs 2000 full-batch Adam updates in float64. The final loss is approximately 0.578083150. The ledger’s points are recorded values, not live training. The exact checkpoint and corpus have separately recorded SHA-256 hashes.
26. Where this analogy ends: Larger systems can have different architectures, richer pretraining, post-training, reasoning behavior, tools and context handling. A next-token interface does not settle what algorithms a learned computation implements. This tiny experiment neither demonstrates nor rules out those larger capabilities.

What has been checked

Analytical reference cases, conservation or transition invariants, finite drawing commands, bounded setup parsing, discovery and route integrity are checked automatically. These checks do not establish anatomical fidelity, learner outcomes or browser/device compatibility. Independent subject review, learner trials, comprehensive accessibility review and browser video encoding checks remain pending.

Each source supports the associated claim. Sources do not certify this implementation or its visuals.

About the cover illustration

Original rendering of actual vector stages from the trained 695-parameter color-selection Transformer. Layer spacing and common compressed value heights are display choices, not physical chip dimensions or an anatomical model.