It is reading this right now
One sequence from the batch it is learning from this very second. Each token is shaded by how much it surprised the model: expected unsure surprised. As it learns, the page should turn paler.
—
Learning curve
Cross-entropy loss per token on the text it trains on (line) and on text it has never seen (dots). Random guessing among 4,096 tokens scores 8.3. Lower is better; the climb down is what learning looks like.
Diary
At each age milestone it was asked to continue “Once upon a time,” and to write something unprompted. Nothing is edited. Read from the top to watch language arrive.
Talk to it
Give it the start of a story and it will continue, with the weights it has at this exact moment. It has only ever read children's stories, so it will be strange, and the younger it is the stranger. Be gentle.
The machine
Nobody connects to the laptop. It keeps one outbound WebSocket to a small Cloudflare Durable Object that fans its narration out to everyone here and passes your questions back. Checkpoints go to disk every five minutes, so a reboot costs at most that.
- architecture
- decoder-only transformer
- parameters
- —
- layers · width · heads
- —
- context
- —
- vocabulary
- 4,096 byte-level BPE tokens, trained here
- optimizer
- AdamW, warmup then cosine, grad clip 1.0
- learning rate now
- —
- corpus
- TinyStories v2 (Eldan & Li), 552M tokens of short stories written for small children
- host
- ThinkPad, AMD Ryzen 7 PRO 4750U, no GPU
- threads in use
- 8 of 16
- cpu now
- —
- die temperature
- —
- memory
- —
- one step
- —
- step
- —
- watching now
- —