Tiny world model generates games and stream ac plays them all on Pico 2W
Readme redone
Sebastian Naeb049f5d67401
diff --git a/README.md b/README.mdindex 604dc1e..3d0bac3 100644--- a/README.md+++ b/README.md@@ -1,20 +1,18 @@<p align="center"><img src="docs/media/banner.png" alt="tinyworldformer: games dreamed by a tiny int8 transformer on a Raspberry Pi Pico 2 W"></p>-Two small neural networks share one Raspberry Pi Pico 2 W. One dreams a game: it takes the current frame and a button press, and predicts the next frame and the reward. The other plays that game: it looks at each dreamed frame and picks the button. There is no game code on the chip; five games (Breakout, Doom-lite, Flappy Bird, Pac-Man lite, Pong) exist only inside the first network.+Two small neural networks share one Raspberry Pi Pico 2 W. The first one dreams a game: given the current frame and a button press, it predicts the next frame and the reward. The second one plays that game: it looks at each dreamed frame and picks a button. There is no game code on the chip. The five games (Breakout, Doom-lite, Flappy Bird, Pac-Man lite, Pong) only exist inside the first network.-+| | | | |+|---|---|---|---|+|  |  |  |  |++Every frame above was drawn by the dream network, and the buttons were pressed by the actor network.++## Why-Every frame above is drawn by the dream network, not by game code. The actor network picks the buttons.+I got interested in looped transformers, where one transformer block is applied several times with the same weights instead of stacking different layers. Reusing weights is cheap on memory, so I wanted to see if a looped world model could be small enough to run on a microcontroller. I picked the Pico 2 W because it is cheap and has an FPU, and small games because I could write the real versions myself and check every dreamed frame against them. Later I added a streaming RL agent so the board could also learn to play inside its own dream.-- [The dream and the actor](#the-dream-and-the-actor)-- [The games](#the-games)-- [How the world model works](#how-the-world-model-works): tokens, the loop, decoding, checking the dream, training-- [The agents: stream-x](#the-agents-stream-x)-- [Quantization and the Pico](#quantization-and-the-pico)-- [On the chip](#on-the-chip)-- [Flashing a game and an agent](#flashing-a-game-and-an-agent)-- [In the browser](#in-the-browser)-- [Reproduce](#reproduce) and [code layout](#code-layout)+Contents: [the two networks](#the-dream-and-the-actor) · [the games](#the-games) · [the world model](#how-the-world-model-works) · [the agents](#the-agents-stream-x) · [quantization](#quantization-and-the-pico) · [on the chip](#on-the-chip) · [flashing](#flashing-a-game-and-an-agent) · [browser](#in-the-browser) · [reproduce](#reproduce) · [code layout](#code-layout)## The dream and the actor@@ -25,17 +23,16 @@ Every frame above is drawn by the dream network, not by game code. The actor net| job | frame + button → next frame + reward | frame → button || architecture | looped transformer, int8 | MLP with two hidden layers, float32 || size | 26k–131k parameters | 9k–16k parameters |-| cost per frame | 2–4M multiply-adds | 9k–16k multiply-adds |+| cost per frame | 2–8M multiply-adds | 9k–16k multiply-adds || trained | on recorded real games ([how](#how-the-world-model-works)) | with stream-x, in the real game or inside the dream ([how](#the-agents-stream-x)) |-- **A closed loop.** On the chip neither network sees the real game. The actor only sees frames the dream drew, and the dream only sees buttons the actor pressed.-- **One core.** Both run on core 1 of the RP2350, one after the other, every frame. Core 0 handles USB and the display.-- **Learning on the chip.** A third network can replace the pretrained actor: a fresh stream-x agent that learns from scratch while it plays inside the dream, on the board.-- **Your turn.** Any button press hands control from the actor to you, so you play the dream yourself.+On the chip neither network ever sees the real game: the actor only sees frames the dream drew, and the dream only sees buttons the actor pressed. Both run on core 1 of the RP2350, one after the other, every frame, while core 0 handles USB and the display.++The pretrained actor can be swapped for a fresh stream-x agent that learns from scratch while it plays inside the dream, on the board. And any button press hands control to you, so you can play the dream yourself.## The games-Each game is a few hundred lines of numpy (`src/tinyworldformer/games/`). It renders to a small grid of token ids, and one frame always holds the whole game state. Hidden state is drawn into the frame: the ball's heading is part of its token, and Doom-lite has a minimap.+Each game is a few hundred lines of numpy in `src/tinyworldformer/games/`. It renders to a small grid of token ids, and one frame always holds the whole game state, so hidden state has to be drawn into the frame: the ball's heading is part of its token, and Doom-lite has a minimap.| game | frame | rules | random event | world model | exact frames, 2 loops ||---|---|---|---|---|---|@@ -43,15 +40,11 @@ Each game is a few hundred lines of numpy (`src/tinyworldformer/games/`). It ren| Doom-lite | 32×32 | raycast first-person view of an 8×8 maze; an enemy walks shortest paths to you; you shoot | the enemy's respawn | 55k params, 4.2M MACs | 83.6% (see below) || Flappy Bird | 24×24 | gravity, flaps, scrolling pipes | the next gap height | 46k params, 2.0M MACs | 100% || Pong | 20×32 | you against a CPU paddle that reacts late, first to 5 | the serve after a point | 46k params, 2.3M MACs | 99.45% |-| Pac-Man lite | 48×48 | 23×21 maze, 249 pellets, two ghosts on shortest paths, 3 lives | the ghosts' respawn | PACMAN_PARAMS | PACMAN_EXACT |--"Exact frames" counts held-out transitions where every cell of the predicted frame matches the real game, leaving out frames after a random event. A random event can only be sampled, not predicted. Doom-lite's 3D view has 768 cells, so one shaded wall pixel off counts the whole frame as wrong. Its game state is almost always right; see [checking the dream](#checking-the-dream).+| Pac-Man lite | 48×48 | 23×21 maze, 249 pellets, two ghosts on shortest paths, 3 lives | the ghosts' respawn | 131k params, 8.2M MACs | 86.4% |-| | | | | |-|---|---|---|---|---|-|  |  |  |  |  |+"Exact frames" is the share of held-out transitions where every cell of the predicted frame matches the real game. Frames right after a random event are left out, since those can only be sampled, not predicted. Doom-lite's 3D view has 768 cells and a single shaded wall pixel off counts the whole frame as wrong, so its number looks worse than it plays; the game state is almost always right (see [checking the dream](#checking-the-dream)).-Play the real games in a terminal: `uv run python -m tinyworldformer.games.play --game pacman` (also `breakout`, `doomlite`, `flappy`, `pong`).+You can play the real games in a terminal: `uv run python -m tinyworldformer.games.play --game pacman` (or `breakout`, `doomlite`, `flappy`, `pong`).## How the world model works@@ -59,21 +52,17 @@ Play the real games in a terminal: `uv run python -m tinyworldformer.games.play-- The frame is cut into square patches: 2×2 cells in Breakout, 4×4 in Doom-lite, Flappy and Pong, 6×6 in Pac-Man. Each game ends up with 36 to 80 tokens.-- A patch's token is the sum of one learned vector per cell. Each position inside the patch has its own table, so the order of the cells is kept.-- Every token also gets a position vector and the action's vector.+The frame is cut into square patches: 2×2 cells in Breakout, 4×4 in Doom-lite, Flappy Bird and Pong, 6×6 in Pac-Man. That gives 36 to 80 tokens per game. A patch's embedding is the sum of one learned vector per cell, with a separate table for each position inside the patch so the cell order isn't lost. Every token also gets a position vector and the action's vector.### One block, looped-- **One block:** a single transformer block (attention plus a ReLU MLP) with one set of weights, applied 1 to 4 times. The input is added back before every pass.-- **Every pass is trained:** the output after each pass is trained against the next frame, so at run time the loop count trades speed for accuracy.-- **On the Pico:** two loops.+The model is a single transformer block (attention plus a ReLU MLP) with one set of weights, applied 1 to 4 times, with the input added back before every pass. The output after every pass is trained against the next frame, so at run time the loop count trades speed for accuracy. The Pico uses two loops.-The animation shows one held-out Doom-lite step. After one pass the model draws an enemy that is not there; each further pass removes part of it: 92, 35, 12, then 0 wrong cells. The bars on its right are the share of held-out frames each loop count gets exactly right.+The animation shows one held-out Doom-lite step. After one pass the model draws an enemy that isn't there, and each further pass removes part of it: 92, 35, 12, then 0 wrong cells. The bars on the right are the share of held-out frames each loop count gets exactly right.-Below, one dreamed Doom-lite step decoded after 1, 2, 3 and 4 loops. Red boxes are cells that differ from the real rules.+Here is a dreamed Doom-lite step decoded after 1, 2, 3 and 4 loops, with red boxes on cells that differ from the real rules:@@ -81,34 +70,30 @@ Exact held-out frames by loop count:| loops | Breakout | Doom-lite | Flappy Bird | Pong | Pac-Man ||---|---|---|---|---|---|-| 1 | 99.68% | 59.8% | 99.99% | 98.47% | PACMAN_L1 |-| 2 | 99.99% | 83.6% | 100% | 99.45% | PACMAN_L2 |-| 3 | 100% | 85.9% | 100% | 99.60% | PACMAN_L3 |-| 4 | 99.96% | 84.7% | 100% | 99.53% | PACMAN_L4 |+| 1 | 99.68% | 59.8% | 99.99% | 98.47% | 79.6% |+| 2 | 99.99% | 83.6% | 100% | 99.45% | 86.4% |+| 3 | 100% | 85.9% | 100% | 99.60% | 85.7% |+| 4 | 99.96% | 84.7% | 100% | 99.53% | 84.4% |### Decoding-The model outputs a score for every cell and every token. Turning the scores into a frame takes three rules, shown above on Pong:+The model outputs a score for every cell and every token. Turning those scores into a frame turned out to matter almost as much as the model, and it ended up as three rules (shown above on Pong):-1. **Background** (walls, bricks, pellets, floor): each cell takes its most likely token on its own.-2. **Singletons** (the ball, the player, each ghost): one choice over all (cell, token) pairs, so there is always exactly one. For a random event the choice is sampled among candidates within 2.2 logits of the best, so the dream invents a new serve or respawn instead of averaging all of them.-3. **Stamps:** Pac-Man and the ghosts are 2×2 sprites. Only one cell of each carries state (the mouth holds the heading; the ghost's lower-left cell holds whether a pellet is underneath). The other three cells are drawn from it instead of being predicted. This took Pac-Man from 14% to 65% exact frames with no retraining.+1. Background cells (walls, bricks, pellets, floor) each take their most likely token independently.+2. Singletons (the ball, the player, each ghost) are placed with one choice over all (cell, token) pairs, so there is always exactly one of each. After a random event the choice is sampled among candidates within 2.2 logits of the best, so the dream invents a new serve or respawn instead of averaging them.+3. Pac-Man and the ghosts are 2×2 sprites, but only one cell of each carries state (the mouth holds the heading; the ghost's lower-left cell holds whether a pellet is underneath). The other three cells are stamped from it instead of predicted. This took Pac-Man from 14% to 65% exact frames without retraining.-Rule 2 replaced per-cell decoding early on. Decoded cell by cell, the first Breakout model drew two balls at once; one global choice took exact frames from 91.6% to 99.7%.+Rule 2 came first. Decoding cell by cell, the first Breakout model drew two balls at once; one global choice took exact frames from 91.6% to 99.7%.### Checking the dream-Frame accuracy on recorded data is not enough. An agent drives the dream into states the data never had, and any mistake there is something it can exploit. So the check runs while an agent plays:--1. Read the game state back off the dreamed frame (`games/<game>.py: read`).-2. Apply the real rules to that state and the agent's action.-3. Compare the result with the next dreamed frame, cell by cell, and compare the rewards.+Accuracy on recorded data wasn't enough. An agent drives the dream into states the data never had, and any mistake there is something it can exploit. So the main check runs while an agent plays: read the game state back off the dreamed frame (`games/<game>.py: read`), apply the real rules to that state and the agent's action, and compare the result with the next dreamed frame and reward.-5,000 dreamed steps per game, at 2 loops, with the shipped agent (`training/evalwm.py --agent`):+5,000 dreamed steps per game at 2 loops, with the shipped agent (`training/evalwm.py --agent`):| game | wrong game states | wrong rewards | wrong cells per step ||---|---|---|---|@@ -116,45 +101,43 @@ Frame accuracy on recorded data is not enough. An agent drives the dream into st| Doom-lite | 1 | 0 | 1.50 || Flappy Bird | 0 | 0 | 0.00 || Pong | 1 | 0 | 0.00 |-| Pac-Man | CHECK_PACMAN |+| Pac-Man | 742 | 38 | 0.39 |++Pac-Man is the weakest dream: about one step in seven reads back to a state the real rules wouldn't produce, although the maze itself is never wrong and the frames stay readable.### Training-**1. Recording.** Agents play the real game, and every step becomes one example: frame, action, next frame, reward. That gives 1–3 million examples per game. Some episodes start from unusual states (bricks with holes, random positions) so the data covers more of the game.+First, agents play the real game and every step is recorded as one example (frame, action, next frame, reward), 1–3 million per game. Some episodes start from unusual states, like bricks with holes or random positions, so the data covers more of the game.-**2. Training.** The world model learns to predict the next frame: a cross-entropy loss per cell, 30k steps, about 45 minutes per game on an RTX 4060 Ti. The first 3–4k steps run in float, then int8 rounding switches on (QAT, see [quantization](#quantization-and-the-pico)). The animation shows one held-out Pong step predicted at six points of a short training run.+Then the world model learns to predict the next frame with a per-cell cross-entropy loss: 30k steps, about 45 minutes per game on an RTX 4060 Ti. The first 3–4k steps run in float, then int8 rounding switches on ([QAT](#quantization-and-the-pico)). The animation shows one held-out Pong step predicted at six points of a short training run.-**3. Deployment.** The int8 weights are written into Rust source with every float stored as its exact bits. `cargo test` checks that the Rust engine reproduces the Python integer spec bit for bit, then the firmware runs the dream on the Pico.+The int8 weights are written into Rust source with every float stored as its exact bits. `cargo test` checks that the Rust engine reproduces the Python integer spec bit for bit, and then the firmware runs the dream on the Pico.-**4. Closing the loop.** An agent trained in the dream reaches states the recorded data never had. Recording real data with that agent and fine-tuning the world model on everything fixes the dream there.+The last step closes the loop. An agent trained in the dream reaches states the recorded data never had, so I record real games with that agent and fine-tune the world model on everything.-Each fix below cost a training round to find:+Most of what I know about this came from things going wrong, each costing a training round to find:-- **Hedging.** Weighting the loss by "this cell changed" biased the model toward predicting change, so it drew two balls and broke two bricks. The fix was to weight by where things can happen in the *input* frame, plus the global choice for singletons.-- **Coverage.** A Breakout agent in the dream got 154 points with 26 bricks left: the dream paid for corner hits without removing the brick. Doom-lite agents snipe from 5 cells away, which scripted play never did. The fix was collecting data with the learned agent.-- **Exploits.** Agents trained only in the dream scored 210 there (the real maximum is 56) and 28 in the real game, as *World Models* (Ha & Schmidhuber) predicts. Each collect-and-fine-tune round closed the exploit that agent had found; Breakout needed four rounds.-- **Copying.** The Doom-lite dream sometimes lost the enemy sprite and never drew it again: the model copied the previous view instead of redrawing it from the minimap. Noise augmentation fixed it (`train --noise 0.3`, as in GameNGen): input views get random pixels and sometimes lose the sprite.-- **Capacity.** `d=32` could not absorb the wider data for Breakout; `d=48` could. Pac-Man's 48×48 frame needed `d=64`.+- Weighting the loss by "this cell changed" biased the model toward predicting change, so it drew two balls and broke two bricks. Weighting by where things *can* happen in the input frame, plus the global singleton choice, fixed it.+- A Breakout agent in the dream got 154 points with 26 bricks left, because the dream paid for corner hits without removing the brick. Doom-lite agents learned to snipe from 5 cells away, which my scripted players never did. Both needed data collected with the learned agent.+- Agents trained only in the dream scored 210 there (the real maximum is 56) and 28 in the real game, as *World Models* (Ha & Schmidhuber) predicts. Each collect-and-fine-tune round closed the exploit that agent had found. Breakout needed four rounds.+- The Doom-lite dream sometimes lost the enemy sprite and never drew it again: the model copied the previous view instead of redrawing it from the minimap. Noise augmentation fixed it (`train --noise 0.3`, as in GameNGen): input views get random pixels and sometimes lose the sprite.+- `d=32` couldn't absorb the wider Breakout data and `d=48` could. Pac-Man's 48×48 frame needed `d=64`.## The agents: stream-x-Stream AC(λ) (Elsayed, Vasan & Mahmood, 2024) learns from one transition at a time:--- **No buffer:** no replay buffer, no batches, no target network.-- **Updates:** eligibility traces carry credit back in time, and ObGD shrinks any step that would overshoot the TD target.-- **Networks:** a policy and a value network, each two hidden layers of 64 units (32 on the Pico) with LayerNorm.+The agents use Stream AC(λ) (Elsayed, Vasan & Mahmood, 2024), which learns from one transition at a time: no replay buffer, no batches, no target network. Eligibility traces carry credit back in time, and ObGD shrinks any step that would overshoot the TD target. The policy and value networks each have two hidden layers of 64 units (32 on the Pico) with LayerNorm.-The input is a feature vector read off the frame (`training/streamx.py: observe_*`, mirrored in Rust in `tinyworldformer-rs/src/agent.rs`). Examples: the ball's row, column and heading in Breakout; in Pac-Man, for each direction, the shortest-path distance to the nearest pellet and how close each ghost is.+The input is a feature vector read off the frame (`training/streamx.py: observe_*`, mirrored in Rust in `tinyworldformer-rs/src/agent.rs`). For example, the ball's row, column and heading in Breakout, or in Pac-Man, for each direction, the shortest-path distance to the nearest pellet and how close each ghost is.-Below, one training run per game (real game, 500k steps) at six stages: untrained, the end, and the snapshots where the score has covered 25, 50, 75 and 95% of the way. Each panel plays greedily from the same start and skips a few steps per frame. "try" counts the episodes played so far, so crashes show.+Below is one training run per game (real game, 500k steps) at six stages: untrained, the end, and the snapshots where the score has covered 25, 50, 75 and 95% of the way. Each panel plays greedily from the same start and skips a few steps per frame. "try" counts the episodes played so far, so you can see the crashes.**Breakout**@@ -171,7 +154,7 @@ Below, one training run per game (real game, 500k steps) at six stages: untraine**Pong**-Greedy return, real game, mean of 10 episodes (snapshots of the same run):+Greedy return in the real game, mean of 10 episodes, at snapshots of the same run:| game | 0 | 1k | 2k | 5k | 10k | 20k | 50k | 100k | 150k | 250k | 500k | max ||---|---|---|---|---|---|---|---|---|---|---|---|---|@@ -181,34 +164,28 @@ Greedy return, real game, mean of 10 episodes (snapshots of the same run):| Pac-Man | 7.0 | 3.8 | 17.0 | 17.7 | 8.2 | 46.8 | 72.9 | 138.9 | 146.6 | 190.3 | 217.4 | 249 || Pong | −4.3 | −3.9 | −3.9 | −3.9 | −1.7 | −2.0 | −0.3 | 1.1 | 1.7 | 1.1 | 1.1 | 5 |-Breakout: bricks minus lost balls. Doom-lite: kills minus hits in 500 steps. Flappy Bird: pipes in 3,000 steps. Pac-Man: pellets minus lives lost. Pong: points won minus lost in 3,000 steps.+Scores are Breakout: bricks minus lost balls; Doom-lite: kills minus hits in 500 steps; Flappy Bird: pipes in 3,000 steps; Pac-Man: pellets minus lives lost; Pong: points won minus lost in 3,000 steps.-- **Flappy Bird** reaches the maximum by 250k steps.-- **Breakout** is still improving at 500k; the 1M-step agents clear the board (56.0).-- **Pong**'s agent learns not to miss, then plateaus: beating the CPU needs angled shots off the paddle's ends, which it rarely finds.+Flappy Bird reaches the maximum by 250k steps. Breakout is still improving at 500k, and the 1M-step agents clear the board (56.0). Pong's agent learns not to miss and then plateaus: beating the CPU needs angled shots off the ends of the paddle, which it rarely finds.-**Trained only in the dream.** These agents never saw the real game. Scores are measured in the real game (greedy, 50 episodes):+Some agents never saw the real game at all. Their scores, measured in the real game (greedy, 50 episodes):| game | agent | real-game score ||---|---|---|| Breakout | 1M dream steps, −5 per lost ball | 54.4 of 56 bricks || Doom-lite | 300k dream steps, idle penalty 0.1 | 81.6 kills per 500 steps |-Without shaping, Doom-lite agents camp in the start corner and fire: the enemy always comes to them. `--idle-penalty 0.1` costs 0.1 for each step spent in a cell visited more than 6 times in the last 20 steps. The position is read off the minimap, so the penalty works inside the dream too. With it the agent patrols and gets twice the kills.+Without shaping, Doom-lite agents camp in the start corner and fire, because the enemy always comes to them. `--idle-penalty 0.1` costs 0.1 for each step spent in a cell visited more than 6 times in the last 20 steps. The position is read off the minimap, so the penalty works inside the dream too. With it the agent patrols and gets twice the kills.-**Learning on the Pico.** `tinyworldformer-rs/src/learner.rs` is the same algorithm in `no_std` Rust, with backpropagation written by hand. It matches 200 PyTorch updates to within 3.5e-6. On the host, a fresh agent inside the Rust int8 Flappy dream averages 6 pipes per episode after 25k steps, 121 after 100k, and 300+ after 250k.+`tinyworldformer-rs/src/learner.rs` is the same algorithm in `no_std` Rust, with the backward pass written by hand. It matches 200 PyTorch updates to within 3.5e-6. On the host, a fresh agent inside the Rust int8 Flappy Bird dream averages 6 pipes per episode after 25k steps, 121 after 100k, and 300+ after 250k.## Quantization and the Pico-- **Weights:** every matrix in the world model is int8, with one scale per output row (`scale = max|w| / 127`).-- **Activations:** int8, with one scale per tensor and per loop index (the loops see different ranges).-- **Matmuls:** int8 × int8 sums into int32 (two multiply-adds per `SMLAD` instruction on the RP2350), then one float multiply by the two scales.-- **Softmax:** a 1024-entry exp table, with probabilities stored as uint8.-- **Float32:** norms, residuals and embeddings. The RP2350 has an FPU.-- **QAT:** training applies the same rounding (straight-through estimator), so the int8 model is the model that was trained.-- **Bit-exact:** `rustexport/intref.py` is the int8 model in numpy with every float operation in a fixed order. The Rust engine reproduces it bit for bit, and `cargo test` checks golden vectors for every exported env.+Every matrix in the world model is int8 with one scale per output row (`scale = max|w| / 127`). Activations are int8 too, with one scale per tensor and per loop index, since the loops see different ranges. Matmuls sum int8 × int8 into int32 (two multiply-adds per `SMLAD` instruction on the RP2350) and then apply one float multiply by the two scales. Softmax uses a 1024-entry exp table, with probabilities stored as uint8. Norms, residuals and embeddings stay in float32, since the RP2350 has an FPU.++Training applies the same rounding (QAT with a straight-through estimator), so the int8 model is the model that was trained. `rustexport/intref.py` is the int8 model in numpy with every float operation in a fixed order; the Rust engine reproduces it bit for bit, and `cargo test` checks golden vectors for every exported env.@@ -218,59 +195,40 @@ Without shaping, Doom-lite agents camp in the start corner and fire: the enemy a| Doom-lite | 33.9 KB | 91.9 KB | 12.0 KB | 61.3 KB | 471 KB | 357 KB || Flappy Bird | 30.1 KB | 69.5 KB | 10.1 KB | 50.8 KB | 409 KB | 227 KB || Pong | 30.1 KB | 70.4 KB | 10.9 KB | 52.6 KB | 414 KB | 244 KB |-| Pac-Man | PACMAN_SIZES |-- **Measuring:** flash and RAM are measured from the firmware ELF (`size -A`), built with the Display Pack driver. They include the code, USB, the display driver and the on-device learner.-- **The Pico 2 W:** 4 MB of flash and 520 KB of SRAM (512 KiB of it in the main memory map).-- **Room to shrink:** the float32 embedding tables are the largest part. They could be int8 too.+Flash and RAM are measured from the firmware ELF (`size -A`), built with the Display Pack driver, so they include the code, USB, the display driver and the on-device learner. The Pico 2 W has 4 MB of flash and 520 KB of SRAM, 512 KiB of it in the main memory map. The float32 embedding tables are the largest part and could be int8 too.-**Reusing the export.** `src/tinyworldformer/rustexport/` imports only numpy and torch, and a test keeps it that way, so other projects can use it:--1. Build a model from `rustexport.qat` (`QLinear`, `LoopedBlock`).-2. Train it with `ActQuant.enabled = True` after a float warm-up.-3. `rustexport.convert.looped_model` turns it into the integer spec.-4. `int_model_arrays` and `mlp_arrays` with `codegen.write_module` write the Rust source; `rustexport.Golden` writes the vectors a Rust test checks.+`src/tinyworldformer/rustexport/` imports only numpy and torch (a test keeps it that way), so other projects can reuse the export: build a model from `rustexport.qat` (`QLinear`, `LoopedBlock`), train it with `ActQuant.enabled = True` after a float warm-up, convert it with `rustexport.convert.looped_model`, then write the Rust source with `int_model_arrays`, `mlp_arrays` and `codegen.write_module`. `rustexport.Golden` writes the vectors a Rust test checks.## On the chipThe firmware (`tinyworldformer-rs/firmware`) is `no_std` Rust on embassy. Nothing is allocated at run time: the weights are `static` arrays in flash, and every buffer is a fixed-size `static` in RAM.-**Core 1 runs the game.** One pass of its loop (`dream_loop` in `firmware/src/main.rs`) is one frame:--1. **The actor picks a button.** It reads features off the current frame (`agent::observe`) and runs its MLP. If on-device learning is on, the learning agent picks instead. If you are pressing buttons, your press is used.-2. **The dream draws the next frame.** The int8 transformer runs 2 loops over the frame and the button (`Engine::forward`). Decoding turns its scores into the next frame: background argmax per cell, each singleton placed once (sampled for random events, with an xorshift RNG), then the sprite stamps. The reward is the argmax of the reward head, and lives and episode ends are counters on top.-3. **The learner updates** (if learning is on), from this one transition, then forgets it.-4. **The frame goes to core 0** through a two-slot channel. If core 0 is still busy with the previous frame, this one is dropped: core 1 never waits for the screen.+Core 1 runs the game. One pass of its loop (`dream_loop` in `firmware/src/main.rs`) is one frame:-When you play, core 1 also slows to a playable speed (66–150 ms per frame, depending on the game). The agent runs as fast as the chip allows.+1. The actor reads features off the current frame (`agent::observe`) and runs its MLP to pick a button. If on-device learning is on, the learning agent picks instead; if you are pressing buttons, your press is used.+2. The int8 transformer runs 2 loops over the frame and the button (`Engine::forward`), and decoding turns its scores into the next frame: background argmax per cell, each singleton placed once (sampled for random events with an xorshift RNG), then the sprite stamps. The reward is the argmax of the reward head; lives and episode ends are counters on top.+3. If learning is on, the learner updates from this one transition and then forgets it.+4. The frame goes to core 0 through a two-slot channel. If core 0 is still busy with the previous frame, this one is dropped, so core 1 never waits for the screen.-**Core 0 talks to the outside world** (embassy async tasks):-- **USB serial:** each frame goes out as coloured ANSI text with a status line (model time, actor time, fps, env and agent names). Keys come back in.-- **Display Pack** (optional, `--features display`): a 320×240 ST7789 over SPI at up to 62.5 MHz. Only cells that changed since the last frame are redrawn.-- **Buttons:** polled every 20 ms.+When you play, core 1 slows down to a playable speed (66–150 ms per frame, depending on the game). The agent runs as fast as the chip allows.-The cores share only the frame channel and a few atomics: who is playing, the pressed key, learning on/off, the loop count.+Core 0 runs embassy async tasks for everything else. Over USB serial, each frame goes out as coloured ANSI text with a status line (model time, actor time, fps, env and agent names), and keys come back in. The optional Display Pack (`--features display`) is a 320×240 ST7789 over SPI at up to 62.5 MHz, and only cells that changed since the last frame are redrawn. Its buttons are polled every 20 ms. The two cores share only the frame channel and a few atomics: who is playing, the pressed key, learning on/off, and the loop count.-**Where the memory goes:**+Where the memory goes:| | in flash | in RAM ||---|---|---|| world model | int8 matrices, float32 embeddings, norms and scales, the game's opening frames and colours (52–138 KB) | scratch buffers for one forward pass: activations for 36–80 tokens, the int8 copies the matmuls read, and the per-cell scores of the next frame (114 KB for Flappy Bird up to 368 KB for Pac-Man, whose per-cell scores alone take 184 KB) || actor | float32 MLP weights and feature normalization (51–61 KB) | its features and hidden layers (about 1 KB) |-| on-device learner | code only | weights, eligibility traces and normalization for two 32-unit networks (about 85 KB) |+| on-device learner | code only | weights, eligibility traces and normalization for two 32-unit networks (49 KB for Pac-Man up to 106 KB for Doom-lite; it scales with the number of input features) || firmware | code, USB, display driver (about 150 KB) | stacks (16 KB for core 1), USB buffers, frame channels |-The memory map gives 4 MB of flash and 512 KiB of RAM. Every game fits: Breakout uses 361 KB of flash and 299 KB of RAM; Pac-Man, the largest, 600 KB and 460 KB.+Every game fits in the 4 MB of flash and 512 KiB of RAM. Breakout uses 361 KB of flash and 299 KB of RAM; Pac-Man, the largest, uses 600 KB and 460 KB.-**The inner loop.** Most of the time goes into int8 dot products (`dot_i8` in `src/lib.rs`):-- **Loads:** one 32-bit load fetches 4 weights.-- **Unpacking:** `SXTB16` unpacks bytes 0/2 and 1/3 into 16-bit halves.-- **Multiply-adds:** `SMLAD` does two multiply-adds in one instruction.-- **Attention:** its uint8 probabilities times int8 values use `UXTB16` instead of `SXTB16`.-- **Float work:** norms, the exp table lookup and the final scale per output stay in float32 on the FPU.-- **Speed:** a frame at 2 loops is 2–4M multiply-adds. On a desktop CPU the same Rust code dreams a frame in 0.7 ms (Flappy Bird) to 1.5 ms (Breakout). **On the Pico the frame rate is not measured yet:** the firmware is built and sized but has not run on a board.+Most of the time goes into int8 dot products (`dot_i8` in `src/lib.rs`). One 32-bit load fetches 4 weights, `SXTB16` unpacks bytes 0/2 and 1/3 into 16-bit halves, and `SMLAD` does two multiply-adds in one instruction. Attention multiplies uint8 probabilities by int8 values, so it uses `UXTB16` instead. Norms, the exp table lookup and the final per-output scale stay in float32 on the FPU. A frame at 2 loops is 2–4M multiply-adds (8.2M for Pac-Man); on a desktop CPU the same Rust code dreams a frame in 0.7 ms (Flappy Bird) to 1.5 ms (Breakout). I haven't measured the frame rate on the Pico itself yet.-**Learning at boot.** The board starts with a fresh agent learning on the device. Press `l` over USB, or hold B for 1 s on the Display Pack, to switch to the pretrained actor, and again to switch back.+The board boots with a fresh agent learning on the device. Press `l` over USB, or hold B for 1 s on the Display Pack, to switch to the pretrained actor, and again to switch back.## Flashing a game and an agent@@ -286,15 +244,13 @@ cd tinyworldformer-rsTWF_ENV=doomlite TWF_AGENT=doom_agent_roam_p0.1 cargo test --release # golden tests for that pair```-- **Default agent:** without `--agent`, the env's game's agent from `models/default` is used if it fits, else the first one by name.-- **Wrong game:** an agent for another game is refused, with the list of valid ones.-- **On screen:** the status line shows which env and agent are running.+Without `--agent`, the agent for the env's game listed in `models/default` is used if there is one, otherwise the first by name. An agent for a different game is refused with a list of the valid ones. The status line on screen shows which env and agent are running.Keys over USB: Breakout a/d/s, Doom-lite w/a/d/space/s, Flappy space, Pac-Man w/a/s/d, Pong w/s. Then `g` hands back to the agent, `1`–`4` set the loop count, `l` toggles on-device learning and `r` restarts it.## In the browser-The same `no_std` int8 engine also compiles to WebAssembly (`tinyworldformer-rs/web/`). The module has no imports and no wasm-bindgen. It runs the firmware's loop: dream, agent, and stream-x learning on the fly. A static page draws the frames on a canvas.+The same `no_std` int8 engine also compiles to WebAssembly (`tinyworldformer-rs/web/`). The module has no imports and no wasm-bindgen, and it runs the firmware's loop: dream, agent, and stream-x learning on the fly. A static page draws the frames on a canvas.```bashrustup target add wasm32-unknown-unknown # once@@ -303,10 +259,7 @@ cd tinyworldformer-rs/webpython3 -m http.server -d www # open http://localhost:8000, ?env=pacman picks a game```-- **Same dream:** with the same seed, the wasm build dreams the same episodes as the native build, frame for frame.-- **Speed:** 1–5 ms per dreamed frame, depending on the game.-- **Agents:** each module carries its game's default agent; `TWF_AGENT=... ./build.sh <env>` picks another.-- **Keys:** the same keys as on the Pico, plus arrow keys and on-screen buttons. `f` runs the dream as fast as it can, which is useful while learning (`l`).+With the same seed, the wasm build dreams the same episodes as the native build, frame for frame, at 1–5 ms per frame depending on the game. Each module carries its game's default agent; `TWF_AGENT=... ./build.sh <env>` picks another. The keys are the same as on the Pico, plus arrow keys and on-screen buttons, and `f` runs the dream as fast as it can, which helps while learning (`l`).`www/` is a plain static site, so it can be hosted as is. To publish it as a Hugging Face Space (the free CPU tier is enough, since everything runs in the visitor's browser):