A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.
CHIP-8 inside a stock Qwen3.5 checkpoint
chipformer/q35/ compiles the CHIP-8 machine into ordinary Qwen3.5 layers: Gated DeltaNet
(linear attention) plus SwiGLU MLPs. The result loads with the stock transformers classes and
needs no trust_remote_code. It runs exactly in bf16 and fp32, with the reference torch kernels
or with flash-linear-attention. The trace format is the same as the original chipformer
(docs/trace_format.md), so chipformer/check.py, session.py and the reference emulator are
reused unchanged.
Ideas
Memory is the delta rule. A DeltaNet head updates its state with
S += k (v - S^T k)^T. Withbeta = 1and a one-hot key, that is an exact overwrite of one row. A one-hot query then reads the latest write. This gives RAM with no positions and no large attention scores, which is why float64 is no longer needed.Norm isolation (ballast).
- Chip tokens carry
2^40inmdims. Withm * 2^80 / hiddena power of 4, every RMSNorm on a chip token divides by an exactly known power of 2, and the emitter folds that factor into the weights. - In the v2 layout every token carries ballast, text included, so the circuit can see text
tokens (as feature rows of a few text classes, see
bridge.py) and is gated onis_textwherever it writes state.
- Chip tokens carry
bf16 exactness.
- Every materialized value has at most 8 significant bits: bits, one-hot nibbles, and integers up to 256. 12-bit quantities are built from a low byte and a high nibble.
silu(K x)withK = 128is an exact ReLU at every value the circuit produces.
fp32 exactness.
- ANDs, keys and queries use half-unit margins (
4 * (sum - n + 1/2)), so a false entry is exactly 0 even when the literals carry noise. - Without the margins, a tiny leak into an unwritten row gets rescaled to full size by the per-head output RMSNorm.
- ANDs, keys and queries use half-unit margins (
Positions, replaced.
- The write index
jcomes from a head whose row halves at every VAL token and resets at STEP/BOOT, sox = 2^-j. - "Latest STEP" and "latest TICK" are rows that get overwritten at those tokens.
- Display clear (CLS) is a decay-0 reset of the display heads.
- The write index
Chip first, then Qwen (v2).
- The chip layers come before Qwen's. Text tokens pass the chip layers, and the last chip MLP
(the exit) clears every chip dim except the mute marker
mu, a tiny output code (KAPPA = 2^-20) andtau, so Qwen's layers receive their usual input. tauis 0 where the circuit owns the next token and2^-40where Qwen speaks. The final norm gains (code2^20,tau2^60) let the owner win the argmax by a wide margin.
- The chip layers come before Qwen's. Text tokens pass the chip layers, and the last chip MLP
(the exit) clears every chip dim except the mute marker
Mute (
mute.py). Columns onmuset Qwen's DeltaNeta,b,zand the attention output gate to -1024 on chip tokens, so Qwen's state passes through a trace unchanged and pure text stays bit-identical. An anti-sink key keeps chip positions from soaking up attention.Overlay (
overlay.py). While the chip layers run, a text token holds its Qwen embedding in Qwen's dims and a chip token holds zeros there. So circuit dims that only chip tokens need share Qwen's dims, and only what the circuit computes on text tokens gets dims of its own.- Units, key channels and output gates that would read or write a shared dim on a text token
get
-2^20 * is_text, so silu is exactly 0 there whatever Qwen's value is (|q| up to ~585). Their handovers and exit clears are gated onmu. - What may be switched off is what nothing on text depends on, or what is 0 on text over a corpus of mixed sessions. The switches are part of the graph, so the IR checks them.
- 4B: hidden 4224 → 3072 (20 → 15 GB). 0.8B: 3072 → 2048.
merge.py --no-overlaykeeps the old layout.
- Units, key channels and output gates that would read or write a shared dim on a text token
get
Qwen uses the computer (
bridge.py,lib/).- The circuit reads Qwen's native tool call (
<tool_call><function=mul><parameter=a>37...): digit tokens and the tool-call tags are text classes, and so are the function names in the spellings Qwen actually generates (=add,ul,=sub,ub). - At
</tool_call>the circuit takes over. It emits a CALL record (argument digits into RAM, PC = the routine's entry), runs the routine, and SCHIP's00FD, redefined as print-and-halt, prints `
- The circuit reads Qwen's native tool call (
Files
| File | Role |
|---|---|
ir.py |
Graph of FFN / DeltaNet-head nodes, and an exact evaluator that also checks bf16 rounding at every intermediate |
prims.py |
Gadgets: ANDs, threshold tests, nibble arithmetic, slot heads |
chip8_q35.py |
The CHIP-8 circuit |
emit.py |
Weights, residual-slot sharing (liveness, with handover units), and the layout |
tiny.py |
Standalone stock Qwen3_5ForCausalLM (python -m chipformer.q35.tiny) |
merge.py |
Widen a Qwen3.5 checkpoint, append the chip layers, extend the tokenizer (python -m chipformer.q35.merge SRC) |
host.py |
Drives a checkpoint with Session (prefill plus cached decode) |
mute.py |
Makes Qwen's layers ignore chip tokens |
bridge.py |
Text classes, the print bridge and tool-call capture |
lib/asm.py, lib/bcd.py, lib/macros.py |
Assembler, the BIOS, the bignum macros |
demo.py |
Merged model vs base on exact add/sub/mul through tool calls |
fast.py, batch.py |
Static-cache decoding: compiled batch-1 step and batched greedy |
record.py |
Plays a game through a checkpoint and renders the verified trace to video |
evals.py, text_eval.py |
Math benchmarks with and without the computer; text regressions |
report.py |
Chip exactness, leakage and text fidelity of a merged checkpoint vs its base |
spike.py |
Phase 0 numerics: the primitives in real HF modules |
Status
- Circuit: the IR evaluator equals the reference emulator on 1.08M tokens across all 119 games, the Timendus test ROMs under every quirk profile, and fuzzed programs.
- Tiny checkpoint (351M, hidden 1024, 15 linear-attention layers plus one inert attention layer): token-exact in bf16 and fp32.
- Merged checkpoints:
| Base | Merged size | Chip | Text fidelity |
|---|---|---|---|
| Qwen3.5-0.8B | hidden 2048, 24 + 15 layers, 2.27B | exact, also inside a chat | KL 7.8e-4 vs 5.4e-4 noise floor |
| Qwen3.5-4B (v2, macros) | hidden 3840, 16 chip + 32 Qwen layers, 8.3B | exact | inside the noise floor |
- Qwen uses its computer (merged 4B, add and mul tools): 9/10 exact vs 5/10 for base Qwen3.5-4B, which got every multiplication wrong. The miss hit the token limit.
- Read bridge (merged 0.8B): the program prints 4817 and Qwen goes on to answer 4818.
- Memory across a trace: with the mute patch Qwen's answers after a trace match the no-trace answers (KL ~0.007).
- Speed (L40S, static cache): tiny ~440 tok/s, merged 0.8B ~105 tok/s.
- Inline arithmetic (merged 4B,
demo --inline --nudge --hard): 12/12 exact on 6-10 digit by 4-7 digit problems, against 4/12 for base Qwen3.5-4B with the same system prompt. Base misses every large product by a few digits; the merged model writes43569773 * 537110 =and the circuit prints23401760776030.