Skip to content

A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.

9.0 KB
Raw Blame

CHIP-8 inside a stock Qwen3.5 checkpoint

chipformer/q35/ compiles the CHIP-8 machine into ordinary Qwen3.5 layers: Gated DeltaNet (linear attention) plus SwiGLU MLPs. The result loads with the stock transformers classes and needs no trust_remote_code. It runs exactly in bf16 and fp32, with the reference torch kernels or with flash-linear-attention. The trace format is the same as the original chipformer (docs/trace_format.md), so chipformer/check.py, session.py and the reference emulator are reused unchanged.

Ideas

  • Memory is the delta rule. A DeltaNet head updates its state with S += k (v - S^T k)^T. With beta = 1 and a one-hot key, that is an exact overwrite of one row. A one-hot query then reads the latest write. This gives RAM with no positions and no large attention scores, which is why float64 is no longer needed.

  • Norm isolation (ballast).

    • Chip tokens carry 2^40 in m dims. With m * 2^80 / hidden a power of 4, every RMSNorm on a chip token divides by an exactly known power of 2, and the emitter folds that factor into the weights.
    • In the v2 layout every token carries ballast, text included, so the circuit can see text tokens (as feature rows of a few text classes, see bridge.py) and is gated on is_text wherever it writes state.
  • bf16 exactness.

    • Every materialized value has at most 8 significant bits: bits, one-hot nibbles, and integers up to 256. 12-bit quantities are built from a low byte and a high nibble.
    • silu(K x) with K = 128 is an exact ReLU at every value the circuit produces.
  • fp32 exactness.

    • ANDs, keys and queries use half-unit margins (4 * (sum - n + 1/2)), so a false entry is exactly 0 even when the literals carry noise.
    • Without the margins, a tiny leak into an unwritten row gets rescaled to full size by the per-head output RMSNorm.
  • Positions, replaced.

    • The write index j comes from a head whose row halves at every VAL token and resets at STEP/BOOT, so x = 2^-j.
    • "Latest STEP" and "latest TICK" are rows that get overwritten at those tokens.
    • Display clear (CLS) is a decay-0 reset of the display heads.
  • Chip first, then Qwen (v2).

    • The chip layers come before Qwen's. Text tokens pass the chip layers, and the last chip MLP (the exit) clears every chip dim except the mute marker mu, a tiny output code (KAPPA = 2^-20) and tau, so Qwen's layers receive their usual input.
    • tau is 0 where the circuit owns the next token and 2^-40 where Qwen speaks. The final norm gains (code 2^20, tau 2^60) let the owner win the argmax by a wide margin.
  • Mute (mute.py). Columns on mu set Qwen's DeltaNet a, b, z and the attention output gate to -1024 on chip tokens, so Qwen's state passes through a trace unchanged and pure text stays bit-identical. An anti-sink key keeps chip positions from soaking up attention.

  • Overlay (overlay.py). While the chip layers run, a text token holds its Qwen embedding in Qwen's dims and a chip token holds zeros there. So circuit dims that only chip tokens need share Qwen's dims, and only what the circuit computes on text tokens gets dims of its own.

    • Units, key channels and output gates that would read or write a shared dim on a text token get -2^20 * is_text, so silu is exactly 0 there whatever Qwen's value is (|q| up to ~585). Their handovers and exit clears are gated on mu.
    • What may be switched off is what nothing on text depends on, or what is 0 on text over a corpus of mixed sessions. The switches are part of the graph, so the IR checks them.
    • 4B: hidden 4224 → 3072 (20 → 15 GB). 0.8B: 3072 → 2048. merge.py --no-overlay keeps the old layout.
  • Qwen uses the computer (bridge.py, lib/).

    • The circuit reads Qwen's native tool call (<tool_call><function=mul><parameter=a>37...): digit tokens and the tool-call tags are text classes, and so are the function names in the spellings Qwen actually generates (=add, ul, =sub, ub).
    • At </tool_call> the circuit takes over. It emits a CALL record (argument digits into RAM, PC = the routine's entry), runs the routine, and SCHIP's 00FD, redefined as print-and-halt, prints `
digits ` as real Qwen tokens. Then Qwen continues and reads the result like any tool response. - The BIOS (`lib/bcd.py`, 56 bytes, assembled by `lib/asm.py`) is booted from the system prompt. - **Inline arithmetic (no tool call).** Outside tool calls the same capture watches Qwen's plain text. When the text since the last breaker token is exactly `A op B =` (op is `+`, `-`, `×`, `*`, ` imes` or `\cdot`, thousands commas allowed, at least 4 digits in total), the circuit takes over at `=`. It predicts STEP, writes the CALL record with MODE = 1, runs the routine and prints ` digits` as plain text, then Qwen carries on. Decimals, chains, negatives and numbers right after a letter, `.` or `^` are left to Qwen. `bridge.InlineScan` is the reference. - Short results are left to Qwen because, right after a trace, Qwen's conv window holds chip tokens instead of the `=`, and a short printed number can read to it like the start of a new expression. - **Bignum macros (`lib/macros.py`).** Instructions in CHIP-8's unused `5XYN` space compute a whole row of digits in one record: `BADD`, `BSUB`, `BRSUB` (B - A), `BZERO`, and `BMUL0` / `BMUL` (one schoolbook row). - A carry or borrow passes between writes only through the digit just emitted: at each VAL token the circuit rebuilds the carry from that digit and the operands it read. - The macros loop and branch themselves, so the BIOS is 22 bytes: - `BMUL` repeats until its last row (PC stays: no PC write) and counts rows in VX. `BMUL0`, the first row, reads R as 0, so no `BZERO` is needed. - `BADD` / `BSUB` / `BRSUB` jump to a target in their XY field. `BSUB` branches on its borrow to `BRSUB`. - The last write sets V0 / I for the print, skipping R[0] when it is 0. - `BTRIM` strips any further leading zero, one per record, then the BIOS prints. - Tokens from `=` to the end of the print (inline): about 50 for a 4+3-digit add, 67 for a 7-4 subtract, 74 for a negative 4-4 subtract, and 87 / 126 / 244 for 4x3 / 6x4 / 8x8 multiplies.

Files

File Role
ir.py Graph of FFN / DeltaNet-head nodes, and an exact evaluator that also checks bf16 rounding at every intermediate
prims.py Gadgets: ANDs, threshold tests, nibble arithmetic, slot heads
chip8_q35.py The CHIP-8 circuit
emit.py Weights, residual-slot sharing (liveness, with handover units), and the layout
tiny.py Standalone stock Qwen3_5ForCausalLM (python -m chipformer.q35.tiny)
merge.py Widen a Qwen3.5 checkpoint, append the chip layers, extend the tokenizer (python -m chipformer.q35.merge SRC)
host.py Drives a checkpoint with Session (prefill plus cached decode)
mute.py Makes Qwen's layers ignore chip tokens
bridge.py Text classes, the print bridge and tool-call capture
lib/asm.py, lib/bcd.py, lib/macros.py Assembler, the BIOS, the bignum macros
demo.py Merged model vs base on exact add/sub/mul through tool calls
fast.py, batch.py Static-cache decoding: compiled batch-1 step and batched greedy
record.py Plays a game through a checkpoint and renders the verified trace to video
evals.py, text_eval.py Math benchmarks with and without the computer; text regressions
report.py Chip exactness, leakage and text fidelity of a merged checkpoint vs its base
spike.py Phase 0 numerics: the primitives in real HF modules

Status

  • Circuit: the IR evaluator equals the reference emulator on 1.08M tokens across all 119 games, the Timendus test ROMs under every quirk profile, and fuzzed programs.
  • Tiny checkpoint (351M, hidden 1024, 15 linear-attention layers plus one inert attention layer): token-exact in bf16 and fp32.
  • Merged checkpoints:
Base Merged size Chip Text fidelity
Qwen3.5-0.8B hidden 2048, 24 + 15 layers, 2.27B exact, also inside a chat KL 7.8e-4 vs 5.4e-4 noise floor
Qwen3.5-4B (v2, macros) hidden 3840, 16 chip + 32 Qwen layers, 8.3B exact inside the noise floor
  • Qwen uses its computer (merged 4B, add and mul tools): 9/10 exact vs 5/10 for base Qwen3.5-4B, which got every multiplication wrong. The miss hit the token limit.
  • Read bridge (merged 0.8B): the program prints 4817 and Qwen goes on to answer 4818.
  • Memory across a trace: with the mute patch Qwen's answers after a trace match the no-trace answers (KL ~0.007).
  • Speed (L40S, static cache): tiny ~440 tok/s, merged 0.8B ~105 tok/s.
  • Inline arithmetic (merged 4B, demo --inline --nudge --hard): 12/12 exact on 6-10 digit by 4-7 digit problems, against 4/12 for base Qwen3.5-4B with the same system prompt. Base misses every large product by a few digits; the merged model writes 43569773 * 537110 = and the circuit prints 23401760776030.