A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.
Running Qwenputer (q4-int)
q4-int is stock Qwen3.5-4B with a small exact computer compiled into extra layers. The computer
is a CHIP-8 machine plus a bignum arithmetic BIOS. Only config numbers change: hidden size, layer
count and vocabulary. It loads as Qwen3_5ForConditionalGeneration (or, in the text-only variant,
Qwen3_5ForCausalLM) in stock transformers. The circuit is bf16-exact, and Qwen's own weights are
unchanged (zero-padded).
This note describes branch q35-int. The checkpoint path, commit and measured numbers are at the
end.
What is in it
| Base | Qwen3.5-4B (32 layers, hidden 2560) |
| Computer | 12 chip layers before Qwen's 32 (layers 0-11), then Qwen's layers |
| Hidden size | 3200 (Qwen's 2560 dims + ballast + the circuit's text-visible dims). Circuit dims that only chip tokens need are overlaid on Qwen's dims (overlay.py) |
| Size | ~14 GB bf16 (text-only variant: ~13 GB, no vision tower) |
| Vocabulary | Qwen's, plus 8708 chip tokens (<|chip…|>) at chip_token_offset |
| Circuit | 12 layers, 3220 dims, 84 heads, 28,253 MLP units (chipformer.json → circuit) |
What the computer does (all exact):
- Inline. When Qwen writes
A op B =, the circuit takes over at the=, runs the BIOS routine and prints the result as ordinary text.- Forms:
\times,\cdot,×,*,+,-,/,÷,\div,mod,\bmod,%andgcd(A, B) =. - Signed operands for mul, div and gcd (
-5,(-5)). - Thresholds: mul/div/mod/gcd need ≥ 4 digits in A and B together, add/sub ≥ 7.
- Forms:
- Whole expressions (v18). For example
(3 + 4) \times 12 / 7 =or2^{10} - 5 =.- The BIOS evaluates them with exact fractions: + − × / ^, brackets, unary minus, decimals.
- The single-op path wins when both match.
- Tools. Qwen's native tool calls
add,sub,mul,div,mod,gcdandcalc(expression).- The circuit answers at
</tool_call>with\n<tool_response>\nR\n</tool_response>\n, then an empty thinking block<think>\n\n</think>\n\n(two\ntokens each), as Qwen's next assistant turn starts. - Negative results print
-R.
- The circuit answers at
- Output format:
- Integers as digits.
- A quotient that ends within 10 decimals prints as that decimal (
0.625). - Otherwise the reduced fraction
p/q(1785/13201915 = 357/2640383). - Division by 0, or a result the BIOS can't represent (more than 24 digits, a malformed
expression), prints
.... - Inline results end with a closer:
., or$/$$inside math.
How to run it
- The BIOS boot record. Every prompt starts with the BIOS's boot record, as chip tokens
(about 5,000 tokens, prefill only):
from chipformer.q35.lib import bcd from chipformer.ref.emu import Chip8 from chipformer.trace import boot_tokens meta = json.load(open(f"{path}/chipformer.json")) bios = [t + meta["chip_token_offset"] for t in boot_tokens(Chip8(bcd.ROM, bcd.QUIRKS).cells)] ids = bios + tok.apply_chat_template(msgs, tools=TOOLS, add_generation_prompt=True, enable_thinking=False)["input_ids"]- The entry points are compiled into the weights, so the BIOS must be the one this checkpoint
was built with. Check
chipformer.json→bios.sha1,ops. bcd.ROM.segmentscarries code below 0x200;Chip8loads it.
- The entry points are compiled into the weights, so the BIOS must be the one this checkpoint
was built with. Check
- The chat template. Qwen's own template (
chat_template.jinja), unchanged.- Tools:
chipformer.q35.demo.TOOLS(add, sub, mul, div, mod, gcd), plusdemo.CALC_TOOL. - With tools, Qwen calls them in its native XML format.
- Without tools, inline mode works on Qwen's plain text.
- Tools:
- System prompts that help (
chipformer/q35/evals.py):- Tool mode:
NUDGE("You have tools for exact arithmetic (...). Use them for every arithmetic step they cover instead of calculating by hand."). - Inline mode:
NUDGE3(single operations, including division). - Whole expressions:
NUDGE4. - Products and quotients inline, sums through tools:
NUDGE_MIX3.
- Tool mode:
- Generation config.
generation_config.jsonbans<|im_start|>and<|endoftext|>(bad_words_ids).- Qwen ends its turns with
<|im_end|>, which stays the EOS. - Without the bans, Qwen sometimes opens a fake user turn, or ends with
<|endoftext|>right after a tool response, before answering. - Greedy decoding (
do_sample=False).
- Qwen ends its turns with
- Decoders:
- Stock
generate()(cache_implementation="static") runs the computer exactly. The circuit needs nothing from the host: onegenerate()call computes and prints.chipformer/q35/demo.pyis a complete example. - The numbers below use
chipformer.q35.batch.BatchDecoderwith two decoder-side options. Both change only position ids and the KV layout, not weights or computer outputs.freeze_chip_positions(fp): Qwen's RoPE positions don't advance on chip tokens.transparent_chip(tc): Qwen's layers don't see chip tokens. Its text computation then equals the same text with the trace deleted.- Inline mode is markedly better with fp + tc. Tool mode works with both decoders.
--fast(skip_qwen) runs only the chip layers for rows on chip tokens. It is token-identical and about 2× faster at batch 16.- flash-linear-attention / causal-conv1d are not needed. The reference kernels are exact.
- Stock
- Cache length. A trace costs hundreds to thousands of chip tokens: a multiplication about
100-400, a division with a fraction 1-9k, an expression 4-15k. Give the cache room, e.g.
--max-len 16384(32768 for whole expressions); with tc, Qwen doesn't see them.
chipformer.json
| Field | Meaning |
|---|---|
layout |
v2: chip layers first |
chip_token_offset, chip_vocab |
Where the 8708 chip tokens start in the vocabulary |
chip_layers, qwen_layers_from |
12: Qwen's layers start at 12 |
hidden, chip_dims_offset, ballast_dims |
The widened residual: Qwen's dims [0, 2560), then ballast and circuit slots |
circuit |
Layers, dims, heads, MLP units of the compiled circuit |
slots, overlay_slots, overlay |
Circuit dims of their own, dims overlaid on Qwen's, and the overlay plan's summary |
muted |
Qwen's layers are muted on chip tokens (no decay, no writes) |
text_ids |
The text classes the circuit reads and prints, by Qwen token id |
bios |
The BIOS the weights expect: sha1, size, segments, entry points |
Release checks (A10G 24 GB, 2026-10-19)
The checkpoint. divagr1925/Qwen3.5-4B-Qwenputer-CHIP8.
- Size: 14.65 GB in 7 shards.
- Contents: config, tokenizer,
chat_template.jinja,generation_config.json,chipformer.json, LICENSE, preprocessor configs. - A text-only variant (13.98 GB,
Qwen3_5ForCausalLM) can be exported withpython -m chipformer.q35.textonly. - Built from
q35-int(see the commit below) withpython -m chipformer.q35.merge <Qwen3.5-4B> --out q4-int(overlay on), thenpython -m chipformer.q35.textonly.
Shape.
- Hidden 3200, ballast 50.
- 12 chip layers + 32 Qwen layers.
- Circuit: 3222 dims, 84 heads, 28,259 MLP units.
- 516 slots of its own, 1129 overlaid on Qwen's dims.
- BIOS sha1
5e885286e8(3072 B at 0x200, plus 174 B at 0x0E0).
Selftest: 17 tool calls, 0 wrong. Calls cover add, sub, mul and signed mul, div (decimal, fraction, signed), mod, gcd and calc, including a negative calc.
Results (fp + tc, BatchDecoder --fast, greedy):
| Run | Correct | Computer results exact |
|---|---|---|
Tool mode (merged_nb_ne_fp_tc), synthetic arith n=20 |
20/20 | 33/33 tool calls, 7/7 inline fires |
Inline (merged_inline_nudge_fp_tc), arith n=20 |
15/20 (4 stopped at the cache length) | 72/72 fires |
| Tool mode, DM mul_big n=10 | 7/10 (4 cache stops) | 6/6 calls, 22/22 fires |
Inline + nudge3, DM mul_div_multiple_longer n=10 |
3/10 (7 cache stops at 16k) | see the IR replay |
Whole expressions (nudge4), DM mul_div_multiple_longer n=10 |
4/10 (2 cache stops at 24k) | 45/45 scored fires |
IR replay (the circuit's IR == the reference BIOS, on every computer-owned position of the DM records above): 30 records, 177,816 positions, 0 mismatches, 0 stray chip tokens.
Notes:
- The DM division sets are short: these runs are 10 problems each.
- Most losses there are cache-length stops. A non-terminating division now runs gcd to print
p/q, which costs 1-9k trace tokens (median 3.6k) instead of about 600.- With tc, Qwen doesn't see those tokens, but the static cache still holds them.
- Use
--max-len 32768or more on these sets, on a GPU with room: an A10G ran out of memory at 32k with the--fastCUDA graphs, and--chip-per-full 0turns those off.
- On a few multi-operation windows (markdown lists,
1.list numbers), the scorer's attribution of an inline fire to an expression is heuristic. The IR replay is the exactness check. - Not run for this release: text KL vs base, GSM8K and MMLU.
Tests (CPU, one file at a time):
- The q35 suite passes: ir, ir_vheads, mute, numerics, macros 19, expr 47, v2, chip8, inline 113.
test_q35_overlaypasses 5/5 (noisy-IR leak checks). The merged-checkpoint overlay test was run on the GPU and passes.test_q35_store: 10/11. One store-only failure,test_circuit_with_arithmetic: with--store, the store's tool response lacks the tool-response thinking tail (owner bits at its end). The store is optional and not in this checkpoint.
Not in this release:
- v17 (decimal operands for the single-op path). Its BIOS (≈550 B) doesn't fit the 4 KB next to v18's evaluator. Decimals work through the expression evaluator instead.
- The maths library (pow, sqrt, logs, …).