Skip to content

A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.

RISC-V model is now the main Qwenputer on HF (CHIP-8 moved to -CHIP8); sync experimental/riscv to rv-p3b c94709a (executor in generation, batching, exact /)

Divyansh Agrawalb25289baa30c
diff --git a/README.md b/README.md
index 4be3ba7..62654af 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,7 @@
> A computer, compiled into the weights of a language model.
-[Model on Hugging Face](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer) · [The standalone CHIP-8 transformer it grew from](https://github.com/divagr18/chipformer) · [MIT License](LICENSE)
+[RISC-V model](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer) · [CHIP-8 model](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer-CHIP8) · [The standalone CHIP-8 transformer it grew from](https://github.com/divagr18/chipformer) · [MIT License](LICENSE)
Qwenputer is Qwen3.5-4B with a working computer compiled directly into its weights. It isn't a
plugin, a tool or a sandbox. It's a CPU, a big-number maths library and its memory, written as
@@ -114,10 +114,13 @@ then reads the printed answer like ordinary text.
## Quickstart
+This runs the CHIP-8 model, which the benchmarks above come from. The RISC-V model is
+[further down](#a-risc-v-computer-inside-qwen).
+
```bash
git clone https://github.com/divagr18/qwenputer && cd qwenputer
uv sync
-hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-4b
+hf download divagr1925/Qwen3.5-4B-Qwenputer-CHIP8 --local-dir qwenputer-4b
uv run python -m chipformer.q35.demo qwenputer-4b # Qwen uses its tool-call syntax
uv run python -m chipformer.q35.demo qwenputer-4b --inline --nudge # plain inline arithmetic
@@ -158,17 +161,42 @@ details:
- the optional decoder settings our benchmarks used (`fp` and `tc` in `BatchDecoder`), which make
inline mode stronger
-## Experimental: a RISC-V computer inside Qwen
+## A RISC-V computer inside Qwen
+
+[`divagr1925/Qwen3.5-4B-Qwenputer`](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer)
+compiles a 32-bit **RV32IM** CPU, with Zba and a bignum coprocessor, into Qwen3.5-4B. Its code
+ROM, RAM, registers and program counter all live in Gated DeltaNet state, in 11 chip layers
+before Qwen's.
+
+The CPU passes the official riscv-tests suites (`rv32ui`, `rv32um`, `rv32uzba`), runs
+gcc-compiled C programs exactly (about 18 million instructions verified) and runs a patched
+MicroPython build exactly. Inside Qwen, it backs Qwen's `calc` tool: Qwen writes the call, the
+RISC-V machine in the weights runs a C calculator firmware, and the model prints the
+`<tool_response>` itself.
+
+| DeepMind set (40 problems) | Qwen + RISC-V in the weights | Qwen + Python tool (host) |
+|---|---|---|
+| arithmetic (mixed) | 39 | 39 |
+| `add_or_sub_big` | 39 | 39 |
+| `add_sub_multiple_longer` | 40 | 40 |
+| `div_big` | 38 | 39 |
+| `mul_big` | 39 | 40 |
-[`experimental/riscv/`](experimental/riscv) takes the idea further. It compiles a 32-bit
-**RV32IM** CPU, with Zba and a bignum coprocessor, into the same Qwen3.5 layer types.
+Replaying all the arithmetic runs and samples from every DeepMind set through the reference
+machine found 0 mismatches. In generation the
+machine runs on a Triton executor at about 3,200 trace tokens a second on an L4.
+
+```bash
+cd experimental/riscv && uv sync
+hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-rv
+uv run python -m chipformer.rv.chat qwenputer-rv --firmware tests/data/rv/calc-bridge.elf --ask "Use the calc tool: what is 123456789 * 987654321?"
+```
-It passes the official riscv-tests suites (`rv32ui`, `rv32um`, `rv32uzba`). It runs
-gcc-compiled C programs exactly, about 18 million instructions verified so far, and it runs a
-patched MicroPython build exactly. Inside Qwen3.5-4B it already answers end to end: Qwen asks,
-the RISC-V machine in the weights computes, Qwen reads the answer. It is a work in progress;
-the compact executor currently runs about 3,700 instructions a second per stream on an L4.
-[`experimental/riscv/docs/rv32.md`](experimental/riscv/docs/rv32.md) has the details.
+Unlike the CHIP-8 model, this one isn't driven by a plain `generate()` call yet. `RvChat` loads
+the firmware image into the DeltaNet state after the prompt, and its decoding loop keeps the
+machine's trace tokens out of Qwen's own layers. The weights alone decide every token.
+[`experimental/riscv/docs/rv32.md`](experimental/riscv/docs/rv32.md) has the design and every
+measurement.
## Citing
diff --git a/docs/running.md b/docs/running.md
index af7fd7b..2143f08 100644
--- a/docs/running.md
+++ b/docs/running.md
@@ -108,7 +108,7 @@ end.
## Release checks (A10G 24 GB, 2026-10-19)
-**The checkpoint.** [divagr1925/Qwen3.5-4B-Qwenputer](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer).
+**The checkpoint.** [divagr1925/Qwen3.5-4B-Qwenputer-CHIP8](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer-CHIP8).
- Size: 14.65 GB in 7 shards.
- Contents: config, tokenizer, `chat_template.jinja`, `generation_config.json`, `chipformer.json`,
LICENSE, preprocessor configs.
diff --git a/experimental/riscv/README.md b/experimental/riscv/README.md
index af5e7e1..0a17455 100644
--- a/experimental/riscv/README.md
+++ b/experimental/riscv/README.md
@@ -1,4 +1,6 @@
-# Experimental: a RISC-V computer inside Qwen
+# A RISC-V computer inside Qwen
+
+Model: [divagr1925/Qwen3.5-4B-Qwenputer](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer).
This folder compiles a 32-bit **RV32IM** CPU into stock Qwen3.5 layers, using the same compiler as
the main project. The CPU also has Zba and a bignum coprocessor. It is a self-contained snapshot:
@@ -20,7 +22,7 @@ merged with the main tree yet.
The machine has 256 KB of code ROM, 128 KB of read-only data and 32 KB of RAM, all held in Gated
DeltaNet state. It takes 11 chip layers and about 1M nonzero weights, and costs about 1.2-1.4
tokens per instruction. A compact executor runs about 3,700 instructions a second per stream on
-an L4.
+an L4, and in generation it runs at about 3,200 trace tokens a second.
[`docs/rv32.md`](docs/rv32.md) has the design and every measurement, and
[`docs/rv32-p3-status.md`](docs/rv32-p3-status.md) has the in-Qwen status.
diff --git a/experimental/riscv/chipformer/rv/batchcheck.py b/experimental/riscv/chipformer/rv/batchcheck.py
new file mode 100644
index 0000000..1a9f20f
--- /dev/null
+++ b/experimental/riscv/chipformer/rv/batchcheck.py
@@ -0,0 +1,79 @@
+"""Batched rv1 decoding (rvbatch) against batch-1 records: the same prompts, decoded N at a time,
+must give each record's tokens.
+
+ python -m chipformer.rv.batchcheck MODEL REC.jsonl:ID [...] --batch 4 [--batch 8 ...]
+"""
+
+import argparse
+import json
+import time
+
+from chipformer.rv import bridge as BR
+from chipformer.rv import elf, ref
+from chipformer.rv.chat import RvChat
+from chipformer.rv.rvbatch import Job, RvBatch
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("model")
+ ap.add_argument("records", nargs="+")
+ ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")
+ ap.add_argument("--batch", type=int, action="append", required=True)
+ ap.add_argument("--budget", type=int, default=4096)
+ ap.add_argument("--max-chip", type=int, default=200000)
+ ap.add_argument("--max-len", type=int, default=6144)
+ ap.add_argument("--local-ref", action="store_true",
+ help="compare with RvChat.generate here instead of the records' tokens")
+ args = ap.parse_args()
+ recs = []
+ for spec in args.records:
+ path, rid = spec.rsplit(":", 1)
+ rows = [json.loads(x) for x in open(path, encoding="utf-8")]
+ recs.append(next(r for r in rows if r["id"] == rid))
+ chat = RvChat(args.model, executor=True)
+
+ def job(r):
+ m = ref.Machine.from_program(elf.program(args.firmware))
+ BR.park(m)
+ m2 = ref.Machine.from_program(elf.program(args.firmware))
+ BR.park(m2)
+ lock = BR.Session(chat.T, m2)
+ lock.history(r["prompt_ids"])
+ return Job(r["id"], r["prompt_ids"], m, lock, {"r": r})
+
+ if args.local_ref: # the reference: RvChat.generate (batch 1) on this GPU
+ for r in recs:
+ j = job(r)
+ t0 = time.time()
+ out = chat.generate(j.ids, j.machine, args.budget, max_chip=args.max_chip,
+ check_head=True, lock=j.lock)
+ same = out["tokens"] == r["gen_ids"]
+ r["gen_ids"] = out["tokens"]
+ print(json.dumps({"batch": "generate", "id": r["id"], "same_as_record": same,
+ "len": len(out["tokens"]), "seconds": round(time.time() - t0, 1),
+ "lock_error": out["lock_error"]}), flush=True)
+ for B in args.batch:
+ rb = RvBatch(chat, B, args.max_len)
+ res = {}
+ t0 = time.time()
+ st = rb.run((job(r) for r in recs), args.budget, args.max_chip,
+ lambda j, out: res.__setitem__(j.key, out))
+ wall = time.time() - t0
+ for r in recs:
+ got, want = res[r["id"]]["tokens"], r["gen_ids"]
+ first = next((i for i, (a, b) in enumerate(zip(got, want)) if a != b),
+ None if len(got) == len(want) else min(len(got), len(want)))
+ print(json.dumps({"batch": B, "id": r["id"], "same_tokens": got == want,
+ "first_diff": first, "len": len(got), "record_len": len(want),
+ "lock_error": res[r["id"]]["lock_error"],
+ "head_mismatch": res[r["id"]]["head_mismatch"],
+ "diff_ctx": chat.tok.decode([t for t in want[max(0, (first or 0) - 8):(first or 0) + 4] if t < chat.v0]) if first is not None else ""}),
+ flush=True)
+ print(json.dumps({"batch": B, "wall_s": round(wall, 1), "stats": st,
+ "problems_per_hour": round(len(recs) / wall * 3600, 1)}), flush=True)
+ del rb
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experimental/riscv/chipformer/rv/chat.py b/experimental/riscv/chipformer/rv/chat.py
index fa69e7e..05b6717 100644
--- a/experimental/riscv/chipformer/rv/chat.py
+++ b/experimental/riscv/chipformer/rv/chat.py
@@ -38,8 +38,67 @@ CALC_TOOL = {"type": "function", "function": {
"required": ["expression"]}}}
+class ExecRow:
+ """One conversation's circuit on the Triton executor (batch 1) with its own CUDA graph."""
+
+ def __init__(self, chat: "RvChat") -> None:
+ from chipformer.rv.tritonexec import Executor2
+
+ self.chat = chat
+ self.ex = Executor2(chat.c, 1, chat.dev)
+ self.heads = [st for st in self.ex.steps if st["kind"] == "head"]
+ self.xbuf = torch.zeros(1, self.ex.D, device=chat.dev)
+ self.kbuf = torch.zeros(len(chat.keep_cols), device=chat.dev)
+ self.graph = None
+
+ def _body(self):
+ self.ex.step(self.xbuf)
+ self.kbuf.copy_(self.xbuf[0, self.chat.keep_ir])
+
+ @torch.no_grad()
+ def _capture(self):
+ """Warm-up runs change the state, so it is saved and put back around them."""
+ saved = [(st["S"].clone(), st["hist"].clone()) for st in self.heads]
+ s = torch.cuda.Stream()
+ s.wait_stream(torch.cuda.current_stream())
+ with torch.cuda.stream(s):
+ for _ in range(3):
+ self._body()
+ torch.cuda.current_stream().wait_stream(s)
+ self.graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(self.graph):
+ self._body()
+ for st, (S_, h) in zip(self.heads, saved):
+ st["S"].copy_(S_)
+ st["hist"].copy_(h)
+
+ @torch.no_grad()
+ def start(self, ids: list[int]) -> None:
+ """As the HF path does it: the prompt through the circuit from zero states, then every
+ head's state set to its init (the snapshot now in the circuit; hf.inject), the lag
+ windows kept."""
+ if self.graph is None:
+ self._capture()
+ for st in self.heads:
+ st["S"].zero_()
+ st["hist"].zero_()
+ emb = self.chat._exec_emb(ids)
+ for t in range(len(ids)):
+ self.xbuf.copy_(emb[t:t + 1])
+ self.graph.replay()
+ self.ex.load_init()
+
+ @torch.no_grad()
+ def step(self, ids: int) -> torch.Tensor:
+ """One token through the circuit; returns the kept values (fp32, the code first)."""
+ self.xbuf.copy_(self.chat._exec_emb([ids]))
+ self.graph.replay()
+ return self.kbuf
+
+
class RvChat:
- def __init__(self, path, device: str | None = None, fast: bool = True) -> None:
+ def __init__(self, path, device: str | None = None, fast: bool = True,
+ executor: bool = False) -> None:
from transformers import AutoTokenizer, Qwen3_5ForCausalLM
from chipformer.q35.fast import exact_matmuls
@@ -61,8 +120,68 @@ class RvChat:
self.code_dims = torch.tensor(self.meta["code_dims"], device=self.dev)
self.eos = {248046, 248044}
self.fast = fast and self.dev == "cuda"
- if self.fast:
+ # executor "verify": both chip paths, the HF view's hc checked against the executor's
+ self.executor = executor if self.dev == "cuda" else False
+ self.verify = self.executor == "verify"
+ if self.executor:
+ self._build_exec()
+ if self.fast and (not self.executor or self.verify):
self._build_fast()
+ self.fast = self.fast and (not self.executor or self.verify)
+
+ # --- the executor path -----------------------------------------------------------------
+ def _build_exec(self) -> None:
+ """The chip layers as the circuit itself (tritonexec.Executor2: the IR's arithmetic in
+ fp32, one CUDA graph per token) instead of their HF modules. After the chip layers the
+ merged residual holds Qwen's embedding dims, the kept dims (output code, tau, mu) and
+ zeros (the exit clears every other chip dim and the ballast), so a text token's hc is
+ built from the embedding row and the executor's kept values; chip tokens need only the
+ code."""
+ self.keep_ir = torch.tensor(self.c.keep, device=self.dev)
+ cols = self.meta["code_dims"] + [self.meta["tau_dim"], self.meta["mu_dim"]]
+ assert len(cols) == len(self.c.keep)
+ self.keep_cols = torch.tensor(cols, device=self.dev)
+ self.d0 = self.meta["chip_dims_offset"]
+ self.emb_cache: dict[int, torch.Tensor] = {}
+ self.erow = ExecRow(self)
+
+ def _exec_emb(self, ids: list[int]) -> torch.Tensor:
+ """Executor input rows for model ids (cached per id; shared by every ExecRow)."""
+ new = sorted({i for i in ids if i not in self.emb_cache})
+ if new:
+ rows = self.erow.ex.embed([i - self.v0 if i >= self.v0 else BR.text_token(i)
+ for i in new])
+ for i, r in zip(new, rows):
+ self.emb_cache[i] = r
+ return torch.stack([self.emb_cache[i] for i in ids])
+
+ def _exec_start(self, ids: list[int]) -> None:
+ self.erow.start(ids)
+
+ def _exec_step(self, ids: int) -> torch.Tensor:
+ return self.erow.step(ids)
+
+ def _exec_hc(self, ids: int, keep: torch.Tensor) -> torch.Tensor:
+ """A text token's residual after the chip layers."""
+ hc = torch.zeros(self.lm.embed_tokens.weight.shape[1], dtype=torch.bfloat16,
+ device=self.dev)
+ hc[: self.d0] = self.lm.embed_tokens.weight[ids, : self.d0]
+ hc[self.keep_cols] = keep.to(torch.bfloat16)
+ return hc
+
+ def code_token(self, code) -> int | None:
+ """circuit_token from the output code values."""
+ code = code.float()
+ if not bool((code != 0).any()):
+ return None
+ bits = (code > 0).long().tolist()
+ nf = core.NFAM
+ fid = sum(b << i for i, b in enumerate(bits[:nf]))
+ p = sum(b << i for i, b in enumerate(bits[nf:]))
+ t = core.decode_code(fid, p, self.T)
+ if t < 0:
+ raise RuntimeError(f"the circuit predicts no token (family {fid}, payload {p})")
+ return t - S.TEXT_TOKEN if S.is_text(t) else self.v0 + t
# --- the fast chip path ----------------------------------------------------------------
def _build_fast(self) -> None:
@@ -190,17 +309,7 @@ class RvChat:
def circuit_token(self, hc) -> int | None:
"""The model id the circuit predicts at a position, None where Qwen predicts."""
- code = hc[self.code_dims].float()
- if not bool((code != 0).any()):
- return None
- bits = (code > 0).long().tolist()
- nf = core.NFAM
- fid = sum(b << i for i, b in enumerate(bits[:nf]))
- p = sum(b << i for i, b in enumerate(bits[nf:]))
- t = core.decode_code(fid, p, self.T)
- if t < 0:
- raise RuntimeError(f"the circuit predicts no token (family {fid}, payload {p})")
- return t - S.TEXT_TOKEN if S.is_text(t) else self.v0 + t
+ return self.code_token(hc[self.code_dims])
# --- a conversation --------------------------------------------------------------------
def prompt_ids(self, messages, tools=(CALC_TOOL,)) -> list[int]:
@@ -222,6 +331,8 @@ class RvChat:
out = self.model(input_ids=x, use_cache=True)
cache = out.past_key_values # (fast: Qwen's layers only)
core.snapshot(self.c, machine)
+ if self.executor:
+ self._exec_start(ids)
if self.fast:
ccache = self._chip_cache()
h = self.cm.embed_tokens(x) # the prompt through the chip layers only
@@ -232,12 +343,12 @@ class RvChat:
if self.graph is None:
self._capture()
g = len(ids) # the chip view's cache position (every token)
- else:
+ elif not self.executor:
hf.inject(self.c, cache)
+ n_verify_bad = 0
logits = out.logits[0, -1]
pos = len(ids) # the next text token's RoPE position
gen, n_text, n_chip_tok = [], 0, 0
- stream_hc = []
# the prompt's last position: Qwen predicts (nothing captured yet)
nxt = int(torch.argmax(logits[: self.v0]))
t0 = time.time()
@@ -273,25 +384,34 @@ class RvChat:
print(f" {len(gen)} tokens ({n_text} text, {n_chip_tok} chip) "
f"{time.time() - t0:.0f}s: {' '.join(tail)}", flush=True)
ts = time.time()
- if nxt >= self.v0: # a chip token
+ chip_in = nxt >= self.v0
+ hc = keep = None
+ if self.executor:
+ keep = self._exec_step(nxt)
+ if not chip_in:
+ hc = self._exec_hc(nxt, keep)
+ if self.fast and (self.verify or not self.executor):
+ hf_hc = self._chip_step(nxt, g, ccache)
+ g += 1
+ if self.verify:
+ same = (bool((hf_hc[self.keep_cols] == keep.to(torch.bfloat16)).all())
+ if chip_in else bool((hf_hc == hc).all()))
+ if not same:
+ n_verify_bad += 1
+ hc = hf_hc
+ elif not self.executor:
+ hc, logits = self._layers(nxt, cache, None if chip_in else pos)
+ if chip_in: # a chip token
n_chip_tok += 1
- if self.fast:
- hc = self._chip_step(nxt, g, ccache)
- g += 1
- else:
- hc, _ = self._layers(nxt, cache, None)
- nxt = self.circuit_token(hc)
+ nxt = (self.code_token(keep[: len(self.code_dims)]) if hc is None
+ else self.circuit_token(hc))
t_chip += time.time() - ts
if nxt is None:
raise RuntimeError("the circuit gave up a chip position")
else:
n_text += 1
- if self.fast:
- hc = self._chip_step(nxt, g, ccache)
- g += 1
+ if self.fast or self.executor:
logits = self._qwen_step(hc, cache, pos)
- else:
- hc, logits = self._layers(nxt, cache, pos)
pos += 1
ct = self.circuit_token(hc)
q = int(torch.argmax(logits))
@@ -299,11 +419,10 @@ class RvChat:
mismatch += 1
nxt = ct if ct is not None else int(torch.argmax(logits[: self.v0]))
t_text += time.time() - ts
- if check_head:
- stream_hc.append(hc[self.code_dims].float().cpu().numpy())
dt = time.time() - t0
return {"tokens": gen, "text_tokens": n_text, "chip_tokens": n_chip_tok,
"seconds": dt, "head_mismatch": mismatch, "lock_error": lock_error,
+ "verify_mismatch": n_verify_bad if self.verify else None,
"chip_seconds": round(t_chip, 2), "text_seconds": round(t_text, 2)}
def to_stream(self, ids: list[int]) -> list[int]:
diff --git a/experimental/riscv/chipformer/rv/evalarith.py b/experimental/riscv/chipformer/rv/evalarith.py
index 34f2190..ca8c417 100644
--- a/experimental/riscv/chipformer/rv/evalarith.py
+++ b/experimental/riscv/chipformer/rv/evalarith.py
@@ -85,6 +85,12 @@ def main() -> None:
ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")
ap.add_argument("--budget", type=int, default=4096, help="Qwen tokens per problem")
ap.add_argument("--max-chip", type=int, default=200000)
+ ap.add_argument("--chip-path", default="executor", choices=("executor", "hf"),
+ help="rv1's chip layers: the circuit's Triton executor, or their HF modules")
+ ap.add_argument("--batch", type=int, default=1, help="rv1: conversations decoded at once")
+ ap.add_argument("--max-len", type=int, default=6144, help="rv1 --batch: Qwen KV slots")
+ ap.add_argument("--shard", default=None, help="i/k: every k-th problem from the i-th")
+ ap.add_argument("--ids", default=None, help="comma-separated problem ids (a subset)")
args = ap.parse_args()
n = args.n or (40 if args.data == "arith" else None)
probs = problems(args.data, n, args.seed)
@@ -93,7 +99,7 @@ def main() -> None:
done = {json.loads(x)["id"] for x in out.read_text().splitlines()} if out.exists() else set()
tool = args.cond != "base_nudge"
if args.cond == "rv1":
- chat = RvChat(args.model)
+ chat = RvChat(args.model, executor=args.chip_path == "executor")
tok = chat.tok
else:
from transformers import AutoModelForImageTextToText, AutoTokenizer
@@ -101,46 +107,132 @@ def main() -> None:
tok = AutoTokenizer.from_pretrained(args.model)
model = AutoModelForImageTextToText.from_pretrained(args.model, dtype=torch.bfloat16)
model = model.to("cuda").eval()
- for p in probs:
- if p["id"] in done:
- continue
- msgs = [{"role": "system", "content": NUDGE if tool else NUDGE_NONE},
+ if args.shard:
+ i, k = map(int, args.shard.split("/"))
+ probs = probs[i::k]
+ if args.ids:
+ want = set(args.ids.split(","))
+ probs = [p for p in probs if p["id"] in want]
+ todo = [p for p in probs if p["id"] not in done]
+ t_all = time.time()
+
+ def msgs_of(p):
+ return [{"role": "system", "content": NUDGE if tool else NUDGE_NONE},
{"role": "user", "content": p["question"] + E.SUFFIX}]
- t0 = time.time()
+
+ def rv1_job(p):
+ ids = chat.prompt_ids(msgs_of(p))
+ m = ref.Machine.from_program(elf.program(args.firmware))
+ BR.park(m)
+ m2 = ref.Machine.from_program(elf.program(args.firmware))
+ BR.park(m2)
+ lock = BR.Session(chat.T, m2) # every circuit token checked against the reference
+ lock.history(ids)
+ return ids, m, lock
+
+ def rv1_fields(ids, lock, r):
+ gen = r["tokens"]
+ outs = [chat.T.decode_bytes(e).decode("utf-8", "replace") for e in lock.runs]
+ return chat.text_of(gen), dict(
+ prompt_ids=ids, gen_ids=gen, calls=len(lock.runs),
+ n_qwen=r["text_tokens"] - sum(len(e) for e in lock.runs),
+ n_chip=r["chip_tokens"], lock_error=r["lock_error"],
+ head_mismatch=r["head_mismatch"], gen_seconds=round(r["seconds"], 1),
+ chip_path=args.chip_path, chip_seconds=r["chip_seconds"],
+ text_seconds=r["text_seconds"], batch=args.batch,
+ results=[o.split("\n")[2] if o.count("\n") > 2 else o for o in outs])
+
+ def emit(p, text, fields, seconds):
rec = {"id": p["id"], "gold": p["gold"], "question": p["question"], "cond": args.cond}
+ rec.update(fields)
+ rec.update(AB.exact_fields(p, text))
+ rec.update(seconds=round(seconds, 1), text=text)
+ with out.open("a", encoding="utf-8") as f:
+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")
+ keys = ("id", "gold", "pred_exact", "exact", "calls", "n_qwen", "n_chip", "seconds",
+ "lock_error", "head_mismatch")
+ print(json.dumps({k: rec[k] for k in keys if k in rec}), flush=True)
+
+ if args.cond == "rv1" and args.batch > 1:
+ from chipformer.rv.rvbatch import Job, RvBatch
+
+ rb = RvBatch(chat, args.batch, args.max_len)
+
+ def jobs():
+ for p in todo:
+ ids, m, lock = rv1_job(p)
+ yield Job(p["id"], ids, m, lock, {"p": p})
+
+ def finished(job, r):
+ text, fields = rv1_fields(job.ids, job.lock, r)
+ emit(job.extra["p"], text, fields, r["seconds"])
+
+ st = rb.run(jobs(), args.budget, args.max_chip, finished)
+ print(json.dumps({"batch_stats": st}), flush=True)
+ if args.cond != "rv1" and args.batch > 1: # base Qwen: q35's BatchDecoder
+ from chipformer.q35.batch import BatchDecoder
+
+ prompts = []
+ for p in todo:
+ ids = tok.apply_chat_template(msgs_of(p), tools=[CALC_TOOL] if tool else None,
+ add_generation_prompt=True, enable_thinking=False,
+ tokenize=True)
+ prompts.append(ids if isinstance(ids, list) else ids["input_ids"])
+ tcc = tok.convert_tokens_to_ids("</tool_call>")
+ state = [{"forced": [], "own": 0, "gen": [], "calls": 0} for _ in todo]
+
+ def hook(i, t):
+ """pytool: after </tool_call> the next tokens are the host's response (forced);
+ the row stops after ``budget`` of Qwen's own tokens, as run_base does."""
+ s = state[i]
+ if s["forced"]:
+ t = s["forced"].pop(0)
+ s["gen"].append(t)
+ return t, False
+ s["own"] += 1
+ s["gen"].append(t)
+ if tool and t == tcc:
+ txt = tok.decode(s["gen"], skip_special_tokens=False)
+ mm = re.findall(r"<parameter=[^>]*>(.*?)</parameter>",
+ txt.split("<tool_call>")[-1], re.S)
+ s["forced"] = tok(calc_response(mm[0] if mm else ""),
+ add_special_tokens=False).input_ids
+ s["calls"] += 1
+ return t, s["own"] >= args.budget and not s["forced"]
+
+ t0 = time.time()
+ max_len = -(-(max(map(len, prompts)) + 2 * args.budget) // 1024) * 1024
+ bd = BatchDecoder(model, batch=args.batch, max_len=max_len)
+
+ def fin(i, toks):
+ s = state[i]
+ text = tok.decode(toks, skip_special_tokens=False)
+ emit(todo[i], text, dict(calls=s["calls"], n_qwen=s["own"], n_chip=0,
+ batch=args.batch), time.time() - t0)
+
+ if prompts:
+ bd.run(prompts, 2 * args.budget, on_done=fin, hook=hook)
+ for p in todo if args.batch == 1 else []:
+ t0 = time.time()
if args.cond == "rv1":
- ids = chat.prompt_ids(msgs)
- m = ref.Machine.from_program(elf.program(args.firmware))
- BR.park(m)
- m2 = ref.Machine.from_program(elf.program(args.firmware))
- BR.park(m2)
- lock = BR.Session(chat.T, m2) # every circuit token checked against the reference
- lock.history(ids)
+ ids, m, lock = rv1_job(p)
r = chat.generate(ids, m, args.budget, max_chip=args.max_chip, check_head=True,
lock=lock)
- gen = r["tokens"]
- text = chat.text_of(gen)
- outs = [chat.T.decode_bytes(e).decode("utf-8", "replace") for e in lock.runs]
- rec.update(prompt_ids=ids, gen_ids=gen, calls=len(lock.runs),
- n_qwen=r["text_tokens"] - sum(len(e) for e in lock.runs),
- n_chip=r["chip_tokens"], lock_error=r["lock_error"],
- head_mismatch=r["head_mismatch"], gen_seconds=round(r["seconds"], 1),
- results=[o.split("\n")[2] if o.count("\n") > 2 else o for o in outs])
+ text, fields = rv1_fields(ids, lock, r)
else:
- ids = tok.apply_chat_template(msgs, tools=[CALC_TOOL] if tool else None,
+ ids = tok.apply_chat_template(msgs_of(p), tools=[CALC_TOOL] if tool else None,
add_generation_prompt=True, enable_thinking=False,
tokenize=True)
ids = ids if isinstance(ids, list) else ids["input_ids"]
gen, calls, own = run_base(model, tok, ids, tool, args.budget)
text = tok.decode(gen, skip_special_tokens=False)
- rec.update(calls=calls, n_qwen=own, n_chip=0)
- rec.update(AB.exact_fields(p, text))
- rec.update(seconds=round(time.time() - t0, 1), text=text)
- with out.open("a", encoding="utf-8") as f:
- f.write(json.dumps(rec, ensure_ascii=False) + "\n")
- keys = ("id", "gold", "pred_exact", "exact", "calls", "n_qwen", "n_chip", "seconds",
- "lock_error", "head_mismatch")
- print(json.dumps({k: rec[k] for k in keys if k in rec}), flush=True)
+ fields = dict(calls=calls, n_qwen=own, n_chip=0)
+ emit(p, text, fields, time.time() - t0)
+ wall = time.time() - t_all
+ if todo:
+ print(json.dumps({"problems": len(todo), "wall_s": round(wall, 1),
+ "problems_per_hour": round(len(todo) / wall * 3600, 1),
+ "batch": args.batch}), flush=True)
rows = [json.loads(x) for x in out.read_text().splitlines()]
print(f"{out.name}: {sum(r['exact'] for r in rows)}/{len(rows)} exact")
diff --git a/experimental/riscv/chipformer/rv/execcheck.py b/experimental/riscv/chipformer/rv/execcheck.py
new file mode 100644
index 0000000..4b1d8a9
--- /dev/null
+++ b/experimental/riscv/chipformer/rv/execcheck.py
@@ -0,0 +1,67 @@
+"""The executor path of RvChat against the HF chip path, on rv1 eval records.
+
+ python -m chipformer.rv.execcheck MODEL REC.jsonl:ID [REC.jsonl:ID ...] [--no-verify]
+
+Per record: (1) "verify" mode: the HF chip view drives generation (it must reproduce the
+record's tokens) and the executor runs beside it, its residual after the chip layers compared
+bitwise at every position (the whole hc on text tokens, the kept dims on chip tokens); (2) the
+executor alone: its tokens must equal the record's. Prints both timings.
+"""
+
+import argparse
+import json
+
+from chipformer.rv import bridge as BR
+from chipformer.rv import elf, ref
+from chipformer.rv.chat import RvChat
+
+
+def run(chat, r, firmware, budget, max_chip):
+ m = ref.Machine.from_program(elf.program(firmware))
+ BR.park(m)
+ m2 = ref.Machine.from_program(elf.program(firmware))
+ BR.park(m2)
+ lock = BR.Session(chat.T, m2)
+ lock.history(r["prompt_ids"])
+ return chat.generate(r["prompt_ids"], m, budget, max_chip=max_chip, check_head=True, lock=lock)
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("model")
+ ap.add_argument("records", nargs="+")
+ ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")
+ ap.add_argument("--budget", type=int, default=4096)
+ ap.add_argument("--max-chip", type=int, default=200000)
+ ap.add_argument("--no-verify", action="store_true")
+ args = ap.parse_args()
+ recs = []
+ for spec in args.records:
+ path, rid = spec.rsplit(":", 1)
+ rows = [json.loads(x) for x in open(path, encoding="utf-8")]
+ recs.append(next(r for r in rows if r["id"] == rid))
+ chat = RvChat(args.model, executor="verify")
+ for r in recs:
+ out = {"id": r["id"], "record_tokens": len(r["gen_ids"])}
+ if not args.no_verify:
+ chat.verify, chat.fast = True, True
+ v = run(chat, r, args.firmware, args.budget, args.max_chip)
+ out["hf"] = {"same_tokens": v["tokens"] == r["gen_ids"],
+ "verify_mismatch": v["verify_mismatch"], "lock_error": v["lock_error"],
+ "chip_tokens": v["chip_tokens"], "chip_s": v["chip_seconds"],
+ "text_s": v["text_seconds"]}
+ chat.verify, chat.fast = False, False
+ e = run(chat, r, args.firmware, args.budget, args.max_chip)
+ first = next((i for i, (a, b) in enumerate(zip(e["tokens"], r["gen_ids"])) if a != b),
+ None if len(e["tokens"]) == len(r["gen_ids"]) else
+ min(len(e["tokens"]), len(r["gen_ids"])))
+ out["exec"] = {"same_tokens": e["tokens"] == r["gen_ids"], "first_diff": first,
+ "lock_error": e["lock_error"], "head_mismatch": e["head_mismatch"],
+ "chip_tokens": e["chip_tokens"], "chip_s": e["chip_seconds"],
+ "text_s": e["text_seconds"], "seconds": round(e["seconds"], 1),
+ "chip_tok_per_s": round(e["chip_tokens"] / max(e["chip_seconds"], 1e-9))}
+ print(json.dumps(out), flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experimental/riscv/chipformer/rv/replay.py b/experimental/riscv/chipformer/rv/replay.py
index d7d9b65..9fb70b2 100644
--- a/experimental/riscv/chipformer/rv/replay.py
+++ b/experimental/riscv/chipformer/rv/replay.py
@@ -47,12 +47,12 @@ def replay_ids(c, T, v0, prompt, gen, firmware) -> dict:
m2 = ref.Machine.from_program(elf.program(firmware))
BR.park(m2)
core.snapshot(c, m2)
- carry, mats = {}, []
+ carry, preds, owns = {}, [], [] # per chunk: a long stream's whole matrix is GBs
for a in range(0, len(stream), 2048):
- mats.append(ir.evaluate(c, stream[a:a + 2048], carry=carry))
- mat = np.concatenate(mats)
- pred = core.predictions(c, mat)
- own = np.rint(ir.ev(c.owner, mat)).astype(int)
+ mat = ir.evaluate(c, stream[a:a + 2048], carry=carry)
+ preds.append(core.predictions(c, mat))
+ owns.append(np.rint(ir.ev(c.owner, mat)).astype(int))
+ pred, own = np.concatenate(preds), np.concatenate(owns)
bad_own = int(sum(own[x] != int(ses.own[x]) for x in range(len(stream))))
bad = int(sum(1 for x in range(len(stream) - 1)
if ses.own[x] and x not in skip and pred[x] != stream[x + 1]))
diff --git a/experimental/riscv/chipformer/rv/rvbatch.py b/experimental/riscv/chipformer/rv/rvbatch.py
new file mode 100644
index 0000000..eccb3fb
--- /dev/null
+++ b/experimental/riscv/chipformer/rv/rvbatch.py
@@ -0,0 +1,191 @@
+"""Batched rv1 decoding: N conversations at once on one GPU.
+
+ rb = RvBatch(chat, batch=8, max_len=8192) # chat: RvChat(path, executor=True)
+ rb.run(jobs, budget, max_chip, done) # jobs: iterable of Job; done(job, result)
+
+Each conversation (a row) keeps its own circuit (chat.ExecRow: the Triton executor at batch 1
+with its own CUDA graph) and runs its chip bursts there on its own, as RvChat.generate does. Its
+text tokens go through Qwen's layers in one batched step for all live rows, on a cache with
+per-row positions (q35.batch._Rows: full attention writes row b's key at its own slot and the
+mask admits slots <= pos[b]; DeltaNet rows are independent). A row's prompt is prefilled alone
+(batch 1, as generate does) and copied into its row. A finished row is refilled from the queue.
+
+Each row is a generator (``_row``) with RvChat.generate's executor-path logic: it yields the
+residual hc of each text token after the chip layers and receives Qwen's two argmaxes (over the
+whole vocabulary, for the head check, and over Qwen's own vocabulary). Only Qwen's GEMMs see the
+batch (M = live rows instead of 1), which can change their bf16 rounding; the circuit is exact
+either way.
+"""
+
+import time
+from dataclasses import dataclass, field
+
+import torch
+
+from chipformer.rv import bridge as BR
+from chipformer.rv import core
+from chipformer.rv import spec as S
+
+
+@dataclass
+class Job:
+ key: object
+ ids: list[int]
+ machine: object # a parked ref.Machine
+ lock: object = None # a BR.Session on a copy (history = the prompt), or None
+ extra: dict = field(default_factory=dict)
+
+
+class RvBatch:
+ def __init__(self, chat, batch: int = 8, max_len: int = 8192) -> None:
+ from chipformer.rv.chat import ExecRow
+
+ assert chat.executor and not chat.verify
+ self.chat, self.B, self.max_len = chat, batch, max_len
+ self.dev = chat.dev
+ self.rows = [chat.erow] + [ExecRow(chat) for _ in range(batch - 1)]
+ self.pos = torch.zeros(batch, dtype=torch.long, device=self.dev)
+ self.slots = torch.arange(max_len, device=self.dev)
+ self.cache = None # _Rows, built from the first prefill's cache (its shapes)
+ cfg = chat.model.config
+ self.kinds = list(cfg.layer_types)
+
+ # --- Qwen's batched step -----------------------------------------------------------------
+ @torch.no_grad()
+ def _qwen(self, H: torch.Tensor):
+ """Qwen's layers on one text token per row (residuals H [B, hidden] after the chip
+ layers, RoPE positions / KV slots self.pos); returns the argmaxes (all, Qwen's)."""
+ chat = self.chat
+ h = H[:, None, :]
+ p4 = self.pos.view(1, -1, 1).expand(4, -1, 1)
+ pe = chat.lm.rotary_emb(h, p4[1:])
+ mask = (self.slots[None, :] <= self.pos[:, None])[:, None, None, :]
+ for i in range(chat.n_chip, len(chat.lm.layers)):
+ full = self.kinds[i] == "full_attention"
+ h = chat.lm.layers[i](h, position_embeddings=pe,
+ attention_mask=mask if full else None,
+ position_ids=p4[0], past_key_values=self.cache.cache)
+ logits = chat.model.lm_head(chat.lm.norm(h))[:, -1]
+ both = torch.stack([logits.argmax(-1), logits[:, : chat.v0].argmax(-1)], 1)
+ return both.tolist()
+
+ @torch.no_grad()
+ def _prefill(self, b: int, ids: list[int]) -> int:
+ from chipformer.q35.batch import _Rows
+
+ chat = self.chat
+ out = chat.model(input_ids=torch.tensor([ids], device=self.dev), use_cache=True)
+ small = out.past_key_values
+ if self.cache is None:
+ self.cache = _Rows(chat.model.config, small, self.B, self.max_len, self.pos)
+ self.cache.load_row(b, small, len(ids))
+ return int(torch.argmax(out.logits[0, -1][: chat.v0]))
+
+ # --- one conversation --------------------------------------------------------------------
+ def _row(self, b: int, job: Job, budget: int, max_chip: int, check_head: bool):
+ """RvChat.generate's executor path as a generator: yields (hc, pos) for each text token,
+ receives (argmax, Qwen-vocabulary argmax); returns generate's result dict."""
+ chat, erow, lock = self.chat, self.rows[b], job.lock
+ ids = job.ids
+ nxt = self._prefill(b, ids)
+ core.snapshot(chat.c, job.machine)
+ erow.start(ids)
+ pos = len(ids)
+ gen, n_text, n_chip_tok = [], 0, 0
+ t0 = time.time()
+ mismatch = 0
+ t_chip = 0.0
+ expect, lock_error = [], None
+ while n_text < budget and n_chip_tok < max_chip:
+ gen.append(nxt)
+ if lock is not None:
+ st = nxt - chat.v0 if nxt >= chat.v0 else BR.text_token(nxt)
+ if expect:
+ e = expect.pop(0)
+ if st != e:
+ lock_error = {"at": len(gen) - 1, "got": S.name(st) if st < S.VOCAB
+ else chat.tok.decode([nxt]),
+ "want": S.name(e) if e < S.VOCAB
+ else chat.tok.decode([e - S.TEXT_TOKEN])}
+ break
+ elif st < S.VOCAB:
+ lock_error = {"at": len(gen) - 1, "got": S.name(st), "want": "text"}
+ break
+ else:
+ k = len(lock.toks)
+ lock.qwen([nxt])
+ expect = list(lock.toks[k + 1:])
+ if nxt in chat.eos:
+ break
+ ts = time.time()
+ keep = erow.step(nxt)
+ if nxt >= chat.v0: # a chip token
+ n_chip_tok += 1
+ nxt = chat.code_token(keep[: len(chat.code_dims)])
+ t_chip += time.time() - ts
+ if nxt is None:
+ raise RuntimeError("the circuit gave up a chip position")
+ else:
+ n_text += 1
+ if pos >= self.max_len:
+ lock_error = {"at": len(gen) - 1, "got": "no KV slot left", "want": "text"}
+ break
+ hc = chat._exec_hc(nxt, keep)
+ ct = chat.circuit_token(hc)
+ q, qv = yield hc, pos
+ pos += 1
+ if ct is not None and check_head and q != ct:
+ mismatch += 1
+ nxt = ct if ct is not None else qv
+ return {"tokens": gen, "text_tokens": n_text, "chip_tokens": n_chip_tok,
+ "seconds": time.time() - t0, "head_mismatch": mismatch,
+ "lock_error": lock_error, "verify_mismatch": None,
+ "chip_seconds": round(t_chip, 2), "text_seconds": None}
+
+ # --- the queue ---------------------------------------------------------------------------
+ @torch.no_grad()
+ def run(self, jobs, budget: int, max_chip: int, done, check_head: bool = True) -> dict:
+ """Decode every job; done(job, result) as each finishes. Returns timing stats."""
+ queue = iter(jobs)
+ live: list = [None] * self.B # (generator, job, pending request)
+ hidden = self.chat.lm.embed_tokens.weight.shape[1]
+ H = torch.zeros(self.B, hidden, dtype=torch.bfloat16, device=self.dev)
+ stats = {"steps": 0, "row_steps": 0, "jobs": 0, "t_step": 0.0}
+
+ def advance(b, gen, job, send):
+ """Run row b until its next text-token request; refill it when it finishes."""
+ while True:
+ try:
+ req = gen.send(send) if send is not None else next(gen)
+ live[b] = (gen, job, req)
+ return
+ except StopIteration as fin:
+ done(job, fin.value)
+ stats["jobs"] += 1
+ nj = next(queue, None)
+ if nj is None:
+ live[b] = None
+ return
+ gen, job, send = self._row(b, nj, budget, max_chip, check_head), nj, None
+
+ for b in range(self.B):
+ j = next(queue, None)
+ if j is None:
+ break
+ advance(b, self._row(b, j, budget, max_chip, check_head), j, None)
+ while any(x is not None for x in live):
+ act = [b for b in range(self.B) if live[b] is not None]
+ for b in act:
+ hc, p = live[b][2]
+ H[b].copy_(hc)
+ self.pos[b] = p
+ # a dead row computes garbage in its own cache row (never read again)
+ ts = time.time()
+ res = self._qwen(H)
+ stats["t_step"] += time.time() - ts
+ stats["steps"] += 1
+ stats["row_steps"] += len(act)
+ for b in act:
+ gen, job, _ = live[b]
+ advance(b, gen, job, tuple(res[b]))
+ return stats