A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.
RISC-V model is now the main Qwenputer on HF (CHIP-8 moved to -CHIP8); sync experimental/riscv to rv-p3b c94709a (executor in generation, batching, exact /)
Divyansh Agrawalb25289baa30c
diff --git a/README.md b/README.mdindex 4be3ba7..62654af 100644--- a/README.md+++ b/README.md@@ -4,7 +4,7 @@> A computer, compiled into the weights of a language model.-[Model on Hugging Face](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer) · [The standalone CHIP-8 transformer it grew from](https://github.com/divagr18/chipformer) · [MIT License](LICENSE)+[RISC-V model](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer) · [CHIP-8 model](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer-CHIP8) · [The standalone CHIP-8 transformer it grew from](https://github.com/divagr18/chipformer) · [MIT License](LICENSE)Qwenputer is Qwen3.5-4B with a working computer compiled directly into its weights. It isn't aplugin, a tool or a sandbox. It's a CPU, a big-number maths library and its memory, written as@@ -114,10 +114,13 @@ then reads the printed answer like ordinary text.## Quickstart+This runs the CHIP-8 model, which the benchmarks above come from. The RISC-V model is+[further down](#a-risc-v-computer-inside-qwen).+```bashgit clone https://github.com/divagr18/qwenputer && cd qwenputeruv sync-hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-4b+hf download divagr1925/Qwen3.5-4B-Qwenputer-CHIP8 --local-dir qwenputer-4buv run python -m chipformer.q35.demo qwenputer-4b # Qwen uses its tool-call syntaxuv run python -m chipformer.q35.demo qwenputer-4b --inline --nudge # plain inline arithmetic@@ -158,17 +161,42 @@ details:- the optional decoder settings our benchmarks used (`fp` and `tc` in `BatchDecoder`), which makeinline mode stronger-## Experimental: a RISC-V computer inside Qwen+## A RISC-V computer inside Qwen++[`divagr1925/Qwen3.5-4B-Qwenputer`](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer)+compiles a 32-bit **RV32IM** CPU, with Zba and a bignum coprocessor, into Qwen3.5-4B. Its code+ROM, RAM, registers and program counter all live in Gated DeltaNet state, in 11 chip layers+before Qwen's.++The CPU passes the official riscv-tests suites (`rv32ui`, `rv32um`, `rv32uzba`), runs+gcc-compiled C programs exactly (about 18 million instructions verified) and runs a patched+MicroPython build exactly. Inside Qwen, it backs Qwen's `calc` tool: Qwen writes the call, the+RISC-V machine in the weights runs a C calculator firmware, and the model prints the+`<tool_response>` itself.++| DeepMind set (40 problems) | Qwen + RISC-V in the weights | Qwen + Python tool (host) |+|---|---|---|+| arithmetic (mixed) | 39 | 39 |+| `add_or_sub_big` | 39 | 39 |+| `add_sub_multiple_longer` | 40 | 40 |+| `div_big` | 38 | 39 |+| `mul_big` | 39 | 40 |-[`experimental/riscv/`](experimental/riscv) takes the idea further. It compiles a 32-bit-**RV32IM** CPU, with Zba and a bignum coprocessor, into the same Qwen3.5 layer types.+Replaying all the arithmetic runs and samples from every DeepMind set through the reference+machine found 0 mismatches. In generation the+machine runs on a Triton executor at about 3,200 trace tokens a second on an L4.++```bash+cd experimental/riscv && uv sync+hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-rv+uv run python -m chipformer.rv.chat qwenputer-rv --firmware tests/data/rv/calc-bridge.elf --ask "Use the calc tool: what is 123456789 * 987654321?"+```-It passes the official riscv-tests suites (`rv32ui`, `rv32um`, `rv32uzba`). It runs-gcc-compiled C programs exactly, about 18 million instructions verified so far, and it runs a-patched MicroPython build exactly. Inside Qwen3.5-4B it already answers end to end: Qwen asks,-the RISC-V machine in the weights computes, Qwen reads the answer. It is a work in progress;-the compact executor currently runs about 3,700 instructions a second per stream on an L4.-[`experimental/riscv/docs/rv32.md`](experimental/riscv/docs/rv32.md) has the details.+Unlike the CHIP-8 model, this one isn't driven by a plain `generate()` call yet. `RvChat` loads+the firmware image into the DeltaNet state after the prompt, and its decoding loop keeps the+machine's trace tokens out of Qwen's own layers. The weights alone decide every token.+[`experimental/riscv/docs/rv32.md`](experimental/riscv/docs/rv32.md) has the design and every+measurement.## Citingdiff --git a/docs/running.md b/docs/running.mdindex af7fd7b..2143f08 100644--- a/docs/running.md+++ b/docs/running.md@@ -108,7 +108,7 @@ end.## Release checks (A10G 24 GB, 2026-10-19)-**The checkpoint.** [divagr1925/Qwen3.5-4B-Qwenputer](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer).+**The checkpoint.** [divagr1925/Qwen3.5-4B-Qwenputer-CHIP8](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer-CHIP8).- Size: 14.65 GB in 7 shards.- Contents: config, tokenizer, `chat_template.jinja`, `generation_config.json`, `chipformer.json`,LICENSE, preprocessor configs.diff --git a/experimental/riscv/README.md b/experimental/riscv/README.mdindex af5e7e1..0a17455 100644--- a/experimental/riscv/README.md+++ b/experimental/riscv/README.md@@ -1,4 +1,6 @@-# Experimental: a RISC-V computer inside Qwen+# A RISC-V computer inside Qwen++Model: [divagr1925/Qwen3.5-4B-Qwenputer](https://huggingface.co/divagr1925/Qwen3.5-4B-Qwenputer).This folder compiles a 32-bit **RV32IM** CPU into stock Qwen3.5 layers, using the same compiler asthe main project. The CPU also has Zba and a bignum coprocessor. It is a self-contained snapshot:@@ -20,7 +22,7 @@ merged with the main tree yet.The machine has 256 KB of code ROM, 128 KB of read-only data and 32 KB of RAM, all held in GatedDeltaNet state. It takes 11 chip layers and about 1M nonzero weights, and costs about 1.2-1.4tokens per instruction. A compact executor runs about 3,700 instructions a second per stream on-an L4.+an L4, and in generation it runs at about 3,200 trace tokens a second.[`docs/rv32.md`](docs/rv32.md) has the design and every measurement, and[`docs/rv32-p3-status.md`](docs/rv32-p3-status.md) has the in-Qwen status.diff --git a/experimental/riscv/chipformer/rv/batchcheck.py b/experimental/riscv/chipformer/rv/batchcheck.pynew file mode 100644index 0000000..1a9f20f--- /dev/null+++ b/experimental/riscv/chipformer/rv/batchcheck.py@@ -0,0 +1,79 @@+"""Batched rv1 decoding (rvbatch) against batch-1 records: the same prompts, decoded N at a time,+must give each record's tokens.++ python -m chipformer.rv.batchcheck MODEL REC.jsonl:ID [...] --batch 4 [--batch 8 ...]+"""++import argparse+import json+import time++from chipformer.rv import bridge as BR+from chipformer.rv import elf, ref+from chipformer.rv.chat import RvChat+from chipformer.rv.rvbatch import Job, RvBatch+++def main() -> None:+ ap = argparse.ArgumentParser()+ ap.add_argument("model")+ ap.add_argument("records", nargs="+")+ ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")+ ap.add_argument("--batch", type=int, action="append", required=True)+ ap.add_argument("--budget", type=int, default=4096)+ ap.add_argument("--max-chip", type=int, default=200000)+ ap.add_argument("--max-len", type=int, default=6144)+ ap.add_argument("--local-ref", action="store_true",+ help="compare with RvChat.generate here instead of the records' tokens")+ args = ap.parse_args()+ recs = []+ for spec in args.records:+ path, rid = spec.rsplit(":", 1)+ rows = [json.loads(x) for x in open(path, encoding="utf-8")]+ recs.append(next(r for r in rows if r["id"] == rid))+ chat = RvChat(args.model, executor=True)++ def job(r):+ m = ref.Machine.from_program(elf.program(args.firmware))+ BR.park(m)+ m2 = ref.Machine.from_program(elf.program(args.firmware))+ BR.park(m2)+ lock = BR.Session(chat.T, m2)+ lock.history(r["prompt_ids"])+ return Job(r["id"], r["prompt_ids"], m, lock, {"r": r})++ if args.local_ref: # the reference: RvChat.generate (batch 1) on this GPU+ for r in recs:+ j = job(r)+ t0 = time.time()+ out = chat.generate(j.ids, j.machine, args.budget, max_chip=args.max_chip,+ check_head=True, lock=j.lock)+ same = out["tokens"] == r["gen_ids"]+ r["gen_ids"] = out["tokens"]+ print(json.dumps({"batch": "generate", "id": r["id"], "same_as_record": same,+ "len": len(out["tokens"]), "seconds": round(time.time() - t0, 1),+ "lock_error": out["lock_error"]}), flush=True)+ for B in args.batch:+ rb = RvBatch(chat, B, args.max_len)+ res = {}+ t0 = time.time()+ st = rb.run((job(r) for r in recs), args.budget, args.max_chip,+ lambda j, out: res.__setitem__(j.key, out))+ wall = time.time() - t0+ for r in recs:+ got, want = res[r["id"]]["tokens"], r["gen_ids"]+ first = next((i for i, (a, b) in enumerate(zip(got, want)) if a != b),+ None if len(got) == len(want) else min(len(got), len(want)))+ print(json.dumps({"batch": B, "id": r["id"], "same_tokens": got == want,+ "first_diff": first, "len": len(got), "record_len": len(want),+ "lock_error": res[r["id"]]["lock_error"],+ "head_mismatch": res[r["id"]]["head_mismatch"],+ "diff_ctx": chat.tok.decode([t for t in want[max(0, (first or 0) - 8):(first or 0) + 4] if t < chat.v0]) if first is not None else ""}),+ flush=True)+ print(json.dumps({"batch": B, "wall_s": round(wall, 1), "stats": st,+ "problems_per_hour": round(len(recs) / wall * 3600, 1)}), flush=True)+ del rb+++if __name__ == "__main__":+ main()diff --git a/experimental/riscv/chipformer/rv/chat.py b/experimental/riscv/chipformer/rv/chat.pyindex fa69e7e..05b6717 100644--- a/experimental/riscv/chipformer/rv/chat.py+++ b/experimental/riscv/chipformer/rv/chat.py@@ -38,8 +38,67 @@ CALC_TOOL = {"type": "function", "function": {"required": ["expression"]}}}+class ExecRow:+ """One conversation's circuit on the Triton executor (batch 1) with its own CUDA graph."""++ def __init__(self, chat: "RvChat") -> None:+ from chipformer.rv.tritonexec import Executor2++ self.chat = chat+ self.ex = Executor2(chat.c, 1, chat.dev)+ self.heads = [st for st in self.ex.steps if st["kind"] == "head"]+ self.xbuf = torch.zeros(1, self.ex.D, device=chat.dev)+ self.kbuf = torch.zeros(len(chat.keep_cols), device=chat.dev)+ self.graph = None++ def _body(self):+ self.ex.step(self.xbuf)+ self.kbuf.copy_(self.xbuf[0, self.chat.keep_ir])++ @torch.no_grad()+ def _capture(self):+ """Warm-up runs change the state, so it is saved and put back around them."""+ saved = [(st["S"].clone(), st["hist"].clone()) for st in self.heads]+ s = torch.cuda.Stream()+ s.wait_stream(torch.cuda.current_stream())+ with torch.cuda.stream(s):+ for _ in range(3):+ self._body()+ torch.cuda.current_stream().wait_stream(s)+ self.graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(self.graph):+ self._body()+ for st, (S_, h) in zip(self.heads, saved):+ st["S"].copy_(S_)+ st["hist"].copy_(h)++ @torch.no_grad()+ def start(self, ids: list[int]) -> None:+ """As the HF path does it: the prompt through the circuit from zero states, then every+ head's state set to its init (the snapshot now in the circuit; hf.inject), the lag+ windows kept."""+ if self.graph is None:+ self._capture()+ for st in self.heads:+ st["S"].zero_()+ st["hist"].zero_()+ emb = self.chat._exec_emb(ids)+ for t in range(len(ids)):+ self.xbuf.copy_(emb[t:t + 1])+ self.graph.replay()+ self.ex.load_init()++ @torch.no_grad()+ def step(self, ids: int) -> torch.Tensor:+ """One token through the circuit; returns the kept values (fp32, the code first)."""+ self.xbuf.copy_(self.chat._exec_emb([ids]))+ self.graph.replay()+ return self.kbuf++class RvChat:- def __init__(self, path, device: str | None = None, fast: bool = True) -> None:+ def __init__(self, path, device: str | None = None, fast: bool = True,+ executor: bool = False) -> None:from transformers import AutoTokenizer, Qwen3_5ForCausalLMfrom chipformer.q35.fast import exact_matmuls@@ -61,8 +120,68 @@ class RvChat:self.code_dims = torch.tensor(self.meta["code_dims"], device=self.dev)self.eos = {248046, 248044}self.fast = fast and self.dev == "cuda"- if self.fast:+ # executor "verify": both chip paths, the HF view's hc checked against the executor's+ self.executor = executor if self.dev == "cuda" else False+ self.verify = self.executor == "verify"+ if self.executor:+ self._build_exec()+ if self.fast and (not self.executor or self.verify):self._build_fast()+ self.fast = self.fast and (not self.executor or self.verify)++ # --- the executor path -----------------------------------------------------------------+ def _build_exec(self) -> None:+ """The chip layers as the circuit itself (tritonexec.Executor2: the IR's arithmetic in+ fp32, one CUDA graph per token) instead of their HF modules. After the chip layers the+ merged residual holds Qwen's embedding dims, the kept dims (output code, tau, mu) and+ zeros (the exit clears every other chip dim and the ballast), so a text token's hc is+ built from the embedding row and the executor's kept values; chip tokens need only the+ code."""+ self.keep_ir = torch.tensor(self.c.keep, device=self.dev)+ cols = self.meta["code_dims"] + [self.meta["tau_dim"], self.meta["mu_dim"]]+ assert len(cols) == len(self.c.keep)+ self.keep_cols = torch.tensor(cols, device=self.dev)+ self.d0 = self.meta["chip_dims_offset"]+ self.emb_cache: dict[int, torch.Tensor] = {}+ self.erow = ExecRow(self)++ def _exec_emb(self, ids: list[int]) -> torch.Tensor:+ """Executor input rows for model ids (cached per id; shared by every ExecRow)."""+ new = sorted({i for i in ids if i not in self.emb_cache})+ if new:+ rows = self.erow.ex.embed([i - self.v0 if i >= self.v0 else BR.text_token(i)+ for i in new])+ for i, r in zip(new, rows):+ self.emb_cache[i] = r+ return torch.stack([self.emb_cache[i] for i in ids])++ def _exec_start(self, ids: list[int]) -> None:+ self.erow.start(ids)++ def _exec_step(self, ids: int) -> torch.Tensor:+ return self.erow.step(ids)++ def _exec_hc(self, ids: int, keep: torch.Tensor) -> torch.Tensor:+ """A text token's residual after the chip layers."""+ hc = torch.zeros(self.lm.embed_tokens.weight.shape[1], dtype=torch.bfloat16,+ device=self.dev)+ hc[: self.d0] = self.lm.embed_tokens.weight[ids, : self.d0]+ hc[self.keep_cols] = keep.to(torch.bfloat16)+ return hc++ def code_token(self, code) -> int | None:+ """circuit_token from the output code values."""+ code = code.float()+ if not bool((code != 0).any()):+ return None+ bits = (code > 0).long().tolist()+ nf = core.NFAM+ fid = sum(b << i for i, b in enumerate(bits[:nf]))+ p = sum(b << i for i, b in enumerate(bits[nf:]))+ t = core.decode_code(fid, p, self.T)+ if t < 0:+ raise RuntimeError(f"the circuit predicts no token (family {fid}, payload {p})")+ return t - S.TEXT_TOKEN if S.is_text(t) else self.v0 + t# --- the fast chip path ----------------------------------------------------------------def _build_fast(self) -> None:@@ -190,17 +309,7 @@ class RvChat:def circuit_token(self, hc) -> int | None:"""The model id the circuit predicts at a position, None where Qwen predicts."""- code = hc[self.code_dims].float()- if not bool((code != 0).any()):- return None- bits = (code > 0).long().tolist()- nf = core.NFAM- fid = sum(b << i for i, b in enumerate(bits[:nf]))- p = sum(b << i for i, b in enumerate(bits[nf:]))- t = core.decode_code(fid, p, self.T)- if t < 0:- raise RuntimeError(f"the circuit predicts no token (family {fid}, payload {p})")- return t - S.TEXT_TOKEN if S.is_text(t) else self.v0 + t+ return self.code_token(hc[self.code_dims])# --- a conversation --------------------------------------------------------------------def prompt_ids(self, messages, tools=(CALC_TOOL,)) -> list[int]:@@ -222,6 +331,8 @@ class RvChat:out = self.model(input_ids=x, use_cache=True)cache = out.past_key_values # (fast: Qwen's layers only)core.snapshot(self.c, machine)+ if self.executor:+ self._exec_start(ids)if self.fast:ccache = self._chip_cache()h = self.cm.embed_tokens(x) # the prompt through the chip layers only@@ -232,12 +343,12 @@ class RvChat:if self.graph is None:self._capture()g = len(ids) # the chip view's cache position (every token)- else:+ elif not self.executor:hf.inject(self.c, cache)+ n_verify_bad = 0logits = out.logits[0, -1]pos = len(ids) # the next text token's RoPE positiongen, n_text, n_chip_tok = [], 0, 0- stream_hc = []# the prompt's last position: Qwen predicts (nothing captured yet)nxt = int(torch.argmax(logits[: self.v0]))t0 = time.time()@@ -273,25 +384,34 @@ class RvChat:print(f" {len(gen)} tokens ({n_text} text, {n_chip_tok} chip) "f"{time.time() - t0:.0f}s: {' '.join(tail)}", flush=True)ts = time.time()- if nxt >= self.v0: # a chip token+ chip_in = nxt >= self.v0+ hc = keep = None+ if self.executor:+ keep = self._exec_step(nxt)+ if not chip_in:+ hc = self._exec_hc(nxt, keep)+ if self.fast and (self.verify or not self.executor):+ hf_hc = self._chip_step(nxt, g, ccache)+ g += 1+ if self.verify:+ same = (bool((hf_hc[self.keep_cols] == keep.to(torch.bfloat16)).all())+ if chip_in else bool((hf_hc == hc).all()))+ if not same:+ n_verify_bad += 1+ hc = hf_hc+ elif not self.executor:+ hc, logits = self._layers(nxt, cache, None if chip_in else pos)+ if chip_in: # a chip tokenn_chip_tok += 1- if self.fast:- hc = self._chip_step(nxt, g, ccache)- g += 1- else:- hc, _ = self._layers(nxt, cache, None)- nxt = self.circuit_token(hc)+ nxt = (self.code_token(keep[: len(self.code_dims)]) if hc is None+ else self.circuit_token(hc))t_chip += time.time() - tsif nxt is None:raise RuntimeError("the circuit gave up a chip position")else:n_text += 1- if self.fast:- hc = self._chip_step(nxt, g, ccache)- g += 1+ if self.fast or self.executor:logits = self._qwen_step(hc, cache, pos)- else:- hc, logits = self._layers(nxt, cache, pos)pos += 1ct = self.circuit_token(hc)q = int(torch.argmax(logits))@@ -299,11 +419,10 @@ class RvChat:mismatch += 1nxt = ct if ct is not None else int(torch.argmax(logits[: self.v0]))t_text += time.time() - ts- if check_head:- stream_hc.append(hc[self.code_dims].float().cpu().numpy())dt = time.time() - t0return {"tokens": gen, "text_tokens": n_text, "chip_tokens": n_chip_tok,"seconds": dt, "head_mismatch": mismatch, "lock_error": lock_error,+ "verify_mismatch": n_verify_bad if self.verify else None,"chip_seconds": round(t_chip, 2), "text_seconds": round(t_text, 2)}def to_stream(self, ids: list[int]) -> list[int]:diff --git a/experimental/riscv/chipformer/rv/evalarith.py b/experimental/riscv/chipformer/rv/evalarith.pyindex 34f2190..ca8c417 100644--- a/experimental/riscv/chipformer/rv/evalarith.py+++ b/experimental/riscv/chipformer/rv/evalarith.py@@ -85,6 +85,12 @@ def main() -> None:ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")ap.add_argument("--budget", type=int, default=4096, help="Qwen tokens per problem")ap.add_argument("--max-chip", type=int, default=200000)+ ap.add_argument("--chip-path", default="executor", choices=("executor", "hf"),+ help="rv1's chip layers: the circuit's Triton executor, or their HF modules")+ ap.add_argument("--batch", type=int, default=1, help="rv1: conversations decoded at once")+ ap.add_argument("--max-len", type=int, default=6144, help="rv1 --batch: Qwen KV slots")+ ap.add_argument("--shard", default=None, help="i/k: every k-th problem from the i-th")+ ap.add_argument("--ids", default=None, help="comma-separated problem ids (a subset)")args = ap.parse_args()n = args.n or (40 if args.data == "arith" else None)probs = problems(args.data, n, args.seed)@@ -93,7 +99,7 @@ def main() -> None:done = {json.loads(x)["id"] for x in out.read_text().splitlines()} if out.exists() else set()tool = args.cond != "base_nudge"if args.cond == "rv1":- chat = RvChat(args.model)+ chat = RvChat(args.model, executor=args.chip_path == "executor")tok = chat.tokelse:from transformers import AutoModelForImageTextToText, AutoTokenizer@@ -101,46 +107,132 @@ def main() -> None:tok = AutoTokenizer.from_pretrained(args.model)model = AutoModelForImageTextToText.from_pretrained(args.model, dtype=torch.bfloat16)model = model.to("cuda").eval()- for p in probs:- if p["id"] in done:- continue- msgs = [{"role": "system", "content": NUDGE if tool else NUDGE_NONE},+ if args.shard:+ i, k = map(int, args.shard.split("/"))+ probs = probs[i::k]+ if args.ids:+ want = set(args.ids.split(","))+ probs = [p for p in probs if p["id"] in want]+ todo = [p for p in probs if p["id"] not in done]+ t_all = time.time()++ def msgs_of(p):+ return [{"role": "system", "content": NUDGE if tool else NUDGE_NONE},{"role": "user", "content": p["question"] + E.SUFFIX}]- t0 = time.time()++ def rv1_job(p):+ ids = chat.prompt_ids(msgs_of(p))+ m = ref.Machine.from_program(elf.program(args.firmware))+ BR.park(m)+ m2 = ref.Machine.from_program(elf.program(args.firmware))+ BR.park(m2)+ lock = BR.Session(chat.T, m2) # every circuit token checked against the reference+ lock.history(ids)+ return ids, m, lock++ def rv1_fields(ids, lock, r):+ gen = r["tokens"]+ outs = [chat.T.decode_bytes(e).decode("utf-8", "replace") for e in lock.runs]+ return chat.text_of(gen), dict(+ prompt_ids=ids, gen_ids=gen, calls=len(lock.runs),+ n_qwen=r["text_tokens"] - sum(len(e) for e in lock.runs),+ n_chip=r["chip_tokens"], lock_error=r["lock_error"],+ head_mismatch=r["head_mismatch"], gen_seconds=round(r["seconds"], 1),+ chip_path=args.chip_path, chip_seconds=r["chip_seconds"],+ text_seconds=r["text_seconds"], batch=args.batch,+ results=[o.split("\n")[2] if o.count("\n") > 2 else o for o in outs])++ def emit(p, text, fields, seconds):rec = {"id": p["id"], "gold": p["gold"], "question": p["question"], "cond": args.cond}+ rec.update(fields)+ rec.update(AB.exact_fields(p, text))+ rec.update(seconds=round(seconds, 1), text=text)+ with out.open("a", encoding="utf-8") as f:+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")+ keys = ("id", "gold", "pred_exact", "exact", "calls", "n_qwen", "n_chip", "seconds",+ "lock_error", "head_mismatch")+ print(json.dumps({k: rec[k] for k in keys if k in rec}), flush=True)++ if args.cond == "rv1" and args.batch > 1:+ from chipformer.rv.rvbatch import Job, RvBatch++ rb = RvBatch(chat, args.batch, args.max_len)++ def jobs():+ for p in todo:+ ids, m, lock = rv1_job(p)+ yield Job(p["id"], ids, m, lock, {"p": p})++ def finished(job, r):+ text, fields = rv1_fields(job.ids, job.lock, r)+ emit(job.extra["p"], text, fields, r["seconds"])++ st = rb.run(jobs(), args.budget, args.max_chip, finished)+ print(json.dumps({"batch_stats": st}), flush=True)+ if args.cond != "rv1" and args.batch > 1: # base Qwen: q35's BatchDecoder+ from chipformer.q35.batch import BatchDecoder++ prompts = []+ for p in todo:+ ids = tok.apply_chat_template(msgs_of(p), tools=[CALC_TOOL] if tool else None,+ add_generation_prompt=True, enable_thinking=False,+ tokenize=True)+ prompts.append(ids if isinstance(ids, list) else ids["input_ids"])+ tcc = tok.convert_tokens_to_ids("</tool_call>")+ state = [{"forced": [], "own": 0, "gen": [], "calls": 0} for _ in todo]++ def hook(i, t):+ """pytool: after </tool_call> the next tokens are the host's response (forced);+ the row stops after ``budget`` of Qwen's own tokens, as run_base does."""+ s = state[i]+ if s["forced"]:+ t = s["forced"].pop(0)+ s["gen"].append(t)+ return t, False+ s["own"] += 1+ s["gen"].append(t)+ if tool and t == tcc:+ txt = tok.decode(s["gen"], skip_special_tokens=False)+ mm = re.findall(r"<parameter=[^>]*>(.*?)</parameter>",+ txt.split("<tool_call>")[-1], re.S)+ s["forced"] = tok(calc_response(mm[0] if mm else ""),+ add_special_tokens=False).input_ids+ s["calls"] += 1+ return t, s["own"] >= args.budget and not s["forced"]++ t0 = time.time()+ max_len = -(-(max(map(len, prompts)) + 2 * args.budget) // 1024) * 1024+ bd = BatchDecoder(model, batch=args.batch, max_len=max_len)++ def fin(i, toks):+ s = state[i]+ text = tok.decode(toks, skip_special_tokens=False)+ emit(todo[i], text, dict(calls=s["calls"], n_qwen=s["own"], n_chip=0,+ batch=args.batch), time.time() - t0)++ if prompts:+ bd.run(prompts, 2 * args.budget, on_done=fin, hook=hook)+ for p in todo if args.batch == 1 else []:+ t0 = time.time()if args.cond == "rv1":- ids = chat.prompt_ids(msgs)- m = ref.Machine.from_program(elf.program(args.firmware))- BR.park(m)- m2 = ref.Machine.from_program(elf.program(args.firmware))- BR.park(m2)- lock = BR.Session(chat.T, m2) # every circuit token checked against the reference- lock.history(ids)+ ids, m, lock = rv1_job(p)r = chat.generate(ids, m, args.budget, max_chip=args.max_chip, check_head=True,lock=lock)- gen = r["tokens"]- text = chat.text_of(gen)- outs = [chat.T.decode_bytes(e).decode("utf-8", "replace") for e in lock.runs]- rec.update(prompt_ids=ids, gen_ids=gen, calls=len(lock.runs),- n_qwen=r["text_tokens"] - sum(len(e) for e in lock.runs),- n_chip=r["chip_tokens"], lock_error=r["lock_error"],- head_mismatch=r["head_mismatch"], gen_seconds=round(r["seconds"], 1),- results=[o.split("\n")[2] if o.count("\n") > 2 else o for o in outs])+ text, fields = rv1_fields(ids, lock, r)else:- ids = tok.apply_chat_template(msgs, tools=[CALC_TOOL] if tool else None,+ ids = tok.apply_chat_template(msgs_of(p), tools=[CALC_TOOL] if tool else None,add_generation_prompt=True, enable_thinking=False,tokenize=True)ids = ids if isinstance(ids, list) else ids["input_ids"]gen, calls, own = run_base(model, tok, ids, tool, args.budget)text = tok.decode(gen, skip_special_tokens=False)- rec.update(calls=calls, n_qwen=own, n_chip=0)- rec.update(AB.exact_fields(p, text))- rec.update(seconds=round(time.time() - t0, 1), text=text)- with out.open("a", encoding="utf-8") as f:- f.write(json.dumps(rec, ensure_ascii=False) + "\n")- keys = ("id", "gold", "pred_exact", "exact", "calls", "n_qwen", "n_chip", "seconds",- "lock_error", "head_mismatch")- print(json.dumps({k: rec[k] for k in keys if k in rec}), flush=True)+ fields = dict(calls=calls, n_qwen=own, n_chip=0)+ emit(p, text, fields, time.time() - t0)+ wall = time.time() - t_all+ if todo:+ print(json.dumps({"problems": len(todo), "wall_s": round(wall, 1),+ "problems_per_hour": round(len(todo) / wall * 3600, 1),+ "batch": args.batch}), flush=True)rows = [json.loads(x) for x in out.read_text().splitlines()]print(f"{out.name}: {sum(r['exact'] for r in rows)}/{len(rows)} exact")diff --git a/experimental/riscv/chipformer/rv/execcheck.py b/experimental/riscv/chipformer/rv/execcheck.pynew file mode 100644index 0000000..4b1d8a9--- /dev/null+++ b/experimental/riscv/chipformer/rv/execcheck.py@@ -0,0 +1,67 @@+"""The executor path of RvChat against the HF chip path, on rv1 eval records.++ python -m chipformer.rv.execcheck MODEL REC.jsonl:ID [REC.jsonl:ID ...] [--no-verify]++Per record: (1) "verify" mode: the HF chip view drives generation (it must reproduce the+record's tokens) and the executor runs beside it, its residual after the chip layers compared+bitwise at every position (the whole hc on text tokens, the kept dims on chip tokens); (2) the+executor alone: its tokens must equal the record's. Prints both timings.+"""++import argparse+import json++from chipformer.rv import bridge as BR+from chipformer.rv import elf, ref+from chipformer.rv.chat import RvChat+++def run(chat, r, firmware, budget, max_chip):+ m = ref.Machine.from_program(elf.program(firmware))+ BR.park(m)+ m2 = ref.Machine.from_program(elf.program(firmware))+ BR.park(m2)+ lock = BR.Session(chat.T, m2)+ lock.history(r["prompt_ids"])+ return chat.generate(r["prompt_ids"], m, budget, max_chip=max_chip, check_head=True, lock=lock)+++def main() -> None:+ ap = argparse.ArgumentParser()+ ap.add_argument("model")+ ap.add_argument("records", nargs="+")+ ap.add_argument("--firmware", default="tests/data/rv/calc-bridge.elf")+ ap.add_argument("--budget", type=int, default=4096)+ ap.add_argument("--max-chip", type=int, default=200000)+ ap.add_argument("--no-verify", action="store_true")+ args = ap.parse_args()+ recs = []+ for spec in args.records:+ path, rid = spec.rsplit(":", 1)+ rows = [json.loads(x) for x in open(path, encoding="utf-8")]+ recs.append(next(r for r in rows if r["id"] == rid))+ chat = RvChat(args.model, executor="verify")+ for r in recs:+ out = {"id": r["id"], "record_tokens": len(r["gen_ids"])}+ if not args.no_verify:+ chat.verify, chat.fast = True, True+ v = run(chat, r, args.firmware, args.budget, args.max_chip)+ out["hf"] = {"same_tokens": v["tokens"] == r["gen_ids"],+ "verify_mismatch": v["verify_mismatch"], "lock_error": v["lock_error"],+ "chip_tokens": v["chip_tokens"], "chip_s": v["chip_seconds"],+ "text_s": v["text_seconds"]}+ chat.verify, chat.fast = False, False+ e = run(chat, r, args.firmware, args.budget, args.max_chip)+ first = next((i for i, (a, b) in enumerate(zip(e["tokens"], r["gen_ids"])) if a != b),+ None if len(e["tokens"]) == len(r["gen_ids"]) else+ min(len(e["tokens"]), len(r["gen_ids"])))+ out["exec"] = {"same_tokens": e["tokens"] == r["gen_ids"], "first_diff": first,+ "lock_error": e["lock_error"], "head_mismatch": e["head_mismatch"],+ "chip_tokens": e["chip_tokens"], "chip_s": e["chip_seconds"],+ "text_s": e["text_seconds"], "seconds": round(e["seconds"], 1),+ "chip_tok_per_s": round(e["chip_tokens"] / max(e["chip_seconds"], 1e-9))}+ print(json.dumps(out), flush=True)+++if __name__ == "__main__":+ main()diff --git a/experimental/riscv/chipformer/rv/replay.py b/experimental/riscv/chipformer/rv/replay.pyindex d7d9b65..9fb70b2 100644--- a/experimental/riscv/chipformer/rv/replay.py+++ b/experimental/riscv/chipformer/rv/replay.py@@ -47,12 +47,12 @@ def replay_ids(c, T, v0, prompt, gen, firmware) -> dict:m2 = ref.Machine.from_program(elf.program(firmware))BR.park(m2)core.snapshot(c, m2)- carry, mats = {}, []+ carry, preds, owns = {}, [], [] # per chunk: a long stream's whole matrix is GBsfor a in range(0, len(stream), 2048):- mats.append(ir.evaluate(c, stream[a:a + 2048], carry=carry))- mat = np.concatenate(mats)- pred = core.predictions(c, mat)- own = np.rint(ir.ev(c.owner, mat)).astype(int)+ mat = ir.evaluate(c, stream[a:a + 2048], carry=carry)+ preds.append(core.predictions(c, mat))+ owns.append(np.rint(ir.ev(c.owner, mat)).astype(int))+ pred, own = np.concatenate(preds), np.concatenate(owns)bad_own = int(sum(own[x] != int(ses.own[x]) for x in range(len(stream))))bad = int(sum(1 for x in range(len(stream) - 1)if ses.own[x] and x not in skip and pred[x] != stream[x + 1]))diff --git a/experimental/riscv/chipformer/rv/rvbatch.py b/experimental/riscv/chipformer/rv/rvbatch.pynew file mode 100644index 0000000..eccb3fb--- /dev/null+++ b/experimental/riscv/chipformer/rv/rvbatch.py@@ -0,0 +1,191 @@+"""Batched rv1 decoding: N conversations at once on one GPU.++ rb = RvBatch(chat, batch=8, max_len=8192) # chat: RvChat(path, executor=True)+ rb.run(jobs, budget, max_chip, done) # jobs: iterable of Job; done(job, result)++Each conversation (a row) keeps its own circuit (chat.ExecRow: the Triton executor at batch 1+with its own CUDA graph) and runs its chip bursts there on its own, as RvChat.generate does. Its+text tokens go through Qwen's layers in one batched step for all live rows, on a cache with+per-row positions (q35.batch._Rows: full attention writes row b's key at its own slot and the+mask admits slots <= pos[b]; DeltaNet rows are independent). A row's prompt is prefilled alone+(batch 1, as generate does) and copied into its row. A finished row is refilled from the queue.++Each row is a generator (``_row``) with RvChat.generate's executor-path logic: it yields the+residual hc of each text token after the chip layers and receives Qwen's two argmaxes (over the+whole vocabulary, for the head check, and over Qwen's own vocabulary). Only Qwen's GEMMs see the+batch (M = live rows instead of 1), which can change their bf16 rounding; the circuit is exact+either way.+"""++import time+from dataclasses import dataclass, field++import torch++from chipformer.rv import bridge as BR+from chipformer.rv import core+from chipformer.rv import spec as S+++@dataclass+class Job:+ key: object+ ids: list[int]+ machine: object # a parked ref.Machine+ lock: object = None # a BR.Session on a copy (history = the prompt), or None+ extra: dict = field(default_factory=dict)+++class RvBatch:+ def __init__(self, chat, batch: int = 8, max_len: int = 8192) -> None:+ from chipformer.rv.chat import ExecRow++ assert chat.executor and not chat.verify+ self.chat, self.B, self.max_len = chat, batch, max_len+ self.dev = chat.dev+ self.rows = [chat.erow] + [ExecRow(chat) for _ in range(batch - 1)]+ self.pos = torch.zeros(batch, dtype=torch.long, device=self.dev)+ self.slots = torch.arange(max_len, device=self.dev)+ self.cache = None # _Rows, built from the first prefill's cache (its shapes)+ cfg = chat.model.config+ self.kinds = list(cfg.layer_types)++ # --- Qwen's batched step -----------------------------------------------------------------+ @torch.no_grad()+ def _qwen(self, H: torch.Tensor):+ """Qwen's layers on one text token per row (residuals H [B, hidden] after the chip+ layers, RoPE positions / KV slots self.pos); returns the argmaxes (all, Qwen's)."""+ chat = self.chat+ h = H[:, None, :]+ p4 = self.pos.view(1, -1, 1).expand(4, -1, 1)+ pe = chat.lm.rotary_emb(h, p4[1:])+ mask = (self.slots[None, :] <= self.pos[:, None])[:, None, None, :]+ for i in range(chat.n_chip, len(chat.lm.layers)):+ full = self.kinds[i] == "full_attention"+ h = chat.lm.layers[i](h, position_embeddings=pe,+ attention_mask=mask if full else None,+ position_ids=p4[0], past_key_values=self.cache.cache)+ logits = chat.model.lm_head(chat.lm.norm(h))[:, -1]+ both = torch.stack([logits.argmax(-1), logits[:, : chat.v0].argmax(-1)], 1)+ return both.tolist()++ @torch.no_grad()+ def _prefill(self, b: int, ids: list[int]) -> int:+ from chipformer.q35.batch import _Rows++ chat = self.chat+ out = chat.model(input_ids=torch.tensor([ids], device=self.dev), use_cache=True)+ small = out.past_key_values+ if self.cache is None:+ self.cache = _Rows(chat.model.config, small, self.B, self.max_len, self.pos)+ self.cache.load_row(b, small, len(ids))+ return int(torch.argmax(out.logits[0, -1][: chat.v0]))++ # --- one conversation --------------------------------------------------------------------+ def _row(self, b: int, job: Job, budget: int, max_chip: int, check_head: bool):+ """RvChat.generate's executor path as a generator: yields (hc, pos) for each text token,+ receives (argmax, Qwen-vocabulary argmax); returns generate's result dict."""+ chat, erow, lock = self.chat, self.rows[b], job.lock+ ids = job.ids+ nxt = self._prefill(b, ids)+ core.snapshot(chat.c, job.machine)+ erow.start(ids)+ pos = len(ids)+ gen, n_text, n_chip_tok = [], 0, 0+ t0 = time.time()+ mismatch = 0+ t_chip = 0.0+ expect, lock_error = [], None+ while n_text < budget and n_chip_tok < max_chip:+ gen.append(nxt)+ if lock is not None:+ st = nxt - chat.v0 if nxt >= chat.v0 else BR.text_token(nxt)+ if expect:+ e = expect.pop(0)+ if st != e:+ lock_error = {"at": len(gen) - 1, "got": S.name(st) if st < S.VOCAB+ else chat.tok.decode([nxt]),+ "want": S.name(e) if e < S.VOCAB+ else chat.tok.decode([e - S.TEXT_TOKEN])}+ break+ elif st < S.VOCAB:+ lock_error = {"at": len(gen) - 1, "got": S.name(st), "want": "text"}+ break+ else:+ k = len(lock.toks)+ lock.qwen([nxt])+ expect = list(lock.toks[k + 1:])+ if nxt in chat.eos:+ break+ ts = time.time()+ keep = erow.step(nxt)+ if nxt >= chat.v0: # a chip token+ n_chip_tok += 1+ nxt = chat.code_token(keep[: len(chat.code_dims)])+ t_chip += time.time() - ts+ if nxt is None:+ raise RuntimeError("the circuit gave up a chip position")+ else:+ n_text += 1+ if pos >= self.max_len:+ lock_error = {"at": len(gen) - 1, "got": "no KV slot left", "want": "text"}+ break+ hc = chat._exec_hc(nxt, keep)+ ct = chat.circuit_token(hc)+ q, qv = yield hc, pos+ pos += 1+ if ct is not None and check_head and q != ct:+ mismatch += 1+ nxt = ct if ct is not None else qv+ return {"tokens": gen, "text_tokens": n_text, "chip_tokens": n_chip_tok,+ "seconds": time.time() - t0, "head_mismatch": mismatch,+ "lock_error": lock_error, "verify_mismatch": None,+ "chip_seconds": round(t_chip, 2), "text_seconds": None}++ # --- the queue ---------------------------------------------------------------------------+ @torch.no_grad()+ def run(self, jobs, budget: int, max_chip: int, done, check_head: bool = True) -> dict:+ """Decode every job; done(job, result) as each finishes. Returns timing stats."""+ queue = iter(jobs)+ live: list = [None] * self.B # (generator, job, pending request)+ hidden = self.chat.lm.embed_tokens.weight.shape[1]+ H = torch.zeros(self.B, hidden, dtype=torch.bfloat16, device=self.dev)+ stats = {"steps": 0, "row_steps": 0, "jobs": 0, "t_step": 0.0}++ def advance(b, gen, job, send):+ """Run row b until its next text-token request; refill it when it finishes."""+ while True:+ try:+ req = gen.send(send) if send is not None else next(gen)+ live[b] = (gen, job, req)+ return+ except StopIteration as fin:+ done(job, fin.value)+ stats["jobs"] += 1+ nj = next(queue, None)+ if nj is None:+ live[b] = None+ return+ gen, job, send = self._row(b, nj, budget, max_chip, check_head), nj, None++ for b in range(self.B):+ j = next(queue, None)+ if j is None:+ break+ advance(b, self._row(b, j, budget, max_chip, check_head), j, None)+ while any(x is not None for x in live):+ act = [b for b in range(self.B) if live[b] is not None]+ for b in act:+ hc, p = live[b][2]+ H[b].copy_(hc)+ self.pos[b] = p+ # a dead row computes garbage in its own cache row (never read again)+ ts = time.time()+ res = self._qwen(H)+ stats["t_step"] += time.time() - ts+ stats["steps"] += 1+ stats["row_steps"] += len(act)+ for b in act:+ gen, job, _ = live[b]+ advance(b, gen, job, tuple(res[b]))+ return stats