Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

bench(hub-pov): runnable harness for the Hub v2 proof of value + smoke run (#1722)

xerj-org0143feac6276
diff --git a/benchmarks/hub-pov/SMOKE.md b/benchmarks/hub-pov/SMOKE.md
new file mode 100644
index 000000000..8af91b121
--- /dev/null
+++ b/benchmarks/hub-pov/SMOKE.md
@@ -0,0 +1,213 @@
+# Hub v2 proof of value: harness smoke run (2026-10-17)
+
+> **These smoke numbers are NOT results.** They come from 1 repeat, 2 tasks (one of
+> them a smoke fixture rather than a pre-registered task), a cheap model, a heavily
+> loaded machine and a locally signed catalogue. They exist only to show that the
+> harness runs and measures what [PLAN.md](./PLAN.md) says it measures. Do not quote
+> them, compare arms with them, or publish them. The §7 gate was not evaluated
+> (`pov.py report` refuses with fewer than 3 repeats).
+
+## What ran
+
+| | |
+|---|---|
+| Matrix | [`matrix/smoke.json`](./matrix/smoke.json): every arm A0–A7 × 2 tasks × 1 repeat = 16 cells, run serially |
+| Tasks | `C1:snappy:T4` (registered suite, `kInputMarginBytes`, mechanical `check.py`); `C2:lz4.smoke:S1` (smoke-only fixture, LLM judge) |
+| Agent | `claude -p` 2.1.302, `--model sonnet` (resolved: `claude-sonnet-5-5`), `--max-turns 15` (A0: 1) |
+| xerj | `v1.0.0-rc.96` built from `main` at `b3753ddb` |
+| Packs | the real releases `pack-lz4-0774d055`, `pack-snappy-9c28114a`, `pack-zstd-01b7154f`. Each archive's released `.sig` verified against the compiled-in `HUB_PUBKEY_HEX` before use. |
+| Sandbox | every `claude -p` ran through [`sandbox.sh`](./sandbox.sh): bwrap, own PID and user namespace, private network namespace, `/home` masked |
+| Run dirs | `/home/claude/hubdev-wt/pov-run/results/smoke-20261017b/` (the valid pass); `smoke-20261017/` is the first pass, kept because it exposed defects 8 and 9 |
+| Spend | $0.40 for the agents across 16 cells; 241 s of agent wall time; about 5 min end to end |
+| Host load (1-min) | **25 to 113** throughout. This is not a quiet machine, so every wall time below is invalid as a timing (PLAN §6). |
+
+### The catalogue override (the live hub catalogue is not deployed)
+
+`https://hub.xerj.org/api/v1/catalogue.json` returns the site's HTML page. The smoke run
+therefore used the override that `tools/hub-packs/consumer_e2e.sh` uses:
+`xerj hub build-catalogue` builds the catalogue and it is served over HTTP as
+`<hub>/api/v1/catalogue.json` + `.sig`, with `XERJ_HUB_URL` pointing at it. Two
+differences from the CI e2e, both forced by running locally:
+
+1. **Throwaway signing key.** The hub seed is a CI secret, so the catalogue is signed
+ with a throwaway ed25519 key, and the client is pointed at that key with
+ `XERJ_HUB_PUBKEY`. This is the local-rehearsal override `consumer_e2e.sh` documents.
+ The packs' signatures are re-made with the same key, because there is only one
+ client key. The pack **bytes** are the released ones: `pack_url` points at the
+ GitHub release, so the v2 arms download real bytes over the real network, and
+ `pack_sha256` in the catalogue pins them. Their genuine release signatures were
+ verified separately (see "Packs" above).
+2. **The catalogue is served inside each jail** (`python3 -m http.server` on the jail's
+ own `127.0.0.1:18080`), because the jail's network namespace cannot reach the host
+ loopback (defect 1). The re-made `.sig` files are served from there too
+ (`sig_url` is rewritten, then the catalogue is re-signed).
+
+Both exports live only in the v2 `xerj` shim (`XERJ_HUB_URL`, `XERJ_HUB_PUBKEY`).
+**Before the full run, switch to the deployed, really-signed catalogue** and drop both
+from the shim.
+
+## Smoke table (run `smoke-20261017b`; NOT results)
+
+| arm | task | valid | grade | cost $ | out tok | turns | wall s* | bytes dl | xerj calls | routing (H2) | contamination check |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| A0 | C1 snappy T4 | yes | pass | 0.0137 | 406 | 1 | 6.3 | 15,824 | 0 | – | clean |
+| A1 | C1 snappy T4 | yes | pass | 0.0168 | 264 | 2 | 7.2 | 1,183,155 | 0 | – | clean (fetched the pin with git) |
+| A2 | C1 snappy T4 | yes | pass | 0.0173 | 268 | 2 | 6.8 | 151,260 | 0 | – | clean (raw.githubusercontent) |
+| A3 | C1 snappy T4 | yes | pass | 0.0191 | 224 | 2 | 6.7 | 25,092 | 0 | – | clean (Sourcegraph only) |
+| A4 | C1 snappy T4 | **NO** | pass | 0.0187 | 441 | 2 | 7.3 | 158,487 | 0 | – | **violation: never called the search API, fetched raw GitHub directly** |
+| A5 | C1 snappy T4 | yes | pass | 0.0413 | 549 | 4 | 43.3 | 1,203,400 | 3 | – | clean (`corpus add` → `index` → `code`) |
+| A6 | C1 snappy T4 | yes | fail† | 0.0293 | 413 | 4 | 21.3 | 11,467,819 | 4 | no_corpus → wrong | clean; pulled snappy (11.3 MB); answered_from local_pack |
+| A7 | C1 snappy T4 | yes | fail† | 0.0294 | 525 | 4 | 14.3 | 18,262 | 4 | no_corpus → wrong | clean; pack pre-pulled (12.0 s setup); answered_from local_pack |
+| A0 | C2 lz4 S1 | yes | unscored‡ | 0.0161 | 633 | 1 | 7.6 | 16,878 | 0 | – | clean |
+| A1 | C2 lz4 S1 | yes | unscored‡ | 0.0215 | 994 | 1 | 9.7 | 18,422 | 0 | – | note: answered with no tool call |
+| A2 | C2 lz4 S1 | yes | unscored‡ | 0.0217 | 941 | 1 | 9.0 | 18,544 | 0 | – | note: answered with no tool call |
+| A3 | C2 lz4 S1 | yes | unscored‡ | 0.0215 | 988 | 1 | 9.8 | 19,372 | 0 | – | note: answered with no tool call |
+| A4 | C2 lz4 S1 | yes | unscored‡ | 0.0200 | 850 | 1 | 9.4 | 17,739 | 0 | – | note: answered with no tool call |
+| A5 | C2 lz4 S1 | yes | unscored‡ | 0.0527 | 1054 | 4 | 37.2 | 488,952 | 3 | – | clean |
+| A6 | C2 lz4 S1 | yes | unscored‡ | 0.0316 | 898 | 3 | 31.3 | 4,700,587 | 2 | no_corpus → wrong | clean; pulled lz4 (4.6 MB); local_pack |
+| A7 | C2 lz4 S1 | yes | unscored‡ | 0.0309 | 929 | 3 | 14.3 | 19,579 | 2 | no_corpus → wrong | clean; pre-pulled (15.3 s); local_pack |
+
+\* Loaded machine: timings are invalid (PLAN §6). † The raw `check.py` grade was a
+pass. PLAN H2 turns it into a fail because the first v2 question did not route (see
+below). ‡ `ANTHROPIC_API_KEY` is not set on this host, so every C2 cell is `unscored`,
+exactly as PLAN §6 requires.
+
+### What the smoke run verified
+
+- **Each arm had only its intended tools.** The init event's `tools` matches the
+ declared set in every cell: A0 has none, A2 adds WebFetch/WebSearch, and the rest
+ have Bash/Read/Grep/Glob. MCP servers and plugins: none. Model: `claude-sonnet-5-5`.
+ An independent grep over all 16 raw transcripts found no `xerj` call outside A5–A7,
+ no `git` outside A1, no clone or archive download in A2–A7, and only Sourcegraph in
+ A3. The check caught one real case of contamination: the A4 C1 cell, which is marked
+ invalid.
+- **The v2 arms really used the pack.** Both A6 and A7 got `answered from: local pack`
+ in every cell. A6 started cold (`packs_before = []`) and its pull shows up in the
+ bytes (11,467,819 B against the 11,338,487 B snappy pack; 4,700,587 B against the
+ 4,623,814 B lz4 pack). A7 started warm (`packs_before = [snappy]` / `[lz4]`), with
+ the pre-pull timed separately (`setup.rx` 11,472,275 / 4,671,184 B). Its downloads
+ while the agent ran were only about 18 KB of baseline noise. No v2 arm touched
+ `corpus add` or `corpus index`, and the v1 arm never touched `ask` or `hub`.
+- **Metrics are captured** for every cell: correct (or unscored), cost and output tokens
+ (summed over every `result` event, with ≥1 event asserted), turns, bytes downloaded
+ (the jail interface delta minus Anthropic-range bytes from an nft counter), wall
+ time with 1/5/15 load before and after, fetches and failed fetches, xerj calls,
+ routing, A7 setup time and bytes. Schema: [`results/schema.json`](./results/schema.json).
+
+### Product observations (seen while validating; NOT results, not measured)
+
+These fall under H2 (routing). They are listed so nobody is surprised by them in the full run:
+
+1. **The router returned NoCorpus for every first v2 question in the smoke run (4/4)**,
+ for example `xerj "In snappy, what is the exact value of kInputMarginBytes?"` →
+ `no hub corpus matches … closest corpora: snappy, lz4` (exit 1). Every agent then
+ re-asked with `--corpus`. With only 3 corpora in the catalogue this is the sketch
+ scoring (§5 step 3: tokens present / tokens total, ×0.8): ordinary question words
+ dilute the score below 0.6.
+2. **Def mode missed `kInputMarginBytes`**: `ask --corpus snappy "what is the exact
+ value of kInputMarginBytes"` exited 1. It is a function-local `const` in
+ `snappy.cc`, so it is not a def record. A plain how-query then found it.
+3. **First use on a cold machine:** in a no-agent dry run at load ~50, the
+ auto-started node did not report healthy within the 60 s wait, so the first `ask`
+ failed. The second one worked. A6 may hit this on a loaded runner.
+
+## Harness defects found and fixed
+
+| # | Defect | Effect if unfixed | Fix |
+|---|---|---|---|
+| 1 | `sandbox.sh` shared the **host network**. `xerj ask` probes `localhost:9200` first and uses whatever answers. | On this host, :9200 is the user's reference-coding node, so a v2 arm would have restored packs **into the user's node** and answered from its corpora. That is both contamination and a write to a node we must never touch. An A5 agent's `corpus add/index` would have hit it too. | Private network namespace via `pasta` (`-t/-u/-T/-U none --no-map-gw`: nothing forwarded either way), plus an nft rule rejecting RFC1918/CGNAT/link-local destinations, which closes the route through `docker0` (10.200.0.1). Verified: a host server on 127.0.0.1 and on 0.0.0.0 is unreachable from the jail, while github.com and the API are reachable. |
+| 2 | `--ro-bind / /` exposed all of `/home`. | Arms could read the gold answers (`benchmarks/corpus-tasks`, the 2026-10-13 run dir), the local corpus clones in `~/.xerj-code`, and `.env` secrets. | `/home` and `/root` are masked with tmpfs. Only HOME, the auth dir and explicit `--ro` paths are visible. |
+| 3 | `CLAUDE_CONFIG_DIR` was bound as-is. | Every arm loaded the user's global `CLAUDE.md`, skills, plugins and Stop hook, and could read other sessions' transcripts and history. `settings.json` `env` **injected `GH_TOKEN` into every arm**, despite `--clearenv`. | Everything in the dir except `.credentials.json` and `.claude.json` is masked, and `settings.json` is replaced by `{}`. The harness also passes `--strict-mcp-config --mcp-config '{"mcpServers":{}}' --disable-slash-commands --tools <arm set>`. |
+| 4 | Inside pasta's user namespace the jail ran as uid 0. | `claude --dangerously-skip-permissions` refuses to run as root. | `bwrap --unshare-user --uid/--gid <real ids>`. |
+| 5 | The host `resolv.conf` points at 127.0.0.53 (host loopback). | No DNS in the jail. | The jail gets its own `resolv.conf` (1.1.1.1, 8.8.8.8). |
+| 6 | Masking files by binding `/dev/null` read back as EACCES. | `CLAUDE.md` read errors. | Masks are bound from regular empty files. |
+| 7 | Catalogue override: `build-catalogue` wants the `.sig` beside each pack and verifies it with the client key; `sig_url` points at the real-key release `.sig`. | The override could not be built with a throwaway key, and a pull would fail signature verification. | Packs are re-signed with the throwaway key, `sig_url` is rewritten to the in-jail server, the catalogue is re-signed, and `XERJ_HUB_PUBKEY` is set for `build-catalogue` and the v2 shim (see above). |
+| 8 | A5 prompt said `xerj corpus index` without the corpus name. | One wasted turn per A5 cell. | `xerj corpus index <slug>`. |
+| 9 | The bytes counter mis-parsed `/proc/net/dev`: interface lines with no leading space shift the columns, so it summed packets instead of bytes. | `bytes_downloaded` was always ≈0. | Fixed parser; the A7 pre-pull is measured separately (`setup.rx`). |
+| 10 | The contamination check let an A4 agent skip its search API. | An A4 cell measured A2. | A3/A4 must call their endpoint (violation otherwise); tool arms that answered with zero tool calls get a note. |
+| 11 | Clone bytes were measured from `.git` dirs in the work dir. | Agents clone into the jail's `/tmp`, so the measure read 0. | Removed. Clone bytes are in the network-namespace `bytes_downloaded`. |
+
+## Ambiguities in PLAN.md (flagged, not silently chosen)
+
+The harness had to pick something to run the smoke. Each item below is what it picked,
+and needs the user's decision at freeze (PLAN §10):
+
+1. **"Nine arms"** (§3), but the table lists eight (A0–A7). The harness implements the eight listed.
+2. **A3/A4 endpoints** are "locked at freeze". Provisional: A3 = Sourcegraph public
+ stream search (`sourcegraph.com/.api/search/stream`, unauthenticated; it searches
+ upstream HEAD, not the pin); A4 = DuckDuckGo HTML (`html.duckduckgo.com/html/`, no
+ key). grep.app was rejected because it answers curl with a Vercel bot checkpoint (429).
+3. **`--max-turns`**: "same for every arm", value not given. Provisional: 15, with A0 = 1
+ and sessions at 15 × questions.
+4. **Pin vs "current development source"**: the C1 questions ask for "current" while
+ §4 grades at the pack pin. The harness adds one line to every arm's prompt: "The
+ reference version is commit `<sha>` (`<date>`) of `<url>`; answer for that
+ version". A1 is told how to fetch that commit.
+5. **A2 "may not git clone"**: also read as no archive/tarball download (a clone by
+ other means). Single-file raw fetches are allowed. A `git` shim refuses
+ clone/fetch/pull/ls-remote.
+6. **A5 "install xerj"**: the binary is placed on PATH for A5–A7 alike, so install
+ cost is not measured. The prior run instead had the agent read llms.txt and install.
+7. **A7 "steady state"**: the harness pre-pulls in the same jail, so the hub node is
+ already up when the agent starts. A6 pays node boot plus pull.
+8. **H2 and NoCorpus**: H2 counts "Confident, or Ambiguous containing the correct
+ slug" as correct. The client actually returned **NoCorpus with a "closest corpora"
+ list that contained the right slug**. Literal reading: not ok, so the answer is
+ counted wrong even when the agent re-asks with `--corpus` and gets it right. The
+ harness does this and records `closest_contains_expected` so the decision can be
+ revisited.
+9. **Corpora**: §4 names lmdb, quinn, leveldb, boringtun, badger, s2n-tls and lz4.
+ Only lz4/snappy/zstd have v2 packs today. **C3 cannot run at all**: no non-git,
+ multi-source corpus has a v2 pack.
+10. **C2/C3 tasks** must be authored by a person and committed before any C2/C3 run
+ (§5). None exist. `tasks/C2/lz4.smoke.json` is a smoke fixture written by the
+ harness agent, and `pov.py` refuses it without `--allow-smoke-fixtures`.
+11. **Judge**: no `ANTHROPIC_API_KEY` on this host, so the judge's live HTTP path is
+ untested (the unscored path is tested). Verify at freeze that `claude-opus-4-8`
+ exists and accepts `temperature: 0`.
+12. **Hosted**: spec §12 puts hosted answers out of scope, so A6's first use is always a
+ synchronous pull, never a hosted answer plus a background pull (§1 describes both).
+13. **"Quiet machine"**: this host ran at load 25–113. The full run needs a host where
+ load stays low, or the wall-time column has to be dropped.
+14. **Failed fetches** is a heuristic: the tool result is empty or an error, or
+ matches captcha/checkpoint/403/429 text.
+15. **C1 grading looseness** (registered suite, unchanged): regexes like `0xf|15` are
+ unanchored, so for example "115" passes. This is pre-existing and noted because it
+ affects C1 accuracy for every arm equally.
+16. `rg` is not installed on this host. A1's "ripgrep" falls back to grep, and the
+ prompt says so.
+
+## Estimated cost and wall time of the full pre-registered run
+
+Assumptions: the 3 available packs; C1 = 3 × 10 registered tasks; C2 = 3 × 5 authored
+tasks (count not fixed in the PLAN); C3 = 0 (no pack); C4 = 3 sessions × 10 questions;
+8 arms; 3 repeats; serial. Cost per question unit is the smoke per-arm mean (about
+$0.015–0.047; A5 is the most expensive).
+
+| | agents | judge (C2 + C4's C2-style, opus) | wall (serial) |
+|---|---|---|---|
+| 3 repeats | **≈ $45** | ≈ $20 | **≈ 8.5 h** |
+| + "+2 repeats on CV > 0.5" for ~30% of cells | ≈ $59 | ≈ $25 | ≈ 11 h |
+
+**This is a floor, not a quote.** The smoke questions were easy: answered in 1–4
+turns, and most C2 cells used no tools at all. Authored cross-file C2 questions and
+10-question C4 sessions will take more turns. If the pinned agent model is Opus
+rather than Sonnet, the per-turn price is higher. A realistic envelope is
+**$150–300 and 10–15 h**. Run one repeat of the frozen matrix first and re-estimate
+from that before committing the rest.
+
+## Running it
+
+```sh
+# once per run dir (downloads + verifies the packs, builds the override, writes shims)
+python3 benchmarks/hub-pov/harness/pov.py prepare --xerj engine/target/release/xerj \
+ --registry <corpus-hub checkout>/tools/xerj-code/hub --registry-commit <sha>
+python3 benchmarks/hub-pov/harness/pov.py verify-c1 # vendored C1 suites == bench/corpus-tasks
+python3 benchmarks/hub-pov/harness/pov.py matrix benchmarks/hub-pov/matrix/smoke.json --run-id <id>
+python3 benchmarks/hub-pov/harness/pov.py score --run-id <id>
+python3 benchmarks/hub-pov/harness/pov.py report --run-id <id>
+python3 -m unittest benchmarks/hub-pov/harness/test_pov.py # offline parser/grader/contamination tests
+```
+
+Results are written under `$POV_RUN_DIR` (default `/home/claude/hubdev-wt/pov-run`) and
+never into the repository. Nothing in the harness publishes anything.
diff --git a/benchmarks/hub-pov/harness/check.py b/benchmarks/hub-pov/harness/check.py
new file mode 100644
index 000000000..7d55ff398
--- /dev/null
+++ b/benchmarks/hub-pov/harness/check.py
@@ -0,0 +1,17 @@
+#!/usr/bin/env python3
+"""Grade one run's ANSWER line against the task's regex set."""
+import json, re, sys
+task = json.load(open(sys.argv[1]))
+if "tasks" in task: # whole suite passed: argv[2] = task id, argv[3] = run
+ task = next(t for t in task["tasks"] if t["id"] == sys.argv[2])
+ run_json = json.load(open(sys.argv[3]))
+else:
+ run_json = json.load(open(sys.argv[2]))
+text = (run_json.get("result") or "") + "\n" + run_json.get("_raw", "")
+m = re.findall(r"ANSWER:\s*(.+)", text)
+ans = m[-1] if m else text[-800:]
+need = task.get("need", 1)
+hits = sum(1 for pat in task["re"] if re.search(pat, ans, re.I))
+print(json.dumps({"id": task["id"], "answered": bool(m), "hits": hits,
+ "need": need, "pass": hits >= need,
+ "answer": ans.strip()[:300]}))
diff --git a/benchmarks/hub-pov/harness/pov.py b/benchmarks/hub-pov/harness/pov.py
new file mode 100644
index 000000000..a6a9bfde7
--- /dev/null
+++ b/benchmarks/hub-pov/harness/pov.py
@@ -0,0 +1,964 @@
+#!/usr/bin/env python3
+"""Corpus Hub v2 proof-of-value harness (benchmarks/hub-pov/PLAN.md).
+
+One harness for all nine arms (PLAN §3: "All arms share one agent harness and
+differ only in the tools on PATH and the prompt's tool instructions"). Every
+`claude -p` runs inside ../sandbox.sh — there is no code path that launches an
+agent outside it.
+
+Subcommands
+ prepare fetch + verify the real signed packs, build the catalogue override,
+ write the shims (once per run dir)
+ cell run one cell: arm x task(s) x repeat (one sandboxed agent)
+ matrix run every cell of a matrix file, serially (PLAN §9: one at a time)
+ score grade + metrics + contamination for every cell of a run
+ report aggregate a scored run into a table (+ the §7 gate, refused < 3 repeats)
+ verify-c1 check the vendored C1 suites against branch bench/corpus-tasks
+
+Nothing here publishes anything. Results stay in the run dir (default
+/home/claude/hubdev-wt/pov-run, override with POV_RUN_DIR).
+"""
+from __future__ import annotations
+
+import argparse
+import base64
+import datetime as dt
+import json
+import os
+import re
+import shlex
+import shutil
+import statistics
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+POV = HERE.parent # benchmarks/hub-pov
+REPO = POV.parent.parent # repo root
+SANDBOX = POV / "sandbox.sh"
+TASKS = POV / "tasks"
+RUN_DIR = Path(os.environ.get("POV_RUN_DIR", "/home/claude/hubdev-wt/pov-run"))
+
+# The catalogue override is served INSIDE each jail (its network namespace cannot
+# reach the host loopback) on this port, from a read-only bind of <run>/hub/site.
+CAT_PORT = 18080
+CAT_URL = f"http://127.0.0.1:{CAT_PORT}"
+# Where the real packs live. pack_url in the override catalogue points HERE, so the
+# v2 arms download the real released bytes over the real network.
+RELEASE_BASE = "https://github.com/xerj-org/xerj/releases/download"
+
+# PLAN §3 leaves these unfixed ("locked at freeze", §10). PROVISIONAL values used
+# by the smoke run only — see SMOKE.md "Ambiguities".
+A3_ENDPOINT = "https://sourcegraph.com/.api/search/stream"
+A4_ENDPOINT = "https://html.duckduckgo.com/html/"
+
+# PLAN §3: "same --max-turns" for every arm but A0 (1). The number is not in the
+# PLAN — PROVISIONAL, flagged in SMOKE.md. Session cells (C4) get the per-question
+# budget times the number of questions.
+MAX_TURNS = 15
+CELL_TIMEOUT_S = 900
+JUDGE_MODEL = "claude-opus-4-8" # PLAN §6, verbatim
+
+BASE_TOOLS = "Bash,Read,Grep,Glob"
+ARMS = {
+ # tools: the --tools list (the built-in tools the arm may call at all)
+ # shims: which PATH shims are exposed; xerj: which xerj shim (None = not on PATH)
+ "A0": {"role": "memory only (floor)", "tools": "", "xerj": None, "git_noclone": False},
+ "A1": {"role": "normal agent + git (gate baseline)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": False},
+ "A2": {"role": "normal agent + web only (gate baseline)", "tools": BASE_TOOLS + ",WebFetch,WebSearch", "xerj": None, "git_noclone": True},
+ "A3": {"role": "hosted regex code search (reference)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": True},
+ "A4": {"role": "web-search API (reference)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": True},
+ "A5": {"role": "XERJ today, v1 corpus add path", "tools": BASE_TOOLS, "xerj": "v1", "git_noclone": False},
+ "A6": {"role": "XERJ v2 first use (cold)", "tools": BASE_TOOLS, "xerj": "v2", "git_noclone": False},
+ "A7": {"role": "XERJ v2 steady state (pack cached)", "tools": BASE_TOOLS, "xerj": "v2", "git_noclone": False},
+}
+XERJ_ARMS = {"A5", "A6", "A7"}
+V2_ARMS = {"A6", "A7"}
+
+
+# --------------------------------------------------------------------------- utils
+def now() -> str:
+ return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def die(msg: str, code: int = 2):
+ print(f"pov.py: {msg}", file=sys.stderr)
+ sys.exit(code)
+
+
+def load_json(p: Path):
+ return json.loads(Path(p).read_text())
+
+
+def write_json(p: Path, obj) -> None:
+ p.parent.mkdir(parents=True, exist_ok=True)
+ p.write_text(json.dumps(obj, indent=1, sort_keys=False) + "\n")
+
+
+def hub_pubkey_hex() -> str:
+ src = (REPO / "engine/crates/xerj-common/src/hub.rs").read_text()
+ m = re.search(r'HUB_PUBKEY_HEX: &str = "([0-9a-f]{64})"', src)
+ if not m:
+ die("cannot read HUB_PUBKEY_HEX from engine/crates/xerj-common/src/hub.rs")
+ return m.group(1)
+
+
+def ed25519(*args: str) -> str:
+ return subprocess.run(["bash", str(REPO / "tools/hub-packs/ed25519.sh"), *args],
+ check=True, capture_output=True, text=True).stdout.strip()
+
+
+PKCS8_PREFIX = "302e020100300506032b657004220420"
+
+
+def ed25519_sign(seed_hex: str, data_file: Path, sig_file: Path, scratch: Path) -> None:
+ """Detached signature in the hub format (128 hex chars), with openssl only."""
+ der = scratch / "seed.der"
+ der.write_bytes(bytes.fromhex(PKCS8_PREFIX + seed_hex))
+ raw = subprocess.run(["openssl", "pkeyutl", "-sign", "-inkey", str(der), "-keyform", "DER",
+ "-rawin", "-in", str(data_file)], check=True, capture_output=True).stdout
+ der.unlink()
+ sig_file.write_text(raw.hex() + "\n")
+
+
+# --------------------------------------------------------------------------- tasks
+def registry_manifest(prep: dict, slug: str) -> dict:
+ return load_json(Path(prep["registry"]) / f"{slug}.json")
+
+
+def load_task(ref: str, allow_smoke: bool) -> dict:
+ """ref = '<class>:<suite>:<id>' e.g. C1:snappy:T4, C2:lz4.smoke:S1."""
+ try:
+ cls, suite, tid = ref.split(":")
+ except ValueError:
+ die(f"task ref must be <class>:<suite>:<id>, got {ref!r}")
+ f = TASKS / cls / f"{suite}.json"
+ if not f.exists():
+ die(f"no task file {f}")
+ doc = load_json(f)
+ if doc.get("smoke_only") and not allow_smoke:
+ die(f"{f} is a smoke-only fixture (not pre-registered, PLAN §5); refusing without --allow-smoke-fixtures")
+ task = next((t for t in doc["tasks"] if t["id"] == tid), None)
+ if task is None:
+ die(f"no task {tid} in {f}")
+ return {"ref": ref, "class": cls, "suite": suite, "corpus": doc["corpus"],
+ "subject": doc.get("subject", doc["corpus"]), "suite_pin": doc.get("pin"),
+ "smoke_only": bool(doc.get("smoke_only")), "task": task}
+
+
+# --------------------------------------------------------------------------- prompts
+COMMON = ("You are answering one technical question about the {subject} source tree. Be precise "
+ "and brief. You MUST end your reply with a single line of the form 'ANSWER: <your answer>'.")
+COMMON_SESSION = ("You are answering {n} technical questions about the {subject} source tree. Be "
+ "precise and brief. You MUST end your reply with one line per question, of the "
+ "form 'ANSWER <id>: <your answer>', in the order given.")
+PIN_LINE = ("The reference version is commit {pin} ({pin_date}) of {url}; answer for that version.")
+
+
+def arm_instructions(arm: str, slug: str, url: str, pin: str, prep: dict) -> str:
+ org_repo = url.removeprefix("https://github.com/")
+ if arm == "A0": # PROTOCOL.md §4 bare prompt, verbatim
+ return ("You are offline: no network access, and its source code is not available to you. "
+ "Answer from your own knowledge; if you do not know, say 'unknown' in the ANSWER "
+ "line — do not guess a fabricated constant.")
+ if arm == "A1":
+ return (f"You have a bash shell with network access. The project's public git repository is "
+ f"{url}. You may clone it (e.g. `git clone --depth 1 {url}`; the reference commit "
+ f"can be fetched with `git fetch --depth 1 origin {pin}`), search it with grep "
+ f"(or ripgrep if installed) and read files.")
+ if arm == "A2":
+ return (f"You have a bash shell with network access and web tools (WebFetch, WebSearch, "
+ f"curl). You may read web pages, including single files on GitHub (for example "
+ f"https://raw.githubusercontent.com/{org_repo}/{pin}/<path> or the GitHub web UI). "
+ f"You may NOT clone or fetch the git repository and may NOT download an "
+ f"archive/tarball of it. The project's home is {url}.")
+ if arm == "A3":
+ return (f"You have a bash shell with network access and ONE tool for finding code: the "
+ f"Sourcegraph public code-search API, via curl only: "
+ f"curl -s -H 'Accept: text/event-stream' '{A3_ENDPOINT}?q=<url-encoded query>&display=20' "
+ f"(query syntax: `repo:^github\\.com/{org_repo}$`, `file:<regex>`, "
+ f"`patternType:regexp`). Do not clone the repository and do not fetch files from "
+ f"any other site.")
+ if arm == "A4":
+ return (f"You have a bash shell with network access and ONE web-search API, via curl: "
+ f"curl -s -A 'Mozilla/5.0' '{A4_ENDPOINT}?q=<url-encoded query>' (DuckDuckGo HTML "
+ f"results). You may then fetch result pages with curl. Do not clone the repository "
+ f"and do not download an archive/tarball of it.")
+ if arm == "A5":
+ manifest = Path(prep["registry"]) / f"{slug}.json"
+ return ("XERJ is installed (`xerj` is on PATH). Use its reference-code flow: (1) start a node "
+ "in the BACKGROUND, never in the foreground: `nohup xerj --insecure --port 9200 "
+ "--data-dir $HOME/xerj-data > $HOME/xerj-node.log 2>&1 &` and wait until "
+ "`curl -s localhost:9200/_cluster/health` answers; (2) `xerj corpus add --from "
+ f"{manifest}`; (3) `xerj corpus index {slug}`; (4) query it: `xerj code {slug} \"<your "
+ "search query>\"`. Do not clone or download the source yourself.")
+ if arm in V2_ARMS:
+ return ("XERJ is installed (`xerj` is on PATH). Ask it in plain English from bash: "
+ "`xerj \"<your question>\"` — it picks the right reference corpus and answers from a "
+ "signed, pre-indexed hub pack (`xerj ask --help` lists options). Use it as much as "
+ "helps; cite what you found. Do not clone or download the source yourself.")
+ die(f"unknown arm {arm}")
+
+
+def build_prompt(arm: str, tasks: list[dict], prep: dict) -> str:
+ slug = tasks[0]["corpus"]
+ if any(t["corpus"] != slug for t in tasks):
+ die("a session cell must stay within one corpus (PLAN §4 C4: 'ten questions of one corpus')")
+ man = registry_manifest(prep, slug)
+ src = man["repos"][0]
+ pin_date = prep["packs"][slug]["pin_date"]
+ subject = tasks[0]["subject"]
+ if len(tasks) == 1:
+ head = COMMON.format(subject=subject)
+ qs = "The question: " + tasks[0]["task"]["q"]
+ else:
+ head = COMMON_SESSION.format(subject=subject, n=len(tasks))
+ qs = "The questions:\n" + "\n".join(f"[{t['task']['id']}] {t['task']['q']}" for t in tasks)
+ pin_line = PIN_LINE.format(pin=src["sha"], pin_date=pin_date, url=src["url"])
+ return " ".join([head, pin_line, arm_instructions(arm, slug, src["url"], src["sha"], prep)]) + "\n" + qs
+
+
+# --------------------------------------------------------------------------- prepare
+def cmd_prepare(a):
+ run = RUN_DIR
+ xerj = Path(a.xerj).resolve()
+ if not xerj.is_file():
+ die(f"no xerj binary at {xerj}")
+ registry = Path(a.registry).resolve()
+ backlog = registry / "backlog/backlog-100.json"
+ hub = run / "hub"
+ (hub / "packs").mkdir(parents=True, exist_ok=True)
+ (hub / "site/api/v1").mkdir(parents=True, exist_ok=True)
+ (hub / "site/sigs").mkdir(parents=True, exist_ok=True)
+ scratch = run / "scratch"; scratch.mkdir(exist_ok=True)
+ pub_real = hub_pubkey_hex()
+ tags = [t.strip() for t in a.tags.split(",") if t.strip()]
+ rows, packs = [], {}
+ for tag in tags:
+ m = re.fullmatch(r"pack-(.+)-([0-9a-f]{8})", tag)
+ if not m:
+ die(f"not a pack release tag: {tag}")
+ slug, pin8 = m.groups()
+ fname = f"{slug}-{pin8}.hubpack.tar.zst"
+ dest = hub / "packs" / tag
+ dest.mkdir(exist_ok=True)
+ if not (dest / fname).exists():
+ subprocess.run(["gh", "release", "download", tag, "-R", "xerj-org/xerj", "-D", str(dest),
+ "--clobber"], check=True)
+ # 1. The released bytes are the genuine, hub-key-signed pack.
+ ed25519("verify", pub_real, str(dest / fname), str(dest / f"{fname}.sig"))
+ pj = json.loads(subprocess.run(f"zstd -dc {shlex.quote(str(dest / fname))} | tar -xOf - pack.json",
+ shell=True, check=True, capture_output=True, text=True).stdout)
+ packs[slug] = {"tag": tag, "file": fname, "pin8": pin8, "pin_sha": pj["pin_sha"],
+ "pin_date": pj["pin_date"], "bytes": (dest / fname).stat().st_size,
+ "licence_class": pj.get("licence_class"),
+ "real_sig_verified_against": pub_real}
+ rows.append(f"{slug}\t{pin8}\t{tag}\n")
+ # 2. A throwaway signing key for the override. The live catalogue is not
+ # deployed and the hub seed is a CI secret, so the override catalogue cannot
+ # be signed with the real key; the client is pointed at the throwaway public
+ # key (XERJ_HUB_PUBKEY, the override consumer_e2e.sh documents for local
+ # rehearsals) and the pack signatures are re-made with it. The pack BYTES are
+ # still the released ones (sha256 in the catalogue; downloaded from GitHub).
+ seed_f = run / "hub/throwaway.seed"
+ if not seed_f.exists():
+ seed_f.write_text(os.urandom(32).hex() + "\n")
+ seed_f.chmod(0o600)
+ pub_tmp = ed25519("derive", str(seed_f))
+ flat = hub / "flat"; flat.mkdir(exist_ok=True)
+ for slug, p in packs.items():
+ src = hub / "packs" / p["tag"] / p["file"]
+ link = flat / p["file"]
+ if not link.exists():
+ os.link(src, link)
+ sig = hub / "site/sigs" / f"{p['file']}.sig"
+ ed25519_sign(seed_f.read_text().strip(), src, sig, scratch)
+ ed25519("verify", pub_tmp, str(src), str(sig))
+ shutil.copy(sig, flat / f"{p['file']}.sig") # build-catalogue reads it beside the pack
+ (hub / "tags.tsv").write_text("".join(rows))
+ cat = hub / "site/api/v1/catalogue.json"
+ subprocess.run([str(xerj), "hub", "build-catalogue", "--registry", str(registry), "--backlog",
+ str(backlog), "--packs", str(flat), "--tags", str(hub / "tags.tsv"),
+ "--release-base", RELEASE_BASE, "--out", str(cat), "--sign-key", str(seed_f)],
+ check=True, env=dict(os.environ, XERJ_HUB_PUBKEY=pub_tmp))
+ doc = load_json(cat)
+ for e in doc["corpora"]:
+ p = packs[e["slug"]]
+ if not e["pack_url"].startswith(RELEASE_BASE):
+ die(f"unexpected pack_url {e['pack_url']}")
+ e["sig_url"] = f"{CAT_URL}/sigs/{p['file']}.sig"
+ cat.write_text(json.dumps(doc, separators=(",", ":")))
+ ed25519_sign(seed_f.read_text().strip(), cat, Path(str(cat) + ".sig"), scratch)
+ ed25519("verify", pub_tmp, str(cat), str(cat) + ".sig")
+ # 3. Shims.
+ shims = run / "shims"
+ for kind in ("v1", "v2"):
+ d = shims / f"xerj-{kind}"; d.mkdir(parents=True, exist_ok=True)
+ exports = "" if kind == "v1" else (f"export XERJ_HUB_URL='{CAT_URL}'\n"
+ f"export XERJ_HUB_PUBKEY='{pub_tmp}'")
+ body = (HERE / "shim_xerj.sh").read_text().replace("@HUB_EXPORTS@", exports).replace("@REAL@", str(xerj))
+ (d / "xerj").write_text(body); (d / "xerj").chmod(0o755)
+ d = shims / "git-noclone"; d.mkdir(parents=True, exist_ok=True)
+ shutil.copy(HERE / "shim_git_noclone.sh", d / "git"); (d / "git").chmod(0o755)
+ claude = Path(shutil.which("claude") or die("claude not on PATH")).resolve()
+ prep = {"prepared_at": now(), "xerj": str(xerj), "xerj_version": subprocess.run(
+ [str(xerj), "--version"], capture_output=True, text=True).stdout.strip(),
+ "claude": str(claude), "claude_version": subprocess.run(
+ [str(claude), "--version"], capture_output=True, text=True).stdout.strip(),
+ "registry": str(registry), "registry_commit": a.registry_commit,
+ "catalogue_override": {"site": str(hub / "site"), "url_in_jail": CAT_URL,
+ "throwaway_pubkey": pub_tmp, "real_pubkey": pub_real,
+ "pack_url_base": RELEASE_BASE},
+ "packs": packs, "shims": str(shims)}
+ write_json(run / "prepared.json", prep)
+ print(json.dumps(prep, indent=1))
+
+
+def load_prep() -> dict:
+ p = RUN_DIR / "prepared.json"
+ if not p.exists():
+ die(f"{p} missing — run `pov.py prepare` first")
+ return load_json(p)
+
+
+# --------------------------------------------------------------------------- cell
+def cell_dir(run_id: str, arm: str, refs: list[str], repeat: int) -> Path:
+ key = "+".join(r.replace(":", "_") for r in refs) if len(refs) <= 2 else \
+ f"{refs[0].split(':')[0]}_{refs[0].split(':')[1]}_session{len(refs)}"
+ return RUN_DIR / "results" / run_id / arm / key / f"r{repeat}"
+
+
+def run_script(arm: str, prompt_file: Path, prep: dict, tasks: list[dict], model: str,
+ max_turns: int) -> str:
+ """The script executed INSIDE the jail (bash). Generated, then run via sandbox.sh."""
+ slug = tasks[0]["corpus"]
+ claude = prep["claude"]
+ tools = ARMS[arm]["tools"]
+ args = [claude, "-p", "--model", model, "--output-format", "stream-json", "--verbose",
+ "--max-turns", str(max_turns), "--tools", tools, "--strict-mcp-config",
+ "--mcp-config", '{"mcpServers":{}}', "--disable-slash-commands",
+ "--dangerously-skip-permissions"]
+ q = shlex.quote
+ co = prep["catalogue_override"]
+ lines = ["set -u", 'S="$HOME/.sandbox"; mkdir -p "$HOME/.pov"',
+ # received bytes on every non-loopback interface of the jail's netns
+ 'netrx() { sed "s/:/ /" /proc/net/dev | awk \'NR>2 && $1!="lo" {s+=$2} END {print s+0}\'; }']
+ if arm in V2_ARMS:
+ lines += [f"python3 -m http.server {CAT_PORT} --bind 127.0.0.1 --directory {q(co['site'])} "
+ f'> "$S/catalogue-http.log" 2>&1 &',
+ f"for i in $(seq 1 50); do curl -fs -m1 {CAT_URL}/api/v1/catalogue.json >/dev/null && break; sleep 0.2; done"]
+ if arm == "A7":
+ # Steady state = the pack is already pulled AND the hub node is up (as on a
+ # machine that has answered before). The pull runs with the REAL binary, not
+ # the shim, so it is not counted as an agent call; it is timed separately.
+ lines += ['t_s0=$(date +%s.%N)', 'srx0=$(netrx)',
+ f"XERJ_HUB_URL={q(CAT_URL)} XERJ_HUB_PUBKEY={q(co['throwaway_pubkey'])} "
+ f"{q(prep['xerj'])} hub pull {q(slug)} > \"$S/setup.log\" 2>&1; echo $? > \"$S/setup.rc\"",
+ 'echo "$(date +%s.%N) $t_s0" > "$S/setup.time"', 'echo $(( $(netrx) - srx0 )) > "$S/setup.rx"']
+ lines += ['ls -1 "$HOME/.xerj/hub/packs" > "$S/packs_before.txt" 2>/dev/null || true',
+ 'cat /proc/loadavg > "$S/load_before"', 'rx0=$(netrx)', 't0=$(date +%s.%N)',
+ f'timeout {CELL_TIMEOUT_S} ' + " ".join(q(x) for x in args) +
+ f' "$(cat {q(str(prompt_file))})" < /dev/null > "$S/transcript.jsonl" 2> "$S/claude.err"',
+ 'rc=$?', 't1=$(date +%s.%N)', 'rx1=$(netrx)', 'cat /proc/loadavg > "$S/load_after"',
+ 'ls -1 "$HOME/.xerj/hub/packs" > "$S/packs_after.txt" 2>/dev/null || true',
+ 'echo "{\\"rc\\": $rc, \\"t0\\": $t0, \\"t1\\": $t1, \\"rx_iface_bytes\\": $((rx1-rx0))}" > "$S/timing.json"',
+ 'exit 0']
+ return "\n".join(lines) + "\n"
+
+
+def cmd_cell(a, prep=None):
+ prep = prep or load_prep()
+ arm = a.arm
+ if arm not in ARMS:
+ die(f"unknown arm {arm}")
+ refs = a.task
+ tasks = [load_task(r, a.allow_smoke_fixtures) for r in refs]
+ for t in tasks:
+ if t["corpus"] not in prep["packs"]:
+ die(f"corpus {t['corpus']} has no prepared v2 pack (prepared: {sorted(prep['packs'])})")
+ cdir = cell_dir(a.run_id, arm, refs, a.repeat)
+ if cdir.exists():
+ if not a.force:
+ print(f"skip (exists): {cdir}"); return
+ shutil.rmtree(cdir)
+ home = cdir / "home"
+ (home / ".sandbox").mkdir(parents=True)
+ max_turns = 1 if arm == "A0" else MAX_TURNS * len(tasks)
+ prompt = build_prompt(arm, tasks, prep)
+ # The prompt is staged OUTSIDE the jail HOME and exposed read-only, so the agent
+ # cannot see other cells; it holds no gold answer.
+ (cdir / "prompt.txt").write_text(prompt)
+ script = run_script(arm, cdir / "prompt.txt", prep, tasks, a.model, max_turns)
+ (cdir / "run.sh").write_text(script)
+ ro = [prep["claude"], str(cdir / "prompt.txt"), str(cdir / "run.sh")]
+ prefix = []
+ if ARMS[arm]["xerj"]:
+ prefix.append(str(Path(prep["shims"]) / f"xerj-{ARMS[arm]['xerj']}"))
+ ro.append(prep["xerj"])
+ if ARMS[arm]["git_noclone"]:
+ prefix.append(str(Path(prep["shims"]) / "git-noclone"))
+ if arm == "A5":
+ ro.append(prep["registry"])
+ if arm in V2_ARMS:
+ ro.append(prep["catalogue_override"]["site"])
+ cmd = ["bash", str(SANDBOX), "--home", str(home)]
+ for p in ro:
+ cmd += ["--ro", p]
+ cmd += ["--", "bash", str(cdir / "run.sh")]
+ env = dict(os.environ, XERJ_POV_PATH_PREFIX=":".join(prefix))
+ meta = {"schema": "hub-pov/cell/1", "run_id": a.run_id, "arm": arm, "role": ARMS[arm]["role"],
+ "tasks": refs, "repeat": a.repeat, "model_requested": a.model, "max_turns": max_turns,
+ "tools": ARMS[arm]["tools"], "path_prefix": prefix, "ro_paths": ro,
+ "started_at": now(), "host_load_before": Path("/proc/loadavg").read_text().split()[:3],
+ "prepared": {"xerj_version": prep["xerj_version"], "claude_version": prep["claude_version"],
+ "catalogue_override": prep["catalogue_override"]}}
+ print(f"[{now()}] cell {arm} {' '.join(refs)} r{a.repeat} -> {cdir}", flush=True)
+ t0 = time.time()
+ p = subprocess.run(cmd, env=env, capture_output=True, text=True, timeout=CELL_TIMEOUT_S + 600)
+ meta["sandbox_rc"] = p.returncode
+ meta["sandbox_wall_s"] = round(time.time() - t0, 2)
+ meta["host_load_after"] = Path("/proc/loadavg").read_text().split()[:3]
+ meta["finished_at"] = now()
+ (cdir / "sandbox.stderr").write_text(p.stderr)
+ write_json(cdir / "cell.json", meta)
+ if p.returncode != 0:
+ print(f" sandbox rc={p.returncode}: {p.stderr[-500:]}", file=sys.stderr)
+
+
+def cmd_matrix(a):
+ prep = load_prep()
+ m = load_json(Path(a.matrix))
+ for rep in range(1, m["repeats"] + 1):
+ for cell in m["cells"]:
+ for arm in m["arms"]:
+ ns = argparse.Namespace(arm=arm, task=cell if isinstance(cell, list) else [cell],
+ repeat=rep, run_id=a.run_id, model=m["model"],
+ allow_smoke_fixtures=m.get("allow_smoke_fixtures", False),
+ force=a.force)
+ cmd_cell(ns, prep)
+
+
+# --------------------------------------------------------------------------- parsing
+def read_events(path: Path) -> list[dict]:
+ out = []
+ if not path.exists():
+ return out
+ for line in path.read_text(errors="replace").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ out.append(json.loads(line))
+ except json.JSONDecodeError:
+ pass
+ return out
+
+
+def transcript_metrics(events: list[dict]) -> dict:
+ """PLAN §6: cost and output tokens are SUMMED over every `result` event."""
+ results = [e for e in events if e.get("type") == "result"]
+ init = next((e for e in events if e.get("type") == "system" and e.get("subtype") == "init"), {})
+ tool_uses, tool_results = [], {}
+ for e in events:
+ msg = e.get("message") or {}
+ content = msg.get("content") if isinstance(msg.get("content"), list) else []
+ if e.get("type") == "assistant":
+ for c in content:
+ if c.get("type") == "tool_use":
+ tool_uses.append({"id": c.get("id"), "name": c.get("name"), "input": c.get("input")})
+ elif e.get("type") == "user":
+ for c in content:
+ if c.get("type") == "tool_result":
+ body = c.get("content")
+ if isinstance(body, list):
+ body = "\n".join(x.get("text", "") for x in body if isinstance(x, dict))
+ tool_results[c.get("tool_use_id")] = {"text": str(body or ""), "is_error": bool(c.get("is_error"))}
+ final = results[-1].get("result", "") if results else ""
+ return {
+ "n_result_events": len(results),
+ "cost_usd": round(sum(float(r.get("total_cost_usd") or 0) for r in results), 6),
+ "output_tokens": sum(int((r.get("usage") or {}).get("output_tokens") or 0) for r in results),
+ "input_tokens_all": sum(sum(int((r.get("usage") or {}).get(k) or 0) for k in
+ ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens"))
+ for r in results),
+ "turns": sum(int(r.get("num_turns") or 0) for r in results),
+ "is_error": any(r.get("is_error") for r in results),
+ "result_subtypes": [r.get("subtype") for r in results],
+ "model": init.get("model"), "init_tools": init.get("tools"), "init_mcp": init.get("mcp_servers"),
+ "init_plugins": [p.get("name") for p in init.get("plugins") or []],
+ "tool_uses": tool_uses, "tool_results": tool_results, "final_text": final,
+ }
+
+
+BLOCK_RE = re.compile(r"captcha|security checkpoint|just a moment|attention required|access denied|"
+ r"cf-chl|too many requests|rate limit|\b(403|429)\b.{0,20}(forbidden|too many)|"
+ r"HTTP/[0-9.]+ (403|429)", re.I)
+
+
+def is_fetch(tu: dict) -> bool:
+ if tu["name"] in ("WebFetch", "WebSearch"):
+ return True
+ if tu["name"] == "Bash":
+ return bool(re.search(r"\b(curl|wget)\b", (tu.get("input") or {}).get("command", "")))
+ return False
+
+
+def failed_fetches(tm: dict) -> tuple[int, int]:
+ """(fetches, failed) — PLAN §6 'failed fetches'. HEURISTIC: a fetch whose result is
+ empty, an error, or looks like a bot-protection / 403 / 429 page. Flagged in SMOKE.md."""
+ n = failed = 0
+ for tu in tm["tool_uses"]:
+ if not is_fetch(tu):
+ continue
+ n += 1
+ r = tm["tool_results"].get(tu["id"], {"text": "", "is_error": True})
+ if r["is_error"] or not r["text"].strip() or BLOCK_RE.search(r["text"][:20000]):
+ failed += 1
+ return n, failed
+
+
+def bash_cmds(tm: dict) -> list[str]:
+ return [(tu.get("input") or {}).get("command", "") for tu in tm["tool_uses"] if tu["name"] == "Bash"]
+
+
+def xerj_calls(home: Path) -> list[dict]:
+ log = home / ".pov/xerj-calls.jsonl"
+ calls = []
+ if log.exists():
+ for line in log.read_text().splitlines():
+ try:
+ c = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ d = home / ".pov/calls"
+ c["out"] = (d / f"{c['n']}.out").read_text(errors="replace") if (d / f"{c['n']}.out").exists() else ""
+ c["err"] = (d / f"{c['n']}.err").read_text(errors="replace") if (d / f"{c['n']}.err").exists() else ""
+ calls.append(c)
+ return calls
+
+
+SUBCOMMANDS = {"ask", "code", "corpus", "hub", "init", "autoindex", "feedback", "mcp", "serve"}
+
+
+def classify_call(argv: list[str]) -> str:
+ if not argv:
+ return "none"
+ a0 = argv[0]
+ if a0 == "ask":
+ return "v2-ask"
+ if a0 == "hub":
+ return "v2-hub-" + (argv[1] if len(argv) > 1 else "")
+ if a0 == "code":
+ return "code-def" if "--def" in argv else "code"
+ if a0 == "corpus":
+ return "v1-corpus-" + (argv[1] if len(argv) > 1 else "")
+ if not a0.startswith("-") and " " in a0 and a0 not in SUBCOMMANDS:
+ return "v2-ask" # the bare `xerj "<question>"` form (spec §5)
+ if a0.startswith("-"):
+ return "node-or-flag"
+ return "other:" + a0
+
+
+def parse_answer_output(c: dict) -> dict:
+ """Routed corpus + answered_from from one v2 call's output (text or --json)."""
+ out = c["out"]
+ r = {"rc": c["rc"], "corpus": None, "answered_from": None, "candidates": []}
+ try:
+ j = json.loads(out)
+ if isinstance(j, dict):
+ r["corpus"] = j.get("corpus"); r["answered_from"] = j.get("answered_from")
+ return r
+ except (json.JSONDecodeError, ValueError):
+ pass
+ m = re.search(r"^answered from:\s*(.+)$", out, re.M)
+ if m:
+ r["answered_from"] = m.group(1).strip().replace(" ", "_")
+ m = re.match(r"\s*([a-z0-9][a-z0-9._-]*)\s+pin\s", out)
+ if m:
+ r["corpus"] = m.group(1)
+ if c["rc"] == 4:
+ r["candidates"] = re.findall(r"^\s*([a-z0-9][a-z0-9._-]*)\s+[—-]\s", out + "\n" + c["err"], re.M)
+ return r
+
+
+def routing(calls: list[dict], expected: str) -> dict:
+ """PLAN §2 H2 / §6: did the router pick the right corpus on the FIRST v2 question?
+ Confident-and-correct, or Ambiguous containing the expected slug, counts as correct;
+ a first call that passes --corpus bypassed routing (recorded, counted not-correct)."""
+ v2 = [c for c in calls if classify_call(c["argv"]) == "v2-ask"]
+ if not v2:
+ return {"status": "no_v2_call", "ok": False}
+ first = v2[0]
+ if "--corpus" in first["argv"]:
+ return {"status": "bypassed (--corpus on first call)", "ok": False}
+ p = parse_answer_output(first)
+ if first["rc"] == 4:
+ ok = expected in p["candidates"]
+ return {"status": "ambiguous", "candidates": p["candidates"], "ok": ok}
+ if p["corpus"]:
+ return {"status": "confident", "corpus": p["corpus"], "ok": p["corpus"] == expected}
+ text = first["out"] + "\n" + first["err"]
+ if "no hub corpus matches" in text:
+ # NoCorpus. The client prints "closest corpora: a, b; re-run with --corpus".
+ # PLAN H2 counts only Confident / Ambiguous-with-expected as correct; whether a
+ # NoCorpus whose closest list holds the expected slug should count is an open
+ # question for the user (SMOKE.md) — recorded, counted NOT ok.
+ m = re.search(r"closest corpora:\s*([^;\n]+)", text)
+ closest = [x.strip() for x in m.group(1).split(",")] if m else []
+ return {"status": "no_corpus", "closest": closest,
+ "closest_contains_expected": expected in closest, "ok": False}
+ return {"status": f"no_answer rc={first['rc']}", "ok": False}
+
+
+# --------------------------------------------------------------------------- grading
+def grade_c1(task: dict, answer_text: str) -> dict:
+ tf = RUN_DIR / "scratch" / "c1_task.json"; rf = RUN_DIR / "scratch" / "c1_run.json"
+ tf.parent.mkdir(parents=True, exist_ok=True)
+ tf.write_text(json.dumps(task)); rf.write_text(json.dumps({"result": answer_text}))
+ out = subprocess.run([sys.executable, str(HERE / "check.py"), str(tf), str(rf)],
+ capture_output=True, text=True, check=True).stdout
+ g = json.loads(out)
+ return {"grader": "check.py", "status": "pass" if g["pass"] else "fail", **g}
+
+
+JUDGE_PROMPT = """You are grading one answer against a fixed rubric. Use ONLY the gold answer and rubric as the authority.
+
+Question:
+{q}
+
+Gold answer:
+{gold}
+
+Rubric (each point must be judged met or not met):
+{rubric}
+
+Pass rule: {pass_rule} (\"all\" = every rubric point met).
+
+Answer to grade:
+<<<
+{answer}
+>>>
+
+Reply with ONLY a JSON object: {{"rubric_met": [true/false per rubric point, in order], "pass": true/false, "note": "<one sentence>"}}"""
+
+
+def grade_judge(task: dict, answer_text: str) -> dict:
+ """PLAN §6: separate inference, claude-opus-4-8, temperature 0; no key or any
+ error => `unscored` (never pass, never fail)."""
+ key = os.environ.get("ANTHROPIC_API_KEY")
+ if not key:
+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "unscored", "reason": "ANTHROPIC_API_KEY unset"}
+ body = {"model": JUDGE_MODEL, "max_tokens": 600, "temperature": 0,
+ "messages": [{"role": "user", "content": JUDGE_PROMPT.format(
+ q=task["q"], gold=task["gold"], rubric="\n".join(f"{i+1}. {r}" for i, r in enumerate(task["rubric"])),
+ pass_rule=task.get("pass_rule", "all"), answer=answer_text)}]}
+ req = urllib.request.Request("https://api.anthropic.com/v1/messages", data=json.dumps(body).encode(),
+ headers={"x-api-key": key, "anthropic-version": "2023-06-01",
+ "content-type": "application/json"})
+ try:
+ with urllib.request.urlopen(req, timeout=120) as r:
+ resp = json.loads(r.read())
+ text = "".join(c.get("text", "") for c in resp.get("content", []))
+ j = json.loads(re.search(r"\{.*\}", text, re.S).group(0))
+ met = [bool(x) for x in j["rubric_met"]]
+ if len(met) != len(task["rubric"]):
+ raise ValueError("rubric_met length mismatch")
+ rule_pass = all(met) if task.get("pass_rule", "all") == "all" else bool(j["pass"])
+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "pass" if rule_pass else "fail",
+ "rubric_met": met, "judge_pass_field": j.get("pass"), "note": j.get("note"),
+ "judge_usage": resp.get("usage")}
+ except (urllib.error.URLError, ValueError, KeyError, AttributeError, json.JSONDecodeError) as e:
+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "unscored", "reason": f"judge error: {e}"}
+
+
+def split_session_answers(text: str, ids: list[str]) -> dict[str, str]:
+ out = {}
+ for i in ids:
+ m = re.findall(rf"ANSWER\s*\[?{re.escape(i)}\]?\s*:\s*(.+)", text)
+ out[i] = ("ANSWER: " + m[-1]) if m else ""
+ return out
+
+
+# --------------------------------------------------------------------------- contamination
+GITHUB_SRC_RE = re.compile(r"(raw\.githubusercontent\.com|codeload\.github\.com|/archive/|/tarball/|/zipball/|"
+ r"github\.com/[^\s'\"]+/(blob|tree|raw)/)", re.I)
+CLONE_RE = re.compile(r"\bgit\s+(clone|fetch|pull)\b")
+ARCHIVE_RE = re.compile(r"(codeload\.github\.com|/archive/|/tarball/|/zipball/|\.tar\.gz|\.zip\b)", re.I)
+URL_HOST_RE = re.compile(r"https?://([A-Za-z0-9.-]+)")
+
+
+def contamination(arm: str, tm: dict, calls: list[dict], home: Path, expected_slug: str) -> dict:
+ """Did the arm have ONLY its intended tools, and did it use the thing under test?
+ Returns violations (hard: the cell is invalid) and notes (soft: reported)."""
+ v, notes = [], []
+ declared = [t for t in ARMS[arm]["tools"].split(",") if t]
+ used = sorted({tu["name"] for tu in tm["tool_uses"]})
+ if tm["init_tools"] is not None and sorted(tm["init_tools"]) != sorted(declared):
+ v.append(f"init tools {tm['init_tools']} != declared {declared}")
+ if tm["init_mcp"]:
+ v.append(f"MCP servers present: {tm['init_mcp']}")
+ extra = [u for u in used if u not in declared]
+ if extra:
+ v.append(f"used undeclared tools {extra}")
+ cmds = "\n".join(bash_cmds(tm))
+ kinds = [classify_call(c["argv"]) for c in calls]
+ if arm == "A0" and tm["tool_uses"]:
+ v.append("memory arm used tools")
+ if arm in ("A0", "A1", "A2", "A3", "A4"):
+ if calls or re.search(r"(^|[\s;&|(])xerj\b", cmds):
+ v.append("non-XERJ arm invoked xerj")
+ if arm != "A0" and not tm["tool_uses"]:
+ notes.append("answered with no tool call at all (model recall, not the arm's tools)")
+ elif arm == "A1" and not CLONE_RE.search(cmds):
+ notes.append("git arm never cloned/fetched (answered without the repo)")
+ # A3/A4 are defined by their ONE endpoint: using the shell for other fetches
+ # without ever calling it makes the cell an A2 cell (seen in the smoke: an A4
+ # agent went straight to raw.githubusercontent.com).
+ if arm == "A3" and tm["tool_uses"] and "sourcegraph.com" not in cmds:
+ v.append("A3 never called the code-search endpoint")
+ if arm == "A4" and tm["tool_uses"]:
+ if "duckduckgo.com" not in cmds:
+ v.append("A4 never called the web-search API (fetched pages directly)")
+ else:
+ other = sorted({h for h in URL_HOST_RE.findall(cmds)} - {"html.duckduckgo.com", "duckduckgo.com"})
+ if other:
+ notes.append(f"A4 fetched pages from {other} (allowed only as search results; check order)")
+ if arm in ("A2", "A3", "A4"):
+ refused = home / ".pov/git-refused.log"
+ if CLONE_RE.search(cmds) or (refused.exists() and refused.read_text().strip()):
+ v.append("git clone/fetch attempted in a no-git arm")
+ if ARCHIVE_RE.search(cmds):
+ v.append("repository archive download in a no-git arm")
+ if arm == "A3":
+ hosts = {h for h in URL_HOST_RE.findall(cmds)} - {"sourcegraph.com"}
+ if hosts:
+ v.append(f"A3 fetched hosts other than the code-search endpoint: {sorted(hosts)}")
+ if any(tu["name"] in ("WebFetch", "WebSearch") for tu in tm["tool_uses"]):
+ v.append("A3 used web tools")
+ if arm in XERJ_ARMS:
+ if CLONE_RE.search(cmds):
+ v.append("XERJ arm ran git clone/fetch itself")
+ if GITHUB_SRC_RE.search(cmds):
+ v.append("XERJ arm fetched source from GitHub directly")
+ if arm == "A5":
+ if any(k.startswith("v2-") for k in kinds):
+ v.append("v1 arm used the v2 surface (ask / hub)")
+ if "v1-corpus-add" not in kinds:
+ notes.append("v1 arm never ran `xerj corpus add`")
+ if not any(k.startswith("code") for k in kinds):
+ notes.append("v1 arm never ran `xerj code`")
+ if arm in V2_ARMS:
+ if any(k.startswith("v1-corpus-add") or k.startswith("v1-corpus-index") for k in kinds):
+ v.append("v2 arm used the v1 corpus add/index path")
+ answers = [parse_answer_output(c) for c in calls if classify_call(c["argv"]) in ("v2-ask", "code-def", "code")]
+ froms = [x["answered_from"] for x in answers if x["answered_from"]]
+ if not any(k == "v2-ask" for k in kinds):
+ v.append("v2 arm made no `xerj \"<q>\"`/`xerj ask` call (a pass would be model recall, PLAN §3)")
+ elif "local_pack" not in froms:
+ v.append(f"no v2 answer came from the local pack (answered_from seen: {froms})")
+ before = (home / ".sandbox/packs_before.txt")
+ before_l = before.read_text().split() if before.exists() else []
+ if arm == "A6" and before_l:
+ v.append(f"A6 was not cold: packs cached before the agent ran: {before_l}")
+ if arm == "A7" and not any(expected_slug in x for x in before_l):
+ v.append(f"A7 was not warm: {expected_slug} pack not cached before the agent ran ({before_l})")
+ return {"violations": v, "notes": notes, "tools_used": used, "xerj_call_kinds": kinds,
+ "n_bash": len(bash_cmds(tm))}
+
+
+# --------------------------------------------------------------------------- score
+def dir_bytes(p: Path) -> int:
+ if not p.exists():
+ return 0
+ return sum(f.stat().st_size for f in p.rglob("*") if f.is_file() and not f.is_symlink())
+
+
+def net_counters(home: Path) -> dict:
+ f = home / ".sandbox/net.json"
+ if not f.exists():
+ return {}
+ out = {}
+ for item in load_json(f).get("nftables", []):
+ c = item.get("counter")
+ if c:
+ out[c["name"]] = c["bytes"]
+ return out
+
+
+def score_cell(cdir: Path) -> dict | None:
+ meta_f = cdir / "cell.json"
+ if not meta_f.exists():
+ return None
+ meta = load_json(meta_f)
+ home = cdir / "home"
+ sb = home / ".sandbox"
+ tm = transcript_metrics(read_events(sb / "transcript.jsonl"))
+ calls = xerj_calls(home)
+ tasks = [load_task(r, True) for r in meta["tasks"]]
+ slug = tasks[0]["corpus"]
+ timing = load_json(sb / "timing.json") if (sb / "timing.json").exists() else {}
+ wall = round(timing["t1"] - timing["t0"], 2) if timing else None
+ net = net_counters(home)
+ model_bytes = net.get("rx_model_api", 0)
+ rx_iface = timing.get("rx_iface_bytes")
+ # bytes downloaded (PLAN §6): everything the jail received while the agent ran,
+ # minus model-API traffic (no setup traffic precedes the agent except A7's pull,
+ # which is reported separately and excluded by the interface delta).
+ bytes_dl = max(0, rx_iface - model_bytes) if rx_iface is not None else None
+ setup = None
+ if (sb / "setup.time").exists():
+ t1s, t0s = map(float, (sb / "setup.time").read_text().split())
+ setup = {"rc": int((sb / "setup.rc").read_text().strip() or 255), "seconds": round(t1s - t0s, 2),
+ "rx_bytes": int((sb / "setup.rx").read_text().strip() or 0) if (sb / "setup.rx").exists() else None,
+ "log_tail": (sb / "setup.log").read_text(errors="replace")[-400:]}
+ pack_bytes_cached = dir_bytes(home / ".xerj/hub/packs")
+ n_fetch, n_failed = failed_fetches(tm)
+ # grading
+ if len(tasks) == 1:
+ per_task = {tasks[0]["task"]["id"]: tm["final_text"]}
+ else:
+ per_task = split_session_answers(tm["final_text"], [t["task"]["id"] for t in tasks])
+ grades = []
+ for t in tasks:
+ ans = per_task.get(t["task"]["id"], "")
+ g = grade_c1(t["task"], ans) if t["class"] in ("C1",) or "re" in t["task"] else grade_judge(t["task"], ans)
+ g["ref"] = t["ref"]
+ grades.append(g)
+ cont = contamination(meta["arm"], tm, calls, home, slug)
+ route = routing(calls, slug) if meta["arm"] in V2_ARMS else None
+ # H2: a mis-route counts the v2 answer wrong.
+ for g in grades:
+ g["status_raw"] = g["status"]
+ if route is not None and not route["ok"] and g["status"] == "pass":
+ g["status"] = "fail"; g["misroute_override"] = True
+ statuses = [g["status"] for g in grades]
+ correct = None if "unscored" in statuses else sum(s == "pass" for s in statuses) / len(statuses)
+ row = {
+ "schema": "hub-pov/score/1", "run_id": meta["run_id"], "arm": meta["arm"], "tasks": meta["tasks"],
+ "repeat": meta["repeat"], "model": tm["model"], "model_requested": meta["model_requested"],
+ "valid": not cont["violations"] and tm["n_result_events"] >= 1 and timing.get("rc") == 0,
+ "correct": correct, "grades": grades,
+ "cost_usd": tm["cost_usd"], "output_tokens": tm["output_tokens"], "input_tokens_all": tm["input_tokens_all"],
+ "turns": tm["turns"], "n_result_events": tm["n_result_events"], "claude_rc": timing.get("rc"),
+ "result_subtypes": tm["result_subtypes"],
+ "wall_s": wall, "load_before": (sb / "load_before").read_text().split()[:3] if (sb / "load_before").exists() else None,
+ "load_after": (sb / "load_after").read_text().split()[:3] if (sb / "load_after").exists() else None,
+ "bytes_downloaded": bytes_dl, "bytes_rx_iface": rx_iface, "bytes_model_api": model_bytes,
+ "bytes_rx_jail_total": net.get("rx_all"), "pack_bytes_cached": pack_bytes_cached,
+ "fetches": n_fetch, "failed_fetches": n_failed,
+ "xerj_calls": len(calls), "routing": route, "setup": setup,
+ "answered_from": [x for x in (parse_answer_output(c)["answered_from"] for c in calls
+ if classify_call(c["argv"]) in ("v2-ask", "code", "code-def")) if x],
+ "contamination": cont, "final_answer_tail": tm["final_text"][-300:],
+ }
+ write_json(cdir / "score.json", row)
+ return row
+
+
+def cells_of(run_id: str) -> list[Path]:
+ root = RUN_DIR / "results" / run_id
+ return sorted(p.parent for p in root.rglob("cell.json"))
+
+
+def cmd_score(a):
+ rows = [r for r in (score_cell(c) for c in cells_of(a.run_id)) if r]
+ out = RUN_DIR / "results" / a.run_id / "scores.jsonl"
+ out.write_text("".join(json.dumps(r) + "\n" for r in rows))
+ print(f"scored {len(rows)} cells -> {out}")
+
+
+# --------------------------------------------------------------------------- report
+def fmt(x, nd=3):
+ if x is None:
+ return "–"
+ if isinstance(x, float):
+ return f"{x:.{nd}f}"
+ return str(x)
+
+
+def cmd_report(a):
+ rows = [json.loads(l) for l in (RUN_DIR / "results" / a.run_id / "scores.jsonl").read_text().splitlines()]
+ hdr = ("| arm | task(s) | rep | valid | correct | grade | cost $ | out tok | turns | wall s | load1 | "
+ "bytes dl | fetches (failed) | xerj calls | routing | answered_from | violations / notes |")
+ print(hdr)
+ print("|" + "---|" * (hdr.count("|") - 1))
+ for r in sorted(rows, key=lambda r: (r["tasks"], r["arm"], r["repeat"])):
+ g = ",".join(x["status"] + ("*" if x.get("misroute_override") else "") for x in r["grades"])
+ froms = ",".join(sorted(set(r.get("answered_from") or []))) or "–"
+ cont = r["contamination"]
+ vio = "; ".join(cont["violations"] + [f"note: {n}" for n in cont["notes"]]) or "clean"
+ route = r["routing"]["status"] + ("/ok" if r["routing"]["ok"] else "/WRONG") if r["routing"] else "–"
+ print(f"| {r['arm']} | {' '.join(r['tasks'])} | {r['repeat']} | {'yes' if r['valid'] else 'NO'} | "
+ f"{fmt(r['correct'], 2)} | {g} | {fmt(r['cost_usd'], 4)} | {r['output_tokens']} | {r['turns']} | "
+ f"{fmt(r['wall_s'], 1)} | {(r['load_before'] or ['–'])[0]} | {fmt(r['bytes_downloaded'])} | "
+ f"{r['fetches']} ({r['failed_fetches']}) | {r['xerj_calls']} | {route} | {froms} | {vio} |")
+ gate(rows)
+
+
+def gate(rows):
+ """PLAN §7, checked only on a complete matrix with >= 3 repeats per cell."""
+ reps = {}
+ for r in rows:
+ reps.setdefault((r["arm"], tuple(r["tasks"])), []).append(r)
+ if not reps or min(len(v) for v in reps.values()) < 3:
+ print("\n§7 gate: NOT EVALUATED — fewer than 3 repeats per cell (PLAN §6/§7). "
+ "These numbers are not results.")
+ return
+ print("\n§7 gate: evaluation over a complete >=3-repeat matrix is implemented in "
+ "gate_eval(); run it on the frozen full matrix only.")
+ for (arm, tasks), v in sorted(reps.items()):
+ for metric in ("cost_usd", "output_tokens"):
+ xs = [x[metric] for x in v]
+ m = statistics.mean(xs)
+ cv = statistics.pstdev(xs) / m if m else 0
+ if cv > 0.5:
+ print(f" CV>{0.5}: {arm} {tasks} {metric} cv={cv:.2f} -> needs +2 repeats (PLAN §6)")
+
+
+# --------------------------------------------------------------------------- verify-c1
+def cmd_verify_c1(a):
+ src = load_json(TASKS / "C1/SOURCE.json")
+ ok = True
+ for rel, blob in src["files"].items():
+ local = TASKS / "C1" / Path(rel).name if rel.startswith("tasks/") else HERE / Path(rel).name
+ h = subprocess.run(["git", "hash-object", str(local)], capture_output=True, text=True).stdout.strip()
+ try:
+ remote = subprocess.run(["git", "-C", str(REPO), "rev-parse",
+ f"origin/{src['source_branch']}:benchmarks/corpus-tasks/{rel}"],
+ capture_output=True, text=True, check=True).stdout.strip()
+ except subprocess.CalledProcessError:
+ remote = "(branch not fetched)"
+ good = h == blob == remote
+ ok &= good
+ print(f"{'ok ' if good else 'BAD'} {rel}: local={h[:12]} recorded={blob[:12]} branch={remote[:12]}")
+ sys.exit(0 if ok else 1)
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ sub = ap.add_subparsers(dest="cmd", required=True)
+ p = sub.add_parser("prepare"); p.add_argument("--xerj", required=True)
+ p.add_argument("--registry", required=True, help="the corpus-hub registry dir (tools/xerj-code/hub)")
+ p.add_argument("--registry-commit", default=None, help="the corpus-hub commit the registry came from")
+ p.add_argument("--tags", default="pack-lz4-0774d055,pack-snappy-9c28114a,pack-zstd-01b7154f")
+ p.set_defaults(fn=cmd_prepare)
+ p = sub.add_parser("cell"); p.add_argument("--arm", required=True)
+ p.add_argument("--task", required=True, action="append", help="<class>:<suite>:<id>; repeat for a session")
+ p.add_argument("--repeat", type=int, default=1); p.add_argument("--run-id", required=True)
+ p.add_argument("--model", required=True); p.add_argument("--allow-smoke-fixtures", action="store_true")
+ p.add_argument("--force", action="store_true"); p.set_defaults(fn=cmd_cell)
+ p = sub.add_parser("matrix"); p.add_argument("matrix"); p.add_argument("--run-id", required=True)
+ p.add_argument("--force", action="store_true"); p.set_defaults(fn=cmd_matrix)
+ p = sub.add_parser("score"); p.add_argument("--run-id", required=True); p.set_defaults(fn=cmd_score)
+ p = sub.add_parser("report"); p.add_argument("--run-id", required=True); p.set_defaults(fn=cmd_report)
+ p = sub.add_parser("verify-c1"); p.set_defaults(fn=cmd_verify_c1)
+ a = ap.parse_args()
+ a.fn(a)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/hub-pov/harness/shim_git_noclone.sh b/benchmarks/hub-pov/harness/shim_git_noclone.sh
new file mode 100755
index 000000000..d28c8f4ef
--- /dev/null
+++ b/benchmarks/hub-pov/harness/shim_git_noclone.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# `git` shim for arm A2 (normal agent + web ONLY): PLAN.md §3 says A2 "may not
+# `git clone` the corpus source". Refuses every verb that fetches a repository and
+# logs the attempt to $HOME/.pov/git-refused.log (the contamination check reads it);
+# anything else is passed to the real git. A2's prompt states the rule, so a refusal
+# is the agent ignoring it, not a trap.
+case " $* " in
+ *" clone "*|*" fetch "*|*" pull "*|*" ls-remote "*|*" submodule "*|*" archive "*--remote*)
+ mkdir -p "$HOME/.pov"; echo "$(date -u +%FT%TZ) $*" >> "$HOME/.pov/git-refused.log"
+ echo "git: fetching repositories is not allowed in this environment (web only)" >&2
+ exit 128 ;;
+esac
+exec /usr/bin/git "$@"
diff --git a/benchmarks/hub-pov/harness/shim_xerj.sh b/benchmarks/hub-pov/harness/shim_xerj.sh
new file mode 100755
index 000000000..4bf2a6493
--- /dev/null
+++ b/benchmarks/hub-pov/harness/shim_xerj.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Logging `xerj` shim for the XERJ arms (PLAN.md §3 note: "invoke xerj through a
+# logging PATH shim that counts invocations"). Generated per run by `pov.py prepare`
+# from this template; the placeholders are filled in there.
+#
+# Every call appends one JSON line to $HOME/.pov/xerj-calls.jsonl and keeps the
+# call's stdout/stderr in $HOME/.pov/calls/<n>.{out,err} so the scorer can read
+# `answered from:` / `answered_from` and the routed corpus (PLAN.md §6 routing).
+# The v2 shim also points the client at the catalogue override (SMOKE.md: the live
+# hub catalogue is not deployed) — those two exports are the ONLY behaviour change.
+@HUB_EXPORTS@
+REAL='@REAL@'
+d="$HOME/.pov/calls"; mkdir -p "$d"
+n=$(( $(ls "$d" 2>/dev/null | grep -c '\.rc$') + 1 ))
+t0=$(date +%s.%N)
+# stdout/stderr are teed, not captured, so the agent sees exactly what it would see.
+# A daemon the call spawns (the hub node) is started detached with its own log, so it
+# does not hold the tee pipes open.
+{ "$REAL" "$@" 2> >(tee "$d/$n.err" >&2); echo $? > "$d/$n.rc"; } | tee "$d/$n.out"
+rc=$(cat "$d/$n.rc" 2>/dev/null || echo 255)
+t1=$(date +%s.%N)
+python3 -c 'import json,sys; print(json.dumps({"n":int(sys.argv[1]),"argv":sys.argv[5:],"rc":int(sys.argv[2]),"t0":float(sys.argv[3]),"t1":float(sys.argv[4])}))' \
+ "$n" "$rc" "$t0" "$t1" "$@" >> "$HOME/.pov/xerj-calls.jsonl"
+exit "$rc"
diff --git a/benchmarks/hub-pov/harness/test_pov.py b/benchmarks/hub-pov/harness/test_pov.py
new file mode 100644
index 000000000..9fb1c28fc
--- /dev/null
+++ b/benchmarks/hub-pov/harness/test_pov.py
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+"""Offline tests for pov.py's parsing, grading and contamination logic (no agent,
+no network): `python3 -m unittest benchmarks/hub-pov/harness/test_pov.py`."""
+import json
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+os.environ.setdefault("POV_RUN_DIR", tempfile.mkdtemp(prefix="pov-test-"))
+import pov # noqa: E402
+
+
+def ev_tool(name, tid, **inp):
+ return {"type": "assistant", "message": {"content": [{"type": "tool_use", "id": tid, "name": name, "input": inp}]}}
+
+
+def ev_res(tid, text, err=False):
+ return {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": tid, "content": text, "is_error": err}]}}
+
+
+def result(cost, out, turns, text="ANSWER: 15"):
+ return {"type": "result", "subtype": "success", "total_cost_usd": cost, "num_turns": turns,
+ "usage": {"output_tokens": out, "input_tokens": 10}, "result": text}
+
+
+INIT = {"type": "system", "subtype": "init", "model": "claude-sonnet-5-5", "tools": ["Bash", "Read", "Grep", "Glob"],
+ "mcp_servers": [], "plugins": []}
+
+
+class Metrics(unittest.TestCase):
+ def test_sums_every_result_event(self):
+ # PLAN §6 summing lesson: a background task emits several result events.
+ tm = pov.transcript_metrics([INIT, result(0.01, 100, 3, "x"), result(0.02, 50, 2, "ANSWER: 8")])
+ self.assertEqual(tm["n_result_events"], 2)
+ self.assertAlmostEqual(tm["cost_usd"], 0.03)
+ self.assertEqual(tm["output_tokens"], 150)
+ self.assertEqual(tm["turns"], 5)
+ self.assertEqual(tm["final_text"], "ANSWER: 8")
+
+ def test_failed_fetch_heuristic(self):
+ evs = [INIT, ev_tool("Bash", "1", command="curl -s https://grep.app/api"),
+ ev_res("1", "<title>Vercel Security Checkpoint</title>"),
+ ev_tool("Bash", "2", command="curl -s https://example.org"), ev_res("2", "hello"),
+ ev_tool("WebFetch", "3", url="https://x"), ev_res("3", "", err=True),
+ ev_tool("Bash", "4", command="ls"), ev_res("4", "")]
+ self.assertEqual(pov.failed_fetches(pov.transcript_metrics(evs)), (3, 2))
+
+
+class Calls(unittest.TestCase):
+ def test_classify(self):
+ c = pov.classify_call
+ self.assertEqual(c(["what is the exact value of kInputMarginBytes in snappy"]), "v2-ask")
+ self.assertEqual(c(["ask", "q"]), "v2-ask")
+ self.assertEqual(c(["hub", "pull", "lz4"]), "v2-hub-pull")
+ self.assertEqual(c(["corpus", "add", "--from", "x.json"]), "v1-corpus-add")
+ self.assertEqual(c(["code", "lz4", "frame checksum"]), "code")
+ self.assertEqual(c(["--insecure", "--port", "9200"]), "node-or-flag")
+
+ def text_call(self, out, rc=0, argv=None):
+ return {"argv": argv or ["how does snappy x"], "rc": rc, "out": out, "err": ""}
+
+ def test_parse_text_answer(self):
+ out = "snappy pin 2026-10-02 (15 days old) BSD-3-Clause snappy.cc:12\n...\nanswered from: local pack\n"
+ p = pov.parse_answer_output(self.text_call(out))
+ self.assertEqual((p["corpus"], p["answered_from"]), ("snappy", "local_pack"))
+
+ def test_parse_json_answer(self):
+ p = pov.parse_answer_output(self.text_call(json.dumps({"corpus": "lz4", "answered_from": "local_pack"})))
+ self.assertEqual((p["corpus"], p["answered_from"]), ("lz4", "local_pack"))
+
+ def test_routing(self):
+ ok = self.text_call("lz4 pin 2026-10-02 (15 days old) x a.c:1\nanswered from: local pack\n")
+ self.assertTrue(pov.routing([ok], "lz4")["ok"])
+ self.assertFalse(pov.routing([ok], "zstd")["ok"])
+ amb = self.text_call("lz4 — an agent ...\nzstd — an agent ...\n", rc=4)
+ self.assertTrue(pov.routing([amb], "zstd")["ok"])
+ byp = self.text_call("lz4 pin x", argv=["ask", "q q", "--corpus", "lz4"])
+ self.assertFalse(pov.routing([byp], "lz4")["ok"])
+ self.assertEqual(pov.routing([], "lz4")["status"], "no_v2_call")
+
+
+class Grading(unittest.TestCase):
+ def test_c1_uses_registered_checker(self):
+ task = {"id": "T4", "re": ["0xf|15"], "need": 1}
+ self.assertEqual(pov.grade_c1(task, "blah\nANSWER: 15")["status"], "pass")
+ self.assertEqual(pov.grade_c1(task, "ANSWER: unknown")["status"], "fail")
+
+ def test_judge_without_key_is_unscored(self):
+ old = os.environ.pop("ANTHROPIC_API_KEY", None)
+ try:
+ g = pov.grade_judge({"q": "q", "gold": "g", "rubric": ["a"]}, "ANSWER: x")
+ self.assertEqual(g["status"], "unscored")
+ finally:
+ if old is not None:
+ os.environ["ANTHROPIC_API_KEY"] = old
+
+ def test_session_split(self):
+ s = pov.split_session_answers("ANSWER T1: 4\nANSWER [T2]: 10\n", ["T1", "T2", "T3"])
+ self.assertEqual(s, {"T1": "ANSWER: 4", "T2": "ANSWER: 10", "T3": ""})
+
+
+class Contamination(unittest.TestCase):
+ def tm(self, tools, *evs):
+ init = dict(INIT, tools=tools)
+ return pov.transcript_metrics([init, *evs, result(0.01, 1, 1)])
+
+ def test_web_arm_clone_is_a_violation(self):
+ tm = self.tm(["Bash", "Read", "Grep", "Glob", "WebFetch", "WebSearch"],
+ ev_tool("Bash", "1", command="git clone --depth 1 https://github.com/google/snappy"))
+ c = pov.contamination("A2", tm, [], Path(tempfile.mkdtemp()), "snappy")
+ self.assertTrue(any("clone" in v for v in c["violations"]))
+
+ def test_git_arm_using_xerj_is_a_violation(self):
+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("Bash", "1", command="xerj 'how does x'"))
+ c = pov.contamination("A1", tm, [], Path(tempfile.mkdtemp()), "snappy")
+ self.assertTrue(any("xerj" in v for v in c["violations"]))
+
+ def test_v2_arm_without_local_pack_answer_is_a_violation(self):
+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("Bash", "1", command="xerj 'how does x y'"))
+ call = {"argv": ["how does x y"], "rc": 1, "out": "", "err": "no hub corpus matches"}
+ c = pov.contamination("A6", tm, [call], Path(tempfile.mkdtemp()), "snappy")
+ self.assertTrue(any("local pack" in v for v in c["violations"]))
+
+ def test_undeclared_tool_is_a_violation(self):
+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("WebFetch", "1", url="https://x"))
+ c = pov.contamination("A1", tm, [], Path(tempfile.mkdtemp()), "snappy")
+ self.assertTrue(any("undeclared" in v for v in c["violations"]))
+
+ def test_search_arm_that_skips_its_endpoint_is_a_violation(self):
+ tm = self.tm(["Bash", "Read", "Grep", "Glob"],
+ ev_tool("Bash", "1", command="curl -s https://raw.githubusercontent.com/google/snappy/x/snappy.cc"))
+ c = pov.contamination("A4", tm, [], Path(tempfile.mkdtemp()), "snappy")
+ self.assertTrue(any("never called the web-search API" in v for v in c["violations"]))
+
+ def test_memory_arm_clean(self):
+ tm = self.tm([])
+ c = pov.contamination("A0", tm, [], Path(tempfile.mkdtemp()), "snappy")
+ self.assertEqual(c["violations"], [])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/benchmarks/hub-pov/matrix/smoke.json b/benchmarks/hub-pov/matrix/smoke.json
new file mode 100644
index 000000000..4cadb9cb7
--- /dev/null
+++ b/benchmarks/hub-pov/matrix/smoke.json
@@ -0,0 +1,8 @@
+{
+ "_comment": "SMOKE ONLY (2026-10-17): 1 repeat x 2 tasks x every arm, cheap model. Validates the harness; its numbers are NOT results (SMOKE.md). C2:lz4.smoke:S1 is a smoke fixture, not a pre-registered task.",
+ "model": "sonnet",
+ "repeats": 1,
+ "allow_smoke_fixtures": true,
+ "arms": ["A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7"],
+ "cells": ["C1:snappy:T4", "C2:lz4.smoke:S1"]
+}
diff --git a/benchmarks/hub-pov/results/schema.json b/benchmarks/hub-pov/results/schema.json
new file mode 100644
index 000000000..1ed71bd28
--- /dev/null
+++ b/benchmarks/hub-pov/results/schema.json
@@ -0,0 +1,272 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "hub-pov/score/1",
+ "title": "One scored cell (arm x task(s) x repeat) of the Corpus Hub v2 proof-of-value run",
+ "description": "Written by `pov.py score` to <run>/results/<run_id>/<arm>/<cell>/r<n>/score.json and appended to scores.jsonl. Metric definitions: PLAN.md \u00a76.",
+ "type": "object",
+ "required": [
+ "schema",
+ "run_id",
+ "arm",
+ "tasks",
+ "repeat",
+ "valid",
+ "correct",
+ "grades",
+ "cost_usd",
+ "output_tokens",
+ "turns",
+ "n_result_events",
+ "wall_s",
+ "load_before",
+ "bytes_downloaded",
+ "fetches",
+ "failed_fetches",
+ "xerj_calls",
+ "routing",
+ "contamination"
+ ],
+ "properties": {
+ "schema": {
+ "const": "hub-pov/score/1"
+ },
+ "run_id": {
+ "type": "string"
+ },
+ "arm": {
+ "enum": [
+ "A0",
+ "A1",
+ "A2",
+ "A3",
+ "A4",
+ "A5",
+ "A6",
+ "A7"
+ ]
+ },
+ "tasks": {
+ "type": "array",
+ "items": {
+ "type": "string",
+ "pattern": "^C[1-4]:[^:]+:[^:]+$"
+ },
+ "minItems": 1,
+ "description": "one ref = single-question cell; >1 = a C4 session in one agent run"
+ },
+ "repeat": {
+ "type": "integer",
+ "minimum": 1
+ },
+ "model": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "description": "resolved model id from the transcript init event"
+ },
+ "model_requested": {
+ "type": "string"
+ },
+ "valid": {
+ "type": "boolean",
+ "description": "no contamination violation, >=1 result event, claude exit 0. An invalid cell is excluded from comparisons and reported."
+ },
+ "correct": {
+ "type": [
+ "number",
+ "null"
+ ],
+ "minimum": 0,
+ "maximum": 1,
+ "description": "fraction of the cell's tasks passed; null when any grade is unscored (PLAN \u00a76)"
+ },
+ "grades": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "required": [
+ "ref",
+ "grader",
+ "status"
+ ],
+ "properties": {
+ "ref": {
+ "type": "string"
+ },
+ "grader": {
+ "type": "string"
+ },
+ "status": {
+ "enum": [
+ "pass",
+ "fail",
+ "unscored"
+ ]
+ },
+ "status_raw": {
+ "enum": [
+ "pass",
+ "fail",
+ "unscored"
+ ]
+ },
+ "misroute_override": {
+ "type": "boolean",
+ "description": "H2: a pass turned into fail because the v2 router picked the wrong corpus"
+ }
+ }
+ }
+ },
+ "cost_usd": {
+ "type": "number",
+ "description": "sum of total_cost_usd over ALL result events"
+ },
+ "output_tokens": {
+ "type": "integer",
+ "description": "sum of usage.output_tokens over ALL result events"
+ },
+ "input_tokens_all": {
+ "type": "integer"
+ },
+ "turns": {
+ "type": "integer",
+ "description": "sum of num_turns over result events"
+ },
+ "n_result_events": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "claude_rc": {
+ "type": [
+ "integer",
+ "null"
+ ]
+ },
+ "wall_s": {
+ "type": [
+ "number",
+ "null"
+ ],
+ "description": "the agent process only (A7's pre-pull is in setup)"
+ },
+ "load_before": {
+ "type": [
+ "array",
+ "null"
+ ],
+ "items": {
+ "type": "string"
+ },
+ "description": "1/5/15-min load average at agent start"
+ },
+ "load_after": {
+ "type": [
+ "array",
+ "null"
+ ],
+ "items": {
+ "type": "string"
+ }
+ },
+ "bytes_downloaded": {
+ "type": [
+ "integer",
+ "null"
+ ],
+ "description": "bytes received by the jail's interface while the agent ran, minus model-API bytes (nft counter on 160.79.104.0/21)"
+ },
+ "bytes_rx_iface": {
+ "type": [
+ "integer",
+ "null"
+ ]
+ },
+ "bytes_model_api": {
+ "type": "integer"
+ },
+ "bytes_rx_jail_total": {
+ "type": [
+ "integer",
+ "null"
+ ]
+ },
+ "pack_bytes_cached": {
+ "type": "integer",
+ "description": "size of ~/.xerj/hub/packs at the end (A6 pull / A7 pre-pull)"
+ },
+ "fetches": {
+ "type": "integer"
+ },
+ "failed_fetches": {
+ "type": "integer",
+ "description": "HEURISTIC: empty, error, captcha/bot page, 403/429 (PLAN \u00a76)"
+ },
+ "xerj_calls": {
+ "type": "integer",
+ "description": "invocations seen by the logging shim"
+ },
+ "answered_from": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
+ "routing": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "description": "A6/A7 only: first v2 question's routing (H2)",
+ "properties": {
+ "status": {
+ "type": "string"
+ },
+ "ok": {
+ "type": "boolean"
+ }
+ }
+ },
+ "setup": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "description": "A7 only: the pre-pull (rc, seconds, rx_bytes) \u2014 excluded from bytes_downloaded, reported for the first-use disclosure (PLAN \u00a77)"
+ },
+ "contamination": {
+ "type": "object",
+ "required": [
+ "violations",
+ "notes",
+ "tools_used"
+ ],
+ "properties": {
+ "violations": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
+ "notes": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
+ "tools_used": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ },
+ "xerj_call_kinds": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/benchmarks/hub-pov/sandbox.sh b/benchmarks/hub-pov/sandbox.sh
index f1c36571e..1f46f86a4 100755
--- a/benchmarks/hub-pov/sandbox.sh
+++ b/benchmarks/hub-pov/sandbox.sh
@@ -12,18 +12,46 @@
# we bind writable, so an arm cannot modify the host, the repo, or another arm.
# * --die-with-parent tears the jail down if the harness dies.
# Every `claude -p` arm (A0-A7 in PLAN.md) MUST be launched through this script.
-# Network is deliberately NOT unshared — arms need it (git clone, web fetch, the
-# hub pull, the search APIs). Isolation here is of the filesystem and the process
-# table, not the network.
+#
+# WHAT THE 2026-10-17 SMOKE RUN ADDED (SMOKE.md "harness defects"), each one a way
+# an arm could leak, be contaminated, or touch the host that the first version left
+# open:
+# * PRIVATE NETWORK NAMESPACE (default `--net isolated`). The jail runs inside
+# `pasta` (user-mode networking: the arm keeps internet access) with NO port
+# forwarding in either direction and no gateway->host-loopback mapping, plus an
+# nftables rule rejecting RFC1918/CGNAT/link-local destinations. Without this an
+# arm shares the host's loopback: `xerj ask` probes http://localhost:9200 FIRST
+# and uses whatever answers — on this host that is the user's reference-coding
+# node, so a v2 arm would have restored a pack INTO the user's node and answered
+# from its corpora (contamination and a write to a node we must never touch).
+# Inside the namespace :9200 is the arm's own, so XERJ arms can bind it safely.
+# * /home and /root are MASKED (tmpfs); only the HOME, the auth dir and the paths
+# passed with --ro are visible. With a bare `--ro-bind / /` every arm could read
+# the gold answers (benchmarks/corpus-tasks, prior run dirs), the local corpus
+# clones in ~/.xerj-code, other sessions' transcripts, and secrets in .env files.
+# * The CLAUDE_CONFIG_DIR is bound writable (token refresh) but everything in it
+# except `.credentials.json` and `.claude.json` is masked: the user's global
+# CLAUDE.md, skills, plugins, hooks, transcripts and history would otherwise be
+# loaded into or readable by every arm, and settings.json `env` would inject the
+# user's GH_TOKEN into the arm despite --clearenv. settings.json is replaced by `{}`.
+# * Inbound bytes are counted per jail (nft counters): all non-loopback bytes, and
+# the subset from the model API (Anthropic, 160.79.104.0/21) so the harness can
+# report "bytes downloaded" net of model traffic. Written to
+# $HOME/.sandbox/net.json when the jail exits.
#
# USAGE:
-# benchmarks/hub-pov/sandbox.sh [--home DIR] [--rm-home] -- CMD [ARGS...]
+# benchmarks/hub-pov/sandbox.sh [--home DIR] [--rm-home] [--ro PATH]...
+# [--net isolated|host] -- CMD [ARGS...]
# --home DIR use DIR as the arm's fresh HOME (default: a new `mktemp -d`).
# Must NOT be the real user's home; the script refuses that.
# --rm-home delete the HOME on exit (default: keep it so the harness can
# collect the transcript/artifacts, then clean up itself).
-# Everything after `--` is the command run inside the jail, e.g.
-# sandbox.sh --home "$cell" -- claude -p --output-format stream-json "$prompt"
+# --ro PATH expose a host PATH read-only at the same path (repeatable). Needed
+# for anything under /home (the claude binary, the xerj binary, shim
+# dirs, the catalogue override). NEVER pass a dir holding gold answers.
+# --net MODE `isolated` (default; needs pasta + nft) or `host` (the old shared
+# network — only for debugging the jail itself, never for an arm).
+# Everything after `--` is the command run inside the jail.
# The resolved HOME is printed to stderr as `sandbox-home: <dir>` for the harness.
#
# ENV (optional, read from the host before the jail clears the environment):
@@ -31,8 +59,8 @@
# Bound writable because the CLI refreshes its token in place;
# do NOT copy .credentials.json into the HOME instead — a copy
# can refresh out from under the main session and log it out.
-# XERJ_POV_PATH_PREFIX dir prepended to PATH inside the jail (e.g. the logging
-# `xerj` shim that counts invocations for the XERJ arms).
+# XERJ_POV_PATH_PREFIX dir(s, ':'-separated) prepended to PATH inside the jail (e.g.
+# the logging `xerj` shim). Each is exposed read-only.
# No host secret is forwarded: the jail starts with --clearenv and we set back only
# HOME, CLAUDE_CONFIG_DIR, PATH, TERM, LANG. GH_TOKEN / ANTHROPIC_API_KEY are NOT
# passed in — arms authenticate via CLAUDE_CONFIG_DIR, and the LLM judge runs
@@ -44,12 +72,19 @@ XERJ_POV_PATH_PREFIX="${XERJ_POV_PATH_PREFIX:-}"
SANDBOX_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
HOME_DIR=""
RM_HOME=0
+NET=isolated
+IN_NETNS=0
+RO_PATHS=()
+ORIG_ARGS=("$@")
# ---- parse args ----------------------------------------------------------
while [ $# -gt 0 ]; do
case "$1" in
--home) HOME_DIR="${2:?--home needs a dir}"; shift 2 ;;
--rm-home) RM_HOME=1; shift ;;
+ --ro) RO_PATHS+=("${2:?--ro needs a path}"); shift 2 ;;
+ --net) NET="${2:?--net needs isolated|host}"; shift 2 ;;
+ --_in-netns) IN_NETNS=1; shift ;; # internal: re-exec'd inside pasta
--) shift; break ;;
*) echo "sandbox.sh: unknown arg '$1' (did you forget '--' before the command?)" >&2; exit 2 ;;
esac
@@ -57,6 +92,7 @@ done
if [ $# -eq 0 ]; then
echo "sandbox.sh: no command after '--'" >&2; exit 2
fi
+case "$NET" in isolated|host) ;; *) echo "sandbox.sh: --net must be isolated or host" >&2; exit 2 ;; esac
# ---- preconditions (fail closed) -----------------------------------------
command -v bwrap >/dev/null 2>&1 || { echo "sandbox.sh: bwrap (bubblewrap) not found — refusing to run an arm unsandboxed" >&2; exit 3; }
@@ -70,19 +106,62 @@ HOME_DIR="$(cd "$HOME_DIR" && pwd -P)" # absolute, resolved
# Guard: the HOME must be a fresh dir, never the real home or the filesystem root.
# Binding the real home writable would defeat the whole point of --ro-bind / /.
case "$HOME_DIR" in
- / | /home/claude | /home/claude/ | /root | /root/ | "${HOME:-/nonexistent}")
+ / | /home | /home/ | /home/claude | /home/claude/ | /root | /root/ | "${HOME:-/nonexistent}")
echo "sandbox.sh: refusing to use '$HOME_DIR' as a scratch HOME (not a fresh dir)" >&2; exit 3 ;;
esac
-mkdir -p "$HOME_DIR/work"
+
+# ---- network namespace (re-exec under pasta) ------------------------------
+if [ "$NET" = isolated ] && [ "$IN_NETNS" -eq 0 ]; then
+ command -v pasta >/dev/null 2>&1 || { echo "sandbox.sh: pasta not found — refusing to run an arm on the host network (use --net host only to debug the jail)" >&2; exit 3; }
+ command -v nft >/dev/null 2>&1 || { echo "sandbox.sh: nft not found — refusing to run an arm without the private-range block" >&2; exit 3; }
+ # -t/-u none: nothing forwarded host->ns. -T/-U none: nothing forwarded ns->host
+ # loopback (pasta's default `auto` would forward every port the host has bound on
+ # 127.0.0.1, i.e. exactly the reference node). --no-map-gw: the gateway address
+ # does not lead to the host either.
+ # pasta puts us in a user namespace as uid 0; remember the real ids so the jail
+ # maps back to them (claude refuses --dangerously-skip-permissions as root).
+ export SANDBOX_HOST_UID="$(id -u)" SANDBOX_HOST_GID="$(id -g)"
+ exec pasta --config-net -q -t none -u none -T none -U none --no-map-gw \
+ -- bash "$(realpath "$0")" --_in-netns --home "$HOME_DIR" "${ORIG_ARGS[@]}"
+fi
+
+mkdir -p "$HOME_DIR/work" "$HOME_DIR/.sandbox"
echo "sandbox-home: $HOME_DIR" >&2
+if [ "$IN_NETNS" -eq 1 ]; then
+ # We are root in pasta's user+net namespace: block private destinations (another
+ # route to a host service bound on 0.0.0.0, e.g. via docker0) and count bytes.
+ nft -f - <<'EOF'
+table inet povjail {
+ counter rx_all {}
+ counter rx_model_api {}
+ chain out {
+ type filter hook output priority 0; policy accept;
+ oifname "lo" accept
+ ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 } reject
+ ip6 daddr { fc00::/7, fe80::/10 } reject
+ }
+ chain in {
+ type filter hook input priority 0; policy accept;
+ iifname "lo" accept
+ counter name rx_all
+ ip saddr 160.79.104.0/21 counter name rx_model_api
+ }
+}
+EOF
+ # The host's resolv.conf points at systemd-resolved on 127.0.0.53, which is the
+ # HOST loopback and unreachable from here: give the jail public resolvers.
+ printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > "$HOME_DIR/.sandbox/resolv.conf"
+fi
+
PATH_IN="$SANDBOX_PATH"
[ -n "$XERJ_POV_PATH_PREFIX" ] && PATH_IN="$XERJ_POV_PATH_PREFIX:$PATH_IN"
# ---- the jail ------------------------------------------------------------
-# ro-bind the whole host read-only; carve out exactly two writable spots:
+# ro-bind the whole host read-only; mask /home and /root; carve out:
# - the fresh HOME (the arm's whole writable world, cwd = $HOME/work)
-# - the auth/config dir (token refresh writes here; kept as narrow as possible)
+# - the auth/config dir (token refresh writes here), with its contents masked
+# - the --ro paths and the PATH-prefix dirs, read-only
# Fresh tmpfs /tmp and minimal /dev /proc; --unshare-pid for the private PID ns;
# --clearenv so no host env (secrets, GH_TOKEN) leaks in, then set back the minimum.
BWRAP=(bwrap
@@ -90,9 +169,35 @@ BWRAP=(bwrap
--dev /dev
--proc /proc
--tmpfs /tmp
- --bind "$HOME_DIR" "$HOME_DIR"
- --bind "$CLAUDE_CONFIG_DIR" "$CLAUDE_CONFIG_DIR"
- --unshare-pid
+ --tmpfs /home
+ --tmpfs /root)
+if [ "$IN_NETNS" -eq 1 ]; then
+ BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/resolv.conf" "$(realpath /etc/resolv.conf)")
+fi
+BWRAP+=(--bind "$HOME_DIR" "$HOME_DIR"
+ --bind "$CLAUDE_CONFIG_DIR" "$CLAUDE_CONFIG_DIR")
+# Mask the config dir's contents except the two files auth needs.
+printf '{}\n' > "$HOME_DIR/.sandbox/settings.json"
+: > "$HOME_DIR/.sandbox/empty" # a regular empty file: a /dev/null bind reads as EACCES here
+for entry in "$CLAUDE_CONFIG_DIR"/* "$CLAUDE_CONFIG_DIR"/.[!.]*; do
+ [ -e "$entry" ] || continue
+ case "$(basename "$entry")" in
+ .credentials.json|.claude.json) continue ;;
+ settings.json) BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/settings.json" "$entry") ;;
+ *) if [ -d "$entry" ] && [ ! -L "$entry" ]; then BWRAP+=(--tmpfs "$entry")
+ elif [ -f "$entry" ] && [ ! -L "$entry" ]; then BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/empty" "$entry"); fi ;;
+ esac
+done
+IFS=: read -r -a prefix_dirs <<< "$XERJ_POV_PATH_PREFIX"
+for p in "${RO_PATHS[@]}" "${prefix_dirs[@]}"; do
+ [ -n "$p" ] || continue
+ [ -e "$p" ] || { echo "sandbox.sh: --ro path does not exist: $p" >&2; exit 2; }
+ BWRAP+=(--ro-bind "$p" "$p")
+done
+if [ "$IN_NETNS" -eq 1 ]; then
+ BWRAP+=(--unshare-user --uid "${SANDBOX_HOST_UID:?}" --gid "${SANDBOX_HOST_GID:?}")
+fi
+BWRAP+=(--unshare-pid
--die-with-parent
--new-session
--chdir "$HOME_DIR/work"
@@ -103,11 +208,17 @@ BWRAP=(bwrap
--setenv TERM xterm
--setenv LANG C.UTF-8)
-# When --rm-home is set we must run bwrap as a child (not exec) so the EXIT trap
-# that deletes the HOME can fire; otherwise exec for clean signal propagation.
-if [ "$RM_HOME" -eq 1 ]; then
- trap 'rm -rf "$HOME_DIR"' EXIT
+write_counters() {
+ [ "$IN_NETNS" -eq 1 ] || return 0
+ nft -j list counters table inet povjail > "$HOME_DIR/.sandbox/net.json" 2>/dev/null || true
+}
+
+# Run bwrap as a child (not exec) when we must act after it: delete the HOME
+# (--rm-home) or read the namespace's byte counters.
+if [ "$RM_HOME" -eq 1 ] || [ "$IN_NETNS" -eq 1 ]; then
+ [ "$RM_HOME" -eq 1 ] && trap 'rm -rf "$HOME_DIR"' EXIT
rc=0; "${BWRAP[@]}" "$@" || rc=$?
+ write_counters
exit "$rc"
else
exec "${BWRAP[@]}" "$@"
diff --git a/benchmarks/hub-pov/tasks/C1/SOURCE.json b/benchmarks/hub-pov/tasks/C1/SOURCE.json
new file mode 100644
index 000000000..53dd2db7d
--- /dev/null
+++ b/benchmarks/hub-pov/tasks/C1/SOURCE.json
@@ -0,0 +1,11 @@
+{
+ "source_branch": "bench/corpus-tasks",
+ "source_commit": "0ca4065a56ae82dba9eb03599063e0ddc277228a",
+ "note": "Verbatim copies of the registered C1 suites (PLAN.md \u00a74 C1: reuse, do not invent). `pov.py verify-c1` re-checks each blob against the branch.",
+ "files": {
+ "tasks/lz4.json": "dbdaef8170594162f43f0fc3d20be44774f7a8e5",
+ "tasks/snappy.json": "4cac10708240eda2cb665ea93c72d33ca94858eb",
+ "tasks/zstd.json": "18ed1f7da441ab7bae0750c8571dad545ec3c105",
+ "harness/check.py": "7d55ff398c2cc9ead5016cf9df88af44a9aa8305"
+ }
+}
\ No newline at end of file
diff --git a/benchmarks/hub-pov/tasks/C1/lz4.json b/benchmarks/hub-pov/tasks/C1/lz4.json
new file mode 100644
index 000000000..dbdaef817
--- /dev/null
+++ b/benchmarks/hub-pov/tasks/C1/lz4.json
@@ -0,0 +1,144 @@
+{
+ "corpus": "lz4",
+ "subject": "lz4",
+ "pin": "0774d05537",
+ "generated": "2026-10-02",
+ "tasks": [
+ {
+ "id": "T1",
+ "class": "drift",
+ "name": "LZ4F_ENDMARK_SIZE",
+ "kind": "added",
+ "changed": "2024-07-24",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of LZ4F_ENDMARK_SIZE? Answer with the value only.",
+ "re": [
+ "0x4|4"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T2",
+ "class": "drift",
+ "name": "LZ4_VERSION_MINOR",
+ "kind": "changed",
+ "changed": "2024-07-14",
+ "olds": [
+ "9"
+ ],
+ "q": "In the current development source of lz4, what is the exact value of LZ4_VERSION_MINOR? Answer with the value only.",
+ "re": [
+ "0xa|10"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T3",
+ "class": "drift",
+ "name": "LZ4_VERSION_RELEASE",
+ "kind": "changed",
+ "changed": "2024-07-14",
+ "olds": [
+ "5"
+ ],
+ "q": "In the current development source of lz4, what is the exact value of LZ4_VERSION_RELEASE? Answer with the value only.",
+ "re": [
+ "0|0x0"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T4",
+ "class": "drift",
+ "name": "ENV_CLEVEL",
+ "kind": "added",
+ "changed": "2024-07-08",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of ENV_CLEVEL? Answer with the value only.",
+ "re": [
+ "LZ4_CLEVEL"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T5",
+ "class": "drift",
+ "name": "ENV_NBTHREADS",
+ "kind": "added",
+ "changed": "2024-07-06",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of ENV_NBTHREADS? Answer with the value only.",
+ "re": [
+ "LZ4_NBWORKERS"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T6",
+ "class": "drift",
+ "name": "OUTBUFF_QUEUE",
+ "kind": "added",
+ "changed": "2024-07-05",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of OUTBUFF_QUEUE? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T7",
+ "class": "drift",
+ "name": "NB_BUFFSETS",
+ "kind": "added",
+ "changed": "2024-07-05",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of NB_BUFFSETS? Answer with the value only.",
+ "re": [
+ "0x4|4"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T8",
+ "class": "drift",
+ "name": "DISTRIB_SIZE_MAX",
+ "kind": "added",
+ "changed": "2024-02-20",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of DISTRIB_SIZE_MAX? Answer with the value only.",
+ "re": [
+ "0x28a|650"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T9",
+ "class": "drift",
+ "name": "LZ4MID_HASHSIZE",
+ "kind": "added",
+ "changed": "2024-02-07",
+ "olds": [],
+ "q": "In the current development source of lz4, what is the exact value of LZ4MID_HASHSIZE? Answer with the value only.",
+ "re": [
+ "0x8|8"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T10",
+ "class": "drift",
+ "name": "LZ4HC_CLEVEL_MIN",
+ "kind": "changed",
+ "changed": "2024-02-07",
+ "olds": [
+ "3"
+ ],
+ "q": "In the current development source of lz4, what is the exact value of LZ4HC_CLEVEL_MIN? Answer with the value only.",
+ "re": [
+ "0x2|2"
+ ],
+ "need": 1
+ }
+ ]
+}
diff --git a/benchmarks/hub-pov/tasks/C1/snappy.json b/benchmarks/hub-pov/tasks/C1/snappy.json
new file mode 100644
index 000000000..4cac10708
--- /dev/null
+++ b/benchmarks/hub-pov/tasks/C1/snappy.json
@@ -0,0 +1,127 @@
+{
+ "corpus": "snappy",
+ "subject": "snappy",
+ "pin": "9c28114a38",
+ "generated": "2026-10-02",
+ "tasks": [
+ {
+ "id": "T1",
+ "class": "drift",
+ "name": "SNAPPY_MIN_COMPRESSION_LEVEL",
+ "kind": "added",
+ "changed": "2026-09-09",
+ "olds": [],
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_MIN_COMPRESSION_LEVEL? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T2",
+ "class": "drift",
+ "name": "SNAPPY_MAX_COMPRESSION_LEVEL",
+ "kind": "added",
+ "changed": "2026-09-09",
+ "olds": [],
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_MAX_COMPRESSION_LEVEL? Answer with the value only.",
+ "re": [
+ "0x2|2"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T3",
+ "class": "drift",
+ "name": "SNAPPY_HAVE_RVV",
+ "kind": "changed",
+ "changed": "2026-05-09",
+ "olds": [
+ "0"
+ ],
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_RVV? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T4",
+ "class": "drift",
+ "name": "kInputMarginBytes",
+ "kind": "added",
+ "changed": "2024-04-04",
+ "olds": [],
+ "q": "In the current development source of snappy, what is the exact value of kInputMarginBytes? Answer with the value only.",
+ "re": [
+ "0xf|15"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T5",
+ "class": "drift",
+ "name": "kMaxHashTableBits",
+ "kind": "changed",
+ "changed": "2024-04-04",
+ "olds": [
+ "14"
+ ],
+ "q": "In the current development source of snappy, what is the exact value of kMaxHashTableBits? Answer with the value only.",
+ "re": [
+ "0xf|15"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T6",
+ "class": "state",
+ "name": "SNAPPY_HAVE_VECTOR_BYTE_SHUFFLE",
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_VECTOR_BYTE_SHUFFLE? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T7",
+ "class": "state",
+ "name": "SNAPPY_HAVE_NEON_CRC32",
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_NEON_CRC32? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T8",
+ "class": "state",
+ "name": "SNAPPY_HAVE_X86_CRC32",
+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_X86_CRC32? Answer with the value only.",
+ "re": [
+ "0x1|1"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T9",
+ "class": "state",
+ "name": "kMinHashTableBits",
+ "q": "In the current development source of snappy, what is the exact value of kMinHashTableBits? Answer with the value only.",
+ "re": [
+ "0x8|8"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T10",
+ "class": "state",
+ "name": "kMaximumTagLength",
+ "q": "In the current development source of snappy, what is the exact value of kMaximumTagLength? Answer with the value only.",
+ "re": [
+ "0x5|5"
+ ],
+ "need": 1
+ }
+ ]
+}
diff --git a/benchmarks/hub-pov/tasks/C1/zstd.json b/benchmarks/hub-pov/tasks/C1/zstd.json
new file mode 100644
index 000000000..18ed1f7da
--- /dev/null
+++ b/benchmarks/hub-pov/tasks/C1/zstd.json
@@ -0,0 +1,138 @@
+{
+ "corpus": "zstd",
+ "subject": "zstd",
+ "pin": "01b7154f11",
+ "generated": "2026-10-02",
+ "tasks": [
+ {
+ "id": "T1",
+ "class": "drift",
+ "name": "X86_FEATURE_BMI2",
+ "kind": "added",
+ "changed": "2026-09-14",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of X86_FEATURE_BMI2? Answer with the value only.",
+ "re": [
+ "0|0x0"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T2",
+ "class": "drift",
+ "name": "FRAME_V01_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V01_SIZE? Answer with the value only.",
+ "re": [
+ "0xbd|189"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T3",
+ "class": "drift",
+ "name": "FRAME_V02_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V02_SIZE? Answer with the value only.",
+ "re": [
+ "0xbb|187"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T4",
+ "class": "drift",
+ "name": "FRAME_V03_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V03_SIZE? Answer with the value only.",
+ "re": [
+ "0xbb|187"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T5",
+ "class": "drift",
+ "name": "FRAME_V04_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V04_SIZE? Answer with the value only.",
+ "re": [
+ "0xc6|198"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T6",
+ "class": "drift",
+ "name": "FRAME_V05_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V05_SIZE? Answer with the value only.",
+ "re": [
+ "0xb8|184"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T7",
+ "class": "drift",
+ "name": "FRAME_V06_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V06_SIZE? Answer with the value only.",
+ "re": [
+ "0xb2|178"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T8",
+ "class": "drift",
+ "name": "FRAME_V07_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V07_SIZE? Answer with the value only.",
+ "re": [
+ "0xb2|178"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T9",
+ "class": "drift",
+ "name": "FRAME_V08_SIZE",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of FRAME_V08_SIZE? Answer with the value only.",
+ "re": [
+ "0xb3|179"
+ ],
+ "need": 1
+ },
+ {
+ "id": "T10",
+ "class": "drift",
+ "name": "OLDEST_STREAMABLE_VERSION",
+ "kind": "added",
+ "changed": "2026-09-11",
+ "olds": [],
+ "q": "In the current development source of zstd, what is the exact value of OLDEST_STREAMABLE_VERSION? Answer with the value only.",
+ "re": [
+ "0x4|4"
+ ],
+ "need": 1
+ }
+ ]
+}
diff --git a/benchmarks/hub-pov/tasks/C2/lz4.smoke.json b/benchmarks/hub-pov/tasks/C2/lz4.smoke.json
new file mode 100644
index 000000000..a60c435dd
--- /dev/null
+++ b/benchmarks/hub-pov/tasks/C2/lz4.smoke.json
@@ -0,0 +1,32 @@
+{
+ "class": "C2",
+ "corpus": "lz4",
+ "subject": "lz4",
+ "pin": "0774d05537f9762f838f7ab541b7765f1a729cb5",
+ "pin_date": "2026-10-02",
+ "smoke_only": true,
+ "provenance": "SMOKE FIXTURE ONLY — NOT a pre-registered C2 task. Written by the harness agent on 2026-10-17 to exercise the C2 judge path end to end; PLAN.md §5 requires C2 tasks to be authored by a person from the pinned source and committed before the first C2 run, so this file must never be used in the full run (pov.py refuses smoke_only tasks unless --allow-smoke-fixtures is passed).",
+ "tasks": [
+ {
+ "id": "S1",
+ "q": "In the lz4 frame format, which integrity checks can a frame carry, which of them are optional, what does each one cover, and how is each computed? Answer concisely.",
+ "gold": "Three checks, all xxHash-32 with seed 0. (1) Header checksum: always present, one byte at the end of the frame descriptor = (xxh32(descriptor) >> 8) & 0xFF, i.e. the second byte of the xxh32 of the full frame descriptor including optional fields. (2) Block checksum: optional (FLG block-checksum flag, bit 4); 4 bytes little-endian after each data block, xxh32 of the raw (compressed/undecoded) block data. (3) Content checksum: optional (FLG content-checksum flag, bit 2); 4 bytes after the EndMark, xxh32 of the whole decompressed content.",
+ "rubric": [
+ "Names xxHash-32 (xxh32) as the checksum function for the checks",
+ "Header (descriptor) checksum: always present, one byte, derived from xxh32 of the frame descriptor (second byte / >>8)",
+ "Block checksum: optional, per data block, computed over the raw/compressed block data",
+ "Content checksum: optional, appended after the end mark, computed over the entire decompressed content"
+ ],
+ "pass_rule": "all",
+ "files": [
+ "doc/lz4_Frame_format.md:91-104 (Content Checksum)",
+ "doc/lz4_Frame_format.md:164-183 (block / content checksum flags)",
+ "doc/lz4_Frame_format.md:243-249 (Header Checksum)",
+ "doc/lz4_Frame_format.md:294-303 (Block checksum)",
+ "lib/lz4frame.c:351-355 (LZ4F_headerChecksum)",
+ "lib/lz4frame.c:918-923 (block checksum over compressed data)",
+ "lib/lz4frame.c:1116 (content checksum on compressEnd)"
+ ]
+ }
+ ]
+}