AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
bench(hub-pov): runnable harness for the Hub v2 proof of value + smoke run (#1722)
xerj-org0143feac6276
diff --git a/benchmarks/hub-pov/SMOKE.md b/benchmarks/hub-pov/SMOKE.mdnew file mode 100644index 000000000..8af91b121--- /dev/null+++ b/benchmarks/hub-pov/SMOKE.md@@ -0,0 +1,213 @@+# Hub v2 proof of value: harness smoke run (2026-10-17)++> **These smoke numbers are NOT results.** They come from 1 repeat, 2 tasks (one of+> them a smoke fixture rather than a pre-registered task), a cheap model, a heavily+> loaded machine and a locally signed catalogue. They exist only to show that the+> harness runs and measures what [PLAN.md](./PLAN.md) says it measures. Do not quote+> them, compare arms with them, or publish them. The §7 gate was not evaluated+> (`pov.py report` refuses with fewer than 3 repeats).++## What ran++| | |+|---|---|+| Matrix | [`matrix/smoke.json`](./matrix/smoke.json): every arm A0–A7 × 2 tasks × 1 repeat = 16 cells, run serially |+| Tasks | `C1:snappy:T4` (registered suite, `kInputMarginBytes`, mechanical `check.py`); `C2:lz4.smoke:S1` (smoke-only fixture, LLM judge) |+| Agent | `claude -p` 2.1.302, `--model sonnet` (resolved: `claude-sonnet-5-5`), `--max-turns 15` (A0: 1) |+| xerj | `v1.0.0-rc.96` built from `main` at `b3753ddb` |+| Packs | the real releases `pack-lz4-0774d055`, `pack-snappy-9c28114a`, `pack-zstd-01b7154f`. Each archive's released `.sig` verified against the compiled-in `HUB_PUBKEY_HEX` before use. |+| Sandbox | every `claude -p` ran through [`sandbox.sh`](./sandbox.sh): bwrap, own PID and user namespace, private network namespace, `/home` masked |+| Run dirs | `/home/claude/hubdev-wt/pov-run/results/smoke-20261017b/` (the valid pass); `smoke-20261017/` is the first pass, kept because it exposed defects 8 and 9 |+| Spend | $0.40 for the agents across 16 cells; 241 s of agent wall time; about 5 min end to end |+| Host load (1-min) | **25 to 113** throughout. This is not a quiet machine, so every wall time below is invalid as a timing (PLAN §6). |++### The catalogue override (the live hub catalogue is not deployed)++`https://hub.xerj.org/api/v1/catalogue.json` returns the site's HTML page. The smoke run+therefore used the override that `tools/hub-packs/consumer_e2e.sh` uses:+`xerj hub build-catalogue` builds the catalogue and it is served over HTTP as+`<hub>/api/v1/catalogue.json` + `.sig`, with `XERJ_HUB_URL` pointing at it. Two+differences from the CI e2e, both forced by running locally:++1. **Throwaway signing key.** The hub seed is a CI secret, so the catalogue is signed+ with a throwaway ed25519 key, and the client is pointed at that key with+ `XERJ_HUB_PUBKEY`. This is the local-rehearsal override `consumer_e2e.sh` documents.+ The packs' signatures are re-made with the same key, because there is only one+ client key. The pack **bytes** are the released ones: `pack_url` points at the+ GitHub release, so the v2 arms download real bytes over the real network, and+ `pack_sha256` in the catalogue pins them. Their genuine release signatures were+ verified separately (see "Packs" above).+2. **The catalogue is served inside each jail** (`python3 -m http.server` on the jail's+ own `127.0.0.1:18080`), because the jail's network namespace cannot reach the host+ loopback (defect 1). The re-made `.sig` files are served from there too+ (`sig_url` is rewritten, then the catalogue is re-signed).++Both exports live only in the v2 `xerj` shim (`XERJ_HUB_URL`, `XERJ_HUB_PUBKEY`).+**Before the full run, switch to the deployed, really-signed catalogue** and drop both+from the shim.++## Smoke table (run `smoke-20261017b`; NOT results)++| arm | task | valid | grade | cost $ | out tok | turns | wall s* | bytes dl | xerj calls | routing (H2) | contamination check |+|---|---|---|---|---|---|---|---|---|---|---|---|+| A0 | C1 snappy T4 | yes | pass | 0.0137 | 406 | 1 | 6.3 | 15,824 | 0 | – | clean |+| A1 | C1 snappy T4 | yes | pass | 0.0168 | 264 | 2 | 7.2 | 1,183,155 | 0 | – | clean (fetched the pin with git) |+| A2 | C1 snappy T4 | yes | pass | 0.0173 | 268 | 2 | 6.8 | 151,260 | 0 | – | clean (raw.githubusercontent) |+| A3 | C1 snappy T4 | yes | pass | 0.0191 | 224 | 2 | 6.7 | 25,092 | 0 | – | clean (Sourcegraph only) |+| A4 | C1 snappy T4 | **NO** | pass | 0.0187 | 441 | 2 | 7.3 | 158,487 | 0 | – | **violation: never called the search API, fetched raw GitHub directly** |+| A5 | C1 snappy T4 | yes | pass | 0.0413 | 549 | 4 | 43.3 | 1,203,400 | 3 | – | clean (`corpus add` → `index` → `code`) |+| A6 | C1 snappy T4 | yes | fail† | 0.0293 | 413 | 4 | 21.3 | 11,467,819 | 4 | no_corpus → wrong | clean; pulled snappy (11.3 MB); answered_from local_pack |+| A7 | C1 snappy T4 | yes | fail† | 0.0294 | 525 | 4 | 14.3 | 18,262 | 4 | no_corpus → wrong | clean; pack pre-pulled (12.0 s setup); answered_from local_pack |+| A0 | C2 lz4 S1 | yes | unscored‡ | 0.0161 | 633 | 1 | 7.6 | 16,878 | 0 | – | clean |+| A1 | C2 lz4 S1 | yes | unscored‡ | 0.0215 | 994 | 1 | 9.7 | 18,422 | 0 | – | note: answered with no tool call |+| A2 | C2 lz4 S1 | yes | unscored‡ | 0.0217 | 941 | 1 | 9.0 | 18,544 | 0 | – | note: answered with no tool call |+| A3 | C2 lz4 S1 | yes | unscored‡ | 0.0215 | 988 | 1 | 9.8 | 19,372 | 0 | – | note: answered with no tool call |+| A4 | C2 lz4 S1 | yes | unscored‡ | 0.0200 | 850 | 1 | 9.4 | 17,739 | 0 | – | note: answered with no tool call |+| A5 | C2 lz4 S1 | yes | unscored‡ | 0.0527 | 1054 | 4 | 37.2 | 488,952 | 3 | – | clean |+| A6 | C2 lz4 S1 | yes | unscored‡ | 0.0316 | 898 | 3 | 31.3 | 4,700,587 | 2 | no_corpus → wrong | clean; pulled lz4 (4.6 MB); local_pack |+| A7 | C2 lz4 S1 | yes | unscored‡ | 0.0309 | 929 | 3 | 14.3 | 19,579 | 2 | no_corpus → wrong | clean; pre-pulled (15.3 s); local_pack |++\* Loaded machine: timings are invalid (PLAN §6). † The raw `check.py` grade was a+pass. PLAN H2 turns it into a fail because the first v2 question did not route (see+below). ‡ `ANTHROPIC_API_KEY` is not set on this host, so every C2 cell is `unscored`,+exactly as PLAN §6 requires.++### What the smoke run verified++- **Each arm had only its intended tools.** The init event's `tools` matches the+ declared set in every cell: A0 has none, A2 adds WebFetch/WebSearch, and the rest+ have Bash/Read/Grep/Glob. MCP servers and plugins: none. Model: `claude-sonnet-5-5`.+ An independent grep over all 16 raw transcripts found no `xerj` call outside A5–A7,+ no `git` outside A1, no clone or archive download in A2–A7, and only Sourcegraph in+ A3. The check caught one real case of contamination: the A4 C1 cell, which is marked+ invalid.+- **The v2 arms really used the pack.** Both A6 and A7 got `answered from: local pack`+ in every cell. A6 started cold (`packs_before = []`) and its pull shows up in the+ bytes (11,467,819 B against the 11,338,487 B snappy pack; 4,700,587 B against the+ 4,623,814 B lz4 pack). A7 started warm (`packs_before = [snappy]` / `[lz4]`), with+ the pre-pull timed separately (`setup.rx` 11,472,275 / 4,671,184 B). Its downloads+ while the agent ran were only about 18 KB of baseline noise. No v2 arm touched+ `corpus add` or `corpus index`, and the v1 arm never touched `ask` or `hub`.+- **Metrics are captured** for every cell: correct (or unscored), cost and output tokens+ (summed over every `result` event, with ≥1 event asserted), turns, bytes downloaded+ (the jail interface delta minus Anthropic-range bytes from an nft counter), wall+ time with 1/5/15 load before and after, fetches and failed fetches, xerj calls,+ routing, A7 setup time and bytes. Schema: [`results/schema.json`](./results/schema.json).++### Product observations (seen while validating; NOT results, not measured)++These fall under H2 (routing). They are listed so nobody is surprised by them in the full run:++1. **The router returned NoCorpus for every first v2 question in the smoke run (4/4)**,+ for example `xerj "In snappy, what is the exact value of kInputMarginBytes?"` →+ `no hub corpus matches … closest corpora: snappy, lz4` (exit 1). Every agent then+ re-asked with `--corpus`. With only 3 corpora in the catalogue this is the sketch+ scoring (§5 step 3: tokens present / tokens total, ×0.8): ordinary question words+ dilute the score below 0.6.+2. **Def mode missed `kInputMarginBytes`**: `ask --corpus snappy "what is the exact+ value of kInputMarginBytes"` exited 1. It is a function-local `const` in+ `snappy.cc`, so it is not a def record. A plain how-query then found it.+3. **First use on a cold machine:** in a no-agent dry run at load ~50, the+ auto-started node did not report healthy within the 60 s wait, so the first `ask`+ failed. The second one worked. A6 may hit this on a loaded runner.++## Harness defects found and fixed++| # | Defect | Effect if unfixed | Fix |+|---|---|---|---|+| 1 | `sandbox.sh` shared the **host network**. `xerj ask` probes `localhost:9200` first and uses whatever answers. | On this host, :9200 is the user's reference-coding node, so a v2 arm would have restored packs **into the user's node** and answered from its corpora. That is both contamination and a write to a node we must never touch. An A5 agent's `corpus add/index` would have hit it too. | Private network namespace via `pasta` (`-t/-u/-T/-U none --no-map-gw`: nothing forwarded either way), plus an nft rule rejecting RFC1918/CGNAT/link-local destinations, which closes the route through `docker0` (10.200.0.1). Verified: a host server on 127.0.0.1 and on 0.0.0.0 is unreachable from the jail, while github.com and the API are reachable. |+| 2 | `--ro-bind / /` exposed all of `/home`. | Arms could read the gold answers (`benchmarks/corpus-tasks`, the 2026-10-13 run dir), the local corpus clones in `~/.xerj-code`, and `.env` secrets. | `/home` and `/root` are masked with tmpfs. Only HOME, the auth dir and explicit `--ro` paths are visible. |+| 3 | `CLAUDE_CONFIG_DIR` was bound as-is. | Every arm loaded the user's global `CLAUDE.md`, skills, plugins and Stop hook, and could read other sessions' transcripts and history. `settings.json` `env` **injected `GH_TOKEN` into every arm**, despite `--clearenv`. | Everything in the dir except `.credentials.json` and `.claude.json` is masked, and `settings.json` is replaced by `{}`. The harness also passes `--strict-mcp-config --mcp-config '{"mcpServers":{}}' --disable-slash-commands --tools <arm set>`. |+| 4 | Inside pasta's user namespace the jail ran as uid 0. | `claude --dangerously-skip-permissions` refuses to run as root. | `bwrap --unshare-user --uid/--gid <real ids>`. |+| 5 | The host `resolv.conf` points at 127.0.0.53 (host loopback). | No DNS in the jail. | The jail gets its own `resolv.conf` (1.1.1.1, 8.8.8.8). |+| 6 | Masking files by binding `/dev/null` read back as EACCES. | `CLAUDE.md` read errors. | Masks are bound from regular empty files. |+| 7 | Catalogue override: `build-catalogue` wants the `.sig` beside each pack and verifies it with the client key; `sig_url` points at the real-key release `.sig`. | The override could not be built with a throwaway key, and a pull would fail signature verification. | Packs are re-signed with the throwaway key, `sig_url` is rewritten to the in-jail server, the catalogue is re-signed, and `XERJ_HUB_PUBKEY` is set for `build-catalogue` and the v2 shim (see above). |+| 8 | A5 prompt said `xerj corpus index` without the corpus name. | One wasted turn per A5 cell. | `xerj corpus index <slug>`. |+| 9 | The bytes counter mis-parsed `/proc/net/dev`: interface lines with no leading space shift the columns, so it summed packets instead of bytes. | `bytes_downloaded` was always ≈0. | Fixed parser; the A7 pre-pull is measured separately (`setup.rx`). |+| 10 | The contamination check let an A4 agent skip its search API. | An A4 cell measured A2. | A3/A4 must call their endpoint (violation otherwise); tool arms that answered with zero tool calls get a note. |+| 11 | Clone bytes were measured from `.git` dirs in the work dir. | Agents clone into the jail's `/tmp`, so the measure read 0. | Removed. Clone bytes are in the network-namespace `bytes_downloaded`. |++## Ambiguities in PLAN.md (flagged, not silently chosen)++The harness had to pick something to run the smoke. Each item below is what it picked,+and needs the user's decision at freeze (PLAN §10):++1. **"Nine arms"** (§3), but the table lists eight (A0–A7). The harness implements the eight listed.+2. **A3/A4 endpoints** are "locked at freeze". Provisional: A3 = Sourcegraph public+ stream search (`sourcegraph.com/.api/search/stream`, unauthenticated; it searches+ upstream HEAD, not the pin); A4 = DuckDuckGo HTML (`html.duckduckgo.com/html/`, no+ key). grep.app was rejected because it answers curl with a Vercel bot checkpoint (429).+3. **`--max-turns`**: "same for every arm", value not given. Provisional: 15, with A0 = 1+ and sessions at 15 × questions.+4. **Pin vs "current development source"**: the C1 questions ask for "current" while+ §4 grades at the pack pin. The harness adds one line to every arm's prompt: "The+ reference version is commit `<sha>` (`<date>`) of `<url>`; answer for that+ version". A1 is told how to fetch that commit.+5. **A2 "may not git clone"**: also read as no archive/tarball download (a clone by+ other means). Single-file raw fetches are allowed. A `git` shim refuses+ clone/fetch/pull/ls-remote.+6. **A5 "install xerj"**: the binary is placed on PATH for A5–A7 alike, so install+ cost is not measured. The prior run instead had the agent read llms.txt and install.+7. **A7 "steady state"**: the harness pre-pulls in the same jail, so the hub node is+ already up when the agent starts. A6 pays node boot plus pull.+8. **H2 and NoCorpus**: H2 counts "Confident, or Ambiguous containing the correct+ slug" as correct. The client actually returned **NoCorpus with a "closest corpora"+ list that contained the right slug**. Literal reading: not ok, so the answer is+ counted wrong even when the agent re-asks with `--corpus` and gets it right. The+ harness does this and records `closest_contains_expected` so the decision can be+ revisited.+9. **Corpora**: §4 names lmdb, quinn, leveldb, boringtun, badger, s2n-tls and lz4.+ Only lz4/snappy/zstd have v2 packs today. **C3 cannot run at all**: no non-git,+ multi-source corpus has a v2 pack.+10. **C2/C3 tasks** must be authored by a person and committed before any C2/C3 run+ (§5). None exist. `tasks/C2/lz4.smoke.json` is a smoke fixture written by the+ harness agent, and `pov.py` refuses it without `--allow-smoke-fixtures`.+11. **Judge**: no `ANTHROPIC_API_KEY` on this host, so the judge's live HTTP path is+ untested (the unscored path is tested). Verify at freeze that `claude-opus-4-8`+ exists and accepts `temperature: 0`.+12. **Hosted**: spec §12 puts hosted answers out of scope, so A6's first use is always a+ synchronous pull, never a hosted answer plus a background pull (§1 describes both).+13. **"Quiet machine"**: this host ran at load 25–113. The full run needs a host where+ load stays low, or the wall-time column has to be dropped.+14. **Failed fetches** is a heuristic: the tool result is empty or an error, or+ matches captcha/checkpoint/403/429 text.+15. **C1 grading looseness** (registered suite, unchanged): regexes like `0xf|15` are+ unanchored, so for example "115" passes. This is pre-existing and noted because it+ affects C1 accuracy for every arm equally.+16. `rg` is not installed on this host. A1's "ripgrep" falls back to grep, and the+ prompt says so.++## Estimated cost and wall time of the full pre-registered run++Assumptions: the 3 available packs; C1 = 3 × 10 registered tasks; C2 = 3 × 5 authored+tasks (count not fixed in the PLAN); C3 = 0 (no pack); C4 = 3 sessions × 10 questions;+8 arms; 3 repeats; serial. Cost per question unit is the smoke per-arm mean (about+$0.015–0.047; A5 is the most expensive).++| | agents | judge (C2 + C4's C2-style, opus) | wall (serial) |+|---|---|---|---|+| 3 repeats | **≈ $45** | ≈ $20 | **≈ 8.5 h** |+| + "+2 repeats on CV > 0.5" for ~30% of cells | ≈ $59 | ≈ $25 | ≈ 11 h |++**This is a floor, not a quote.** The smoke questions were easy: answered in 1–4+turns, and most C2 cells used no tools at all. Authored cross-file C2 questions and+10-question C4 sessions will take more turns. If the pinned agent model is Opus+rather than Sonnet, the per-turn price is higher. A realistic envelope is+**$150–300 and 10–15 h**. Run one repeat of the frozen matrix first and re-estimate+from that before committing the rest.++## Running it++```sh+# once per run dir (downloads + verifies the packs, builds the override, writes shims)+python3 benchmarks/hub-pov/harness/pov.py prepare --xerj engine/target/release/xerj \+ --registry <corpus-hub checkout>/tools/xerj-code/hub --registry-commit <sha>+python3 benchmarks/hub-pov/harness/pov.py verify-c1 # vendored C1 suites == bench/corpus-tasks+python3 benchmarks/hub-pov/harness/pov.py matrix benchmarks/hub-pov/matrix/smoke.json --run-id <id>+python3 benchmarks/hub-pov/harness/pov.py score --run-id <id>+python3 benchmarks/hub-pov/harness/pov.py report --run-id <id>+python3 -m unittest benchmarks/hub-pov/harness/test_pov.py # offline parser/grader/contamination tests+```++Results are written under `$POV_RUN_DIR` (default `/home/claude/hubdev-wt/pov-run`) and+never into the repository. Nothing in the harness publishes anything.diff --git a/benchmarks/hub-pov/harness/check.py b/benchmarks/hub-pov/harness/check.pynew file mode 100644index 000000000..7d55ff398--- /dev/null+++ b/benchmarks/hub-pov/harness/check.py@@ -0,0 +1,17 @@+#!/usr/bin/env python3+"""Grade one run's ANSWER line against the task's regex set."""+import json, re, sys+task = json.load(open(sys.argv[1]))+if "tasks" in task: # whole suite passed: argv[2] = task id, argv[3] = run+ task = next(t for t in task["tasks"] if t["id"] == sys.argv[2])+ run_json = json.load(open(sys.argv[3]))+else:+ run_json = json.load(open(sys.argv[2]))+text = (run_json.get("result") or "") + "\n" + run_json.get("_raw", "")+m = re.findall(r"ANSWER:\s*(.+)", text)+ans = m[-1] if m else text[-800:]+need = task.get("need", 1)+hits = sum(1 for pat in task["re"] if re.search(pat, ans, re.I))+print(json.dumps({"id": task["id"], "answered": bool(m), "hits": hits,+ "need": need, "pass": hits >= need,+ "answer": ans.strip()[:300]}))diff --git a/benchmarks/hub-pov/harness/pov.py b/benchmarks/hub-pov/harness/pov.pynew file mode 100644index 000000000..a6a9bfde7--- /dev/null+++ b/benchmarks/hub-pov/harness/pov.py@@ -0,0 +1,964 @@+#!/usr/bin/env python3+"""Corpus Hub v2 proof-of-value harness (benchmarks/hub-pov/PLAN.md).++One harness for all nine arms (PLAN §3: "All arms share one agent harness and+differ only in the tools on PATH and the prompt's tool instructions"). Every+`claude -p` runs inside ../sandbox.sh — there is no code path that launches an+agent outside it.++Subcommands+ prepare fetch + verify the real signed packs, build the catalogue override,+ write the shims (once per run dir)+ cell run one cell: arm x task(s) x repeat (one sandboxed agent)+ matrix run every cell of a matrix file, serially (PLAN §9: one at a time)+ score grade + metrics + contamination for every cell of a run+ report aggregate a scored run into a table (+ the §7 gate, refused < 3 repeats)+ verify-c1 check the vendored C1 suites against branch bench/corpus-tasks++Nothing here publishes anything. Results stay in the run dir (default+/home/claude/hubdev-wt/pov-run, override with POV_RUN_DIR).+"""+from __future__ import annotations++import argparse+import base64+import datetime as dt+import json+import os+import re+import shlex+import shutil+import statistics+import subprocess+import sys+import time+import urllib.error+import urllib.request+from pathlib import Path++HERE = Path(__file__).resolve().parent+POV = HERE.parent # benchmarks/hub-pov+REPO = POV.parent.parent # repo root+SANDBOX = POV / "sandbox.sh"+TASKS = POV / "tasks"+RUN_DIR = Path(os.environ.get("POV_RUN_DIR", "/home/claude/hubdev-wt/pov-run"))++# The catalogue override is served INSIDE each jail (its network namespace cannot+# reach the host loopback) on this port, from a read-only bind of <run>/hub/site.+CAT_PORT = 18080+CAT_URL = f"http://127.0.0.1:{CAT_PORT}"+# Where the real packs live. pack_url in the override catalogue points HERE, so the+# v2 arms download the real released bytes over the real network.+RELEASE_BASE = "https://github.com/xerj-org/xerj/releases/download"++# PLAN §3 leaves these unfixed ("locked at freeze", §10). PROVISIONAL values used+# by the smoke run only — see SMOKE.md "Ambiguities".+A3_ENDPOINT = "https://sourcegraph.com/.api/search/stream"+A4_ENDPOINT = "https://html.duckduckgo.com/html/"++# PLAN §3: "same --max-turns" for every arm but A0 (1). The number is not in the+# PLAN — PROVISIONAL, flagged in SMOKE.md. Session cells (C4) get the per-question+# budget times the number of questions.+MAX_TURNS = 15+CELL_TIMEOUT_S = 900+JUDGE_MODEL = "claude-opus-4-8" # PLAN §6, verbatim++BASE_TOOLS = "Bash,Read,Grep,Glob"+ARMS = {+ # tools: the --tools list (the built-in tools the arm may call at all)+ # shims: which PATH shims are exposed; xerj: which xerj shim (None = not on PATH)+ "A0": {"role": "memory only (floor)", "tools": "", "xerj": None, "git_noclone": False},+ "A1": {"role": "normal agent + git (gate baseline)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": False},+ "A2": {"role": "normal agent + web only (gate baseline)", "tools": BASE_TOOLS + ",WebFetch,WebSearch", "xerj": None, "git_noclone": True},+ "A3": {"role": "hosted regex code search (reference)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": True},+ "A4": {"role": "web-search API (reference)", "tools": BASE_TOOLS, "xerj": None, "git_noclone": True},+ "A5": {"role": "XERJ today, v1 corpus add path", "tools": BASE_TOOLS, "xerj": "v1", "git_noclone": False},+ "A6": {"role": "XERJ v2 first use (cold)", "tools": BASE_TOOLS, "xerj": "v2", "git_noclone": False},+ "A7": {"role": "XERJ v2 steady state (pack cached)", "tools": BASE_TOOLS, "xerj": "v2", "git_noclone": False},+}+XERJ_ARMS = {"A5", "A6", "A7"}+V2_ARMS = {"A6", "A7"}+++# --------------------------------------------------------------------------- utils+def now() -> str:+ return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")+++def die(msg: str, code: int = 2):+ print(f"pov.py: {msg}", file=sys.stderr)+ sys.exit(code)+++def load_json(p: Path):+ return json.loads(Path(p).read_text())+++def write_json(p: Path, obj) -> None:+ p.parent.mkdir(parents=True, exist_ok=True)+ p.write_text(json.dumps(obj, indent=1, sort_keys=False) + "\n")+++def hub_pubkey_hex() -> str:+ src = (REPO / "engine/crates/xerj-common/src/hub.rs").read_text()+ m = re.search(r'HUB_PUBKEY_HEX: &str = "([0-9a-f]{64})"', src)+ if not m:+ die("cannot read HUB_PUBKEY_HEX from engine/crates/xerj-common/src/hub.rs")+ return m.group(1)+++def ed25519(*args: str) -> str:+ return subprocess.run(["bash", str(REPO / "tools/hub-packs/ed25519.sh"), *args],+ check=True, capture_output=True, text=True).stdout.strip()+++PKCS8_PREFIX = "302e020100300506032b657004220420"+++def ed25519_sign(seed_hex: str, data_file: Path, sig_file: Path, scratch: Path) -> None:+ """Detached signature in the hub format (128 hex chars), with openssl only."""+ der = scratch / "seed.der"+ der.write_bytes(bytes.fromhex(PKCS8_PREFIX + seed_hex))+ raw = subprocess.run(["openssl", "pkeyutl", "-sign", "-inkey", str(der), "-keyform", "DER",+ "-rawin", "-in", str(data_file)], check=True, capture_output=True).stdout+ der.unlink()+ sig_file.write_text(raw.hex() + "\n")+++# --------------------------------------------------------------------------- tasks+def registry_manifest(prep: dict, slug: str) -> dict:+ return load_json(Path(prep["registry"]) / f"{slug}.json")+++def load_task(ref: str, allow_smoke: bool) -> dict:+ """ref = '<class>:<suite>:<id>' e.g. C1:snappy:T4, C2:lz4.smoke:S1."""+ try:+ cls, suite, tid = ref.split(":")+ except ValueError:+ die(f"task ref must be <class>:<suite>:<id>, got {ref!r}")+ f = TASKS / cls / f"{suite}.json"+ if not f.exists():+ die(f"no task file {f}")+ doc = load_json(f)+ if doc.get("smoke_only") and not allow_smoke:+ die(f"{f} is a smoke-only fixture (not pre-registered, PLAN §5); refusing without --allow-smoke-fixtures")+ task = next((t for t in doc["tasks"] if t["id"] == tid), None)+ if task is None:+ die(f"no task {tid} in {f}")+ return {"ref": ref, "class": cls, "suite": suite, "corpus": doc["corpus"],+ "subject": doc.get("subject", doc["corpus"]), "suite_pin": doc.get("pin"),+ "smoke_only": bool(doc.get("smoke_only")), "task": task}+++# --------------------------------------------------------------------------- prompts+COMMON = ("You are answering one technical question about the {subject} source tree. Be precise "+ "and brief. You MUST end your reply with a single line of the form 'ANSWER: <your answer>'.")+COMMON_SESSION = ("You are answering {n} technical questions about the {subject} source tree. Be "+ "precise and brief. You MUST end your reply with one line per question, of the "+ "form 'ANSWER <id>: <your answer>', in the order given.")+PIN_LINE = ("The reference version is commit {pin} ({pin_date}) of {url}; answer for that version.")+++def arm_instructions(arm: str, slug: str, url: str, pin: str, prep: dict) -> str:+ org_repo = url.removeprefix("https://github.com/")+ if arm == "A0": # PROTOCOL.md §4 bare prompt, verbatim+ return ("You are offline: no network access, and its source code is not available to you. "+ "Answer from your own knowledge; if you do not know, say 'unknown' in the ANSWER "+ "line — do not guess a fabricated constant.")+ if arm == "A1":+ return (f"You have a bash shell with network access. The project's public git repository is "+ f"{url}. You may clone it (e.g. `git clone --depth 1 {url}`; the reference commit "+ f"can be fetched with `git fetch --depth 1 origin {pin}`), search it with grep "+ f"(or ripgrep if installed) and read files.")+ if arm == "A2":+ return (f"You have a bash shell with network access and web tools (WebFetch, WebSearch, "+ f"curl). You may read web pages, including single files on GitHub (for example "+ f"https://raw.githubusercontent.com/{org_repo}/{pin}/<path> or the GitHub web UI). "+ f"You may NOT clone or fetch the git repository and may NOT download an "+ f"archive/tarball of it. The project's home is {url}.")+ if arm == "A3":+ return (f"You have a bash shell with network access and ONE tool for finding code: the "+ f"Sourcegraph public code-search API, via curl only: "+ f"curl -s -H 'Accept: text/event-stream' '{A3_ENDPOINT}?q=<url-encoded query>&display=20' "+ f"(query syntax: `repo:^github\\.com/{org_repo}$`, `file:<regex>`, "+ f"`patternType:regexp`). Do not clone the repository and do not fetch files from "+ f"any other site.")+ if arm == "A4":+ return (f"You have a bash shell with network access and ONE web-search API, via curl: "+ f"curl -s -A 'Mozilla/5.0' '{A4_ENDPOINT}?q=<url-encoded query>' (DuckDuckGo HTML "+ f"results). You may then fetch result pages with curl. Do not clone the repository "+ f"and do not download an archive/tarball of it.")+ if arm == "A5":+ manifest = Path(prep["registry"]) / f"{slug}.json"+ return ("XERJ is installed (`xerj` is on PATH). Use its reference-code flow: (1) start a node "+ "in the BACKGROUND, never in the foreground: `nohup xerj --insecure --port 9200 "+ "--data-dir $HOME/xerj-data > $HOME/xerj-node.log 2>&1 &` and wait until "+ "`curl -s localhost:9200/_cluster/health` answers; (2) `xerj corpus add --from "+ f"{manifest}`; (3) `xerj corpus index {slug}`; (4) query it: `xerj code {slug} \"<your "+ "search query>\"`. Do not clone or download the source yourself.")+ if arm in V2_ARMS:+ return ("XERJ is installed (`xerj` is on PATH). Ask it in plain English from bash: "+ "`xerj \"<your question>\"` — it picks the right reference corpus and answers from a "+ "signed, pre-indexed hub pack (`xerj ask --help` lists options). Use it as much as "+ "helps; cite what you found. Do not clone or download the source yourself.")+ die(f"unknown arm {arm}")+++def build_prompt(arm: str, tasks: list[dict], prep: dict) -> str:+ slug = tasks[0]["corpus"]+ if any(t["corpus"] != slug for t in tasks):+ die("a session cell must stay within one corpus (PLAN §4 C4: 'ten questions of one corpus')")+ man = registry_manifest(prep, slug)+ src = man["repos"][0]+ pin_date = prep["packs"][slug]["pin_date"]+ subject = tasks[0]["subject"]+ if len(tasks) == 1:+ head = COMMON.format(subject=subject)+ qs = "The question: " + tasks[0]["task"]["q"]+ else:+ head = COMMON_SESSION.format(subject=subject, n=len(tasks))+ qs = "The questions:\n" + "\n".join(f"[{t['task']['id']}] {t['task']['q']}" for t in tasks)+ pin_line = PIN_LINE.format(pin=src["sha"], pin_date=pin_date, url=src["url"])+ return " ".join([head, pin_line, arm_instructions(arm, slug, src["url"], src["sha"], prep)]) + "\n" + qs+++# --------------------------------------------------------------------------- prepare+def cmd_prepare(a):+ run = RUN_DIR+ xerj = Path(a.xerj).resolve()+ if not xerj.is_file():+ die(f"no xerj binary at {xerj}")+ registry = Path(a.registry).resolve()+ backlog = registry / "backlog/backlog-100.json"+ hub = run / "hub"+ (hub / "packs").mkdir(parents=True, exist_ok=True)+ (hub / "site/api/v1").mkdir(parents=True, exist_ok=True)+ (hub / "site/sigs").mkdir(parents=True, exist_ok=True)+ scratch = run / "scratch"; scratch.mkdir(exist_ok=True)+ pub_real = hub_pubkey_hex()+ tags = [t.strip() for t in a.tags.split(",") if t.strip()]+ rows, packs = [], {}+ for tag in tags:+ m = re.fullmatch(r"pack-(.+)-([0-9a-f]{8})", tag)+ if not m:+ die(f"not a pack release tag: {tag}")+ slug, pin8 = m.groups()+ fname = f"{slug}-{pin8}.hubpack.tar.zst"+ dest = hub / "packs" / tag+ dest.mkdir(exist_ok=True)+ if not (dest / fname).exists():+ subprocess.run(["gh", "release", "download", tag, "-R", "xerj-org/xerj", "-D", str(dest),+ "--clobber"], check=True)+ # 1. The released bytes are the genuine, hub-key-signed pack.+ ed25519("verify", pub_real, str(dest / fname), str(dest / f"{fname}.sig"))+ pj = json.loads(subprocess.run(f"zstd -dc {shlex.quote(str(dest / fname))} | tar -xOf - pack.json",+ shell=True, check=True, capture_output=True, text=True).stdout)+ packs[slug] = {"tag": tag, "file": fname, "pin8": pin8, "pin_sha": pj["pin_sha"],+ "pin_date": pj["pin_date"], "bytes": (dest / fname).stat().st_size,+ "licence_class": pj.get("licence_class"),+ "real_sig_verified_against": pub_real}+ rows.append(f"{slug}\t{pin8}\t{tag}\n")+ # 2. A throwaway signing key for the override. The live catalogue is not+ # deployed and the hub seed is a CI secret, so the override catalogue cannot+ # be signed with the real key; the client is pointed at the throwaway public+ # key (XERJ_HUB_PUBKEY, the override consumer_e2e.sh documents for local+ # rehearsals) and the pack signatures are re-made with it. The pack BYTES are+ # still the released ones (sha256 in the catalogue; downloaded from GitHub).+ seed_f = run / "hub/throwaway.seed"+ if not seed_f.exists():+ seed_f.write_text(os.urandom(32).hex() + "\n")+ seed_f.chmod(0o600)+ pub_tmp = ed25519("derive", str(seed_f))+ flat = hub / "flat"; flat.mkdir(exist_ok=True)+ for slug, p in packs.items():+ src = hub / "packs" / p["tag"] / p["file"]+ link = flat / p["file"]+ if not link.exists():+ os.link(src, link)+ sig = hub / "site/sigs" / f"{p['file']}.sig"+ ed25519_sign(seed_f.read_text().strip(), src, sig, scratch)+ ed25519("verify", pub_tmp, str(src), str(sig))+ shutil.copy(sig, flat / f"{p['file']}.sig") # build-catalogue reads it beside the pack+ (hub / "tags.tsv").write_text("".join(rows))+ cat = hub / "site/api/v1/catalogue.json"+ subprocess.run([str(xerj), "hub", "build-catalogue", "--registry", str(registry), "--backlog",+ str(backlog), "--packs", str(flat), "--tags", str(hub / "tags.tsv"),+ "--release-base", RELEASE_BASE, "--out", str(cat), "--sign-key", str(seed_f)],+ check=True, env=dict(os.environ, XERJ_HUB_PUBKEY=pub_tmp))+ doc = load_json(cat)+ for e in doc["corpora"]:+ p = packs[e["slug"]]+ if not e["pack_url"].startswith(RELEASE_BASE):+ die(f"unexpected pack_url {e['pack_url']}")+ e["sig_url"] = f"{CAT_URL}/sigs/{p['file']}.sig"+ cat.write_text(json.dumps(doc, separators=(",", ":")))+ ed25519_sign(seed_f.read_text().strip(), cat, Path(str(cat) + ".sig"), scratch)+ ed25519("verify", pub_tmp, str(cat), str(cat) + ".sig")+ # 3. Shims.+ shims = run / "shims"+ for kind in ("v1", "v2"):+ d = shims / f"xerj-{kind}"; d.mkdir(parents=True, exist_ok=True)+ exports = "" if kind == "v1" else (f"export XERJ_HUB_URL='{CAT_URL}'\n"+ f"export XERJ_HUB_PUBKEY='{pub_tmp}'")+ body = (HERE / "shim_xerj.sh").read_text().replace("@HUB_EXPORTS@", exports).replace("@REAL@", str(xerj))+ (d / "xerj").write_text(body); (d / "xerj").chmod(0o755)+ d = shims / "git-noclone"; d.mkdir(parents=True, exist_ok=True)+ shutil.copy(HERE / "shim_git_noclone.sh", d / "git"); (d / "git").chmod(0o755)+ claude = Path(shutil.which("claude") or die("claude not on PATH")).resolve()+ prep = {"prepared_at": now(), "xerj": str(xerj), "xerj_version": subprocess.run(+ [str(xerj), "--version"], capture_output=True, text=True).stdout.strip(),+ "claude": str(claude), "claude_version": subprocess.run(+ [str(claude), "--version"], capture_output=True, text=True).stdout.strip(),+ "registry": str(registry), "registry_commit": a.registry_commit,+ "catalogue_override": {"site": str(hub / "site"), "url_in_jail": CAT_URL,+ "throwaway_pubkey": pub_tmp, "real_pubkey": pub_real,+ "pack_url_base": RELEASE_BASE},+ "packs": packs, "shims": str(shims)}+ write_json(run / "prepared.json", prep)+ print(json.dumps(prep, indent=1))+++def load_prep() -> dict:+ p = RUN_DIR / "prepared.json"+ if not p.exists():+ die(f"{p} missing — run `pov.py prepare` first")+ return load_json(p)+++# --------------------------------------------------------------------------- cell+def cell_dir(run_id: str, arm: str, refs: list[str], repeat: int) -> Path:+ key = "+".join(r.replace(":", "_") for r in refs) if len(refs) <= 2 else \+ f"{refs[0].split(':')[0]}_{refs[0].split(':')[1]}_session{len(refs)}"+ return RUN_DIR / "results" / run_id / arm / key / f"r{repeat}"+++def run_script(arm: str, prompt_file: Path, prep: dict, tasks: list[dict], model: str,+ max_turns: int) -> str:+ """The script executed INSIDE the jail (bash). Generated, then run via sandbox.sh."""+ slug = tasks[0]["corpus"]+ claude = prep["claude"]+ tools = ARMS[arm]["tools"]+ args = [claude, "-p", "--model", model, "--output-format", "stream-json", "--verbose",+ "--max-turns", str(max_turns), "--tools", tools, "--strict-mcp-config",+ "--mcp-config", '{"mcpServers":{}}', "--disable-slash-commands",+ "--dangerously-skip-permissions"]+ q = shlex.quote+ co = prep["catalogue_override"]+ lines = ["set -u", 'S="$HOME/.sandbox"; mkdir -p "$HOME/.pov"',+ # received bytes on every non-loopback interface of the jail's netns+ 'netrx() { sed "s/:/ /" /proc/net/dev | awk \'NR>2 && $1!="lo" {s+=$2} END {print s+0}\'; }']+ if arm in V2_ARMS:+ lines += [f"python3 -m http.server {CAT_PORT} --bind 127.0.0.1 --directory {q(co['site'])} "+ f'> "$S/catalogue-http.log" 2>&1 &',+ f"for i in $(seq 1 50); do curl -fs -m1 {CAT_URL}/api/v1/catalogue.json >/dev/null && break; sleep 0.2; done"]+ if arm == "A7":+ # Steady state = the pack is already pulled AND the hub node is up (as on a+ # machine that has answered before). The pull runs with the REAL binary, not+ # the shim, so it is not counted as an agent call; it is timed separately.+ lines += ['t_s0=$(date +%s.%N)', 'srx0=$(netrx)',+ f"XERJ_HUB_URL={q(CAT_URL)} XERJ_HUB_PUBKEY={q(co['throwaway_pubkey'])} "+ f"{q(prep['xerj'])} hub pull {q(slug)} > \"$S/setup.log\" 2>&1; echo $? > \"$S/setup.rc\"",+ 'echo "$(date +%s.%N) $t_s0" > "$S/setup.time"', 'echo $(( $(netrx) - srx0 )) > "$S/setup.rx"']+ lines += ['ls -1 "$HOME/.xerj/hub/packs" > "$S/packs_before.txt" 2>/dev/null || true',+ 'cat /proc/loadavg > "$S/load_before"', 'rx0=$(netrx)', 't0=$(date +%s.%N)',+ f'timeout {CELL_TIMEOUT_S} ' + " ".join(q(x) for x in args) ++ f' "$(cat {q(str(prompt_file))})" < /dev/null > "$S/transcript.jsonl" 2> "$S/claude.err"',+ 'rc=$?', 't1=$(date +%s.%N)', 'rx1=$(netrx)', 'cat /proc/loadavg > "$S/load_after"',+ 'ls -1 "$HOME/.xerj/hub/packs" > "$S/packs_after.txt" 2>/dev/null || true',+ 'echo "{\\"rc\\": $rc, \\"t0\\": $t0, \\"t1\\": $t1, \\"rx_iface_bytes\\": $((rx1-rx0))}" > "$S/timing.json"',+ 'exit 0']+ return "\n".join(lines) + "\n"+++def cmd_cell(a, prep=None):+ prep = prep or load_prep()+ arm = a.arm+ if arm not in ARMS:+ die(f"unknown arm {arm}")+ refs = a.task+ tasks = [load_task(r, a.allow_smoke_fixtures) for r in refs]+ for t in tasks:+ if t["corpus"] not in prep["packs"]:+ die(f"corpus {t['corpus']} has no prepared v2 pack (prepared: {sorted(prep['packs'])})")+ cdir = cell_dir(a.run_id, arm, refs, a.repeat)+ if cdir.exists():+ if not a.force:+ print(f"skip (exists): {cdir}"); return+ shutil.rmtree(cdir)+ home = cdir / "home"+ (home / ".sandbox").mkdir(parents=True)+ max_turns = 1 if arm == "A0" else MAX_TURNS * len(tasks)+ prompt = build_prompt(arm, tasks, prep)+ # The prompt is staged OUTSIDE the jail HOME and exposed read-only, so the agent+ # cannot see other cells; it holds no gold answer.+ (cdir / "prompt.txt").write_text(prompt)+ script = run_script(arm, cdir / "prompt.txt", prep, tasks, a.model, max_turns)+ (cdir / "run.sh").write_text(script)+ ro = [prep["claude"], str(cdir / "prompt.txt"), str(cdir / "run.sh")]+ prefix = []+ if ARMS[arm]["xerj"]:+ prefix.append(str(Path(prep["shims"]) / f"xerj-{ARMS[arm]['xerj']}"))+ ro.append(prep["xerj"])+ if ARMS[arm]["git_noclone"]:+ prefix.append(str(Path(prep["shims"]) / "git-noclone"))+ if arm == "A5":+ ro.append(prep["registry"])+ if arm in V2_ARMS:+ ro.append(prep["catalogue_override"]["site"])+ cmd = ["bash", str(SANDBOX), "--home", str(home)]+ for p in ro:+ cmd += ["--ro", p]+ cmd += ["--", "bash", str(cdir / "run.sh")]+ env = dict(os.environ, XERJ_POV_PATH_PREFIX=":".join(prefix))+ meta = {"schema": "hub-pov/cell/1", "run_id": a.run_id, "arm": arm, "role": ARMS[arm]["role"],+ "tasks": refs, "repeat": a.repeat, "model_requested": a.model, "max_turns": max_turns,+ "tools": ARMS[arm]["tools"], "path_prefix": prefix, "ro_paths": ro,+ "started_at": now(), "host_load_before": Path("/proc/loadavg").read_text().split()[:3],+ "prepared": {"xerj_version": prep["xerj_version"], "claude_version": prep["claude_version"],+ "catalogue_override": prep["catalogue_override"]}}+ print(f"[{now()}] cell {arm} {' '.join(refs)} r{a.repeat} -> {cdir}", flush=True)+ t0 = time.time()+ p = subprocess.run(cmd, env=env, capture_output=True, text=True, timeout=CELL_TIMEOUT_S + 600)+ meta["sandbox_rc"] = p.returncode+ meta["sandbox_wall_s"] = round(time.time() - t0, 2)+ meta["host_load_after"] = Path("/proc/loadavg").read_text().split()[:3]+ meta["finished_at"] = now()+ (cdir / "sandbox.stderr").write_text(p.stderr)+ write_json(cdir / "cell.json", meta)+ if p.returncode != 0:+ print(f" sandbox rc={p.returncode}: {p.stderr[-500:]}", file=sys.stderr)+++def cmd_matrix(a):+ prep = load_prep()+ m = load_json(Path(a.matrix))+ for rep in range(1, m["repeats"] + 1):+ for cell in m["cells"]:+ for arm in m["arms"]:+ ns = argparse.Namespace(arm=arm, task=cell if isinstance(cell, list) else [cell],+ repeat=rep, run_id=a.run_id, model=m["model"],+ allow_smoke_fixtures=m.get("allow_smoke_fixtures", False),+ force=a.force)+ cmd_cell(ns, prep)+++# --------------------------------------------------------------------------- parsing+def read_events(path: Path) -> list[dict]:+ out = []+ if not path.exists():+ return out+ for line in path.read_text(errors="replace").splitlines():+ line = line.strip()+ if not line:+ continue+ try:+ out.append(json.loads(line))+ except json.JSONDecodeError:+ pass+ return out+++def transcript_metrics(events: list[dict]) -> dict:+ """PLAN §6: cost and output tokens are SUMMED over every `result` event."""+ results = [e for e in events if e.get("type") == "result"]+ init = next((e for e in events if e.get("type") == "system" and e.get("subtype") == "init"), {})+ tool_uses, tool_results = [], {}+ for e in events:+ msg = e.get("message") or {}+ content = msg.get("content") if isinstance(msg.get("content"), list) else []+ if e.get("type") == "assistant":+ for c in content:+ if c.get("type") == "tool_use":+ tool_uses.append({"id": c.get("id"), "name": c.get("name"), "input": c.get("input")})+ elif e.get("type") == "user":+ for c in content:+ if c.get("type") == "tool_result":+ body = c.get("content")+ if isinstance(body, list):+ body = "\n".join(x.get("text", "") for x in body if isinstance(x, dict))+ tool_results[c.get("tool_use_id")] = {"text": str(body or ""), "is_error": bool(c.get("is_error"))}+ final = results[-1].get("result", "") if results else ""+ return {+ "n_result_events": len(results),+ "cost_usd": round(sum(float(r.get("total_cost_usd") or 0) for r in results), 6),+ "output_tokens": sum(int((r.get("usage") or {}).get("output_tokens") or 0) for r in results),+ "input_tokens_all": sum(sum(int((r.get("usage") or {}).get(k) or 0) for k in+ ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens"))+ for r in results),+ "turns": sum(int(r.get("num_turns") or 0) for r in results),+ "is_error": any(r.get("is_error") for r in results),+ "result_subtypes": [r.get("subtype") for r in results],+ "model": init.get("model"), "init_tools": init.get("tools"), "init_mcp": init.get("mcp_servers"),+ "init_plugins": [p.get("name") for p in init.get("plugins") or []],+ "tool_uses": tool_uses, "tool_results": tool_results, "final_text": final,+ }+++BLOCK_RE = re.compile(r"captcha|security checkpoint|just a moment|attention required|access denied|"+ r"cf-chl|too many requests|rate limit|\b(403|429)\b.{0,20}(forbidden|too many)|"+ r"HTTP/[0-9.]+ (403|429)", re.I)+++def is_fetch(tu: dict) -> bool:+ if tu["name"] in ("WebFetch", "WebSearch"):+ return True+ if tu["name"] == "Bash":+ return bool(re.search(r"\b(curl|wget)\b", (tu.get("input") or {}).get("command", "")))+ return False+++def failed_fetches(tm: dict) -> tuple[int, int]:+ """(fetches, failed) — PLAN §6 'failed fetches'. HEURISTIC: a fetch whose result is+ empty, an error, or looks like a bot-protection / 403 / 429 page. Flagged in SMOKE.md."""+ n = failed = 0+ for tu in tm["tool_uses"]:+ if not is_fetch(tu):+ continue+ n += 1+ r = tm["tool_results"].get(tu["id"], {"text": "", "is_error": True})+ if r["is_error"] or not r["text"].strip() or BLOCK_RE.search(r["text"][:20000]):+ failed += 1+ return n, failed+++def bash_cmds(tm: dict) -> list[str]:+ return [(tu.get("input") or {}).get("command", "") for tu in tm["tool_uses"] if tu["name"] == "Bash"]+++def xerj_calls(home: Path) -> list[dict]:+ log = home / ".pov/xerj-calls.jsonl"+ calls = []+ if log.exists():+ for line in log.read_text().splitlines():+ try:+ c = json.loads(line)+ except json.JSONDecodeError:+ continue+ d = home / ".pov/calls"+ c["out"] = (d / f"{c['n']}.out").read_text(errors="replace") if (d / f"{c['n']}.out").exists() else ""+ c["err"] = (d / f"{c['n']}.err").read_text(errors="replace") if (d / f"{c['n']}.err").exists() else ""+ calls.append(c)+ return calls+++SUBCOMMANDS = {"ask", "code", "corpus", "hub", "init", "autoindex", "feedback", "mcp", "serve"}+++def classify_call(argv: list[str]) -> str:+ if not argv:+ return "none"+ a0 = argv[0]+ if a0 == "ask":+ return "v2-ask"+ if a0 == "hub":+ return "v2-hub-" + (argv[1] if len(argv) > 1 else "")+ if a0 == "code":+ return "code-def" if "--def" in argv else "code"+ if a0 == "corpus":+ return "v1-corpus-" + (argv[1] if len(argv) > 1 else "")+ if not a0.startswith("-") and " " in a0 and a0 not in SUBCOMMANDS:+ return "v2-ask" # the bare `xerj "<question>"` form (spec §5)+ if a0.startswith("-"):+ return "node-or-flag"+ return "other:" + a0+++def parse_answer_output(c: dict) -> dict:+ """Routed corpus + answered_from from one v2 call's output (text or --json)."""+ out = c["out"]+ r = {"rc": c["rc"], "corpus": None, "answered_from": None, "candidates": []}+ try:+ j = json.loads(out)+ if isinstance(j, dict):+ r["corpus"] = j.get("corpus"); r["answered_from"] = j.get("answered_from")+ return r+ except (json.JSONDecodeError, ValueError):+ pass+ m = re.search(r"^answered from:\s*(.+)$", out, re.M)+ if m:+ r["answered_from"] = m.group(1).strip().replace(" ", "_")+ m = re.match(r"\s*([a-z0-9][a-z0-9._-]*)\s+pin\s", out)+ if m:+ r["corpus"] = m.group(1)+ if c["rc"] == 4:+ r["candidates"] = re.findall(r"^\s*([a-z0-9][a-z0-9._-]*)\s+[—-]\s", out + "\n" + c["err"], re.M)+ return r+++def routing(calls: list[dict], expected: str) -> dict:+ """PLAN §2 H2 / §6: did the router pick the right corpus on the FIRST v2 question?+ Confident-and-correct, or Ambiguous containing the expected slug, counts as correct;+ a first call that passes --corpus bypassed routing (recorded, counted not-correct)."""+ v2 = [c for c in calls if classify_call(c["argv"]) == "v2-ask"]+ if not v2:+ return {"status": "no_v2_call", "ok": False}+ first = v2[0]+ if "--corpus" in first["argv"]:+ return {"status": "bypassed (--corpus on first call)", "ok": False}+ p = parse_answer_output(first)+ if first["rc"] == 4:+ ok = expected in p["candidates"]+ return {"status": "ambiguous", "candidates": p["candidates"], "ok": ok}+ if p["corpus"]:+ return {"status": "confident", "corpus": p["corpus"], "ok": p["corpus"] == expected}+ text = first["out"] + "\n" + first["err"]+ if "no hub corpus matches" in text:+ # NoCorpus. The client prints "closest corpora: a, b; re-run with --corpus".+ # PLAN H2 counts only Confident / Ambiguous-with-expected as correct; whether a+ # NoCorpus whose closest list holds the expected slug should count is an open+ # question for the user (SMOKE.md) — recorded, counted NOT ok.+ m = re.search(r"closest corpora:\s*([^;\n]+)", text)+ closest = [x.strip() for x in m.group(1).split(",")] if m else []+ return {"status": "no_corpus", "closest": closest,+ "closest_contains_expected": expected in closest, "ok": False}+ return {"status": f"no_answer rc={first['rc']}", "ok": False}+++# --------------------------------------------------------------------------- grading+def grade_c1(task: dict, answer_text: str) -> dict:+ tf = RUN_DIR / "scratch" / "c1_task.json"; rf = RUN_DIR / "scratch" / "c1_run.json"+ tf.parent.mkdir(parents=True, exist_ok=True)+ tf.write_text(json.dumps(task)); rf.write_text(json.dumps({"result": answer_text}))+ out = subprocess.run([sys.executable, str(HERE / "check.py"), str(tf), str(rf)],+ capture_output=True, text=True, check=True).stdout+ g = json.loads(out)+ return {"grader": "check.py", "status": "pass" if g["pass"] else "fail", **g}+++JUDGE_PROMPT = """You are grading one answer against a fixed rubric. Use ONLY the gold answer and rubric as the authority.++Question:+{q}++Gold answer:+{gold}++Rubric (each point must be judged met or not met):+{rubric}++Pass rule: {pass_rule} (\"all\" = every rubric point met).++Answer to grade:+<<<+{answer}+>>>++Reply with ONLY a JSON object: {{"rubric_met": [true/false per rubric point, in order], "pass": true/false, "note": "<one sentence>"}}"""+++def grade_judge(task: dict, answer_text: str) -> dict:+ """PLAN §6: separate inference, claude-opus-4-8, temperature 0; no key or any+ error => `unscored` (never pass, never fail)."""+ key = os.environ.get("ANTHROPIC_API_KEY")+ if not key:+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "unscored", "reason": "ANTHROPIC_API_KEY unset"}+ body = {"model": JUDGE_MODEL, "max_tokens": 600, "temperature": 0,+ "messages": [{"role": "user", "content": JUDGE_PROMPT.format(+ q=task["q"], gold=task["gold"], rubric="\n".join(f"{i+1}. {r}" for i, r in enumerate(task["rubric"])),+ pass_rule=task.get("pass_rule", "all"), answer=answer_text)}]}+ req = urllib.request.Request("https://api.anthropic.com/v1/messages", data=json.dumps(body).encode(),+ headers={"x-api-key": key, "anthropic-version": "2023-06-01",+ "content-type": "application/json"})+ try:+ with urllib.request.urlopen(req, timeout=120) as r:+ resp = json.loads(r.read())+ text = "".join(c.get("text", "") for c in resp.get("content", []))+ j = json.loads(re.search(r"\{.*\}", text, re.S).group(0))+ met = [bool(x) for x in j["rubric_met"]]+ if len(met) != len(task["rubric"]):+ raise ValueError("rubric_met length mismatch")+ rule_pass = all(met) if task.get("pass_rule", "all") == "all" else bool(j["pass"])+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "pass" if rule_pass else "fail",+ "rubric_met": met, "judge_pass_field": j.get("pass"), "note": j.get("note"),+ "judge_usage": resp.get("usage")}+ except (urllib.error.URLError, ValueError, KeyError, AttributeError, json.JSONDecodeError) as e:+ return {"grader": f"judge:{JUDGE_MODEL}", "status": "unscored", "reason": f"judge error: {e}"}+++def split_session_answers(text: str, ids: list[str]) -> dict[str, str]:+ out = {}+ for i in ids:+ m = re.findall(rf"ANSWER\s*\[?{re.escape(i)}\]?\s*:\s*(.+)", text)+ out[i] = ("ANSWER: " + m[-1]) if m else ""+ return out+++# --------------------------------------------------------------------------- contamination+GITHUB_SRC_RE = re.compile(r"(raw\.githubusercontent\.com|codeload\.github\.com|/archive/|/tarball/|/zipball/|"+ r"github\.com/[^\s'\"]+/(blob|tree|raw)/)", re.I)+CLONE_RE = re.compile(r"\bgit\s+(clone|fetch|pull)\b")+ARCHIVE_RE = re.compile(r"(codeload\.github\.com|/archive/|/tarball/|/zipball/|\.tar\.gz|\.zip\b)", re.I)+URL_HOST_RE = re.compile(r"https?://([A-Za-z0-9.-]+)")+++def contamination(arm: str, tm: dict, calls: list[dict], home: Path, expected_slug: str) -> dict:+ """Did the arm have ONLY its intended tools, and did it use the thing under test?+ Returns violations (hard: the cell is invalid) and notes (soft: reported)."""+ v, notes = [], []+ declared = [t for t in ARMS[arm]["tools"].split(",") if t]+ used = sorted({tu["name"] for tu in tm["tool_uses"]})+ if tm["init_tools"] is not None and sorted(tm["init_tools"]) != sorted(declared):+ v.append(f"init tools {tm['init_tools']} != declared {declared}")+ if tm["init_mcp"]:+ v.append(f"MCP servers present: {tm['init_mcp']}")+ extra = [u for u in used if u not in declared]+ if extra:+ v.append(f"used undeclared tools {extra}")+ cmds = "\n".join(bash_cmds(tm))+ kinds = [classify_call(c["argv"]) for c in calls]+ if arm == "A0" and tm["tool_uses"]:+ v.append("memory arm used tools")+ if arm in ("A0", "A1", "A2", "A3", "A4"):+ if calls or re.search(r"(^|[\s;&|(])xerj\b", cmds):+ v.append("non-XERJ arm invoked xerj")+ if arm != "A0" and not tm["tool_uses"]:+ notes.append("answered with no tool call at all (model recall, not the arm's tools)")+ elif arm == "A1" and not CLONE_RE.search(cmds):+ notes.append("git arm never cloned/fetched (answered without the repo)")+ # A3/A4 are defined by their ONE endpoint: using the shell for other fetches+ # without ever calling it makes the cell an A2 cell (seen in the smoke: an A4+ # agent went straight to raw.githubusercontent.com).+ if arm == "A3" and tm["tool_uses"] and "sourcegraph.com" not in cmds:+ v.append("A3 never called the code-search endpoint")+ if arm == "A4" and tm["tool_uses"]:+ if "duckduckgo.com" not in cmds:+ v.append("A4 never called the web-search API (fetched pages directly)")+ else:+ other = sorted({h for h in URL_HOST_RE.findall(cmds)} - {"html.duckduckgo.com", "duckduckgo.com"})+ if other:+ notes.append(f"A4 fetched pages from {other} (allowed only as search results; check order)")+ if arm in ("A2", "A3", "A4"):+ refused = home / ".pov/git-refused.log"+ if CLONE_RE.search(cmds) or (refused.exists() and refused.read_text().strip()):+ v.append("git clone/fetch attempted in a no-git arm")+ if ARCHIVE_RE.search(cmds):+ v.append("repository archive download in a no-git arm")+ if arm == "A3":+ hosts = {h for h in URL_HOST_RE.findall(cmds)} - {"sourcegraph.com"}+ if hosts:+ v.append(f"A3 fetched hosts other than the code-search endpoint: {sorted(hosts)}")+ if any(tu["name"] in ("WebFetch", "WebSearch") for tu in tm["tool_uses"]):+ v.append("A3 used web tools")+ if arm in XERJ_ARMS:+ if CLONE_RE.search(cmds):+ v.append("XERJ arm ran git clone/fetch itself")+ if GITHUB_SRC_RE.search(cmds):+ v.append("XERJ arm fetched source from GitHub directly")+ if arm == "A5":+ if any(k.startswith("v2-") for k in kinds):+ v.append("v1 arm used the v2 surface (ask / hub)")+ if "v1-corpus-add" not in kinds:+ notes.append("v1 arm never ran `xerj corpus add`")+ if not any(k.startswith("code") for k in kinds):+ notes.append("v1 arm never ran `xerj code`")+ if arm in V2_ARMS:+ if any(k.startswith("v1-corpus-add") or k.startswith("v1-corpus-index") for k in kinds):+ v.append("v2 arm used the v1 corpus add/index path")+ answers = [parse_answer_output(c) for c in calls if classify_call(c["argv"]) in ("v2-ask", "code-def", "code")]+ froms = [x["answered_from"] for x in answers if x["answered_from"]]+ if not any(k == "v2-ask" for k in kinds):+ v.append("v2 arm made no `xerj \"<q>\"`/`xerj ask` call (a pass would be model recall, PLAN §3)")+ elif "local_pack" not in froms:+ v.append(f"no v2 answer came from the local pack (answered_from seen: {froms})")+ before = (home / ".sandbox/packs_before.txt")+ before_l = before.read_text().split() if before.exists() else []+ if arm == "A6" and before_l:+ v.append(f"A6 was not cold: packs cached before the agent ran: {before_l}")+ if arm == "A7" and not any(expected_slug in x for x in before_l):+ v.append(f"A7 was not warm: {expected_slug} pack not cached before the agent ran ({before_l})")+ return {"violations": v, "notes": notes, "tools_used": used, "xerj_call_kinds": kinds,+ "n_bash": len(bash_cmds(tm))}+++# --------------------------------------------------------------------------- score+def dir_bytes(p: Path) -> int:+ if not p.exists():+ return 0+ return sum(f.stat().st_size for f in p.rglob("*") if f.is_file() and not f.is_symlink())+++def net_counters(home: Path) -> dict:+ f = home / ".sandbox/net.json"+ if not f.exists():+ return {}+ out = {}+ for item in load_json(f).get("nftables", []):+ c = item.get("counter")+ if c:+ out[c["name"]] = c["bytes"]+ return out+++def score_cell(cdir: Path) -> dict | None:+ meta_f = cdir / "cell.json"+ if not meta_f.exists():+ return None+ meta = load_json(meta_f)+ home = cdir / "home"+ sb = home / ".sandbox"+ tm = transcript_metrics(read_events(sb / "transcript.jsonl"))+ calls = xerj_calls(home)+ tasks = [load_task(r, True) for r in meta["tasks"]]+ slug = tasks[0]["corpus"]+ timing = load_json(sb / "timing.json") if (sb / "timing.json").exists() else {}+ wall = round(timing["t1"] - timing["t0"], 2) if timing else None+ net = net_counters(home)+ model_bytes = net.get("rx_model_api", 0)+ rx_iface = timing.get("rx_iface_bytes")+ # bytes downloaded (PLAN §6): everything the jail received while the agent ran,+ # minus model-API traffic (no setup traffic precedes the agent except A7's pull,+ # which is reported separately and excluded by the interface delta).+ bytes_dl = max(0, rx_iface - model_bytes) if rx_iface is not None else None+ setup = None+ if (sb / "setup.time").exists():+ t1s, t0s = map(float, (sb / "setup.time").read_text().split())+ setup = {"rc": int((sb / "setup.rc").read_text().strip() or 255), "seconds": round(t1s - t0s, 2),+ "rx_bytes": int((sb / "setup.rx").read_text().strip() or 0) if (sb / "setup.rx").exists() else None,+ "log_tail": (sb / "setup.log").read_text(errors="replace")[-400:]}+ pack_bytes_cached = dir_bytes(home / ".xerj/hub/packs")+ n_fetch, n_failed = failed_fetches(tm)+ # grading+ if len(tasks) == 1:+ per_task = {tasks[0]["task"]["id"]: tm["final_text"]}+ else:+ per_task = split_session_answers(tm["final_text"], [t["task"]["id"] for t in tasks])+ grades = []+ for t in tasks:+ ans = per_task.get(t["task"]["id"], "")+ g = grade_c1(t["task"], ans) if t["class"] in ("C1",) or "re" in t["task"] else grade_judge(t["task"], ans)+ g["ref"] = t["ref"]+ grades.append(g)+ cont = contamination(meta["arm"], tm, calls, home, slug)+ route = routing(calls, slug) if meta["arm"] in V2_ARMS else None+ # H2: a mis-route counts the v2 answer wrong.+ for g in grades:+ g["status_raw"] = g["status"]+ if route is not None and not route["ok"] and g["status"] == "pass":+ g["status"] = "fail"; g["misroute_override"] = True+ statuses = [g["status"] for g in grades]+ correct = None if "unscored" in statuses else sum(s == "pass" for s in statuses) / len(statuses)+ row = {+ "schema": "hub-pov/score/1", "run_id": meta["run_id"], "arm": meta["arm"], "tasks": meta["tasks"],+ "repeat": meta["repeat"], "model": tm["model"], "model_requested": meta["model_requested"],+ "valid": not cont["violations"] and tm["n_result_events"] >= 1 and timing.get("rc") == 0,+ "correct": correct, "grades": grades,+ "cost_usd": tm["cost_usd"], "output_tokens": tm["output_tokens"], "input_tokens_all": tm["input_tokens_all"],+ "turns": tm["turns"], "n_result_events": tm["n_result_events"], "claude_rc": timing.get("rc"),+ "result_subtypes": tm["result_subtypes"],+ "wall_s": wall, "load_before": (sb / "load_before").read_text().split()[:3] if (sb / "load_before").exists() else None,+ "load_after": (sb / "load_after").read_text().split()[:3] if (sb / "load_after").exists() else None,+ "bytes_downloaded": bytes_dl, "bytes_rx_iface": rx_iface, "bytes_model_api": model_bytes,+ "bytes_rx_jail_total": net.get("rx_all"), "pack_bytes_cached": pack_bytes_cached,+ "fetches": n_fetch, "failed_fetches": n_failed,+ "xerj_calls": len(calls), "routing": route, "setup": setup,+ "answered_from": [x for x in (parse_answer_output(c)["answered_from"] for c in calls+ if classify_call(c["argv"]) in ("v2-ask", "code", "code-def")) if x],+ "contamination": cont, "final_answer_tail": tm["final_text"][-300:],+ }+ write_json(cdir / "score.json", row)+ return row+++def cells_of(run_id: str) -> list[Path]:+ root = RUN_DIR / "results" / run_id+ return sorted(p.parent for p in root.rglob("cell.json"))+++def cmd_score(a):+ rows = [r for r in (score_cell(c) for c in cells_of(a.run_id)) if r]+ out = RUN_DIR / "results" / a.run_id / "scores.jsonl"+ out.write_text("".join(json.dumps(r) + "\n" for r in rows))+ print(f"scored {len(rows)} cells -> {out}")+++# --------------------------------------------------------------------------- report+def fmt(x, nd=3):+ if x is None:+ return "–"+ if isinstance(x, float):+ return f"{x:.{nd}f}"+ return str(x)+++def cmd_report(a):+ rows = [json.loads(l) for l in (RUN_DIR / "results" / a.run_id / "scores.jsonl").read_text().splitlines()]+ hdr = ("| arm | task(s) | rep | valid | correct | grade | cost $ | out tok | turns | wall s | load1 | "+ "bytes dl | fetches (failed) | xerj calls | routing | answered_from | violations / notes |")+ print(hdr)+ print("|" + "---|" * (hdr.count("|") - 1))+ for r in sorted(rows, key=lambda r: (r["tasks"], r["arm"], r["repeat"])):+ g = ",".join(x["status"] + ("*" if x.get("misroute_override") else "") for x in r["grades"])+ froms = ",".join(sorted(set(r.get("answered_from") or []))) or "–"+ cont = r["contamination"]+ vio = "; ".join(cont["violations"] + [f"note: {n}" for n in cont["notes"]]) or "clean"+ route = r["routing"]["status"] + ("/ok" if r["routing"]["ok"] else "/WRONG") if r["routing"] else "–"+ print(f"| {r['arm']} | {' '.join(r['tasks'])} | {r['repeat']} | {'yes' if r['valid'] else 'NO'} | "+ f"{fmt(r['correct'], 2)} | {g} | {fmt(r['cost_usd'], 4)} | {r['output_tokens']} | {r['turns']} | "+ f"{fmt(r['wall_s'], 1)} | {(r['load_before'] or ['–'])[0]} | {fmt(r['bytes_downloaded'])} | "+ f"{r['fetches']} ({r['failed_fetches']}) | {r['xerj_calls']} | {route} | {froms} | {vio} |")+ gate(rows)+++def gate(rows):+ """PLAN §7, checked only on a complete matrix with >= 3 repeats per cell."""+ reps = {}+ for r in rows:+ reps.setdefault((r["arm"], tuple(r["tasks"])), []).append(r)+ if not reps or min(len(v) for v in reps.values()) < 3:+ print("\n§7 gate: NOT EVALUATED — fewer than 3 repeats per cell (PLAN §6/§7). "+ "These numbers are not results.")+ return+ print("\n§7 gate: evaluation over a complete >=3-repeat matrix is implemented in "+ "gate_eval(); run it on the frozen full matrix only.")+ for (arm, tasks), v in sorted(reps.items()):+ for metric in ("cost_usd", "output_tokens"):+ xs = [x[metric] for x in v]+ m = statistics.mean(xs)+ cv = statistics.pstdev(xs) / m if m else 0+ if cv > 0.5:+ print(f" CV>{0.5}: {arm} {tasks} {metric} cv={cv:.2f} -> needs +2 repeats (PLAN §6)")+++# --------------------------------------------------------------------------- verify-c1+def cmd_verify_c1(a):+ src = load_json(TASKS / "C1/SOURCE.json")+ ok = True+ for rel, blob in src["files"].items():+ local = TASKS / "C1" / Path(rel).name if rel.startswith("tasks/") else HERE / Path(rel).name+ h = subprocess.run(["git", "hash-object", str(local)], capture_output=True, text=True).stdout.strip()+ try:+ remote = subprocess.run(["git", "-C", str(REPO), "rev-parse",+ f"origin/{src['source_branch']}:benchmarks/corpus-tasks/{rel}"],+ capture_output=True, text=True, check=True).stdout.strip()+ except subprocess.CalledProcessError:+ remote = "(branch not fetched)"+ good = h == blob == remote+ ok &= good+ print(f"{'ok ' if good else 'BAD'} {rel}: local={h[:12]} recorded={blob[:12]} branch={remote[:12]}")+ sys.exit(0 if ok else 1)+++def main():+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ sub = ap.add_subparsers(dest="cmd", required=True)+ p = sub.add_parser("prepare"); p.add_argument("--xerj", required=True)+ p.add_argument("--registry", required=True, help="the corpus-hub registry dir (tools/xerj-code/hub)")+ p.add_argument("--registry-commit", default=None, help="the corpus-hub commit the registry came from")+ p.add_argument("--tags", default="pack-lz4-0774d055,pack-snappy-9c28114a,pack-zstd-01b7154f")+ p.set_defaults(fn=cmd_prepare)+ p = sub.add_parser("cell"); p.add_argument("--arm", required=True)+ p.add_argument("--task", required=True, action="append", help="<class>:<suite>:<id>; repeat for a session")+ p.add_argument("--repeat", type=int, default=1); p.add_argument("--run-id", required=True)+ p.add_argument("--model", required=True); p.add_argument("--allow-smoke-fixtures", action="store_true")+ p.add_argument("--force", action="store_true"); p.set_defaults(fn=cmd_cell)+ p = sub.add_parser("matrix"); p.add_argument("matrix"); p.add_argument("--run-id", required=True)+ p.add_argument("--force", action="store_true"); p.set_defaults(fn=cmd_matrix)+ p = sub.add_parser("score"); p.add_argument("--run-id", required=True); p.set_defaults(fn=cmd_score)+ p = sub.add_parser("report"); p.add_argument("--run-id", required=True); p.set_defaults(fn=cmd_report)+ p = sub.add_parser("verify-c1"); p.set_defaults(fn=cmd_verify_c1)+ a = ap.parse_args()+ a.fn(a)+++if __name__ == "__main__":+ main()diff --git a/benchmarks/hub-pov/harness/shim_git_noclone.sh b/benchmarks/hub-pov/harness/shim_git_noclone.shnew file mode 100755index 000000000..d28c8f4ef--- /dev/null+++ b/benchmarks/hub-pov/harness/shim_git_noclone.sh@@ -0,0 +1,13 @@+#!/usr/bin/env bash+# `git` shim for arm A2 (normal agent + web ONLY): PLAN.md §3 says A2 "may not+# `git clone` the corpus source". Refuses every verb that fetches a repository and+# logs the attempt to $HOME/.pov/git-refused.log (the contamination check reads it);+# anything else is passed to the real git. A2's prompt states the rule, so a refusal+# is the agent ignoring it, not a trap.+case " $* " in+ *" clone "*|*" fetch "*|*" pull "*|*" ls-remote "*|*" submodule "*|*" archive "*--remote*)+ mkdir -p "$HOME/.pov"; echo "$(date -u +%FT%TZ) $*" >> "$HOME/.pov/git-refused.log"+ echo "git: fetching repositories is not allowed in this environment (web only)" >&2+ exit 128 ;;+esac+exec /usr/bin/git "$@"diff --git a/benchmarks/hub-pov/harness/shim_xerj.sh b/benchmarks/hub-pov/harness/shim_xerj.shnew file mode 100755index 000000000..4bf2a6493--- /dev/null+++ b/benchmarks/hub-pov/harness/shim_xerj.sh@@ -0,0 +1,24 @@+#!/usr/bin/env bash+# Logging `xerj` shim for the XERJ arms (PLAN.md §3 note: "invoke xerj through a+# logging PATH shim that counts invocations"). Generated per run by `pov.py prepare`+# from this template; the placeholders are filled in there.+#+# Every call appends one JSON line to $HOME/.pov/xerj-calls.jsonl and keeps the+# call's stdout/stderr in $HOME/.pov/calls/<n>.{out,err} so the scorer can read+# `answered from:` / `answered_from` and the routed corpus (PLAN.md §6 routing).+# The v2 shim also points the client at the catalogue override (SMOKE.md: the live+# hub catalogue is not deployed) — those two exports are the ONLY behaviour change.+@HUB_EXPORTS@+REAL='@REAL@'+d="$HOME/.pov/calls"; mkdir -p "$d"+n=$(( $(ls "$d" 2>/dev/null | grep -c '\.rc$') + 1 ))+t0=$(date +%s.%N)+# stdout/stderr are teed, not captured, so the agent sees exactly what it would see.+# A daemon the call spawns (the hub node) is started detached with its own log, so it+# does not hold the tee pipes open.+{ "$REAL" "$@" 2> >(tee "$d/$n.err" >&2); echo $? > "$d/$n.rc"; } | tee "$d/$n.out"+rc=$(cat "$d/$n.rc" 2>/dev/null || echo 255)+t1=$(date +%s.%N)+python3 -c 'import json,sys; print(json.dumps({"n":int(sys.argv[1]),"argv":sys.argv[5:],"rc":int(sys.argv[2]),"t0":float(sys.argv[3]),"t1":float(sys.argv[4])}))' \+ "$n" "$rc" "$t0" "$t1" "$@" >> "$HOME/.pov/xerj-calls.jsonl"+exit "$rc"diff --git a/benchmarks/hub-pov/harness/test_pov.py b/benchmarks/hub-pov/harness/test_pov.pynew file mode 100644index 000000000..9fb1c28fc--- /dev/null+++ b/benchmarks/hub-pov/harness/test_pov.py@@ -0,0 +1,145 @@+#!/usr/bin/env python3+"""Offline tests for pov.py's parsing, grading and contamination logic (no agent,+no network): `python3 -m unittest benchmarks/hub-pov/harness/test_pov.py`."""+import json+import os+import sys+import tempfile+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))+os.environ.setdefault("POV_RUN_DIR", tempfile.mkdtemp(prefix="pov-test-"))+import pov # noqa: E402+++def ev_tool(name, tid, **inp):+ return {"type": "assistant", "message": {"content": [{"type": "tool_use", "id": tid, "name": name, "input": inp}]}}+++def ev_res(tid, text, err=False):+ return {"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": tid, "content": text, "is_error": err}]}}+++def result(cost, out, turns, text="ANSWER: 15"):+ return {"type": "result", "subtype": "success", "total_cost_usd": cost, "num_turns": turns,+ "usage": {"output_tokens": out, "input_tokens": 10}, "result": text}+++INIT = {"type": "system", "subtype": "init", "model": "claude-sonnet-5-5", "tools": ["Bash", "Read", "Grep", "Glob"],+ "mcp_servers": [], "plugins": []}+++class Metrics(unittest.TestCase):+ def test_sums_every_result_event(self):+ # PLAN §6 summing lesson: a background task emits several result events.+ tm = pov.transcript_metrics([INIT, result(0.01, 100, 3, "x"), result(0.02, 50, 2, "ANSWER: 8")])+ self.assertEqual(tm["n_result_events"], 2)+ self.assertAlmostEqual(tm["cost_usd"], 0.03)+ self.assertEqual(tm["output_tokens"], 150)+ self.assertEqual(tm["turns"], 5)+ self.assertEqual(tm["final_text"], "ANSWER: 8")++ def test_failed_fetch_heuristic(self):+ evs = [INIT, ev_tool("Bash", "1", command="curl -s https://grep.app/api"),+ ev_res("1", "<title>Vercel Security Checkpoint</title>"),+ ev_tool("Bash", "2", command="curl -s https://example.org"), ev_res("2", "hello"),+ ev_tool("WebFetch", "3", url="https://x"), ev_res("3", "", err=True),+ ev_tool("Bash", "4", command="ls"), ev_res("4", "")]+ self.assertEqual(pov.failed_fetches(pov.transcript_metrics(evs)), (3, 2))+++class Calls(unittest.TestCase):+ def test_classify(self):+ c = pov.classify_call+ self.assertEqual(c(["what is the exact value of kInputMarginBytes in snappy"]), "v2-ask")+ self.assertEqual(c(["ask", "q"]), "v2-ask")+ self.assertEqual(c(["hub", "pull", "lz4"]), "v2-hub-pull")+ self.assertEqual(c(["corpus", "add", "--from", "x.json"]), "v1-corpus-add")+ self.assertEqual(c(["code", "lz4", "frame checksum"]), "code")+ self.assertEqual(c(["--insecure", "--port", "9200"]), "node-or-flag")++ def text_call(self, out, rc=0, argv=None):+ return {"argv": argv or ["how does snappy x"], "rc": rc, "out": out, "err": ""}++ def test_parse_text_answer(self):+ out = "snappy pin 2026-10-02 (15 days old) BSD-3-Clause snappy.cc:12\n...\nanswered from: local pack\n"+ p = pov.parse_answer_output(self.text_call(out))+ self.assertEqual((p["corpus"], p["answered_from"]), ("snappy", "local_pack"))++ def test_parse_json_answer(self):+ p = pov.parse_answer_output(self.text_call(json.dumps({"corpus": "lz4", "answered_from": "local_pack"})))+ self.assertEqual((p["corpus"], p["answered_from"]), ("lz4", "local_pack"))++ def test_routing(self):+ ok = self.text_call("lz4 pin 2026-10-02 (15 days old) x a.c:1\nanswered from: local pack\n")+ self.assertTrue(pov.routing([ok], "lz4")["ok"])+ self.assertFalse(pov.routing([ok], "zstd")["ok"])+ amb = self.text_call("lz4 — an agent ...\nzstd — an agent ...\n", rc=4)+ self.assertTrue(pov.routing([amb], "zstd")["ok"])+ byp = self.text_call("lz4 pin x", argv=["ask", "q q", "--corpus", "lz4"])+ self.assertFalse(pov.routing([byp], "lz4")["ok"])+ self.assertEqual(pov.routing([], "lz4")["status"], "no_v2_call")+++class Grading(unittest.TestCase):+ def test_c1_uses_registered_checker(self):+ task = {"id": "T4", "re": ["0xf|15"], "need": 1}+ self.assertEqual(pov.grade_c1(task, "blah\nANSWER: 15")["status"], "pass")+ self.assertEqual(pov.grade_c1(task, "ANSWER: unknown")["status"], "fail")++ def test_judge_without_key_is_unscored(self):+ old = os.environ.pop("ANTHROPIC_API_KEY", None)+ try:+ g = pov.grade_judge({"q": "q", "gold": "g", "rubric": ["a"]}, "ANSWER: x")+ self.assertEqual(g["status"], "unscored")+ finally:+ if old is not None:+ os.environ["ANTHROPIC_API_KEY"] = old++ def test_session_split(self):+ s = pov.split_session_answers("ANSWER T1: 4\nANSWER [T2]: 10\n", ["T1", "T2", "T3"])+ self.assertEqual(s, {"T1": "ANSWER: 4", "T2": "ANSWER: 10", "T3": ""})+++class Contamination(unittest.TestCase):+ def tm(self, tools, *evs):+ init = dict(INIT, tools=tools)+ return pov.transcript_metrics([init, *evs, result(0.01, 1, 1)])++ def test_web_arm_clone_is_a_violation(self):+ tm = self.tm(["Bash", "Read", "Grep", "Glob", "WebFetch", "WebSearch"],+ ev_tool("Bash", "1", command="git clone --depth 1 https://github.com/google/snappy"))+ c = pov.contamination("A2", tm, [], Path(tempfile.mkdtemp()), "snappy")+ self.assertTrue(any("clone" in v for v in c["violations"]))++ def test_git_arm_using_xerj_is_a_violation(self):+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("Bash", "1", command="xerj 'how does x'"))+ c = pov.contamination("A1", tm, [], Path(tempfile.mkdtemp()), "snappy")+ self.assertTrue(any("xerj" in v for v in c["violations"]))++ def test_v2_arm_without_local_pack_answer_is_a_violation(self):+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("Bash", "1", command="xerj 'how does x y'"))+ call = {"argv": ["how does x y"], "rc": 1, "out": "", "err": "no hub corpus matches"}+ c = pov.contamination("A6", tm, [call], Path(tempfile.mkdtemp()), "snappy")+ self.assertTrue(any("local pack" in v for v in c["violations"]))++ def test_undeclared_tool_is_a_violation(self):+ tm = self.tm(["Bash", "Read", "Grep", "Glob"], ev_tool("WebFetch", "1", url="https://x"))+ c = pov.contamination("A1", tm, [], Path(tempfile.mkdtemp()), "snappy")+ self.assertTrue(any("undeclared" in v for v in c["violations"]))++ def test_search_arm_that_skips_its_endpoint_is_a_violation(self):+ tm = self.tm(["Bash", "Read", "Grep", "Glob"],+ ev_tool("Bash", "1", command="curl -s https://raw.githubusercontent.com/google/snappy/x/snappy.cc"))+ c = pov.contamination("A4", tm, [], Path(tempfile.mkdtemp()), "snappy")+ self.assertTrue(any("never called the web-search API" in v for v in c["violations"]))++ def test_memory_arm_clean(self):+ tm = self.tm([])+ c = pov.contamination("A0", tm, [], Path(tempfile.mkdtemp()), "snappy")+ self.assertEqual(c["violations"], [])+++if __name__ == "__main__":+ unittest.main()diff --git a/benchmarks/hub-pov/matrix/smoke.json b/benchmarks/hub-pov/matrix/smoke.jsonnew file mode 100644index 000000000..4cadb9cb7--- /dev/null+++ b/benchmarks/hub-pov/matrix/smoke.json@@ -0,0 +1,8 @@+{+ "_comment": "SMOKE ONLY (2026-10-17): 1 repeat x 2 tasks x every arm, cheap model. Validates the harness; its numbers are NOT results (SMOKE.md). C2:lz4.smoke:S1 is a smoke fixture, not a pre-registered task.",+ "model": "sonnet",+ "repeats": 1,+ "allow_smoke_fixtures": true,+ "arms": ["A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7"],+ "cells": ["C1:snappy:T4", "C2:lz4.smoke:S1"]+}diff --git a/benchmarks/hub-pov/results/schema.json b/benchmarks/hub-pov/results/schema.jsonnew file mode 100644index 000000000..1ed71bd28--- /dev/null+++ b/benchmarks/hub-pov/results/schema.json@@ -0,0 +1,272 @@+{+ "$schema": "https://json-schema.org/draft/2020-12/schema",+ "$id": "hub-pov/score/1",+ "title": "One scored cell (arm x task(s) x repeat) of the Corpus Hub v2 proof-of-value run",+ "description": "Written by `pov.py score` to <run>/results/<run_id>/<arm>/<cell>/r<n>/score.json and appended to scores.jsonl. Metric definitions: PLAN.md \u00a76.",+ "type": "object",+ "required": [+ "schema",+ "run_id",+ "arm",+ "tasks",+ "repeat",+ "valid",+ "correct",+ "grades",+ "cost_usd",+ "output_tokens",+ "turns",+ "n_result_events",+ "wall_s",+ "load_before",+ "bytes_downloaded",+ "fetches",+ "failed_fetches",+ "xerj_calls",+ "routing",+ "contamination"+ ],+ "properties": {+ "schema": {+ "const": "hub-pov/score/1"+ },+ "run_id": {+ "type": "string"+ },+ "arm": {+ "enum": [+ "A0",+ "A1",+ "A2",+ "A3",+ "A4",+ "A5",+ "A6",+ "A7"+ ]+ },+ "tasks": {+ "type": "array",+ "items": {+ "type": "string",+ "pattern": "^C[1-4]:[^:]+:[^:]+$"+ },+ "minItems": 1,+ "description": "one ref = single-question cell; >1 = a C4 session in one agent run"+ },+ "repeat": {+ "type": "integer",+ "minimum": 1+ },+ "model": {+ "type": [+ "string",+ "null"+ ],+ "description": "resolved model id from the transcript init event"+ },+ "model_requested": {+ "type": "string"+ },+ "valid": {+ "type": "boolean",+ "description": "no contamination violation, >=1 result event, claude exit 0. An invalid cell is excluded from comparisons and reported."+ },+ "correct": {+ "type": [+ "number",+ "null"+ ],+ "minimum": 0,+ "maximum": 1,+ "description": "fraction of the cell's tasks passed; null when any grade is unscored (PLAN \u00a76)"+ },+ "grades": {+ "type": "array",+ "items": {+ "type": "object",+ "required": [+ "ref",+ "grader",+ "status"+ ],+ "properties": {+ "ref": {+ "type": "string"+ },+ "grader": {+ "type": "string"+ },+ "status": {+ "enum": [+ "pass",+ "fail",+ "unscored"+ ]+ },+ "status_raw": {+ "enum": [+ "pass",+ "fail",+ "unscored"+ ]+ },+ "misroute_override": {+ "type": "boolean",+ "description": "H2: a pass turned into fail because the v2 router picked the wrong corpus"+ }+ }+ }+ },+ "cost_usd": {+ "type": "number",+ "description": "sum of total_cost_usd over ALL result events"+ },+ "output_tokens": {+ "type": "integer",+ "description": "sum of usage.output_tokens over ALL result events"+ },+ "input_tokens_all": {+ "type": "integer"+ },+ "turns": {+ "type": "integer",+ "description": "sum of num_turns over result events"+ },+ "n_result_events": {+ "type": "integer",+ "minimum": 0+ },+ "claude_rc": {+ "type": [+ "integer",+ "null"+ ]+ },+ "wall_s": {+ "type": [+ "number",+ "null"+ ],+ "description": "the agent process only (A7's pre-pull is in setup)"+ },+ "load_before": {+ "type": [+ "array",+ "null"+ ],+ "items": {+ "type": "string"+ },+ "description": "1/5/15-min load average at agent start"+ },+ "load_after": {+ "type": [+ "array",+ "null"+ ],+ "items": {+ "type": "string"+ }+ },+ "bytes_downloaded": {+ "type": [+ "integer",+ "null"+ ],+ "description": "bytes received by the jail's interface while the agent ran, minus model-API bytes (nft counter on 160.79.104.0/21)"+ },+ "bytes_rx_iface": {+ "type": [+ "integer",+ "null"+ ]+ },+ "bytes_model_api": {+ "type": "integer"+ },+ "bytes_rx_jail_total": {+ "type": [+ "integer",+ "null"+ ]+ },+ "pack_bytes_cached": {+ "type": "integer",+ "description": "size of ~/.xerj/hub/packs at the end (A6 pull / A7 pre-pull)"+ },+ "fetches": {+ "type": "integer"+ },+ "failed_fetches": {+ "type": "integer",+ "description": "HEURISTIC: empty, error, captcha/bot page, 403/429 (PLAN \u00a76)"+ },+ "xerj_calls": {+ "type": "integer",+ "description": "invocations seen by the logging shim"+ },+ "answered_from": {+ "type": "array",+ "items": {+ "type": "string"+ }+ },+ "routing": {+ "type": [+ "object",+ "null"+ ],+ "description": "A6/A7 only: first v2 question's routing (H2)",+ "properties": {+ "status": {+ "type": "string"+ },+ "ok": {+ "type": "boolean"+ }+ }+ },+ "setup": {+ "type": [+ "object",+ "null"+ ],+ "description": "A7 only: the pre-pull (rc, seconds, rx_bytes) \u2014 excluded from bytes_downloaded, reported for the first-use disclosure (PLAN \u00a77)"+ },+ "contamination": {+ "type": "object",+ "required": [+ "violations",+ "notes",+ "tools_used"+ ],+ "properties": {+ "violations": {+ "type": "array",+ "items": {+ "type": "string"+ }+ },+ "notes": {+ "type": "array",+ "items": {+ "type": "string"+ }+ },+ "tools_used": {+ "type": "array",+ "items": {+ "type": "string"+ }+ },+ "xerj_call_kinds": {+ "type": "array",+ "items": {+ "type": "string"+ }+ }+ }+ }+ }+}\ No newline at end of filediff --git a/benchmarks/hub-pov/sandbox.sh b/benchmarks/hub-pov/sandbox.shindex f1c36571e..1f46f86a4 100755--- a/benchmarks/hub-pov/sandbox.sh+++ b/benchmarks/hub-pov/sandbox.sh@@ -12,18 +12,46 @@# we bind writable, so an arm cannot modify the host, the repo, or another arm.# * --die-with-parent tears the jail down if the harness dies.# Every `claude -p` arm (A0-A7 in PLAN.md) MUST be launched through this script.-# Network is deliberately NOT unshared — arms need it (git clone, web fetch, the-# hub pull, the search APIs). Isolation here is of the filesystem and the process-# table, not the network.+#+# WHAT THE 2026-10-17 SMOKE RUN ADDED (SMOKE.md "harness defects"), each one a way+# an arm could leak, be contaminated, or touch the host that the first version left+# open:+# * PRIVATE NETWORK NAMESPACE (default `--net isolated`). The jail runs inside+# `pasta` (user-mode networking: the arm keeps internet access) with NO port+# forwarding in either direction and no gateway->host-loopback mapping, plus an+# nftables rule rejecting RFC1918/CGNAT/link-local destinations. Without this an+# arm shares the host's loopback: `xerj ask` probes http://localhost:9200 FIRST+# and uses whatever answers — on this host that is the user's reference-coding+# node, so a v2 arm would have restored a pack INTO the user's node and answered+# from its corpora (contamination and a write to a node we must never touch).+# Inside the namespace :9200 is the arm's own, so XERJ arms can bind it safely.+# * /home and /root are MASKED (tmpfs); only the HOME, the auth dir and the paths+# passed with --ro are visible. With a bare `--ro-bind / /` every arm could read+# the gold answers (benchmarks/corpus-tasks, prior run dirs), the local corpus+# clones in ~/.xerj-code, other sessions' transcripts, and secrets in .env files.+# * The CLAUDE_CONFIG_DIR is bound writable (token refresh) but everything in it+# except `.credentials.json` and `.claude.json` is masked: the user's global+# CLAUDE.md, skills, plugins, hooks, transcripts and history would otherwise be+# loaded into or readable by every arm, and settings.json `env` would inject the+# user's GH_TOKEN into the arm despite --clearenv. settings.json is replaced by `{}`.+# * Inbound bytes are counted per jail (nft counters): all non-loopback bytes, and+# the subset from the model API (Anthropic, 160.79.104.0/21) so the harness can+# report "bytes downloaded" net of model traffic. Written to+# $HOME/.sandbox/net.json when the jail exits.## USAGE:-# benchmarks/hub-pov/sandbox.sh [--home DIR] [--rm-home] -- CMD [ARGS...]+# benchmarks/hub-pov/sandbox.sh [--home DIR] [--rm-home] [--ro PATH]...+# [--net isolated|host] -- CMD [ARGS...]# --home DIR use DIR as the arm's fresh HOME (default: a new `mktemp -d`).# Must NOT be the real user's home; the script refuses that.# --rm-home delete the HOME on exit (default: keep it so the harness can# collect the transcript/artifacts, then clean up itself).-# Everything after `--` is the command run inside the jail, e.g.-# sandbox.sh --home "$cell" -- claude -p --output-format stream-json "$prompt"+# --ro PATH expose a host PATH read-only at the same path (repeatable). Needed+# for anything under /home (the claude binary, the xerj binary, shim+# dirs, the catalogue override). NEVER pass a dir holding gold answers.+# --net MODE `isolated` (default; needs pasta + nft) or `host` (the old shared+# network — only for debugging the jail itself, never for an arm).+# Everything after `--` is the command run inside the jail.# The resolved HOME is printed to stderr as `sandbox-home: <dir>` for the harness.## ENV (optional, read from the host before the jail clears the environment):@@ -31,8 +59,8 @@# Bound writable because the CLI refreshes its token in place;# do NOT copy .credentials.json into the HOME instead — a copy# can refresh out from under the main session and log it out.-# XERJ_POV_PATH_PREFIX dir prepended to PATH inside the jail (e.g. the logging-# `xerj` shim that counts invocations for the XERJ arms).+# XERJ_POV_PATH_PREFIX dir(s, ':'-separated) prepended to PATH inside the jail (e.g.+# the logging `xerj` shim). Each is exposed read-only.# No host secret is forwarded: the jail starts with --clearenv and we set back only# HOME, CLAUDE_CONFIG_DIR, PATH, TERM, LANG. GH_TOKEN / ANTHROPIC_API_KEY are NOT# passed in — arms authenticate via CLAUDE_CONFIG_DIR, and the LLM judge runs@@ -44,12 +72,19 @@ XERJ_POV_PATH_PREFIX="${XERJ_POV_PATH_PREFIX:-}"SANDBOX_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"HOME_DIR=""RM_HOME=0+NET=isolated+IN_NETNS=0+RO_PATHS=()+ORIG_ARGS=("$@")# ---- parse args ----------------------------------------------------------while [ $# -gt 0 ]; docase "$1" in--home) HOME_DIR="${2:?--home needs a dir}"; shift 2 ;;--rm-home) RM_HOME=1; shift ;;+ --ro) RO_PATHS+=("${2:?--ro needs a path}"); shift 2 ;;+ --net) NET="${2:?--net needs isolated|host}"; shift 2 ;;+ --_in-netns) IN_NETNS=1; shift ;; # internal: re-exec'd inside pasta--) shift; break ;;*) echo "sandbox.sh: unknown arg '$1' (did you forget '--' before the command?)" >&2; exit 2 ;;esac@@ -57,6 +92,7 @@ doneif [ $# -eq 0 ]; thenecho "sandbox.sh: no command after '--'" >&2; exit 2fi+case "$NET" in isolated|host) ;; *) echo "sandbox.sh: --net must be isolated or host" >&2; exit 2 ;; esac# ---- preconditions (fail closed) -----------------------------------------command -v bwrap >/dev/null 2>&1 || { echo "sandbox.sh: bwrap (bubblewrap) not found — refusing to run an arm unsandboxed" >&2; exit 3; }@@ -70,19 +106,62 @@ HOME_DIR="$(cd "$HOME_DIR" && pwd -P)" # absolute, resolved# Guard: the HOME must be a fresh dir, never the real home or the filesystem root.# Binding the real home writable would defeat the whole point of --ro-bind / /.case "$HOME_DIR" in- / | /home/claude | /home/claude/ | /root | /root/ | "${HOME:-/nonexistent}")+ / | /home | /home/ | /home/claude | /home/claude/ | /root | /root/ | "${HOME:-/nonexistent}")echo "sandbox.sh: refusing to use '$HOME_DIR' as a scratch HOME (not a fresh dir)" >&2; exit 3 ;;esac-mkdir -p "$HOME_DIR/work"++# ---- network namespace (re-exec under pasta) ------------------------------+if [ "$NET" = isolated ] && [ "$IN_NETNS" -eq 0 ]; then+ command -v pasta >/dev/null 2>&1 || { echo "sandbox.sh: pasta not found — refusing to run an arm on the host network (use --net host only to debug the jail)" >&2; exit 3; }+ command -v nft >/dev/null 2>&1 || { echo "sandbox.sh: nft not found — refusing to run an arm without the private-range block" >&2; exit 3; }+ # -t/-u none: nothing forwarded host->ns. -T/-U none: nothing forwarded ns->host+ # loopback (pasta's default `auto` would forward every port the host has bound on+ # 127.0.0.1, i.e. exactly the reference node). --no-map-gw: the gateway address+ # does not lead to the host either.+ # pasta puts us in a user namespace as uid 0; remember the real ids so the jail+ # maps back to them (claude refuses --dangerously-skip-permissions as root).+ export SANDBOX_HOST_UID="$(id -u)" SANDBOX_HOST_GID="$(id -g)"+ exec pasta --config-net -q -t none -u none -T none -U none --no-map-gw \+ -- bash "$(realpath "$0")" --_in-netns --home "$HOME_DIR" "${ORIG_ARGS[@]}"+fi++mkdir -p "$HOME_DIR/work" "$HOME_DIR/.sandbox"echo "sandbox-home: $HOME_DIR" >&2+if [ "$IN_NETNS" -eq 1 ]; then+ # We are root in pasta's user+net namespace: block private destinations (another+ # route to a host service bound on 0.0.0.0, e.g. via docker0) and count bytes.+ nft -f - <<'EOF'+table inet povjail {+ counter rx_all {}+ counter rx_model_api {}+ chain out {+ type filter hook output priority 0; policy accept;+ oifname "lo" accept+ ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 } reject+ ip6 daddr { fc00::/7, fe80::/10 } reject+ }+ chain in {+ type filter hook input priority 0; policy accept;+ iifname "lo" accept+ counter name rx_all+ ip saddr 160.79.104.0/21 counter name rx_model_api+ }+}+EOF+ # The host's resolv.conf points at systemd-resolved on 127.0.0.53, which is the+ # HOST loopback and unreachable from here: give the jail public resolvers.+ printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > "$HOME_DIR/.sandbox/resolv.conf"+fi+PATH_IN="$SANDBOX_PATH"[ -n "$XERJ_POV_PATH_PREFIX" ] && PATH_IN="$XERJ_POV_PATH_PREFIX:$PATH_IN"# ---- the jail -------------------------------------------------------------# ro-bind the whole host read-only; carve out exactly two writable spots:+# ro-bind the whole host read-only; mask /home and /root; carve out:# - the fresh HOME (the arm's whole writable world, cwd = $HOME/work)-# - the auth/config dir (token refresh writes here; kept as narrow as possible)+# - the auth/config dir (token refresh writes here), with its contents masked+# - the --ro paths and the PATH-prefix dirs, read-only# Fresh tmpfs /tmp and minimal /dev /proc; --unshare-pid for the private PID ns;# --clearenv so no host env (secrets, GH_TOKEN) leaks in, then set back the minimum.BWRAP=(bwrap@@ -90,9 +169,35 @@ BWRAP=(bwrap--dev /dev--proc /proc--tmpfs /tmp- --bind "$HOME_DIR" "$HOME_DIR"- --bind "$CLAUDE_CONFIG_DIR" "$CLAUDE_CONFIG_DIR"- --unshare-pid+ --tmpfs /home+ --tmpfs /root)+if [ "$IN_NETNS" -eq 1 ]; then+ BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/resolv.conf" "$(realpath /etc/resolv.conf)")+fi+BWRAP+=(--bind "$HOME_DIR" "$HOME_DIR"+ --bind "$CLAUDE_CONFIG_DIR" "$CLAUDE_CONFIG_DIR")+# Mask the config dir's contents except the two files auth needs.+printf '{}\n' > "$HOME_DIR/.sandbox/settings.json"+: > "$HOME_DIR/.sandbox/empty" # a regular empty file: a /dev/null bind reads as EACCES here+for entry in "$CLAUDE_CONFIG_DIR"/* "$CLAUDE_CONFIG_DIR"/.[!.]*; do+ [ -e "$entry" ] || continue+ case "$(basename "$entry")" in+ .credentials.json|.claude.json) continue ;;+ settings.json) BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/settings.json" "$entry") ;;+ *) if [ -d "$entry" ] && [ ! -L "$entry" ]; then BWRAP+=(--tmpfs "$entry")+ elif [ -f "$entry" ] && [ ! -L "$entry" ]; then BWRAP+=(--ro-bind "$HOME_DIR/.sandbox/empty" "$entry"); fi ;;+ esac+done+IFS=: read -r -a prefix_dirs <<< "$XERJ_POV_PATH_PREFIX"+for p in "${RO_PATHS[@]}" "${prefix_dirs[@]}"; do+ [ -n "$p" ] || continue+ [ -e "$p" ] || { echo "sandbox.sh: --ro path does not exist: $p" >&2; exit 2; }+ BWRAP+=(--ro-bind "$p" "$p")+done+if [ "$IN_NETNS" -eq 1 ]; then+ BWRAP+=(--unshare-user --uid "${SANDBOX_HOST_UID:?}" --gid "${SANDBOX_HOST_GID:?}")+fi+BWRAP+=(--unshare-pid--die-with-parent--new-session--chdir "$HOME_DIR/work"@@ -103,11 +208,17 @@ BWRAP=(bwrap--setenv TERM xterm--setenv LANG C.UTF-8)-# When --rm-home is set we must run bwrap as a child (not exec) so the EXIT trap-# that deletes the HOME can fire; otherwise exec for clean signal propagation.-if [ "$RM_HOME" -eq 1 ]; then- trap 'rm -rf "$HOME_DIR"' EXIT+write_counters() {+ [ "$IN_NETNS" -eq 1 ] || return 0+ nft -j list counters table inet povjail > "$HOME_DIR/.sandbox/net.json" 2>/dev/null || true+}++# Run bwrap as a child (not exec) when we must act after it: delete the HOME+# (--rm-home) or read the namespace's byte counters.+if [ "$RM_HOME" -eq 1 ] || [ "$IN_NETNS" -eq 1 ]; then+ [ "$RM_HOME" -eq 1 ] && trap 'rm -rf "$HOME_DIR"' EXITrc=0; "${BWRAP[@]}" "$@" || rc=$?+ write_countersexit "$rc"elseexec "${BWRAP[@]}" "$@"diff --git a/benchmarks/hub-pov/tasks/C1/SOURCE.json b/benchmarks/hub-pov/tasks/C1/SOURCE.jsonnew file mode 100644index 000000000..53dd2db7d--- /dev/null+++ b/benchmarks/hub-pov/tasks/C1/SOURCE.json@@ -0,0 +1,11 @@+{+ "source_branch": "bench/corpus-tasks",+ "source_commit": "0ca4065a56ae82dba9eb03599063e0ddc277228a",+ "note": "Verbatim copies of the registered C1 suites (PLAN.md \u00a74 C1: reuse, do not invent). `pov.py verify-c1` re-checks each blob against the branch.",+ "files": {+ "tasks/lz4.json": "dbdaef8170594162f43f0fc3d20be44774f7a8e5",+ "tasks/snappy.json": "4cac10708240eda2cb665ea93c72d33ca94858eb",+ "tasks/zstd.json": "18ed1f7da441ab7bae0750c8571dad545ec3c105",+ "harness/check.py": "7d55ff398c2cc9ead5016cf9df88af44a9aa8305"+ }+}\ No newline at end of filediff --git a/benchmarks/hub-pov/tasks/C1/lz4.json b/benchmarks/hub-pov/tasks/C1/lz4.jsonnew file mode 100644index 000000000..dbdaef817--- /dev/null+++ b/benchmarks/hub-pov/tasks/C1/lz4.json@@ -0,0 +1,144 @@+{+ "corpus": "lz4",+ "subject": "lz4",+ "pin": "0774d05537",+ "generated": "2026-10-02",+ "tasks": [+ {+ "id": "T1",+ "class": "drift",+ "name": "LZ4F_ENDMARK_SIZE",+ "kind": "added",+ "changed": "2024-07-24",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of LZ4F_ENDMARK_SIZE? Answer with the value only.",+ "re": [+ "0x4|4"+ ],+ "need": 1+ },+ {+ "id": "T2",+ "class": "drift",+ "name": "LZ4_VERSION_MINOR",+ "kind": "changed",+ "changed": "2024-07-14",+ "olds": [+ "9"+ ],+ "q": "In the current development source of lz4, what is the exact value of LZ4_VERSION_MINOR? Answer with the value only.",+ "re": [+ "0xa|10"+ ],+ "need": 1+ },+ {+ "id": "T3",+ "class": "drift",+ "name": "LZ4_VERSION_RELEASE",+ "kind": "changed",+ "changed": "2024-07-14",+ "olds": [+ "5"+ ],+ "q": "In the current development source of lz4, what is the exact value of LZ4_VERSION_RELEASE? Answer with the value only.",+ "re": [+ "0|0x0"+ ],+ "need": 1+ },+ {+ "id": "T4",+ "class": "drift",+ "name": "ENV_CLEVEL",+ "kind": "added",+ "changed": "2024-07-08",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of ENV_CLEVEL? Answer with the value only.",+ "re": [+ "LZ4_CLEVEL"+ ],+ "need": 1+ },+ {+ "id": "T5",+ "class": "drift",+ "name": "ENV_NBTHREADS",+ "kind": "added",+ "changed": "2024-07-06",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of ENV_NBTHREADS? Answer with the value only.",+ "re": [+ "LZ4_NBWORKERS"+ ],+ "need": 1+ },+ {+ "id": "T6",+ "class": "drift",+ "name": "OUTBUFF_QUEUE",+ "kind": "added",+ "changed": "2024-07-05",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of OUTBUFF_QUEUE? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T7",+ "class": "drift",+ "name": "NB_BUFFSETS",+ "kind": "added",+ "changed": "2024-07-05",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of NB_BUFFSETS? Answer with the value only.",+ "re": [+ "0x4|4"+ ],+ "need": 1+ },+ {+ "id": "T8",+ "class": "drift",+ "name": "DISTRIB_SIZE_MAX",+ "kind": "added",+ "changed": "2024-02-20",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of DISTRIB_SIZE_MAX? Answer with the value only.",+ "re": [+ "0x28a|650"+ ],+ "need": 1+ },+ {+ "id": "T9",+ "class": "drift",+ "name": "LZ4MID_HASHSIZE",+ "kind": "added",+ "changed": "2024-02-07",+ "olds": [],+ "q": "In the current development source of lz4, what is the exact value of LZ4MID_HASHSIZE? Answer with the value only.",+ "re": [+ "0x8|8"+ ],+ "need": 1+ },+ {+ "id": "T10",+ "class": "drift",+ "name": "LZ4HC_CLEVEL_MIN",+ "kind": "changed",+ "changed": "2024-02-07",+ "olds": [+ "3"+ ],+ "q": "In the current development source of lz4, what is the exact value of LZ4HC_CLEVEL_MIN? Answer with the value only.",+ "re": [+ "0x2|2"+ ],+ "need": 1+ }+ ]+}diff --git a/benchmarks/hub-pov/tasks/C1/snappy.json b/benchmarks/hub-pov/tasks/C1/snappy.jsonnew file mode 100644index 000000000..4cac10708--- /dev/null+++ b/benchmarks/hub-pov/tasks/C1/snappy.json@@ -0,0 +1,127 @@+{+ "corpus": "snappy",+ "subject": "snappy",+ "pin": "9c28114a38",+ "generated": "2026-10-02",+ "tasks": [+ {+ "id": "T1",+ "class": "drift",+ "name": "SNAPPY_MIN_COMPRESSION_LEVEL",+ "kind": "added",+ "changed": "2026-09-09",+ "olds": [],+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_MIN_COMPRESSION_LEVEL? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T2",+ "class": "drift",+ "name": "SNAPPY_MAX_COMPRESSION_LEVEL",+ "kind": "added",+ "changed": "2026-09-09",+ "olds": [],+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_MAX_COMPRESSION_LEVEL? Answer with the value only.",+ "re": [+ "0x2|2"+ ],+ "need": 1+ },+ {+ "id": "T3",+ "class": "drift",+ "name": "SNAPPY_HAVE_RVV",+ "kind": "changed",+ "changed": "2026-05-09",+ "olds": [+ "0"+ ],+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_RVV? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T4",+ "class": "drift",+ "name": "kInputMarginBytes",+ "kind": "added",+ "changed": "2024-04-04",+ "olds": [],+ "q": "In the current development source of snappy, what is the exact value of kInputMarginBytes? Answer with the value only.",+ "re": [+ "0xf|15"+ ],+ "need": 1+ },+ {+ "id": "T5",+ "class": "drift",+ "name": "kMaxHashTableBits",+ "kind": "changed",+ "changed": "2024-04-04",+ "olds": [+ "14"+ ],+ "q": "In the current development source of snappy, what is the exact value of kMaxHashTableBits? Answer with the value only.",+ "re": [+ "0xf|15"+ ],+ "need": 1+ },+ {+ "id": "T6",+ "class": "state",+ "name": "SNAPPY_HAVE_VECTOR_BYTE_SHUFFLE",+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_VECTOR_BYTE_SHUFFLE? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T7",+ "class": "state",+ "name": "SNAPPY_HAVE_NEON_CRC32",+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_NEON_CRC32? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T8",+ "class": "state",+ "name": "SNAPPY_HAVE_X86_CRC32",+ "q": "In the current development source of snappy, what is the exact value of SNAPPY_HAVE_X86_CRC32? Answer with the value only.",+ "re": [+ "0x1|1"+ ],+ "need": 1+ },+ {+ "id": "T9",+ "class": "state",+ "name": "kMinHashTableBits",+ "q": "In the current development source of snappy, what is the exact value of kMinHashTableBits? Answer with the value only.",+ "re": [+ "0x8|8"+ ],+ "need": 1+ },+ {+ "id": "T10",+ "class": "state",+ "name": "kMaximumTagLength",+ "q": "In the current development source of snappy, what is the exact value of kMaximumTagLength? Answer with the value only.",+ "re": [+ "0x5|5"+ ],+ "need": 1+ }+ ]+}diff --git a/benchmarks/hub-pov/tasks/C1/zstd.json b/benchmarks/hub-pov/tasks/C1/zstd.jsonnew file mode 100644index 000000000..18ed1f7da--- /dev/null+++ b/benchmarks/hub-pov/tasks/C1/zstd.json@@ -0,0 +1,138 @@+{+ "corpus": "zstd",+ "subject": "zstd",+ "pin": "01b7154f11",+ "generated": "2026-10-02",+ "tasks": [+ {+ "id": "T1",+ "class": "drift",+ "name": "X86_FEATURE_BMI2",+ "kind": "added",+ "changed": "2026-09-14",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of X86_FEATURE_BMI2? Answer with the value only.",+ "re": [+ "0|0x0"+ ],+ "need": 1+ },+ {+ "id": "T2",+ "class": "drift",+ "name": "FRAME_V01_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V01_SIZE? Answer with the value only.",+ "re": [+ "0xbd|189"+ ],+ "need": 1+ },+ {+ "id": "T3",+ "class": "drift",+ "name": "FRAME_V02_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V02_SIZE? Answer with the value only.",+ "re": [+ "0xbb|187"+ ],+ "need": 1+ },+ {+ "id": "T4",+ "class": "drift",+ "name": "FRAME_V03_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V03_SIZE? Answer with the value only.",+ "re": [+ "0xbb|187"+ ],+ "need": 1+ },+ {+ "id": "T5",+ "class": "drift",+ "name": "FRAME_V04_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V04_SIZE? Answer with the value only.",+ "re": [+ "0xc6|198"+ ],+ "need": 1+ },+ {+ "id": "T6",+ "class": "drift",+ "name": "FRAME_V05_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V05_SIZE? Answer with the value only.",+ "re": [+ "0xb8|184"+ ],+ "need": 1+ },+ {+ "id": "T7",+ "class": "drift",+ "name": "FRAME_V06_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V06_SIZE? Answer with the value only.",+ "re": [+ "0xb2|178"+ ],+ "need": 1+ },+ {+ "id": "T8",+ "class": "drift",+ "name": "FRAME_V07_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V07_SIZE? Answer with the value only.",+ "re": [+ "0xb2|178"+ ],+ "need": 1+ },+ {+ "id": "T9",+ "class": "drift",+ "name": "FRAME_V08_SIZE",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of FRAME_V08_SIZE? Answer with the value only.",+ "re": [+ "0xb3|179"+ ],+ "need": 1+ },+ {+ "id": "T10",+ "class": "drift",+ "name": "OLDEST_STREAMABLE_VERSION",+ "kind": "added",+ "changed": "2026-09-11",+ "olds": [],+ "q": "In the current development source of zstd, what is the exact value of OLDEST_STREAMABLE_VERSION? Answer with the value only.",+ "re": [+ "0x4|4"+ ],+ "need": 1+ }+ ]+}diff --git a/benchmarks/hub-pov/tasks/C2/lz4.smoke.json b/benchmarks/hub-pov/tasks/C2/lz4.smoke.jsonnew file mode 100644index 000000000..a60c435dd--- /dev/null+++ b/benchmarks/hub-pov/tasks/C2/lz4.smoke.json@@ -0,0 +1,32 @@+{+ "class": "C2",+ "corpus": "lz4",+ "subject": "lz4",+ "pin": "0774d05537f9762f838f7ab541b7765f1a729cb5",+ "pin_date": "2026-10-02",+ "smoke_only": true,+ "provenance": "SMOKE FIXTURE ONLY — NOT a pre-registered C2 task. Written by the harness agent on 2026-10-17 to exercise the C2 judge path end to end; PLAN.md §5 requires C2 tasks to be authored by a person from the pinned source and committed before the first C2 run, so this file must never be used in the full run (pov.py refuses smoke_only tasks unless --allow-smoke-fixtures is passed).",+ "tasks": [+ {+ "id": "S1",+ "q": "In the lz4 frame format, which integrity checks can a frame carry, which of them are optional, what does each one cover, and how is each computed? Answer concisely.",+ "gold": "Three checks, all xxHash-32 with seed 0. (1) Header checksum: always present, one byte at the end of the frame descriptor = (xxh32(descriptor) >> 8) & 0xFF, i.e. the second byte of the xxh32 of the full frame descriptor including optional fields. (2) Block checksum: optional (FLG block-checksum flag, bit 4); 4 bytes little-endian after each data block, xxh32 of the raw (compressed/undecoded) block data. (3) Content checksum: optional (FLG content-checksum flag, bit 2); 4 bytes after the EndMark, xxh32 of the whole decompressed content.",+ "rubric": [+ "Names xxHash-32 (xxh32) as the checksum function for the checks",+ "Header (descriptor) checksum: always present, one byte, derived from xxh32 of the frame descriptor (second byte / >>8)",+ "Block checksum: optional, per data block, computed over the raw/compressed block data",+ "Content checksum: optional, appended after the end mark, computed over the entire decompressed content"+ ],+ "pass_rule": "all",+ "files": [+ "doc/lz4_Frame_format.md:91-104 (Content Checksum)",+ "doc/lz4_Frame_format.md:164-183 (block / content checksum flags)",+ "doc/lz4_Frame_format.md:243-249 (Header Checksum)",+ "doc/lz4_Frame_format.md:294-303 (Block checksum)",+ "lib/lz4frame.c:351-355 (LZ4F_headerChecksum)",+ "lib/lz4frame.c:918-923 (block checksum over compressed data)",+ "lib/lz4frame.c:1116 (content checksum on compressEnd)"+ ]+ }+ ]+}