A self-improving RLM agent for coding workflows and long-running autonomous tasks.
Rust
[RSI] swarm-fanout capability eval harness (#2283)
Seth Karten6af0e1a4efdf
diff --git a/packages/coding-agent/docs/development.md b/packages/coding-agent/docs/development.mdindex 56aa1a74a4..443834c371 100644--- a/packages/coding-agent/docs/development.md+++ b/packages/coding-agent/docs/development.md@@ -81,3 +81,15 @@ npx tsx ../../node_modules/vitest/dist/cli.js --run test/specific.test.ts```If you create or modify a test file, run that file and iterate until it passes. Coding-agent suite regressions belong under `test/suite/regressions/` and use the suite harness and faux provider rather than live provider credentials.++## Capability Evals++End-to-end capability evals live under `scripts/evals/` and are not part of CI: a real-model run is a manual step with credentials in the environment. Each harness ships model-free self-tests that validate its fixtures and rubric without any model call - run them from the eval directory:++```bash+cd scripts/evals/swarm_fanout+uv run --locked ruff check .+uv run --locked python -m unittest discover -s tests -v+```++See `scripts/evals/README.md` for the rubric and the real-model run instructions.diff --git a/scripts/evals/README.md b/scripts/evals/README.mdnew file mode 100644index 0000000000..4ed8d1270b--- /dev/null+++ b/scripts/evals/README.md@@ -0,0 +1,93 @@+# Prime Agent capability evals++These harnesses measure end-to-end agent capability with a real model.+They complement the repo's unit tests (which never call a model) and the+PR performance benchmarks (which measure startup, not capability). They+never run in CI and never need credentials to validate: every harness is+exercised by model-free self-tests, and a real-model run is a manual+step.++## swarm-fanout++Measures multi-agent orchestration under one prompt: decompose a job+into per-shard child tasks, fan out one RLM subagent per shard, fan the+answers back in, and account for child failures. Everything is scored+from on-disk artifacts - the RLM spawn ledger, sub-* child session+dirs, the parent session transcript, and the combined answer artifact -+never from the agent's own claims.++1. **Fixtures** (`swarm_fanout/fixtures/`) are two self-contained sets+ (JSONL event logs, CSV host metrics) of eight stdlib-only data shards+ each, with a machine-checkable question and expected answer per shard.+ `task.txt` is the orchestration prompt; `fixture.json` is the+ scoring manifest and stays out of the agent's repo copy.+2. **Runner** (`swarm_fanout/runner.py`) copies a fixture to a temp git+ repo, runs the parent agent headless (`--mode json`, `--daemon-socket`,+ `--session-dir`, isolated agent home), then records the post-state.+ The ledger path is computed exactly as the product does:+ `<agentHome>/rlm-ledger/<sha256(canonical sessions dir)[:16]>.jsonl`.+3. **Scorer** (`swarm_fanout/scorer.py`) applies the rubric:+ - **coverage**: every shard's answer in `combined-index.md` matches+ the machine-computed expected value;+ - **delegation evidence**: a live depth-1 ledger edge for every+ shard's worker name, each edge backed by its real child session dir+ (the edge's recorded child file must exist in the collected sub-*+ dirs) - helper-named spawns do not substitute, and a parent that+ answers everything itself fails here;+ - **dedup**: no shard's worker name is spawned twice while another+ child for that shard is live, and total depth-1 spawns stay within+ one retry per shard (2x the shard count);+ - **receipts**: every depth-1 child has a parent-visible reply or an+ explicit failure/terminal notice in the parent transcript;+ - **efficiency**: tokens, turns, and wall time (informational until a+ single-agent baseline datapoint exists).++A run counts as resolved only when coverage, delegation, dedup, and+receipts all pass.++### Running a real-model eval++The eval needs a model selector and provider credentials in the+environment (the agent's normal auth):++```+cd scripts/evals/swarm_fanout+uv run --locked python runner.py --fixture fixtures/json-events --model anthropic/claude-sonnet-4-5+uv run --locked python runner.py --fixture fixtures/csv-metrics --model anthropic/claude-sonnet-4-5+```++The runner prints the scored outcome as JSON and exits 0 only when the+run is resolved. `--timeout` (default 1800s) bounds the agent run.++Real-model runs should be planned against one constraint: a one-shot+`--mode json` session is client-owned, and when the CLI process exits+the daemon stops its worker (after a 30s grace), taking still-running+children with it. The task prompt therefore requires the parent to keep+its run active until every child has replied. If a real-model datapoint+shows models reliably ending their turn before the fan-in completes,+drive the parent as a resident session over the daemon protocol+(`create` + `prompt` + `wait_for_headless_completion` with+`waitForRlmQuiescence: true`) instead and re-check; the scorer needs no+changes either way, since it reads only the artifacts.++### Self-tests++Validate the harness with the model-free self-tests (fixture integrity+with independently recomputed answers, every scorer rule plus the+no-spawn, double-spawn, fabricated-answer, and silent-drop cheats,+ledger-path computation, and a stub-agent end-to-end run) - never a+model call:++```+uv run --locked ruff check .+uv run --locked ruff format --check .+uv run --locked python -m unittest discover -s tests -v+```++### Adding a fixture++Copy an existing fixture directory, keep the shape: `shards/` (data+files with machine-checkable answers), `fixture.json` (name, artifact,+shards with file/worker/question/expected), and `task.txt` (orchestration+prompt naming every shard and its question). Extend the fixture+integrity tests with an independent recompute for the new questions.diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.json b/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.jsonnew file mode 100644index 0000000000..888ddaf122--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.json@@ -0,0 +1,54 @@+{+ "name": "csv-metrics",+ "artifact": "combined-index.md",+ "shards": [+ {+ "file": "metrics-01.csv",+ "worker": "worker-metrics-01",+ "question": "What is the maximum value of the cpu column?",+ "expected": 85+ },+ {+ "file": "metrics-02.csv",+ "worker": "worker-metrics-02",+ "question": "What is the total of the latency_ms column over rows with cpu >= 90?",+ "expected": 37+ },+ {+ "file": "metrics-03.csv",+ "worker": "worker-metrics-03",+ "question": "How many rows have memory > 80?",+ "expected": 3+ },+ {+ "file": "metrics-04.csv",+ "worker": "worker-metrics-04",+ "question": "What is the total of the latency_ms column over all rows?",+ "expected": 287+ },+ {+ "file": "metrics-05.csv",+ "worker": "worker-metrics-05",+ "question": "How many rows have cpu >= 90?",+ "expected": 1+ },+ {+ "file": "metrics-06.csv",+ "worker": "worker-metrics-06",+ "question": "What is the total of the memory column over all rows?",+ "expected": 746+ },+ {+ "file": "metrics-07.csv",+ "worker": "worker-metrics-07",+ "question": "What is the maximum memory among rows with latency_ms < 10?",+ "expected": 72+ },+ {+ "file": "metrics-08.csv",+ "worker": "worker-metrics-08",+ "question": "What is the total of the cpu column over rows with memory >= 50?",+ "expected": 485+ }+ ]+}diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csvnew file mode 100644index 0000000000..8261191260--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csv@@ -0,0 +1,12 @@+host,cpu,memory,latency_ms+core-2,34,46,14+db-1,51,69,21+db-2,68,92,28+api-1,85,25,35+api-2,12,48,3+gw-1,29,71,10+gw-2,46,94,17+edge-1,63,27,24+edge-2,80,50,31+core-1,7,73,38+core-2,24,6,6diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csvnew file mode 100644index 0000000000..0f6b23427c--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csv@@ -0,0 +1,13 @@+host,cpu,memory,latency_ms+api-1,63,87,27+api-2,80,20,34+gw-1,7,43,2+gw-2,24,66,9+edge-1,41,89,16+edge-2,58,22,23+core-1,75,45,30+core-2,92,68,37+db-1,19,91,5+db-2,36,24,12+api-1,53,47,19+api-2,70,70,26diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csvnew file mode 100644index 0000000000..66631ff9a8--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csv@@ -0,0 +1,14 @@+host,cpu,memory,latency_ms+gw-2,92,38,1+edge-1,19,61,8+edge-2,36,84,15+core-1,53,17,22+core-2,70,40,29+db-1,87,63,36+db-2,14,86,4+api-1,31,19,11+api-2,48,42,18+gw-1,65,65,25+gw-2,82,88,32+edge-1,9,21,39+edge-2,26,44,7diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csvnew file mode 100644index 0000000000..80b5f84bc6--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csv@@ -0,0 +1,15 @@+host,cpu,memory,latency_ms+core-1,31,79,14+core-2,48,12,21+db-1,65,35,28+db-2,82,58,35+api-1,9,81,3+api-2,26,14,10+gw-1,43,37,17+gw-2,60,60,24+edge-1,77,83,31+edge-2,94,16,38+core-1,21,39,6+core-2,38,62,13+db-1,55,85,20+db-2,72,18,27diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csvnew file mode 100644index 0000000000..2d31325e3d--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csv@@ -0,0 +1,16 @@+host,cpu,memory,latency_ms+db-2,60,30,27+api-1,77,53,34+api-2,94,76,2+gw-1,21,9,9+gw-2,38,32,16+edge-1,55,55,23+edge-2,72,78,30+core-1,89,11,37+core-2,16,34,5+db-1,33,57,12+db-2,50,80,19+api-1,67,13,26+api-2,84,36,33+gw-1,11,59,1+gw-2,28,82,8diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csvnew file mode 100644index 0000000000..8a6fd276f2--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csv@@ -0,0 +1,17 @@+host,cpu,memory,latency_ms+gw-1,89,71,1+gw-2,16,94,8+edge-1,33,27,15+edge-2,50,50,22+core-1,67,73,29+core-2,84,6,36+db-1,11,29,4+db-2,28,52,11+api-1,45,75,18+api-2,62,8,25+gw-1,79,31,32+gw-2,6,54,39+edge-1,23,77,7+edge-2,40,10,14+core-1,57,33,21+core-2,74,56,28diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csvnew file mode 100644index 0000000000..c0d0d2ea46--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csv@@ -0,0 +1,18 @@+host,cpu,memory,latency_ms+edge-2,28,22,14+core-1,45,45,21+core-2,62,68,28+db-1,79,91,35+db-2,6,24,3+api-1,23,47,10+api-2,40,70,17+gw-1,57,93,24+gw-2,74,26,31+edge-1,91,49,38+edge-2,18,72,6+core-1,35,5,13+core-2,52,28,20+db-1,69,51,27+db-2,86,74,34+api-1,13,7,2+api-2,30,30,9diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csvnew file mode 100644index 0000000000..c6079b42f8--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csv@@ -0,0 +1,19 @@+host,cpu,memory,latency_ms+db-1,57,63,27+db-2,74,86,34+api-1,91,19,2+api-2,18,42,9+gw-1,35,65,16+gw-2,52,88,23+edge-1,69,21,30+edge-2,86,44,37+core-1,13,67,5+core-2,30,90,12+db-1,47,23,19+db-2,64,46,26+api-1,81,69,33+api-2,8,92,1+gw-1,25,25,8+gw-2,42,48,15+edge-1,59,71,22+edge-2,76,94,29diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txt b/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txtnew file mode 100644index 0000000000..b6b9a59073--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txt@@ -0,0 +1,38 @@+You are the orchestrator of a data-analysis fan-out. This repository+holds 8 read-only data shards under shards/. Answer each shard's+question by delegating it to a dedicated RLM subagent - never by+computing an answer yourself.++Shard questions:+- shards/metrics-01.csv: What is the maximum value of the cpu column?+- shards/metrics-02.csv: What is the total of the latency_ms column over rows with cpu >= 90?+- shards/metrics-03.csv: How many rows have memory > 80?+- shards/metrics-04.csv: What is the total of the latency_ms column over all rows?+- shards/metrics-05.csv: How many rows have cpu >= 90?+- shards/metrics-06.csv: What is the total of the memory column over all rows?+- shards/metrics-07.csv: What is the maximum memory among rows with latency_ms < 10?+- shards/metrics-08.csv: What is the total of the cpu column over rows with memory >= 50?++How to run the fan-out:++1. Spawn exactly one RLM subagent per shard from the Python kernel+ with await rlm.spawn('<task prompt>', name='worker-<shard stem>');+ for shards/metrics-01.csv the child name is worker-metrics-01. The+ child's task prompt must name the shard's file path and question+ and require the child to answer with only the number.+2. Each child must send its answer back to you with+ await agent_message.send(answer, receiver_role='parent').+3. Do not spawn two children for the same shard at once. If a child+ fails, you may retry once: wait for the failure notice, delete the+ failed child with await rlm.delete_subagent(child), then spawn a+ replacement with the same worker name.+4. Keep your current run active until every child has replied or+ clearly failed: replies and failure notices are delivered into+ your active run, and rlm.list_subagents() shows child status while+ you wait. Do not end your run early.+5. When every child has reported, write combined-index.md in the+ repository root with exactly one bullet line per shard, in this+ exact format:+ - <shard-file-name>: <answer>+ Copy each answer verbatim from the child's reply. The file must+ cover all 8 shards.diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/fixture.json b/scripts/evals/swarm_fanout/fixtures/json-events/fixture.jsonnew file mode 100644index 0000000000..72bf4d8c2b--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/fixture.json@@ -0,0 +1,54 @@+{+ "name": "json-events",+ "artifact": "combined-index.md",+ "shards": [+ {+ "file": "events-01.jsonl",+ "worker": "worker-events-01",+ "question": "How many records have action \"deploy\"?",+ "expected": 3+ },+ {+ "file": "events-02.jsonl",+ "worker": "worker-events-02",+ "question": "What is the total of the bytes field over all records with action \"upload\"?",+ "expected": 2044+ },+ {+ "file": "events-03.jsonl",+ "worker": "worker-events-03",+ "question": "How many distinct users have at least one record with action \"login\"?",+ "expected": 3+ },+ {+ "file": "events-04.jsonl",+ "worker": "worker-events-04",+ "question": "What is the maximum value of the bytes field?",+ "expected": 871+ },+ {+ "file": "events-05.jsonl",+ "worker": "worker-events-05",+ "question": "How many records have action \"logout\"?",+ "expected": 5+ },+ {+ "file": "events-06.jsonl",+ "worker": "worker-events-06",+ "question": "How many records have bytes greater than 500?",+ "expected": 5+ },+ {+ "file": "events-07.jsonl",+ "worker": "worker-events-07",+ "question": "What is the total of the bytes field over all records with action \"deploy\"?",+ "expected": 1912+ },+ {+ "file": "events-08.jsonl",+ "worker": "worker-events-08",+ "question": "How many distinct users appear in the file?",+ "expected": 3+ }+ ]+}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonlnew file mode 100644index 0000000000..6fea7b43db--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonl@@ -0,0 +1,13 @@+{"ts": "2026-09-01T100:00:00Z", "user": "u2", "action": "logout", "bytes": 0}+{"ts": "2026-09-01T101:00:00Z", "user": "u4", "action": "deploy", "bytes": 278}+{"ts": "2026-09-01T102:00:00Z", "user": "u6", "action": "upload", "bytes": 369}+{"ts": "2026-09-01T103:00:00Z", "user": "u2", "action": "login", "bytes": 0}+{"ts": "2026-09-01T104:00:00Z", "user": "u4", "action": "logout", "bytes": 0}+{"ts": "2026-09-01T105:00:00Z", "user": "u6", "action": "deploy", "bytes": 642}+{"ts": "2026-09-01T106:00:00Z", "user": "u2", "action": "upload", "bytes": 733}+{"ts": "2026-09-01T107:00:00Z", "user": "u4", "action": "login", "bytes": 0}+{"ts": "2026-09-01T108:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-01T109:00:00Z", "user": "u2", "action": "deploy", "bytes": 106}+{"ts": "2026-09-01T110:00:00Z", "user": "u4", "action": "upload", "bytes": 197}+{"ts": "2026-09-01T111:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-01T112:00:00Z", "user": "u2", "action": "logout", "bytes": 0}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonlnew file mode 100644index 0000000000..83588d449d--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonl@@ -0,0 +1,14 @@+{"ts": "2026-09-02T100:00:00Z", "user": "u3", "action": "deploy", "bytes": 324}+{"ts": "2026-09-02T101:00:00Z", "user": "u5", "action": "upload", "bytes": 415}+{"ts": "2026-09-02T102:00:00Z", "user": "u1", "action": "login", "bytes": 0}+{"ts": "2026-09-02T103:00:00Z", "user": "u3", "action": "logout", "bytes": 0}+{"ts": "2026-09-02T104:00:00Z", "user": "u5", "action": "deploy", "bytes": 688}+{"ts": "2026-09-02T105:00:00Z", "user": "u1", "action": "upload", "bytes": 779}+{"ts": "2026-09-02T106:00:00Z", "user": "u3", "action": "login", "bytes": 0}+{"ts": "2026-09-02T107:00:00Z", "user": "u5", "action": "logout", "bytes": 0}+{"ts": "2026-09-02T108:00:00Z", "user": "u1", "action": "deploy", "bytes": 152}+{"ts": "2026-09-02T109:00:00Z", "user": "u3", "action": "upload", "bytes": 243}+{"ts": "2026-09-02T110:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-02T111:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-02T112:00:00Z", "user": "u3", "action": "deploy", "bytes": 516}+{"ts": "2026-09-02T113:00:00Z", "user": "u5", "action": "upload", "bytes": 607}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonlnew file mode 100644index 0000000000..2621376603--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonl@@ -0,0 +1,15 @@+{"ts": "2026-09-03T100:00:00Z", "user": "u4", "action": "upload", "bytes": 461}+{"ts": "2026-09-03T101:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-03T102:00:00Z", "user": "u2", "action": "logout", "bytes": 0}+{"ts": "2026-09-03T103:00:00Z", "user": "u4", "action": "deploy", "bytes": 734}+{"ts": "2026-09-03T104:00:00Z", "user": "u6", "action": "upload", "bytes": 825}+{"ts": "2026-09-03T105:00:00Z", "user": "u2", "action": "login", "bytes": 0}+{"ts": "2026-09-03T106:00:00Z", "user": "u4", "action": "logout", "bytes": 0}+{"ts": "2026-09-03T107:00:00Z", "user": "u6", "action": "deploy", "bytes": 198}+{"ts": "2026-09-03T108:00:00Z", "user": "u2", "action": "upload", "bytes": 289}+{"ts": "2026-09-03T109:00:00Z", "user": "u4", "action": "login", "bytes": 0}+{"ts": "2026-09-03T110:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-03T111:00:00Z", "user": "u2", "action": "deploy", "bytes": 562}+{"ts": "2026-09-03T112:00:00Z", "user": "u4", "action": "upload", "bytes": 653}+{"ts": "2026-09-03T113:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-03T114:00:00Z", "user": "u2", "action": "logout", "bytes": 0}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonlnew file mode 100644index 0000000000..20ad634eaa--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonl@@ -0,0 +1,16 @@+{"ts": "2026-09-04T100:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-04T101:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-04T102:00:00Z", "user": "u3", "action": "deploy", "bytes": 780}+{"ts": "2026-09-04T103:00:00Z", "user": "u5", "action": "upload", "bytes": 871}+{"ts": "2026-09-04T104:00:00Z", "user": "u1", "action": "login", "bytes": 0}+{"ts": "2026-09-04T105:00:00Z", "user": "u3", "action": "logout", "bytes": 0}+{"ts": "2026-09-04T106:00:00Z", "user": "u5", "action": "deploy", "bytes": 244}+{"ts": "2026-09-04T107:00:00Z", "user": "u1", "action": "upload", "bytes": 335}+{"ts": "2026-09-04T108:00:00Z", "user": "u3", "action": "login", "bytes": 0}+{"ts": "2026-09-04T109:00:00Z", "user": "u5", "action": "logout", "bytes": 0}+{"ts": "2026-09-04T110:00:00Z", "user": "u1", "action": "deploy", "bytes": 608}+{"ts": "2026-09-04T111:00:00Z", "user": "u3", "action": "upload", "bytes": 699}+{"ts": "2026-09-04T112:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-04T113:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-04T114:00:00Z", "user": "u3", "action": "deploy", "bytes": 72}+{"ts": "2026-09-04T115:00:00Z", "user": "u5", "action": "upload", "bytes": 163}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonlnew file mode 100644index 0000000000..5374d2063b--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonl@@ -0,0 +1,17 @@+{"ts": "2026-09-05T100:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-05T101:00:00Z", "user": "u2", "action": "deploy", "bytes": 826}+{"ts": "2026-09-05T102:00:00Z", "user": "u4", "action": "upload", "bytes": 917}+{"ts": "2026-09-05T103:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-05T104:00:00Z", "user": "u2", "action": "logout", "bytes": 0}+{"ts": "2026-09-05T105:00:00Z", "user": "u4", "action": "deploy", "bytes": 290}+{"ts": "2026-09-05T106:00:00Z", "user": "u6", "action": "upload", "bytes": 381}+{"ts": "2026-09-05T107:00:00Z", "user": "u2", "action": "login", "bytes": 0}+{"ts": "2026-09-05T108:00:00Z", "user": "u4", "action": "logout", "bytes": 0}+{"ts": "2026-09-05T109:00:00Z", "user": "u6", "action": "deploy", "bytes": 654}+{"ts": "2026-09-05T110:00:00Z", "user": "u2", "action": "upload", "bytes": 745}+{"ts": "2026-09-05T111:00:00Z", "user": "u4", "action": "login", "bytes": 0}+{"ts": "2026-09-05T112:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-05T113:00:00Z", "user": "u2", "action": "deploy", "bytes": 118}+{"ts": "2026-09-05T114:00:00Z", "user": "u4", "action": "upload", "bytes": 209}+{"ts": "2026-09-05T115:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-05T116:00:00Z", "user": "u2", "action": "logout", "bytes": 0}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonlnew file mode 100644index 0000000000..42f321c352--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonl@@ -0,0 +1,18 @@+{"ts": "2026-09-06T100:00:00Z", "user": "u1", "action": "deploy", "bytes": 872}+{"ts": "2026-09-06T101:00:00Z", "user": "u3", "action": "upload", "bytes": 63}+{"ts": "2026-09-06T102:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-06T103:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-06T104:00:00Z", "user": "u3", "action": "deploy", "bytes": 336}+{"ts": "2026-09-06T105:00:00Z", "user": "u5", "action": "upload", "bytes": 427}+{"ts": "2026-09-06T106:00:00Z", "user": "u1", "action": "login", "bytes": 0}+{"ts": "2026-09-06T107:00:00Z", "user": "u3", "action": "logout", "bytes": 0}+{"ts": "2026-09-06T108:00:00Z", "user": "u5", "action": "deploy", "bytes": 700}+{"ts": "2026-09-06T109:00:00Z", "user": "u1", "action": "upload", "bytes": 791}+{"ts": "2026-09-06T110:00:00Z", "user": "u3", "action": "login", "bytes": 0}+{"ts": "2026-09-06T111:00:00Z", "user": "u5", "action": "logout", "bytes": 0}+{"ts": "2026-09-06T112:00:00Z", "user": "u1", "action": "deploy", "bytes": 164}+{"ts": "2026-09-06T113:00:00Z", "user": "u3", "action": "upload", "bytes": 255}+{"ts": "2026-09-06T114:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-06T115:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-06T116:00:00Z", "user": "u3", "action": "deploy", "bytes": 528}+{"ts": "2026-09-06T117:00:00Z", "user": "u5", "action": "upload", "bytes": 619}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonlnew file mode 100644index 0000000000..1133b269ad--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonl@@ -0,0 +1,19 @@+{"ts": "2026-09-07T100:00:00Z", "user": "u2", "action": "upload", "bytes": 109}+{"ts": "2026-09-07T101:00:00Z", "user": "u4", "action": "login", "bytes": 0}+{"ts": "2026-09-07T102:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-07T103:00:00Z", "user": "u2", "action": "deploy", "bytes": 382}+{"ts": "2026-09-07T104:00:00Z", "user": "u4", "action": "upload", "bytes": 473}+{"ts": "2026-09-07T105:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-07T106:00:00Z", "user": "u2", "action": "logout", "bytes": 0}+{"ts": "2026-09-07T107:00:00Z", "user": "u4", "action": "deploy", "bytes": 746}+{"ts": "2026-09-07T108:00:00Z", "user": "u6", "action": "upload", "bytes": 837}+{"ts": "2026-09-07T109:00:00Z", "user": "u2", "action": "login", "bytes": 0}+{"ts": "2026-09-07T110:00:00Z", "user": "u4", "action": "logout", "bytes": 0}+{"ts": "2026-09-07T111:00:00Z", "user": "u6", "action": "deploy", "bytes": 210}+{"ts": "2026-09-07T112:00:00Z", "user": "u2", "action": "upload", "bytes": 301}+{"ts": "2026-09-07T113:00:00Z", "user": "u4", "action": "login", "bytes": 0}+{"ts": "2026-09-07T114:00:00Z", "user": "u6", "action": "logout", "bytes": 0}+{"ts": "2026-09-07T115:00:00Z", "user": "u2", "action": "deploy", "bytes": 574}+{"ts": "2026-09-07T116:00:00Z", "user": "u4", "action": "upload", "bytes": 665}+{"ts": "2026-09-07T117:00:00Z", "user": "u6", "action": "login", "bytes": 0}+{"ts": "2026-09-07T118:00:00Z", "user": "u2", "action": "logout", "bytes": 0}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonlnew file mode 100644index 0000000000..cb9bedd8b4--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonl@@ -0,0 +1,20 @@+{"ts": "2026-09-08T100:00:00Z", "user": "u3", "action": "login", "bytes": 0}+{"ts": "2026-09-08T101:00:00Z", "user": "u5", "action": "logout", "bytes": 0}+{"ts": "2026-09-08T102:00:00Z", "user": "u1", "action": "deploy", "bytes": 428}+{"ts": "2026-09-08T103:00:00Z", "user": "u3", "action": "upload", "bytes": 519}+{"ts": "2026-09-08T104:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-08T105:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-08T106:00:00Z", "user": "u3", "action": "deploy", "bytes": 792}+{"ts": "2026-09-08T107:00:00Z", "user": "u5", "action": "upload", "bytes": 883}+{"ts": "2026-09-08T108:00:00Z", "user": "u1", "action": "login", "bytes": 0}+{"ts": "2026-09-08T109:00:00Z", "user": "u3", "action": "logout", "bytes": 0}+{"ts": "2026-09-08T110:00:00Z", "user": "u5", "action": "deploy", "bytes": 256}+{"ts": "2026-09-08T111:00:00Z", "user": "u1", "action": "upload", "bytes": 347}+{"ts": "2026-09-08T112:00:00Z", "user": "u3", "action": "login", "bytes": 0}+{"ts": "2026-09-08T113:00:00Z", "user": "u5", "action": "logout", "bytes": 0}+{"ts": "2026-09-08T114:00:00Z", "user": "u1", "action": "deploy", "bytes": 620}+{"ts": "2026-09-08T115:00:00Z", "user": "u3", "action": "upload", "bytes": 711}+{"ts": "2026-09-08T116:00:00Z", "user": "u5", "action": "login", "bytes": 0}+{"ts": "2026-09-08T117:00:00Z", "user": "u1", "action": "logout", "bytes": 0}+{"ts": "2026-09-08T118:00:00Z", "user": "u3", "action": "deploy", "bytes": 84}+{"ts": "2026-09-08T119:00:00Z", "user": "u5", "action": "upload", "bytes": 175}diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/task.txt b/scripts/evals/swarm_fanout/fixtures/json-events/task.txtnew file mode 100644index 0000000000..2eac761899--- /dev/null+++ b/scripts/evals/swarm_fanout/fixtures/json-events/task.txt@@ -0,0 +1,38 @@+You are the orchestrator of a data-analysis fan-out. This repository+holds 8 read-only data shards under shards/. Answer each shard's+question by delegating it to a dedicated RLM subagent - never by+computing an answer yourself.++Shard questions:+- shards/events-01.jsonl: How many records have action "deploy"?+- shards/events-02.jsonl: What is the total of the bytes field over all records with action "upload"?+- shards/events-03.jsonl: How many distinct users have at least one record with action "login"?+- shards/events-04.jsonl: What is the maximum value of the bytes field?+- shards/events-05.jsonl: How many records have action "logout"?+- shards/events-06.jsonl: How many records have bytes greater than 500?+- shards/events-07.jsonl: What is the total of the bytes field over all records with action "deploy"?+- shards/events-08.jsonl: How many distinct users appear in the file?++How to run the fan-out:++1. Spawn exactly one RLM subagent per shard from the Python kernel+ with await rlm.spawn('<task prompt>', name='worker-<shard stem>');+ for shards/events-01.jsonl the child name is worker-events-01. The+ child's task prompt must name the shard's file path and question+ and require the child to answer with only the number.+2. Each child must send its answer back to you with+ await agent_message.send(answer, receiver_role='parent').+3. Do not spawn two children for the same shard at once. If a child+ fails, you may retry once: wait for the failure notice, delete the+ failed child with await rlm.delete_subagent(child), then spawn a+ replacement with the same worker name.+4. Keep your current run active until every child has replied or+ clearly failed: replies and failure notices are delivered into+ your active run, and rlm.list_subagents() shows child status while+ you wait. Do not end your run early.+5. When every child has reported, write combined-index.md in the+ repository root with exactly one bullet line per shard, in this+ exact format:+ - <shard-file-name>: <answer>+ Copy each answer verbatim from the child's reply. The file must+ cover all 8 shards.diff --git a/scripts/evals/swarm_fanout/pyproject.toml b/scripts/evals/swarm_fanout/pyproject.tomlnew file mode 100644index 0000000000..df6d1cf8aa--- /dev/null+++ b/scripts/evals/swarm_fanout/pyproject.toml@@ -0,0 +1,19 @@+[project]+name = "prime-agent-evals"+version = "0.1.0"+requires-python = ">=3.11"+dependencies = []++[dependency-groups]+dev = ["ruff==0.16.5"]++[tool.uv]+package = false+exclude-newer = "2026-09-02T00:00:00Z"++[tool.ruff]+line-length = 110+target-version = "py311"++[tool.ruff.lint]+select = ["E", "F", "I", "UP", "B"]diff --git a/scripts/evals/swarm_fanout/runner.py b/scripts/evals/swarm_fanout/runner.pynew file mode 100644index 0000000000..8862a33489--- /dev/null+++ b/scripts/evals/swarm_fanout/runner.py@@ -0,0 +1,414 @@+"""Runner for the swarm-fanout eval.++Copies a fixture repo to a temp dir, runs the parent agent headless+against it, records the post-state (RLM spawn ledger, parent transcript,+child session dirs, combined artifact), and prints the scored outcome+as JSON.++The parent is expected to fan out one RLM child per data shard and fan+the answers back in. Real-model runs are manual: pass --model and ensure+provider auth is configured in the environment. The harness itself is+validated model-free by tests/test_swarm_fanout.py.++Usage:+ uv run --locked python runner.py --fixture fixtures/json-events --model anthropic/claude-sonnet-4-5+"""++from __future__ import annotations++import argparse+import hashlib+import json+import os+import shutil+import socket+import subprocess+import sys+import tempfile+import time+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))+import scorer # noqa: E402++RLM_LEDGER_DIR = "rlm-ledger"+++def _captured_text(output: str | bytes | None) -> str:+ """Decode captured subprocess output.++ TimeoutExpired output arrives as bytes even with text=True.+ """+ if output is None:+ return ""+ if isinstance(output, bytes):+ return output.decode(errors="replace")+ return output+++def canonicalize_dir_path(dir_path: str) -> str:+ """Mirror the product ledger's canonicalization.++ rlmLedgerPath realpaths a sessions dir that exists and falls back to+ plain resolve otherwise, so the runner must reproduce both branches+ to land on the same ledger file the daemon writes.+ """+ resolved = os.path.abspath(dir_path)+ if os.path.exists(resolved):+ return os.path.realpath(resolved)+ return resolved+++def ledger_file_candidates(agent_home: Path, sessions_dir: str) -> list[Path]:+ """Ledger path candidates for a sessions dir, most likely first.++ The daemon canonicalizes (realpaths) the sessions dir before hashing+ it into the ledger file name; the plain-resolved variant is checked+ second in case the directory did not exist when the daemon first+ constructed its ledger.+ """+ ledger_dir = Path(agent_home) / RLM_LEDGER_DIR+ candidates = []+ for variant in (canonicalize_dir_path(sessions_dir), os.path.abspath(sessions_dir)):+ digest = hashlib.sha256(variant.encode("utf-8")).hexdigest()[:16]+ candidate = ledger_dir / f"{digest}.jsonl"+ if candidate not in candidates:+ candidates.append(candidate)+ return candidates+++def find_ledger_file(agent_home: Path, sessions_dir: str) -> Path | None:+ for candidate in ledger_file_candidates(agent_home, sessions_dir):+ if candidate.is_file():+ return candidate+ return None+++def read_ledger_text(agent_home: Path, sessions_dir: str) -> str:+ ledger_file = find_ledger_file(agent_home, sessions_dir)+ if ledger_file is None:+ return ""+ try:+ return ledger_file.read_text()+ except OSError:+ return ""+++def find_parent_session_file(sessions_dir: Path, ledger_text: str) -> Path | None:+ """Locate the parent session file in the sessions dir.++ The spawn records name the parent session file directly; when the+ ledger is empty (a no-spawn run) the newest session file is the+ parent - the eval launches exactly one client session.+ """+ session_files = sorted(sessions_dir.glob("*.jsonl"))+ if not session_files:+ return None+ records, _malformed = scorer.parse_ledger(ledger_text)+ parents = set()+ for entry in records:+ if entry.get("op") != "spawn" or not isinstance(entry.get("parent"), str):+ continue+ segments = scorer._path_segments(entry["parent"])+ if segments:+ parents.add(segments[-1])+ matching = [path for path in session_files if path.name in parents]+ if matching:+ return matching[0]+ return max(session_files, key=lambda path: path.stat().st_mtime)+++def collect_child_session_dirs(sessions_dir: Path, parent_session_file: Path | None) -> dict[str, list[str]]:+ """sub-* child dirs under the parent's session artifact dir, by name.++ Child sessions live next to their parent's artifacts+ (session-artifacts/<parent-id>/sub-<uuid8>/), each holding the+ child's session JSONL. The dir-to-files mapping lets the scorer+ verify each ledger edge's recorded child file exists on disk, so a+ dir unrelated to the ledger's children cannot stand in for one.+ """+ if parent_session_file is None:+ return {}+ artifacts_root = sessions_dir.parent / "session-artifacts"+ parent_artifacts = artifacts_root / parent_session_file.stem+ if not parent_artifacts.is_dir():+ return {}+ child_dirs: dict[str, list[str]] = {}+ for child_dir in sorted(parent_artifacts.iterdir()):+ if not child_dir.is_dir() or not child_dir.name.startswith("sub-"):+ continue+ session_files = sorted(path.name for path in child_dir.glob("*.jsonl"))+ if session_files:+ child_dirs[child_dir.name] = session_files+ return child_dirs+++def read_artifact_text(repo_dir: Path, artifact: str) -> str:+ try:+ return (repo_dir / artifact).read_text()+ except OSError:+ return ""+++def first_agent_error(agent_log: str) -> str | None:+ """The first error message the agent reported, for quick diagnosis.++ A zero-token unresolved run almost always means a launch or auth+ failure; surfacing the error in the result JSON saves the+ workdir-digging this field was created for.+ """+ for line in agent_log.splitlines():+ try:+ entry = json.loads(line)+ except ValueError:+ continue+ message = entry.get("message") if isinstance(entry, dict) else None+ if isinstance(message, dict) and message.get("role") == "assistant":+ error = message.get("errorMessage")+ if isinstance(error, str) and error:+ return error+ return None+++def first_stderr_error(stderr: str) -> str | None:+ """The first stderr line, where launch and auth failures surface."""+ for line in stderr.splitlines():+ stripped = line.strip()+ if stripped:+ return stripped+ return None+++def shutdown_agent_daemon(socket_path: Path, retry_window_s: float = 0.0) -> None:+ """Stop the daemon the agent run leaves listening on the eval socket.++ The CLI spawns a detached daemon per --daemon-socket; without this,+ repeated evals accumulate orphan daemons and a timed-out run keeps+ its worker going. Client commands ride in a protocol envelope; the+ shutdown command closes the daemon's sessions before it exits.++ A timed-out launch can die before the detached daemon finished+ binding its socket; the daemon then comes up orphaned and never sees+ the shutdown. With a retry window the teardown waits for the socket+ to appear and still delivers the shutdown.+ """+ envelope = json.dumps(+ {+ "type": "command",+ "id": "eval-shutdown",+ "protocol": {"name": "prime-agent.daemon", "version": 7},+ "command": {"type": "shutdown"},+ }+ )+ deadline = time.monotonic() + retry_window_s+ while True:+ try:+ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client:+ client.settimeout(10)+ client.connect(str(socket_path))+ client.sendall(envelope.encode() + b"\n")+ while client.recv(4096):+ pass+ return+ except OSError:+ if time.monotonic() >= deadline:+ # No daemon on the socket (launch failure or a stub+ # agent): done.+ return+ time.sleep(0.25)+++def agent_env(agent_home: Path, sessions_dir: str) -> dict:+ """A clean, isolated environment for the agent subprocess.++ The eval agent must be an independent root session: every PRIME_AGENT_INTERNAL_*+ variable an embedding session might leak is stripped (a child inheriting+ them would try to attach to the parent's worker), the agent runs under+ its own PRIME_AGENT_CODING_AGENT_DIR / PI_CODING_AGENT_DIR (the workspace+ launcher's env prefix derives from the package config name) so it never+ touches a production agent dir, and the credential file is copied in so+ model auth works without sharing any state.++ The session dir env override matters beyond the --session-dir flag: the+ detached daemon is launched without that flag and keys its RLM spawn+ ledger on the env override, so the scorer can compute the ledger path+ deterministically.+ """+ env = {key: value for key, value in os.environ.items() if not key.startswith("PRIME_AGENT_INTERNAL_")}+ for key in (+ "PRIME_AGENT_BASH_SHELL",+ "PRIME_AGENT_BASH_COMMAND_PREFIX",+ # Depth overrides from an embedding RLM session would start the+ # eval parent at the wrong depth or block spawning shard children.+ "RLM_DEPTH",+ "RLM_MAX_DEPTH",+ ):+ env.pop(key, None)+ source_agent_dir = Path(env.get("PRIME_AGENT_CODING_AGENT_DIR") or Path.home() / ".prime" / "agent")+ source_auth = source_agent_dir / "auth.json"+ if source_auth.is_file():+ agent_home.mkdir(parents=True, exist_ok=True)+ shutil.copy2(source_auth, agent_home / "auth.json")+ env["PRIME_AGENT_CODING_AGENT_DIR"] = str(agent_home)+ env["PI_CODING_AGENT_DIR"] = str(agent_home)+ env["PRIME_AGENT_SESSION_DIR"] = sessions_dir+ env["PI_SESSION_DIR"] = sessions_dir+ env["PRIME_AGENT_CODING_AGENT_SESSION_DIR"] = sessions_dir+ env["PI_CODING_AGENT_SESSION_DIR"] = sessions_dir+ return env+++def init_fixture_repo(fixture_dir: Path, repo_dir: Path) -> None:+ """Copy the fixture into a fresh git repo.++ The fixture manifest and task prompt stay out of the repo: the agent+ works on the shards alone, and expected answers never reach it.+ """+ shutil.copytree(+ fixture_dir,+ repo_dir,+ ignore=shutil.ignore_patterns("fixture.json", "task.txt"),+ )+ # Strip git location variables before initializing the temp repo: with+ # GIT_DIR, GIT_INDEX_FILE, or GIT_WORK_TREE inherited from the caller,+ # git add/commit would operate on the caller-selected repository instead+ # of repo_dir. The environment is built fresh (not layered on+ # os.environ) so a stripped variable cannot re-enter via the merge.+ git_env = {+ key: value+ for key, value in os.environ.items()+ if key+ not in {+ "GIT_DIR",+ "GIT_INDEX_FILE",+ "GIT_WORK_TREE",+ "GIT_OBJECT_DIRECTORY",+ "GIT_COMMON_DIR",+ }+ }+ git_env.update(+ {+ "GIT_AUTHOR_NAME": "eval",+ "GIT_AUTHOR_EMAIL": "eval@eval",+ "GIT_COMMITTER_NAME": "eval",+ "GIT_COMMITTER_EMAIL": "eval@eval",+ }+ )+ for command in (+ ["git", "init", "-q"],+ ["git", "add", "-A"],+ ["git", "-c", "commit.gpgsign=false", "commit", "-qm", "fixture"],+ ):+ subprocess.run(command, cwd=repo_dir, check=True, env=git_env)+++def collect_outcome(sessions_dir: Path, agent_home: Path, repo_dir: Path, agent_log: str) -> dict:+ """Gather every artifact the rubric scores from the post-run state."""+ ledger_text = read_ledger_text(agent_home, str(sessions_dir))+ parent_session_file = find_parent_session_file(sessions_dir, ledger_text)+ parent_transcript_text = ""+ if parent_session_file is not None:+ try:+ parent_transcript_text = parent_session_file.read_text()+ except OSError:+ parent_transcript_text = ""+ return {+ "ledger_text": ledger_text,+ "parent_transcript_text": parent_transcript_text,+ "child_session_dirs": collect_child_session_dirs(sessions_dir, parent_session_file),+ "artifact_text": read_artifact_text(repo_dir, "combined-index.md"),+ "usage": scorer.summarize_usage(agent_log),+ }+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--fixture", required=True, help="Fixture directory (contains fixture.json)")+ parser.add_argument("--model", required=True, help="Model selector under test, provider/model-id")+ parser.add_argument("--timeout", type=int, default=1800, help="Agent run timeout in seconds")+ parser.add_argument("--agent-bin", default="prime-agent", help="Agent binary to invoke")+ args = parser.parse_args(argv)++ if sys.platform == "win32":+ # The eval contract (per-workdir Unix daemon socket plus an+ # AF_UNIX shutdown client) has no Windows named-pipe support;+ # refuse loudly instead of failing obscurely mid-run.+ print(+ "swarm-fanout eval requires a Unix-domain-socket platform; Windows named pipes are not supported",+ file=sys.stderr,+ )+ return 1++ fixture_dir = Path(args.fixture).resolve()+ fixture = json.loads((fixture_dir / "fixture.json").read_text())+ task = (fixture_dir / "task.txt").read_text()++ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-"))+ repo_dir = workdir / "repo"+ sessions_dir = workdir / "sessions"+ sessions_dir.mkdir()+ sessions_dir = Path(canonicalize_dir_path(str(sessions_dir)))+ agent_home = workdir / "agent-home"+ init_fixture_repo(fixture_dir, repo_dir)++ timed_out = False+ started = time.monotonic()+ try:+ completed = subprocess.run(+ [+ args.agent_bin,+ "--mode",+ "json",+ "--daemon-socket",+ str(workdir / "daemon.sock"),+ "--cwd",+ str(repo_dir),+ "--session-dir",+ str(sessions_dir),+ "--model",+ args.model,+ "--",+ task,+ ],+ env=agent_env(agent_home, str(sessions_dir)),+ capture_output=True,+ text=True,+ timeout=args.timeout,+ )+ agent_log = completed.stdout+ stderr_text = completed.stderr+ exit_code = completed.returncode+ except subprocess.TimeoutExpired as exc:+ # A timed-out run still scores; keep whatever transcript exists.+ timed_out = True+ agent_log = _captured_text(exc.stdout)+ stderr_text = _captured_text(exc.stderr)+ exit_code = None+ except OSError as exc:+ # A missing or non-executable agent binary still produces a result.+ agent_log = ""+ stderr_text = str(exc)+ exit_code = None+ finally:+ wall_time_s = time.monotonic() - started+ # A timed-out run may have died mid daemon-spawn: give the+ # detached daemon time to bind before shutting it down.+ shutdown_agent_daemon(workdir / "daemon.sock", retry_window_s=8.0 if timed_out else 0.0)+ (workdir / "agent.log").write_text(agent_log)+ # stderr is where launch and auth failures land; keep it with the result.+ (workdir / "agent.stderr").write_text(stderr_text)++ outcome = collect_outcome(sessions_dir, agent_home, repo_dir, agent_log)+ outcome["wall_time_s"] = wall_time_s+ result = scorer.score_fixture(fixture, outcome)+ result["exit_code"] = exit_code+ result["timed_out"] = timed_out+ result["workdir"] = str(workdir)+ result["model"] = args.model+ result["agent_error"] = first_agent_error(agent_log) or first_stderr_error(stderr_text)+ print(json.dumps(result, indent=2))+ return 0 if result["resolved"] else 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/evals/swarm_fanout/scorer.py b/scripts/evals/swarm_fanout/scorer.pynew file mode 100644index 0000000000..96e2fba27e--- /dev/null+++ b/scripts/evals/swarm_fanout/scorer.py@@ -0,0 +1,424 @@+"""Pure scorer for the swarm-fanout eval.++The scorer consumes a fixture manifest plus a recorded outcome dict and+applies the orchestration rubric. It never runs the agent; the runner+records the outcome and calls in, so every rule below is unit-testable+without a model or network.++Rubric (all four are required for a resolved run):+ - coverage: every shard's answer appears in combined-index.md and+ matches the machine-computed expected value from the fixture.+ - delegation evidence: at least one live depth-1 spawn edge per shard,+ each with a distinct childId and distinct name in the RLM ledger, and+ at least one sub-* child session dir per shard holding a session+ file. A parent that answers everything itself fails here.+ - dedup: no shard's worker name is spawned twice while another child+ for that shard is still live, and total depth-1 spawns stay within+ one retry per shard (2x the shard count).+ - receipt completeness: every depth-1 child (live or deleted) has a+ parent-visible reply or an explicit child failure/terminal notice in+ the parent transcript. A silently dropped child fails here.++Efficiency (tokens, turns, wall time) is informational until a+single-agent baseline datapoint exists.+"""++from __future__ import annotations++import json+import os+import re++LEDGER_OPS = {"meta", "spawn", "rename", "delete"}+NOTICE_CUSTOM_TYPES = {"rlm_child_terminal_notice", "rlm_child_failure"}+REPLY_CUSTOM_TYPE = "agent_message"++# One "- <shard>: <answer>" bullet per line; the shard name may not+# contain a colon, the answer is everything after the first colon.+ANSWER_LINE = re.compile(r"^\s*[-*]\s+(.+?):\s*(.*?)\s*$")+++def score_fixture(fixture: dict, outcome: dict) -> dict:+ """Apply the swarm-fanout rubric to one recorded outcome."""+ records, malformed = parse_ledger(outcome.get("ledger_text", ""))+ edges = replay_edges(records)+ answers = parse_answers(outcome.get("artifact_text", ""))+ coverage = score_coverage(fixture, answers)+ delegation = score_delegation(fixture, edges, outcome.get("child_session_dirs", {}))+ dedup = score_dedup(fixture, records)+ receipts = score_receipts(edges, outcome.get("parent_transcript_text", ""))+ usage = outcome.get("usage") or {}+ resolved = (+ coverage["coverage"] and delegation["delegation_evidence"] and dedup["dedup"] and receipts["receipts"]+ )+ return {+ "fixture": fixture.get("name"),+ "resolved": resolved,+ "coverage": coverage["coverage"],+ "missing_shards": coverage["missing_shards"],+ "wrong_answers": coverage["wrong_answers"],+ "delegation_evidence": delegation["delegation_evidence"],+ "live_depth1_edges": delegation["live_depth1_edges"],+ "distinct_worker_names": delegation["distinct_worker_names"],+ "child_session_dirs": delegation["child_session_dirs"],+ "verified_shard_workers": delegation["verified_shard_workers"],+ "dedup": dedup["dedup"],+ "duplicate_spawns": dedup["duplicate_spawns"],+ "total_spawns": dedup["total_spawns"],+ "spawn_budget": dedup["spawn_budget"],+ "receipts": receipts["receipts"],+ "missing_receipts": receipts["missing_receipts"],+ "ledger_malformed_lines": malformed,+ "tokens_used": int(usage.get("tokens", 0)),+ "turns": int(usage.get("turns", 0)),+ "wall_time_s": float(outcome.get("wall_time_s", 0.0)),+ }+++def parse_ledger(ledger_text: str) -> tuple[list[dict], int]:+ """Parse ledger JSONL, skipping blank and malformed lines.++ The product ledger fails closed on malformed lines; the scorer is a+ passive reader of a possibly torn artifact, so malformed lines are+ skipped and counted in the result for diagnosis. Each record's fields+ are validated against the product schema (parseLedgerLine) before it+ is replayed: a structurally invalid spawn line must not count as+ delegation evidence. Records with a valid envelope but an unknown op+ are skipped silently, the same forward-compat carve-out the product+ reader makes.+ """+ records: list[dict] = []+ malformed = 0+ for line in ledger_text.splitlines():+ stripped = line.strip()+ if not stripped:+ continue+ try:+ record = json.loads(stripped)+ except ValueError:+ malformed += 1+ continue+ if not isinstance(record, dict) or record.get("op") not in LEDGER_OPS:+ if isinstance(record, dict) and record.get("v") == 1 and isinstance(record.get("at"), str):+ continue # unknown op: forward-compat, not a defect+ malformed += 1+ continue+ if not _valid_ledger_record(record):+ malformed += 1+ continue+ records.append(record)+ return records, malformed+++def _is_string(value: object) -> bool:+ return isinstance(value, str)+++def _valid_ledger_record(record: dict) -> bool:+ """Field-level validation, mirroring the product's parseLedgerLine."""+ if record.get("v") != 1 or not _is_string(record.get("at")):+ return False+ op = record.get("op")+ if op == "meta":+ return _is_string(record.get("sessionsDir"))+ if not (_is_string(record.get("childId")) and _is_string(record.get("child"))):+ return False+ if op == "spawn":+ return (+ _is_string(record.get("parent"))+ and _is_string(record.get("name"))+ and isinstance(record.get("depth"), int)+ and not isinstance(record.get("depth"), bool)+ )+ if op == "rename":+ return _is_string(record.get("name"))+ return _is_string(record.get("reason"))+++def replay_edges(records: list[dict]) -> dict[tuple[str, str], dict]:+ """Replay ledger records into edges, last-writer-wins per childId+child.++ The product ledger keys an edge by (childId, canonical child session+ path) - two records sharing a childId but naming different child+ sessions are two distinct edges, and a rename or delete applies only+ to the edge its own child path names. The scorer mirrors that key so+ a shadowed child cannot slip out of delegation and receipt checks.+ """+ edges: dict[tuple[str, str], dict] = {}+ for record in records:+ op = record.get("op")+ child_id = record.get("childId")+ child = record.get("child")+ if not isinstance(child_id, str) or not isinstance(child, str):+ continue+ key = (child_id, _canonical_session_path(child))+ if op == "spawn":+ edges[key] = {+ "childId": child_id,+ "parent": record.get("parent"),+ "child": record.get("child"),+ "depth": record.get("depth"),+ "name": record.get("name"),+ "deleted": None,+ }+ elif op == "rename":+ edge = edges.get(key)+ if edge is not None and isinstance(record.get("name"), str):+ edge["name"] = record["name"]+ elif op == "delete":+ edge = edges.get(key)+ if edge is not None:+ edge["deleted"] = record.get("reason")+ return edges+++def _canonical_session_path(session_path: str) -> str:+ """Mirror the product's canonicalSessionPath for edge keys.++ The path is real path'd when it exists, falling back to the parent's+ real path joined with the file name, so a canonical and a plain+ spelling of the same child session share one edge key.+ """+ resolved = os.path.abspath(session_path)+ return os.path.realpath(resolved)+++def parse_answers(artifact_text: str) -> dict[str, list[str]]:+ """Map shard file name to every answer given in the index bullets.++ Duplicate bullets are kept, not overwritten: a shard answered twice+ with conflicting values must not pass coverage just because one of+ the two happens to be right. Coverage requires every occurrence to+ match the expected answer.+ """+ answers: dict[str, list[str]] = {}+ for line in artifact_text.splitlines():+ match = ANSWER_LINE.match(line)+ if match is None:+ continue+ shard, answer = match.group(1), match.group(2)+ answers.setdefault(shard, []).append(answer)+ return answers+++def score_coverage(fixture: dict, answers: dict[str, list[str]]) -> dict:+ """Every shard's answer must match the machine-computed expected value.++ A shard listed more than once is covered only when every occurrence+ matches: conflicting duplicates are fabricated answers, not noise.+ """+ missing: list[str] = []+ wrong: dict[str, dict] = {}+ for shard in fixture.get("shards", []):+ file = shard["file"]+ expected = str(shard["expected"])+ found = answers.get(file)+ if not found:+ missing.append(file)+ elif any(value != expected for value in found):+ wrong[file] = {"expected": expected, "found": found}+ return {+ "coverage": not missing and not wrong,+ "missing_shards": missing,+ "wrong_answers": wrong,+ }+++def score_delegation(fixture: dict, edges: dict[str, dict], child_session_dirs: dict[str, list[str]]) -> dict:+ """Spawn evidence must exist per shard: ledger edges plus child dirs.++ Every shard's worker name must have a live depth-1 ledger edge, and+ every such edge must be backed by its real child session dir: the+ edge's recorded child file must exist in the collected sub-* dirs.+ A parent that answers every shard without spawning one child per+ shard fails delegation even though coverage may pass - the no-spawn+ cheat is the blind-answer analog of swe-fix-loop's blind patch, and+ helper-named children do not substitute for shard workers.+ """+ required_workers = {shard["worker"] for shard in fixture.get("shards", [])}+ live_depth1 = [edge for edge in edges.values() if edge.get("depth") == 1 and edge.get("deleted") is None]+ worker_names = {edge.get("name") for edge in live_depth1 if edge.get("name") in required_workers}+ verified_workers = 0+ claimed_sessions: set[tuple[str, str]] = set()+ for worker in required_workers:+ for edge in live_depth1:+ if edge.get("name") != worker:+ continue+ session = _edge_session_dir_and_file(edge)+ if session is None or session in claimed_sessions:+ continue+ if _session_dir_has_file(child_session_dirs, session):+ claimed_sessions.add(session)+ verified_workers += 1+ break+ passed = worker_names == required_workers and verified_workers == len(required_workers)+ return {+ "delegation_evidence": passed,+ "live_depth1_edges": len({edge["childId"] for edge in live_depth1}),+ "distinct_worker_names": len({edge.get("name") for edge in live_depth1 if edge.get("name")}),+ "child_session_dirs": len(child_session_dirs),+ "verified_shard_workers": verified_workers,+ }+++def _path_segments(path: str) -> list[str]:+ """Split a recorded path on both separators, platform-independent.++ The product writes ledger paths with the host separator; on Windows+ they are backslash-joined, so splitting on "/" alone would reject+ every valid child session there.+ """+ return [segment for segment in path.replace("\\", "/").split("/") if segment]+++def _edge_session_dir_and_file(edge: dict) -> tuple[str, str] | None:+ """The edge's recorded (child session dir, session file) pair."""+ child_file = edge.get("child")+ if not isinstance(child_file, str):+ return None+ segments = _path_segments(child_file)+ if len(segments) < 2:+ return None+ return segments[-2], segments[-1]+++def _session_dir_has_file(child_session_dirs: dict[str, list[str]], session: tuple[str, str]) -> bool:+ """True when the recorded session file exists in the collected dirs.++ The ledger's child path points into the child's own sub-* dir; the+ runner collects the dir-to-files mapping, so a fabricated edge with+ no persisted session behind it cannot pass delegation.+ """+ child_dir, session_file = session+ return session_file in child_session_dirs.get(child_dir, [])+++def score_dedup(fixture: dict, records: list[dict]) -> dict:+ """No shard may be worked by two live children; one retry per shard.++ The replay mirrors the daemon's ledger semantics: a spawn adds the+ childId to its name's live set, a delete removes it. A spawn for a+ shard's worker name while another child already holds that name is+ duplicate work. Total depth-1 spawns may not exceed two per shard,+ which is the retry budget the task prompt grants.+ """+ n_shards = len(fixture.get("shards", []))+ shard_workers = {shard["worker"] for shard in fixture.get("shards", [])}+ # A child's shard assignment is its name at spawn and survives renames:+ # renaming the first worker-events-01 child away must not free the name+ # for a second spawn while the first child is still live. Deletes are+ # the only legitimate way a shard's worker name becomes reusable.+ spawn_name: dict[str, str] = {}+ live_depth1_ids: set[str] = set()+ duplicates: list[dict] = []+ total_spawns = 0+ for record in records:+ op = record.get("op")+ child_id = record.get("childId")+ if not isinstance(child_id, str):+ continue+ if op == "spawn" and record.get("depth") == 1:+ total_spawns += 1+ name = record.get("name")+ spawn_name[child_id] = name+ if name in shard_workers and any(spawn_name.get(live_id) == name for live_id in live_depth1_ids):+ duplicates.append({"name": name, "childId": child_id})+ live_depth1_ids.add(child_id)+ elif op == "delete":+ live_depth1_ids.discard(child_id)+ budget = 2 * n_shards+ return {+ "dedup": not duplicates and total_spawns <= budget,+ "duplicate_spawns": duplicates,+ "total_spawns": total_spawns,+ "spawn_budget": budget,+ }+++def score_receipts(edges: dict[str, dict], parent_transcript_text: str) -> dict:+ """Every depth-1 child must be accounted for in the parent transcript.++ A receipt is a child reply (an agent_message custom record naming the+ child's session id) or an explicit failure/terminal notice record+ naming the childId. Deleted children are still checked: deleting a+ silently failed child hides the drop, it does not receipt it.+ """+ replies: set[str] = set()+ notices: set[str] = set()+ for entry in _transcript_records(parent_transcript_text):+ if entry.get("customType") == REPLY_CUSTOM_TYPE:+ details = entry.get("details") or {}+ # Only an explicit child reply receipts a child: the daemon+ # always stamps fromRelationship on delivered messages, so a+ # record missing it (or claiming another relationship) with a+ # matching session id is not the child's reply.+ if details.get("fromRelationship") != "child":+ continue+ sender = details.get("from") or {}+ session_id = sender.get("sessionId")+ if isinstance(session_id, str) and session_id:+ replies.add(session_id)+ elif entry.get("customType") in NOTICE_CUSTOM_TYPES:+ details = entry.get("details") or {}+ child_id = details.get("childId")+ if isinstance(child_id, str) and child_id:+ notices.add(child_id)+ missing: list[dict] = []+ for edge in edges.values():+ if edge.get("depth") != 1:+ continue+ child_session_id = _session_id_from_file(edge.get("child"))+ if child_session_id in replies or edge["childId"] in notices:+ continue+ missing.append({"childId": edge["childId"], "name": edge.get("name")})+ return {"receipts": not missing, "missing_receipts": missing}+++def summarize_usage(session_text: str) -> dict:+ """Sum assistant tokens and assistant turns from the session JSONL.++ In --mode json each completed assistant message is emitted once on+ message_end; turn_end repeats the same message, so only message_end+ events are counted.+ """+ tokens = 0+ turns = 0+ for line in session_text.splitlines():+ try:+ entry = json.loads(line)+ except ValueError:+ continue+ if not isinstance(entry, dict) or entry.get("type") != "message_end":+ continue+ message = entry.get("message")+ if not isinstance(message, dict) or message.get("role") != "assistant":+ continue+ usage = message.get("usage") or {}+ total = usage.get("totalTokens", 0)+ if not isinstance(total, int) or total <= 0:+ continue+ tokens += total+ turns += 1+ return {"tokens": tokens, "turns": turns}+++def _transcript_records(transcript_text: str):+ """Yield parsed custom_message records from a session JSONL."""+ for line in transcript_text.splitlines():+ try:+ entry = json.loads(line)+ except ValueError:+ continue+ if isinstance(entry, dict) and entry.get("type") == "custom_message":+ yield entry+++def _session_id_from_file(session_file: object) -> str | None:+ """The ledger child path is a <session-id>.jsonl file; extract the id."""+ if not isinstance(session_file, str):+ return None+ segments = _path_segments(session_file)+ if not segments:+ return None+ name = segments[-1]+ return name[:-6] if name.endswith(".jsonl") else namediff --git a/scripts/evals/swarm_fanout/tests/test_swarm_fanout.py b/scripts/evals/swarm_fanout/tests/test_swarm_fanout.pynew file mode 100644index 0000000000..9967ba9b31--- /dev/null+++ b/scripts/evals/swarm_fanout/tests/test_swarm_fanout.py@@ -0,0 +1,1084 @@+"""Model-free self-tests for the swarm-fanout eval harness.++Validates fixture integrity (expected answers recomputed independently+from the shard data, task prompts naming every shard) and the scorer+rubric with synthetic ledgers, transcripts, and artifacts - including+the no-spawn, double-spawn, fabricated-answer, and silent-drop cheats.+The runner is exercised end to end with a stub agent binary that writes+a full fan-out artifact set. No agent, model, or network is invoked.+"""++from __future__ import annotations++import contextlib+import hashlib+import io+import json+import os+import shutil+import socket+import subprocess+import sys+import tempfile+import threading+import time+import unittest+import unittest.mock+from pathlib import Path++HARNESS = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(HARNESS))+import runner # noqa: E402+import scorer # noqa: E402++FIXTURES = HARNESS / "fixtures"+++def fixture_manifest(name: str) -> dict:+ return json.loads((FIXTURES / name / "fixture.json").read_text())+++def recompute_answer(fixture_name: str, shard_file: str, question: str) -> int:+ """Recompute a shard's expected answer independently of the manifest."""+ shard_path = FIXTURES / fixture_name / "shards" / shard_file+ if fixture_name == "json-events":+ records = [json.loads(line) for line in shard_path.read_text().splitlines() if line.strip()]+ if question == 'How many records have action "deploy"?':+ return sum(1 for record in records if record["action"] == "deploy")+ if question == 'What is the total of the bytes field over all records with action "upload"?':+ return sum(record["bytes"] for record in records if record["action"] == "upload")+ if question == 'How many distinct users have at least one record with action "login"?':+ return len({record["user"] for record in records if record["action"] == "login"})+ if question == "What is the maximum value of the bytes field?":+ return max(record["bytes"] for record in records)+ if question == 'How many records have action "logout"?':+ return sum(1 for record in records if record["action"] == "logout")+ if question == "How many records have bytes greater than 500?":+ return sum(1 for record in records if record["bytes"] > 500)+ if question == 'What is the total of the bytes field over all records with action "deploy"?':+ return sum(record["bytes"] for record in records if record["action"] == "deploy")+ if question == "How many distinct users appear in the file?":+ return len({record["user"] for record in records})+ rows = [line.split(",") for line in shard_path.read_text().splitlines() if line.strip()]+ header, data = rows[0], rows[1:]+ columns = {name: index for index, name in enumerate(header)}+ if question == "What is the maximum value of the cpu column?":+ return max(int(row[columns["cpu"]]) for row in data)+ if question == "What is the total of the latency_ms column over rows with cpu >= 90?":+ return sum(int(row[columns["latency_ms"]]) for row in data if int(row[columns["cpu"]]) >= 90)+ if question == "How many rows have memory > 80?":+ return sum(1 for row in data if int(row[columns["memory"]]) > 80)+ if question == "What is the total of the latency_ms column over all rows?":+ return sum(int(row[columns["latency_ms"]]) for row in data)+ if question == "How many rows have cpu >= 90?":+ return sum(1 for row in data if int(row[columns["cpu"]]) >= 90)+ if question == "What is the total of the memory column over all rows?":+ return sum(int(row[columns["memory"]]) for row in data)+ if question == "What is the maximum memory among rows with latency_ms < 10?":+ return max(int(row[columns["memory"]]) for row in data if int(row[columns["latency_ms"]]) < 10)+ if question == "What is the total of the cpu column over rows with memory >= 50?":+ return sum(int(row[columns["cpu"]]) for row in data if int(row[columns["memory"]]) >= 50)+ raise AssertionError(f"unknown question: {question}")+++# --- synthetic artifact builders -------------------------------------------+++def spawn_line(child_id: str, name: str, child_file: str, parent_file: str, depth: int = 1) -> str:+ return json.dumps(+ {+ "v": 1,+ "op": "spawn",+ "at": "2026-09-12T00:00:00.000Z",+ "childId": child_id,+ "parent": parent_file,+ "child": child_file,+ "depth": depth,+ "name": name,+ }+ )+++def delete_line(child_id: str, child_file: str, reason: str = "parent-teardown") -> str:+ return json.dumps(+ {+ "v": 1,+ "op": "delete",+ "at": "2026-09-12T00:00:01.000Z",+ "childId": child_id,+ "child": child_file,+ "reason": reason,+ }+ )+++def rename_line(child_id: str, child_file: str, name: str) -> str:+ return json.dumps(+ {+ "v": 1,+ "op": "rename",+ "at": "2026-09-12T00:00:01.000Z",+ "childId": child_id,+ "child": child_file,+ "name": name,+ }+ )+++def reply_line(child_session_id: str, session_name: str = "worker") -> str:+ """An agent_message custom record delivered into the parent transcript."""+ return json.dumps(+ {+ "type": "custom_message",+ "customType": "agent_message",+ "content": f"[agent-message from child:{session_name}]\n\n17",+ "display": True,+ "details": {+ "id": "agentmsg-1",+ "message": "17",+ "from": {+ "activeSessionId": "active-1",+ "sessionId": child_session_id,+ "sessionName": session_name,+ "runtimeKind": "subagent",+ },+ "fromRelationship": "child",+ "target": {"activeSessionId": "active-parent", "sessionId": "parent-id"},+ },+ "id": "entry-1",+ "parentId": None,+ "timestamp": "2026-09-12T00:00:02.000Z",+ }+ )+++def notice_line(child_id: str, session_name: str, kind: str = "completed_without_reply") -> str:+ return json.dumps(+ {+ "type": "custom_message",+ "customType": "rlm_child_terminal_notice",+ "content": f"[child-exited: no-reply child:{session_name}]",+ "display": True,+ "details": {"kind": kind, "childId": child_id, "sessionName": session_name},+ "id": "entry-2",+ "parentId": "entry-1",+ "timestamp": "2026-09-12T00:00:03.000Z",+ }+ )+++def failure_line(child_id: str, session_name: str) -> str:+ return json.dumps(+ {+ "type": "custom_message",+ "customType": "rlm_child_failure",+ "content": f"[child-failed child:{session_name}]\n\nboom",+ "display": True,+ "details": {"childId": child_id, "sessionName": session_name, "error": "boom"},+ "id": "entry-3",+ "parentId": "entry-2",+ "timestamp": "2026-09-12T00:00:04.000Z",+ }+ )+++def full_ledger(fixture: dict, parent_file: str, artifacts_root: str) -> tuple[str, list[dict]]:+ """A well-formed per-shard fan-out ledger plus its child metadata."""+ lines = [+ json.dumps({"v": 1, "op": "meta", "at": "2026-09-12T00:00:00.000Z", "sessionsDir": "/tmp/sessions"})+ ]+ children = []+ for shard in fixture["shards"]:+ child_id = f"sub-{hashlib.sha256(shard['worker'].encode()).hexdigest()[:8]}"+ child_session = f"child-{hashlib.sha256(shard['file'].encode()).hexdigest()[:12]}"+ child_file = f"{artifacts_root}/parent-id/{child_id}/{child_session}.jsonl"+ lines.append(spawn_line(child_id, shard["worker"], child_file, parent_file))+ children.append(+ {"shard": shard, "childId": child_id, "session": child_session, "child_file": child_file}+ )+ return "\n".join(lines) + "\n", children+++def full_transcript(children: list[dict]) -> str:+ """A parent transcript in which every listed child replied with its answer."""+ header = {"type": "session", "version": 3, "id": "parent-id", "timestamp": "2026-09-12T00:00:00.000Z"}+ lines = [json.dumps(header)]+ for child in children:+ lines.append(reply_line(child["session"], child["shard"]["worker"]))+ return "\n".join(lines) + "\n"+++def full_index(fixture: dict, wrong: dict | None = None) -> str:+ wrong = wrong or {}+ lines = []+ for shard in fixture["shards"]:+ answer = wrong.get(shard["file"], str(shard["expected"]))+ lines.append(f"- {shard['file']}: {answer}")+ return "\n".join(lines) + "\n"+++def ledger_spawn_records(ledger_text: str) -> list[dict]:+ return [+ json.loads(line)+ for line in ledger_text.splitlines()+ if line.strip() and json.loads(line).get("op") == "spawn"+ ]+++def child_dir_mapping(children: list[dict]) -> dict[str, list[str]]:+ """The dir-to-session-files mapping the runner collects post-run."""+ return {child["childId"]: [child["session"] + ".jsonl"] for child in children}+++def passing_outcome(fixture: dict) -> dict:+ ledger_text, children = full_ledger(fixture, "/tmp/sessions/parent-id.jsonl", "/tmp/artifacts")+ return {+ "ledger_text": ledger_text,+ "parent_transcript_text": full_transcript(children),+ "child_session_dirs": child_dir_mapping(children),+ "artifact_text": full_index(fixture),+ "usage": {"tokens": 40_000, "turns": 12},+ "wall_time_s": 123.4,+ }+++class FixtureIntegrity(unittest.TestCase):+ def test_json_events_expected_answers_recompute(self):+ manifest = fixture_manifest("json-events")+ for shard in manifest["shards"]:+ recomputed = recompute_answer("json-events", shard["file"], shard["question"])+ self.assertEqual(recomputed, shard["expected"], shard["file"])++ def test_csv_metrics_expected_answers_recompute(self):+ manifest = fixture_manifest("csv-metrics")+ for shard in manifest["shards"]:+ recomputed = recompute_answer("csv-metrics", shard["file"], shard["question"])+ self.assertEqual(recomputed, shard["expected"], shard["file"])++ def test_task_txt_names_every_shard_and_question(self):+ for name in ("json-events", "csv-metrics"):+ manifest = fixture_manifest(name)+ task = (FIXTURES / name / "task.txt").read_text()+ for shard in manifest["shards"]:+ self.assertIn(f"shards/{shard['file']}: {shard['question']}", task, name)++ def test_task_txt_example_names_its_own_fixture(self):+ # The worked example must reference this fixture's shards: the+ # csv prompt naming events-* shards (or vice versa) would send a+ # compliant model looking for files that do not exist.+ for name, other_prefix in (("json-events", "metrics"), ("csv-metrics", "events-")):+ manifest = fixture_manifest(name)+ task = (FIXTURES / name / "task.txt").read_text()+ first = manifest["shards"][0]+ self.assertIn(f"for shards/{first['file']} the child name is {first['worker']}", task)+ self.assertNotIn(f"{other_prefix}-01", task)++ def test_worker_names_and_artifact_are_wellformed(self):+ for name in ("json-events", "csv-metrics"):+ manifest = fixture_manifest(name)+ self.assertEqual(manifest["artifact"], "combined-index.md")+ self.assertEqual(len(manifest["shards"]), 8)+ workers = [shard["worker"] for shard in manifest["shards"]]+ self.assertEqual(len(set(workers)), len(workers), name)+ for shard in manifest["shards"]:+ stem = shard["file"].rsplit(".", 1)[0]+ self.assertEqual(shard["worker"], f"worker-{stem}")+++class ScorerTests(unittest.TestCase):+ def setUp(self):+ self.fixture = fixture_manifest("json-events")++ def test_full_resolution(self):+ result = scorer.score_fixture(self.fixture, passing_outcome(self.fixture))+ self.assertTrue(result["resolved"])+ self.assertTrue(result["coverage"])+ self.assertTrue(result["delegation_evidence"])+ self.assertTrue(result["dedup"])+ self.assertTrue(result["receipts"])+ self.assertEqual(result["tokens_used"], 40_000)+ self.assertEqual(result["turns"], 12)+ self.assertEqual(result["wall_time_s"], 123.4)++ def test_no_spawn_blocks_resolution(self):+ # The parent answered everything itself: no ledger, no children.+ outcome = passing_outcome(self.fixture)+ outcome["ledger_text"] = ""+ outcome["parent_transcript_text"] = ""+ outcome["child_session_dirs"] = {}+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertTrue(result["coverage"])+ self.assertFalse(result["delegation_evidence"])+ self.assertFalse(result["resolved"])++ def test_fabricated_and_missing_answers_block_coverage(self):+ outcome = passing_outcome(self.fixture)+ wrong = {self.fixture["shards"][0]["file"]: "999999"}+ outcome["artifact_text"] = full_index(self.fixture, wrong)+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["coverage"])+ self.assertEqual(len(result["wrong_answers"]), 1)+ # A dropped shard bullet is also a coverage failure.+ partial = full_index(self.fixture)+ outcome["artifact_text"] = "\n".join(partial.splitlines()[:-1]) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["coverage"])+ self.assertEqual(len(result["missing_shards"]), 1)++ def test_delegation_requires_child_session_dirs(self):+ # Ledger edges exist but the child session dirs were never written.+ outcome = passing_outcome(self.fixture)+ outcome["child_session_dirs"] = {}+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertFalse(result["resolved"])++ def test_delegation_requires_distinct_names(self):+ outcome = passing_outcome(self.fixture)+ # Rename every child to one name: eight live edges, one distinct+ # name, so the fan-out degenerated to a pile of duplicate workers.+ lines = outcome["ledger_text"].splitlines()+ renames = [+ rename_line(record["childId"], record["child"], "worker-same")+ for record in ledger_spawn_records(outcome["ledger_text"])+ ]+ outcome["ledger_text"] = "\n".join(lines + renames) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertEqual(result["distinct_worker_names"], 1)++ def test_delegation_requires_shard_worker_edges(self):+ # Eight live depth-1 children named helper-N (none matching a+ # shard's worker) with real session dirs still fail delegation:+ # helper spawns cannot substitute for per-shard delegation.+ outcome = passing_outcome(self.fixture)+ records = [json.loads(line) for line in outcome["ledger_text"].splitlines()]+ rewritten = []+ for record in records:+ if record.get("op") == "spawn":+ record = {**record, "name": "helper-" + record["childId"][-4:]}+ rewritten.append(json.dumps(record))+ outcome["ledger_text"] = "\n".join(rewritten) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertEqual(result["verified_shard_workers"], 0)++ def test_delegation_requires_edge_backed_session_dirs(self):+ # The collected dirs exist but belong to sessions the ledger edges+ # never recorded, so no worker edge is backed by a real child+ # session dir and delegation fails.+ outcome = passing_outcome(self.fixture)+ outcome["child_session_dirs"] = {+ f"sub-unrelated{index:02d}": [f"session-{index:02d}.jsonl"] for index in range(8)+ }+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertEqual(result["verified_shard_workers"], 0)++ def test_depth2_rename_does_not_flag_depth1_spawn(self):+ # A grandchild renamed to a shard worker name must not shadow the+ # parent's own depth-1 spawn of that worker: rename bookkeeping+ # applies only to the fan-out's depth-1 children.+ ledger = "\n".join(+ [+ spawn_line("sub-grand0000", "worker-tmp", "/tmp/g.jsonl", "/tmp/gp.jsonl", depth=2),+ rename_line("sub-grand0000", "/tmp/g.jsonl", self.fixture["shards"][0]["worker"]),+ spawn_line(+ "sub-kid000000",+ self.fixture["shards"][0]["worker"],+ "/tmp/artifacts/parent-id/sub-kid000000/c000000000001.jsonl",+ "/tmp/sessions/parent-id.jsonl",+ ),+ ]+ )+ records, _malformed = scorer.parse_ledger(ledger)+ result = scorer.score_dedup(self.fixture, records)+ self.assertTrue(result["dedup"])+ self.assertEqual(result["total_spawns"], 1)++ def test_parse_ledger_rejects_type_invalid_records(self):+ # Records that fail the product ledger schema must not replay as+ # delegation evidence: a spawn missing parent/child, or with a+ # string depth, is malformed and skipped.+ bad_spawn = json.dumps(+ {+ "v": 1,+ "op": "spawn",+ "at": "2026-09-12T00:00:00.000Z",+ "childId": "sub-x",+ "depth": 1,+ "name": "worker-events-01",+ }+ )+ bad_depth = json.dumps(+ {+ "v": 1,+ "op": "spawn",+ "at": "2026-09-12T00:00:00.000Z",+ "childId": "sub-x",+ "parent": "/tmp/p.jsonl",+ "child": "/tmp/c.jsonl",+ "depth": "1",+ "name": "worker-events-01",+ }+ )+ unknown_op = json.dumps({"v": 1, "op": "future", "at": "2026-09-12T00:00:00.000Z"})+ good_spawn = spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl")+ records, malformed = scorer.parse_ledger(+ "\n".join([bad_spawn, bad_depth, unknown_op, good_spawn]) + "\n"+ )+ self.assertEqual(malformed, 2)+ self.assertEqual(len(records), 1)+ self.assertEqual(records[0]["childId"], "sub-1")++ def test_score_fixture_tolerates_missing_child_dirs_key(self):+ # An outcome without child session dirs must score (delegation+ # false), not crash the dict-based rubric.+ outcome = passing_outcome(self.fixture)+ del outcome["child_session_dirs"]+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertFalse(result["resolved"])++ def test_delegation_requires_distinct_session_claims(self):+ # Two worker edges pointing at the same child session file is one+ # child masquerading as a per-shard fan-out: each verified worker+ # must claim its own session dir and file.+ outcome = passing_outcome(self.fixture)+ records = ledger_spawn_records(outcome["ledger_text"])+ first = records[0]+ shared_file = first["child"]+ rewritten = []+ for record in [json.loads(line) for line in outcome["ledger_text"].splitlines()]:+ if record.get("op") == "spawn" and record["name"] != first["name"]:+ record = {**record, "child": shared_file}+ rewritten.append(json.dumps(record))+ outcome["ledger_text"] = "\n".join(rewritten) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["delegation_evidence"])+ self.assertEqual(result["verified_shard_workers"], 1)++ def test_renamed_worker_does_not_free_name_for_duplicate(self):+ # Renaming the first worker-events-01 child away must not free the+ # name for a second spawn while the first child is still live: the+ # shard assignment is the spawn name, not the current name.+ outcome = passing_outcome(self.fixture)+ lines = outcome["ledger_text"].splitlines()+ first = ledger_spawn_records(outcome["ledger_text"])[0]+ duplicate = spawn_line("sub-secondspawn", first["name"], first["child"] + "2", first["parent"])+ outcome["ledger_text"] = (+ "\n".join(+ lines + [rename_line(first["childId"], first["child"], "worker-renamed-away"), duplicate]+ )+ + "\n"+ )+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["dedup"])+ self.assertEqual(len(result["duplicate_spawns"]), 1)+ self.assertEqual(result["total_spawns"], 9)++ def test_conflicting_duplicate_answers_block_coverage(self):+ # Order-independent: a wrong bullet must fail coverage whether it+ # lands before or after the correct one. (Last-wins scoring would+ # pass this when the correction lands last.)+ outcome = passing_outcome(self.fixture)+ artifact = outcome["artifact_text"].rstrip()+ first_shard = self.fixture["shards"][0]["file"]+ expected = str(self.fixture["shards"][0]["expected"])+ outcome["artifact_text"] = artifact + "\n" + f"- {first_shard}: 999999\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["coverage"])+ # A correction after a wrong bullet still fails: the artifact+ # carried a fabricated answer, and the scorer keeps every bullet.+ outcome["artifact_text"] = (+ artifact + "\n" + f"- {first_shard}: 999999\n" + f"- {first_shard}: {expected}\n"+ )+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["coverage"])+ # Identical duplicate bullets that all match stay covered.+ outcome["artifact_text"] = artifact + "\n" + f"- {first_shard}: {expected}\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertTrue(result["coverage"])++ def test_replay_edges_keyed_by_child_id_and_path(self):+ # Two spawns sharing a childId but naming different child+ # sessions are two distinct edges (the product keys an edge by+ # childId plus canonical child path); a delete names its own+ # edge and must not tombstone the sibling.+ ledger = "\n".join(+ [+ spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl"),+ spawn_line("sub-1", "worker-b", "/tmp/b.jsonl", "/tmp/p.jsonl"),+ delete_line("sub-1", "/tmp/a.jsonl"),+ ]+ )+ records, _malformed = scorer.parse_ledger(ledger)+ edges = scorer.replay_edges(records)+ self.assertEqual(len(edges), 2)+ live = [edge for edge in edges.values() if edge["deleted"] is None]+ self.assertEqual(len(live), 1)+ self.assertEqual(live[0]["name"], "worker-b")++ def test_edge_session_paths_use_platform_separators(self):+ # The product writes ledger paths with the host separator; a+ # Windows backslash-joined child path must verify delegation and+ # receipts exactly like a POSIX one.+ fixture = self.fixture+ ledger_text, children = full_ledger(fixture, "\\tmp\\sessions\\parent-id.jsonl", "\\tmp\\artifacts")+ backslash_children = []+ lines = [json.loads(line) for line in ledger_text.splitlines()]+ rewritten = []+ for record in lines:+ if record.get("op") == "spawn":+ record = {**record, "child": record["child"].replace("/", "\\")}+ backslash_children.append(record)+ if record.get("op") == "meta":+ record = {**record, "sessionsDir": record["sessionsDir"]}+ rewritten.append(json.dumps(record))+ outcome = {+ "ledger_text": "\n".join(rewritten) + "\n",+ "parent_transcript_text": full_transcript(children),+ "child_session_dirs": child_dir_mapping(+ [+ {+ "childId": child["childId"],+ "session": child["session"],+ }+ for child in children+ ]+ ),+ "artifact_text": full_index(fixture),+ "usage": {"tokens": 1, "turns": 1},+ "wall_time_s": 1.0,+ }+ result = scorer.score_fixture(fixture, outcome)+ self.assertTrue(result["delegation_evidence"])+ self.assertTrue(result["receipts"])+ self.assertTrue(result["resolved"])++ def test_receipts_require_explicit_child_relationship(self):+ # A reply record with the relationship field missing is not a+ # receipt: the daemon always stamps it on delivered messages.+ outcome = passing_outcome(self.fixture)+ outcome["parent_transcript_text"] = (+ "\n".join(+ line.replace('"fromRelationship": "child",', "")+ for line in outcome["parent_transcript_text"].splitlines()+ )+ + "\n"+ )+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["receipts"])++ def test_double_spawn_blocks_dedup(self):+ outcome = passing_outcome(self.fixture)+ lines = outcome["ledger_text"].splitlines()+ first = ledger_spawn_records(outcome["ledger_text"])[0]+ duplicate = spawn_line("sub-duplicate00", first["name"], first["child"] + "2", first["parent"])+ outcome["ledger_text"] = "\n".join(lines + [duplicate]) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["dedup"])+ self.assertEqual(len(result["duplicate_spawns"]), 1)+ self.assertEqual(result["total_spawns"], 9)++ def test_retry_after_delete_stays_within_dedup(self):+ # One child failed, was deleted, and a replacement with the same+ # worker name was spawned in its own session dir; the replacement+ # also replied. This is the retry the task prompt grants, and it+ # resolves end to end.+ outcome = passing_outcome(self.fixture)+ lines = outcome["ledger_text"].splitlines()+ first = ledger_spawn_records(outcome["ledger_text"])[0]+ replacement_file = "/tmp/artifacts/parent-id/sub-retry00000/child-retry00000.jsonl"+ replacement = spawn_line("sub-retry00000", first["name"], replacement_file, first["parent"])+ outcome["ledger_text"] = (+ "\n".join(lines + [delete_line(first["childId"], first["child"]), replacement]) + "\n"+ )+ outcome["child_session_dirs"]["sub-retry00000"] = ["child-retry00000.jsonl"]+ transcript = outcome["parent_transcript_text"].rstrip()+ outcome["parent_transcript_text"] = (+ transcript + "\n" + reply_line("child-retry00000", first["name"]) + "\n"+ )+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertTrue(result["dedup"])+ self.assertTrue(result["receipts"])+ self.assertTrue(result["delegation_evidence"])+ self.assertEqual(result["total_spawns"], 9)+ self.assertTrue(result["resolved"])++ def test_spawn_budget_caps_total_spawns(self):+ # Eight shard workers plus nine helper spawns: seventeen depth-1+ # spawns exceed the one-retry-per-shard budget (2 x 8 = 16).+ outcome = passing_outcome(self.fixture)+ lines = outcome["ledger_text"].splitlines()+ first = ledger_spawn_records(outcome["ledger_text"])[0]+ helpers = []+ for index in range(9):+ helpers.append(+ spawn_line(+ f"sub-helper{index:05d}",+ f"helper-{index}",+ f"/tmp/artifacts/parent-id/sub-helper{index:05d}/child{index}.jsonl",+ first["parent"],+ )+ )+ outcome["ledger_text"] = "\n".join(lines + helpers) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertEqual(result["total_spawns"], 17)+ self.assertFalse(result["dedup"])++ def test_silent_drop_blocks_receipts(self):+ outcome = passing_outcome(self.fixture)+ records = ledger_spawn_records(outcome["ledger_text"])+ children = [+ {+ "shard": {"worker": record["name"]},+ "session": record["child"].rsplit("/", 1)[-1][:-6],+ }+ for record in records+ ]+ # Every child replied except the last, which neither replied nor+ # left a notice: a dropped child must fail receipt completeness.+ outcome["parent_transcript_text"] = full_transcript(children[:-1])+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["receipts"])+ self.assertEqual(len(result["missing_receipts"]), 1)++ def test_terminal_and_failure_notices_count_as_receipts(self):+ outcome = passing_outcome(self.fixture)+ records = ledger_spawn_records(outcome["ledger_text"])+ lines = [outcome["parent_transcript_text"].rstrip()]+ # Two children never replied; one left a completed-without-reply+ # notice, the other a failure notice. Both are accounted for.+ lines.append(notice_line(records[0]["childId"], records[0]["name"]))+ lines.append(failure_line(records[1]["childId"], records[1]["name"]))+ outcome["parent_transcript_text"] = "\n".join(lines) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertTrue(result["receipts"])++ def test_deleted_child_without_notice_blocks_receipts(self):+ # Deleting a child that never replied hides the drop: the delete+ # record lands in the ledger, but no parent-visible receipt exists.+ # A deleted child that did reply stays receipted, so the drop+ # must be a child with neither reply nor notice.+ outcome = passing_outcome(self.fixture)+ records = ledger_spawn_records(outcome["ledger_text"])+ dropped = records[-1]+ kept = [+ {"shard": {"worker": record["name"]}, "session": record["child"].rsplit("/", 1)[-1][:-6]}+ for record in records[:-1]+ ]+ outcome["parent_transcript_text"] = full_transcript(kept)+ lines = outcome["ledger_text"].splitlines()+ outcome["ledger_text"] = "\n".join(lines + [delete_line(dropped["childId"], dropped["child"])]) + "\n"+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["receipts"])+ self.assertEqual(len(result["missing_receipts"]), 1)+ self.assertFalse(result["delegation_evidence"])++ def test_receipt_requires_child_relationship(self):+ outcome = passing_outcome(self.fixture)+ # A sibling-labeled message carrying the child's session id is not+ # a receipt: the child's reply never actually arrived.+ outcome["parent_transcript_text"] = (+ "\n".join(+ line.replace('"fromRelationship": "child"', '"fromRelationship": "sibling"')+ for line in outcome["parent_transcript_text"].splitlines()+ )+ + "\n"+ )+ result = scorer.score_fixture(self.fixture, outcome)+ self.assertFalse(result["receipts"])++ def test_replay_edges_last_writer_wins(self):+ ledger = "\n".join(+ [+ spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl"),+ spawn_line("sub-2", "worker-b", "/tmp/b.jsonl", "/tmp/p.jsonl"),+ rename_line("sub-2", "/tmp/b.jsonl", "worker-c"),+ delete_line("sub-1", "/tmp/a.jsonl"),+ ]+ )+ records, malformed = scorer.parse_ledger(ledger)+ self.assertEqual(malformed, 0)+ edges = scorer.replay_edges(records)+ self.assertEqual(len(edges), 2)+ by_child = {edge["childId"]: edge for edge in edges.values()}+ self.assertEqual(by_child["sub-2"]["name"], "worker-c")+ self.assertIsNone(by_child["sub-2"]["deleted"])+ self.assertEqual(by_child["sub-1"]["deleted"], "parent-teardown")++ def test_parse_ledger_skips_malformed_lines(self):+ ledger = "{not json}\n" + spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl") + "\n\n"+ records, malformed = scorer.parse_ledger(ledger)+ self.assertEqual(malformed, 1)+ self.assertEqual(len(records), 1)++ def test_summarize_usage_counts_each_assistant_message_once(self):+ message = {"role": "assistant", "usage": {"input": 60, "output": 40, "totalTokens": 100}}+ session = (+ json.dumps({"type": "message_end", "message": message})+ + "\n"+ + json.dumps({"type": "turn_end", "message": message, "toolResults": []})+ )+ self.assertEqual(scorer.summarize_usage(session), {"tokens": 100, "turns": 1})++ def test_summarize_usage_ignores_malformed_lines(self):+ session = "not json\n" + json.dumps({"type": "message_end"}) + "\n"+ self.assertEqual(scorer.summarize_usage(session), {"tokens": 0, "turns": 0})+++# --- stub agent for end-to-end runner tests ---------------------------------+#+# The stub is a stand-in for the real CLI: it receives the same launch+# arguments the runner would give prime-agent, reads the isolated agent+# home from PRIME_AGENT_CODING_AGENT_DIR, and writes the exact artifact+# set a correct fan-out leaves behind (ledger, parent transcript, child+# session dirs, combined index, json-mode stdout events).++STUB_BODY = """\+#!/usr/bin/env python3+# Stub agent for the swarm-fanout eval self-tests.+import hashlib+import json+import os+import sys+from pathlib import Path++ANSWERS = __ANSWERS__+MODE = "__MODE__"+PARENT_ID = "019aaaaa-aaaa-4aaa-8aaa-00000000000a"+++def assistant_event(tokens):+ usage = {"role": "assistant", "usage": {"input": 10, "output": tokens - 10, "totalTokens": tokens}}+ return json.dumps({"type": "message_end", "message": usage})+++argv = sys.argv[1:]+# --mode json --daemon-socket SOCK --cwd REPO --session-dir SESSIONS --model M -- PROMPT+repo = Path(argv[5])+sessions_dir = Path(os.path.realpath(argv[7]))+agent_home = Path(os.environ["PRIME_AGENT_CODING_AGENT_DIR"])+parent_file = sessions_dir / (PARENT_ID + ".jsonl")++index_lines = [f"- {shard_file}: {answer}" for shard_file, answer in ANSWERS.items()]+(repo / "combined-index.md").write_text("\\n".join(index_lines) + "\\n")++if MODE == "artifact-only":+ # The no-spawn cheat: correct answers, zero delegation.+ print(assistant_event(1000))+ sys.exit(0)++artifacts_root = sessions_dir.parent / "session-artifacts"+ledger_lines = [+ json.dumps({"v": 1, "op": "meta", "at": "2026-09-12T00:00:00.000Z", "sessionsDir": str(sessions_dir)})+]+session_header = {+ "type": "session",+ "version": 3,+ "id": PARENT_ID,+ "timestamp": "2026-09-12T00:00:00.000Z",+ "cwd": str(repo),+}+transcript_lines = [json.dumps(session_header)]+for shard_file, answer in ANSWERS.items():+ stem = shard_file.rsplit(".", 1)[0]+ child_id = "sub-" + hashlib.sha256(stem.encode()).hexdigest()[:8]+ child_session = hashlib.sha256(("s" + shard_file).encode()).hexdigest()[:16]+ child_dir = artifacts_root / PARENT_ID / child_id+ child_dir.mkdir(parents=True, exist_ok=True)+ child_file = child_dir / (child_session + ".jsonl")+ child_header = {+ "type": "session",+ "version": 3,+ "id": child_session,+ "timestamp": "2026-09-12T00:00:00.000Z",+ }+ child_file.write_text(json.dumps(child_header) + "\\n")+ ledger_lines.append(+ json.dumps(+ {+ "v": 1,+ "op": "spawn",+ "at": "2026-09-12T00:00:00.500Z",+ "childId": child_id,+ "parent": str(parent_file),+ "child": str(child_file),+ "depth": 1,+ "name": "worker-" + stem,+ }+ )+ )+ transcript_lines.append(+ json.dumps(+ {+ "type": "custom_message",+ "customType": "agent_message",+ "content": "[agent-message from child:worker-" + stem + "]\\n\\n" + str(answer),+ "display": True,+ "details": {+ "id": "agentmsg-" + child_session,+ "message": str(answer),+ "from": {+ "activeSessionId": "active-" + child_session[:8],+ "sessionId": child_session,+ "sessionName": "worker-" + stem,+ "runtimeKind": "subagent",+ },+ "fromRelationship": "child",+ "target": {"activeSessionId": "active-parent", "sessionId": PARENT_ID},+ },+ "id": "entry-" + child_session[:8],+ "parentId": None,+ "timestamp": "2026-09-12T00:00:02.000Z",+ }+ )+ )++ledger_dir = agent_home / "rlm-ledger"+ledger_dir.mkdir(parents=True, exist_ok=True)+digest = hashlib.sha256(str(sessions_dir).encode("utf-8")).hexdigest()[:16]+(ledger_dir / (digest + ".jsonl")).write_text("\\n".join(ledger_lines) + "\\n")+parent_file.write_text("\\n".join(transcript_lines) + "\\n")+print(assistant_event(1000))+print(assistant_event(500))+sys.exit(0)+"""+++def write_stub_agent(fixture: dict, mode: str = "full") -> Path:+ answers = {shard["file"]: shard["expected"] for shard in fixture["shards"]}+ script_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-stub-"))+ script = script_dir / "agent"+ body = STUB_BODY.replace("__ANSWERS__", repr(answers)).replace("__MODE__", mode)+ script.write_text(body)+ script.chmod(0o755)+ return script+++class LedgerPathTests(unittest.TestCase):+ def test_ledger_hash_matches_product_computation(self):+ # rlmLedgerPath hashes the canonical sessions dir into the ledger+ # file name: sha256(canonical)[:16] under <agentDir>/rlm-ledger/.+ agent_home = Path("/tmp/agent-home")+ sessions_dir = "/tmp/sessions"+ candidates = runner.ledger_file_candidates(agent_home, sessions_dir)+ canonical = runner.canonicalize_dir_path(sessions_dir)+ digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]+ self.assertEqual(candidates[0], agent_home / "rlm-ledger" / f"{digest}.jsonl")++ def test_canonicalize_realpaths_existing_dirs(self):+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-canon-"))+ try:+ symlink = workdir / "linked"+ real = workdir / "real"+ real.mkdir()+ symlink.symlink_to(real, target_is_directory=True)+ self.assertEqual(runner.canonicalize_dir_path(str(symlink)), str(real.resolve()))+ finally:+ shutil.rmtree(workdir, ignore_errors=True)++ def test_canonicalize_missing_dir_stays_resolved(self):+ # rlmLedgerPath falls back to plain resolve when the dir does not+ # exist yet, without resolving ancestor symlinks.+ missing = str(Path(tempfile.gettempdir()) / "swarm-fanout-missing-dir")+ self.assertEqual(runner.canonicalize_dir_path(missing), os.path.abspath(missing))++ def test_ledger_candidates_cover_both_variants(self):+ agent_home = Path(tempfile.mkdtemp(prefix="swarm-fanout-home-"))+ try:+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-cand-"))+ try:+ real = workdir / "real"+ real.mkdir()+ symlinked = workdir / "link"+ symlinked.symlink_to(real, target_is_directory=True)+ candidates = runner.ledger_file_candidates(agent_home, str(symlinked))+ self.assertEqual(len(candidates), 2)+ canonical = hashlib.sha256(str(real.resolve()).encode("utf-8")).hexdigest()[:16]+ plain = hashlib.sha256(str(symlinked).encode("utf-8")).hexdigest()[:16]+ self.assertEqual(candidates[0].name, f"{canonical}.jsonl")+ self.assertEqual(candidates[1].name, f"{plain}.jsonl")+ finally:+ shutil.rmtree(workdir, ignore_errors=True)+ finally:+ shutil.rmtree(agent_home, ignore_errors=True)+++class RunnerTests(unittest.TestCase):+ """The runner must emit a scored result in every launch failure mode."""++ def run_runner(self, argv: list[str]) -> tuple[int, dict]:+ captured = io.StringIO()+ with contextlib.redirect_stdout(captured):+ exit_code = runner.main(argv)+ return exit_code, json.loads(captured.getvalue())++ def test_stub_agent_full_fanout_resolves(self):+ fixture = fixture_manifest("json-events")+ stub = write_stub_agent(fixture, mode="full")+ argv = ["--fixture", str(FIXTURES / "json-events"), "--model", "test/fake", "--agent-bin", str(stub)]+ exit_code, result = self.run_runner(argv)+ self.assertEqual(exit_code, 0)+ self.assertTrue(result["resolved"])+ self.assertTrue(result["coverage"])+ self.assertTrue(result["delegation_evidence"])+ self.assertEqual(result["child_session_dirs"], 8)+ self.assertEqual(result["total_spawns"], 8)+ self.assertTrue(result["dedup"])+ self.assertTrue(result["receipts"])+ self.assertEqual(result["tokens_used"], 1500)+ self.assertEqual(result["turns"], 2)+ self.assertFalse(result["timed_out"])++ def test_stub_agent_without_spawns_scores_unresolved(self):+ # The artifact-only stub answers everything without spawning: the+ # no-spawn cheat must fail delegation evidence even though every+ # answer is correct.+ fixture = fixture_manifest("csv-metrics")+ stub = write_stub_agent(fixture, mode="artifact-only")+ argv = ["--fixture", str(FIXTURES / "csv-metrics"), "--model", "test/fake", "--agent-bin", str(stub)]+ exit_code, result = self.run_runner(argv)+ self.assertEqual(exit_code, 1)+ self.assertTrue(result["coverage"])+ self.assertFalse(result["delegation_evidence"])+ self.assertFalse(result["resolved"])++ def test_missing_agent_bin_scores_unresolved(self):+ argv = [+ "--fixture",+ str(FIXTURES / "json-events"),+ "--model",+ "test/fake",+ "--agent-bin",+ "/nonexistent-agent",+ ]+ exit_code, result = self.run_runner(argv)+ self.assertEqual(exit_code, 1)+ self.assertFalse(result["resolved"])+ self.assertTrue(result["agent_error"])++ def test_agent_timeout_still_scores(self):+ script_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-agent-"))+ script = script_dir / "agent"+ script.write_text("#!/bin/sh\nexec sleep 30\n")+ script.chmod(0o755)+ argv = [+ "--fixture",+ str(FIXTURES / "json-events"),+ "--model",+ "test/fake",+ "--agent-bin",+ str(script),+ "--timeout",+ "1",+ ]+ exit_code, result = self.run_runner(argv)+ self.assertEqual(exit_code, 1)+ self.assertFalse(result["resolved"])+ self.assertTrue(result["timed_out"])+ self.assertIsNone(result["exit_code"])++ def test_init_fixture_repo_ignores_git_location_env(self):+ # GIT_DIR / GIT_INDEX_FILE / GIT_WORK_TREE from the caller must not+ # redirect the fixture init into another repository.+ decoy = Path(tempfile.mkdtemp(prefix="swarm-fanout-gitenv-"))+ subprocess.run(["git", "init", "-q", "-b", "main"], cwd=decoy, check=True)+ (decoy / "marker.txt").write_text("decoy\n")+ subprocess.run(["git", "add", "marker.txt"], cwd=decoy, check=True)+ subprocess.run(["git", "-c", "commit.gpgsign=false", "commit", "-qm", "decoy"], cwd=decoy, check=True)+ ref = decoy / ".git" / "refs" / "heads" / "main"+ ref_before = ref.read_text()+ fixture = fixture_manifest("json-events")+ stub = write_stub_agent(fixture, mode="artifact-only")+ argv = ["--fixture", str(FIXTURES / "json-events"), "--model", "test/fake", "--agent-bin", str(stub)]+ git_env = {+ "GIT_DIR": str(decoy / ".git"),+ "GIT_WORK_TREE": str(decoy),+ "GIT_INDEX_FILE": str(decoy / ".git" / "index"),+ }+ saved = {key: os.environ.get(key) for key in git_env}+ try:+ os.environ.update(git_env)+ exit_code, result = self.run_runner(argv)+ finally:+ for key, value in saved.items():+ if value is None:+ os.environ.pop(key, None)+ else:+ os.environ[key] = value+ self.assertEqual(exit_code, 1) # artifact-only stub: unresolved, but the repo must init+ repo = Path(result["workdir"]) / "repo"+ self.assertTrue((repo / ".git").is_dir(), "fixture repo was not initialized in the workdir")+ self.assertTrue((repo / "shards" / "events-01.jsonl").is_file())+ self.assertEqual(ref.read_text(), ref_before, "fixture commit leaked into the decoy repo")+ self.assertFalse((decoy / "shards").exists())++ def test_agent_env_strips_embedding_overrides(self):+ # RLM depth overrides from an embedding session must not leak into+ # the eval agent: the eval parent must be an independent root.+ env = dict(os.environ)+ env.update({"RLM_DEPTH": "2", "RLM_MAX_DEPTH": "2", "PRIME_AGENT_INTERNAL_TOKEN": "x"})+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-env-"))+ try:+ with unittest.mock.patch.dict(os.environ, env, clear=True):+ result = runner.agent_env(workdir / "agent-home", str(workdir / "sessions"))+ self.assertNotIn("RLM_DEPTH", result)+ self.assertNotIn("RLM_MAX_DEPTH", result)+ self.assertNotIn("PRIME_AGENT_INTERNAL_TOKEN", result)+ self.assertEqual(result["PRIME_AGENT_SESSION_DIR"], str(workdir / "sessions"))+ self.assertEqual(result["PRIME_AGENT_CODING_AGENT_DIR"], str(workdir / "agent-home"))+ finally:+ shutil.rmtree(workdir, ignore_errors=True)++ def test_shutdown_agent_daemon_waits_for_late_socket(self):+ # A timed-out launch can die before the detached daemon binds; the+ # retry window must still deliver the shutdown once it appears.+ socket_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-late-"))+ socket_path = socket_dir / "daemon.sock"+ server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)+ received = []++ def serve() -> None:+ time.sleep(1.0)+ server.bind(str(socket_path))+ server.listen(1)+ connection, _ = server.accept()+ received.append(connection.recv(1024).decode())+ connection.close()++ thread = threading.Thread(target=serve)+ thread.start()+ runner.shutdown_agent_daemon(socket_path, retry_window_s=5.0)+ thread.join(timeout=10)+ server.close()+ self.assertEqual(len(received), 1)+ self.assertEqual(json.loads(received[0])["command"]["type"], "shutdown")++ def test_shutdown_agent_daemon_sends_shutdown_command(self):+ server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)+ socket_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-sock-"))+ socket_path = socket_dir / "daemon.sock"+ server.bind(str(socket_path))+ server.listen(1)+ received = []++ def serve() -> None:+ connection, _ = server.accept()+ received.append(connection.recv(1024).decode())+ connection.close()++ thread = threading.Thread(target=serve)+ thread.start()+ runner.shutdown_agent_daemon(socket_path)+ thread.join(timeout=5)+ server.close()+ self.assertEqual(len(received), 1)+ envelope = json.loads(received[0])+ self.assertEqual(envelope["type"], "command")+ self.assertEqual(envelope["command"]["type"], "shutdown")+++if __name__ == "__main__":+ unittest.main()diff --git a/scripts/evals/swarm_fanout/uv.lock b/scripts/evals/swarm_fanout/uv.locknew file mode 100644index 0000000000..905047fec0--- /dev/null+++ b/scripts/evals/swarm_fanout/uv.lock@@ -0,0 +1,46 @@+version = 1+revision = 3+requires-python = ">=3.11"++[options]+exclude-newer = "2026-09-02T00:00:00Z"++[[package]]+name = "prime-agent-evals"+version = "0.1.0"+source = { virtual = "." }++[package.dev-dependencies]+dev = [+ { name = "ruff" },+]++[package.metadata]++[package.metadata.requires-dev]+dev = [{ name = "ruff", specifier = "==0.16.5" }]++[[package]]+name = "ruff"+version = "0.16.5"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/f3/85/c8e12473c93018f92d19dd988a294202e1c27426c47ec4de53ffb847b8d8/ruff-0.16.5.tar.gz", hash = "sha256:1b88500f9ffbcab3dedb0082c9f9492e91ec3d618aac1236a3e0189938f7040b", size = 4912003, upload-time = "2026-08-27T16:34:18.258Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/c6/b6/77c90a970fe2dae17a723acbd011043ea97c98d7deacccefdc4ba74ec512/ruff-0.16.5-py3-none-linux_armv6l.whl", hash = "sha256:12e5f673e774c35fbb62f288809c7653b73445f8ecec6b6063fd6ea3521aa14b", size = 10011941, upload-time = "2026-08-27T16:33:41.287Z" },+ { url = "https://files.pythonhosted.org/packages/4b/46/6cf67cf6411885a1d6f7f6d801682f155536a85176d10b605e2ceffed8bd/ruff-0.16.5-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:eda58a5802de40e7ed5b32b64e0b32539338cc6fcd2c78f61e3ad6a0d79f51c3", size = 10204049, upload-time = "2026-08-27T16:33:44.056Z" },+ { url = "https://files.pythonhosted.org/packages/46/fd/c8720ca7a090abf0c2fef4abe8a5ef6e5127ed15196d8886ff75a2b370e2/ruff-0.16.5-py3-none-macosx_11_0_arm64.whl", hash = "sha256:c5ae9a7b9a8875131f40f8fe967cc86abf899779efd663cb7ce3d572d01da7eb", size = 9809037, upload-time = "2026-08-27T16:33:46.257Z" },+ { url = "https://files.pythonhosted.org/packages/43/45/a684caacdedaca180f52bacccc40bf0789d2c5a7c75f25324853e9eaedb5/ruff-0.16.5-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7b719b0a1f4d59710d283ab2965f621684a108a9e41da622e3b23f0326cd0025", size = 9964129, upload-time = "2026-08-27T16:33:48.352Z" },+ { url = "https://files.pythonhosted.org/packages/9e/f2/5d2bcdaca6b5b93d1b4dfc166cd2aebf7680143a1b38a28759df13a94d31/ruff-0.16.5-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2298f2780ed1be0c5cb1361e32ab7b1467f3cce7dabe101d2210a314f2fe42e9", size = 9821518, upload-time = "2026-08-27T16:33:50.57Z" },+ { url = "https://files.pythonhosted.org/packages/aa/ff/011cce29accf9257d5974145b733fc653a37985ed6825413a3987cefbfe0/ruff-0.16.5-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:258f29035a2dd021e7861e631b227a5b3f14e50c1184c9a6a122c5f4576154d7", size = 10534835, upload-time = "2026-08-27T16:33:52.522Z" },+ { url = "https://files.pythonhosted.org/packages/d7/5a/f0cf109bada9bba0e96c90c21c9f9251803f57225c32d293327a03c710d6/ruff-0.16.5-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b9a4f0432966834019c74d1b7e5c51224305d7713f3d7faf3e7451f1a3be3cde", size = 11252550, upload-time = "2026-08-27T16:33:54.521Z" },+ { url = "https://files.pythonhosted.org/packages/63/4d/1d481aaea2046c6a7ed7c291f9004c669cce3c087b6b376ed5b08271e3fe/ruff-0.16.5-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b5eb3a8c3d0ade9cea42b591fd530368e8798380e30e0a308b85a5cf718f09ea", size = 10777949, upload-time = "2026-08-27T16:33:56.88Z" },+ { url = "https://files.pythonhosted.org/packages/ee/34/ee245ca55f64443233034b3d02b03236b19242004281247c079390b7facd/ruff-0.16.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ef0f69e191a13a3c9816f63163c88790cb12cd157bbbb384e9c44745702ab105", size = 10311656, upload-time = "2026-08-27T16:33:59.12Z" },+ { url = "https://files.pythonhosted.org/packages/a7/4d/c33a333e341c0a2b96c715b52d89a606f5a34cd4ac493cd9b8d0187186b8/ruff-0.16.5-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:0eeab41fbea2c42f98dfb9822cdccda9d24ba38d49f6dc945b5c236d48f0ef29", size = 10532125, upload-time = "2026-08-27T16:34:01.166Z" },+ { url = "https://files.pythonhosted.org/packages/30/e1/a64cef78b40192497bb98a27a8aa8f2c98ee9ee15bc97f7712d94ef32937/ruff-0.16.5-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:f0768e9df4300713fff30733c87575f68b6f1d8de41184e505b7fdd9c0c95eaf", size = 10097648, upload-time = "2026-08-27T16:34:03.16Z" },+ { url = "https://files.pythonhosted.org/packages/cc/4e/4cdc9ed3c3e109d2f71e62572a37457298d7bc7501ec3138babb7ed32bbd/ruff-0.16.5-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:95cc70cdc7aa80c338de356279d2adbeb2de0f520b9ecd8aba75b94e95e02f91", size = 9829344, upload-time = "2026-08-27T16:34:05.134Z" },+ { url = "https://files.pythonhosted.org/packages/39/4a/31ed35ce31729955fc583ee0d176d6e784c1290cb0b0a75cb2134c1ab72a/ruff-0.16.5-py3-none-musllinux_1_2_i686.whl", hash = "sha256:d185c8398ded1bfd91c0c2cb258346307571eccc473a8490af8c3977399c384a", size = 10277117, upload-time = "2026-08-27T16:34:07.425Z" },+ { url = "https://files.pythonhosted.org/packages/a8/a0/60356d86687b4b666d593df213f4dc3041750d024cb7bf2cfa81cfd65c2e/ruff-0.16.5-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:fb8e3a3c4c6a784150a7ced53b015f4b253fc2bf97a610886419ead64b4756ef", size = 10711653, upload-time = "2026-08-27T16:34:09.712Z" },+ { url = "https://files.pythonhosted.org/packages/ed/20/656d67f5b25ca9bda4e02b1de25867b2954e1d19e03648060f167ad0f4cc/ruff-0.16.5-py3-none-win32.whl", hash = "sha256:288b0a5f080492fe5635db849f9e2e84aa3cce7b7f0e955997d416c507c76a26", size = 10034250, upload-time = "2026-08-27T16:34:11.8Z" },+ { url = "https://files.pythonhosted.org/packages/5b/42/ee8e68a207b9127fcde6c3d7e197def432f346cb1af159e1fa14ca0d1cdc/ruff-0.16.5-py3-none-win_amd64.whl", hash = "sha256:ddc6385fb2137f616357ca03d6c74f4be987f80fed4008566b754f6032b8546f", size = 10516714, upload-time = "2026-08-27T16:34:13.963Z" },+ { url = "https://files.pythonhosted.org/packages/73/e3/7df5a396e445b9ba49ce9a9437439a4d80042c61c0ade199abf8d16de1ac/ruff-0.16.5-py3-none-win_arm64.whl", hash = "sha256:a64abe90968719b851bb7cedffaa8753fbdbdadab483089682db623f3edc587e", size = 10391564, upload-time = "2026-08-27T16:34:16.064Z" },+]