Skip to content

A self-improving RLM agent for coding workflows and long-running autonomous tasks.

Rust

[RSI] swarm-fanout capability eval harness (#2283)

Seth Karten6af0e1a4efdf
diff --git a/packages/coding-agent/docs/development.md b/packages/coding-agent/docs/development.md
index 56aa1a74a4..443834c371 100644
--- a/packages/coding-agent/docs/development.md
+++ b/packages/coding-agent/docs/development.md
@@ -81,3 +81,15 @@ npx tsx ../../node_modules/vitest/dist/cli.js --run test/specific.test.ts
```
If you create or modify a test file, run that file and iterate until it passes. Coding-agent suite regressions belong under `test/suite/regressions/` and use the suite harness and faux provider rather than live provider credentials.
+
+## Capability Evals
+
+End-to-end capability evals live under `scripts/evals/` and are not part of CI: a real-model run is a manual step with credentials in the environment. Each harness ships model-free self-tests that validate its fixtures and rubric without any model call - run them from the eval directory:
+
+```bash
+cd scripts/evals/swarm_fanout
+uv run --locked ruff check .
+uv run --locked python -m unittest discover -s tests -v
+```
+
+See `scripts/evals/README.md` for the rubric and the real-model run instructions.
diff --git a/scripts/evals/README.md b/scripts/evals/README.md
new file mode 100644
index 0000000000..4ed8d1270b
--- /dev/null
+++ b/scripts/evals/README.md
@@ -0,0 +1,93 @@
+# Prime Agent capability evals
+
+These harnesses measure end-to-end agent capability with a real model.
+They complement the repo's unit tests (which never call a model) and the
+PR performance benchmarks (which measure startup, not capability). They
+never run in CI and never need credentials to validate: every harness is
+exercised by model-free self-tests, and a real-model run is a manual
+step.
+
+## swarm-fanout
+
+Measures multi-agent orchestration under one prompt: decompose a job
+into per-shard child tasks, fan out one RLM subagent per shard, fan the
+answers back in, and account for child failures. Everything is scored
+from on-disk artifacts - the RLM spawn ledger, sub-* child session
+dirs, the parent session transcript, and the combined answer artifact -
+never from the agent's own claims.
+
+1. **Fixtures** (`swarm_fanout/fixtures/`) are two self-contained sets
+ (JSONL event logs, CSV host metrics) of eight stdlib-only data shards
+ each, with a machine-checkable question and expected answer per shard.
+ `task.txt` is the orchestration prompt; `fixture.json` is the
+ scoring manifest and stays out of the agent's repo copy.
+2. **Runner** (`swarm_fanout/runner.py`) copies a fixture to a temp git
+ repo, runs the parent agent headless (`--mode json`, `--daemon-socket`,
+ `--session-dir`, isolated agent home), then records the post-state.
+ The ledger path is computed exactly as the product does:
+ `<agentHome>/rlm-ledger/<sha256(canonical sessions dir)[:16]>.jsonl`.
+3. **Scorer** (`swarm_fanout/scorer.py`) applies the rubric:
+ - **coverage**: every shard's answer in `combined-index.md` matches
+ the machine-computed expected value;
+ - **delegation evidence**: a live depth-1 ledger edge for every
+ shard's worker name, each edge backed by its real child session dir
+ (the edge's recorded child file must exist in the collected sub-*
+ dirs) - helper-named spawns do not substitute, and a parent that
+ answers everything itself fails here;
+ - **dedup**: no shard's worker name is spawned twice while another
+ child for that shard is live, and total depth-1 spawns stay within
+ one retry per shard (2x the shard count);
+ - **receipts**: every depth-1 child has a parent-visible reply or an
+ explicit failure/terminal notice in the parent transcript;
+ - **efficiency**: tokens, turns, and wall time (informational until a
+ single-agent baseline datapoint exists).
+
+A run counts as resolved only when coverage, delegation, dedup, and
+receipts all pass.
+
+### Running a real-model eval
+
+The eval needs a model selector and provider credentials in the
+environment (the agent's normal auth):
+
+```
+cd scripts/evals/swarm_fanout
+uv run --locked python runner.py --fixture fixtures/json-events --model anthropic/claude-sonnet-4-5
+uv run --locked python runner.py --fixture fixtures/csv-metrics --model anthropic/claude-sonnet-4-5
+```
+
+The runner prints the scored outcome as JSON and exits 0 only when the
+run is resolved. `--timeout` (default 1800s) bounds the agent run.
+
+Real-model runs should be planned against one constraint: a one-shot
+`--mode json` session is client-owned, and when the CLI process exits
+the daemon stops its worker (after a 30s grace), taking still-running
+children with it. The task prompt therefore requires the parent to keep
+its run active until every child has replied. If a real-model datapoint
+shows models reliably ending their turn before the fan-in completes,
+drive the parent as a resident session over the daemon protocol
+(`create` + `prompt` + `wait_for_headless_completion` with
+`waitForRlmQuiescence: true`) instead and re-check; the scorer needs no
+changes either way, since it reads only the artifacts.
+
+### Self-tests
+
+Validate the harness with the model-free self-tests (fixture integrity
+with independently recomputed answers, every scorer rule plus the
+no-spawn, double-spawn, fabricated-answer, and silent-drop cheats,
+ledger-path computation, and a stub-agent end-to-end run) - never a
+model call:
+
+```
+uv run --locked ruff check .
+uv run --locked ruff format --check .
+uv run --locked python -m unittest discover -s tests -v
+```
+
+### Adding a fixture
+
+Copy an existing fixture directory, keep the shape: `shards/` (data
+files with machine-checkable answers), `fixture.json` (name, artifact,
+shards with file/worker/question/expected), and `task.txt` (orchestration
+prompt naming every shard and its question). Extend the fixture
+integrity tests with an independent recompute for the new questions.
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.json b/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.json
new file mode 100644
index 0000000000..888ddaf122
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/fixture.json
@@ -0,0 +1,54 @@
+{
+ "name": "csv-metrics",
+ "artifact": "combined-index.md",
+ "shards": [
+ {
+ "file": "metrics-01.csv",
+ "worker": "worker-metrics-01",
+ "question": "What is the maximum value of the cpu column?",
+ "expected": 85
+ },
+ {
+ "file": "metrics-02.csv",
+ "worker": "worker-metrics-02",
+ "question": "What is the total of the latency_ms column over rows with cpu >= 90?",
+ "expected": 37
+ },
+ {
+ "file": "metrics-03.csv",
+ "worker": "worker-metrics-03",
+ "question": "How many rows have memory > 80?",
+ "expected": 3
+ },
+ {
+ "file": "metrics-04.csv",
+ "worker": "worker-metrics-04",
+ "question": "What is the total of the latency_ms column over all rows?",
+ "expected": 287
+ },
+ {
+ "file": "metrics-05.csv",
+ "worker": "worker-metrics-05",
+ "question": "How many rows have cpu >= 90?",
+ "expected": 1
+ },
+ {
+ "file": "metrics-06.csv",
+ "worker": "worker-metrics-06",
+ "question": "What is the total of the memory column over all rows?",
+ "expected": 746
+ },
+ {
+ "file": "metrics-07.csv",
+ "worker": "worker-metrics-07",
+ "question": "What is the maximum memory among rows with latency_ms < 10?",
+ "expected": 72
+ },
+ {
+ "file": "metrics-08.csv",
+ "worker": "worker-metrics-08",
+ "question": "What is the total of the cpu column over rows with memory >= 50?",
+ "expected": 485
+ }
+ ]
+}
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csv
new file mode 100644
index 0000000000..8261191260
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-01.csv
@@ -0,0 +1,12 @@
+host,cpu,memory,latency_ms
+core-2,34,46,14
+db-1,51,69,21
+db-2,68,92,28
+api-1,85,25,35
+api-2,12,48,3
+gw-1,29,71,10
+gw-2,46,94,17
+edge-1,63,27,24
+edge-2,80,50,31
+core-1,7,73,38
+core-2,24,6,6
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csv
new file mode 100644
index 0000000000..0f6b23427c
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-02.csv
@@ -0,0 +1,13 @@
+host,cpu,memory,latency_ms
+api-1,63,87,27
+api-2,80,20,34
+gw-1,7,43,2
+gw-2,24,66,9
+edge-1,41,89,16
+edge-2,58,22,23
+core-1,75,45,30
+core-2,92,68,37
+db-1,19,91,5
+db-2,36,24,12
+api-1,53,47,19
+api-2,70,70,26
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csv
new file mode 100644
index 0000000000..66631ff9a8
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-03.csv
@@ -0,0 +1,14 @@
+host,cpu,memory,latency_ms
+gw-2,92,38,1
+edge-1,19,61,8
+edge-2,36,84,15
+core-1,53,17,22
+core-2,70,40,29
+db-1,87,63,36
+db-2,14,86,4
+api-1,31,19,11
+api-2,48,42,18
+gw-1,65,65,25
+gw-2,82,88,32
+edge-1,9,21,39
+edge-2,26,44,7
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csv
new file mode 100644
index 0000000000..80b5f84bc6
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-04.csv
@@ -0,0 +1,15 @@
+host,cpu,memory,latency_ms
+core-1,31,79,14
+core-2,48,12,21
+db-1,65,35,28
+db-2,82,58,35
+api-1,9,81,3
+api-2,26,14,10
+gw-1,43,37,17
+gw-2,60,60,24
+edge-1,77,83,31
+edge-2,94,16,38
+core-1,21,39,6
+core-2,38,62,13
+db-1,55,85,20
+db-2,72,18,27
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csv
new file mode 100644
index 0000000000..2d31325e3d
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-05.csv
@@ -0,0 +1,16 @@
+host,cpu,memory,latency_ms
+db-2,60,30,27
+api-1,77,53,34
+api-2,94,76,2
+gw-1,21,9,9
+gw-2,38,32,16
+edge-1,55,55,23
+edge-2,72,78,30
+core-1,89,11,37
+core-2,16,34,5
+db-1,33,57,12
+db-2,50,80,19
+api-1,67,13,26
+api-2,84,36,33
+gw-1,11,59,1
+gw-2,28,82,8
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csv
new file mode 100644
index 0000000000..8a6fd276f2
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-06.csv
@@ -0,0 +1,17 @@
+host,cpu,memory,latency_ms
+gw-1,89,71,1
+gw-2,16,94,8
+edge-1,33,27,15
+edge-2,50,50,22
+core-1,67,73,29
+core-2,84,6,36
+db-1,11,29,4
+db-2,28,52,11
+api-1,45,75,18
+api-2,62,8,25
+gw-1,79,31,32
+gw-2,6,54,39
+edge-1,23,77,7
+edge-2,40,10,14
+core-1,57,33,21
+core-2,74,56,28
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csv
new file mode 100644
index 0000000000..c0d0d2ea46
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-07.csv
@@ -0,0 +1,18 @@
+host,cpu,memory,latency_ms
+edge-2,28,22,14
+core-1,45,45,21
+core-2,62,68,28
+db-1,79,91,35
+db-2,6,24,3
+api-1,23,47,10
+api-2,40,70,17
+gw-1,57,93,24
+gw-2,74,26,31
+edge-1,91,49,38
+edge-2,18,72,6
+core-1,35,5,13
+core-2,52,28,20
+db-1,69,51,27
+db-2,86,74,34
+api-1,13,7,2
+api-2,30,30,9
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csv b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csv
new file mode 100644
index 0000000000..c6079b42f8
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/shards/metrics-08.csv
@@ -0,0 +1,19 @@
+host,cpu,memory,latency_ms
+db-1,57,63,27
+db-2,74,86,34
+api-1,91,19,2
+api-2,18,42,9
+gw-1,35,65,16
+gw-2,52,88,23
+edge-1,69,21,30
+edge-2,86,44,37
+core-1,13,67,5
+core-2,30,90,12
+db-1,47,23,19
+db-2,64,46,26
+api-1,81,69,33
+api-2,8,92,1
+gw-1,25,25,8
+gw-2,42,48,15
+edge-1,59,71,22
+edge-2,76,94,29
diff --git a/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txt b/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txt
new file mode 100644
index 0000000000..b6b9a59073
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/csv-metrics/task.txt
@@ -0,0 +1,38 @@
+You are the orchestrator of a data-analysis fan-out. This repository
+holds 8 read-only data shards under shards/. Answer each shard's
+question by delegating it to a dedicated RLM subagent - never by
+computing an answer yourself.
+
+Shard questions:
+- shards/metrics-01.csv: What is the maximum value of the cpu column?
+- shards/metrics-02.csv: What is the total of the latency_ms column over rows with cpu >= 90?
+- shards/metrics-03.csv: How many rows have memory > 80?
+- shards/metrics-04.csv: What is the total of the latency_ms column over all rows?
+- shards/metrics-05.csv: How many rows have cpu >= 90?
+- shards/metrics-06.csv: What is the total of the memory column over all rows?
+- shards/metrics-07.csv: What is the maximum memory among rows with latency_ms < 10?
+- shards/metrics-08.csv: What is the total of the cpu column over rows with memory >= 50?
+
+How to run the fan-out:
+
+1. Spawn exactly one RLM subagent per shard from the Python kernel
+ with await rlm.spawn('<task prompt>', name='worker-<shard stem>');
+ for shards/metrics-01.csv the child name is worker-metrics-01. The
+ child's task prompt must name the shard's file path and question
+ and require the child to answer with only the number.
+2. Each child must send its answer back to you with
+ await agent_message.send(answer, receiver_role='parent').
+3. Do not spawn two children for the same shard at once. If a child
+ fails, you may retry once: wait for the failure notice, delete the
+ failed child with await rlm.delete_subagent(child), then spawn a
+ replacement with the same worker name.
+4. Keep your current run active until every child has replied or
+ clearly failed: replies and failure notices are delivered into
+ your active run, and rlm.list_subagents() shows child status while
+ you wait. Do not end your run early.
+5. When every child has reported, write combined-index.md in the
+ repository root with exactly one bullet line per shard, in this
+ exact format:
+ - <shard-file-name>: <answer>
+ Copy each answer verbatim from the child's reply. The file must
+ cover all 8 shards.
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/fixture.json b/scripts/evals/swarm_fanout/fixtures/json-events/fixture.json
new file mode 100644
index 0000000000..72bf4d8c2b
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/fixture.json
@@ -0,0 +1,54 @@
+{
+ "name": "json-events",
+ "artifact": "combined-index.md",
+ "shards": [
+ {
+ "file": "events-01.jsonl",
+ "worker": "worker-events-01",
+ "question": "How many records have action \"deploy\"?",
+ "expected": 3
+ },
+ {
+ "file": "events-02.jsonl",
+ "worker": "worker-events-02",
+ "question": "What is the total of the bytes field over all records with action \"upload\"?",
+ "expected": 2044
+ },
+ {
+ "file": "events-03.jsonl",
+ "worker": "worker-events-03",
+ "question": "How many distinct users have at least one record with action \"login\"?",
+ "expected": 3
+ },
+ {
+ "file": "events-04.jsonl",
+ "worker": "worker-events-04",
+ "question": "What is the maximum value of the bytes field?",
+ "expected": 871
+ },
+ {
+ "file": "events-05.jsonl",
+ "worker": "worker-events-05",
+ "question": "How many records have action \"logout\"?",
+ "expected": 5
+ },
+ {
+ "file": "events-06.jsonl",
+ "worker": "worker-events-06",
+ "question": "How many records have bytes greater than 500?",
+ "expected": 5
+ },
+ {
+ "file": "events-07.jsonl",
+ "worker": "worker-events-07",
+ "question": "What is the total of the bytes field over all records with action \"deploy\"?",
+ "expected": 1912
+ },
+ {
+ "file": "events-08.jsonl",
+ "worker": "worker-events-08",
+ "question": "How many distinct users appear in the file?",
+ "expected": 3
+ }
+ ]
+}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonl
new file mode 100644
index 0000000000..6fea7b43db
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-01.jsonl
@@ -0,0 +1,13 @@
+{"ts": "2026-09-01T100:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
+{"ts": "2026-09-01T101:00:00Z", "user": "u4", "action": "deploy", "bytes": 278}
+{"ts": "2026-09-01T102:00:00Z", "user": "u6", "action": "upload", "bytes": 369}
+{"ts": "2026-09-01T103:00:00Z", "user": "u2", "action": "login", "bytes": 0}
+{"ts": "2026-09-01T104:00:00Z", "user": "u4", "action": "logout", "bytes": 0}
+{"ts": "2026-09-01T105:00:00Z", "user": "u6", "action": "deploy", "bytes": 642}
+{"ts": "2026-09-01T106:00:00Z", "user": "u2", "action": "upload", "bytes": 733}
+{"ts": "2026-09-01T107:00:00Z", "user": "u4", "action": "login", "bytes": 0}
+{"ts": "2026-09-01T108:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-01T109:00:00Z", "user": "u2", "action": "deploy", "bytes": 106}
+{"ts": "2026-09-01T110:00:00Z", "user": "u4", "action": "upload", "bytes": 197}
+{"ts": "2026-09-01T111:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-01T112:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonl
new file mode 100644
index 0000000000..83588d449d
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-02.jsonl
@@ -0,0 +1,14 @@
+{"ts": "2026-09-02T100:00:00Z", "user": "u3", "action": "deploy", "bytes": 324}
+{"ts": "2026-09-02T101:00:00Z", "user": "u5", "action": "upload", "bytes": 415}
+{"ts": "2026-09-02T102:00:00Z", "user": "u1", "action": "login", "bytes": 0}
+{"ts": "2026-09-02T103:00:00Z", "user": "u3", "action": "logout", "bytes": 0}
+{"ts": "2026-09-02T104:00:00Z", "user": "u5", "action": "deploy", "bytes": 688}
+{"ts": "2026-09-02T105:00:00Z", "user": "u1", "action": "upload", "bytes": 779}
+{"ts": "2026-09-02T106:00:00Z", "user": "u3", "action": "login", "bytes": 0}
+{"ts": "2026-09-02T107:00:00Z", "user": "u5", "action": "logout", "bytes": 0}
+{"ts": "2026-09-02T108:00:00Z", "user": "u1", "action": "deploy", "bytes": 152}
+{"ts": "2026-09-02T109:00:00Z", "user": "u3", "action": "upload", "bytes": 243}
+{"ts": "2026-09-02T110:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-02T111:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-02T112:00:00Z", "user": "u3", "action": "deploy", "bytes": 516}
+{"ts": "2026-09-02T113:00:00Z", "user": "u5", "action": "upload", "bytes": 607}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonl
new file mode 100644
index 0000000000..2621376603
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-03.jsonl
@@ -0,0 +1,15 @@
+{"ts": "2026-09-03T100:00:00Z", "user": "u4", "action": "upload", "bytes": 461}
+{"ts": "2026-09-03T101:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-03T102:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
+{"ts": "2026-09-03T103:00:00Z", "user": "u4", "action": "deploy", "bytes": 734}
+{"ts": "2026-09-03T104:00:00Z", "user": "u6", "action": "upload", "bytes": 825}
+{"ts": "2026-09-03T105:00:00Z", "user": "u2", "action": "login", "bytes": 0}
+{"ts": "2026-09-03T106:00:00Z", "user": "u4", "action": "logout", "bytes": 0}
+{"ts": "2026-09-03T107:00:00Z", "user": "u6", "action": "deploy", "bytes": 198}
+{"ts": "2026-09-03T108:00:00Z", "user": "u2", "action": "upload", "bytes": 289}
+{"ts": "2026-09-03T109:00:00Z", "user": "u4", "action": "login", "bytes": 0}
+{"ts": "2026-09-03T110:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-03T111:00:00Z", "user": "u2", "action": "deploy", "bytes": 562}
+{"ts": "2026-09-03T112:00:00Z", "user": "u4", "action": "upload", "bytes": 653}
+{"ts": "2026-09-03T113:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-03T114:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonl
new file mode 100644
index 0000000000..20ad634eaa
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-04.jsonl
@@ -0,0 +1,16 @@
+{"ts": "2026-09-04T100:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-04T101:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-04T102:00:00Z", "user": "u3", "action": "deploy", "bytes": 780}
+{"ts": "2026-09-04T103:00:00Z", "user": "u5", "action": "upload", "bytes": 871}
+{"ts": "2026-09-04T104:00:00Z", "user": "u1", "action": "login", "bytes": 0}
+{"ts": "2026-09-04T105:00:00Z", "user": "u3", "action": "logout", "bytes": 0}
+{"ts": "2026-09-04T106:00:00Z", "user": "u5", "action": "deploy", "bytes": 244}
+{"ts": "2026-09-04T107:00:00Z", "user": "u1", "action": "upload", "bytes": 335}
+{"ts": "2026-09-04T108:00:00Z", "user": "u3", "action": "login", "bytes": 0}
+{"ts": "2026-09-04T109:00:00Z", "user": "u5", "action": "logout", "bytes": 0}
+{"ts": "2026-09-04T110:00:00Z", "user": "u1", "action": "deploy", "bytes": 608}
+{"ts": "2026-09-04T111:00:00Z", "user": "u3", "action": "upload", "bytes": 699}
+{"ts": "2026-09-04T112:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-04T113:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-04T114:00:00Z", "user": "u3", "action": "deploy", "bytes": 72}
+{"ts": "2026-09-04T115:00:00Z", "user": "u5", "action": "upload", "bytes": 163}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonl
new file mode 100644
index 0000000000..5374d2063b
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-05.jsonl
@@ -0,0 +1,17 @@
+{"ts": "2026-09-05T100:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-05T101:00:00Z", "user": "u2", "action": "deploy", "bytes": 826}
+{"ts": "2026-09-05T102:00:00Z", "user": "u4", "action": "upload", "bytes": 917}
+{"ts": "2026-09-05T103:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-05T104:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
+{"ts": "2026-09-05T105:00:00Z", "user": "u4", "action": "deploy", "bytes": 290}
+{"ts": "2026-09-05T106:00:00Z", "user": "u6", "action": "upload", "bytes": 381}
+{"ts": "2026-09-05T107:00:00Z", "user": "u2", "action": "login", "bytes": 0}
+{"ts": "2026-09-05T108:00:00Z", "user": "u4", "action": "logout", "bytes": 0}
+{"ts": "2026-09-05T109:00:00Z", "user": "u6", "action": "deploy", "bytes": 654}
+{"ts": "2026-09-05T110:00:00Z", "user": "u2", "action": "upload", "bytes": 745}
+{"ts": "2026-09-05T111:00:00Z", "user": "u4", "action": "login", "bytes": 0}
+{"ts": "2026-09-05T112:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-05T113:00:00Z", "user": "u2", "action": "deploy", "bytes": 118}
+{"ts": "2026-09-05T114:00:00Z", "user": "u4", "action": "upload", "bytes": 209}
+{"ts": "2026-09-05T115:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-05T116:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonl
new file mode 100644
index 0000000000..42f321c352
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-06.jsonl
@@ -0,0 +1,18 @@
+{"ts": "2026-09-06T100:00:00Z", "user": "u1", "action": "deploy", "bytes": 872}
+{"ts": "2026-09-06T101:00:00Z", "user": "u3", "action": "upload", "bytes": 63}
+{"ts": "2026-09-06T102:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-06T103:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-06T104:00:00Z", "user": "u3", "action": "deploy", "bytes": 336}
+{"ts": "2026-09-06T105:00:00Z", "user": "u5", "action": "upload", "bytes": 427}
+{"ts": "2026-09-06T106:00:00Z", "user": "u1", "action": "login", "bytes": 0}
+{"ts": "2026-09-06T107:00:00Z", "user": "u3", "action": "logout", "bytes": 0}
+{"ts": "2026-09-06T108:00:00Z", "user": "u5", "action": "deploy", "bytes": 700}
+{"ts": "2026-09-06T109:00:00Z", "user": "u1", "action": "upload", "bytes": 791}
+{"ts": "2026-09-06T110:00:00Z", "user": "u3", "action": "login", "bytes": 0}
+{"ts": "2026-09-06T111:00:00Z", "user": "u5", "action": "logout", "bytes": 0}
+{"ts": "2026-09-06T112:00:00Z", "user": "u1", "action": "deploy", "bytes": 164}
+{"ts": "2026-09-06T113:00:00Z", "user": "u3", "action": "upload", "bytes": 255}
+{"ts": "2026-09-06T114:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-06T115:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-06T116:00:00Z", "user": "u3", "action": "deploy", "bytes": 528}
+{"ts": "2026-09-06T117:00:00Z", "user": "u5", "action": "upload", "bytes": 619}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonl
new file mode 100644
index 0000000000..1133b269ad
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-07.jsonl
@@ -0,0 +1,19 @@
+{"ts": "2026-09-07T100:00:00Z", "user": "u2", "action": "upload", "bytes": 109}
+{"ts": "2026-09-07T101:00:00Z", "user": "u4", "action": "login", "bytes": 0}
+{"ts": "2026-09-07T102:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-07T103:00:00Z", "user": "u2", "action": "deploy", "bytes": 382}
+{"ts": "2026-09-07T104:00:00Z", "user": "u4", "action": "upload", "bytes": 473}
+{"ts": "2026-09-07T105:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-07T106:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
+{"ts": "2026-09-07T107:00:00Z", "user": "u4", "action": "deploy", "bytes": 746}
+{"ts": "2026-09-07T108:00:00Z", "user": "u6", "action": "upload", "bytes": 837}
+{"ts": "2026-09-07T109:00:00Z", "user": "u2", "action": "login", "bytes": 0}
+{"ts": "2026-09-07T110:00:00Z", "user": "u4", "action": "logout", "bytes": 0}
+{"ts": "2026-09-07T111:00:00Z", "user": "u6", "action": "deploy", "bytes": 210}
+{"ts": "2026-09-07T112:00:00Z", "user": "u2", "action": "upload", "bytes": 301}
+{"ts": "2026-09-07T113:00:00Z", "user": "u4", "action": "login", "bytes": 0}
+{"ts": "2026-09-07T114:00:00Z", "user": "u6", "action": "logout", "bytes": 0}
+{"ts": "2026-09-07T115:00:00Z", "user": "u2", "action": "deploy", "bytes": 574}
+{"ts": "2026-09-07T116:00:00Z", "user": "u4", "action": "upload", "bytes": 665}
+{"ts": "2026-09-07T117:00:00Z", "user": "u6", "action": "login", "bytes": 0}
+{"ts": "2026-09-07T118:00:00Z", "user": "u2", "action": "logout", "bytes": 0}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonl b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonl
new file mode 100644
index 0000000000..cb9bedd8b4
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/shards/events-08.jsonl
@@ -0,0 +1,20 @@
+{"ts": "2026-09-08T100:00:00Z", "user": "u3", "action": "login", "bytes": 0}
+{"ts": "2026-09-08T101:00:00Z", "user": "u5", "action": "logout", "bytes": 0}
+{"ts": "2026-09-08T102:00:00Z", "user": "u1", "action": "deploy", "bytes": 428}
+{"ts": "2026-09-08T103:00:00Z", "user": "u3", "action": "upload", "bytes": 519}
+{"ts": "2026-09-08T104:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-08T105:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-08T106:00:00Z", "user": "u3", "action": "deploy", "bytes": 792}
+{"ts": "2026-09-08T107:00:00Z", "user": "u5", "action": "upload", "bytes": 883}
+{"ts": "2026-09-08T108:00:00Z", "user": "u1", "action": "login", "bytes": 0}
+{"ts": "2026-09-08T109:00:00Z", "user": "u3", "action": "logout", "bytes": 0}
+{"ts": "2026-09-08T110:00:00Z", "user": "u5", "action": "deploy", "bytes": 256}
+{"ts": "2026-09-08T111:00:00Z", "user": "u1", "action": "upload", "bytes": 347}
+{"ts": "2026-09-08T112:00:00Z", "user": "u3", "action": "login", "bytes": 0}
+{"ts": "2026-09-08T113:00:00Z", "user": "u5", "action": "logout", "bytes": 0}
+{"ts": "2026-09-08T114:00:00Z", "user": "u1", "action": "deploy", "bytes": 620}
+{"ts": "2026-09-08T115:00:00Z", "user": "u3", "action": "upload", "bytes": 711}
+{"ts": "2026-09-08T116:00:00Z", "user": "u5", "action": "login", "bytes": 0}
+{"ts": "2026-09-08T117:00:00Z", "user": "u1", "action": "logout", "bytes": 0}
+{"ts": "2026-09-08T118:00:00Z", "user": "u3", "action": "deploy", "bytes": 84}
+{"ts": "2026-09-08T119:00:00Z", "user": "u5", "action": "upload", "bytes": 175}
diff --git a/scripts/evals/swarm_fanout/fixtures/json-events/task.txt b/scripts/evals/swarm_fanout/fixtures/json-events/task.txt
new file mode 100644
index 0000000000..2eac761899
--- /dev/null
+++ b/scripts/evals/swarm_fanout/fixtures/json-events/task.txt
@@ -0,0 +1,38 @@
+You are the orchestrator of a data-analysis fan-out. This repository
+holds 8 read-only data shards under shards/. Answer each shard's
+question by delegating it to a dedicated RLM subagent - never by
+computing an answer yourself.
+
+Shard questions:
+- shards/events-01.jsonl: How many records have action "deploy"?
+- shards/events-02.jsonl: What is the total of the bytes field over all records with action "upload"?
+- shards/events-03.jsonl: How many distinct users have at least one record with action "login"?
+- shards/events-04.jsonl: What is the maximum value of the bytes field?
+- shards/events-05.jsonl: How many records have action "logout"?
+- shards/events-06.jsonl: How many records have bytes greater than 500?
+- shards/events-07.jsonl: What is the total of the bytes field over all records with action "deploy"?
+- shards/events-08.jsonl: How many distinct users appear in the file?
+
+How to run the fan-out:
+
+1. Spawn exactly one RLM subagent per shard from the Python kernel
+ with await rlm.spawn('<task prompt>', name='worker-<shard stem>');
+ for shards/events-01.jsonl the child name is worker-events-01. The
+ child's task prompt must name the shard's file path and question
+ and require the child to answer with only the number.
+2. Each child must send its answer back to you with
+ await agent_message.send(answer, receiver_role='parent').
+3. Do not spawn two children for the same shard at once. If a child
+ fails, you may retry once: wait for the failure notice, delete the
+ failed child with await rlm.delete_subagent(child), then spawn a
+ replacement with the same worker name.
+4. Keep your current run active until every child has replied or
+ clearly failed: replies and failure notices are delivered into
+ your active run, and rlm.list_subagents() shows child status while
+ you wait. Do not end your run early.
+5. When every child has reported, write combined-index.md in the
+ repository root with exactly one bullet line per shard, in this
+ exact format:
+ - <shard-file-name>: <answer>
+ Copy each answer verbatim from the child's reply. The file must
+ cover all 8 shards.
diff --git a/scripts/evals/swarm_fanout/pyproject.toml b/scripts/evals/swarm_fanout/pyproject.toml
new file mode 100644
index 0000000000..df6d1cf8aa
--- /dev/null
+++ b/scripts/evals/swarm_fanout/pyproject.toml
@@ -0,0 +1,19 @@
+[project]
+name = "prime-agent-evals"
+version = "0.1.0"
+requires-python = ">=3.11"
+dependencies = []
+
+[dependency-groups]
+dev = ["ruff==0.16.5"]
+
+[tool.uv]
+package = false
+exclude-newer = "2026-09-02T00:00:00Z"
+
+[tool.ruff]
+line-length = 110
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
diff --git a/scripts/evals/swarm_fanout/runner.py b/scripts/evals/swarm_fanout/runner.py
new file mode 100644
index 0000000000..8862a33489
--- /dev/null
+++ b/scripts/evals/swarm_fanout/runner.py
@@ -0,0 +1,414 @@
+"""Runner for the swarm-fanout eval.
+
+Copies a fixture repo to a temp dir, runs the parent agent headless
+against it, records the post-state (RLM spawn ledger, parent transcript,
+child session dirs, combined artifact), and prints the scored outcome
+as JSON.
+
+The parent is expected to fan out one RLM child per data shard and fan
+the answers back in. Real-model runs are manual: pass --model and ensure
+provider auth is configured in the environment. The harness itself is
+validated model-free by tests/test_swarm_fanout.py.
+
+Usage:
+ uv run --locked python runner.py --fixture fixtures/json-events --model anthropic/claude-sonnet-4-5
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import scorer # noqa: E402
+
+RLM_LEDGER_DIR = "rlm-ledger"
+
+
+def _captured_text(output: str | bytes | None) -> str:
+ """Decode captured subprocess output.
+
+ TimeoutExpired output arrives as bytes even with text=True.
+ """
+ if output is None:
+ return ""
+ if isinstance(output, bytes):
+ return output.decode(errors="replace")
+ return output
+
+
+def canonicalize_dir_path(dir_path: str) -> str:
+ """Mirror the product ledger's canonicalization.
+
+ rlmLedgerPath realpaths a sessions dir that exists and falls back to
+ plain resolve otherwise, so the runner must reproduce both branches
+ to land on the same ledger file the daemon writes.
+ """
+ resolved = os.path.abspath(dir_path)
+ if os.path.exists(resolved):
+ return os.path.realpath(resolved)
+ return resolved
+
+
+def ledger_file_candidates(agent_home: Path, sessions_dir: str) -> list[Path]:
+ """Ledger path candidates for a sessions dir, most likely first.
+
+ The daemon canonicalizes (realpaths) the sessions dir before hashing
+ it into the ledger file name; the plain-resolved variant is checked
+ second in case the directory did not exist when the daemon first
+ constructed its ledger.
+ """
+ ledger_dir = Path(agent_home) / RLM_LEDGER_DIR
+ candidates = []
+ for variant in (canonicalize_dir_path(sessions_dir), os.path.abspath(sessions_dir)):
+ digest = hashlib.sha256(variant.encode("utf-8")).hexdigest()[:16]
+ candidate = ledger_dir / f"{digest}.jsonl"
+ if candidate not in candidates:
+ candidates.append(candidate)
+ return candidates
+
+
+def find_ledger_file(agent_home: Path, sessions_dir: str) -> Path | None:
+ for candidate in ledger_file_candidates(agent_home, sessions_dir):
+ if candidate.is_file():
+ return candidate
+ return None
+
+
+def read_ledger_text(agent_home: Path, sessions_dir: str) -> str:
+ ledger_file = find_ledger_file(agent_home, sessions_dir)
+ if ledger_file is None:
+ return ""
+ try:
+ return ledger_file.read_text()
+ except OSError:
+ return ""
+
+
+def find_parent_session_file(sessions_dir: Path, ledger_text: str) -> Path | None:
+ """Locate the parent session file in the sessions dir.
+
+ The spawn records name the parent session file directly; when the
+ ledger is empty (a no-spawn run) the newest session file is the
+ parent - the eval launches exactly one client session.
+ """
+ session_files = sorted(sessions_dir.glob("*.jsonl"))
+ if not session_files:
+ return None
+ records, _malformed = scorer.parse_ledger(ledger_text)
+ parents = set()
+ for entry in records:
+ if entry.get("op") != "spawn" or not isinstance(entry.get("parent"), str):
+ continue
+ segments = scorer._path_segments(entry["parent"])
+ if segments:
+ parents.add(segments[-1])
+ matching = [path for path in session_files if path.name in parents]
+ if matching:
+ return matching[0]
+ return max(session_files, key=lambda path: path.stat().st_mtime)
+
+
+def collect_child_session_dirs(sessions_dir: Path, parent_session_file: Path | None) -> dict[str, list[str]]:
+ """sub-* child dirs under the parent's session artifact dir, by name.
+
+ Child sessions live next to their parent's artifacts
+ (session-artifacts/<parent-id>/sub-<uuid8>/), each holding the
+ child's session JSONL. The dir-to-files mapping lets the scorer
+ verify each ledger edge's recorded child file exists on disk, so a
+ dir unrelated to the ledger's children cannot stand in for one.
+ """
+ if parent_session_file is None:
+ return {}
+ artifacts_root = sessions_dir.parent / "session-artifacts"
+ parent_artifacts = artifacts_root / parent_session_file.stem
+ if not parent_artifacts.is_dir():
+ return {}
+ child_dirs: dict[str, list[str]] = {}
+ for child_dir in sorted(parent_artifacts.iterdir()):
+ if not child_dir.is_dir() or not child_dir.name.startswith("sub-"):
+ continue
+ session_files = sorted(path.name for path in child_dir.glob("*.jsonl"))
+ if session_files:
+ child_dirs[child_dir.name] = session_files
+ return child_dirs
+
+
+def read_artifact_text(repo_dir: Path, artifact: str) -> str:
+ try:
+ return (repo_dir / artifact).read_text()
+ except OSError:
+ return ""
+
+
+def first_agent_error(agent_log: str) -> str | None:
+ """The first error message the agent reported, for quick diagnosis.
+
+ A zero-token unresolved run almost always means a launch or auth
+ failure; surfacing the error in the result JSON saves the
+ workdir-digging this field was created for.
+ """
+ for line in agent_log.splitlines():
+ try:
+ entry = json.loads(line)
+ except ValueError:
+ continue
+ message = entry.get("message") if isinstance(entry, dict) else None
+ if isinstance(message, dict) and message.get("role") == "assistant":
+ error = message.get("errorMessage")
+ if isinstance(error, str) and error:
+ return error
+ return None
+
+
+def first_stderr_error(stderr: str) -> str | None:
+ """The first stderr line, where launch and auth failures surface."""
+ for line in stderr.splitlines():
+ stripped = line.strip()
+ if stripped:
+ return stripped
+ return None
+
+
+def shutdown_agent_daemon(socket_path: Path, retry_window_s: float = 0.0) -> None:
+ """Stop the daemon the agent run leaves listening on the eval socket.
+
+ The CLI spawns a detached daemon per --daemon-socket; without this,
+ repeated evals accumulate orphan daemons and a timed-out run keeps
+ its worker going. Client commands ride in a protocol envelope; the
+ shutdown command closes the daemon's sessions before it exits.
+
+ A timed-out launch can die before the detached daemon finished
+ binding its socket; the daemon then comes up orphaned and never sees
+ the shutdown. With a retry window the teardown waits for the socket
+ to appear and still delivers the shutdown.
+ """
+ envelope = json.dumps(
+ {
+ "type": "command",
+ "id": "eval-shutdown",
+ "protocol": {"name": "prime-agent.daemon", "version": 7},
+ "command": {"type": "shutdown"},
+ }
+ )
+ deadline = time.monotonic() + retry_window_s
+ while True:
+ try:
+ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client:
+ client.settimeout(10)
+ client.connect(str(socket_path))
+ client.sendall(envelope.encode() + b"\n")
+ while client.recv(4096):
+ pass
+ return
+ except OSError:
+ if time.monotonic() >= deadline:
+ # No daemon on the socket (launch failure or a stub
+ # agent): done.
+ return
+ time.sleep(0.25)
+
+
+def agent_env(agent_home: Path, sessions_dir: str) -> dict:
+ """A clean, isolated environment for the agent subprocess.
+
+ The eval agent must be an independent root session: every PRIME_AGENT_INTERNAL_*
+ variable an embedding session might leak is stripped (a child inheriting
+ them would try to attach to the parent's worker), the agent runs under
+ its own PRIME_AGENT_CODING_AGENT_DIR / PI_CODING_AGENT_DIR (the workspace
+ launcher's env prefix derives from the package config name) so it never
+ touches a production agent dir, and the credential file is copied in so
+ model auth works without sharing any state.
+
+ The session dir env override matters beyond the --session-dir flag: the
+ detached daemon is launched without that flag and keys its RLM spawn
+ ledger on the env override, so the scorer can compute the ledger path
+ deterministically.
+ """
+ env = {key: value for key, value in os.environ.items() if not key.startswith("PRIME_AGENT_INTERNAL_")}
+ for key in (
+ "PRIME_AGENT_BASH_SHELL",
+ "PRIME_AGENT_BASH_COMMAND_PREFIX",
+ # Depth overrides from an embedding RLM session would start the
+ # eval parent at the wrong depth or block spawning shard children.
+ "RLM_DEPTH",
+ "RLM_MAX_DEPTH",
+ ):
+ env.pop(key, None)
+ source_agent_dir = Path(env.get("PRIME_AGENT_CODING_AGENT_DIR") or Path.home() / ".prime" / "agent")
+ source_auth = source_agent_dir / "auth.json"
+ if source_auth.is_file():
+ agent_home.mkdir(parents=True, exist_ok=True)
+ shutil.copy2(source_auth, agent_home / "auth.json")
+ env["PRIME_AGENT_CODING_AGENT_DIR"] = str(agent_home)
+ env["PI_CODING_AGENT_DIR"] = str(agent_home)
+ env["PRIME_AGENT_SESSION_DIR"] = sessions_dir
+ env["PI_SESSION_DIR"] = sessions_dir
+ env["PRIME_AGENT_CODING_AGENT_SESSION_DIR"] = sessions_dir
+ env["PI_CODING_AGENT_SESSION_DIR"] = sessions_dir
+ return env
+
+
+def init_fixture_repo(fixture_dir: Path, repo_dir: Path) -> None:
+ """Copy the fixture into a fresh git repo.
+
+ The fixture manifest and task prompt stay out of the repo: the agent
+ works on the shards alone, and expected answers never reach it.
+ """
+ shutil.copytree(
+ fixture_dir,
+ repo_dir,
+ ignore=shutil.ignore_patterns("fixture.json", "task.txt"),
+ )
+ # Strip git location variables before initializing the temp repo: with
+ # GIT_DIR, GIT_INDEX_FILE, or GIT_WORK_TREE inherited from the caller,
+ # git add/commit would operate on the caller-selected repository instead
+ # of repo_dir. The environment is built fresh (not layered on
+ # os.environ) so a stripped variable cannot re-enter via the merge.
+ git_env = {
+ key: value
+ for key, value in os.environ.items()
+ if key
+ not in {
+ "GIT_DIR",
+ "GIT_INDEX_FILE",
+ "GIT_WORK_TREE",
+ "GIT_OBJECT_DIRECTORY",
+ "GIT_COMMON_DIR",
+ }
+ }
+ git_env.update(
+ {
+ "GIT_AUTHOR_NAME": "eval",
+ "GIT_AUTHOR_EMAIL": "eval@eval",
+ "GIT_COMMITTER_NAME": "eval",
+ "GIT_COMMITTER_EMAIL": "eval@eval",
+ }
+ )
+ for command in (
+ ["git", "init", "-q"],
+ ["git", "add", "-A"],
+ ["git", "-c", "commit.gpgsign=false", "commit", "-qm", "fixture"],
+ ):
+ subprocess.run(command, cwd=repo_dir, check=True, env=git_env)
+
+
+def collect_outcome(sessions_dir: Path, agent_home: Path, repo_dir: Path, agent_log: str) -> dict:
+ """Gather every artifact the rubric scores from the post-run state."""
+ ledger_text = read_ledger_text(agent_home, str(sessions_dir))
+ parent_session_file = find_parent_session_file(sessions_dir, ledger_text)
+ parent_transcript_text = ""
+ if parent_session_file is not None:
+ try:
+ parent_transcript_text = parent_session_file.read_text()
+ except OSError:
+ parent_transcript_text = ""
+ return {
+ "ledger_text": ledger_text,
+ "parent_transcript_text": parent_transcript_text,
+ "child_session_dirs": collect_child_session_dirs(sessions_dir, parent_session_file),
+ "artifact_text": read_artifact_text(repo_dir, "combined-index.md"),
+ "usage": scorer.summarize_usage(agent_log),
+ }
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--fixture", required=True, help="Fixture directory (contains fixture.json)")
+ parser.add_argument("--model", required=True, help="Model selector under test, provider/model-id")
+ parser.add_argument("--timeout", type=int, default=1800, help="Agent run timeout in seconds")
+ parser.add_argument("--agent-bin", default="prime-agent", help="Agent binary to invoke")
+ args = parser.parse_args(argv)
+
+ if sys.platform == "win32":
+ # The eval contract (per-workdir Unix daemon socket plus an
+ # AF_UNIX shutdown client) has no Windows named-pipe support;
+ # refuse loudly instead of failing obscurely mid-run.
+ print(
+ "swarm-fanout eval requires a Unix-domain-socket platform; Windows named pipes are not supported",
+ file=sys.stderr,
+ )
+ return 1
+
+ fixture_dir = Path(args.fixture).resolve()
+ fixture = json.loads((fixture_dir / "fixture.json").read_text())
+ task = (fixture_dir / "task.txt").read_text()
+
+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-"))
+ repo_dir = workdir / "repo"
+ sessions_dir = workdir / "sessions"
+ sessions_dir.mkdir()
+ sessions_dir = Path(canonicalize_dir_path(str(sessions_dir)))
+ agent_home = workdir / "agent-home"
+ init_fixture_repo(fixture_dir, repo_dir)
+
+ timed_out = False
+ started = time.monotonic()
+ try:
+ completed = subprocess.run(
+ [
+ args.agent_bin,
+ "--mode",
+ "json",
+ "--daemon-socket",
+ str(workdir / "daemon.sock"),
+ "--cwd",
+ str(repo_dir),
+ "--session-dir",
+ str(sessions_dir),
+ "--model",
+ args.model,
+ "--",
+ task,
+ ],
+ env=agent_env(agent_home, str(sessions_dir)),
+ capture_output=True,
+ text=True,
+ timeout=args.timeout,
+ )
+ agent_log = completed.stdout
+ stderr_text = completed.stderr
+ exit_code = completed.returncode
+ except subprocess.TimeoutExpired as exc:
+ # A timed-out run still scores; keep whatever transcript exists.
+ timed_out = True
+ agent_log = _captured_text(exc.stdout)
+ stderr_text = _captured_text(exc.stderr)
+ exit_code = None
+ except OSError as exc:
+ # A missing or non-executable agent binary still produces a result.
+ agent_log = ""
+ stderr_text = str(exc)
+ exit_code = None
+ finally:
+ wall_time_s = time.monotonic() - started
+ # A timed-out run may have died mid daemon-spawn: give the
+ # detached daemon time to bind before shutting it down.
+ shutdown_agent_daemon(workdir / "daemon.sock", retry_window_s=8.0 if timed_out else 0.0)
+ (workdir / "agent.log").write_text(agent_log)
+ # stderr is where launch and auth failures land; keep it with the result.
+ (workdir / "agent.stderr").write_text(stderr_text)
+
+ outcome = collect_outcome(sessions_dir, agent_home, repo_dir, agent_log)
+ outcome["wall_time_s"] = wall_time_s
+ result = scorer.score_fixture(fixture, outcome)
+ result["exit_code"] = exit_code
+ result["timed_out"] = timed_out
+ result["workdir"] = str(workdir)
+ result["model"] = args.model
+ result["agent_error"] = first_agent_error(agent_log) or first_stderr_error(stderr_text)
+ print(json.dumps(result, indent=2))
+ return 0 if result["resolved"] else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/evals/swarm_fanout/scorer.py b/scripts/evals/swarm_fanout/scorer.py
new file mode 100644
index 0000000000..96e2fba27e
--- /dev/null
+++ b/scripts/evals/swarm_fanout/scorer.py
@@ -0,0 +1,424 @@
+"""Pure scorer for the swarm-fanout eval.
+
+The scorer consumes a fixture manifest plus a recorded outcome dict and
+applies the orchestration rubric. It never runs the agent; the runner
+records the outcome and calls in, so every rule below is unit-testable
+without a model or network.
+
+Rubric (all four are required for a resolved run):
+ - coverage: every shard's answer appears in combined-index.md and
+ matches the machine-computed expected value from the fixture.
+ - delegation evidence: at least one live depth-1 spawn edge per shard,
+ each with a distinct childId and distinct name in the RLM ledger, and
+ at least one sub-* child session dir per shard holding a session
+ file. A parent that answers everything itself fails here.
+ - dedup: no shard's worker name is spawned twice while another child
+ for that shard is still live, and total depth-1 spawns stay within
+ one retry per shard (2x the shard count).
+ - receipt completeness: every depth-1 child (live or deleted) has a
+ parent-visible reply or an explicit child failure/terminal notice in
+ the parent transcript. A silently dropped child fails here.
+
+Efficiency (tokens, turns, wall time) is informational until a
+single-agent baseline datapoint exists.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+
+LEDGER_OPS = {"meta", "spawn", "rename", "delete"}
+NOTICE_CUSTOM_TYPES = {"rlm_child_terminal_notice", "rlm_child_failure"}
+REPLY_CUSTOM_TYPE = "agent_message"
+
+# One "- <shard>: <answer>" bullet per line; the shard name may not
+# contain a colon, the answer is everything after the first colon.
+ANSWER_LINE = re.compile(r"^\s*[-*]\s+(.+?):\s*(.*?)\s*$")
+
+
+def score_fixture(fixture: dict, outcome: dict) -> dict:
+ """Apply the swarm-fanout rubric to one recorded outcome."""
+ records, malformed = parse_ledger(outcome.get("ledger_text", ""))
+ edges = replay_edges(records)
+ answers = parse_answers(outcome.get("artifact_text", ""))
+ coverage = score_coverage(fixture, answers)
+ delegation = score_delegation(fixture, edges, outcome.get("child_session_dirs", {}))
+ dedup = score_dedup(fixture, records)
+ receipts = score_receipts(edges, outcome.get("parent_transcript_text", ""))
+ usage = outcome.get("usage") or {}
+ resolved = (
+ coverage["coverage"] and delegation["delegation_evidence"] and dedup["dedup"] and receipts["receipts"]
+ )
+ return {
+ "fixture": fixture.get("name"),
+ "resolved": resolved,
+ "coverage": coverage["coverage"],
+ "missing_shards": coverage["missing_shards"],
+ "wrong_answers": coverage["wrong_answers"],
+ "delegation_evidence": delegation["delegation_evidence"],
+ "live_depth1_edges": delegation["live_depth1_edges"],
+ "distinct_worker_names": delegation["distinct_worker_names"],
+ "child_session_dirs": delegation["child_session_dirs"],
+ "verified_shard_workers": delegation["verified_shard_workers"],
+ "dedup": dedup["dedup"],
+ "duplicate_spawns": dedup["duplicate_spawns"],
+ "total_spawns": dedup["total_spawns"],
+ "spawn_budget": dedup["spawn_budget"],
+ "receipts": receipts["receipts"],
+ "missing_receipts": receipts["missing_receipts"],
+ "ledger_malformed_lines": malformed,
+ "tokens_used": int(usage.get("tokens", 0)),
+ "turns": int(usage.get("turns", 0)),
+ "wall_time_s": float(outcome.get("wall_time_s", 0.0)),
+ }
+
+
+def parse_ledger(ledger_text: str) -> tuple[list[dict], int]:
+ """Parse ledger JSONL, skipping blank and malformed lines.
+
+ The product ledger fails closed on malformed lines; the scorer is a
+ passive reader of a possibly torn artifact, so malformed lines are
+ skipped and counted in the result for diagnosis. Each record's fields
+ are validated against the product schema (parseLedgerLine) before it
+ is replayed: a structurally invalid spawn line must not count as
+ delegation evidence. Records with a valid envelope but an unknown op
+ are skipped silently, the same forward-compat carve-out the product
+ reader makes.
+ """
+ records: list[dict] = []
+ malformed = 0
+ for line in ledger_text.splitlines():
+ stripped = line.strip()
+ if not stripped:
+ continue
+ try:
+ record = json.loads(stripped)
+ except ValueError:
+ malformed += 1
+ continue
+ if not isinstance(record, dict) or record.get("op") not in LEDGER_OPS:
+ if isinstance(record, dict) and record.get("v") == 1 and isinstance(record.get("at"), str):
+ continue # unknown op: forward-compat, not a defect
+ malformed += 1
+ continue
+ if not _valid_ledger_record(record):
+ malformed += 1
+ continue
+ records.append(record)
+ return records, malformed
+
+
+def _is_string(value: object) -> bool:
+ return isinstance(value, str)
+
+
+def _valid_ledger_record(record: dict) -> bool:
+ """Field-level validation, mirroring the product's parseLedgerLine."""
+ if record.get("v") != 1 or not _is_string(record.get("at")):
+ return False
+ op = record.get("op")
+ if op == "meta":
+ return _is_string(record.get("sessionsDir"))
+ if not (_is_string(record.get("childId")) and _is_string(record.get("child"))):
+ return False
+ if op == "spawn":
+ return (
+ _is_string(record.get("parent"))
+ and _is_string(record.get("name"))
+ and isinstance(record.get("depth"), int)
+ and not isinstance(record.get("depth"), bool)
+ )
+ if op == "rename":
+ return _is_string(record.get("name"))
+ return _is_string(record.get("reason"))
+
+
+def replay_edges(records: list[dict]) -> dict[tuple[str, str], dict]:
+ """Replay ledger records into edges, last-writer-wins per childId+child.
+
+ The product ledger keys an edge by (childId, canonical child session
+ path) - two records sharing a childId but naming different child
+ sessions are two distinct edges, and a rename or delete applies only
+ to the edge its own child path names. The scorer mirrors that key so
+ a shadowed child cannot slip out of delegation and receipt checks.
+ """
+ edges: dict[tuple[str, str], dict] = {}
+ for record in records:
+ op = record.get("op")
+ child_id = record.get("childId")
+ child = record.get("child")
+ if not isinstance(child_id, str) or not isinstance(child, str):
+ continue
+ key = (child_id, _canonical_session_path(child))
+ if op == "spawn":
+ edges[key] = {
+ "childId": child_id,
+ "parent": record.get("parent"),
+ "child": record.get("child"),
+ "depth": record.get("depth"),
+ "name": record.get("name"),
+ "deleted": None,
+ }
+ elif op == "rename":
+ edge = edges.get(key)
+ if edge is not None and isinstance(record.get("name"), str):
+ edge["name"] = record["name"]
+ elif op == "delete":
+ edge = edges.get(key)
+ if edge is not None:
+ edge["deleted"] = record.get("reason")
+ return edges
+
+
+def _canonical_session_path(session_path: str) -> str:
+ """Mirror the product's canonicalSessionPath for edge keys.
+
+ The path is real path'd when it exists, falling back to the parent's
+ real path joined with the file name, so a canonical and a plain
+ spelling of the same child session share one edge key.
+ """
+ resolved = os.path.abspath(session_path)
+ return os.path.realpath(resolved)
+
+
+def parse_answers(artifact_text: str) -> dict[str, list[str]]:
+ """Map shard file name to every answer given in the index bullets.
+
+ Duplicate bullets are kept, not overwritten: a shard answered twice
+ with conflicting values must not pass coverage just because one of
+ the two happens to be right. Coverage requires every occurrence to
+ match the expected answer.
+ """
+ answers: dict[str, list[str]] = {}
+ for line in artifact_text.splitlines():
+ match = ANSWER_LINE.match(line)
+ if match is None:
+ continue
+ shard, answer = match.group(1), match.group(2)
+ answers.setdefault(shard, []).append(answer)
+ return answers
+
+
+def score_coverage(fixture: dict, answers: dict[str, list[str]]) -> dict:
+ """Every shard's answer must match the machine-computed expected value.
+
+ A shard listed more than once is covered only when every occurrence
+ matches: conflicting duplicates are fabricated answers, not noise.
+ """
+ missing: list[str] = []
+ wrong: dict[str, dict] = {}
+ for shard in fixture.get("shards", []):
+ file = shard["file"]
+ expected = str(shard["expected"])
+ found = answers.get(file)
+ if not found:
+ missing.append(file)
+ elif any(value != expected for value in found):
+ wrong[file] = {"expected": expected, "found": found}
+ return {
+ "coverage": not missing and not wrong,
+ "missing_shards": missing,
+ "wrong_answers": wrong,
+ }
+
+
+def score_delegation(fixture: dict, edges: dict[str, dict], child_session_dirs: dict[str, list[str]]) -> dict:
+ """Spawn evidence must exist per shard: ledger edges plus child dirs.
+
+ Every shard's worker name must have a live depth-1 ledger edge, and
+ every such edge must be backed by its real child session dir: the
+ edge's recorded child file must exist in the collected sub-* dirs.
+ A parent that answers every shard without spawning one child per
+ shard fails delegation even though coverage may pass - the no-spawn
+ cheat is the blind-answer analog of swe-fix-loop's blind patch, and
+ helper-named children do not substitute for shard workers.
+ """
+ required_workers = {shard["worker"] for shard in fixture.get("shards", [])}
+ live_depth1 = [edge for edge in edges.values() if edge.get("depth") == 1 and edge.get("deleted") is None]
+ worker_names = {edge.get("name") for edge in live_depth1 if edge.get("name") in required_workers}
+ verified_workers = 0
+ claimed_sessions: set[tuple[str, str]] = set()
+ for worker in required_workers:
+ for edge in live_depth1:
+ if edge.get("name") != worker:
+ continue
+ session = _edge_session_dir_and_file(edge)
+ if session is None or session in claimed_sessions:
+ continue
+ if _session_dir_has_file(child_session_dirs, session):
+ claimed_sessions.add(session)
+ verified_workers += 1
+ break
+ passed = worker_names == required_workers and verified_workers == len(required_workers)
+ return {
+ "delegation_evidence": passed,
+ "live_depth1_edges": len({edge["childId"] for edge in live_depth1}),
+ "distinct_worker_names": len({edge.get("name") for edge in live_depth1 if edge.get("name")}),
+ "child_session_dirs": len(child_session_dirs),
+ "verified_shard_workers": verified_workers,
+ }
+
+
+def _path_segments(path: str) -> list[str]:
+ """Split a recorded path on both separators, platform-independent.
+
+ The product writes ledger paths with the host separator; on Windows
+ they are backslash-joined, so splitting on "/" alone would reject
+ every valid child session there.
+ """
+ return [segment for segment in path.replace("\\", "/").split("/") if segment]
+
+
+def _edge_session_dir_and_file(edge: dict) -> tuple[str, str] | None:
+ """The edge's recorded (child session dir, session file) pair."""
+ child_file = edge.get("child")
+ if not isinstance(child_file, str):
+ return None
+ segments = _path_segments(child_file)
+ if len(segments) < 2:
+ return None
+ return segments[-2], segments[-1]
+
+
+def _session_dir_has_file(child_session_dirs: dict[str, list[str]], session: tuple[str, str]) -> bool:
+ """True when the recorded session file exists in the collected dirs.
+
+ The ledger's child path points into the child's own sub-* dir; the
+ runner collects the dir-to-files mapping, so a fabricated edge with
+ no persisted session behind it cannot pass delegation.
+ """
+ child_dir, session_file = session
+ return session_file in child_session_dirs.get(child_dir, [])
+
+
+def score_dedup(fixture: dict, records: list[dict]) -> dict:
+ """No shard may be worked by two live children; one retry per shard.
+
+ The replay mirrors the daemon's ledger semantics: a spawn adds the
+ childId to its name's live set, a delete removes it. A spawn for a
+ shard's worker name while another child already holds that name is
+ duplicate work. Total depth-1 spawns may not exceed two per shard,
+ which is the retry budget the task prompt grants.
+ """
+ n_shards = len(fixture.get("shards", []))
+ shard_workers = {shard["worker"] for shard in fixture.get("shards", [])}
+ # A child's shard assignment is its name at spawn and survives renames:
+ # renaming the first worker-events-01 child away must not free the name
+ # for a second spawn while the first child is still live. Deletes are
+ # the only legitimate way a shard's worker name becomes reusable.
+ spawn_name: dict[str, str] = {}
+ live_depth1_ids: set[str] = set()
+ duplicates: list[dict] = []
+ total_spawns = 0
+ for record in records:
+ op = record.get("op")
+ child_id = record.get("childId")
+ if not isinstance(child_id, str):
+ continue
+ if op == "spawn" and record.get("depth") == 1:
+ total_spawns += 1
+ name = record.get("name")
+ spawn_name[child_id] = name
+ if name in shard_workers and any(spawn_name.get(live_id) == name for live_id in live_depth1_ids):
+ duplicates.append({"name": name, "childId": child_id})
+ live_depth1_ids.add(child_id)
+ elif op == "delete":
+ live_depth1_ids.discard(child_id)
+ budget = 2 * n_shards
+ return {
+ "dedup": not duplicates and total_spawns <= budget,
+ "duplicate_spawns": duplicates,
+ "total_spawns": total_spawns,
+ "spawn_budget": budget,
+ }
+
+
+def score_receipts(edges: dict[str, dict], parent_transcript_text: str) -> dict:
+ """Every depth-1 child must be accounted for in the parent transcript.
+
+ A receipt is a child reply (an agent_message custom record naming the
+ child's session id) or an explicit failure/terminal notice record
+ naming the childId. Deleted children are still checked: deleting a
+ silently failed child hides the drop, it does not receipt it.
+ """
+ replies: set[str] = set()
+ notices: set[str] = set()
+ for entry in _transcript_records(parent_transcript_text):
+ if entry.get("customType") == REPLY_CUSTOM_TYPE:
+ details = entry.get("details") or {}
+ # Only an explicit child reply receipts a child: the daemon
+ # always stamps fromRelationship on delivered messages, so a
+ # record missing it (or claiming another relationship) with a
+ # matching session id is not the child's reply.
+ if details.get("fromRelationship") != "child":
+ continue
+ sender = details.get("from") or {}
+ session_id = sender.get("sessionId")
+ if isinstance(session_id, str) and session_id:
+ replies.add(session_id)
+ elif entry.get("customType") in NOTICE_CUSTOM_TYPES:
+ details = entry.get("details") or {}
+ child_id = details.get("childId")
+ if isinstance(child_id, str) and child_id:
+ notices.add(child_id)
+ missing: list[dict] = []
+ for edge in edges.values():
+ if edge.get("depth") != 1:
+ continue
+ child_session_id = _session_id_from_file(edge.get("child"))
+ if child_session_id in replies or edge["childId"] in notices:
+ continue
+ missing.append({"childId": edge["childId"], "name": edge.get("name")})
+ return {"receipts": not missing, "missing_receipts": missing}
+
+
+def summarize_usage(session_text: str) -> dict:
+ """Sum assistant tokens and assistant turns from the session JSONL.
+
+ In --mode json each completed assistant message is emitted once on
+ message_end; turn_end repeats the same message, so only message_end
+ events are counted.
+ """
+ tokens = 0
+ turns = 0
+ for line in session_text.splitlines():
+ try:
+ entry = json.loads(line)
+ except ValueError:
+ continue
+ if not isinstance(entry, dict) or entry.get("type") != "message_end":
+ continue
+ message = entry.get("message")
+ if not isinstance(message, dict) or message.get("role") != "assistant":
+ continue
+ usage = message.get("usage") or {}
+ total = usage.get("totalTokens", 0)
+ if not isinstance(total, int) or total <= 0:
+ continue
+ tokens += total
+ turns += 1
+ return {"tokens": tokens, "turns": turns}
+
+
+def _transcript_records(transcript_text: str):
+ """Yield parsed custom_message records from a session JSONL."""
+ for line in transcript_text.splitlines():
+ try:
+ entry = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(entry, dict) and entry.get("type") == "custom_message":
+ yield entry
+
+
+def _session_id_from_file(session_file: object) -> str | None:
+ """The ledger child path is a <session-id>.jsonl file; extract the id."""
+ if not isinstance(session_file, str):
+ return None
+ segments = _path_segments(session_file)
+ if not segments:
+ return None
+ name = segments[-1]
+ return name[:-6] if name.endswith(".jsonl") else name
diff --git a/scripts/evals/swarm_fanout/tests/test_swarm_fanout.py b/scripts/evals/swarm_fanout/tests/test_swarm_fanout.py
new file mode 100644
index 0000000000..9967ba9b31
--- /dev/null
+++ b/scripts/evals/swarm_fanout/tests/test_swarm_fanout.py
@@ -0,0 +1,1084 @@
+"""Model-free self-tests for the swarm-fanout eval harness.
+
+Validates fixture integrity (expected answers recomputed independently
+from the shard data, task prompts naming every shard) and the scorer
+rubric with synthetic ledgers, transcripts, and artifacts - including
+the no-spawn, double-spawn, fabricated-answer, and silent-drop cheats.
+The runner is exercised end to end with a stub agent binary that writes
+a full fan-out artifact set. No agent, model, or network is invoked.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import hashlib
+import io
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import unittest
+import unittest.mock
+from pathlib import Path
+
+HARNESS = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(HARNESS))
+import runner # noqa: E402
+import scorer # noqa: E402
+
+FIXTURES = HARNESS / "fixtures"
+
+
+def fixture_manifest(name: str) -> dict:
+ return json.loads((FIXTURES / name / "fixture.json").read_text())
+
+
+def recompute_answer(fixture_name: str, shard_file: str, question: str) -> int:
+ """Recompute a shard's expected answer independently of the manifest."""
+ shard_path = FIXTURES / fixture_name / "shards" / shard_file
+ if fixture_name == "json-events":
+ records = [json.loads(line) for line in shard_path.read_text().splitlines() if line.strip()]
+ if question == 'How many records have action "deploy"?':
+ return sum(1 for record in records if record["action"] == "deploy")
+ if question == 'What is the total of the bytes field over all records with action "upload"?':
+ return sum(record["bytes"] for record in records if record["action"] == "upload")
+ if question == 'How many distinct users have at least one record with action "login"?':
+ return len({record["user"] for record in records if record["action"] == "login"})
+ if question == "What is the maximum value of the bytes field?":
+ return max(record["bytes"] for record in records)
+ if question == 'How many records have action "logout"?':
+ return sum(1 for record in records if record["action"] == "logout")
+ if question == "How many records have bytes greater than 500?":
+ return sum(1 for record in records if record["bytes"] > 500)
+ if question == 'What is the total of the bytes field over all records with action "deploy"?':
+ return sum(record["bytes"] for record in records if record["action"] == "deploy")
+ if question == "How many distinct users appear in the file?":
+ return len({record["user"] for record in records})
+ rows = [line.split(",") for line in shard_path.read_text().splitlines() if line.strip()]
+ header, data = rows[0], rows[1:]
+ columns = {name: index for index, name in enumerate(header)}
+ if question == "What is the maximum value of the cpu column?":
+ return max(int(row[columns["cpu"]]) for row in data)
+ if question == "What is the total of the latency_ms column over rows with cpu >= 90?":
+ return sum(int(row[columns["latency_ms"]]) for row in data if int(row[columns["cpu"]]) >= 90)
+ if question == "How many rows have memory > 80?":
+ return sum(1 for row in data if int(row[columns["memory"]]) > 80)
+ if question == "What is the total of the latency_ms column over all rows?":
+ return sum(int(row[columns["latency_ms"]]) for row in data)
+ if question == "How many rows have cpu >= 90?":
+ return sum(1 for row in data if int(row[columns["cpu"]]) >= 90)
+ if question == "What is the total of the memory column over all rows?":
+ return sum(int(row[columns["memory"]]) for row in data)
+ if question == "What is the maximum memory among rows with latency_ms < 10?":
+ return max(int(row[columns["memory"]]) for row in data if int(row[columns["latency_ms"]]) < 10)
+ if question == "What is the total of the cpu column over rows with memory >= 50?":
+ return sum(int(row[columns["cpu"]]) for row in data if int(row[columns["memory"]]) >= 50)
+ raise AssertionError(f"unknown question: {question}")
+
+
+# --- synthetic artifact builders -------------------------------------------
+
+
+def spawn_line(child_id: str, name: str, child_file: str, parent_file: str, depth: int = 1) -> str:
+ return json.dumps(
+ {
+ "v": 1,
+ "op": "spawn",
+ "at": "2026-09-12T00:00:00.000Z",
+ "childId": child_id,
+ "parent": parent_file,
+ "child": child_file,
+ "depth": depth,
+ "name": name,
+ }
+ )
+
+
+def delete_line(child_id: str, child_file: str, reason: str = "parent-teardown") -> str:
+ return json.dumps(
+ {
+ "v": 1,
+ "op": "delete",
+ "at": "2026-09-12T00:00:01.000Z",
+ "childId": child_id,
+ "child": child_file,
+ "reason": reason,
+ }
+ )
+
+
+def rename_line(child_id: str, child_file: str, name: str) -> str:
+ return json.dumps(
+ {
+ "v": 1,
+ "op": "rename",
+ "at": "2026-09-12T00:00:01.000Z",
+ "childId": child_id,
+ "child": child_file,
+ "name": name,
+ }
+ )
+
+
+def reply_line(child_session_id: str, session_name: str = "worker") -> str:
+ """An agent_message custom record delivered into the parent transcript."""
+ return json.dumps(
+ {
+ "type": "custom_message",
+ "customType": "agent_message",
+ "content": f"[agent-message from child:{session_name}]\n\n17",
+ "display": True,
+ "details": {
+ "id": "agentmsg-1",
+ "message": "17",
+ "from": {
+ "activeSessionId": "active-1",
+ "sessionId": child_session_id,
+ "sessionName": session_name,
+ "runtimeKind": "subagent",
+ },
+ "fromRelationship": "child",
+ "target": {"activeSessionId": "active-parent", "sessionId": "parent-id"},
+ },
+ "id": "entry-1",
+ "parentId": None,
+ "timestamp": "2026-09-12T00:00:02.000Z",
+ }
+ )
+
+
+def notice_line(child_id: str, session_name: str, kind: str = "completed_without_reply") -> str:
+ return json.dumps(
+ {
+ "type": "custom_message",
+ "customType": "rlm_child_terminal_notice",
+ "content": f"[child-exited: no-reply child:{session_name}]",
+ "display": True,
+ "details": {"kind": kind, "childId": child_id, "sessionName": session_name},
+ "id": "entry-2",
+ "parentId": "entry-1",
+ "timestamp": "2026-09-12T00:00:03.000Z",
+ }
+ )
+
+
+def failure_line(child_id: str, session_name: str) -> str:
+ return json.dumps(
+ {
+ "type": "custom_message",
+ "customType": "rlm_child_failure",
+ "content": f"[child-failed child:{session_name}]\n\nboom",
+ "display": True,
+ "details": {"childId": child_id, "sessionName": session_name, "error": "boom"},
+ "id": "entry-3",
+ "parentId": "entry-2",
+ "timestamp": "2026-09-12T00:00:04.000Z",
+ }
+ )
+
+
+def full_ledger(fixture: dict, parent_file: str, artifacts_root: str) -> tuple[str, list[dict]]:
+ """A well-formed per-shard fan-out ledger plus its child metadata."""
+ lines = [
+ json.dumps({"v": 1, "op": "meta", "at": "2026-09-12T00:00:00.000Z", "sessionsDir": "/tmp/sessions"})
+ ]
+ children = []
+ for shard in fixture["shards"]:
+ child_id = f"sub-{hashlib.sha256(shard['worker'].encode()).hexdigest()[:8]}"
+ child_session = f"child-{hashlib.sha256(shard['file'].encode()).hexdigest()[:12]}"
+ child_file = f"{artifacts_root}/parent-id/{child_id}/{child_session}.jsonl"
+ lines.append(spawn_line(child_id, shard["worker"], child_file, parent_file))
+ children.append(
+ {"shard": shard, "childId": child_id, "session": child_session, "child_file": child_file}
+ )
+ return "\n".join(lines) + "\n", children
+
+
+def full_transcript(children: list[dict]) -> str:
+ """A parent transcript in which every listed child replied with its answer."""
+ header = {"type": "session", "version": 3, "id": "parent-id", "timestamp": "2026-09-12T00:00:00.000Z"}
+ lines = [json.dumps(header)]
+ for child in children:
+ lines.append(reply_line(child["session"], child["shard"]["worker"]))
+ return "\n".join(lines) + "\n"
+
+
+def full_index(fixture: dict, wrong: dict | None = None) -> str:
+ wrong = wrong or {}
+ lines = []
+ for shard in fixture["shards"]:
+ answer = wrong.get(shard["file"], str(shard["expected"]))
+ lines.append(f"- {shard['file']}: {answer}")
+ return "\n".join(lines) + "\n"
+
+
+def ledger_spawn_records(ledger_text: str) -> list[dict]:
+ return [
+ json.loads(line)
+ for line in ledger_text.splitlines()
+ if line.strip() and json.loads(line).get("op") == "spawn"
+ ]
+
+
+def child_dir_mapping(children: list[dict]) -> dict[str, list[str]]:
+ """The dir-to-session-files mapping the runner collects post-run."""
+ return {child["childId"]: [child["session"] + ".jsonl"] for child in children}
+
+
+def passing_outcome(fixture: dict) -> dict:
+ ledger_text, children = full_ledger(fixture, "/tmp/sessions/parent-id.jsonl", "/tmp/artifacts")
+ return {
+ "ledger_text": ledger_text,
+ "parent_transcript_text": full_transcript(children),
+ "child_session_dirs": child_dir_mapping(children),
+ "artifact_text": full_index(fixture),
+ "usage": {"tokens": 40_000, "turns": 12},
+ "wall_time_s": 123.4,
+ }
+
+
+class FixtureIntegrity(unittest.TestCase):
+ def test_json_events_expected_answers_recompute(self):
+ manifest = fixture_manifest("json-events")
+ for shard in manifest["shards"]:
+ recomputed = recompute_answer("json-events", shard["file"], shard["question"])
+ self.assertEqual(recomputed, shard["expected"], shard["file"])
+
+ def test_csv_metrics_expected_answers_recompute(self):
+ manifest = fixture_manifest("csv-metrics")
+ for shard in manifest["shards"]:
+ recomputed = recompute_answer("csv-metrics", shard["file"], shard["question"])
+ self.assertEqual(recomputed, shard["expected"], shard["file"])
+
+ def test_task_txt_names_every_shard_and_question(self):
+ for name in ("json-events", "csv-metrics"):
+ manifest = fixture_manifest(name)
+ task = (FIXTURES / name / "task.txt").read_text()
+ for shard in manifest["shards"]:
+ self.assertIn(f"shards/{shard['file']}: {shard['question']}", task, name)
+
+ def test_task_txt_example_names_its_own_fixture(self):
+ # The worked example must reference this fixture's shards: the
+ # csv prompt naming events-* shards (or vice versa) would send a
+ # compliant model looking for files that do not exist.
+ for name, other_prefix in (("json-events", "metrics"), ("csv-metrics", "events-")):
+ manifest = fixture_manifest(name)
+ task = (FIXTURES / name / "task.txt").read_text()
+ first = manifest["shards"][0]
+ self.assertIn(f"for shards/{first['file']} the child name is {first['worker']}", task)
+ self.assertNotIn(f"{other_prefix}-01", task)
+
+ def test_worker_names_and_artifact_are_wellformed(self):
+ for name in ("json-events", "csv-metrics"):
+ manifest = fixture_manifest(name)
+ self.assertEqual(manifest["artifact"], "combined-index.md")
+ self.assertEqual(len(manifest["shards"]), 8)
+ workers = [shard["worker"] for shard in manifest["shards"]]
+ self.assertEqual(len(set(workers)), len(workers), name)
+ for shard in manifest["shards"]:
+ stem = shard["file"].rsplit(".", 1)[0]
+ self.assertEqual(shard["worker"], f"worker-{stem}")
+
+
+class ScorerTests(unittest.TestCase):
+ def setUp(self):
+ self.fixture = fixture_manifest("json-events")
+
+ def test_full_resolution(self):
+ result = scorer.score_fixture(self.fixture, passing_outcome(self.fixture))
+ self.assertTrue(result["resolved"])
+ self.assertTrue(result["coverage"])
+ self.assertTrue(result["delegation_evidence"])
+ self.assertTrue(result["dedup"])
+ self.assertTrue(result["receipts"])
+ self.assertEqual(result["tokens_used"], 40_000)
+ self.assertEqual(result["turns"], 12)
+ self.assertEqual(result["wall_time_s"], 123.4)
+
+ def test_no_spawn_blocks_resolution(self):
+ # The parent answered everything itself: no ledger, no children.
+ outcome = passing_outcome(self.fixture)
+ outcome["ledger_text"] = ""
+ outcome["parent_transcript_text"] = ""
+ outcome["child_session_dirs"] = {}
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertTrue(result["coverage"])
+ self.assertFalse(result["delegation_evidence"])
+ self.assertFalse(result["resolved"])
+
+ def test_fabricated_and_missing_answers_block_coverage(self):
+ outcome = passing_outcome(self.fixture)
+ wrong = {self.fixture["shards"][0]["file"]: "999999"}
+ outcome["artifact_text"] = full_index(self.fixture, wrong)
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["coverage"])
+ self.assertEqual(len(result["wrong_answers"]), 1)
+ # A dropped shard bullet is also a coverage failure.
+ partial = full_index(self.fixture)
+ outcome["artifact_text"] = "\n".join(partial.splitlines()[:-1]) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["coverage"])
+ self.assertEqual(len(result["missing_shards"]), 1)
+
+ def test_delegation_requires_child_session_dirs(self):
+ # Ledger edges exist but the child session dirs were never written.
+ outcome = passing_outcome(self.fixture)
+ outcome["child_session_dirs"] = {}
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertFalse(result["resolved"])
+
+ def test_delegation_requires_distinct_names(self):
+ outcome = passing_outcome(self.fixture)
+ # Rename every child to one name: eight live edges, one distinct
+ # name, so the fan-out degenerated to a pile of duplicate workers.
+ lines = outcome["ledger_text"].splitlines()
+ renames = [
+ rename_line(record["childId"], record["child"], "worker-same")
+ for record in ledger_spawn_records(outcome["ledger_text"])
+ ]
+ outcome["ledger_text"] = "\n".join(lines + renames) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertEqual(result["distinct_worker_names"], 1)
+
+ def test_delegation_requires_shard_worker_edges(self):
+ # Eight live depth-1 children named helper-N (none matching a
+ # shard's worker) with real session dirs still fail delegation:
+ # helper spawns cannot substitute for per-shard delegation.
+ outcome = passing_outcome(self.fixture)
+ records = [json.loads(line) for line in outcome["ledger_text"].splitlines()]
+ rewritten = []
+ for record in records:
+ if record.get("op") == "spawn":
+ record = {**record, "name": "helper-" + record["childId"][-4:]}
+ rewritten.append(json.dumps(record))
+ outcome["ledger_text"] = "\n".join(rewritten) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertEqual(result["verified_shard_workers"], 0)
+
+ def test_delegation_requires_edge_backed_session_dirs(self):
+ # The collected dirs exist but belong to sessions the ledger edges
+ # never recorded, so no worker edge is backed by a real child
+ # session dir and delegation fails.
+ outcome = passing_outcome(self.fixture)
+ outcome["child_session_dirs"] = {
+ f"sub-unrelated{index:02d}": [f"session-{index:02d}.jsonl"] for index in range(8)
+ }
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertEqual(result["verified_shard_workers"], 0)
+
+ def test_depth2_rename_does_not_flag_depth1_spawn(self):
+ # A grandchild renamed to a shard worker name must not shadow the
+ # parent's own depth-1 spawn of that worker: rename bookkeeping
+ # applies only to the fan-out's depth-1 children.
+ ledger = "\n".join(
+ [
+ spawn_line("sub-grand0000", "worker-tmp", "/tmp/g.jsonl", "/tmp/gp.jsonl", depth=2),
+ rename_line("sub-grand0000", "/tmp/g.jsonl", self.fixture["shards"][0]["worker"]),
+ spawn_line(
+ "sub-kid000000",
+ self.fixture["shards"][0]["worker"],
+ "/tmp/artifacts/parent-id/sub-kid000000/c000000000001.jsonl",
+ "/tmp/sessions/parent-id.jsonl",
+ ),
+ ]
+ )
+ records, _malformed = scorer.parse_ledger(ledger)
+ result = scorer.score_dedup(self.fixture, records)
+ self.assertTrue(result["dedup"])
+ self.assertEqual(result["total_spawns"], 1)
+
+ def test_parse_ledger_rejects_type_invalid_records(self):
+ # Records that fail the product ledger schema must not replay as
+ # delegation evidence: a spawn missing parent/child, or with a
+ # string depth, is malformed and skipped.
+ bad_spawn = json.dumps(
+ {
+ "v": 1,
+ "op": "spawn",
+ "at": "2026-09-12T00:00:00.000Z",
+ "childId": "sub-x",
+ "depth": 1,
+ "name": "worker-events-01",
+ }
+ )
+ bad_depth = json.dumps(
+ {
+ "v": 1,
+ "op": "spawn",
+ "at": "2026-09-12T00:00:00.000Z",
+ "childId": "sub-x",
+ "parent": "/tmp/p.jsonl",
+ "child": "/tmp/c.jsonl",
+ "depth": "1",
+ "name": "worker-events-01",
+ }
+ )
+ unknown_op = json.dumps({"v": 1, "op": "future", "at": "2026-09-12T00:00:00.000Z"})
+ good_spawn = spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl")
+ records, malformed = scorer.parse_ledger(
+ "\n".join([bad_spawn, bad_depth, unknown_op, good_spawn]) + "\n"
+ )
+ self.assertEqual(malformed, 2)
+ self.assertEqual(len(records), 1)
+ self.assertEqual(records[0]["childId"], "sub-1")
+
+ def test_score_fixture_tolerates_missing_child_dirs_key(self):
+ # An outcome without child session dirs must score (delegation
+ # false), not crash the dict-based rubric.
+ outcome = passing_outcome(self.fixture)
+ del outcome["child_session_dirs"]
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertFalse(result["resolved"])
+
+ def test_delegation_requires_distinct_session_claims(self):
+ # Two worker edges pointing at the same child session file is one
+ # child masquerading as a per-shard fan-out: each verified worker
+ # must claim its own session dir and file.
+ outcome = passing_outcome(self.fixture)
+ records = ledger_spawn_records(outcome["ledger_text"])
+ first = records[0]
+ shared_file = first["child"]
+ rewritten = []
+ for record in [json.loads(line) for line in outcome["ledger_text"].splitlines()]:
+ if record.get("op") == "spawn" and record["name"] != first["name"]:
+ record = {**record, "child": shared_file}
+ rewritten.append(json.dumps(record))
+ outcome["ledger_text"] = "\n".join(rewritten) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["delegation_evidence"])
+ self.assertEqual(result["verified_shard_workers"], 1)
+
+ def test_renamed_worker_does_not_free_name_for_duplicate(self):
+ # Renaming the first worker-events-01 child away must not free the
+ # name for a second spawn while the first child is still live: the
+ # shard assignment is the spawn name, not the current name.
+ outcome = passing_outcome(self.fixture)
+ lines = outcome["ledger_text"].splitlines()
+ first = ledger_spawn_records(outcome["ledger_text"])[0]
+ duplicate = spawn_line("sub-secondspawn", first["name"], first["child"] + "2", first["parent"])
+ outcome["ledger_text"] = (
+ "\n".join(
+ lines + [rename_line(first["childId"], first["child"], "worker-renamed-away"), duplicate]
+ )
+ + "\n"
+ )
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["dedup"])
+ self.assertEqual(len(result["duplicate_spawns"]), 1)
+ self.assertEqual(result["total_spawns"], 9)
+
+ def test_conflicting_duplicate_answers_block_coverage(self):
+ # Order-independent: a wrong bullet must fail coverage whether it
+ # lands before or after the correct one. (Last-wins scoring would
+ # pass this when the correction lands last.)
+ outcome = passing_outcome(self.fixture)
+ artifact = outcome["artifact_text"].rstrip()
+ first_shard = self.fixture["shards"][0]["file"]
+ expected = str(self.fixture["shards"][0]["expected"])
+ outcome["artifact_text"] = artifact + "\n" + f"- {first_shard}: 999999\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["coverage"])
+ # A correction after a wrong bullet still fails: the artifact
+ # carried a fabricated answer, and the scorer keeps every bullet.
+ outcome["artifact_text"] = (
+ artifact + "\n" + f"- {first_shard}: 999999\n" + f"- {first_shard}: {expected}\n"
+ )
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["coverage"])
+ # Identical duplicate bullets that all match stay covered.
+ outcome["artifact_text"] = artifact + "\n" + f"- {first_shard}: {expected}\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertTrue(result["coverage"])
+
+ def test_replay_edges_keyed_by_child_id_and_path(self):
+ # Two spawns sharing a childId but naming different child
+ # sessions are two distinct edges (the product keys an edge by
+ # childId plus canonical child path); a delete names its own
+ # edge and must not tombstone the sibling.
+ ledger = "\n".join(
+ [
+ spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl"),
+ spawn_line("sub-1", "worker-b", "/tmp/b.jsonl", "/tmp/p.jsonl"),
+ delete_line("sub-1", "/tmp/a.jsonl"),
+ ]
+ )
+ records, _malformed = scorer.parse_ledger(ledger)
+ edges = scorer.replay_edges(records)
+ self.assertEqual(len(edges), 2)
+ live = [edge for edge in edges.values() if edge["deleted"] is None]
+ self.assertEqual(len(live), 1)
+ self.assertEqual(live[0]["name"], "worker-b")
+
+ def test_edge_session_paths_use_platform_separators(self):
+ # The product writes ledger paths with the host separator; a
+ # Windows backslash-joined child path must verify delegation and
+ # receipts exactly like a POSIX one.
+ fixture = self.fixture
+ ledger_text, children = full_ledger(fixture, "\\tmp\\sessions\\parent-id.jsonl", "\\tmp\\artifacts")
+ backslash_children = []
+ lines = [json.loads(line) for line in ledger_text.splitlines()]
+ rewritten = []
+ for record in lines:
+ if record.get("op") == "spawn":
+ record = {**record, "child": record["child"].replace("/", "\\")}
+ backslash_children.append(record)
+ if record.get("op") == "meta":
+ record = {**record, "sessionsDir": record["sessionsDir"]}
+ rewritten.append(json.dumps(record))
+ outcome = {
+ "ledger_text": "\n".join(rewritten) + "\n",
+ "parent_transcript_text": full_transcript(children),
+ "child_session_dirs": child_dir_mapping(
+ [
+ {
+ "childId": child["childId"],
+ "session": child["session"],
+ }
+ for child in children
+ ]
+ ),
+ "artifact_text": full_index(fixture),
+ "usage": {"tokens": 1, "turns": 1},
+ "wall_time_s": 1.0,
+ }
+ result = scorer.score_fixture(fixture, outcome)
+ self.assertTrue(result["delegation_evidence"])
+ self.assertTrue(result["receipts"])
+ self.assertTrue(result["resolved"])
+
+ def test_receipts_require_explicit_child_relationship(self):
+ # A reply record with the relationship field missing is not a
+ # receipt: the daemon always stamps it on delivered messages.
+ outcome = passing_outcome(self.fixture)
+ outcome["parent_transcript_text"] = (
+ "\n".join(
+ line.replace('"fromRelationship": "child",', "")
+ for line in outcome["parent_transcript_text"].splitlines()
+ )
+ + "\n"
+ )
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["receipts"])
+
+ def test_double_spawn_blocks_dedup(self):
+ outcome = passing_outcome(self.fixture)
+ lines = outcome["ledger_text"].splitlines()
+ first = ledger_spawn_records(outcome["ledger_text"])[0]
+ duplicate = spawn_line("sub-duplicate00", first["name"], first["child"] + "2", first["parent"])
+ outcome["ledger_text"] = "\n".join(lines + [duplicate]) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["dedup"])
+ self.assertEqual(len(result["duplicate_spawns"]), 1)
+ self.assertEqual(result["total_spawns"], 9)
+
+ def test_retry_after_delete_stays_within_dedup(self):
+ # One child failed, was deleted, and a replacement with the same
+ # worker name was spawned in its own session dir; the replacement
+ # also replied. This is the retry the task prompt grants, and it
+ # resolves end to end.
+ outcome = passing_outcome(self.fixture)
+ lines = outcome["ledger_text"].splitlines()
+ first = ledger_spawn_records(outcome["ledger_text"])[0]
+ replacement_file = "/tmp/artifacts/parent-id/sub-retry00000/child-retry00000.jsonl"
+ replacement = spawn_line("sub-retry00000", first["name"], replacement_file, first["parent"])
+ outcome["ledger_text"] = (
+ "\n".join(lines + [delete_line(first["childId"], first["child"]), replacement]) + "\n"
+ )
+ outcome["child_session_dirs"]["sub-retry00000"] = ["child-retry00000.jsonl"]
+ transcript = outcome["parent_transcript_text"].rstrip()
+ outcome["parent_transcript_text"] = (
+ transcript + "\n" + reply_line("child-retry00000", first["name"]) + "\n"
+ )
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertTrue(result["dedup"])
+ self.assertTrue(result["receipts"])
+ self.assertTrue(result["delegation_evidence"])
+ self.assertEqual(result["total_spawns"], 9)
+ self.assertTrue(result["resolved"])
+
+ def test_spawn_budget_caps_total_spawns(self):
+ # Eight shard workers plus nine helper spawns: seventeen depth-1
+ # spawns exceed the one-retry-per-shard budget (2 x 8 = 16).
+ outcome = passing_outcome(self.fixture)
+ lines = outcome["ledger_text"].splitlines()
+ first = ledger_spawn_records(outcome["ledger_text"])[0]
+ helpers = []
+ for index in range(9):
+ helpers.append(
+ spawn_line(
+ f"sub-helper{index:05d}",
+ f"helper-{index}",
+ f"/tmp/artifacts/parent-id/sub-helper{index:05d}/child{index}.jsonl",
+ first["parent"],
+ )
+ )
+ outcome["ledger_text"] = "\n".join(lines + helpers) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertEqual(result["total_spawns"], 17)
+ self.assertFalse(result["dedup"])
+
+ def test_silent_drop_blocks_receipts(self):
+ outcome = passing_outcome(self.fixture)
+ records = ledger_spawn_records(outcome["ledger_text"])
+ children = [
+ {
+ "shard": {"worker": record["name"]},
+ "session": record["child"].rsplit("/", 1)[-1][:-6],
+ }
+ for record in records
+ ]
+ # Every child replied except the last, which neither replied nor
+ # left a notice: a dropped child must fail receipt completeness.
+ outcome["parent_transcript_text"] = full_transcript(children[:-1])
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["receipts"])
+ self.assertEqual(len(result["missing_receipts"]), 1)
+
+ def test_terminal_and_failure_notices_count_as_receipts(self):
+ outcome = passing_outcome(self.fixture)
+ records = ledger_spawn_records(outcome["ledger_text"])
+ lines = [outcome["parent_transcript_text"].rstrip()]
+ # Two children never replied; one left a completed-without-reply
+ # notice, the other a failure notice. Both are accounted for.
+ lines.append(notice_line(records[0]["childId"], records[0]["name"]))
+ lines.append(failure_line(records[1]["childId"], records[1]["name"]))
+ outcome["parent_transcript_text"] = "\n".join(lines) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertTrue(result["receipts"])
+
+ def test_deleted_child_without_notice_blocks_receipts(self):
+ # Deleting a child that never replied hides the drop: the delete
+ # record lands in the ledger, but no parent-visible receipt exists.
+ # A deleted child that did reply stays receipted, so the drop
+ # must be a child with neither reply nor notice.
+ outcome = passing_outcome(self.fixture)
+ records = ledger_spawn_records(outcome["ledger_text"])
+ dropped = records[-1]
+ kept = [
+ {"shard": {"worker": record["name"]}, "session": record["child"].rsplit("/", 1)[-1][:-6]}
+ for record in records[:-1]
+ ]
+ outcome["parent_transcript_text"] = full_transcript(kept)
+ lines = outcome["ledger_text"].splitlines()
+ outcome["ledger_text"] = "\n".join(lines + [delete_line(dropped["childId"], dropped["child"])]) + "\n"
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["receipts"])
+ self.assertEqual(len(result["missing_receipts"]), 1)
+ self.assertFalse(result["delegation_evidence"])
+
+ def test_receipt_requires_child_relationship(self):
+ outcome = passing_outcome(self.fixture)
+ # A sibling-labeled message carrying the child's session id is not
+ # a receipt: the child's reply never actually arrived.
+ outcome["parent_transcript_text"] = (
+ "\n".join(
+ line.replace('"fromRelationship": "child"', '"fromRelationship": "sibling"')
+ for line in outcome["parent_transcript_text"].splitlines()
+ )
+ + "\n"
+ )
+ result = scorer.score_fixture(self.fixture, outcome)
+ self.assertFalse(result["receipts"])
+
+ def test_replay_edges_last_writer_wins(self):
+ ledger = "\n".join(
+ [
+ spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl"),
+ spawn_line("sub-2", "worker-b", "/tmp/b.jsonl", "/tmp/p.jsonl"),
+ rename_line("sub-2", "/tmp/b.jsonl", "worker-c"),
+ delete_line("sub-1", "/tmp/a.jsonl"),
+ ]
+ )
+ records, malformed = scorer.parse_ledger(ledger)
+ self.assertEqual(malformed, 0)
+ edges = scorer.replay_edges(records)
+ self.assertEqual(len(edges), 2)
+ by_child = {edge["childId"]: edge for edge in edges.values()}
+ self.assertEqual(by_child["sub-2"]["name"], "worker-c")
+ self.assertIsNone(by_child["sub-2"]["deleted"])
+ self.assertEqual(by_child["sub-1"]["deleted"], "parent-teardown")
+
+ def test_parse_ledger_skips_malformed_lines(self):
+ ledger = "{not json}\n" + spawn_line("sub-1", "worker-a", "/tmp/a.jsonl", "/tmp/p.jsonl") + "\n\n"
+ records, malformed = scorer.parse_ledger(ledger)
+ self.assertEqual(malformed, 1)
+ self.assertEqual(len(records), 1)
+
+ def test_summarize_usage_counts_each_assistant_message_once(self):
+ message = {"role": "assistant", "usage": {"input": 60, "output": 40, "totalTokens": 100}}
+ session = (
+ json.dumps({"type": "message_end", "message": message})
+ + "\n"
+ + json.dumps({"type": "turn_end", "message": message, "toolResults": []})
+ )
+ self.assertEqual(scorer.summarize_usage(session), {"tokens": 100, "turns": 1})
+
+ def test_summarize_usage_ignores_malformed_lines(self):
+ session = "not json\n" + json.dumps({"type": "message_end"}) + "\n"
+ self.assertEqual(scorer.summarize_usage(session), {"tokens": 0, "turns": 0})
+
+
+# --- stub agent for end-to-end runner tests ---------------------------------
+#
+# The stub is a stand-in for the real CLI: it receives the same launch
+# arguments the runner would give prime-agent, reads the isolated agent
+# home from PRIME_AGENT_CODING_AGENT_DIR, and writes the exact artifact
+# set a correct fan-out leaves behind (ledger, parent transcript, child
+# session dirs, combined index, json-mode stdout events).
+
+STUB_BODY = """\
+#!/usr/bin/env python3
+# Stub agent for the swarm-fanout eval self-tests.
+import hashlib
+import json
+import os
+import sys
+from pathlib import Path
+
+ANSWERS = __ANSWERS__
+MODE = "__MODE__"
+PARENT_ID = "019aaaaa-aaaa-4aaa-8aaa-00000000000a"
+
+
+def assistant_event(tokens):
+ usage = {"role": "assistant", "usage": {"input": 10, "output": tokens - 10, "totalTokens": tokens}}
+ return json.dumps({"type": "message_end", "message": usage})
+
+
+argv = sys.argv[1:]
+# --mode json --daemon-socket SOCK --cwd REPO --session-dir SESSIONS --model M -- PROMPT
+repo = Path(argv[5])
+sessions_dir = Path(os.path.realpath(argv[7]))
+agent_home = Path(os.environ["PRIME_AGENT_CODING_AGENT_DIR"])
+parent_file = sessions_dir / (PARENT_ID + ".jsonl")
+
+index_lines = [f"- {shard_file}: {answer}" for shard_file, answer in ANSWERS.items()]
+(repo / "combined-index.md").write_text("\\n".join(index_lines) + "\\n")
+
+if MODE == "artifact-only":
+ # The no-spawn cheat: correct answers, zero delegation.
+ print(assistant_event(1000))
+ sys.exit(0)
+
+artifacts_root = sessions_dir.parent / "session-artifacts"
+ledger_lines = [
+ json.dumps({"v": 1, "op": "meta", "at": "2026-09-12T00:00:00.000Z", "sessionsDir": str(sessions_dir)})
+]
+session_header = {
+ "type": "session",
+ "version": 3,
+ "id": PARENT_ID,
+ "timestamp": "2026-09-12T00:00:00.000Z",
+ "cwd": str(repo),
+}
+transcript_lines = [json.dumps(session_header)]
+for shard_file, answer in ANSWERS.items():
+ stem = shard_file.rsplit(".", 1)[0]
+ child_id = "sub-" + hashlib.sha256(stem.encode()).hexdigest()[:8]
+ child_session = hashlib.sha256(("s" + shard_file).encode()).hexdigest()[:16]
+ child_dir = artifacts_root / PARENT_ID / child_id
+ child_dir.mkdir(parents=True, exist_ok=True)
+ child_file = child_dir / (child_session + ".jsonl")
+ child_header = {
+ "type": "session",
+ "version": 3,
+ "id": child_session,
+ "timestamp": "2026-09-12T00:00:00.000Z",
+ }
+ child_file.write_text(json.dumps(child_header) + "\\n")
+ ledger_lines.append(
+ json.dumps(
+ {
+ "v": 1,
+ "op": "spawn",
+ "at": "2026-09-12T00:00:00.500Z",
+ "childId": child_id,
+ "parent": str(parent_file),
+ "child": str(child_file),
+ "depth": 1,
+ "name": "worker-" + stem,
+ }
+ )
+ )
+ transcript_lines.append(
+ json.dumps(
+ {
+ "type": "custom_message",
+ "customType": "agent_message",
+ "content": "[agent-message from child:worker-" + stem + "]\\n\\n" + str(answer),
+ "display": True,
+ "details": {
+ "id": "agentmsg-" + child_session,
+ "message": str(answer),
+ "from": {
+ "activeSessionId": "active-" + child_session[:8],
+ "sessionId": child_session,
+ "sessionName": "worker-" + stem,
+ "runtimeKind": "subagent",
+ },
+ "fromRelationship": "child",
+ "target": {"activeSessionId": "active-parent", "sessionId": PARENT_ID},
+ },
+ "id": "entry-" + child_session[:8],
+ "parentId": None,
+ "timestamp": "2026-09-12T00:00:02.000Z",
+ }
+ )
+ )
+
+ledger_dir = agent_home / "rlm-ledger"
+ledger_dir.mkdir(parents=True, exist_ok=True)
+digest = hashlib.sha256(str(sessions_dir).encode("utf-8")).hexdigest()[:16]
+(ledger_dir / (digest + ".jsonl")).write_text("\\n".join(ledger_lines) + "\\n")
+parent_file.write_text("\\n".join(transcript_lines) + "\\n")
+print(assistant_event(1000))
+print(assistant_event(500))
+sys.exit(0)
+"""
+
+
+def write_stub_agent(fixture: dict, mode: str = "full") -> Path:
+ answers = {shard["file"]: shard["expected"] for shard in fixture["shards"]}
+ script_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-stub-"))
+ script = script_dir / "agent"
+ body = STUB_BODY.replace("__ANSWERS__", repr(answers)).replace("__MODE__", mode)
+ script.write_text(body)
+ script.chmod(0o755)
+ return script
+
+
+class LedgerPathTests(unittest.TestCase):
+ def test_ledger_hash_matches_product_computation(self):
+ # rlmLedgerPath hashes the canonical sessions dir into the ledger
+ # file name: sha256(canonical)[:16] under <agentDir>/rlm-ledger/.
+ agent_home = Path("/tmp/agent-home")
+ sessions_dir = "/tmp/sessions"
+ candidates = runner.ledger_file_candidates(agent_home, sessions_dir)
+ canonical = runner.canonicalize_dir_path(sessions_dir)
+ digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]
+ self.assertEqual(candidates[0], agent_home / "rlm-ledger" / f"{digest}.jsonl")
+
+ def test_canonicalize_realpaths_existing_dirs(self):
+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-canon-"))
+ try:
+ symlink = workdir / "linked"
+ real = workdir / "real"
+ real.mkdir()
+ symlink.symlink_to(real, target_is_directory=True)
+ self.assertEqual(runner.canonicalize_dir_path(str(symlink)), str(real.resolve()))
+ finally:
+ shutil.rmtree(workdir, ignore_errors=True)
+
+ def test_canonicalize_missing_dir_stays_resolved(self):
+ # rlmLedgerPath falls back to plain resolve when the dir does not
+ # exist yet, without resolving ancestor symlinks.
+ missing = str(Path(tempfile.gettempdir()) / "swarm-fanout-missing-dir")
+ self.assertEqual(runner.canonicalize_dir_path(missing), os.path.abspath(missing))
+
+ def test_ledger_candidates_cover_both_variants(self):
+ agent_home = Path(tempfile.mkdtemp(prefix="swarm-fanout-home-"))
+ try:
+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-cand-"))
+ try:
+ real = workdir / "real"
+ real.mkdir()
+ symlinked = workdir / "link"
+ symlinked.symlink_to(real, target_is_directory=True)
+ candidates = runner.ledger_file_candidates(agent_home, str(symlinked))
+ self.assertEqual(len(candidates), 2)
+ canonical = hashlib.sha256(str(real.resolve()).encode("utf-8")).hexdigest()[:16]
+ plain = hashlib.sha256(str(symlinked).encode("utf-8")).hexdigest()[:16]
+ self.assertEqual(candidates[0].name, f"{canonical}.jsonl")
+ self.assertEqual(candidates[1].name, f"{plain}.jsonl")
+ finally:
+ shutil.rmtree(workdir, ignore_errors=True)
+ finally:
+ shutil.rmtree(agent_home, ignore_errors=True)
+
+
+class RunnerTests(unittest.TestCase):
+ """The runner must emit a scored result in every launch failure mode."""
+
+ def run_runner(self, argv: list[str]) -> tuple[int, dict]:
+ captured = io.StringIO()
+ with contextlib.redirect_stdout(captured):
+ exit_code = runner.main(argv)
+ return exit_code, json.loads(captured.getvalue())
+
+ def test_stub_agent_full_fanout_resolves(self):
+ fixture = fixture_manifest("json-events")
+ stub = write_stub_agent(fixture, mode="full")
+ argv = ["--fixture", str(FIXTURES / "json-events"), "--model", "test/fake", "--agent-bin", str(stub)]
+ exit_code, result = self.run_runner(argv)
+ self.assertEqual(exit_code, 0)
+ self.assertTrue(result["resolved"])
+ self.assertTrue(result["coverage"])
+ self.assertTrue(result["delegation_evidence"])
+ self.assertEqual(result["child_session_dirs"], 8)
+ self.assertEqual(result["total_spawns"], 8)
+ self.assertTrue(result["dedup"])
+ self.assertTrue(result["receipts"])
+ self.assertEqual(result["tokens_used"], 1500)
+ self.assertEqual(result["turns"], 2)
+ self.assertFalse(result["timed_out"])
+
+ def test_stub_agent_without_spawns_scores_unresolved(self):
+ # The artifact-only stub answers everything without spawning: the
+ # no-spawn cheat must fail delegation evidence even though every
+ # answer is correct.
+ fixture = fixture_manifest("csv-metrics")
+ stub = write_stub_agent(fixture, mode="artifact-only")
+ argv = ["--fixture", str(FIXTURES / "csv-metrics"), "--model", "test/fake", "--agent-bin", str(stub)]
+ exit_code, result = self.run_runner(argv)
+ self.assertEqual(exit_code, 1)
+ self.assertTrue(result["coverage"])
+ self.assertFalse(result["delegation_evidence"])
+ self.assertFalse(result["resolved"])
+
+ def test_missing_agent_bin_scores_unresolved(self):
+ argv = [
+ "--fixture",
+ str(FIXTURES / "json-events"),
+ "--model",
+ "test/fake",
+ "--agent-bin",
+ "/nonexistent-agent",
+ ]
+ exit_code, result = self.run_runner(argv)
+ self.assertEqual(exit_code, 1)
+ self.assertFalse(result["resolved"])
+ self.assertTrue(result["agent_error"])
+
+ def test_agent_timeout_still_scores(self):
+ script_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-agent-"))
+ script = script_dir / "agent"
+ script.write_text("#!/bin/sh\nexec sleep 30\n")
+ script.chmod(0o755)
+ argv = [
+ "--fixture",
+ str(FIXTURES / "json-events"),
+ "--model",
+ "test/fake",
+ "--agent-bin",
+ str(script),
+ "--timeout",
+ "1",
+ ]
+ exit_code, result = self.run_runner(argv)
+ self.assertEqual(exit_code, 1)
+ self.assertFalse(result["resolved"])
+ self.assertTrue(result["timed_out"])
+ self.assertIsNone(result["exit_code"])
+
+ def test_init_fixture_repo_ignores_git_location_env(self):
+ # GIT_DIR / GIT_INDEX_FILE / GIT_WORK_TREE from the caller must not
+ # redirect the fixture init into another repository.
+ decoy = Path(tempfile.mkdtemp(prefix="swarm-fanout-gitenv-"))
+ subprocess.run(["git", "init", "-q", "-b", "main"], cwd=decoy, check=True)
+ (decoy / "marker.txt").write_text("decoy\n")
+ subprocess.run(["git", "add", "marker.txt"], cwd=decoy, check=True)
+ subprocess.run(["git", "-c", "commit.gpgsign=false", "commit", "-qm", "decoy"], cwd=decoy, check=True)
+ ref = decoy / ".git" / "refs" / "heads" / "main"
+ ref_before = ref.read_text()
+ fixture = fixture_manifest("json-events")
+ stub = write_stub_agent(fixture, mode="artifact-only")
+ argv = ["--fixture", str(FIXTURES / "json-events"), "--model", "test/fake", "--agent-bin", str(stub)]
+ git_env = {
+ "GIT_DIR": str(decoy / ".git"),
+ "GIT_WORK_TREE": str(decoy),
+ "GIT_INDEX_FILE": str(decoy / ".git" / "index"),
+ }
+ saved = {key: os.environ.get(key) for key in git_env}
+ try:
+ os.environ.update(git_env)
+ exit_code, result = self.run_runner(argv)
+ finally:
+ for key, value in saved.items():
+ if value is None:
+ os.environ.pop(key, None)
+ else:
+ os.environ[key] = value
+ self.assertEqual(exit_code, 1) # artifact-only stub: unresolved, but the repo must init
+ repo = Path(result["workdir"]) / "repo"
+ self.assertTrue((repo / ".git").is_dir(), "fixture repo was not initialized in the workdir")
+ self.assertTrue((repo / "shards" / "events-01.jsonl").is_file())
+ self.assertEqual(ref.read_text(), ref_before, "fixture commit leaked into the decoy repo")
+ self.assertFalse((decoy / "shards").exists())
+
+ def test_agent_env_strips_embedding_overrides(self):
+ # RLM depth overrides from an embedding session must not leak into
+ # the eval agent: the eval parent must be an independent root.
+ env = dict(os.environ)
+ env.update({"RLM_DEPTH": "2", "RLM_MAX_DEPTH": "2", "PRIME_AGENT_INTERNAL_TOKEN": "x"})
+ workdir = Path(tempfile.mkdtemp(prefix="swarm-fanout-env-"))
+ try:
+ with unittest.mock.patch.dict(os.environ, env, clear=True):
+ result = runner.agent_env(workdir / "agent-home", str(workdir / "sessions"))
+ self.assertNotIn("RLM_DEPTH", result)
+ self.assertNotIn("RLM_MAX_DEPTH", result)
+ self.assertNotIn("PRIME_AGENT_INTERNAL_TOKEN", result)
+ self.assertEqual(result["PRIME_AGENT_SESSION_DIR"], str(workdir / "sessions"))
+ self.assertEqual(result["PRIME_AGENT_CODING_AGENT_DIR"], str(workdir / "agent-home"))
+ finally:
+ shutil.rmtree(workdir, ignore_errors=True)
+
+ def test_shutdown_agent_daemon_waits_for_late_socket(self):
+ # A timed-out launch can die before the detached daemon binds; the
+ # retry window must still deliver the shutdown once it appears.
+ socket_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-late-"))
+ socket_path = socket_dir / "daemon.sock"
+ server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ received = []
+
+ def serve() -> None:
+ time.sleep(1.0)
+ server.bind(str(socket_path))
+ server.listen(1)
+ connection, _ = server.accept()
+ received.append(connection.recv(1024).decode())
+ connection.close()
+
+ thread = threading.Thread(target=serve)
+ thread.start()
+ runner.shutdown_agent_daemon(socket_path, retry_window_s=5.0)
+ thread.join(timeout=10)
+ server.close()
+ self.assertEqual(len(received), 1)
+ self.assertEqual(json.loads(received[0])["command"]["type"], "shutdown")
+
+ def test_shutdown_agent_daemon_sends_shutdown_command(self):
+ server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ socket_dir = Path(tempfile.mkdtemp(prefix="swarm-fanout-sock-"))
+ socket_path = socket_dir / "daemon.sock"
+ server.bind(str(socket_path))
+ server.listen(1)
+ received = []
+
+ def serve() -> None:
+ connection, _ = server.accept()
+ received.append(connection.recv(1024).decode())
+ connection.close()
+
+ thread = threading.Thread(target=serve)
+ thread.start()
+ runner.shutdown_agent_daemon(socket_path)
+ thread.join(timeout=5)
+ server.close()
+ self.assertEqual(len(received), 1)
+ envelope = json.loads(received[0])
+ self.assertEqual(envelope["type"], "command")
+ self.assertEqual(envelope["command"]["type"], "shutdown")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/scripts/evals/swarm_fanout/uv.lock b/scripts/evals/swarm_fanout/uv.lock
new file mode 100644
index 0000000000..905047fec0
--- /dev/null
+++ b/scripts/evals/swarm_fanout/uv.lock
@@ -0,0 +1,46 @@
+version = 1
+revision = 3
+requires-python = ">=3.11"
+
+[options]
+exclude-newer = "2026-09-02T00:00:00Z"
+
+[[package]]
+name = "prime-agent-evals"
+version = "0.1.0"
+source = { virtual = "." }
+
+[package.dev-dependencies]
+dev = [
+ { name = "ruff" },
+]
+
+[package.metadata]
+
+[package.metadata.requires-dev]
+dev = [{ name = "ruff", specifier = "==0.16.5" }]
+
+[[package]]
+name = "ruff"
+version = "0.16.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f3/85/c8e12473c93018f92d19dd988a294202e1c27426c47ec4de53ffb847b8d8/ruff-0.16.5.tar.gz", hash = "sha256:1b88500f9ffbcab3dedb0082c9f9492e91ec3d618aac1236a3e0189938f7040b", size = 4912003, upload-time = "2026-08-27T16:34:18.258Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c6/b6/77c90a970fe2dae17a723acbd011043ea97c98d7deacccefdc4ba74ec512/ruff-0.16.5-py3-none-linux_armv6l.whl", hash = "sha256:12e5f673e774c35fbb62f288809c7653b73445f8ecec6b6063fd6ea3521aa14b", size = 10011941, upload-time = "2026-08-27T16:33:41.287Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/46/6cf67cf6411885a1d6f7f6d801682f155536a85176d10b605e2ceffed8bd/ruff-0.16.5-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:eda58a5802de40e7ed5b32b64e0b32539338cc6fcd2c78f61e3ad6a0d79f51c3", size = 10204049, upload-time = "2026-08-27T16:33:44.056Z" },
+ { url = "https://files.pythonhosted.org/packages/46/fd/c8720ca7a090abf0c2fef4abe8a5ef6e5127ed15196d8886ff75a2b370e2/ruff-0.16.5-py3-none-macosx_11_0_arm64.whl", hash = "sha256:c5ae9a7b9a8875131f40f8fe967cc86abf899779efd663cb7ce3d572d01da7eb", size = 9809037, upload-time = "2026-08-27T16:33:46.257Z" },
+ { url = "https://files.pythonhosted.org/packages/43/45/a684caacdedaca180f52bacccc40bf0789d2c5a7c75f25324853e9eaedb5/ruff-0.16.5-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7b719b0a1f4d59710d283ab2965f621684a108a9e41da622e3b23f0326cd0025", size = 9964129, upload-time = "2026-08-27T16:33:48.352Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/f2/5d2bcdaca6b5b93d1b4dfc166cd2aebf7680143a1b38a28759df13a94d31/ruff-0.16.5-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2298f2780ed1be0c5cb1361e32ab7b1467f3cce7dabe101d2210a314f2fe42e9", size = 9821518, upload-time = "2026-08-27T16:33:50.57Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/ff/011cce29accf9257d5974145b733fc653a37985ed6825413a3987cefbfe0/ruff-0.16.5-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:258f29035a2dd021e7861e631b227a5b3f14e50c1184c9a6a122c5f4576154d7", size = 10534835, upload-time = "2026-08-27T16:33:52.522Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/5a/f0cf109bada9bba0e96c90c21c9f9251803f57225c32d293327a03c710d6/ruff-0.16.5-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b9a4f0432966834019c74d1b7e5c51224305d7713f3d7faf3e7451f1a3be3cde", size = 11252550, upload-time = "2026-08-27T16:33:54.521Z" },
+ { url = "https://files.pythonhosted.org/packages/63/4d/1d481aaea2046c6a7ed7c291f9004c669cce3c087b6b376ed5b08271e3fe/ruff-0.16.5-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b5eb3a8c3d0ade9cea42b591fd530368e8798380e30e0a308b85a5cf718f09ea", size = 10777949, upload-time = "2026-08-27T16:33:56.88Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/34/ee245ca55f64443233034b3d02b03236b19242004281247c079390b7facd/ruff-0.16.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ef0f69e191a13a3c9816f63163c88790cb12cd157bbbb384e9c44745702ab105", size = 10311656, upload-time = "2026-08-27T16:33:59.12Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/4d/c33a333e341c0a2b96c715b52d89a606f5a34cd4ac493cd9b8d0187186b8/ruff-0.16.5-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:0eeab41fbea2c42f98dfb9822cdccda9d24ba38d49f6dc945b5c236d48f0ef29", size = 10532125, upload-time = "2026-08-27T16:34:01.166Z" },
+ { url = "https://files.pythonhosted.org/packages/30/e1/a64cef78b40192497bb98a27a8aa8f2c98ee9ee15bc97f7712d94ef32937/ruff-0.16.5-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:f0768e9df4300713fff30733c87575f68b6f1d8de41184e505b7fdd9c0c95eaf", size = 10097648, upload-time = "2026-08-27T16:34:03.16Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/4e/4cdc9ed3c3e109d2f71e62572a37457298d7bc7501ec3138babb7ed32bbd/ruff-0.16.5-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:95cc70cdc7aa80c338de356279d2adbeb2de0f520b9ecd8aba75b94e95e02f91", size = 9829344, upload-time = "2026-08-27T16:34:05.134Z" },
+ { url = "https://files.pythonhosted.org/packages/39/4a/31ed35ce31729955fc583ee0d176d6e784c1290cb0b0a75cb2134c1ab72a/ruff-0.16.5-py3-none-musllinux_1_2_i686.whl", hash = "sha256:d185c8398ded1bfd91c0c2cb258346307571eccc473a8490af8c3977399c384a", size = 10277117, upload-time = "2026-08-27T16:34:07.425Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/a0/60356d86687b4b666d593df213f4dc3041750d024cb7bf2cfa81cfd65c2e/ruff-0.16.5-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:fb8e3a3c4c6a784150a7ced53b015f4b253fc2bf97a610886419ead64b4756ef", size = 10711653, upload-time = "2026-08-27T16:34:09.712Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/20/656d67f5b25ca9bda4e02b1de25867b2954e1d19e03648060f167ad0f4cc/ruff-0.16.5-py3-none-win32.whl", hash = "sha256:288b0a5f080492fe5635db849f9e2e84aa3cce7b7f0e955997d416c507c76a26", size = 10034250, upload-time = "2026-08-27T16:34:11.8Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/42/ee8e68a207b9127fcde6c3d7e197def432f346cb1af159e1fa14ca0d1cdc/ruff-0.16.5-py3-none-win_amd64.whl", hash = "sha256:ddc6385fb2137f616357ca03d6c74f4be987f80fed4008566b754f6032b8546f", size = 10516714, upload-time = "2026-08-27T16:34:13.963Z" },
+ { url = "https://files.pythonhosted.org/packages/73/e3/7df5a396e445b9ba49ce9a9437439a4d80042c61c0ade199abf8d16de1ac/ruff-0.16.5-py3-none-win_arm64.whl", hash = "sha256:a64abe90968719b851bb7cedffaa8753fbdbdadab483089682db623f3edc587e", size = 10391564, upload-time = "2026-08-27T16:34:16.064Z" },
+]