Skip to content

Experience-Driven Black-Box Cost Control for Coding Agents

Init

Weijia Xuac866f5c5f80
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
new file mode 100644
index 0000000..fbc93c0
--- /dev/null
+++ b/.claude-plugin/marketplace.json
@@ -0,0 +1,14 @@
+{
+ "name": "tokeneconomizer",
+ "owner": {
+ "name": "TokenEconomizer authors"
+ },
+ "description": "TokenEconomizer: runtime cost control for coding agents.",
+ "plugins": [
+ {
+ "name": "tokeneconomizer",
+ "source": "./plugin",
+ "description": "Learned per-repo facts at session start, approved unused-tool deny lists, and a spiral guard for failing loops."
+ }
+ ]
+}
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..256833d
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,39 @@
+# --- Python ---
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# Local virtualenvs. experiments/docs/ENVIRONMENT.md tells you to create `.venv` right here,
+# and CPython's venv only started writing a self-excluding .gitignore in 3.13 — so ignore it
+# explicitly rather than relying on the tool that happened to create it.
+.venv/
+venv/
+
+# --- Research artifacts (regenerated, never committed) ---
+# Run outputs: per-task transcripts/rows/grades, streams, summaries.
+results/
+artifacts/
+runs/
+*.jsonl
+
+# Rendered figures and other local exports are never tracked.
+experiments/figures/
+*.png
+*.pdf
+*.svg
+
+# Aggregated tables derived from results/.
+*.csv
+
+# --- Secrets / local env ---
+.secrets.env
+.env
+.env.*
+!.env.example
+*.key
diff --git a/README.md b/README.md
index bc1e65b..1fe2ad8 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,130 @@
-# Repository setup required :wave:
-
-Please visit the website URL :point_right: for this repository to complete the setup of this repository and configure access controls.
\ No newline at end of file
+# TokenEconomizer
+
+Runtime cost control for coding agents, from outside the agent. TokenEconomizer is a plugin
+for **Claude Code**, **Codex**, the **GitHub Copilot CLI** and **Pi**. It watches your sessions
+in each repository and, from one session to the next, targets three kinds of token waste:
+
+| Mechanism | What it does | Setting |
+|---|---|---|
+| **Learned facts** | Environment facts your sessions keep rediscovering the hard way (for example, that `tox` is missing here and `python -m pytest` works) are added to the next session's context. | `inject` |
+| **Unused-tool deny list** | Tools that stay loaded but go unused for several sessions are proposed as a deny list. Nothing changes until you apply it. | `untool` |
+| **Stuck-turn guard** | A turn that keeps repeating the same failing edit-and-test cycle, well past how long this repository's turns usually take, gets one mid-turn redirect. | `spiral` |
+
+Each mechanism can be turned off or tuned on its own ([Settings](#settings)). Every action is
+announced in one line, every hook fails open, and by default the plugin makes no model calls of
+its own. What it learns stays on your machine, under `~/.tokeneconomizer`.
+
+## Install
+
+You need Python 3.10 or newer on `PATH`. Clone the repository:
+
+```bash
+git clone https://github.com/microsoft/TokenEconomizer.git
+```
+
+Then install it into your agent (use the path of your clone):
+
+| Agent | Install |
+|---|---|
+| Claude Code | `claude plugin marketplace add /path/to/TokenEconomizer`<br>`claude plugin install tokeneconomizer@tokeneconomizer` |
+| Copilot CLI | `copilot plugin marketplace add /path/to/TokenEconomizer`<br>`copilot plugin install tokeneconomizer@tokeneconomizer` |
+| Codex | `python /path/to/TokenEconomizer/scripts/build_plugin.py --out ~/te-marketplace`<br>`codex plugin marketplace add ~/te-marketplace`<br>`codex plugin add tokeneconomizer@tokeneconomizer`<br>then approve its hooks once in `/hooks` inside Codex |
+| Pi | `pi install /path/to/TokenEconomizer/plugin` |
+
+Plugins load when a session starts, so start a new one. On Claude Code and the Copilot CLI, run
+`/tokeneconomizer:untool refresh-catalog` once in each repository (Copilot from a terminal:
+`~/.tokeneconomizer/bin/te --host copilot untool refresh-catalog`); without it the deny list
+cannot see the tools every session loads.
+
+**[`plugin/README.md`](plugin/README.md)** walks through the first run on each agent, updating
+and uninstalling, using Copilot with your own model key, and troubleshooting.
+
+## Use
+
+Work as usual; the plugin needs nothing from you. To see what it is doing, in Claude Code or
+the Copilot CLI's interactive mode:
+
+| Command | What it shows or does |
+|---|---|
+| `/tokeneconomizer:status` | what it learned in this repository, what it did, and what your sessions cost |
+| `/tokeneconomizer:facts` | the facts it injects at session start, and those still forming |
+| `/tokeneconomizer:untool` | the proposed deny list and its evidence; `… apply` accepts it, `… restore` undoes it |
+| `/tokeneconomizer:config` | the settings: what is on, and every value in effect |
+
+On Codex, run the same commands from a terminal inside the repository, as
+`~/.tokeneconomizer/bin/te --host codex <command>` (for example `… untool apply`). On Pi they
+are `/te <command>`.
+
+## Settings
+
+Turn a mechanism on or off, in this repository or, with `--global`, in all of them:
+
+```text
+/tokeneconomizer:config set spiral=off
+/tokeneconomizer:config set untool=off inject=on
+```
+
+Tune one by hand:
+
+```text
+/tokeneconomizer:config set inject_max_facts=2 # inject at most two facts
+/tokeneconomizer:config set untool_evidence=10 # wait ten unused sessions before proposing
+/tokeneconomizer:config set spiral_nudges=2 # allow two redirects per turn
+/tokeneconomizer:config set spiral.min_turns=30 # let a turn run 30 tool calls before it can count as stuck
+/tokeneconomizer:config unset spiral.min_turns # back to the default
+```
+
+`/tokeneconomizer:config` shows every value in effect. Every setting, its default, and how to
+set one for a single session: [Settings in `plugin/README.md`](plugin/README.md#settings).
+
+## What it does on each agent
+
+| | Learned facts | Unused-tool deny list | Stuck-turn guard |
+|---|---|---|---|
+| Claude Code | at session start | tools, in `.claude/settings.local.json` | redirect; can stop a headless turn |
+| Codex | at session start | capability packages, in the project's `.codex/config.toml` | redirect, then a notice |
+| Copilot CLI | at session start | whole MCP servers, in `.github/copilot/settings.local.json` | redirect, then a notice |
+| Pi | at session start | — (Pi loads four tools) | redirect; can stop a headless turn |
+
+Nothing is written to your settings until you approve it, and `untool restore` removes only
+what the plugin added. The design, and how each agent's hooks and settings were verified to
+behave, are in **[`docs/PLUGIN.md`](docs/PLUGIN.md)**.
+
+## Repository layout
+
+```
+plugin/ the installable plugin: manifests and hook files per agent, the Pi
+ extension, launchers, slash commands
+te_live/ the live harness the plugin runs: hook dispatch, per-repository state,
+ host adapters, the `te` command line
+tokeneconomizer/ the shared core: the three mechanisms, the trace reader, the cost ledger
+ (stdlib-only, no dependencies)
+tests/ offline tests for the plugin and the core
+docs/ PLUGIN.md (design and verified agent behaviour), METHOD.md (the method
+ and the core's seams)
+scripts/ build_plugin.py: a self-contained bundle for marketplace installs
+experiments/ the paper's reproduction code (see below)
+```
+
+## Development
+
+Python ≥ 3.10, no runtime dependencies. Run the plugin and core tests with
+`python -m pytest tests`, or `python -m pytest` to include the experiments' suite. Build an
+installable bundle with `python scripts/build_plugin.py --out dist/marketplace`.
+
+## Reproducing the paper
+
+The benchmark experiments behind the research — the Docker harness that runs sealed agents on
+SWE-bench and Web-Bench, the sweep and aggregation pipeline, the evaluated task sets, and the
+run book — live in **[`experiments/`](experiments/README.md)**. They are separate
+from the plugin and not packaged with it; they share only the core in `tokeneconomizer/`.
+
+The same research code in its original layout, without the plugin, is on the
+[`paper`](https://github.com/microsoft/TokenEconomizer/tree/paper) branch.
+
+## Limitations
+TokenEconomizer was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios.
+
+There has not been a systematic effort to ensure that systems using TokenEconomizer are protected from security vulnerabilities such as indirect prompt injection attacks. Content in a repository, issue text, or tool output could influence a synthesized micro-skill that persists across later tasks. Any systems using it should take proactive measures to harden their systems as appropriate.
+
+Downstream developers remain responsible for ensuring that the underlying coding agent, its tool permissions, execution sandbox, generated code, and deployment environment are appropriate to their use case.
diff --git a/docs/METHOD.md b/docs/METHOD.md
new file mode 100644
index 0000000..45ccb69
--- /dev/null
+++ b/docs/METHOD.md
@@ -0,0 +1,80 @@
+# Method — thesis, amortized setting, and the seams
+
+## Thesis
+
+Hold the **sealed agent fixed** (Claude Code, Copilot CLI, Codex) and make the
+**environment learn a cheaper interface to itself**, so the agent solves tasks at lower
+token cost with **no regression in task success**. The optimization is delivered
+**externally** (no agent changes) and must **transfer across agents**.
+
+The cost we optimize is amortized over a **stream** of tasks on one repository, not any
+single task. Some mechanism cost is paid once (a between-task proposer tick) and reused by
+every later task, so the metric is the **amortized accuracy–cost curve** over the stream,
+not per-task overhead in isolation.
+
+## Design constraints (learned empirically)
+
+- **Optimize turns, not bytes.** Cost is driven by round-trips × re-read context, not
+ per-command output size — so compressing command output alone is not enough.
+- **Displacement is hard.** Sealed agents under-adopt nudged tools; prefer environment-side
+ levers the agent already uses over ones it must opt into.
+- **Cache-aware cost or nothing.** Most raw tokens in an agent session are cached re-reads,
+ so comparing on `tokens_total` flatters methods that prune cheap re-reads. Everything is
+ measured on `pricing.effective_tokens + ledger.overhead`.
+- **Resolve is a hard guardrail.** A cheaper arm that loses task success is not shipped. The
+ spiral finalize/abort rules are gated to be resolve-safe by construction.
+
+## The three mechanisms
+
+Each acts at a different point in a task's lifecycle and is independently switchable:
+
+| Mechanism | Lifecycle point | What it does |
+|---|---|---|
+| **injection** | before the task | Inject a firing-gated, learned per-repo operating manual (canonical test command, "a repro is redundant", required env var, …). Two layers share one lifecycle: an LLM-free environment-fact layer (`envfacts`: recurring failure→repair shapes, refused hosts, empty probes; vocabulary from the domain adapter) and a paid proposer that authors open-vocabulary facts + extractors over a compact digest. Profiles `distiller` (Claude/Copilot default), `envfacts` (Codex default, no paid call) and `envfacts+distiller` (both; the proposer is told what is already injected). A validated lifecycle (formation / decay / firing gate) keeps only facts that actually fire. |
+| **tooling** | at task construction | Drop inapplicable tool schemas. Generic: evidence untooling (drop a tool loaded-but-unused for N tasks) + an optional per-task instruction-aware judge. Codex-native: remove whole inapplicable capability packages from the request contract, evidence-gated and resolve-safe. |
+| **spiral** | in flight / at finalize | Kill or nudge a stuck rollout against a learned per-repo resolved-turn envelope. Generic: cheap gate → sampled LLM judge → nudge-then-abort. Codex-native: one same-thread finalize nudge once a trustworthy green passes and every failed scope is cleared. |
+
+## The two seams that keep the core self-contained
+
+The mechanisms are harness- and domain-agnostic. Everything environment-specific enters
+through two small, named surfaces:
+
+### 1. Trajectory + cost — vendored, stdlib-only
+
+- `tokeneconomizer.trace` — the ONE place a saved rollout transcript is read
+ (`load_trajectory` / `tool_usage` / `detect` / `parse_catalog`); auto-detects the
+ supported agent transcript schemas.
+- `tokeneconomizer.ledger` — the ONE definition of amortized cost, over the vendored
+ `pricing` table. Mechanisms write per-task overhead fields; aggregation reads them back
+ through the same module, so emitted and reported ledgers cannot drift.
+- `tokeneconomizer.vendor` — `pricing` / `reflect` / `spiral`, copied in verbatim
+ (stdlib-only), so `import tokeneconomizer` needs no external benchmark tree.
+
+### 2. Execution + domain — behind interfaces
+
+- `tokeneconomizer.runner.AgentRunner` — the seam to the task-execution harness. The
+ economizer never launches Docker, calls an LLM API, or shells out to an agent CLI itself;
+ it needs only (a) a paid **optimizer call** for the proposer/judge/nudge, and (b) an
+ optional **task executor**. A concrete Docker + CLI + gold-grading runner is an external
+ plugin (see [`experiments/`](../experiments/README.md)), with image/dataset paths as
+ placeholders.
+- `tokeneconomizer.adapters.DomainAdapter` — the seam to the task domain. The mechanisms
+ read the domain only through this (trajectory reading, waste signals, untool tool-sets,
+ spiral snapshot, per-task issue text). A new benchmark = a new adapter, no mechanism
+ edits. Ships with SWE-bench (canonical), Web-Bench, CredData, ReposVul, CyberGym, and a
+ WebArena stub; dataset roots are env-var placeholders.
+
+## Entry point
+
+```python
+from tokeneconomizer import Economizer, EconomizerConfig
+
+econ = Economizer.from_args(args, repo=repo, results_root=root, fresh=True,
+ runner=my_runner) # my_runner: AgentRunner or optimizer callable
+# per task:
+injected = econ.injection_prefix(task)
+exclude_tools = econ.build_tools(task, i)
+run_kwargs = econ.run_kwargs(task)
+# ... execute the task via the runner with those decisions ...
+opt_tokens, reason = econ.after_task(task, row, run_dir_path, srow)
+```
diff --git a/docs/PLUGIN.md b/docs/PLUGIN.md
new file mode 100644
index 0000000..376dc79
--- /dev/null
+++ b/docs/PLUGIN.md
@@ -0,0 +1,414 @@
+# The plugin — the three mechanisms in your own agent sessions
+
+The experiments in this repository hold a sealed agent fixed and run it, headless, over streams
+of benchmark tasks. This document is about the other way to use the same controller: as a
+**plugin** that runs inside the sessions you already have — Claude Code, Codex, the GitHub
+Copilot CLI and Pi — through the hooks (or, on Pi, the extension events) the agent exposes.
+
+It covers what the plugin does at each hook, what changes relative to the experimental setting
+and why, the agent behaviour each design choice rests on, how to install and configure it, and
+what is not done yet. The code is `te_live/` (the live harness: the session-oriented controller,
+a sibling of `experiments/te_harness/`) and `plugin/` (the host packaging).
+
+> **The plugin ships the mechanisms, not results.** The experiments ran on sealed, headless,
+> graded SWE-bench / Web-Bench streams; a real session is interactive, has no grader, and
+> differs in the ways listed in §2. To see what the plugin would do on your own history before
+> turning anything on, run `te replay` (§4).
+
+---
+
+## 1. What it does, hook by hook
+
+| Hook | What the plugin does | Mechanism it drives |
+|---|---|---|
+| `SessionStart` | Injects the repository's learned manual (a cached render) as `additionalContext`; announces a changed manual or a new untool proposal in one `systemMessage` line; folds any earlier session whose end was never seen | injection |
+| `UserPromptSubmit` | Opens a prompt-turn for the spiral guard | spiral |
+| `PostToolUse` | Folds the call into the turn's guard; delivers a nudge as `additionalContext`, or stops the turn | spiral |
+| `Stop` | Closes the turn; a turn that *ended green* feeds the per-repository envelope | spiral |
+| `SessionEnd` | Hands the session to a detached `te fold` and returns at once | injection, untooling, ledger |
+
+`te fold` runs the **same classes the experiments ran** over the finished session's log:
+`PrefixStage` (envfacts, plus the paid proposer when allowed) and `ToolingStage` (evidence
+untooling). The spiral guard (`te_live/guard.py`) is `vendor.spiral.NudgeGuard`'s control flow —
+`SpiralState.update`, `assess_abort`, `is_stuck`, `nudge_text`, the cumulative backstop — one
+tool call at a time. None of the thresholds were re-tuned. Three things it adds:
+
+* It counts **edits made through the shell**: `sed -i`, `perl -pi`, a redirect or `tee` into a
+ source file, an inline Python script that writes a file back, `patch`/`git apply`. The vendored
+ state counts only edit tools, and agents that edit through Bash then looked as if they had
+ edited nothing.
+* **The nudge always comes before the stop.** The vendored ladder checks a hard backstop (a
+ multiple of the learned envelope: 2.5 × at the default level, 1.8 × at `aggressive`) before
+ anything else. With the large envelopes of the benchmark streams, the backstop lay well past
+ the stuck rule's own floor (20 calls at the default level, 14 at `aggressive`), and the nudge
+ came first. An interactive repository learns a small envelope. Its backstop then falls inside
+ that floor, so the first call that counts as stuck has also tripped the backstop, and the turn
+ would be stopped (or, in nudge mode, merely flagged) without ever being redirected. Now an
+ unspent nudge goes first, and the stop needs a second stuck segment after it.
+* With `paid` on, the **stuck judge and nudge writer run in a detached `te judge` job**, not in
+ the hook. The hook starts the job and returns; a later PostToolUse picks up the verdict, with
+ the redirect already written. Each job's spend is charged to the ledger when it is spent.
+
+Every hook fails open: any error becomes an empty output (no change) and a traceback in
+`~/.tokeneconomizer/errors.log`. The hooks are stdlib only, and none waits on a model call.
+
+## 2. What changes relative to the experimental setting
+
+| | Experiments | Plugin | Why |
+|---|---|---|---|
+| Unit of work | one benchmark task | a **session** for injection and untooling; a **prompt-turn** for the spiral guard | In interactive use a person already splits work into short turns. A whole session folded as one trajectory would look stuck merely by being long |
+| Learning signal | the official grader's `resolved` | the envelope learns from turns that **ended green**: the agent stopped on its own, was not stopped by the guard, and no verification after its last edit failed | There is no grader. Injection and evidence untooling never needed one |
+| Injection channel | a prefix on the task prompt | `SessionStart` `additionalContext` | Verified to reach the model (§3) |
+| Tool removal | `--disallowed-tools` / Codex config at launch | a **deny list you approve**, written to the repository's own settings file — tools in `.claude/settings.local.json`, Codex capability packages in `.codex/config.toml`, Copilot MCP servers in `.github/copilot/settings.local.json` — effective from the **next** session | The plugin launches nothing. Tools are fixed at session start, and changing them mid-session would re-write the cached prompt prefix. Each agent's file can express a different unit (§3) |
+| Untool candidates | `reflect.INAPPLICABLE_TOOLS` — orchestration, Q&A, plan mode, web — "inapplicable by construction" for a sealed headless task | nothing is inapplicable by construction; only **evidence** (loaded but unused for N sessions) proposes a tool, and a person's tools — AskUserQuestion, plan mode, sub-agents, the web — are protected | A person asks questions, plans and browses in a real session. A deny rule the plugin wrote for a tool that has since become protected is given back at the next fold |
+| Network | every arm, baseline included, carries an answer-key guard — a `PreToolUse` hook and an `/etc/hosts` block refuse the benchmarks' upstream sources — and drops the web tools | **no guard**: no `PreToolUse` hook (on Pi, no `tool_call` handler), no blocked command or host; the web tools stay | A benchmark's answer is a fetch away. A real session has no answer key, and the web is part of the work |
+| Nudge | kill the agent, resume it with a continuation prompt | a mid-turn reminder attached to the tool result | Different intervention, so its effect is not inherited from the experiments |
+| Abort | terminal, in autonomous runs | a `continue: false` stop only in **headless** sessions with `spiral: guard`; in an interactive session, a one-line notice — the person is already the abort | The vendored module says so itself: "interactive callers should map it to a hand-off" |
+| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`, default `haiku`), and only when `paid` is on | Every paid call is still charged to the session ledger |
+
+## 3. Verified agent behaviour
+
+What the code relies on about each agent, and how it was checked. Each block names the version
+and date it was verified on; §7 says what has not been exercised.
+
+**Claude Code 2.1.286 — verified with a local plugin (`--plugin-dir`) on Haiku, 2026-09-30:**
+
+* `SessionStart` `additionalContext` reaches the model. End to end through this plugin: four
+ synthetic sessions taught it that bare `pytest` is missing; a real `claude -p` session in that
+ checkout then answered "Tests should be run with `python -m pytest` since `pytest` command is
+ not found in this environment", and the detached fold priced that session from its transcript.
+* `PostToolUse` `additionalContext` is followed **in the same turn** — no restart needed.
+* `PostToolUse` `{"continue": false}` halts the loop after the current tool (one of three
+ requested commands ran) and the session ends cleanly.
+* A bare-name `permissions.deny` rule **removes the tool from the session's tool list**, the same
+ list `--disallowed-tools` produces, whether the rule comes from `--settings` or from a
+ project's `.claude/settings.local.json`. A plugin's own `settings.json` cannot carry deny rules
+ (only `agent` and `subagentStatusLine` take effect), which is why the untool writer edits the
+ project-local file, with approval.
+* An interactive transcript does not record the tool list a session loaded. Without a recorded
+ catalog, evidence untooling sees only the tools Claude Code defers (one listing line each) and
+ those a session used; `te untool refresh-catalog` records the always-loaded set, with no model
+ call.
+* Bash `tool_response` is `{stdout, stderr, interrupted, isImage, noOutputExpected}` — **no exit
+ code**, so on Claude Code failure is read from the output text, as the vendored detectors do.
+* Interactive transcripts parse with the existing `trace.load_trajectory`, but write **one entry
+ per content block, each repeating its message's usage**, so an assistant message id can repeat
+ many times; the ledger de-duplicates by message id. Subagents log to
+ `<session>/subagents/`, and their calls and tokens are counted with their session.
+* A `claude -p` session's hooks see `CLAUDE_CODE_SESSION_ATTENDED=0` and
+ `CLAUDE_CODE_ENTRYPOINT=sdk-cli`; `mode: auto` uses them to tell headless from interactive.
+
+**Codex 0.153.1 — verified live with `codex exec`** (`gpt-5.6-sol`, direct OpenAI API key, an
+isolated `CODEX_HOME`, `--dangerously-bypass-hook-trust`; 2026-09-30):
+
+* All five hooks fire — `SessionStart`, `UserPromptSubmit`, `PostToolUse`, `Stop`, `SessionEnd` —
+ and both `SessionStart` and `PostToolUse` `additionalContext` reach the model (a probe's
+ codeword and its mid-turn instruction both came back in the answer).
+* `PostToolUse` fires for shell commands as `tool_name: "Bash"`, `tool_input: {"command": "..."}`,
+ with `tool_response` a plain string and no exit code. Issue #38850 (PostToolUse not firing in
+ Code Mode) did not reproduce on this build and model.
+* The plugin installs through Codex's own commands: `codex plugin marketplace add <dir>` reads
+ `.claude-plugin/marketplace.json`, and `codex plugin add tokeneconomizer@tokeneconomizer`
+ **copies** the plugin into `$CODEX_HOME/plugins/cache/…` and enables it — so Codex needs the
+ bundle, exactly as a Claude Code marketplace install does. `"${PLUGIN_ROOT:-…}"` in the hook
+ command resolves to that copy.
+* End to end: four synthetic rollouts taught it that bare `pytest` is missing; a real Codex
+ session then answered "Run tests with `python -m pytest …` because the `pytest` command is
+ unavailable here", the guard folded the turn's tool call, and SessionEnd's detached fold priced
+ the session from its rollout: fresh, cached, written and output tokens matched the stream's own
+ usage block, converted as `te_harness.codex.row_usage` converts it.
+* Hooks inherit the agent's environment, including a parent Claude Code session's `CLAUDE_*`
+ variables, which is why the host is detected from the payload (`turn_id`), never the env.
+* `CODEX_HOME` under `/tmp` makes Codex refuse to install its helper binaries, and its sandbox then
+ fails every command (`codex-linux-sandbox: No such file`); keep it out of temporary dirs.
+
+Hooks are **trust-gated**: skipped until approved in `/hooks` (stored as
+`[hooks.state.<key>].trusted_hash`), unless the run passes `--dangerously-bypass-hook-trust`.
+That gate, kebab-case event names and an argv-list handler are why `te_harness` once recorded
+that Codex "honours no PreToolUse hook"; with PascalCase events, a command string and trust,
+every hook fires. Not exercised, from the binary's schemas and the docs: `PostToolUse`
+`continue: false` does **not** stop a Codex turn mid-flight, so on Codex the guard's last rung
+is a notice, matching the Codex-native spiral policies, which never abort. Code Mode presents
+one `exec` tool with nested descriptions and defers MCP tools, so Codex untooling works on
+capability packages (below), not on single tool schemas.
+
+**Codex untooling — verified on 0.153.1, 2026-10-07** (`codex debug prompt-input` renders a
+session's prompt input locally, with no model call; `codex features list` prints the effective
+feature flags):
+
+* A project's `.codex/config.toml` is read **only in a trusted project**. The same toggles
+ changed the rendered prompt input under a `CODEX_HOME` that trusted the checkout, and changed
+ nothing in an untrusted one, including a subdirectory of a trusted path. `te --host codex
+ untool apply` checks trust and runs `codex features list` to confirm.
+* The project-local toggles for `collaboration` (`[agents] enabled`, `[features] multi_agent`,
+ `include_collaboration_mode_instructions`), `goals` and `image_media` take effect. `goals` and
+ `view_image` remove declarations inside the code-mode tool description, which the render does
+ not show, so `untool measure` understates them, and its output says so.
+* `[[skills.config]] enabled = false` hides the bundled skills listing from the user-level
+ config only. A per-repository decision does not belong there, so `skills` is not offered.
+* The plugin writes each toggle as a line marked `# tokeneconomizer:<capability>` into the
+ file. It never overrides a key you already set, and `restore` removes exactly its own lines.
+* Code mode routes every call through one `exec` tool (`tools.view_image(…)`,
+ `tools.web__run(…)`), recorded as `response_item` `custom_tool_call` records, not as rollout
+ items. The fold therefore reads a capability trace that includes them.
+* In a live session (gpt-5.6-sol on a Codex login), after `te --host codex untool apply
+ collaboration goals image_media` the model reported no sub-agents, image viewer or goals, and
+ could no longer view an image in the repository; `untool restore` returned the folder to its
+ earlier state. Codex loaded those runs' hooks from the repository's `.codex/hooks.json`. A run
+ trusts a project with `-c 'projects={"<path>"={trust_level="trusted"}}'`; the dotted `-c` form
+ does not take.
+
+**Copilot with a real model — 2026-10-07, over BYOK.** This machine's organization policy
+refuses the Copilot subscription's models ("Access denied by policy settings"). Copilot's BYOK
+mode (`COPILOT_PROVIDER_TYPE=anthropic`, `_BASE_URL`, `_API_KEY`) bypasses that, and these
+checks ran on `claude-haiku-4-5`, through `te --host copilot probe` (`te_live/probe.py`, which
+any user can run on their own setup):
+
+* **Settings apply only in a trusted folder.** Copilot ignores a repository's `.github/copilot`
+ settings outside its `trustedFolders`: a deny list applied in an untrusted directory under
+ `/tmp` changed nothing, and a local capture showed the same file taking effect under a trusted
+ folder. `untool apply` therefore warns about an untrusted folder and confirms the effect with a
+ local capture, `doctor` reports trust, and the probe builds its repository inside a trusted
+ folder.
+* Copilot's `session.shutdown` records the request's tool-definition size as
+ `toolDefinitionsTokens`; the probe reports it before and after the deny list.
+* **Usage under BYOK** keeps the grand-total form: with an Anthropic provider, `inputTokens`
+ includes the `cacheWriteTokens`, which the ledger splits out.
+* **Paid side calls under BYOK.** With `COPILOT_MODEL=auto` (the subscription's router) in the
+ environment, `copilot -p` without a model answered nothing. The side call now names a model:
+ `paid_model`, else a concrete `COPILOT_MODEL`, else `claude-haiku-4-5` on an Anthropic
+ provider. It answered, and was priced from its own session log. The local capture behind
+ `refresh-catalog` and `measure` mimics the person's model family, so it records the same
+ built-in tools (`edit`/`create` for Claude models, not `apply_patch`).
+
+**GitHub Copilot CLI 1.0.83 — verified live, 2026-10-07**, against a scripted OpenAI-compatible
+endpoint on localhost (Copilot's BYOK mode, `COPILOT_PROVIDER_BASE_URL`). Every request the CLI
+built was recorded, so whether a hook's output reached the model was read off the request
+itself, with no model and no quota:
+
+* Copilot loads a Claude-format plugin, and prefers `.github/plugin/plugin.json` to
+ `.claude-plugin/plugin.json`: with both present, only the hook file the first one names
+ ran. That is how the plugin gives Copilot its own `hooks/copilot-hooks.json` (`--host
+ copilot`, plus `PostToolUseFailure`) while Claude Code keeps `hooks/hooks.json`.
+ `copilot plugin marketplace add <checkout>` and `copilot plugin install` load the plugin
+ live from the checkout ("nothing was copied"), and the installed copy's hooks all fired.
+* PascalCase events receive Claude-shaped snake_case payloads: `session_id`, `cwd`,
+ `hook_event_name`. `SessionStart` has `source: "new"`, and there is no turn id. Only `Stop`
+ carries `transcript_path` (`~/.copilot/session-state/<id>/events.jsonl`). Tool names are
+ Claude's (`Bash`, `Read`). A shell result is `tool_result.text_result_for_llm`, ending in
+ `<shellId: N completed with exit code X>`. A failed tool fires `PostToolUseFailure` with an
+ `error` string.
+* Context reaches the model only as a **top-level `additionalContext`**, on `SessionStart`,
+ `UserPromptSubmit` and `PostToolUse` (the last as "Additional guidance from postToolUse
+ hooks"). `hookSpecificOutput.additionalContext` is ignored.
+* `continue: false` from `PostToolUse` did **not** stop a turn: all three scripted calls ran.
+ As on Codex, the guard's last rung is a notice.
+* The session log is the `tool.execution_*` stream the trace seam already reads. Usage arrives
+ in `session.shutdown` (`modelMetrics[].usage`), written just after the `SessionEnd` hook
+ returns, so the detached fold waits for it. `inputTokens` is the grand input total: a scripted
+ 4 × (1,000 prompt, 600 cached) became `inputTokens` 4,000 and `cacheReadTokens` 2,400.
+* End to end, through the installed plugin: four sessions that hit "bare `pytest` is missing"
+ taught it the fact, and in the sessions after them the fact was in the request the model
+ received. Every session was folded and priced from its own `session.shutdown`.
+* **The interactive TUI** (driven in a pseudo-terminal, same local endpoint). `SessionStart`
+ fires lazily, at the first prompt; `SessionEnd` fires **once per session**, at `/exit`, not per
+ prompt. `systemMessage` is **never shown**, so a notice was invisible until the plugin printed
+ it as a `{"type": "progress", "message": …}` line ahead of its JSON. Copilot displays that as a
+ status line (verified on SessionStart, UserPromptSubmit and PostToolUse), and the final object
+ still applies. Plugin slash commands **do** run in the TUI, as skills. `-p` does not expand
+ them. Copilot substitutes only `$ARGUMENTS` in a command's text: `${CLAUDE_PLUGIN_ROOT}`,
+ `${PLUGIN_ROOT}`, `${COPILOT_PLUGIN_ROOT}` and `{{plugin_root}}` arrive literally, and its shell
+ exports none of them. So every session start now keeps a launcher at
+ `~/.tokeneconomizer/bin/te`, and the commands fall back to it when the substituted path is not
+ there. Through it, `/tokeneconomizer:status` printed Copilot's status in Copilot, and Claude
+ Code's still printed Claude's.
+* A side effect to know when scripting Copilot's TUI: under an editor's terminal (`TERM_PROGRAM`
+ set), its first run offers to add a shift+enter binding and writes that editor's
+ `keybindings.json` if accepted.
+* Untooling. Copilot has no persistent per-tool filter: `--excluded-tools` is a launch flag,
+ and `permissions.deny` refuses a call but leaves the schema in the request. What does work,
+ from the repository's `.github/copilot/settings.local.json`, is `disabledMcpServers`: listing
+ the built-in `github-mcp-server` removes all of that server's tools from the request.
+ `githubMcpTools`, `disabledSkills` and `builtInAgents` changed nothing. The plugin therefore
+ proposes a server only when none of its tools was used. With the catalog recorded
+ (`refresh-catalog`, captured the same way, with no model call), five sessions that never used
+ the GitHub tools produced that proposal, and the next session's request no longer carried the
+ server's tools.
+
+**Pi 0.99.2 — verified live with `pi --print`** (`@earendil-works/pi-coding-agent`, the
+extension loaded with `--extension`, gpt-5.6-sol through an `openai-responses` provider in an
+isolated `PI_CODING_AGENT_DIR`):
+
+* Pi has no hook protocol; `plugin/pi/tokeneconomizer.ts` speaks the dispatcher's on Pi's
+ behalf — `session_start`, `before_agent_start`, `tool_result`, `agent_end` and
+ `session_shutdown` each become one `te-hook --host pi` call — and applies the answers with
+ Pi's own levers: a system-prompt section, a steering message, `ctx.abort()`, a UI notice.
+* End to end: four synthetic Pi session files taught it that bare `pytest` is missing; in a real
+ session the `<tokeneconomizer>` section appeared in the session's recorded system prompt, the
+ model answered "Tests should be run with `python -m pytest …` because the `pytest` command is
+ unavailable here", the guard folded its `bash` call, and the detached fold priced the session
+ from Pi's own session file.
+* The section is re-applied on every prompt: Pi diffs prompt sections between runs and records
+ the change, so one left out would be recorded as removed. Unchanged, it adds nothing after
+ the first prompt.
+* Pi's bash result carries a real exit code (`structuredContent.exit_code`), which the
+ ended-green proxy uses; its session file is a tree, and only the active branch is read.
+* Not exercised live: a nudge (`sendMessage` with `deliverAs: "steer"`) and a stop
+ (`ctx.abort()`) — the dispatcher's side is covered by the offline tests, Pi's by its API.
+ Untooling does not apply: Pi declares four tools by default.
+
+## 4. Install
+
+The plugin directory is `plugin/`; the repository root is also a Claude Code marketplace
+(`.claude-plugin/marketplace.json`). Requires Python ≥ 3.10 on `PATH` (or `TE_PYTHON`).
+
+**Claude Code, from a checkout**
+(`git clone https://github.com/microsoft/TokenEconomizer.git`). This loads in place,
+so edits apply at the next session. Then run `/tokeneconomizer:untool refresh-catalog` once per
+repository; without it, evidence untooling sees only deferred tools:
+
+```bash
+claude --plugin-dir /path/to/TokenEconomizer/plugin # one session
+# or, persistently:
+/plugin marketplace add /path/to/TokenEconomizer
+/plugin install tokeneconomizer@tokeneconomizer
+```
+
+**From a bundle** — a marketplace install copies only the plugin directory, so it needs the core
+vendored inside it. `python scripts/build_plugin.py` writes a self-contained marketplace to
+`dist/marketplace/` (add it with `/plugin marketplace add`, publish it as a branch or repository,
+or `--zip` it for `claude --plugin-url`).
+
+**Codex** — the plugin carries `.codex-plugin/plugin.json` and `hooks/codex-hooks.json`. Codex
+always copies an installed plugin, so install from the bundle:
+
+```bash
+python scripts/build_plugin.py --out ~/te-marketplace
+codex plugin marketplace add ~/te-marketplace
+codex plugin add tokeneconomizer@tokeneconomizer
+```
+
+Then approve its hooks once in the TUI (headless runs: `codex exec
+--dangerously-bypass-hook-trust`, for automation that vets its hook sources). Verified end to
+end on 0.153.1 (2026-10-08, in an isolated `CODEX_HOME`): the bundle installs as a copy with the
+core vendored in `lib/`. Until trusted, the hooks do not run, and a session leaves no plugin
+state. At start the TUI says "Hooks need review: 5 hooks are new or changed". Its hooks screen
+lists the plugin's five hooks; `t` trusts all, recorded as
+`[hooks.state."tokeneconomizer@tokeneconomizer:hooks/codex-hooks.json:…"] trusted_hash`.
+After that, a plain `codex exec` ran them, folded and priced the session, and wrote
+`~/.tokeneconomizer/bin/te` pointing into the plugin cache. Through that launcher, `untool
+apply` removed sub-agents, goals and the image viewer from the next live session, by the
+model's own report, and `restore` gave them back. The deny list
+(`te --host codex untool …`) writes the project's `.codex/config.toml`, which Codex reads only
+once you trust the project.
+
+**Copilot CLI** — the plugin carries `.github/plugin/plugin.json` and `hooks/copilot-hooks.json`.
+Copilot loads a local marketplace in place, so the checkout itself is the install:
+
+```bash
+copilot plugin marketplace add /path/to/TokenEconomizer
+copilot plugin install tokeneconomizer@tokeneconomizer
+te --host copilot untool refresh-catalog # once: the tools a session loads, no model call
+```
+
+Copilot runs the plugin's slash commands in its interactive mode, not under `copilot -p`. From
+a terminal, use `~/.tokeneconomizer/bin/te --host copilot <command>` (or `plugin/scripts/te`;
+in Copilot's own shell, `te` detects Copilot without `--host`).
+
+**Pi** — `plugin/` is also a Pi package (`package.json` → `pi.extensions`). Pi loads a local
+package in place, so from a checkout:
+
+```bash
+pi install /path/to/TokenEconomizer/plugin # or, for one run: pi -e plugin/pi/tokeneconomizer.ts
+```
+
+Elsewhere, install the bundle's plugin directory the same way (it carries the core in `lib/`).
+Inside Pi, `/te status` (and `facts`, `replay`, `config`) runs the command line.
+
+**First run.** Before turning anything on, see what it would do on your own history:
+
+```text
+/tokeneconomizer:replay # or: plugin/scripts/te replay
+```
+
+## 5. Configure
+
+Each mechanism has its own switch, and each takes a plain `on` / `off` (`on` = the default
+below). The defaults are the conservative end of each:
+
+| Key | Default | Values |
+|---|---|---|
+| `inject` | `envfacts` | `off`, `envfacts`, `envfacts+distiller`, `distiller` (the last two make paid calls) |
+| `untool` | `propose` | `off`, `propose`, `auto` (apply proposals as they form) |
+| `spiral` | `nudge` | `off`, `nudge`, `guard` (nudge, then stop — headless only) |
+| `mode` | `auto` | `auto`, `interactive`, `headless` |
+| `paid` | `false` | gate for every LLM call a mechanism would make |
+| `paid_model` | `""` | model for paid calls (`haiku` on Claude Code when empty) |
+| `shadow` | `false` | compute and log every decision, act on none |
+
+Hyperparameters, set by hand:
+
+| Key | Default | Sets |
+|---|---|---|
+| `inject_max_facts`, `inject_max_tokens` | `4`, `150` | caps on the injected fact block (`InjectionConfig.max_facts` / `max_tokens`, passed to `EnvFactsDistiller`) |
+| `propose_every` | `5` | sessions between paid proposer runs |
+| `untool_evidence` | `5` | sessions a tool must go unused (×3 for NotebookEdit and Codex's `image_media`; WebFetch / WebSearch are never proposed) |
+| `spiral_aggr` | `default` | `conservative`, `default`, `aggressive` (the vendored `SPIRAL_LEVELS`) |
+| `spiral_nudges` | `1` | the guard's nudge budget per prompt-turn |
+| `spiral.<name>` | the preset | one key of `DEFAULT_THRESHOLDS`, pinned: `te_live.guard.thresholds` applies it over the preset and over what the repository's envelope learned; the `envelope_*` keys go to the `EnvelopeTracker` itself |
+
+Set them with `te config set KEY=VALUE` (this repository) or `--global`, `te config unset KEY`
+to fall back, or per session with `TE_<KEY>` (`TE_SPIRAL_THRESHOLDS` takes JSON);
+`TE_DISABLE=1` turns the plugin off. `te config show` prints each mechanism's state and the
+guard's thresholds in effect. Commands: `/tokeneconomizer:status`, `:facts`, `:untool`,
+`:replay`, `:config` in Claude Code and Copilot; `plugin/scripts/te <command>` anywhere. The
+user-facing reference is the Settings section of [`plugin/README.md`](../plugin/README.md#settings).
+
+## 6. State and privacy
+
+All state is local, under `$TOKENECONOMIZER_HOME` (default `~/.tokeneconomizer`), one directory
+per repository and host: the mechanisms' own state, `ledger.jsonl` (one priced record per
+session), `events.jsonl` (every injection, nudge, stop), and per-session guard state. A
+repository is identified by its `origin` remote, so worktrees and re-clones share what was
+learned. Uninstalling the plugin leaves this directory; delete it to forget. Like the agents'
+own session logs it is derived from, it is private to you: the plugin creates it `0700` and its
+files `0600`, and tightens a looser one it finds.
+
+Nothing leaves the machine unless `paid` is on. Then the proposer, judge and nudge writer send
+**digests** — command templates, edited file basenames, error signatures, never file contents —
+through your own agent CLI (`claude -p` / `codex exec` / `copilot -p`), billed where your
+session is.
+
+The untool writer adds and removes only deny rules it recorded adding, never one you wrote, and
+adds a `settings.local.json` it creates to `.git/info/exclude`. A repository decides what its
+`.claude`, `.codex` and `.github` paths are, so the writer refuses one that is, or goes through,
+a symlink leading out of the repository, and refuses to edit a settings file that does not
+parse rather than write over it.
+
+A learned fact is text injected into your agent's context, derived from your sessions' command
+output — which a repository controls. The free layer only fills fixed templates with observed
+command forms and hosts, and lints out completion advice; the paid proposer's facts are
+open-vocabulary. Read them with `te facts` before enabling the proposer on a repository you do
+not trust.
+
+## 7. Not done yet
+
+* **Copilot CLI** — the real-model checks used one model (Claude Haiku 4.5, over BYOK). A
+ per-tool deny list waits on Copilot gaining a persistent per-tool filter.
+* **The stuck-turn guard, live** — the stop rung has not run on a real spiral, and whether a
+ nudge changes a looping model's course is not established. The guard counts edits and
+ repeated failures, not understanding, so with a small learned envelope it can fire on a turn
+ that has already found its cause and is working around it.
+* **Pi** — a live nudge and stop (steer message, `ctx.abort()`); untooling is not planned there,
+ since Pi declares only four tools by default.
+* **Known gaps:**
+ * A turn that ran no recognised verification counts as ended green, so a failed turn without
+ one can enter the envelope and loosen the guard for every later turn.
+ * The free fact layer has no family for "this tool is absent here, with no working
+ alternative" (a missing `ruff`, `mypy` or `pre-commit`, say).
+ * Test runs launched from a script are not recognised as verification.
+ * `refresh-catalog` is manual; until it runs in a repository, evidence untooling sees only
+ deferred tools. Running it at a repository's first session would close that gap.
+* Windows (the launcher is POSIX `sh`), Python 3.9, and compressed Codex rollouts (`.zst`).
+* Transcript formats are internal to each agent and change between versions; the readers are
+ pinned by tests against the shapes observed on the versions above.
diff --git a/experiments/.env.example b/experiments/.env.example
new file mode 100644
index 0000000..6f3a0c1
--- /dev/null
+++ b/experiments/.env.example
@@ -0,0 +1,135 @@
+# TokenEconomizer — local credentials + run environment (TEMPLATE, tracked in git).
+#
+# cp .env.example .env # then fill in the real values; .env is git-ignored
+#
+# Format: plain `KEY=value`, one per line. A `#` starts a comment ONLY at the start of a
+# line -- a trailing `# ...` after a value is part of the VALUE, not a comment. Quote values containing
+# spaces. No `$VAR` interpolation and no command substitution — both the Python loader
+# (scripts/_env.py) and the shell loader (scripts/launch/_env.sh) treat values literally.
+# Loaders NEVER overwrite a variable already exported in your shell, so a one-off
+# `ANTHROPIC_API_KEY=... bash scripts/launch/...` still wins.
+
+# --- Agent credentials ---------------------------------------------------------------------
+# Each agent bills its own vendor's API directly:
+# Claude Code -> the Anthropic API on ANTHROPIC_API_KEY
+# Codex -> the OpenAI API on OPENAI_API_KEY
+# Copilot CLI -> the GitHub Copilot subscription (or Copilot's own BYOK, further down)
+# ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN are not read: the launchers say so and unset
+# them if they are set, because Claude Code would otherwise honour them.
+ANTHROPIC_API_KEY=
+OPENAI_API_KEY=
+# Regional OpenAI endpoint, ONLY for a key issued against one: a key scoped to a region
+# 401s on the global endpoint. Leave it unset for a global key. The launchers refuse any
+# host that is not OpenAI's own API.
+# OPENAI_BASE_URL=https://us.api.openai.com/v1
+
+# --- Models billed by the sealed agents ---------------------------------------------------
+# These label the cost basis and are what the harness passes to the CLIs. Keep an A/B's two
+# arms on the SAME model — the comparison is meaningless otherwise.
+#
+# The main sweep runs claude-opus-4-8 and claude-sonnet-5 on both agents, plus 4 required
+# arms on claude-opus-5 (docs/RUNNING.md). main_sweep.sh passes --model per lane
+# and that WINS over this variable, so for the sweep leave it commented out.
+# ANTHROPIC_MODEL=claude-opus-5
+# ANTHROPIC_MODEL=claude-sonnet-5
+# ANTHROPIC_MODEL=claude-opus-4-8
+#
+# SPELLING. The older Opus has two spellings and the catalogs disagree: the Anthropic API
+# serves the DASHED `claude-opus-4-8` and 404s the dotted form; the Copilot CLI's own
+# catalog on a subscription lists the DOTTED `claude-opus-4.8`. Write the dashed form
+# everywhere: the launchers spell it for the backend (te_model_for_agent in
+# scripts/launch/_env.sh, logged as `[env] model id: ... -> ...`), the Copilot probe
+# verifies it against the CLI and switches spelling if a vendor rotates it, and
+# te_model_tag folds both to one tree tag (opus48) so the two spellings can never fork a
+# tree. `claude-opus-5` and `claude-sonnet-5` have one spelling and are untouched.
+# Reasoning effort the sealed agent runs at: low|medium|high|xhigh|max. Part of the cost
+# basis, so keep BOTH arms of an A/B on the same level.
+# COMMENT THIS OUT for the main sweep: its baseline-low/-medium arms set the level from the
+# arm name, and an inherited value makes the driver log an override for every cell.
+TE_EFFORT_LEVEL=high
+# Small/fast model for Claude Code's own background calls. An unserved id is a per-call
+# FAILURE, not a fallback, and te_harness forwards this var into the task container, so a
+# wrong value reaches the CLI. Leave it unset to keep Claude Code's own default.
+# ANTHROPIC_SMALL_FAST_MODEL=claude-haiku-4-5
+# Codex model. `gpt-5.6-sol` is served on /v1/responses by the OpenAI API (probed
+# 2026-09-09). THE TRAP is the `-codex` ALIASES (gpt-5.1-codex, gpt-5.2-codex, ...):
+# /v1/models lists them and /v1/responses 404s them, so pinning one fails once per task.
+# Pin a base id.
+# Both gpt-5.6-terra and gpt-5.6-sol are evaluated, each in its OWN results tree (the model
+# is part of the cost basis). This picks the default for a command that does not pass MODEL=;
+# docs/RUNNING.md section 5 loops the arm catalog over both.
+CODEX_MODEL=gpt-5.6-sol
+
+# --- GitHub Copilot CLI (AGENT=copilot) -----------------------------------------------------
+# The Copilot CLI is its own sealed agent and needs no API key: it authenticates against
+# GitHub from ~/.copilot (or a token in COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN).
+# Full guide: docs/COPILOT.md.
+#
+# Copilot serves its own model catalog, so an Anthropic id is NOT valid here. 'auto' lets
+# the CLI route; a pinned id fixes the cost basis but must be one your subscription serves
+# (`copilot -p ok --model <id>` tells you). Caveat: 'auto' REJECTS a reasoning-effort flag,
+# so TE_EFFORT_LEVEL above is recorded as null and not applied unless you pin a model.
+COPILOT_MODEL=auto
+# headless/unattended alternative to `copilot /login`
+# COPILOT_GITHUB_TOKEN=
+#
+# BYOK -- Copilot's own built-in "bring your own key": bill your OWN provider instead of the
+# Copilot subscription. THE SUBSCRIPTION IS THE DEFAULT; this is the fallback for a host with no subscription, or one whose monthly quota
+# is spent (which surfaces as HTTP 402 `quota_exceeded` and a tree of turns=1/eff_tok=0
+# rows). COPILOT_PROVIDER_BASE_URL is the ONE switch: set it and the CLI bills that provider
+# and does not use GitHub auth at all. Full variable list: `copilot help providers`.
+#
+# A BYOK row and a subscription row are DIFFERENT COST BASES. The tree name says
+# `copilot_*` either way, so run_config.json -> routing.base_url is what tells them apart:
+# the provider URL under BYOK, the literal `github-copilot` on a subscription. Do not pool
+# the two (docs/COPILOT.md).
+# COPILOT_PROVIDER_TYPE=anthropic # openai (default) | azure | anthropic
+# COPILOT_PROVIDER_BASE_URL=https://api.anthropic.com
+# COPILOT_PROVIDER_API_KEY= # same key as ANTHROPIC_API_KEY above
+# NOTE: a trailing `# comment` on the same line becomes part of the value -- both loaders
+# treat only a WHOLE-LINE `#` as a comment. Uncomment the bare assignment, not these lines.
+
+# --- Data + results roots (see docs/BENCHMARKS.md, te_data/env.sh) ------------------------
+# These are site-specific paths with NO built-in default: the SWE-bench harness raises
+# at construction when one is missing, instead of resolving to a path that only exists on
+# whichever machine wrote the code. DATA is the root holding swebench/swebench_grading.json
+# and is the shorthand TE_SWEBENCH_GRADING falls back to.
+DATA=
+TE_SWEBENCH_REPO=
+TE_SWEBENCH_GRADING=
+# External tool baselines (tokeneconomizer/baselines/): where the pinned rtk release binary is
+# staged host-side, because the sealed task container cannot reach GitHub. Defaults to
+# $DATA/scaffolds when DATA is set. Only the rtk-bearing arms read it: rtk, trimech-rtk,
+# trimech-rtk-web and codex-trimech-rtk.
+TE_SCAFFOLD_CACHE=
+TOKENECONOMIZER_SWEBENCH_TASKS=
+TOKENECONOMIZER_WEBBENCH_TASKS=
+TOKENECONOMIZER_WEBBENCH_PROJECTS=
+TOKENECONOMIZER_RESULTS=
+
+# Every other harness knob -- turn caps, timeouts, the probes, the container CPU limit, the
+# answer-key override -- is in the config table in docs/HARNESS.md. Nothing there is
+# required: each has a default the harness states.
+
+# --- Host resource budget ------------------------------------------------------------------
+# TE_SWEBENCH_MAX_CPUS caps ONE task container (default 16) plus the OMP/MKL/OPENBLAS/NUMEXPR/
+# LOKY thread caps that must agree with it -- `--cpus` alone does not change what `nproc`
+# reports inside, so a parallel test suite would still size its pool to the whole host and
+# thrash. 0 = unlimited. Recorded per row in run_config.json -> cpu_limit.
+#
+# TE_FLEET_CPUS is the one that bounds the HOST: N concurrent lanes are an N x cap ceiling, so
+# the per-container value cannot do it. scripts/launch/fleet_budget.sh writes this onto the
+# cgroup every rootless-docker container scope shares, which binds lanes that are ALREADY
+# running and needs no restart. 0/none = do not enforce. The Codex launcher asserts it.
+# TE_SWEBENCH_MAX_CPUS=4
+# TE_FLEET_CPUS=16
+
+# --- Harness seam -------------------------------------------------------------------------
+# `stub` (offline StubRunner, no Docker/LLM) or `module:factory` returning an AgentRunner.
+# stub runs NO agent and grades nothing -- its rows are synthetic, for wiring checks only,
+# and the launchers now say so loudly at startup. For a real run point this at a harness --
+# the repo's own needs no PYTHONPATH, since the driver puts experiments/ and the core on
+# sys.path:
+# RUNNER=te_harness.swebench:make_runner # SWE-bench
+# RUNNER=te_harness.webbench:make_runner # Web-Bench (all three agents)
+RUNNER=stub
diff --git a/experiments/README.md b/experiments/README.md
new file mode 100644
index 0000000..3be78d9
--- /dev/null
+++ b/experiments/README.md
@@ -0,0 +1,248 @@
+# TokenEconomizer — reproducing the paper's experiments
+
+This directory holds the code behind the paper: the Docker benchmark harness, the sweep and
+aggregation scripts, the evaluated task sets, and the run book. It is **not part of the
+plugin** and is not packaged with it. The plugin, and the shared core both use, live at the
+repository root; see [`../README.md`](../README.md).
+
+Everything here keeps the layout the experiments were run with: `te_harness/`, `scripts/`,
+`workloads/` and `docs/` sit side by side. The shared core, `tokeneconomizer`, stays at the
+repository root, one level up, and the scripts put both directories on `sys.path`
+themselves (an editable install, `pip install -e ..`, works too). Run commands from this
+directory (`cd experiments`), or by path from anywhere; the scripts locate their own root. The `.env` the scripts read is
+`experiments/.env`, else the repository root's.
+
+### Proactive runtime cost control for sealed coding agents
+
+**Externally-delivered, agent-agnostic middleware that aims to cut a sealed coding agent's
+amortized token cost without hurting task success.**
+
+---
+
+## The problem
+
+A sealed coding agent — Claude Code, the Copilot CLI, Codex — is optimized for **task
+success, not the token bill**. You cannot retrain it, patch it, or see inside it. What you
+get is a fixed interface: a launch command, an instruction channel, a tool contract, and a
+stream of events. Meanwhile its transcript grows monotonically and is replayed into every
+model call.
+
+That shape produces three costs that recur on *every task in a stream*, not once:
+
+| | Cost region | What it looks like |
+|---|---|---|
+| 1 | **Repeated environment discovery** | Re-probing the test command, the build, the repo layout — things the agent already worked out on the previous task, and the task before that. |
+| 2 | **The fixed tool contract** | Every tool schema re-sent on every turn, including the ones this repository will never need. |
+| 3 | **The unproductive tail** | Tokens burned after a rollout has stopped making progress but before it gives up. |
+
+An experienced engineer working in a familiar repository avoids all three. They already know
+the test command, they don't reach for tools the project doesn't use, and they abandon a dead
+approach early.
+
+## The question this repository asks
+
+**Can an external controller supply that familiarity while leaving the agent completely
+fixed** — no weight updates, no fine-tuning, no changes to agent code?
+
+A static plugin is not enough, and that is the substantive claim. How much there is to save,
+and where, depends on the repository, the harness, the agent's tool surface, and the
+trajectory currently unfolding. So the controller has to be **stateful and observational**:
+it reads completed and live traces, maintains per-repository state across a task stream, and
+acts only through interfaces the harness already exposes — the launch call, the instruction
+channel, and the event stream.
+
+Three independently-switchable mechanisms each target one cost region, at a different point
+in a task's lifecycle:
+
+```
+task lifecycle: ── inject ──▶ construct tools ──▶ ═══ rollout ═══▶ finalize ──
+ │ │ │
+ injection tooling spiral
+ learned per-repo drop inapplicable kill/nudge a stuck
+ operating manual tool schemas / rollout vs a learned
+ before the task capabilities resolved-turn envelope
+ (between tasks) (before each task) (during rollout)
+```
+
+Because they are independently switchable, each can be measured alone as an ablation of the
+combined stack rather than as a separate system.
+
+| Mechanism | Module | What it does |
+|---|---|---|
+| **injection** | `mechanisms/injection.py` | A paid proposer learns a firing-gated per-repo operating manual (the canonical test command, "a repro script is redundant here", a required env var) over the task stream, and injects it before each task. A formation / decay / firing-gate lifecycle keeps only facts that actually fire. |
+| **tooling** | `mechanisms/tooling.py`, `mechanisms/codex_untool.py` | Drops inapplicable tool schemas: evidence untooling (loaded-but-unused for *N* consecutive tasks) plus an optional instruction-aware judge. Never drops a protected tool — without an edit path and a shell there is no task. |
+| **spiral** | `mechanisms/spiral.py`, `mechanisms/codex_spiral.py` | Kills or nudges a stuck rollout against a learned per-repo resolved-turn envelope: cheap gate → sampled LLM judge → nudge, then abort. |
+
+## What is actually measured
+
+The objective is to **minimize cache-aware effective cost while holding resolve rate** (or
+F1, on the classification domains).
+
+Two properties make that honest rather than rhetorical:
+
+- **Cache-aware cost.** Most raw transcript tokens are cached re-reads. Counting them at full
+ weight would let a method look good merely by pruning cheap repetition, so cost is
+ measured in fresh-input-token equivalents.
+- **The controller pays for itself.** Every token the controller spends — proposer, tool
+ judge, spiral judge, nudge writer — is charged back to TokenEconomizer in the same ledger
+ as the task. Reported savings are **net of the mechanisms' own overhead**, so a mechanism
+ that buys savings with paid calls has to win on the balance.
+
+One consequence worth stating up front: a mechanism's overhead line is *not* a saving
+available by deleting it, because the mechanisms shape the trajectory that produces every
+other number in the row. Use the ledger to price a mechanism, not to predict its removal.
+
+Results are not part of this repository; the code here produces them.
+
+---
+
+## How the repository is organised
+
+The layout follows one rule: **the controller must not be able to reach the things it is
+supposed to be independent of.** That is enforced structurally, by what sits underneath
+what, rather than by convention.
+
+### Three layers
+
+**Layer 1 — `tokeneconomizer/`, the controller.** The mechanisms, the cost definition, and
+the trace reader. It is **stdlib-only with zero runtime dependencies**: `import
+tokeneconomizer` cannot touch Docker, an agent CLI, or an LLM SDK, because none of that code
+lives beneath it. This is what makes the agent-agnostic claim checkable instead of
+aspirational.
+
+**Layer 2 — `te_harness/`, the reference harness.** The heavy machinery: launching a sealed
+agent inside a benchmark's Docker image, applying the controller's per-task decisions,
+streaming events back, and grading with the benchmark's *official* grader. It is a
+**sibling** package of `tokeneconomizer`, not a submodule, and is excluded from the
+distributed wheel — which is precisely what keeps Layer 1's guarantee structural. Swapping in
+your own harness is a supported path, not a fallback: nothing above the seam changes, because
+the controller never learns which harness it is talking to.
+
+`te_live/` (at the repository root) is the second harness, and the proof of that claim: the
+**live harness** runs the same mechanism classes inside a user's own Claude Code, Codex, Copilot
+CLI or Pi sessions, through the hooks the agent exposes, and `plugin/` packages it for
+installation. It is a sibling of the core for the same reason `te_harness` is — it reaches an
+agent CLI — but it ships in the wheel, because it is what `te` runs. See
+[`../docs/PLUGIN.md`](../docs/PLUGIN.md).
+
+**Layer 3 — `scripts/`, the reproduction pipeline.** Planning a sweep, driving one arm,
+aggregating row trees into CSVs, and comparing arms. Nothing here is imported by Layers 1
+or 2.
+
+### The four seams
+
+Every crossing between layers goes through exactly one interface, which is why a new agent,
+harness, or benchmark is an addition rather than an edit:
+
+| Seam | Module | Separates |
+|---|---|---|
+| `AgentRunner` | `runner.py` | the controller from task execution and paid LLM calls |
+| `DomainAdapter` | `adapters/` | the controller from what a "task" is |
+| trace | `trace.py` | every mechanism from the shape of a rollout's event stream |
+| ledger | `ledger.py` | every mechanism from the definition of cost |
+
+The mechanisms read rollouts through the one trace seam and charge cost through the one
+ledger, so a change to either is a change in one place, and no mechanism can quietly adopt
+its own accounting.
+
+### The tree
+
+```
+experiments/ the paper's reproduction code (this directory)
+ te_harness/ Layer 2 — reference AgentRunner: Docker + agent CLIs + official graders
+ swebench.py agent in the instance's pinned image, graded by get_eval_report
+ webbench.py agent at the Node evaluator's HTTP seam, graded by its own suites
+ copilot.py Copilot CLI overrides (GitHub auth, own catalog, own accounting)
+ codex.py Codex CLI seam — the OpenAI API, the capability→config
+ mapping, own accounting
+ guards/ measurement-integrity guards — refusing fetches of a benchmark's
+ own answer key, enforced twice: a PreToolUse hook in each CLI's
+ own hook file, and an /etc/hosts block in the task container
+ capture_catalog.py mines the real tool catalog off a live run's wire
+ scripts/ Layer 3 — reproduction pipeline (planning, driving, aggregating, comparing)
+ workloads/ the evaluated task sets as committed JSON (ids only, ~28 KB)
+ docs/ setup, provisioning, the harness, the run book (see below)
+ tests/ offline suite for the harness and pipeline — no Docker, no datasets
+
+../tokeneconomizer/ Layer 1 — the controller (stdlib-only, zero deps), shared with the plugin
+ facade.py Economizer, the orchestration entry point
+ config.py three independent on/off switches, one per mechanism
+ arms.py _armspec.py named arm → exact configuration (the experiment's vocabulary)
+ ledger.py trace.py the one cost definition + the one trace seam
+ runner.py AgentRunner interface — the harness seam
+ workloads.py the evaluated task set, as data (which instances, per domain)
+ mechanisms/ injection · tooling · spiral (+ the Codex-native variants)
+ baselines/ third-party tool scaffolds (rtk · serena · aider repo-map) —
+ comparison points, not mechanisms
+ adapters/ DomainAdapter + swebench · webbench · creddata · reposvul ·
+ cybergym · webarena
+ vendor/ pricing · reflect · spiral, vendored verbatim, stdlib-only
+../te_live/ ../plugin/ the plugin (see ../README.md)
+
+── and deliberately NOT in the repository ───────────────────────────────────────
+te_data/ the data root, beside the repository: benchmark checkouts, task
+ specs, grading tables, Docker-image manifests, every results tree.
+ Reached only through TOKENECONOMIZER_* / DATA env vars.
+```
+
+### The data root lives *outside* the repository
+
+Nothing large that the project reads or writes belongs in the checkout. Benchmark
+checkouts, task specifications, grading tables, and every results tree live under **one data
+root of your choosing, outside the repo** — conventionally `te_data/` beside it — and the
+code reaches it only through `TOKENECONOMIZER_*` / `DATA` environment variables.
+
+The reason is scale, and then three practical ones. A working root for this project runs to
+tens of gigabytes (the Web-Bench monorepo alone is ~24 GB, results trees a few GB more),
+on top of ~165 GB of pinned per-instance Docker images that live in the daemon's own store.
+None of that is versionable, and none of it should be one `git add -A` away from a commit.
+Beyond size:
+
+- **A clone becomes disposable.** Delete or re-clone the repo and your results are untouched
+ — which is what makes it safe to test a change in a throwaway clone against the corpus you
+ already have.
+- **Several checkouts can share one root.** Two clones, or a git worktree, can run against
+ the same provisioned data and the same results tree without re-provisioning any of it.
+- **No machine-specific path is baked into the code.** These variables have no built-in
+ defaults on purpose: the harness raises at construction when one is missing rather than
+ silently resolving to a path that only exists on whichever machine wrote the code.
+
+`.gitignore` is the belt-and-braces half of the same decision — `results/`, `*.csv`,
+`*.jsonl`, rendered images and `.env` are all excluded — but the primary defence is that
+the data was never inside the tree to begin with. [`docs/ENVIRONMENT.md`](docs/ENVIRONMENT.md)
+and [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) cover the layout and what to put where.
+
+### Provenance, kept as data
+
+The identities of the evaluated tasks live in the core package (and are mirrored as committed
+JSON under `workloads/`), not in a results tree. So the exact set a published number was
+computed over can be re-provisioned and re-run without inheriting anyone's output directory.
+The published corpus ran a fixed subset per domain rather than a whole benchmark; two
+SWE-bench streams have since been extended because every arm had saturated them, so a *new*
+run's task set is deliberately not identical to the recorded one — [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md)
+spells out both.
+
+---
+
+## Where things are documented
+
+| Document | What it covers |
+|---|---|
+| [`../docs/METHOD.md`](../docs/METHOD.md) | why the mechanisms are shaped this way |
+| [`docs/SETUP.md`](docs/SETUP.md) | fresh clone → running, gated step by step |
+| [`docs/ENVIRONMENT.md`](docs/ENVIRONMENT.md) | machine requirements and Docker setup |
+| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | per-benchmark provisioning and on-disk layout |
+| [`docs/HARNESS.md`](docs/HARNESS.md) | the `AgentRunner` seam, and writing your own harness |
+| [`docs/COPILOT.md`](docs/COPILOT.md) | running the Copilot CLI as the sealed agent |
+| [`docs/RUNNING.md`](docs/RUNNING.md) | the run book: authentication per agent, the arm catalog, one command per lane |
+
+**The plugin** — the mechanisms in your own sessions — is documented at the repository root:
+[`../README.md`](../README.md) and [`../plugin/README.md`](../plugin/README.md).
+
+**Start here:** [`../docs/METHOD.md`](../docs/METHOD.md) for the ideas,
+[`docs/SETUP.md`](docs/SETUP.md) to get it running, and
+[`docs/RUNNING.md`](docs/RUNNING.md) to reproduce the experiments.
+
+Requires Python ≥ 3.10. The core has no runtime dependencies; `requirements.txt` here covers
+the test suite (`python -m pytest tests`, from this directory, or
+`python -m pytest` at the repository root for both suites).
diff --git a/experiments/docs/BENCHMARKS.md b/experiments/docs/BENCHMARKS.md
new file mode 100644
index 0000000..e237653
--- /dev/null
+++ b/experiments/docs/BENCHMARKS.md
@@ -0,0 +1,494 @@
+# Benchmark provisioning — images, datasets, and env vars
+
+> Per-benchmark reference detail. If you are setting up from a fresh
+> clone, start with [`SETUP.md`](SETUP.md) and come here at its data step.
+
+This is the operational companion to [`../scripts/launch/README.md`](../scripts/launch/README.md).
+It documents, per benchmark, **exactly what you must pull/fetch/build and where it must land**
+before an economized A/B can run.
+
+> **Set up your environment first.** [`ENVIRONMENT.md`](ENVIRONMENT.md) covers the two layers
+> (the core package vs. the harness and what it drives), the `requirements.txt`, and Docker
+> install/update for beginners. The per-benchmark provisioning below is the harness layer.
+
+## What lives where (and why this doc exists)
+
+TokenEconomizer is middleware: it observes traces, maintains per-repo state, and decides
+per-task interventions. It **never pulls an image or downloads a dataset itself** — that is
+the job of the task-execution harness below the `AgentRunner` seam
+(`RUNNER=te_harness.swebench:make_runner`, or your own; see [`HARNESS.md`](HARNESS.md)).
+Nothing provisioned below is committed: the harness is code, and these are payloads.
+
+### The two kinds of path you configure (do not confuse them)
+
+Every domain has **two** independent inputs, read by **two** different components:
+
+| # | What | Read by | Value | Env var pattern |
+|---|---|---|---|---|
+| A | **Task list** | `run_experiment.py` (the driver) — `json.load()`s it | a **JSON file** (a list of `{instance_id, problem_statement, repo}`) | `TOKENECONOMIZER_<DOMAIN>_TASKS` |
+| B | **Dataset root** | the domain **adapter** + your harness | a **directory / suite file** with the raw data | `TOKENECONOMIZER_<DOMAIN>_ROOT` (or `_PROJECTS` / `_SUITE`) |
+
+- **A (task list)** is what the driver iterates and what the per-task tool judge reads. It is
+ **always a JSON file path** — pointing it at a directory will crash with `IsADirectoryError`.
+- **B (dataset root)** is where the actual source/labels live, used by the adapter to enumerate
+ instances and by your harness to materialize each task's workspace.
+
+For **SWE-bench only**, A and B are the *same file*: `TOKENECONOMIZER_SWEBENCH_TASKS` is both
+the driver's task list and the adapter's instance source. For every other domain they are
+**different** — you export a small task-list JSON (A) *and* set the dataset root (B).
+
+The commands below are the reference provisioning recipe; adapt paths to your infra.
+**All fetched data is git-ignored** — none of it is committed to this repo.
+
+### Which tasks to actually run — `tokeneconomizer.workloads`
+
+The experiments do **not** run a whole benchmark. Each domain uses a fixed subset, and that subset is data in the core package — importable on a bare clone, with
+no results tree, no datasets, and no Docker:
+
+```python
+from tokeneconomizer import workloads
+
+workloads.task_counts("swebench") # {'django/django': 40, ..., 'pydata/xarray': 39}
+workloads.total_tasks("swebench") # 229 tasks per seed
+workloads.instances("swebench", "pydata/xarray") # the exact 39 instance ids
+workloads.swebench_image_tag("django__django-11532") # the image to pull for one task
+```
+
+| Domain | Workload | Seeds |
+|---|---|---|
+| SWE-bench | **229** recorded over 6 repos (django 40, sphinx 44, sympy 40, xarray 39, matplotlib 34, sklearn 32); a NEW run executes **277** — `workloads.SWEBENCH_EXCLUDED` drops one sklearn instance and `workloads.SWEBENCH_ADDED` extends the two saturated streams, so `runnable_instances()` reports **sklearn 60** and **sympy 60** | `0 1 2` |
+| Web-Bench | **102** tasks over 5 projects (calculator 21, table 21, expressjs/react/zustand 20 each) | `A B C` |
+| CredData | **100** of the 333 repo ids | `0 1 2` |
+| ReposVul | **100** instances, all from the **Python** split (`py-*`) | `0 1 2` |
+
+Provision only what the workload names — it is a small fraction of each full benchmark, and
+for SWE-bench it is the difference between ~277 and 2,294 multi-GB images.
+
+The same lists ship as committed JSON manifests under [`../workloads/`](../workloads/) —
+ids only, ~16 KB for all four, readable without importing the package. The launchers default
+`WORKLOAD` to `workloads/$DOMAIN.json` and pass it to the driver, so a launch is pinned to the
+evaluated set even when your `TASKS_FILE` holds a superset (`WORKLOAD=none` opts out):
+
+```bash
+# uses workloads/swebench.json automatically:
+AGENT=claude DOMAIN=swebench ARM=trimech REPOS="pydata/xarray" SEEDS="0 1 2" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/trimech_ab.sh
+
+# or explicitly, straight to the driver:
+python scripts/run_experiment.py --workload workloads/swebench.json --repo pydata/xarray ...
+```
+
+To pin a run to exactly the evaluated set from Python instead, filter the task list yourself:
+
+```python
+tasks = json.load(open(os.environ["TOKENECONOMIZER_SWEBENCH_TASKS"]))
+tasks = workloads.filter_tasks(tasks, "swebench", "pydata/xarray") # exactly 39
+```
+
+> Prefer `filter_tasks` over the driver's `--repo` flag. That flag prefix-matches the repo's
+> short name against `instance_id` and **falls back to the entire task list when nothing
+> matches** (`run_experiment.py:188`) — which is what happens for any repo whose org differs
+> from its name, `pydata/xarray` included.
+
+### Set a base directory first
+
+Every command below writes under `$DATA`. Pick any writable path and export it once
+(`$TOKENECONOMIZER_RESULTS` is where runs land — also your choice):
+
+```bash
+export DATA="$HOME/te_data" # anywhere you like; holds datasets + task lists
+export TOKENECONOMIZER_RESULTS="$DATA/results" # git-ignored results tree
+mkdir -p "$DATA" "$TOKENECONOMIZER_RESULTS"
+```
+
+### Quick map of every placeholder
+
+| Env var | Kind | Consumed by | Points at |
+|---|---|---|---|
+| `TOKENECONOMIZER_SWEBENCH_TASKS` | A + B | driver + swebench adapter | SWE-bench instances JSON (`{instance_id, problem_statement, repo}`) |
+| `TOKENECONOMIZER_WEBBENCH_TASKS` | A | driver | Web-Bench task-list JSON you export (below) |
+| `TOKENECONOMIZER_WEBBENCH_PROJECTS` | B | webbench adapter | Web-Bench `projects/` tree (each has `tasks.jsonl`) |
+| `TOKENECONOMIZER_CREDDATA_TASKS` | A | driver | CredData task-list JSON you export (below) |
+| `TOKENECONOMIZER_CREDDATA_ROOT` | B | creddata adapter | CredData checkout (`data/`, `meta/`) |
+| `TOKENECONOMIZER_REPOSVUL_TASKS` | A | driver | ReposVul task-list JSON you export (below) |
+| `TOKENECONOMIZER_REPOSVUL_ROOT` | B | reposvul adapter | prepared ReposVul (`instances/*.json`) |
+| `TOKENECONOMIZER_CYBERGYM_SUITE` | B | cybergym adapter | CyberGym suite JSON |
+| `TOKENECONOMIZER_RESULTS` | — | aggregate + harness | results trees the harness writes and the aggregator reads |
+| `<DOCKER_IMAGE>` | — | your harness (not this repo) | the era-pinned benchmark image |
+
+---
+
+## 1. SWE-bench (primary — Claude / Copilot `trimech`, Codex arms)
+
+**Dataset.** SWE-bench Verified (500 instances) from Hugging Face. The economizer only needs
+the issue text; the grading gold patch stays inside your harness. Here the task list (A) and
+the adapter's instance source (B) are the **same file**.
+
+> **Verified alone is not enough.** A new run executes 277 instances and **79 of them are
+> not in Verified**: the recent `pydata/xarray` tail (30, `workloads.SWEBENCH_FULL_SPLIT_ONLY`)
+> plus all 49 de-saturation additions (`workloads.SWEBENCH_ADDED_FULL_SPLIT_ONLY` — both
+> repos' Verified rows were already spent). Build the task file from both splits and take
+> the union named by `workloads.runnable_instances("swebench")`; those two frozensets list
+> exactly which instances force the second fetch.
+
+```bash
+mkdir -p "$DATA/swebench"
+python - <<'PY'
+import json, os
+from datasets import load_dataset # pip install datasets (provisioning only)
+from tokeneconomizer import workloads
+
+wanted = {i for ids in workloads.runnable_instances("swebench").values() for i in ids} # 277
+rows, seen = [], set()
+for name in ("princeton-nlp/SWE-bench_Verified", "princeton-nlp/SWE-bench"):
+ for r in load_dataset(name, split="test"): # Verified first, then the
+ iid = r["instance_id"] # full split for the 30 extras
+ if iid in wanted and iid not in seen:
+ seen.add(iid)
+ rows.append({"instance_id": iid, "repo": r["repo"],
+ "problem_statement": r["problem_statement"]})
+ if seen >= wanted:
+ break
+missing = wanted - seen
+out = os.path.join(os.environ["DATA"], "swebench", "swebench_tasks.json")
+json.dump(rows, open(out, "w"))
+print(len(rows), "->", out, "| missing:", sorted(missing) or "none")
+PY
+export TOKENECONOMIZER_SWEBENCH_TASKS="$DATA/swebench/swebench_tasks.json"
+```
+
+**Per-instance execution images.** The official SWE-bench harness runs each instance in a
+pinned Docker image (repo + deps + test bed frozen at the issue's era). Your harness pulls
+these; they are large, so pre-pull the repos you'll run:
+
+Note the tag convention: the `__` in an instance id becomes `_1776_` in the image name, so
+`django__django-11532` lives at `swebench/sweb.eval.x86_64.django_1776_django-11532:latest`.
+`workloads.swebench_image_tag()` does that conversion, so the whole set is one loop:
+
+```bash
+python - > "$DATA/swebench/image_tags.txt" <<'PY'
+from tokeneconomizer import workloads
+for ids in workloads.runnable_instances("swebench").values():
+ for iid in ids:
+ print(workloads.swebench_image_tag(iid))
+PY
+wc -l "$DATA/swebench/image_tags.txt" # 277
+xargs -a "$DATA/swebench/image_tags.txt" -P 4 -n 1 docker pull -q
+# ...or let the SWE-bench harness build/pull them: python -m swebench.harness.run_evaluation ...
+```
+
+Budget roughly **150–200 GB** for the 277 images: they share a ~1.4 GB base plus a
+~1–3 GB environment layer per repo/era, with each instance adding only ~0.1–0.5 GB on top.
+Unauthenticated Docker Hub pulls are rate-limited to 100 per 6 h, so `docker login` first
+or expect the pull to span several windows.
+
+**Task-spec schema** (what both the driver and the adapter read):
+```json
+[{"instance_id": "pydata__xarray-1234", "repo": "pydata/xarray",
+ "problem_statement": "…issue text…"}]
+```
+The adapter redacts gold-solution URLs (PR/commit/blob links) from `problem_statement`
+automatically.
+
+**Run** (`$TOKENECONOMIZER_RESULTS` was set in the preamble):
+```bash
+AGENT=claude DOMAIN=swebench ARM=trimech \
+ REPOS="pydata/xarray django/django sympy/sympy scikit-learn/scikit-learn sphinx-doc/sphinx" \
+ SEEDS="0 1 2" RUNNER=te_harness.swebench:make_runner \
+ bash scripts/launch/trimech_ab.sh
+# Codex arms use the same task-spec, a different launcher, and one arm per command,
+# billed to OPENAI_API_KEY. Both models are evaluated, each in its own tree:
+# gpt-5.6-terra and gpt-5.6-sol.
+MODEL=gpt-5.6-terra \
+ ARM=codex-untool RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
+# Twelve arms, one per mechanism plus the naive spine, the all-three arm (and its rtk and
+# repomap variants) and all three scaffolds: baseline, baseline-medium, baseline-low,
+# repomap, rtk, serena, codex-inject, codex-untool, codex-spiral, codex-trimech,
+# codex-trimech-rtk, codex-trimech-repomap.
+# rtk on Codex is INSTRUCTION-based (read rtk_compliance); serena is wired through
+# config.toml (read serena_tool_calls). A configured scaffold is not a used one.
+# Full catalog, one command per lane: docs/RUNNING.md section 5.
+```
+
+---
+
+## 2. Web-Bench (primary — Copilot / Claude `trimech-web`)
+
+> **All three agents run here** (Codex through `CodexWebBenchRunner`). Because Web-Bench runs
+> the agent on the host with no per-task container, the Codex lane cannot use the PreToolUse
+> hook the other two are guarded by — it ignores hooks — so it carries the SWE-bench
+> `/etc/hosts` block inside an unprivileged user+mount namespace instead. Its rows report
+> `net_guard_installed: false` and `answer_key_hosts_blocked: true`; filter on the second.
+>
+> ```bash
+> DOMAIN=webbench MODEL=gpt-5.6-terra \
+> ARM=codex-untool RUNNER=te_harness.webbench:make_runner bash scripts/launch/codex_ab.sh
+> ```
+
+**Dataset + evaluator.** Web-Bench (ByteDance, `bytedance/web-bench`) is 50 Node/TypeScript
+projects, each a stream of 20 sequentially dependent tasks. Its **Node/Playwright evaluator
+owns the task loop and grading** and lives in your harness; the adapter only reads each
+project's `tasks.jsonl`.
+
+```bash
+git clone https://github.com/bytedance/web-bench.git "$DATA/web-bench"
+cd "$DATA/web-bench"
+# It is a RUSH monorepo, not an npm project: `npm install` at the root does nothing useful.
+npm i -g @microsoft/rush # or use ./common/scripts/install-run-rush.js
+rush install # installs every workspace
+rush build --to @web-bench/eval # the Node evaluator YOUR harness drives
+rush build --to @web-bench/calculator --to @web-bench/expressjs --to @web-bench/react \
+ --to @web-bench/zustand --to @web-bench/table
+# Playwright browsers must match the version the projects PIN (@playwright/test 1.57.0
+# wants browser build 1200); install from inside a project so the pinned version resolves:
+cd projects/calculator && npx playwright install chromium chromium-headless-shell && cd -
+# (B) dataset root the adapter reads:
+export TOKENECONOMIZER_WEBBENCH_PROJECTS="$DATA/web-bench/projects" # <project>/tasks.jsonl
+```
+
+> Use a **canonical** path here. The evaluator hands this root to each project's dev server
+> while node reports `process.cwd()` already symlink-resolved; reached through a symlinked
+> home (`/home/x` → `/srv/home/x`) the two spellings disagree and Vite cannot resolve the
+> `react` project's entry module — every test fails with an app that never rendered, which
+> looks exactly like an agent that wrote nothing. `readlink -f` the path before exporting it.
+
+**Check the checkout before spending anything.** Run a project's own reference solution
+through its own suite; a project that cannot pass its tests with the answer key cannot grade
+an agent:
+
+```bash
+R="$TOKENECONOMIZER_WEBBENCH_PROJECTS"
+cd "$R/calculator" && MAX_TEST_WORKERS=4 EVAL_PROJECT_ROOT="$R/calculator/src" \
+ IS_EVAL_PRODUCTION=true EVAL=true \
+ EVAL_PROJECT_PORT=3931 npm run test -- -a # expect: 68 passed
+```
+`MAX_TEST_WORKERS=4` matches the runner's own `TE_WEBBENCH_MAX_WORKERS` (default 4). Leave
+it off and Playwright uses half the CPU count of workers against one dev server; on a
+many-core host the reference solutions then fail their own suites (observed: expressjs
+88/92, react 51/59, prisma 90/91 — prisma has since been retired from the sweep) with
+nothing actually wrong.
+
+Expected totals: calculator 68 · react 59 · expressjs 92 · zustand ? · table ? (the two new
+projects are unmeasured; run the gate command above once to record them)
+(calculator measured 2026-09-07; react and expressjs carried over from the 2026-08
+provisioning). No project in the current five needs a pre-build — `react`/`zustand` are a plain `vite build`
+and `table` and `calculator`/`expressjs` build nothing. `expressjs` still fails one test if a stale `src/test.sqlite` is left behind.
+
+**Expected dataset layout (B)** — `TOKENECONOMIZER_WEBBENCH_PROJECTS/<project>/tasks.jsonl`,
+one object per line:
+```json
+{"id": "init", "date": "…", "level": "easy", "description": "…"}
+{"id": "task-1", "date": "…", "level": "easy", "description": "…"}
+```
+
+**Export the driver task list (A).** The driver needs its own JSON list of the tasks to
+stream (one entry per Web-Bench task, `repo` = the project name). It is a projection of
+`workloads.WEBBENCH_TASKS`, so a script builds it rather than a copy-pasted snippet:
+
+```bash
+export TOKENECONOMIZER_WEBBENCH_TASKS="$DATA/webbench_tasks.json" # (A) driver task list
+python scripts/export_webbench_tasks.py # write it
+python scripts/export_webbench_tasks.py --check # verify only; exit 1 if stale
+```
+
+> **Three registries have to agree, and only one of them is the authority.**
+> `tokeneconomizer/workloads.py` decides which projects and tasks are in the sweep; (A) the
+> driver task list and (C) `workloads/webbench.json` are both projections of it, and (B) the
+> projects tree has to contain a `tasks.jsonl` for every project named. Changing the project
+> list is therefore three commands, in this order:
+>
+> ```bash
+> python scripts/webbench_tasks_from_yaml.py --all # B: projects that ship only tasks.yml
+> python scripts/export_workloads.py # C
+> python scripts/export_webbench_tasks.py # A
+> ```
+>
+> A stale (A) does not fail loudly at launch — the driver streams the *old* project's tasks
+> into the *new* project's evaluator session and the harness raises
+> `session is running 'X' but the task belongs to 'Y'` partway into the run.
+>
+> Some Web-Bench projects (`zustand` among the evaluated five) ship only `tasks.yml`, while
+> `te_harness.webbench` and `WebBenchAdapter` both read `tasks.jsonl` and nothing else.
+> `webbench_tasks_from_yaml.py` derives the jsonl and **leaves the yml in place on purpose**:
+> Web-Bench's own evaluator prefers the yml when both exist, so deleting or editing one
+> without the other silently desyncs the graded task order from the driver's.
+
+
+**Run** (short streams → `trimech-web`, which adds the LLM nudge + frozen envelope). Each
+value in `REPOS` names a Web-Bench **project**; the driver selects that project's adapter and
+filters the task list to it:
+```bash
+AGENT=claude DOMAIN=webbench ARM=trimech-web SEEDS="0" \
+ REPOS="calculator expressjs react zustand table" \
+ RUNNER=te_harness.webbench:make_runner bash scripts/launch/trimech_ab.sh
+```
+
+Two things a Web-Bench harness has to get right, both because the streams are sequentially
+dependent:
+
+* **The evaluator owns the loop.** It walks a project's whole stream itself, so it must be
+ started once per stream rather than per task. The driver exposes the optional
+ `begin_stream(tasks, ...)` / `end_stream()` hooks on `AgentRunner` for exactly this; a
+ harness that does not define them is called the usual way.
+* **A failed task ends the stream.** In the evaluator's `sequential` mode the project stops
+ at the first task that fails every attempt. The driver still asks for the tasks after it,
+ and a harness should mark those rows unreached (`stream_broken` / `unreached`) with zero
+ tokens rather than letting them read as cheap failures in the cost aggregation.
+
+**Two resolve metrics, and they differ.** Each task gets up to `TE_WEBBENCH_RETRY`
+attempts (default 2), so a row carries `resolved_p1` (first attempt — Web-Bench's own
+headline metric) and `resolved_p2` (any attempt), alongside the per-attempt verdict list
+`webbench_attempt_results`. `resolved` stays pass@N because that is what the evaluator
+acts on when it decides to end a stream — and because a row's cost is the sum of every
+attempt, pass@N is the metric the cost columns actually match. The retry can rescue several tasks
+per arm, so never quote one metric as the other. Unreached tasks carry `None` for both —
+never attempted is not a miss.
+
+---
+
+## 3. CredData (secondary — security / secret detection)
+
+**Dataset.** Samsung CredData: 333 repos of obfuscated source, each `data/<repoid>/` with a
+per-repo label CSV `meta/<repoid>.csv`.
+
+```bash
+git clone https://github.com/Samsung/CredData.git "$DATA/CredData"
+cd "$DATA/CredData" && python download_data.py --data_dir data && cd - # per upstream README
+export TOKENECONOMIZER_CREDDATA_ROOT="$DATA/CredData" # (B) dir with data/ + meta/
+```
+
+**Expected dataset layout (B):**
+```
+$TOKENECONOMIZER_CREDDATA_ROOT/
+ data/<repoid>/… # obfuscated source to audit (harness copies a disposable clone)
+ meta/<repoid>.csv # line-level secret labels → graded to F1 (the quality guardrail)
+```
+Every repo gets the **same** read-only audit prompt; grading is line-level detection **F1**
+(the CredData analog of `resolved`). No Docker image — the harness runs the agent over a
+disposable copy of each repo directory.
+
+**Export the driver task list (A).** The audit prompt is fixed, so build one entry per repoid
+with that prompt as `problem_statement` (the exact prompt lives in
+`tokeneconomizer/adapters/creddata.py:CREDDATA_TASK_PROMPT`):
+```bash
+python - <<'PY'
+import glob, json, os
+from tokeneconomizer.adapters.creddata import CREDDATA_TASK_PROMPT
+from tokeneconomizer import workloads
+root = os.environ["TOKENECONOMIZER_CREDDATA_ROOT"]
+have = {os.path.basename(p)[:-4] for p in glob.glob(os.path.join(root, "meta", "*.csv"))}
+ids = [rid for rid in workloads.CREDDATA_INSTANCES if rid in have] # 100 of the 333 repos
+print("missing from checkout:", sorted(set(workloads.CREDDATA_INSTANCES) - have) or "none")
+rows = [{"instance_id": rid, "problem_statement": CREDDATA_TASK_PROMPT, "repo": rid}
+ for rid in ids]
+out = os.path.join(os.environ["DATA"], "creddata_tasks.json")
+json.dump(rows, open(out, "w"))
+print(len(rows), "->", out)
+PY
+export TOKENECONOMIZER_CREDDATA_TASKS="$DATA/creddata_tasks.json" # (A) JSON list, NOT a dir
+```
+
+**Run:**
+```bash
+AGENT=claude DOMAIN=creddata ARM=trimech SEEDS="0 1 2" \
+ RUNNER=your_pkg.harness:make_runner bash scripts/launch/trimech_ab.sh
+```
+
+---
+
+## 4. ReposVul (secondary — vulnerability detection)
+
+**Dataset.** ReposVul (ICSE'24, `Eshe0922/ReposVul`): each instance is a CVE with a fixing
+commit; you reconstruct the **pre-fix (vulnerable)** files at their real repo paths and
+grade line-level F1 against the fix-removed lines.
+
+```bash
+git clone https://github.com/Eshe0922/ReposVul.git "$DATA/ReposVul-src"
+# Prepare per-instance vulnerable-file snapshots into instances/*.json with YOUR prep step
+# (checkout parent commit of each fixing commit, capture the removed lines as labels):
+python your_prep/reposvul_prepare.py --src "$DATA/ReposVul-src" --out "$DATA/ReposVul/instances"
+export TOKENECONOMIZER_REPOSVUL_ROOT="$DATA/ReposVul" # (B) dir containing instances/
+```
+
+**Expected dataset layout (B):** `$TOKENECONOMIZER_REPOSVUL_ROOT/instances/<iid>.json` (one
+prepared instance each). Every instance gets the **same** vulnerability-audit prompt; grading
+is line-level **F1**. No Docker image.
+
+**Export the driver task list (A)** (prompt from
+`tokeneconomizer/adapters/reposvul.py:REPOSVUL_TASK_PROMPT`):
+```bash
+python - <<'PY'
+import glob, json, os
+from tokeneconomizer.adapters.reposvul import REPOSVUL_TASK_PROMPT
+from tokeneconomizer import workloads
+root = os.environ["TOKENECONOMIZER_REPOSVUL_ROOT"]
+have = {os.path.basename(p)[:-5] for p in glob.glob(os.path.join(root, "instances", "*.json"))}
+ids = [iid for iid in workloads.REPOSVUL_INSTANCES if iid in have] # 100 Python-split CVEs
+print("missing from prep:", sorted(set(workloads.REPOSVUL_INSTANCES) - have) or "none")
+rows = [{"instance_id": iid, "problem_statement": REPOSVUL_TASK_PROMPT, "repo": iid}
+ for iid in ids]
+out = os.path.join(os.environ["DATA"], "reposvul_tasks.json")
+json.dump(rows, open(out, "w"))
+print(len(rows), "->", out)
+PY
+export TOKENECONOMIZER_REPOSVUL_TASKS="$DATA/reposvul_tasks.json" # (A) JSON list, NOT a dir
+```
+
+**Run:**
+```bash
+AGENT=claude DOMAIN=reposvul ARM=trimech SEEDS="0 1 2" \
+ RUNNER=your_pkg.harness:make_runner bash scripts/launch/trimech_ab.sh
+```
+
+---
+
+## 5. CyberGym (exploratory — PoC authoring)
+
+**Dataset.** CyberGym tasks give a vulnerable workspace and ask the agent to author a
+proof-of-concept input that triggers the vulnerability. Provisioning is via a single **suite
+JSON** listing tasks; per-task workspaces are materialized by your harness.
+
+```bash
+# Fetch/prepare the suite per your CyberGym source, then:
+export TOKENECONOMIZER_CYBERGYM_SUITE="$DATA/cybergym/suite.json" # (B) adapter reads this
+```
+
+CyberGym has **no `trimech_ab.sh` domain case** — the launchers cover swebench / webbench /
+creddata / reposvul only. Run it by calling the driver directly, supplying the driver task
+list (A) yourself and `--adapter cybergym`:
+
+```bash
+# (A) a JSON list of {instance_id, problem_statement, repo}; problem_statement can be the
+# fixed CYBERGYM_TASK_PROMPT (see tokeneconomizer/adapters/cybergym.py).
+python scripts/run_experiment.py \
+ --arm trimech --agent claude --adapter cybergym \
+ --tasks-file "$DATA/cybergym_tasks.json" --repo "$PROJECT" --seed 0 \
+ --results-root "$TOKENECONOMIZER_RESULTS/claude_cybergym" \
+ --runner your_pkg.harness:make_runner
+python scripts/aggregate.py "$TOKENECONOMIZER_RESULTS/claude_cybergym" --arms baseline trimech
+```
+The adapter reads the suite JSON to key the per-task judge; PoC building + triggering live in
+the harness.
+
+---
+
+## 6. WebArena — not implemented
+
+`WebArenaAdapter` is a **stub**: every method raises `NotImplementedError` naming the exact
+surface a WebArena integration must supply (episode→trajectory parser, per-action usage
+counter, action-space catalog, waste detector, protected/inapplicable sets). It documents the
+contract; there is nothing to provision until it is implemented.
+
+---
+
+## Offline check (no images, no datasets)
+
+Before wiring any real harness, confirm the whole pipeline runs end-to-end with the built-in
+`StubRunner` — it launches no agent, needs no image or dataset, and cleans up after itself:
+
+```bash
+bash scripts/launch/smoke_test.sh # exercises every arm; asserts CSVs; deletes its temp dir
+```
+
+Its numbers are synthetic (a wiring check, not a benchmark). Real results require your
+`AgentRunner`.
diff --git a/experiments/docs/COPILOT.md b/experiments/docs/COPILOT.md
new file mode 100644
index 0000000..9d496eb
--- /dev/null
+++ b/experiments/docs/COPILOT.md
@@ -0,0 +1,312 @@
+# Running the GitHub Copilot CLI as the sealed agent
+
+TokenEconomizer's mechanisms are agent-agnostic, but each sealed agent needs its own runner
+below the [`AgentRunner`](../../tokeneconomizer/runner.py) seam. This page is the Copilot path,
+end to end, for **both** SWE-bench and Web-Bench. It assumes you have already done
+[`SETUP.md`](SETUP.md) steps 1–3 (venv, the two outside-the-repo directories, datasets);
+everything Copilot-specific is here.
+
+Read [`SETUP.md`](SETUP.md) first if you have not. The gates below are cumulative — do not
+skip one because the next looks more interesting.
+
+---
+
+## What is different about Copilot
+
+Four things, and each one has bitten this harness at least once:
+
+| | Claude Code | Copilot CLI |
+|---|---|---|
+| **Auth** | `ANTHROPIC_API_KEY` | GitHub, from on-disk `~/.copilot` or `COPILOT_GITHUB_TOKEN`; or Copilot's own BYOK |
+| **Token accounting** | in the stream's terminal `result` event | **only** in the CLI's debug log |
+| **Turn cap** | `--max-turns N` | none — wall clock and the spiral guard are the only bounds |
+| **Tool exclusion** | one comma-joined `--disallowed-tools` | `--excluded-tools` repeated per tool |
+
+The accounting difference is the dangerous one. Copilot's `result` event carries
+`premiumRequests` and durations but **no token fields at all**, so a runner that reads usage
+the Claude way records a row of zeros — every cost number downstream is then wrong rather
+than missing. That is why every Copilot invocation here runs with `--log-level all
+--log-dir …` and mines the log
+([`te_harness/copilot.py`](../te_harness/copilot.py)).
+
+---
+
+## Step 1 — install and sign in
+
+```bash
+curl -fsSL https://gh.io/copilot-install | bash # -> ~/.local/bin/copilot
+copilot # then: /login
+```
+
+`/login` runs an OAuth device flow and stores the token in your system credential store, or
+in plain text under `~/.copilot/` if there is no credential store. For a headless box,
+export a token instead — the CLI checks `COPILOT_GITHUB_TOKEN`, then `GH_TOKEN`, then
+`GITHUB_TOKEN`. Fine-grained (v2) PATs need the **Copilot Requests** permission; classic
+`ghp_` tokens are not supported.
+
+**Gate:**
+
+```bash
+copilot -p "say ok" --allow-all-tools --no-color --log-level none
+```
+
+Expect `ok`. If you get `No authentication information found` on a box that has been idle,
+**run it a second time** — see the token-staleness note in Step 5; the first call refreshes
+the token as a side effect.
+
+### BYOK — Copilot's own provider keys
+
+Without a subscription, or with its quota spent, the CLI can bill **your own provider**
+instead (`copilot help providers`). `COPILOT_PROVIDER_BASE_URL` is the one switch — set it
+and GitHub auth is not used at all:
+
+```bash
+COPILOT_PROVIDER_TYPE=anthropic
+COPILOT_PROVIDER_BASE_URL=https://api.anthropic.com
+COPILOT_PROVIDER_API_KEY=<your Anthropic key> # the CLI never reads ANTHROPIC_API_KEY
+```
+
+The launcher then prints `routing: Copilot CLI BYOK -> anthropic at …` and skips the GitHub
+auth probe. With the base URL set and neither `COPILOT_PROVIDER_API_KEY` nor
+`COPILOT_PROVIDER_BEARER_TOKEN`, it warns, and every task would fail — read the banner.
+
+* **It is still the Copilot CLI.** BYOK keeps the agent `copilot`, its own trajectory shape,
+ tool names and `copilot_<domain>_<tag>` trees; it is not Claude Code on the same key.
+* **`--effort` is accepted under BYOK**, so the effort arms run.
+* **`--model` keeps the first-party id** on the Anthropic wire (`claude-opus-4-8`, dashed).
+ `COPILOT_PROVIDER_MODEL_ID` / `COPILOT_PROVIDER_WIRE_MODEL` are for a provider whose served
+ id differs from the one you want recorded (a router that prefixes `anthropic/`, for
+ example); leave them unset otherwise.
+* **A BYOK row and a subscription row are different cost bases, and the tree name does not
+ separate them.** The discriminator is `run_config.json` → `routing.base_url`: the provider
+ URL under BYOK, the literal `github-copilot` on a subscription. Never pool the two; if you
+ switch mid-sweep, move the finished cells to their own tree first.
+
+The harness forwards the provider variables into the SWE-bench task container and passes
+the credential by name only, so it never reaches argv; Web-Bench's agent runs on the host and
+inherits them directly.
+
+## Step 2 — pick a model, and know what that costs you
+
+Copilot serves its own catalog, which depends on your subscription. An Anthropic model id is
+not a valid value here. Check what yours serves before a sweep — a bad id fails **per task**,
+not at launch:
+
+```bash
+copilot -p "say ok" --model <the-id-you-want> --effort high --no-color --log-level none
+```
+
+This is the same probe the launcher runs, so its verdict is the one you will get:
+
+* `ok` — the model is served **and** takes your effort level. Pin it.
+* `Model "<id>" … is not available.` — pick another id.
+* `Model "<id>" does not support reasoning effort configuration` — the model is served but
+ has no effort knob. Usable, but effort will not be applied (see below).
+
+If nothing is available, use `--model auto` and let the CLI route.
+
+### Reasoning effort
+
+**If you want `TE_EFFORT_LEVEL=high` to actually take effect, you must pin a model.** The
+`auto` router rejects the flag outright — `--model auto --effort high` exits with *"Model
+auto does not support reasoning effort configuration"* and the task never runs. Some pinned
+models refuse it too.
+
+The launcher settles this for you with one probe before the sweep starts
+(`te_check_copilot_model`), so neither mistake can cost you a run:
+
+| Probe result | What happens |
+|---|---|
+| model served, effort accepted | `[env] copilot model '<id>' serves; effort=high accepted` — runs with effort, row records it |
+| model served, effort refused | **warns**, exports `TE_EFFORT_APPLIES=0`, runs without the flag, row records `effort: null` |
+| model not served | **hard fail, exit 2** — nothing could have run |
+
+**A pinned model must also be priced.** `pricing.MODEL_WEIGHTS` refuses to guess ratios for
+an id it does not know — deliberately, so a new model can never be silently mispriced. The
+runner now checks this at construction, so an unpriced id fails at startup:
+
+```
+ValueError: pricing: no weight entry for billed model 'my-model' (agent='copilot').
+ Add it to pricing.MODEL_WEIGHTS
+```
+
+Add the id to `tokeneconomizer/vendor/pricing.py` before pinning it. Ids containing
+`claude` fall back to the Claude curve automatically and need no entry. `auto` needs none
+either — it is a routing directive, not a billed id, so it is recorded as `null` and priced
+on the Claude curve, which is the historical behaviour for Copilot rows.
+
+Set both in the repo's `.env`:
+
+```bash
+COPILOT_MODEL=gpt-5.6-sol # a model id YOUR subscription serves; 'auto' = router, no effort
+TE_EFFORT_LEVEL=high # takes effect only when COPILOT_MODEL is a pinned id
+```
+
+> Where effort is not applied, the row's `effort` is `null` rather than `high`, and the
+> reasoning effort is whatever the model or router defaulted to. Those rows stay internally
+> comparable, but must **not** be compared against effort-pinned Claude numbers as though
+> effort were held equal. Check `effort` in `run_config.json` before you compare.
+
+If you bypass the launcher and call `scripts/run_experiment.py` directly, there is no probe —
+but the runner still fails fast: a refused model or effort raises on the first segment rather
+than writing a tree of 0-turn rows that look like genuine task failures.
+
+## Step 3 — point the driver at the Copilot runner
+
+The runner factories are agent-aware: the driver passes `--agent` to them, and an unknown
+label raises rather than falling back to Claude. Nothing to configure beyond
+`AGENT=copilot`.
+
+```bash
+# in .env
+RUNNER=te_harness.swebench:make_runner # or te_harness.webbench:make_runner
+```
+
+**Gate** — the label really does select the Copilot runner:
+
+```bash
+python3 - <<'EOF'
+import sys; sys.path.insert(0, "scripts")
+from run_experiment import load_runner
+r = load_runner("te_harness.swebench:make_runner", "copilot")
+print(type(r).__name__, r.agent_bin)
+EOF
+```
+
+Expect `CopilotSwebenchRunner /…/bin/copilot`. If you see `ClaudeSwebenchRunner`, the label
+is not reaching the factory and **every row would be Claude's work filed under Copilot's
+name** — stop and fix that before spending anything.
+
+## Step 4 — launch
+
+Same launcher as Claude; only `AGENT` changes.
+
+```bash
+# SWE-bench
+AGENT=copilot DOMAIN=swebench ARM=trimech REPOS="pydata/xarray" SEEDS="0" \
+RUNNER=te_harness.swebench:make_runner \
+ bash scripts/launch/trimech_ab.sh
+
+# Web-Bench (short streams -> the tuned arm)
+AGENT=copilot DOMAIN=webbench ARM=trimech-web REPOS="calculator" SEEDS="0" \
+RUNNER=te_harness.webbench:make_runner \
+ bash scripts/launch/trimech_ab.sh
+```
+
+The preflight prints which auth path it resolved and warms the token:
+
+```
+[env] routing: native Copilot (on-disk GitHub auth, token warm)
+[trimech_ab] agent=copilot routing=direct domain=swebench arms=(baseline trimech) ...
+```
+
+Results land in `copilot_swebench/` and `copilot_webbench/`, separate from the Claude trees —
+a different agent is a different cost basis and the two must never average into one cell.
+
+## Step 5 — keep the token warm for a long sweep
+
+Copilot's token is short-lived and is refreshed into `~/.copilot/config.json` **only when a
+copilot process runs on the host**. SWE-bench containers authenticate from a read-only mount
+of that file, so during a long unattended sweep the token goes stale (~75 min was the
+observed lifetime) and every later container copies the dead token and fast-fails —
+*"No model backend available"*, a 0-byte patch, ~30s wall. Hundreds of paid-looking failures
+that are really one expired credential.
+
+Run the keepalive for the duration of any sweep longer than about an hour:
+
+```bash
+/path/to/AgentEO/swebench/copilot_auth_keepalive.sh 600 & # refresh every 10 min
+# ... run the sweep ...
+kill %1
+```
+
+Setting `COPILOT_GITHUB_TOKEN` sidesteps this entirely — the token is passed into the
+container by environment and never goes stale mid-run. Prefer that for unattended work.
+
+---
+
+## Verifying a finished run
+
+Three checks, in order of how quietly they fail:
+
+```bash
+RUN=$(ls -d <results>/copilot_swebench/baseline/*/2*__copilot__* | head -1)
+
+# 1. the debug log exists -> token accounting and the untooling catalog both work
+ls -la "$RUN/copilot_debug.log"
+
+# 2. the row's tokens came from the log, not the stream fallback
+python3 -c "import json;r=json.load(open('$RUN/row.json'));print(r['tokens_source'], r['input_tokens'], r['cache_read_input_tokens'])"
+
+# 3. the tool catalog is non-empty (an empty one makes untooling a paid no-op)
+python3 -c "
+from tokeneconomizer import trace
+c = trace.parse_catalog('$RUN', 'copilot')
+print(len(c), 'tools;', sum(1 for v in c.values() if v), 'with descriptions')"
+```
+
+`tokens_source` must be `debug_log`. `stream_output_only` means the log was not recovered:
+the row then has output tokens and nothing else, so its cost is a large underestimate rather
+than a gap you would notice in a summary. `mixed` means one spiral segment was accounted and
+another was not.
+
+Check 3 should report ~17 tools. **`0 with descriptions` is expected on a current CLI** and
+is not a failure — see the next section for what it costs you.
+
+### Tool descriptions and which untooling flag to use
+
+Older Copilot builds dumped the full tool **schemas** into the debug log, and
+`reflect.parse_tool_catalog` read `{name: description}` straight out of it. Current builds
+(verified on 1.0.82) log only the tool *calls*; the loaded tool **names** moved to the
+stream's `session.usage_checkpoint` event. `trace.parse_catalog` tries the log first and
+falls back to those names with empty descriptions, so you get names on every version.
+
+That distinction decides which mechanism to run:
+
+| Flag | Needs | On a current CLI |
+|---|---|---|
+| `--untool-evidence N` | names only (drop what was loaded but unused for N tasks) | **works** |
+| `--untool-judge` | descriptions (reason about what each tool is *for*) | **degrades to a paid no-op** — it drops nothing and still bills for the call |
+
+So on a current CLI prefer `--untool-evidence` alone, and treat a `trimech` run that
+includes `--untool-judge` as paying for a judge that cannot contribute. Restoring the judge
+means capturing a description snapshot the way `vendor/claude_tool_catalog.json` does for
+Claude (see `te_harness/capture_catalog.py` for the pattern).
+
+## Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `no copilot binary found` | Not installed, or installed somewhere else — set `COPILOT_HOST_BIN`. |
+| `copilot host install incomplete: …/.cache/copilot` | The package cache is missing. Run `copilot` once interactively to populate it; docker would otherwise invent an empty dir and every task would fail. |
+| Every task: 0 turns, empty patch, ~30s | Stale or missing auth in the container. See Step 5. |
+| `Model "auto" does not support reasoning effort` | You pinned effort against the router. The launcher probe and the runner both guard this; if you see it raw, you built the argv yourself. |
+| `copilot refused this configuration: …` (run aborts) | `COPILOT_MODEL` is not served, or refuses `--effort`. Fix it per Step 2 — the run stopped deliberately rather than logging hundreds of 0-turn failures. |
+| `tokens_source: stream_output_only` on every row | `--log-level all --log-dir` was dropped, or the log dir was unreadable. |
+| Untooling drops nothing but still bills | Expected for `--untool-judge` on a current CLI (no descriptions). If `--untool-evidence` also drops nothing, check the catalog is non-empty (check 3). |
+| Rows say `copilot` but look like Claude | The runner factory ignored `--agent`. Re-run the Step-3 gate. |
+
+## Known limitations
+
+* **No turn cap.** `max_turns` is recorded in `run_config.json` and the spiral envelope is
+ counted in turns, but nothing enforces a cap on the agent. `TE_TASK_TIMEOUT` and the
+ spiral guard are the real bounds. A Copilot baseline is therefore not turn-capped the way
+ a Claude baseline is.
+* **No vendor USD.** Copilot reports a premium-request count, not a dollar figure, so
+ `cost_usd_vendor` stays null. Cost comparisons run on effective tokens, which is the
+ basis the study uses anyway. `aggregate.py` knows this: its `vendor_cost_missing` flag
+ skips any agent in `AGENTS_WITHOUT_VENDOR_USD` (`copilot`, `codex`), so a Copilot tree
+ reports zero of them rather than flagging every row.
+* **Effort on `auto`** — see Step 2.
+* **No tool descriptions** on current CLI builds, so the instruction-aware tool judge has
+ nothing to reason over. Evidence untooling is unaffected.
+
+## Verified
+
+A `baseline` task on each domain has been run end to end on the subscription: each resolved
+by that benchmark's own official grader (the SWE-bench harness and Web-Bench's Node
+evaluator), with `tokens_source=debug_log` and the full cache-aware token breakdown
+recovered from the debug log — a real cost basis rather than a row of zeros.
+
+Check the same three things on your first row (see "Verifying a finished run" above) before
+launching a lane.
diff --git a/experiments/docs/ENVIRONMENT.md b/experiments/docs/ENVIRONMENT.md
new file mode 100644
index 0000000..541482d
--- /dev/null
+++ b/experiments/docs/ENVIRONMENT.md
@@ -0,0 +1,165 @@
+# Environment & requirements
+
+Everything you need on a **fresh clone**, split into two independent layers. The layers do
+**not** overlap: the repo layer is tiny and pure-Python; the harness layer (Docker, agent
+CLIs, datasets) is external infrastructure you provision only when you want to reproduce the
+real cost numbers.
+
+```
+┌─ Layer 1: THIS REPO ────────────────────────────────────────────────┐
+│ Python >= 3.10. Core = stdlib-only (zero deps). │
+│ + pytest (tests). requirements.txt. │
+│ Lets you: import the package, run the test suite, run the offline │
+│ smoke test. NO Docker, NO datasets. │
+└─────────────────────────────────────────────────────────────────────┘
+┌─ Layer 2: THE HARNESS (te_harness/, + what it drives) ──────────────┐
+│ Docker + the sealed agent CLIs + dataset checkouts. The runner │
+│ itself is in the repo; none of what it drives can be. Lets you: │
+│ run each benchmark for real and measure cost and resolve. │
+│ See docs/HARNESS.md and docs/BENCHMARKS.md. │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+## Layer 1 — the repo (always required)
+
+| Requirement | Version | Why |
+|---|---|---|
+| Python | ≥ 3.10 | the package + scripts |
+| pip / venv | any recent | install the package |
+| pytest | ≥ 7.0 | test suite — optional |
+
+Core `import tokeneconomizer` needs **nothing beyond the standard library**. `pytest` is
+only for the tests.
+
+**Install (recommended — editable, with the test extra):**
+```bash
+python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
+pip install -e '..[test]' # the repository root's pyproject.toml
+```
+
+**Or with the convenience requirements file** (same pins, no build step):
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+pip install -e .. # still install the package itself (editable)
+```
+
+**Verify Layer 1 with zero external infra:**
+```bash
+python -c "import tokeneconomizer as te; print(te.Economizer)" # self-contained import
+pytest # offline suite, no Docker/datasets
+bash scripts/launch/smoke_test.sh # every arm wired end-to-end, auto-cleans
+```
+If those three pass, the repo layer is healthy. **A fresh user can do all of this with no
+Docker, no datasets, and no API keys.**
+
+## Layer 2 — the harness (only for real runs)
+
+> This section is the **what**. For the **how**, in order, with a gate after every step, see
+> [`SETUP.md`](SETUP.md) — venv, the data directory, credentials, the runner, then
+> per-benchmark data.
+
+A real run launches a **sealed agent inside a benchmark Docker image** and grades it against
+a gold patch. The code that does this is in the repo
+([`HARNESS.md`](HARNESS.md)); everything it *drives* is not, and cannot be. To reproduce for
+real you supply:
+
+1. **Docker** — to run the per-benchmark execution images (see install guide below).
+2. **A sealed agent CLI** — Claude Code, GitHub Copilot CLI, or Codex, on its own native
+ authentication: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or a Copilot subscription or
+ Copilot BYOK. Keys go in `.env` (see [`../.env.example`](../.env.example)); the per-agent
+ table is in [`RUNNING.md`](RUNNING.md).
+3. **Dataset checkouts** — per [`docs/BENCHMARKS.md`](BENCHMARKS.md) (SWE-bench, Web-Bench,
+ CredData, ReposVul, CyberGym).
+4. **Nothing else.** The `AgentRunner` implementation — the glue that launches the agent in
+ the image, applies the economizer's per-task decisions, grades, and writes `row.json` +
+ `transcript.jsonl` — ships in the repo as `te_harness`, so this is a flag rather than a
+ deliverable: `RUNNER=te_harness.swebench:make_runner`. Pass
+ `RUNNER=your_pkg.harness:make_runner` to substitute your own.
+
+Some dataset **provisioning** steps need extra tools that are **not** repo dependencies
+(don't put them in `requirements.txt`):
+
+| Tool | Needed for | Install |
+|---|---|---|
+| `git` | cloning Web-Bench / CredData / ReposVul | your OS package manager |
+| `datasets` (HF) | fetching SWE-bench Verified instances | `pip install datasets` (provisioning only) |
+| Node.js + npm | Web-Bench evaluator + Playwright | nodejs.org / `nvm` |
+
+---
+
+## Installing & updating Docker (for beginners)
+
+You only need this for **Layer 2**. Skip it entirely if you're just running the repo layer
+(import / tests / smoke test).
+
+### Check what you already have
+```bash
+docker --version # e.g. "Docker version 27.x"
+docker run --rm hello-world # prints a success message if the daemon is working
+```
+If `docker` is missing or `hello-world` fails, install/fix it below.
+
+### Linux (Ubuntu/Debian) — Docker Engine
+```bash
+# 1. Remove any old distro packages
+sudo apt-get remove -y docker docker-engine docker.io containerd runc 2>/dev/null || true
+
+# 2. Add Docker's official apt repository
+sudo apt-get update
+sudo apt-get install -y ca-certificates curl gnupg
+sudo install -m 0755 -d /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
+ sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+sudo chmod a+r /etc/apt/keyrings/docker.gpg
+echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+ https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
+ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+
+# 3. Install
+sudo apt-get update
+sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
+ docker-buildx-plugin docker-compose-plugin
+
+# 4. Run docker without sudo (log out/in afterward)
+sudo usermod -aG docker "$USER"
+```
+
+### macOS / Windows — Docker Desktop
+- **macOS:** install [Docker Desktop](https://docs.docker.com/desktop/install/mac-install/)
+ (Apple-Silicon or Intel build), or `brew install --cask docker`, then launch Docker.app.
+- **Windows:** install [Docker Desktop](https://docs.docker.com/desktop/install/windows-install/)
+ with the **WSL 2** backend, and run the launchers from inside a WSL 2 Ubuntu shell.
+
+### Updating Docker
+```bash
+# Linux (Docker Engine): just re-pull the packages
+sudo apt-get update && sudo apt-get install -y \
+ docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+# macOS / Windows (Docker Desktop): Docker menu → "Check for Updates…", or reinstall the cask
+brew upgrade --cask docker
+```
+
+### Common first-run fixes
+- **`permission denied … /var/run/docker.sock`** — you're not in the `docker` group yet: run
+ the `usermod -aG docker` line above, then log out and back in (or `newgrp docker`).
+- **`Cannot connect to the Docker daemon`** — the daemon isn't running:
+ `sudo systemctl start docker` (Linux) or launch Docker Desktop (macOS/Windows).
+- **Benchmark images are large** — pre-pull the instances you'll run and give Docker enough
+ disk (SWE-bench per-instance images are multi-GB each).
+
+---
+
+## What a fresh user can and cannot reproduce
+
+| Goal | Needs | Doable on a bare clone? |
+|---|---|---|
+| Import the package | Layer 1 | ✅ yes |
+| Run the test suite | Layer 1 | ✅ yes |
+| Run the offline smoke test (all arms wired) | Layer 1 | ✅ yes — synthetic numbers |
+| Measure cost and resolve per arm | Layer 2 (Docker + agent CLI + credentials + datasets) | ⚠️ yes, once Layer 2 is provisioned |
+
+The boundary: **the repo exercises the *method and wiring* out of the box; real numbers need
+Docker, an agent CLI with its credentials, and the datasets**, which are machine- and
+account-specific and therefore not shipped here.
diff --git a/experiments/docs/HARNESS.md b/experiments/docs/HARNESS.md
new file mode 100644
index 0000000..ce2f173
--- /dev/null
+++ b/experiments/docs/HARNESS.md
@@ -0,0 +1,546 @@
+# te_harness — Layer-2 task-execution harness for TokenEconomizer
+
+Everything that touches Docker, an agent CLI, or a credential lives **below**
+`tokeneconomizer.runner.AgentRunner`. [`te_harness/`](../te_harness) is this repo's
+implementation of that seam. Supplying your own instead is a supported path, not a
+fallback: pass `--runner your_pkg.harness:make_runner` and nothing above the seam changes.
+
+**It is a sibling package of `tokeneconomizer`, not a submodule of it**, and `pyproject.toml`
+excludes it from the distributed wheel. That boundary is the point: `import tokeneconomizer`
+cannot reach Docker or an agent CLI because the code that shells out to them is not
+underneath it, so the core's stdlib-only guarantee is structural rather than a convention.
+It costs nothing at install time either — this package is stdlib-only at import time too
+(the official SWE-bench grader is imported lazily inside `swebench._import_swebench`, from
+the checkout `TE_SWEBENCH_REPO` names), which is why the `harness` extra installs no
+packages. What it needs cannot come from pip: Docker, an agent CLI on `PATH`, an API key,
+the benchmark images, and for Web-Bench a built Rush monorepo.
+
+Because the harness is versioned alongside the mechanisms, a result is attributable to a
+harness commit and not merely to "whatever was on disk that day". Anything that changes
+recorded cost, `resolved`, or turn counts is a reviewable diff — which matters more than it
+sounds, since the accounting these rows are built from lives here, not above the seam.
+
+It holds one runner per domain, each executing tasks with a **real sealed coding agent** and
+grading with **that benchmark's own official grader**, returning the row + transcript the
+economizer's mechanisms consume:
+
+| Runner | Domain | Agent runs | Graded by |
+|---|---|---|---|
+| `te_harness.swebench:make_runner` | SWE-bench | inside the instance's official Docker image | the official SWE-bench harness (`get_eval_report`) |
+| `te_harness.webbench:make_runner` | Web-Bench | on the host, over the task's file set | Web-Bench's own Node evaluator + Playwright suites |
+| `te_harness.webbench_smoke:make_runner` | Web-Bench | **not at all** — the reference solution stands in | same as above (offline wiring check) |
+
+## Agents
+
+Both real factories take the driver's `--agent` label and return that agent's runner. Each
+agent runs on its own native authentication: Claude Code on `ANTHROPIC_API_KEY`, Codex on
+`OPENAI_API_KEY`, the Copilot CLI on its GitHub subscription or its own BYOK.
+
+| `--agent` | SWE-bench | Web-Bench |
+|---|---|---|
+| `claude` (default) | `ClaudeSwebenchRunner` | `WebBenchRunner` |
+| `copilot` | `CopilotSwebenchRunner` | `CopilotWebBenchRunner` |
+| `codex` | `CodexSwebenchRunner` | `CodexWebBenchRunner` |
+
+An unknown label **raises**. It must never fall back: a factory that quietly returned the
+Claude runner for `--agent copilot` would produce a full tree of Claude rows filed under
+Copilot's name — plausible numbers for an agent that never ran, which is worse than a crash.
+
+Everything Copilot-specific — GitHub auth instead of an API key, its own model catalog,
+token recovery from the debug log, and the absence of a turn cap — lives in
+[`te_harness/copilot.py`](../te_harness/copilot.py); setup is in [`COPILOT.md`](COPILOT.md).
+The Codex seam — the capability→config mapping and its own accounting — is
+[`te_harness/codex.py`](../te_harness/codex.py). The two Copilot runners
+subclass the Claude ones and override only construction, launch, and accounting; the task
+lifecycle (container, spiral segments, grading, evaluator round-trip) is shared.
+
+## Use
+
+```bash
+cd /path/to/TokenEconomizer
+set -a; . ./.env; set +a # ANTHROPIC_API_KEY etc.
+
+python3 scripts/run_experiment.py \
+ --arm baseline --agent claude --repo pydata/xarray --seed 0 --limit 1 \
+ --adapter swebench --tasks-file "$TOKENECONOMIZER_SWEBENCH_TASKS" \
+ --workload workloads/swebench.json \
+ --results-root "$TOKENECONOMIZER_RESULTS/claude_swebench" \
+ --runner te_harness.swebench:make_runner --fresh
+```
+
+The launchers take the same seam: `RUNNER=te_harness.swebench:make_runner bash
+scripts/launch/trimech_ab.sh`.
+
+No `PYTHONPATH` is needed for either form. `scripts/run_experiment.py` puts `experiments/`
+and the repository root (the shared core) on `sys.path` itself, so `--runner te_harness.…` resolves from any working directory. (While the
+harness lived outside the repo this required exporting `PYTHONPATH`, and forgetting it failed
+a launch at import time — or, worse, appeared to work by namespace-package accident whenever
+the shell happened to be sitting in the harness's parent directory.)
+
+## What one task does
+
+1. **Start** `swebench/sweb.eval.x86_64.<instance>:latest` detached, with the host's
+ Claude Code binary bind-mounted in. It is a static ELF, so it runs in the image
+ unmodified — no image rebuild, no Node, no network install.
+2. **Run the agent** at `/testbed` via `claude -p --output-format stream-json`, applying the
+ economizer's per-task decisions: the injected manual prefix goes in the prompt, the
+ untooling mask becomes `--disallowed-tools`, and the spiral guard folds the live stream
+ through `NudgeGuard.monitor`. Every raw stream line is written to `transcript.jsonl` —
+ the exact schema `tokeneconomizer.trace` re-reads afterwards.
+3. **Spiral rung 1**: on a guard nudge, the agent is re-invoked in the *same* container on
+ top of its own on-disk edits, with the leakage-safe redirect appended. On an abort, the
+ agent is killed inside the container (killing the `docker exec` client alone would leave
+ it running and billing).
+4. **Grade** with `swebench.harness.test_spec` + `.grading` — the same `get_eval_report`
+ the official `run_evaluation` calls — running the instance's real eval script in the
+ same container. The eval script resets test files and applies the instance's own
+ `test_patch`, so agent edits to tests are discarded exactly as upstream discards them.
+5. **Return** `(row, run_dir)` with the cache-aware token breakdown from the CLI's final
+ `result` event, `resolved`, `num_turns`, and the spiral/untool telemetry the mechanisms
+ read. When the agent was killed before that event, the breakdown is recovered from the
+ transcript and the row is flagged — see **Accounting** below.
+
+### Accounting: rows that are a lower bound
+
+Claude Code reports its token breakdown **only** in the stream's terminal
+`{"type":"result"}` event, and both of `_segment`'s break paths — a spiral guard stop and
+`TE_TASK_TIMEOUT` — kill the CLI before it arrives. The segment used to return `{}`, so the
+row recorded **zero** for every token field and for `num_turns`, describing a rollout that
+had really run for tens of turns and hundreds of thousands of tokens as free.
+
+Only economized arms install a guard, so the understatement was **one-directional**: it
+flattered exactly the arms under test and could never touch a baseline. Grading was
+unaffected (it reads the patch), but cost and turn-based claims were not.
+
+`_UsageRecovery` now folds usage in as the stream arrives and rebuilds a `result`-shaped
+event when none comes. What that can and cannot recover:
+
+| Field | Recoverable? | How |
+|---|---|---|
+| `input_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens` | **exactly** | per-message `usage` snapshots deduplicated on `message.id` (they are cumulative and repeat per content block, so summing every event double-counts). Validated against 60 healthy rows that *do* have a `result` event: 60/60 exact on all three |
+| `num_turns` | approximately | deduplicated assistant-message count — exact on 107 of 120 healthy rows, within 1 on 12 more |
+| `output_tokens` | **no** | the per-message figure is the `message_start` placeholder, understating the truth 7.5×–134.7× (median 28×). Reported as 0 and flagged, never estimated |
+
+So a recovered row is honest but incomplete and says so: `usage_incomplete` with
+`output_tokens_missing`, `num_turns_recovered`, and `usage_incomplete_reason`. The flags are
+present-and-`False` on a healthy row, so an absent flag means "written before this check
+existed", not "verified complete". They OR across spiral segments, because degradation is a
+property of the sum — the usual shape is a first segment that completes and a second the
+guard kills.
+
+**A flagged row's cost is a floor, not a measurement.** `pricing.effective_tokens` is
+deliberately left alone: this reports the defect rather than papering over it.
+`scripts/aggregate.py` surfaces `usage_incomplete` in `all_rows.csv`, counts it per cell as
+`n_usage_incomplete` in `summary_by_cell.csv`, and prints a warning naming the affected arms
+— so a distorted mean cannot be read without the distortion beside it. It also recognises
+rows stored *before* the fix by their signature (no tokens, yet the task resolved or left a
+patch) and flags them `legacy_zero_usage`.
+
+Copilot was never affected: it mines its debug log after the kill (`cat {logdir}/*.log`),
+which is why the audit found zero affected rows in 2,571. Its own degraded case is a log it
+could not parse, which `copilot.row_usage` already reports as
+`tokens_source="stream_output_only"` with `None` — not 0 — in the unaccounted fields. Both
+Copilot runners map that onto the same `usage_incomplete` flag, so one filter covers every
+row this harness writes.
+
+Each run dir therefore holds:
+
+| File | What it is |
+|---|---|
+| `transcript.jsonl` | raw `stream-json` events, the schema `tokeneconomizer.trace` re-reads |
+| `run_config.json` | **everything that shaped the run** — effort, model, turn cap, CLI build, routing mode, the tool mask actually applied, spiral params, injected-manual hash |
+| `model_patch.diff` | the agent's working-tree diff, i.e. what was graded |
+| `eval.log` | the official eval script's output, i.e. why it graded that way |
+| `row.json` | the normalized row (written by the driver, not the harness) |
+
+`run_config.json` exists because the CLI records **neither** the effort level nor the tool
+mask in its own transcript — without it, a finished run cannot answer "what settings
+produced this?", and an effort default that silently changed would be undetectable after
+the fact.
+
+Root note: SWE-bench images run as root, where the CLI refuses
+`--dangerously-skip-permissions` unless `IS_SANDBOX=1` says it is containerized. These
+disposable per-task containers are exactly that, so the harness sets it.
+
+## Web-Bench
+
+```bash
+cd /path/to/TokenEconomizer
+set -a; . ./.env; set +a
+
+python3 scripts/run_experiment.py \
+ --arm baseline --agent claude --repo calculator --seed 0 \
+ --adapter webbench --tasks-file "$TOKENECONOMIZER_WEBBENCH_TASKS" \
+ --workload workloads/webbench.json \
+ --results-root "$TOKENECONOMIZER_RESULTS/claude_webbench" \
+ --runner te_harness.webbench:make_runner --fresh
+```
+
+Or a full A/B over the five evaluated projects (`trimech-web` is the arm tuned for
+Web-Bench's short streams):
+
+```bash
+AGENT=claude DOMAIN=webbench ARM=trimech-web \
+ REPOS="calculator expressjs react zustand table" SEEDS="0" \
+ RUNNER=te_harness.webbench:make_runner \
+ bash scripts/launch/trimech_ab.sh
+```
+
+`--repo` names the Web-Bench **project**, which is one stream of 20–21 sequentially
+dependent tasks (`init`, `task-1` … `task-20`). One driver invocation runs one project.
+
+### Why this one is different
+
+Web-Bench inverts the control flow. Its Node evaluator owns the task loop — copy the
+previous task's workspace forward, collect the file set, ask the agent for files, write them
+back, run the project's build, screenshot, then grade with the project's Playwright suite
+**cumulatively** (`init` + `task-1..task-N`, so a task passes only if every earlier task
+still passes). The economizer's driver owns *its* loop and expects one `run_task` per task.
+
+The runner joins the two at the evaluator's own HTTP agent seam, so neither side is
+reimplemented. Per project it starts a serialized HTTP server speaking the evaluator's
+agent contract and `@web-bench/eval` restricted to that project, pointed at that server.
+`run_task` then blocks on a rendezvous: take the evaluator's request for this task, run
+Claude Code over the request's files in a scratch workspace, answer with the files that
+changed, and wait for the evaluator's verdict line for this task before returning the row.
+`resolved` is Web-Bench's own graded verdict.
+
+The driver reaches this through two optional `AgentRunner` hooks — `begin_stream(tasks)` /
+`end_stream()` — because the evaluator has to be started once for the whole stream, not
+per task. Harnesses that do not define them are unaffected.
+
+Two consequences of Web-Bench's contract shape what the mechanisms can observe, and neither
+is a harness limitation:
+
+* **The agent gets files, not an installed project.** Web-Bench's own reference agent is a
+ single files-in/files-out LLM call; the request carries no `node_modules` and no test
+ suite (the tests *are* the grader). Claude Code here is strictly more capable — it reads,
+ greps and writes across the whole file set — but it cannot build or run the project, so
+ trajectories are far shorter than SWE-bench's edit–test–fail loops. That is why the
+ `trimech-web` arm exists.
+* **A failed task ends the stream.** In the evaluator's `sequential` mode the project stops
+ at the first task that fails every attempt. The driver still asks for the tasks after it;
+ those rows come back `resolved: false` with `stream_broken: true`, `unreached: true` and
+ zero tokens — unreached, not cheap. Aggregations must not read them as low-cost failures.
+
+**Two resolve metrics, and they differ.** Each task gets up to `TE_WEBBENCH_RETRY`
+attempts (default 2), so a row carries `resolved_p1` (first attempt — Web-Bench's own
+headline metric) and `resolved_p2` (any attempt), alongside the per-attempt verdict list
+`webbench_attempt_results`. `resolved` stays pass@N because that is what the evaluator
+acts on when it decides to end a stream — and because a row's cost is the sum of every
+attempt, pass@N is the metric the cost columns actually match. The retry can rescue several tasks
+per arm, so never quote one metric as the other. Unreached tasks carry `None` for both —
+never attempted is not a miss.
+
+Each run dir holds the same artifacts as a SWE-bench run (`transcript.jsonl`,
+`run_config.json`, `row.json`) plus `agent_request_<n>.json` — what the evaluator actually
+asked for on each attempt. The evaluator's own stdout is teed to
+`<results>/<arm>/<cell>/_webbench_<project>/evaluator.log`, and its report tree lands beside
+it.
+
+### Wiring check (no agent, no spend)
+
+```bash
+python3 scripts/run_experiment.py --arm baseline --agent claude --repo calculator --seed 0 \
+ --adapter webbench --tasks-file "$TOKENECONOMIZER_WEBBENCH_TASKS" \
+ --workload workloads/webbench.json --results-root /tmp/wb_smoke \
+ --runner te_harness.webbench_smoke:make_runner --fresh
+```
+
+This is the production runner with `_segment` replaced by "copy the project's own reference
+solution into the workspace". It exercises the real evaluator, the real HTTP contract and
+the real Playwright grading, so **every task should come back resolved** — anything else is
+a harness fault, not a model result. Run it after touching this file or upgrading the
+Web-Bench checkout.
+
+### Provisioning the checkout
+
+The evaluator is a Rush monorepo and must be installed *and built*; the projects' Playwright
+browsers must match the version they pin (`@playwright/test` 1.57.0 → browser build 1200).
+
+```bash
+cd "$(dirname "$TOKENECONOMIZER_WEBBENCH_PROJECTS")"
+rush install # once
+rush build --to @web-bench/eval # the evaluator + its deps
+rush build --to @web-bench/calculator --to @web-bench/expressjs --to @web-bench/react \
+ --to @web-bench/zustand --to @web-bench/table
+cd projects/calculator && npx playwright install chromium chromium-headless-shell
+```
+
+Verify the checkout independently of any agent by running a project's reference solution
+through its own suite — this is what the wiring check automates, and what to reach for when
+a real run's grades look impossible:
+
+```bash
+R="$TOKENECONOMIZER_WEBBENCH_PROJECTS"
+cd "$R/calculator" && MAX_TEST_WORKERS=4 EVAL_PROJECT_ROOT="$R/calculator/src" \
+ IS_EVAL_PRODUCTION=true EVAL=true \
+ EVAL_PROJECT_PORT=3931 npm run test -- -a # expect: 68 passed
+```
+
+`MAX_TEST_WORKERS=4` matches what the runner passes per grade
+(`TE_WEBBENCH_MAX_WORKERS`, default 4, recorded in the row as `playwright_workers`).
+Without it Playwright picks its own default — half the CPU count — and on a many-core host
+the **reference solution** fails its own suite on the larger projects, which reads like a
+broken checkout and is not one.
+
+No project in the current five needs a special pre-build — `react` and `zustand`
+are plain `vite build`, and `calculator`/`expressjs` build nothing. (The retired `prisma`
+did: its `npm run build` generated the client and seeded `test.sqlite`, and skipping it
+failed ~55 tests.) `expressjs` still fails one test if a stale `src/test.sqlite` is left
+lying around.
+
+> **Paths must be canonical.** Every path the evaluator derives comes from its entry
+> script's `__dirname`, and it passes the project root to each project's dev server through
+> `EVAL_PROJECT_ROOT` — while node reports `process.cwd()` already symlink-resolved.
+> Reached through a symlinked home (`/home/x` → `/srv/home/x`), the two spellings disagree
+> and Vite fails to resolve the `react` project's entry module against a root that is not
+> the root it thinks it is: **every** test fails with an app that never rendered, which
+> reads exactly like an agent that wrote nothing. `webbench_repo()` therefore
+> `realpath`-resolves the checkout, and anything invoking these commands by hand should too.
+
+## Config
+
+Everything comes from the repo's `.env` (see its `.env.example`) plus:
+
+| Var | Default | Meaning |
+|---|---|---|
+| `ANTHROPIC_API_KEY` | — | Claude Code's key, and the mechanisms' calls on a Claude lane. `ANTHROPIC_BASE_URL` / `ANTHROPIC_AUTH_TOKEN` are reported once and ignored: the agent and the mechanisms bill `api.anthropic.com` directly |
+| `OPENAI_API_KEY` | — | Codex's key. `OPENAI_BASE_URL` only for a region-scoped key on an OpenAI host |
+| `COPILOT_GITHUB_TOKEN` / `GH_TOKEN` / `GITHUB_TOKEN`, `COPILOT_PROVIDER_*` | — | Copilot's GitHub auth, or its own BYOK; see [`COPILOT.md`](COPILOT.md) |
+| `TE_ENV_ACTIVATION` | `1` | `0` disables [project-environment activation](#project-environment-activation) and reproduces the pre-2026-09-11 (inactive `testbed` env) behaviour. Recorded per row |
+| `ANTHROPIC_MODEL` | `claude-opus-5` | agent model (was `claude-opus-4-8`; still accepted) |
+| `TE_EFFORT_LEVEL` | `high` | reasoning effort (`low`/`medium`/`high`/`xhigh`/`max`), passed as `--effort`. Validated at construction — a typo raises instead of silently using the CLI default |
+| ~~`TE_OPTIMIZER_MODEL`~~ | **refused** | the between-task proposer / judge / nudge-writer runs on the **agent's own model**, on the agent's own authentication, in every harness (`routing.optimizer_transport` records the wire). Not a knob: charging mechanism overhead to a cheaper model would understate what the arm spends, against a baseline that made no such calls. Setting this variable now raises at construction rather than being half-honoured |
+| `TE_CLAUDE_BIN` | from `which claude` | host Claude Code binary to mount |
+| `TE_SWEBENCH_GRADING` | — (`$DATA/swebench/swebench_grading.json` if `DATA` is set) | `instance_id` → grading fields. **Required**: no default — unset raises at construction |
+| `TE_SWEBENCH_REPO` | — | checkout supplying the official grader. **Required**: no default — unset raises at construction, rather than failing as an `ImportError` deep in grading after a task has been paid for |
+| `TE_MAX_TURNS` | `80` | agent turn cap |
+| `TE_TASK_TIMEOUT` / `TE_EVAL_TIMEOUT` | `1800` | wall-clock caps, seconds |
+| `TE_KEEP_CONTAINERS` | unset | `1` leaves containers up for post-mortem |
+| `TE_ALLOW_NETWORK_TOOLS` | unset | `1` lifts EVERY answer-key layer at once — the PreToolUse guard, the `/etc/hosts` block, and Codex's `web_search = "disabled"`. For a run that genuinely wants a browsing agent; a graded benchmark run must leave it unset (the answers are a fetch away) |
+| `TE_CODEX_VENDOR` | resolved from the Node shim's install tree | **Codex lanes only.** The vendor directory bind-mounted read-only into the task container (the static-PIE ELF plus its siblings). Resolved and checked at runner construction, because Docker invents an *empty* directory for a missing bind-mount source rather than erroring — every task would fail `codex: not found` behind a green launch banner |
+| `TE_SWEBENCH_MAX_CPUS` | `16` | CPU cap for ONE task container, plus the matching `OMP`/`MKL`/`OPENBLAS`/`NUMEXPR`/`LOKY` thread caps — which are the half that actually bounds a joblib pool, since `--cpus` does not change what `nproc` reports inside the container. `0` = unlimited. Recorded per row in `run_config.json → cpu_limit`. The launchers set it to `4` |
+| `TE_FLEET_CPUS` | `16` | CPU budget for **every lane's containers together**, written by `scripts/launch/fleet_budget.sh` onto the cgroup all rootless-docker container scopes share. The variable above is per container, so *N* lanes are an *N*×cap ceiling; this is the one that bounds the host. It binds lanes that are already running, needs no restart, and is work-conserving inside the budget. `0`/`none` = do not enforce. An un-delegated or read-only cgroup warns and never fails a launch |
+| `TE_OPTIMIZER_MAX_TOKENS` | `16384` | output cap for optimizer calls, the same on every runner (both agents, both domains). Too low truncates a judge's per-tool JSON into an unparseable answer, which reads downstream as "no verdict" while still billing — truncation is now logged, not silent. Applies on the API transport only: `copilot -p` has no output-cap flag, so on the CLI transport the value is read by nothing |
+| `TE_OPTIMIZER_PREFLIGHT` | set by the driver | `0` skips the mechanism-endpoint probe at runner construction. `run_experiment.py` sets it from `config.needs_optimizer`, so arms that make no paid between-task call neither bill the probe nor depend on the endpoint. Unset means "assume it might", so a direct caller of `make_runner` keeps the check |
+| `TE_OPTIMIZER_LOG` | `$TOKENECONOMIZER_RESULTS/logs/optimizer_calls.jsonl` | where between-task optimizer calls (prompt + response + usage) are recorded; not recorded at all if unset and no results root |
+
+### Project-environment activation
+
+The official SWE-bench images install the instance's dependencies into the `testbed` conda
+env and activate it only from `/root/.bashrc`. No lane runs with `HOME=/root` and no CLI's
+shell sources that file, so until 2026-09-11 a bare `python` / `pytest` in the agent's shell
+hit the **base** interpreter (Python 3.11, no project deps) while `TASK_TEMPLATE` told the
+agent the env was active. A bare `python` then failed to import the project's own
+package, and each failure cost a multi-step detour before the agent found
+`/opt/miniconda3/envs/testbed/bin/python` — or gave up running tests. Copilot's
+`--bash-env=on` was inert because no `BASH_ENV` was set.
+
+Every SWE-bench runner (Claude, Copilot, Codex) now activates the env for the agent
+**process**: at container start it lists `/opt/miniconda3/envs/*/bin`, picks `testbed`
+(or the only env), reads the image's `PATH`, and hands the agent `PATH=<env>/bin:<image
+PATH>`, `CONDA_PREFIX` and `CONDA_DEFAULT_ENV` via `docker exec -e`. That reaches every
+shell shape the CLIs spawn (`bash -lc`, `bash -c`, any `HOME` — measured), which a
+`BASH_ENV` hook does not for Claude Code. `python` is resolved once under the new `PATH`
+and the result is recorded in `run_config.json → env_activation`
+(`mode`, `env_bin`, `python`, `verified`) and on the row as `env_activation`.
+
+`TE_ENV_ACTIVATION=0` restores the old behaviour. **Rows taken with and without activation
+are not comparable on cost** and must not be pooled; the field exists so an aggregator can tell them apart.
+
+### The tool catalog
+
+`reflect.parse_tool_catalog_claude` needs `{tool_name: description}` for the tools a run
+loaded. The Copilot CLI dumps tool schemas into its debug log; Claude Code does not — its
+transcript carries only the tool *names* — so the descriptions are captured off the wire:
+
+```bash
+python -m te_harness.capture_catalog # -> tokeneconomizer/vendor/claude_tool_catalog.json
+```
+
+It runs the agent exactly as a task run does (same image, container, flags, and turn cap —
+the CLI's offered tool set depends on the cap) with a stdlib recorder inside the container
+recording the `tools` array off the request and forwarding to the real API. Captures merge
+into any existing snapshot rather than replacing it, since the loaded set drifts slightly
+between runs. Re-run it after a Claude Code upgrade.
+
+### The grading file
+
+`swebench_grading.json` maps each workload instance to `base_commit`,
+`version`, `test_patch`, `FAIL_TO_PASS`, `PASS_TO_PASS`. The repo's `swebench_tasks.json`
+carries only `instance_id` / `problem_statement` / `repo`, which cannot be graded. Rebuild
+it from the HF parquet exports (all but 30 are in SWE-bench_Verified; those 30 xarray
+instances come from the full SWE-bench test split). A new run executes the 277 of
+`workloads.runnable_instances()`, which drops the one instance `SWEBENCH_EXCLUDED` names
+and adds the 49 `SWEBENCH_ADDED` uses to de-saturate the sklearn and sympy streams:
+
+```bash
+cd "$DATA/swebench"
+curl -sL -o verified.parquet "https://huggingface.co/api/datasets/princeton-nlp/SWE-bench_Verified/parquet/default/test/0.parquet"
+curl -sL -o full_test.parquet "https://huggingface.co/api/datasets/princeton-nlp/SWE-bench/parquet/default/test/0.parquet"
+python3 -c "
+import pyarrow.parquet as pq, json
+K=('repo','base_commit','environment_setup_commit','version','test_patch','patch','FAIL_TO_PASS','PASS_TO_PASS')
+out={}
+for f in ('full_test.parquet','verified.parquet'):
+ for r in pq.read_table(f).to_pylist():
+ out.setdefault(r['instance_id'], {k: r[k] for k in K})
+ids=[t['instance_id'] for t in json.load(open('swebench_tasks.json'))]
+json.dump({i: out[i] for i in ids if i in out}, open('swebench_grading.json','w'))"
+```
+
+## External tool scaffolds (the baseline arms)
+
+The baselines themselves live in the CORE package, at `tokeneconomizer/baselines/`, and
+are stdlib-only: they reach a container only through the callables this harness hands them
+(`exec_`, `copy_in`, `read_bytes`, `host_run` on `ScaffoldContext`). What follows is what
+*this* harness has to get right to drive them. Any other harness that can implement those
+four callables can run the same baselines unchanged.
+
+They install somebody else's off-the-shelf tool for the sealed agent so its cost and
+resolve land on the same honest basis as a mechanism. "Into the container" is the
+SWE-bench case, not the contract: Web-Bench runs the agent on the HOST, and the same
+baseline installs into a per-task `$HOME` and a per-task `bin/` instead. What differs is
+carried by `AgentProfile.home` / `AgentProfile.bin_dir`, so a scaffold never branches on
+the harness -- and one that hardcodes `/usr/local/bin` silently no-ops off-container
+(`chmod` fails, `&&` short-circuits, the row reads `rtk_installed: false`). They are **baselines, not
+mechanisms**: nothing in `tokeneconomizer/` imports the package on its own — not
+`__init__.py`, not anything under `mechanisms/` — so `import tokeneconomizer` stays
+dependency-free and no mechanism can see a baseline.
+
+| Scaffold | Arms | What it installs | Host requirement |
+|---|---|---|---|
+| `rtk` | `rtk`, `trimech-rtk`, `trimech-rtk-web`, `codex-trimech-rtk` | pinned rtk **0.45.0** musl binary, plus the integration that agent has: a PreToolUse hook on Claude (`settings.json`) and Copilot (`$COPILOT_HOME/hooks/`), instruction files on Codex | staged binary cache (`TE_SCAFFOLD_CACHE`, else `$DATA/scaffolds`) |
+| `serena` | `serena` | `uv tool install serena-agent` in-container, an MCP server, and a nav-tool nudge | none (needs network *from* the container) |
+| `repomap` | `repomap`, `trimech-repomap`, `codex-trimech-repomap` | aider's token-budgeted repo map, written as project instructions | `aider` on the host — `pip install -e '.[baselines]'` |
+
+There is deliberately **no `trimech-serena`**: Serena's contract is that the agent navigates
+through its MCP tools, which is exactly what evidence untooling prunes when a tool goes
+unused, so the combined arm would measure that contention rather than either baseline. RTK
+hooks `Bash` (always in `PROTECTED_TOOLS`, so untooling can never drop it) and the repo map
+writes a file and touches no tool — neither contends, which is why those two combinations
+are well-defined.
+
+### Which baselines each domain can host
+
+`rtk` is the only one Web-Bench can run (`te_harness/webbench.py` `WEBBENCH_SCAFFOLDS`);
+`scripts/sweep_plan.py` `SWEEP_ARMS_WEBBENCH_UNAVAILABLE` is the planner-side twin, and
+`tests/test_sweep_arms.py` asserts the two agree.
+
+| Scaffold | SWE-bench | Web-Bench | Why not |
+|---|---|---|---|
+| `rtk` | yes | **yes** | — it needs no container and no language |
+| `repomap` | yes | no | its generator walks `*.py`; Web-Bench projects are TS/JS |
+| `serena` | yes | no | an MCP navigator for a codebase; Web-Bench ships a file set |
+
+Two Web-Bench-specific differences, both handled by the runner rather than the baseline:
+
+* **Artifacts are excluded from the ANSWER, not from a diff.** There is no `git diff` to
+ subtract from — the evaluator applies whatever files the runner returns — so `_collect`
+ refuses to ship the scaffold's own names, recording them in `run_config.json` →
+ `scaffold.answer_excludes`.
+* **`COPILOT_HOME` is isolated per task-run when a scaffold is active.** The Copilot
+ Web-Bench runner deliberately keeps the real `$HOME` (a fresh one would orphan the
+ ~170 MB package cache), but both hook writers land in the CLI's config dir — rtk's
+ `hooks/rtk-rewrite.json` and the net guard's `settings.json` — which would be the
+ operator's own `~/.copilot`. Only that variable is redirected, seeded with the real auth,
+ so a sweep cannot leave a hook behind on the machine. The scaffold resolves the directory
+ as `${COPILOT_HOME:-$HOME/.copilot}` in the shell, which is right on both lanes.
+
+### Three things that make a scaffold arm silently fake
+
+* **The graded diff.** A scaffold writes project instructions *into* `/testbed`, so
+ `git diff` picks them up and the official grader would apply the scaffold's own prompt
+ text as part of the agent's fix. Each scaffold declares `patch_excludes`; the runner
+ subtracts them (`git diff -- . ':(exclude,top)CLAUDE.md' …`) and records what it
+ subtracted in `run_config.json`. With no scaffold the diff command is left byte-identical
+ to the pre-scaffold form, so stored baseline rows stay reproducible.
+* **RTK's hook.** `rtk init -g` **declines** to patch the agent's settings
+ non-interactively — it prints a "MANUAL STEP" notice and exits 0. Trusting it yields an
+ arm where `rtk --version` works, every provenance field looks healthy, and the hook never
+ fires. The scaffold registers the hook itself and `rtk_hook_registered` records whether
+ that took; a `true` install with a `false` hook logs a warning naming the failure.
+* **Serena is a nudge, not an exclusion.** The original study asked the agent to stop using
+ the native navigation tools via an instruction and did not pass `--disallowed-tools`.
+ That is kept: `Read` is in `PROTECTED_TOOLS`, so an exclusion would be filtered out and
+ the arm would run native with an unused MCP server attached. If the agent keeps grepping,
+ that **is** the measurement.
+
+A scaffold that fails is recorded, not raised: the row carries `scaffold_error` (or
+`rtk_installed: false`, `serena_installed: false`, `map_ok: false`) and the task runs
+native. A lane of flagged rows is diagnosable; a lane of missing rows is not. **Filter those
+rows out before quoting a scaffold arm's mean** — they are native rows wearing a scaffold's
+name.
+
+### Adding one
+
+Subclass `Scaffold` (`tokeneconomizer/baselines/base.py`), implement `setup(ctx)` returning a
+notes dict, declare `patch_excludes`, register it in `SCAFFOLDS`, and map its arm(s) in
+`ARM_SCAFFOLDS`. Per-agent differences (instructions path, native tool names, `$HOME`, how
+an MCP config is wired) come from `ctx.profile`, never from a hardcoded assumption — writing
+`CLAUDE.md` on a Copilot run produces an arm that is native with a stray file in it. Add
+the arm to `ARMS` in `scripts/run_experiment.py` with `scaffold="<name>"`;
+`tests/test_sweep_arms.py` asserts the two maps agree.
+
+## The answer-key guard
+
+A SWE-bench instance's fix commit is public on GitHub, so an agent that can fetch it is not
+solving the task — it is reading the answer key, and the resolve verdict it produces is
+worthless. So this applies to **every arm, `baseline` included**: it is a property of the
+measurement, not of an arm. Leaving it to untooling would be worse than useless, since
+untooling drops a tool only after observing disuse — precisely the agents that browse would
+keep the ability.
+
+**One policy.** A single case-insensitive substring, `github`. Re-derived 2026-09-09 over
+3,538 stored run dirs and 558 network-shell calls: it matches 352, every one of them reading
+a repository's source, history, PR diffs or `.patch` files, with **0 false positives** (no
+`pip`/`npm`/`npx`/`yarn` install blocked; pip alone accounts for 180 of the legitimate
+calls) and **0 misses** (nothing naming a `.patch`, a `/commit/`, a `/pull/`, a mirror, or a
+benchmark repo escaped it). That is the argument for keeping it this narrow rather than
+cutting the container's network: the wider rule also deletes the tool surface the untooling
+mechanism is measured on, and charges the baseline for a restriction the cheat route never
+needed.
+
+**Two enforcement points**, because a hook is only as good as the CLI's cooperation:
+
+| layer | how | Claude | Copilot | Codex |
+|---|---|---|---|---|
+| `PreToolUse` hook | exit 2 + reason on stderr | ✅ | ✅ | ✅ since 2026-09-30 — `$CODEX_HOME/hooks.json` + `--dangerously-bypass-hook-trust` |
+| `/etc/hosts` block | answer-key hosts → `127.0.0.1` in the task container | ✅ | ✅ | ✅ |
+
+The hook is the layer that tells the **model why**, which a DNS failure cannot. The hosts
+block needs no cooperation from any CLI, so it is identical on all three lanes, and it was
+the only layer Codex had until 2026-09-30. Verified in a real SWE-bench container: afterwards `curl`, `wget`,
+`git clone` and `python -c "import urllib.request"` all fail with `Connection refused` for
+the answer-key hosts, while `pypi.org`, `files.pythonhosted.org` and a real `pip download`
+keep working. `127.0.0.1` rather than `0.0.0.0` so the connection is refused at once instead
+of hanging to a timeout. The stanza is appended idempotently behind a marker comment and
+then **verified** (`getent hosts github.com` must answer `127.0.0.1`) — a write can succeed
+into a container whose resolver ignores `/etc/hosts`, and that failure is otherwise
+invisible.
+
+`TE_ALLOW_NETWORK_TOOLS=1` lifts every layer, or none.
+
+**Neither layer is bullet-proof**, and the row telemetry is what makes that manageable:
+
+| field | means |
+|---|---|
+| `net_guard_installed` | the hook is armed *and* proved it refuses a probe |
+| `net_guard_blocks` | hook refusals this task |
+| `answer_key_hosts_blocked` | the DNS stanza is installed *and* `github.com` resolves to loopback |
+| `answer_key_attempts` | answer-key-shaped commands found in the transcript afterwards — **agent-independent**, so it is the number to watch on any row whose `net_guard_installed` is false (every Codex row before 2026-09-30); `None` when there is no transcript, never `0` |
+
+An IP literal, a redirect, a mirror or a base64'd URL walks past both layers. A model
+routing around them shows up as `answer_key_attempts` climbing while `net_guard_blocks`
+stays flat — which is why attempts are counted from the trace and not inferred from
+refusals.
+
+## Not covered yet
+
+* **Two tool descriptions are unobtainable.** The catalog is captured (below), but `Task`
+ and `ToolSearch` stay name-only: they exist *only* in first-party mode, and pointing the
+ CLI at any base URL — including the capture shim — removes them from the offered set.
+ Their schemas reach the model's context but never the transcript (a `ToolSearch`
+ tool_result is empty by design), so there is nowhere to read them from short of
+ intercepting first-party TLS. The judge tolerates a name with no description.
diff --git a/experiments/docs/RUNNING.md b/experiments/docs/RUNNING.md
new file mode 100644
index 0000000..194238c
--- /dev/null
+++ b/experiments/docs/RUNNING.md
@@ -0,0 +1,209 @@
+# Running the experiments
+
+The sweep runs three agents — Claude Code, the GitHub Copilot CLI and Codex — each on its own
+native authentication, against SWE-bench and Web-Bench, once per arm. This page is the run
+book: how each agent is billed, how to check the wiring before paying for anything, the arm
+catalog with one command per lane, and what to do with the rows afterwards.
+
+Before this page: [`SETUP.md`](SETUP.md) (venv, data root, Docker, `.env`) and
+[`BENCHMARKS.md`](BENCHMARKS.md) (task images and datasets). Commands below run from
+`experiments/`.
+
+## 1. Authentication — each agent's own
+
+| Agent | Billed through | What to set (in `.env` or the shell) |
+|---|---|---|
+| Claude Code | the Anthropic API | `ANTHROPIC_API_KEY` |
+| Codex | the OpenAI API | `OPENAI_API_KEY`; `OPENAI_BASE_URL` only for a key issued against a regional OpenAI endpoint (`https://<region>.api.openai.com/v1`) |
+| Copilot CLI | your GitHub Copilot subscription | `/login` in `copilot` on the host, or `COPILOT_GITHUB_TOKEN` / `GH_TOKEN` |
+| Copilot CLI | Copilot's own BYOK | `COPILOT_PROVIDER_BASE_URL`, `COPILOT_PROVIDER_TYPE`, `COPILOT_PROVIDER_API_KEY` or `_BEARER_TOKEN`, and a pinned model (see [`COPILOT.md`](COPILOT.md)) |
+
+Each agent bills its own vendor directly. The launchers ignore `ANTHROPIC_BASE_URL` and
+`ANTHROPIC_AUTH_TOKEN` (they say so once and unset them), and refuse an `OPENAI_BASE_URL`
+that is not an OpenAI host. Copilot-specific steps — signing in, keeping a token warm through
+a long sweep, and checking a finished row — are in [`COPILOT.md`](COPILOT.md).
+
+The model is part of the cost basis, so every lane pins one: the results tree is named
+`<agent>_<domain>_<model-tag>` (for example `claude_swebench_opus48`), and two models never
+share a tree.
+
+## 2. Check the wiring for free
+
+```bash
+bash scripts/launch/smoke_test.sh # offline: StubRunner, temp dirs, no Docker, no model
+AGENT=claude MODEL=claude-sonnet-5 DOMAIN=swebench DRYRUN=1 \
+ bash scripts/launch/main_sweep.sh # the plan and its size, nothing run
+```
+
+`DRYRUN=1` works on a fresh clone with nothing configured; the billed probes skip themselves.
+
+## 3. One paid cell first
+
+One task, one arm, in the foreground, before any lane:
+
+```bash
+AGENT=claude MODEL=claude-sonnet-5 DOMAIN=swebench ARMS=baseline \
+ REPOS=sympy/sympy SEEDS=0 LIMIT=1 DETACH=0 \
+ RUNNER=te_harness.swebench:make_runner \
+ bash scripts/launch/main_sweep.sh
+```
+
+Then read the row (`$TOKENECONOMIZER_RESULTS/<tree>/baseline/sympy_s0/*/row.json`):
+`num_turns` must be above zero and the token fields populated. A mechanism-bearing arm needs
+more than one task to fire anything — injection proposes on a 5-task cadence, untooling
+needs 3 consecutive unused tasks, and the spiral guard waits for its minimum turn count — so
+pilot `trimech` with `LIMIT=6` before trusting it.
+
+## 4. Claude Code and Copilot — `main_sweep.sh`
+
+One invocation fills one tree: one agent, one model, one domain, every arm × group × seed.
+The main sweep is both agents × `claude-opus-4-8` and `claude-sonnet-5` × both domains, eight
+lanes; `scripts/launch/full_sweep.sh` launches all eight. Each lane is detached by default
+and resumes where it stopped (`SKIP_DONE=1`).
+
+### swebench — 12 arms
+
+| Arm | What runs |
+|---|---|
+| `baseline` | no mechanisms, harness-default effort (`high`) — the cost basis every arm pairs against |
+| `baseline-medium` | `baseline` at `TE_EFFORT_LEVEL=medium` (the arm sets it) |
+| `baseline-low` | `baseline` at `TE_EFFORT_LEVEL=low` |
+| `inject-only` | micro-skill synthesis and injection only |
+| `untool-only` | evidence untooling and the tool judge only |
+| `spiral-only` | spiral guard, judge and nudge only |
+| `trimech` | all three mechanisms together |
+| `rtk` | `baseline` + rtk 0.45.0 |
+| `serena` | `baseline` + Serena |
+| `repomap` | `baseline` + aider's repo map |
+| `trimech-rtk` | `trimech` + rtk |
+| `trimech-repomap` | `trimech` + the repo map |
+
+```bash
+for AGENT in claude copilot; do for MODEL in claude-opus-4-8 claude-sonnet-5; do
+ AGENT=$AGENT MODEL=$MODEL DOMAIN=swebench \
+ RUNNER=te_harness.swebench:make_runner \
+ bash scripts/launch/main_sweep.sh
+done; done
+```
+
+### webbench — 5 arms
+
+| Arm | What runs |
+|---|---|
+| `baseline` | no mechanisms, harness-default effort |
+| `baseline-medium` | `baseline` at `TE_EFFORT_LEVEL=medium` |
+| `baseline-low` | `baseline` at `TE_EFFORT_LEVEL=low` |
+| `trimech-web` | `trimech` with the spiral stage tuned for short streams |
+| `rtk` | `baseline` + rtk 0.45.0 |
+
+```bash
+for AGENT in claude copilot; do for MODEL in claude-opus-4-8 claude-sonnet-5; do
+ AGENT=$AGENT MODEL=$MODEL DOMAIN=webbench \
+ RUNNER=te_harness.webbench:make_runner \
+ bash scripts/launch/main_sweep.sh
+done; done
+```
+
+Web-Bench reaches the agent through the evaluator's HTTP seam with no per-task repository
+container, so `serena`, `repomap` and `trimech-repomap` cannot run there, and the planner
+refuses them. The single-mechanism arms are answered on SWE-bench and are not planned for
+Web-Bench; they still run if named in `ARMS`.
+
+### The frontier lane
+
+`claude-opus-5` runs four arms per domain on both agents, in trees of its own (never pooled
+with the main sweep's):
+
+```bash
+for AGENT in claude copilot; do
+ AGENT=$AGENT MODEL=claude-opus-5 DOMAIN=swebench \
+ ARMS="baseline baseline-medium baseline-low trimech" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
+ AGENT=$AGENT MODEL=claude-opus-5 DOMAIN=webbench \
+ ARMS="baseline baseline-medium baseline-low trimech-web" \
+ RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
+done
+```
+
+`full_sweep.sh` does not launch these two.
+
+## 5. Codex — `codex_ab.sh`
+
+Codex runs one arm per invocation, through its own launcher, on `gpt-5.6-terra` and
+`gpt-5.6-sol`. `BASELINE=0` runs only the named arm, so the loop below fills one tree per
+model and domain with every arm. The Codex mechanisms are native to it: untooling works on
+capability packages rather than single tool schemas, and the spiral guard nudges in the same
+thread rather than stopping a turn.
+
+### swebench — 12 arms
+
+| Arm | What runs |
+|---|---|
+| `baseline` | no mechanisms, harness-default effort (`high`) |
+| `baseline-medium` | `baseline` at `TE_EFFORT_LEVEL=medium` |
+| `baseline-low` | `baseline` at `TE_EFFORT_LEVEL=low` |
+| `codex-inject` | environment facts plus the distiller, injected at session start (the proposer bills) |
+| `codex-untool` | evidence-gated capability untooling |
+| `codex-spiral` | the defensive spiral profile: same-thread nudges, no abort, no effort change |
+| `codex-trimech` | all three Codex mechanisms |
+| `rtk` | `baseline` + rtk 0.45.0 in its Codex instruction mode |
+| `serena` | `baseline` + Serena |
+| `repomap` | `baseline` + aider's repo map |
+| `codex-trimech-rtk` | `codex-trimech` + rtk |
+| `codex-trimech-repomap` | `codex-trimech` + the repo map |
+
+```bash
+REPOS="django/django matplotlib/matplotlib scikit-learn/scikit-learn sphinx-doc/sphinx sympy/sympy pydata/xarray"
+for MODEL in gpt-5.6-terra gpt-5.6-sol; do
+ for ARM in baseline baseline-medium baseline-low codex-inject codex-untool codex-spiral \
+ codex-trimech rtk serena repomap codex-trimech-rtk codex-trimech-repomap; do
+ MODEL=$MODEL ARM=$ARM BASELINE=0 DOMAIN=swebench REPOS="$REPOS" SEEDS="0 1 2 3 4" \
+ RUNNER=te_harness.swebench:make_runner DETACH=0 \
+ bash scripts/launch/codex_ab.sh
+ done
+done
+```
+
+### webbench — 5 arms
+
+| Arm | What runs |
+|---|---|
+| `baseline` | no mechanisms, harness-default effort |
+| `baseline-medium` | `baseline` at `TE_EFFORT_LEVEL=medium` |
+| `baseline-low` | `baseline` at `TE_EFFORT_LEVEL=low` |
+| `codex-trimech` | all three Codex mechanisms |
+| `rtk` | `baseline` + rtk 0.45.0 in its Codex instruction mode |
+
+```bash
+PROJECTS="calculator expressjs react zustand table"
+for MODEL in gpt-5.6-terra gpt-5.6-sol; do
+ for ARM in baseline baseline-medium baseline-low codex-trimech rtk; do
+ MODEL=$MODEL ARM=$ARM BASELINE=0 DOMAIN=webbench REPOS="$PROJECTS" SEEDS="0 1 2 3 4" \
+ RUNNER=te_harness.webbench:make_runner DETACH=0 \
+ bash scripts/launch/codex_ab.sh
+ done
+done
+```
+
+On Web-Bench the Codex agent runs on the host; the runner blocks the answer-key hosts in an
+unprivileged user and mount namespace and refuses to start if it cannot.
+
+## 6. Narrowing a lane
+
+Every launcher takes the same knobs: `ARMS` (or `ARM`), `REPOS`, `SEEDS`, `LIMIT` (tasks per
+cell), `DRYRUN=1`, `DETACH=0`, `ON_FAIL=abort`. The variables and their defaults are listed at
+the top of each script and in [`../scripts/launch/README.md`](../scripts/launch/README.md).
+
+## 7. Aggregating and comparing
+
+```bash
+python scripts/aggregate.py $TOKENECONOMIZER_RESULTS/<tree> # -> <tree>/index/*.csv
+python scripts/compare_arms.py \
+ --baseline $TOKENECONOMIZER_RESULTS/<tree>/baseline/<cell> \
+ --arm $TOKENECONOMIZER_RESULTS/<tree>/<arm>/<cell>
+```
+
+`aggregate.py` rewrites the whole index on every call and refuses an `--arms` subset that
+would drop arms from it. `compare_arms.py` pairs tasks by instance id and prices them on the
+cache-aware effective-token basis, with each arm's own mechanism calls charged to it. Never
+average two trees into one cell: the model and the agent are both part of the cost basis.
diff --git a/experiments/docs/SETUP.md b/experiments/docs/SETUP.md
new file mode 100644
index 0000000..73da692
--- /dev/null
+++ b/experiments/docs/SETUP.md
@@ -0,0 +1,339 @@
+# Setup — from `git clone` to a real run
+
+A sequential walkthrough of the four things a fresh clone needs: a **venv**, a
+**data directory**, **credentials**, and a **runner**. Every step ends with a
+gate that tells you immediately whether it worked, so a failure lands on the step that
+caused it instead of surfacing hours later as an impossible benchmark result.
+
+Reference detail lives elsewhere and is linked rather than repeated:
+[`ENVIRONMENT.md`](ENVIRONMENT.md) for requirements and Docker install,
+[`BENCHMARKS.md`](BENCHMARKS.md) for per-benchmark dataset provisioning,
+[`../scripts/launch/README.md`](../scripts/launch/README.md) for launcher flags.
+
+```
+Step 1 venv → pytest + smoke test pass (no Docker, no keys)
+Step 2 data directory → $DATA exists, canonical
+Step 3 credentials → each agent's key answers, free
+Step 4 runner → one real task produces a graded row
+Step 5 per-benchmark data → the benchmark can grade its own answer key
+Step 6 first A/B → index CSVs compare_arms.py reads
+```
+
+---
+
+## Step 1 — the venv (Layer 1)
+
+```bash
+git clone <repo-url> TokenEconomizer && cd TokenEconomizer
+python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
+pip install -e '.[test]'
+cd experiments # every command below runs from here
+```
+
+The core is stdlib-only; the `test` extra adds `pytest`. Other install shapes
+(`requirements.txt`) are in [`ENVIRONMENT.md`](ENVIRONMENT.md).
+
+**Running the full arm catalog needs two more extras.** The run book lists 12 SWE-bench arms
+per agent, and three of them carry an external tool baseline with a host dependency:
+
+```bash
+pip install -e '..[test,harness,baselines]'
+```
+
+| extra | what it adds | arms that need it |
+|---|---|---|
+| `harness` | nothing on PyPI — it names the intent to run `te_harness/` (Docker, an agent CLI, a key, benchmark images) | every arm |
+| `baselines` | `aider-chat`, for the repo map generated on the HOST | `repomap`, `trimech-repomap`, `codex-trimech-repomap` |
+
+`rtk` and `serena` need no host package — rtk ships a static binary (staged into
+`TE_SCAFFOLD_CACHE`, else `$DATA/scaffolds`, from Step 2) and Serena installs itself inside
+the task container. Skip `baselines` only if you will never run a repo-map arm: without it
+those arms raise `ScaffoldError` at runner construction, naming this command, so a lane fails
+at launch rather than 200 tasks in.
+
+**Gate:**
+
+```bash
+python -c "import tokeneconomizer as te; print(te.Economizer)" # self-contained import
+pytest # offline suite
+bash scripts/launch/smoke_test.sh # every arm wired
+```
+
+Expect `pytest` all-pass and `SMOKE PASS: all arms wired; no artifacts left.` This works with
+no Docker, no datasets and no API keys — if it passes, the repo layer is healthy and
+everything after this point is external infrastructure.
+
+> Use **this** venv for the driver later, and install the SWE-bench grader into it (Step 4).
+> A venv that can run `pytest` but cannot import the grader will fail only once a real run
+> reaches grading.
+
+---
+
+## Step 2 — the one directory outside the repo
+
+The `AgentRunner` implementation is **in** the repo — [`te_harness/`](../te_harness) — so
+there is no harness directory to create and no
+`PYTHONPATH` to export: `scripts/run_experiment.py` puts `experiments/` and the repository
+root (the shared core) on `sys.path` itself,
+so `--runner te_harness.swebench:make_runner` resolves from any working directory. (Writing
+your own remains supported; put it anywhere importable and pass
+`--runner your_pkg:make_runner`.)
+
+What does **not** belong in the repo is data — what a benchmark ships and what runs produce.
+One directory, created wherever you like:
+
+| Variable | Holds | Contents |
+|---|---|---|
+| `$DATA` | **what a benchmark ships + what runs produce** | task lists, grading keys, benchmark checkouts, results |
+
+Two things deliberately live in neither: **Docker images** (they belong to Docker's own
+store — `docker images` is the inventory, never copy them into `$DATA`) and the **SWE-bench
+grader checkout**, which is a pip-installable library and can sit anywhere you point
+`TE_SWEBENCH_REPO`.
+
+```bash
+export DATA="$(readlink -f ~/te_data)"
+mkdir -p "$DATA/results"
+```
+
+`readlink -f` is not cosmetic. If your home is a symlink (`/home/you` → `/srv/home/you`),
+Web-Bench's Vite-based project resolves its entry module against a root that disagrees with
+node's already-resolved `process.cwd()`, and **every** test fails with an app that never
+rendered — indistinguishable from an agent that wrote nothing. Canonicalize once, here.
+
+Keep the exports in one sourceable file so every shell, launcher and hand-run command agrees:
+
+```bash
+cat > "$DATA/env.sh" <<'EOF'
+# TokenEconomizer data root — source me
+export DATA="$(readlink -f ~/te_data)"
+export TOKENECONOMIZER_RESULTS="$DATA/results"
+export PLAYWRIGHT_BROWSERS_PATH="$HOME/.cache/ms-playwright"
+EOF
+. "$DATA/env.sh"
+```
+
+### Why a benchmark checkout goes in `$DATA`, not the repo
+
+Some benchmarks ship their grader *with* their data, inseparably. Web-Bench is one monorepo
+holding both its Node/Playwright evaluator **and** its task data
+(`projects/<project>/tasks.jsonl`, `src-init/`, `test/`), and its evaluator resolves
+`projects/` relative to its own location — so the tree cannot be split. Of its ~1.9 GB,
+~1.7 GB is regenerable `node_modules`/pnpm store and ~158 MB is the actual dataset. Moving
+it into the repo would commit a real dataset in order to evict build artifacts from the data
+directory, and would leave the dataset variable pointing inside a git tree. So: **a benchmark
+checkout is a benchmark payload — it lives in `$DATA`**, and the repo stays code only.
+
+Results are also `$DATA` (they are what runs produce) and are git-ignored by the repo.
+Old run archives are worth parking under `$DATA/archive/` rather than at the top level.
+
+---
+
+## Step 3 — credentials
+
+```bash
+cp .env.example .env # .env is git-ignored
+```
+
+`.env` is plain `KEY=value`, no interpolation, and **a variable already exported in your
+shell always wins** — so a one-off `ANTHROPIC_API_KEY=... bash scripts/launch/...` overrides
+the file. Each agent runs on its own native authentication; fill in the ones you will run:
+
+| Key | For | Notes |
+|---|---|---|
+| `ANTHROPIC_API_KEY` | Claude Code | `ANTHROPIC_BASE_URL` / `ANTHROPIC_AUTH_TOKEN` are ignored and unset by the launchers |
+| `OPENAI_API_KEY` | Codex | |
+| `OPENAI_BASE_URL` | Codex | **only** for a region-scoped key (`https://<region>.api.openai.com/v1`) — such a key 401s on the global endpoint; any non-OpenAI host is refused |
+| `COPILOT_GITHUB_TOKEN` / `GH_TOKEN`, or `COPILOT_PROVIDER_*` | Copilot CLI | the GitHub subscription, or Copilot's own BYOK — see [`COPILOT.md`](COPILOT.md) |
+| `ANTHROPIC_MODEL`, `CODEX_MODEL` | cost basis | must exist in `pricing.MODEL_WEIGHTS`, which hard-fails rather than mispricing an unknown id |
+| `TE_EFFORT_LEVEL` | cost basis | `low\|medium\|high\|xhigh\|max`. Keep **both arms of an A/B identical**: results at different levels are not directly comparable |
+
+**Gate — prove the keys without spending a token.** The model-list endpoints are free:
+
+```bash
+set -a; . ./.env; set +a
+curl -s -o /dev/null -w 'anthropic %{http_code}\n' https://api.anthropic.com/v1/models \
+ -H "x-api-key: $ANTHROPIC_API_KEY" -H "anthropic-version: 2023-06-01"
+curl -s -o /dev/null -w 'openai %{http_code}\n' "${OPENAI_BASE_URL:-https://api.openai.com/v1}/models" \
+ -H "Authorization: Bearer $OPENAI_API_KEY"
+```
+
+Each key you set should give `200`. Fetch the listings in full once and check that your
+configured `ANTHROPIC_MODEL` / `CODEX_MODEL` ids actually appear — a typo here fails much
+later, inside a container, after you have paid for the containers that came before it. The
+Copilot gate is in [`COPILOT.md`](COPILOT.md) Step 1.
+
+---
+
+## Step 4 — the runner
+
+The heavy machinery — launching a sealed agent, applying the economizer's per-task
+decisions, grading against gold — sits **below `AgentRunner`**
+([`../tokeneconomizer/runner.py`](../../tokeneconomizer/runner.py), which is the authoritative
+contract). The repo ships a reference implementation, [`te_harness/`](../te_harness) — see
+[`HARNESS.md`](HARNESS.md) — so this step is a check
+rather than something to write. Every launch selects a runner with `RUNNER=module:factory`
+(a zero-arg callable returning the runner), which is also how you'd point it at your own.
+
+What a runner owes the economizer:
+
+| Member | Contract |
+|---|---|
+| `optimizer_call(prompt, model)` | one small paid LLM request → `(text, effective_tokens)`; used by the between-task proposer / judge / nudge-writer. Must never raise — return `("", 0.0)` on a transport failure so the mechanisms fall back to their LLM-free path |
+| `run_task(task, *, arm, injected, exclude_tools, run_dir_path, **kw)` | run one task with those decisions applied → `(row, run_dir)` |
+| `begin_stream(tasks, **info)` / `end_stream()` | **optional.** For a harness that owns a whole stream rather than one task (Web-Bench's evaluator drives its project's 20 sequentially dependent tasks itself). The driver calls them `hasattr`-guarded, so runners without them are unaffected |
+
+`run_task` must write `transcript.jsonl` under `run_dir_path` (the schema
+`tokeneconomizer.trace` re-reads) and return a row carrying the four-field token breakdown
+`pricing.effective_tokens` reads, plus `resolved` and `num_turns`. Recording a
+`run_config.json` beside it is strongly advised: agent CLIs record neither the reasoning
+effort nor the tool mask in their own transcripts, so without it a finished run cannot answer
+"what settings produced this?".
+
+Pick it per launch. No `PYTHONPATH` is needed for the in-repo harness — the driver puts
+`experiments/` and the repository root on `sys.path` itself, so `te_harness.*` resolves from any working directory; a
+harness of your own just has to be importable by the same interpreter.
+
+`.env` ships `RUNNER=stub`, the built-in offline `StubRunner`. It runs **no agent and grades
+nothing** — its `resolved` is a fixed deterministic pattern — so set `RUNNER` explicitly on
+every real launch. With more than one domain you will have more than one runner, which is
+why this is a per-launch flag rather than a value you set once:
+
+```bash
+RUNNER=te_harness.swebench:make_runner # SWE-bench
+RUNNER=te_harness.webbench:make_runner # Web-Bench
+```
+
+> The factory is handed the driver's `--agent` label (`factory(agent=...)`, falling back to
+> a zero-arg call for older harnesses, with the label also exported as `TE_AGENT`). A factory
+> that ignores it will run whatever agent it hardcodes while the driver files the rows under
+> `--agent`'s name — plausible numbers attributed to an agent that never ran. If your harness
+> supports one agent only, raise on an unexpected label rather than ignoring it. Running the
+> **Copilot CLI** instead of Claude Code: [`COPILOT.md`](COPILOT.md).
+
+**Gate:** launch with `RUNNER=stub` and confirm the banner
+`[env] RUNNER=stub -- SYNTHETIC run: no agent, no grading, numbers are NOT a benchmark`.
+That line is your standing check that you are pointed at a real harness; if you ever see it
+on a run you meant to be real, stop.
+
+### Installing the SWE-bench grader
+
+The official grader is a library the runner imports, so it must be importable from the **same
+interpreter that runs the driver**:
+
+```bash
+git clone https://github.com/princeton-nlp/SWE-bench.git "$DATA/SWE-bench"
+pip install -e "$DATA/SWE-bench" # into the Step-1 venv
+echo 'export TE_SWEBENCH_REPO="$DATA/SWE-bench"' >> "$DATA/env.sh"
+```
+
+**Gate:** `python -c "from swebench.harness.grading import get_eval_report; print('ok')"`.
+Its import chain pulls `bs4`/`datasets`, so a venv that skipped `pip install -e` on the
+checkout raises `ModuleNotFoundError` — and only at grading time, after the agent has run and
+billed.
+
+---
+
+## Step 5 — per-benchmark data
+
+Full fetch-and-layout instructions per benchmark are in
+[`BENCHMARKS.md`](BENCHMARKS.md); provision only what the workload names
+(`workloads/*.json`), which is a small fraction of each full benchmark. What belongs here is
+the discipline that saves money:
+
+> **Verify a benchmark can grade its own answer key before you spend anything on an agent.**
+> A benchmark that cannot pass its own reference solution cannot grade a model, and the
+> failure looks identical to a model that produced nothing.
+
+For **SWE-bench** that means, in order: the grading file covers every workload id you intend
+to run; every one of those ids has a local image (`docker images`); then one task at
+`--arm baseline --limit 1` whose `row.json` carries a `resolved` from the official report and
+a full token breakdown.
+
+For **Web-Bench** that means installing *and building* the monorepo (an installed-but-unbuilt
+checkout has no `dist/` to run — they are separate Rush commands), installing the Playwright
+browser build the projects **pin** (a mismatch surfaces as `Executable doesn't exist`, not as
+a version warning), and then running each project's own `src` through its own suite and
+recording the expected pass counts. Some projects need their build step first to generate a
+client or seed a database — exactly what the evaluator does per task — so a bare test run
+under-reports.
+
+Whatever your harness offers as an offline wiring check, run it before the first paid task:
+it should exercise the real grader with the agent call replaced, so **every task comes back
+resolved** and anything else is a harness fault rather than a model result.
+
+---
+
+## Step 6 — the first A/B
+
+```bash
+. "$DATA/env.sh"
+AGENT=claude DOMAIN=swebench ARM=trimech SEEDS="0" \
+ REPOS="pydata/xarray django/django sympy/sympy" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/trimech_ab.sh
+```
+
+Each launcher runs `baseline` then the economized arm over your repos and seeds, then
+aggregates into `index/all_rows.csv` + `index/summary_by_cell.csv`; compare the two arms
+cell by cell with `scripts/compare_arms.py`. Launches **detach by
+default** and print the log path plus the `kill -TERM -<pid>` line that stops the whole
+process group (`DETACH=0` to stay in the foreground).
+
+Read [`RUNNING.md`](RUNNING.md) for the arm catalog and one command per lane,
+[`../scripts/launch/README.md`](../scripts/launch/README.md) for every env var, and
+[`../scripts/README.md`](../scripts/README.md) for the chain from rows to comparisons.
+
+### Where the traces land
+
+```
+$TOKENECONOMIZER_RESULTS/<agent>_<domain>_<model>/ e.g. claude_swebench_opus48
+├── <arm>/<repo-or-project>_s<seed>/ the CELL: one arm x one stream x one seed
+│ ├── 20260826T204556Z__000__claude__<instance>__<arm>__s<seed>__<mode>/
+│ │ ├── transcript.jsonl the raw agent stream — the primary artifact
+│ │ ├── row.json the normalized row (tokens, resolved, telemetry)
+│ │ ├── run_config.json everything that shaped the run (effort, tool mask, …)
+│ │ └── … domain extras: model_patch.diff + eval.log (SWE-bench),
+│ │ agent_request_<n>.json (Web-Bench)
+│ ├── _streams/<stream>__<mode>__stream.jsonl the per-cell amortized cost curve
+│ └── _webbench_<project>/ Web-Bench only: the evaluator's log, config and report
+└── index/ aggregate.py output: all_rows.csv, summary_by_cell.csv
+```
+
+The run-dir name is `<UTC>__<index>__<agent>__<instance>__<arm>__s<seed>__<mode>`:
+
+* **the UTC stamp comes first**, so every run dir is unique and the tree sorts
+ chronologically. This is the part that matters operationally — **re-running a cell
+ archives the previous attempt beside the new one instead of overwriting its transcript.**
+ A trace cannot be reproduced after the fact, so losing one to a re-run loses it for good.
+* **the stream index** (`000`, `001`, …) keeps stream order explicit — which is essential
+ for a sequentially dependent Web-Bench stream — and breaks ties when two tasks land in
+ the same second (an unreached Web-Bench task costs no wall-clock at all).
+* **`<mode>`** stays in the name because `aggregate.py` recovers it from there.
+
+Because re-runs accumulate, `aggregate.py` reports **one row per (arm, cell, instance),
+taken from the newest attempt**, and prints `superseded runs skipped: N` when it passed over
+older ones. So the CSVs always describe the latest run of each task while the earlier traces
+stay on disk. To retire a bad run rather than supersede it, move its dir under
+`<tree>/provenance/` — the aggregator only walks the arm dirs.
+
+**One interpretation rule before you read any Web-Bench numbers.** Its streams are
+sequentially dependent, so the evaluator stops a project at the first task that fails every
+attempt; the driver still asks for the tasks after it, and a correct harness returns those as
+unreached with zero tokens. They are **not** cheap failures — exclude them from any per-task
+cost mean.
+
+---
+
+## Troubleshooting — symptom → cause
+
+| Symptom | Cause |
+|---|---|
+| `ModuleNotFoundError: No module named 'bs4'` at grading | SWE-bench grader not installed into the driver's interpreter (Step 4) |
+| Rows look plausible but resolve follows a fixed pattern | `RUNNER=stub`; check the launch banner (Step 4) |
+| Every Web-Bench test fails, app never rendered | non-canonical path — `readlink -f` your roots (Step 2) |
+| `Executable doesn't exist at .../chromium_headless_shell-<n>` | Playwright browser does not match the version the projects pin (Step 5) |
+| "the evaluator is not built" / no `dist/` | `rush install` ran but `rush build` did not (Step 5) |
+| A whole project's tests fail that passed before | a build/seed step was skipped, or a stale database file was left behind (Step 5) |
+| Codex 401s despite a valid key | a regional OpenAI key is hitting the global endpoint; set `OPENAI_BASE_URL` (Step 3) |
+| `pricing: no weight entry for billed model …` | the billed model id is not in `pricing.MODEL_WEIGHTS`; add it rather than letting it be mispriced |
+| Two runs differ in cost at equal resolve | `TE_EFFORT_LEVEL` differs between them (Step 3) |
diff --git a/experiments/requirements.txt b/experiments/requirements.txt
new file mode 100644
index 0000000..830576c
--- /dev/null
+++ b/experiments/requirements.txt
@@ -0,0 +1,14 @@
+# TokenEconomizer — repo-layer requirements (Layer 1).
+#
+# The CORE package (`import tokeneconomizer`) is STDLIB-ONLY and needs nothing here.
+# This pin covers the one optional repo capability: running the test suite. It mirrors the
+# [test] extra in pyproject.toml.
+#
+# pip install -r requirements.txt # tests
+# pip install -e . # then install the package itself (editable)
+#
+# NOT included on purpose (these belong to the EXTERNAL harness, Layer 2 — see
+# docs/ENVIRONMENT.md and docs/BENCHMARKS.md): Docker, the sealed agent CLIs, and the
+# provisioning-only tools (git, Node/npm, the HuggingFace `datasets` package).
+
+pytest>=7.0 # offline test suite (no Docker, no datasets)
diff --git a/experiments/scripts/README.md b/experiments/scripts/README.md
new file mode 100644
index 0000000..e9c9f97
--- /dev/null
+++ b/experiments/scripts/README.md
@@ -0,0 +1,109 @@
+# scripts/ — reproduction pipeline
+
+The scripts here plan, drive, aggregate and compare the experiment arms. The results trees
+they write are **outside the repository and regenerated per run**. Dataset, Docker-image,
+and results paths below are **placeholders**: set the environment variables to point at
+your own runs. The run book is [`../docs/RUNNING.md`](../docs/RUNNING.md).
+
+## The end-to-end chain
+
+```
+ build a task stream run each arm aggregate compare
+ ───────────────────── ─────────────────────── ───────────────── ──────────────
+ <your stream builder> ──▶ run_experiment.py ──▶ aggregate.py ──▶ compare_arms.py
+ (chronological task list) (facade + AgentRunner; row.json ─▶ paired per-instance
+ Docker/CLI harness: te_harness/) all_rows.csv + cost and resolve
+ │ summary_by_cell.csv
+ ▼
+ _streams/*.jsonl ──▶ summarize_stream.py (amortized cost curve)
+```
+
+### 0. Launchers — `launch/` (reproduce the main experiments)
+
+`run_experiment.py` is the one-arm driver that maps each **proven arm** to an exact
+`EconomizerConfig` and drives the facade per task (see [`launch/README.md`](launch/README.md)
+for the full arm→flags table). The `launch/*.sh` scripts wrap it into lanes and A/Bs:
+
+```bash
+# Claude/Copilot tri-mechanism A/B on a domain (baseline vs trimech), then aggregate:
+AGENT=claude DOMAIN=swebench ARM=trimech REPOS="pydata/xarray" SEEDS="0 1 2" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/trimech_ab.sh
+# Codex-native A/B (codex-untool | codex-spiral | codex-inject | codex-trimech | ...):
+ARM=codex-untool RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
+# Offline wiring check (StubRunner; writes only to a temp dir, cleans up):
+bash scripts/launch/smoke_test.sh
+```
+
+The task-execution harness is selected with `RUNNER=module:factory`: this repo's own is
+[`te_harness/`](../te_harness), and `RUNNER=stub` runs a no-agent stand-in for wiring checks
+only.
+
+### 1. Run each arm
+
+Task execution — launching a sealed agent (Claude Code / Copilot CLI / Codex) inside its
+Docker image, applying the economizer's per-task decisions, and grading against the gold
+patch — lives **below the `tokeneconomizer.runner.AgentRunner` interface**, in
+[`te_harness/`](../te_harness) (details: [`../docs/HARNESS.md`](../docs/HARNESS.md)). It is a
+sibling package, never imported from the core and not shipped in the wheel, so the core stays
+free of Docker and CLI dependencies.
+
+Bringing your own is a supported path, not a fallback: pass `--runner module:factory` for a
+zero-arg factory returning an `AgentRunner` that:
+
+- builds an `Economizer` per repo stream (`tokeneconomizer.facade.Economizer`);
+- per task calls `injection_prefix` → `build_tools` → `run_kwargs`, runs the agent with
+ those decisions, then `after_task`;
+- writes one `row.json` per task-run under `<RESULTS>/<arm>/<repo>_s<seed>/<run>/` and a
+ per-task `transcript.jsonl` the mechanisms read;
+- optionally appends each task's row to `<RESULTS>/<arm>/_streams/<repo>__<mode>__<ts>.jsonl`.
+
+Placeholders to set for your harness (see [`../docs/BENCHMARKS.md`](../docs/BENCHMARKS.md)
+for the full recipe and the driver-task-list vs. dataset-root distinction):
+
+| Placeholder | What it is |
+|---|---|
+| `TOKENECONOMIZER_SWEBENCH_TASKS` | JSON list of `{instance_id, problem_statement, repo}` (SWE-bench; driver task list **and** adapter source) |
+| `TOKENECONOMIZER_WEBBENCH_TASKS` / `_CREDDATA_TASKS` / `_REPOSVUL_TASKS` | driver task-list JSON **file** for those domains (not a dir) |
+| `TOKENECONOMIZER_WEBBENCH_PROJECTS` | Web-Bench `projects/` tree (adapter dataset root) |
+| `TOKENECONOMIZER_CREDDATA_ROOT` / `_REPOSVUL_ROOT` / `_CYBERGYM_SUITE` | dataset roots for those adapters |
+| `<DOCKER_IMAGE>` | era-pinned benchmark image (`te_harness.swebench.image_for`, or your runner) |
+| `TOKENECONOMIZER_RESULTS` | root of the results trees the aggregator reads |
+
+The arm catalog, per agent and domain, is in [`../docs/RUNNING.md`](../docs/RUNNING.md).
+
+### 2. Aggregate — `aggregate.py`
+
+Run dirs are timestamped (`<UTC>__<index>__…`), so re-running a cell adds a run dir rather
+than overwriting one. The aggregator therefore keeps **one row per (arm, cell, instance),
+from the newest attempt**, and prints `superseded runs skipped: N` when older attempts were
+passed over — they stay on disk. Move a run under `provenance/` to retire it entirely; only
+the arm dirs are walked.
+
+```
+python scripts/aggregate.py "$TOKENECONOMIZER_RESULTS" [--arms baseline trimech ...]
+```
+
+Walks `<root>/<arm>/<repo>_s<seed>/**/row.json` → writes `<root>/index/all_rows.csv`
+(one row per task-run) and `summary_by_cell.csv` (one row per arm×repo×seed). Cost is the
+honest cache-aware basis `pricing.effective_tokens(row) + ledger.overhead(row)` (vendored).
+
+### 3. Amortized curve — `summarize_stream.py`
+
+```
+python scripts/summarize_stream.py "$TOKENECONOMIZER_RESULTS"/<arm>/_streams/*.jsonl
+python scripts/summarize_stream.py --curve run.jsonl # per-position amortized curve
+```
+
+Folds each task's execution cost + mechanism overhead into one amortized-per-task number,
+cache-aware by default (`--raw` for legacy full-weight `tokens_total`).
+
+### 4. Compare arms — `compare_arms.py`
+
+```
+python scripts/compare_arms.py --baseline "$TOKENECONOMIZER_RESULTS"/<tree>/baseline/<cell> \
+ --arm "$TOKENECONOMIZER_RESULTS"/<tree>/<arm>/<cell>
+```
+
+Pairs the two cells on instance id (a task missing from either side is reported and
+excluded) and reports the pooled mean, the median, the paired per-task delta and resolve
+counts, on the same cache-aware basis with each arm's mechanism spend charged to it.
diff --git a/experiments/scripts/_env.py b/experiments/scripts/_env.py
new file mode 100644
index 0000000..b2c21ce
--- /dev/null
+++ b/experiments/scripts/_env.py
@@ -0,0 +1,83 @@
+"""Load the repo-root `.env` into `os.environ` (stdlib only, no python-dotenv).
+
+The core package is stdlib-only and credential-free by design; this loader lives in
+`scripts/` because credentials belong to the *harness* layer (Layer 2 — Docker + agent CLIs
++ API keys), not to the economizer. `run_experiment.py` calls `load_env()` at startup so a
+real run picks up ANTHROPIC_API_KEY / OPENAI_API_KEY without every launcher having to
+re-export them.
+
+Contract (matches scripts/launch/_env.sh so both loaders agree):
+ * plain ``KEY=value`` lines, optional leading ``export``; ``#`` starts a comment;
+ * surrounding single/double quotes are stripped; values are otherwise LITERAL — no
+ ``$VAR`` interpolation, no command substitution;
+ * an existing ``os.environ`` entry always WINS (``override=False``), so a one-off
+ ``FOO=bar python scripts/run_experiment.py ...`` still overrides the file;
+ * an empty value (``KEY=``) is treated as unset and skipped, so the committed
+ ``.env.example`` placeholders never mask a real shell export.
+
+`.env` is git-ignored; `.env.example` is the tracked template.
+"""
+import os
+
+__all__ = ["env_path", "load_env", "require_keys"]
+
+_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def env_path(path=None):
+ """The env file to read: explicit arg, else $TOKENECONOMIZER_ENV_FILE, else
+ experiments/.env, else the repository root's .env (where it lived before experiments/
+ moved under the plugin repository; an existing one keeps working)."""
+ explicit = path or os.environ.get("TOKENECONOMIZER_ENV_FILE")
+ if explicit:
+ return explicit
+ local = os.path.join(_REPO_ROOT, ".env")
+ parent = os.path.join(os.path.dirname(_REPO_ROOT), ".env")
+ return local if os.path.exists(local) or not os.path.exists(parent) else parent
+
+
+def _parse(text):
+ out = {}
+ for raw in text.splitlines():
+ line = raw.strip()
+ if not line or line.startswith("#") or "=" not in line:
+ continue
+ if line.startswith("export "):
+ line = line[len("export "):].lstrip()
+ key, _, val = line.partition("=")
+ key = key.strip()
+ if not key:
+ continue
+ val = val.strip()
+ if len(val) >= 2 and val[0] == val[-1] and val[0] in ("'", '"'):
+ val = val[1:-1]
+ if val: # KEY= means "not set here"; don't mask a real export
+ out[key] = val
+ return out
+
+
+def load_env(path=None, override=False):
+ """Read the env file into os.environ. Returns the dict of keys actually applied.
+
+ A missing file is not an error -- the offline/stub paths need no credentials at all."""
+ p = env_path(path)
+ try:
+ with open(p, encoding="utf-8") as fh:
+ parsed = _parse(fh.read())
+ except FileNotFoundError:
+ return {}
+ applied = {}
+ for key, val in parsed.items():
+ if override or not os.environ.get(key):
+ os.environ[key] = val
+ applied[key] = val
+ return applied
+
+
+def require_keys(*names):
+ """Exit with a pointed message if any of `names` is missing/empty in os.environ."""
+ missing = [n for n in names if not os.environ.get(n)]
+ if missing:
+ raise SystemExit(
+ "missing credential(s): " + ", ".join(missing)
+ + f"\nset them in {env_path()} (copy .env.example) or export them in your shell")
diff --git a/experiments/scripts/aggregate.py b/experiments/scripts/aggregate.py
new file mode 100644
index 0000000..57f8105
--- /dev/null
+++ b/experiments/scripts/aggregate.py
@@ -0,0 +1,474 @@
+#!/usr/bin/env python3
+"""Flatten a results tree into per-task and per-cell CSV tables.
+
+Walks a results ROOT of the shape ``<arm>/<repo>_s<seed>/**/row.json`` (the layout the
+runner writes and the curated exports keep) and emits, next to it:
+
+ * all_rows.csv -- one row per task-run (every field the comparisons read)
+ * summary_by_cell.csv -- one row per (arm, repo, seed) with resolve rate + summed cost
+
+Cost is the HONEST amortized basis (cache-aware effective execution + mechanism overhead),
+computed via the vendored pricing table and the ledger, so a method that mainly prunes
+cheap cached re-reads cannot look better than its real spend:
+
+ cost_effective_tokens = pricing.effective_tokens(row) + ledger.overhead(row)
+
+Usage:
+ python scripts/aggregate.py <results_root> [--arms baseline trimech ...] [--out DIR]
+
+`--arms` defaults to the immediate sub-directories of the results root that contain any
+`row.json`. It narrows the REWRITE, not just the reading, so naming a subset of the tree
+while writing to the canonical `<root>/index` is REFUSED -- it would leave the index the
+comparisons read describing fewer arms than the tree holds. Pass `--out` to put a subset
+index somewhere of its own. The results tree itself is git-ignored (regenerated per run);
+this script and its output are placeholders you point at your own runs.
+"""
+import argparse
+import collections
+import csv
+import glob
+import json
+import os
+import re
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+from tokeneconomizer import ledger # noqa: E402
+from tokeneconomizer.vendor import pricing # noqa: E402
+
+FIELDS = [
+ "arm", "repo", "seed", "instance", "repo_full", "mode",
+ "resolved",
+ # Web-Bench grades a task with up to `retry` attempts, so it carries two resolve
+ # metrics: pass@1 (first attempt -- Web-Bench's own headline) and pass@N (any attempt).
+ # Empty on SWE-bench, which is single-attempt.
+ "resolved_p1", "resolved_p2", "webbench_attempts",
+ "is_error", "aborted", "num_turns",
+ # A row whose token fields could not be read in full (the agent was killed before its
+ # terminal usage event, or its debug log was unparseable). Such a row is a LOWER BOUND
+ # on cost, so it must be visible in the CSV the comparisons read -- an unflagged zero
+ # was previously indistinguishable from a genuinely free task.
+ "usage_incomplete", "usage_incomplete_reason", "output_tokens_missing",
+ # A row whose eval script never RAN (the grading `docker exec` failed -- usually the
+ # container went away under us). `resolved` is False on such a row, which is
+ # indistinguishable from a graded miss, so the distinction has to be its own column:
+ # a not-graded row must be dropped from a resolve rate, not counted as a failure.
+ "not_graded", "not_graded_reason",
+ "tokens_total", "input_tokens", "output_tokens",
+ "cache_read_input_tokens", "cache_creation_input_tokens",
+ # `cost_usd` is the VENDOR's own dollar figure for the task, read from the
+ # `cost_usd_vendor` the runners write. A row that ran but carries no figure is a
+ # silent under-count when the column is summed -- its tokens land in the denominator
+ # while its dollars do not -- so the absence gets its own column rather than reading
+ # as a free task.
+ "cost_usd", "vendor_cost_missing", "vendor_cost_missing_reason",
+ "effective_tokens", "overhead_tokens", "cost_effective_tokens",
+ "wall_clock_s", "patch_bytes", "model", "effort",
+ "run_dir", "source_path",
+]
+
+# USD per effective (fresh-input-equivalent) token, for a $5/1M-output-tier model. Adjust
+# for your billed model; it only scales the reported dollar figure, not the comparison.
+USD_PER_EFFECTIVE_TOKEN = 5.0 / 1_000_000
+
+
+def discover_arms(root):
+ arms = []
+ for name in sorted(os.listdir(root)):
+ d = os.path.join(root, name)
+ if os.path.isdir(d) and glob.glob(os.path.join(d, "**", "row.json"),
+ recursive=True):
+ arms.append(name)
+ return arms
+
+
+def cell_of(path, root, arm):
+ """(repo, seed) from a row.json path under <root>/<arm>/<repo>_s<seed>/..."""
+ rel = os.path.relpath(path, os.path.join(root, arm))
+ cell = rel.split(os.sep)[0]
+ if "_s" in cell:
+ repo, seed = cell.rsplit("_s", 1)
+ return repo, seed
+ return cell, ""
+
+
+def normalize_repo(rs, inst):
+ rs = rs or (inst.rsplit("-", 1)[0] if inst else "")
+ rs = rs.replace("scikit-learn", "sklearn").replace("pydata__xarray", "xarray")
+ return rs.split("__")[0] if "__" in rs else rs
+
+
+_STAMP_RE = re.compile(r"^(\d{8}T\d{6}Z)__")
+
+
+def _run_order(rj):
+ """Sort key putting the NEWEST run of a task last.
+
+ Run dirs are named `<UTC>__<index>__...`, so the stamp orders them directly; a dir from
+ an older layout with no stamp falls back to its mtime. Both are compared as strings
+ against a fixed-width stamp, so an unstamped dir never outranks a stamped one by
+ accident -- it is ordered by its own mtime instead."""
+ name = os.path.basename(os.path.dirname(rj))
+ m = _STAMP_RE.match(name)
+ if m:
+ return (1, m.group(1), name)
+ try:
+ return (0, str(int(os.path.getmtime(rj))), name)
+ except OSError:
+ return (0, "", name)
+
+
+def usage_incomplete(d):
+ """(is_incomplete, reason) for one row: are its token fields a LOWER BOUND?
+
+ Two sources, because the flag has to cover rows written before it existed:
+
+ * The harness now says so itself. `te_harness` sets `usage_incomplete` when the agent
+ was killed before its terminal usage event and the row had to be rebuilt from the
+ transcript (input/cache exact, `output_tokens` unrecoverable), or when Copilot's
+ debug log could not be parsed.
+ * ROWS STORED BEFORE THAT FIX carry no flag at all -- they recorded a flat zero for
+ every token field and for `num_turns`. They are recognised by their signature: no
+ tokens, yet the task demonstrably ran (it resolved, or it left a patch). The
+ alternative reading, a task that really cost nothing and still produced a diff, is
+ not a thing that happens. Only guarded arms could produce these, so leaving them
+ unmarked biases every economized arm's mean DOWNWARD.
+
+ `pricing.effective_tokens` is deliberately left alone: this reports the defect, it does
+ not paper over it. A recovered figure is a floor and gets labelled as one.
+ """
+ if d.get("usage_incomplete"):
+ return True, d.get("usage_incomplete_reason") or "harness_flagged"
+ ran = bool(d.get("resolved")) or (d.get("patch_bytes") or 0) > 0
+ if ran and not (d.get("tokens_total") or 0) and not (d.get("input_tokens") or 0):
+ return True, "legacy_zero_usage"
+ return False, ""
+
+
+def not_graded(d):
+ """(never_graded, reason) for one row: did the official eval script actually run?
+
+ `_grade` flags `eval_exec_failed` when the grading `docker exec` came back nonzero
+ with nothing on stdout -- the eval did not run, so the patch was never scored. The
+ row still carries `resolved: False`, because there is no third value for it, and that
+ is precisely the trap: an ungraded row read as a graded miss depresses a resolve rate
+ with a task nobody evaluated. Counted separately and excluded from the denominator.
+ """
+ rep = d.get("eval_report") or {}
+ if isinstance(rep, dict) and rep.get("eval_exec_failed"):
+ return True, (rep.get("eval_exec_error")
+ or f"eval exec rc={rep.get('eval_exec_rc')}")
+ if isinstance(rep, dict) and rep.get("grading_error"):
+ return True, f"grading_error: {rep['grading_error']}"
+ return False, ""
+
+
+def vendor_usd(d):
+ """The vendor's own dollar figure for one task, or "" when the row carries none.
+
+ The runners write it as `cost_usd_vendor` (Claude Code's `total_cost_usd`, copied
+ from its final `result` event); only the older curated exports carry it under the
+ plain name `cost_usd`. Reading just `cost_usd` -- which this script did until the
+ fix -- left the column EMPTY for every tree this repo generates. The COLUMN keeps its
+ name so its consumers keep working; only the source key is fixed.
+ """
+ for k in ("cost_usd_vendor", "cost_usd"):
+ v = d.get(k)
+ if isinstance(v, (int, float)) and not isinstance(v, bool):
+ return v
+ return ""
+
+
+# Agents that never report a per-task dollar figure at all: Copilot meters premium
+# requests and Codex bills a subscription, so a null `cost_usd_vendor` on those lanes is
+# the documented normal case (docs/COPILOT.md), not a defect. Matched on the agent prefix,
+# so a suffixed label is covered without a new entry.
+AGENTS_WITHOUT_VENDOR_USD = ("copilot", "codex")
+
+
+def vendor_cost_missing(d):
+ """(missing, reason) for one row: did a task that RAN lose its vendor dollar figure?
+
+ When the agent's final `result` event arrives without `total_cost_usd` the row stores
+ None, and until this flag nothing marked it. The row is not otherwise damaged, which
+ is what makes it dangerous: token accounting is complete (so `usage_incomplete` stays
+ False and the effective-token basis is exact) and the task grades normally, so the
+ row looks healthy from every other column. Summing `cost_usd` across the cell then
+ UNDER-counts -- the row's tokens are in the denominator while its dollars are not.
+
+ It concentrates in the longest trajectories, i.e. the most expensive tasks, so a few
+ such rows can move an arm's billed total enough to change its rank in a comparison.
+
+ The effective-token basis is immune, which is the argument for reporting on it; this
+ flag is what keeps the vendor column honest when someone sums it anyway.
+ """
+ agent = str(d.get("agent") or "")
+ if agent.startswith(AGENTS_WITHOUT_VENDOR_USD):
+ return False, ""
+ if vendor_usd(d) != "":
+ return False, ""
+ # "Ran" on the same evidence usage_incomplete() uses: a verdict, a patch, or tokens.
+ ran = (d.get("resolved") is True or (d.get("patch_bytes") or 0) > 0
+ or (d.get("tokens_total") or 0) > 0 or (d.get("input_tokens") or 0) > 0)
+ if not ran:
+ return False, ""
+ return True, f"{agent or 'agent'} reported no total_cost_usd"
+
+
+def collect(root, arms):
+ rows = []
+ for arm in arms:
+ for rj in glob.glob(os.path.join(root, arm, "**", "row.json"), recursive=True):
+ try:
+ d = json.load(open(rj))
+ except Exception as e:
+ print("SKIP (unreadable):", rj, e)
+ continue
+ repo_cell, seed = cell_of(rj, root, arm)
+ inst = d.get("instance") or d.get("instance_id", "")
+ repo = normalize_repo(repo_cell, inst)
+ rd = d.get("run_dir", "")
+ mode = ("manual4" if "manual4" in rd
+ else "standalone" if "standalone" in rd else "")
+ eff = pricing.effective_tokens(d)
+ oh = ledger.overhead(d)
+ incomplete, reason = usage_incomplete(d)
+ ungraded, ungraded_reason = not_graded(d)
+ vmissing, vreason = vendor_cost_missing(d)
+ rows.append({
+ "arm": arm, "repo": repo, "seed": d.get("seed", seed),
+ "instance": inst, "repo_full": d.get("repo", ""), "mode": mode,
+ "resolved": d.get("resolved", ""),
+ # Web-Bench grades a task with up to `retry` attempts, so it has two
+ # resolve metrics: pass@1 (first attempt -- Web-Bench's own headline) and
+ # pass@N (any attempt). SWE-bench is single-attempt and leaves these empty,
+ # where they fall back to `resolved` in the cell rollup below.
+ "resolved_p1": d.get("resolved_p1", ""),
+ "resolved_p2": d.get("resolved_p2", ""),
+ "webbench_attempts": d.get("webbench_attempts", ""),
+ "is_error": d.get("is_error", ""),
+ "aborted": d.get("aborted", ""), "num_turns": d.get("num_turns", ""),
+ "tokens_total": d.get("tokens_total", ""),
+ "input_tokens": d.get("input_tokens", ""),
+ "output_tokens": d.get("output_tokens", ""),
+ "cache_read_input_tokens": d.get("cache_read_input_tokens", ""),
+ "cache_creation_input_tokens": d.get("cache_creation_input_tokens", ""),
+ "cost_usd": vendor_usd(d),
+ "vendor_cost_missing": vmissing,
+ "vendor_cost_missing_reason": vreason,
+ "effective_tokens": round(eff, 1),
+ "overhead_tokens": round(oh, 1),
+ "cost_effective_tokens": round(eff + oh, 1),
+ "wall_clock_s": d.get("wall_clock_s", ""),
+ "patch_bytes": d.get("patch_bytes", ""),
+ "model": d.get("model", ""), "effort": d.get("effort", ""),
+ "usage_incomplete": incomplete,
+ "usage_incomplete_reason": reason,
+ "not_graded": ungraded,
+ "not_graded_reason": ungraded_reason,
+ "output_tokens_missing": d.get("output_tokens_missing", ""),
+ "run_dir": rd, "source_path": os.path.relpath(rj, root),
+ "_order": _run_order(rj),
+ })
+ # One row per (arm, cell, instance): keep the NEWEST run and leave the earlier
+ # attempts on disk. Re-running a cell adds timestamped run dirs beside the old ones, so
+ # without this a re-run would double the cell's task count and average a task's cost
+ # with a superseded attempt of itself. A tree that was never re-run is unaffected.
+ newest = {}
+ superseded = 0
+ for r in rows:
+ key = (r["arm"], str(r["repo"]), str(r["seed"]), r["instance"])
+ prev = newest.get(key)
+ if prev is None:
+ newest[key] = r
+ else:
+ superseded += 1
+ if r["_order"] > prev["_order"]:
+ newest[key] = r
+ if superseded:
+ print(f"superseded runs skipped: {superseded} "
+ f"(older attempts of a re-run task; still on disk)")
+ rows = list(newest.values())
+ for r in rows:
+ r.pop("_order", None)
+ rows.sort(key=lambda r: (r["arm"], str(r["repo"]), str(r["seed"]), r["instance"]))
+ return rows
+
+
+def summarize(rows):
+ agg = collections.OrderedDict()
+ for r in rows:
+ key = (r["arm"], r["repo"], str(r["seed"]))
+ a = agg.setdefault(key, {"n": 0, "graded": 0, "resolved": 0, "errors": 0,
+ "eff": 0.0, "turns": 0, "incomplete": 0,
+ "ungraded": 0, "p1": 0, "p2": 0, "graded_pn": 0,
+ "vendor_missing": 0})
+ a["n"] += 1
+ if r.get("usage_incomplete") is True:
+ a["incomplete"] += 1
+ if r.get("vendor_cost_missing") is True:
+ a["vendor_missing"] += 1
+ if r.get("not_graded") is True:
+ # The eval never ran. Not a miss -- an absence. Leaving it in `graded` would
+ # let a lost container look like the model failing the task.
+ a["ungraded"] += 1
+ elif r["resolved"] != "":
+ a["graded"] += 1
+ if r["resolved"] is True:
+ a["resolved"] += 1
+ # pass@1 / pass@N. A row with no per-attempt verdicts (SWE-bench, or a
+ # Web-Bench row written before this field existed) falls back to `resolved`,
+ # which for a single-attempt domain is the same number. An UNREACHED row has
+ # resolved_p1 = None: never attempted, so it counts in neither numerator nor
+ # denominator -- scoring it as a miss would blame the agent for a task the
+ # evaluator never ran.
+ p1, p2 = r.get("resolved_p1", ""), r.get("resolved_p2", "")
+ att = r.get("webbench_attempts", "")
+ if p1 == "" and p2 == "":
+ if att != "":
+ # A Web-Bench row written before resolved_p1/p2 existed. The metrics
+ # are still RECOVERABLE: the runner breaks out of the attempt loop on
+ # the first pass, so a resolved row that took 2 attempts necessarily
+ # failed attempt 1. Deriving them beats falling back to `resolved`,
+ # which would silently report pass@N under the pass@1 column.
+ att = int(att or 0)
+ if att == 0:
+ p1 = p2 = None # unreached: never attempted
+ else:
+ p2 = r["resolved"] is True
+ p1 = p2 and att == 1
+ else:
+ # Single-attempt domain (SWE-bench): one attempt, so the two coincide.
+ p1 = p2 = r["resolved"]
+ if p1 is not None and p1 != "":
+ a["graded_pn"] += 1
+ if p1 is True:
+ a["p1"] += 1
+ if p2 is True:
+ a["p2"] += 1
+ if r["is_error"] is True:
+ a["errors"] += 1
+ try:
+ a["eff"] += float(r["cost_effective_tokens"])
+ except (TypeError, ValueError):
+ pass
+ try:
+ a["turns"] += int(r["num_turns"])
+ except (TypeError, ValueError):
+ pass
+ return agg
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__)
+ ap.add_argument("root", help="results tree root (<arm>/<repo>_s<seed>/**/row.json)")
+ ap.add_argument("--arms", nargs="*", default=None, help="arm dirs (default: auto)")
+ ap.add_argument("--out", default=None, help="output dir (default: <root>/index)")
+ args = ap.parse_args()
+
+ root = os.path.abspath(args.root)
+ present = discover_arms(root)
+ if not present:
+ ap.error(f"no arm dirs with row.json under {root}")
+ arms = args.arms or present
+ out = args.out or os.path.join(root, "index")
+
+ # `--arms` limits the REWRITE, not just the reading. The two CSVs are written whole
+ # every time, so narrowing the arm list while writing to the canonical <root>/index
+ # replaces an index describing the tree with one describing a subset -- at the path
+ # compare_arms.py reads, under a name that says nothing about being partial. A lane
+ # that runs one arm per invocation (the Codex lane) is the easy way to do it, and a
+ # tree was once left holding one arm of eight that way. So a subset write
+ # to the canonical path is refused rather than warned about -- pass --out to put the
+ # subset somewhere of its own, or drop --arms and aggregate the tree.
+ if args.arms and args.out is None:
+ dropped = [a for a in present if a not in arms]
+ if dropped:
+ ap.error(
+ "--arms would rewrite the canonical index with a SUBSET of the tree.\n"
+ f" index: {os.path.join(root, 'index')}\n"
+ f" keeping: {' '.join(arms)}\n"
+ f" DROPPED: {' '.join(dropped)}\n"
+ " The index is what compare_arms.py reads, so this would\n"
+ " silently discard those arms' rows. Either\n"
+ f" aggregate the whole tree: {os.path.basename(sys.argv[0])} {args.root}\n"
+ " or write the subset apart: --out <dir>")
+
+ unknown = [a for a in arms if a not in present]
+ if unknown:
+ print(f"!! --arms named {len(unknown)} dir(s) with no row.json under {root}: "
+ f"{' '.join(unknown)}")
+
+ os.makedirs(out, exist_ok=True)
+
+ rows = collect(root, arms)
+ with open(os.path.join(out, "all_rows.csv"), "w", newline="") as f:
+ w = csv.DictWriter(f, fieldnames=FIELDS)
+ w.writeheader()
+ w.writerows(rows)
+
+ agg = summarize(rows)
+ with open(os.path.join(out, "summary_by_cell.csv"), "w", newline="") as f:
+ w = csv.writer(f)
+ # n_usage_incomplete: how many of this cell's rows are a cost LOWER BOUND. A cell
+ # with any is not comparable to one without on cost until they are accounted for,
+ # so the count travels with the mean instead of being a separate audit step.
+ w.writerow(["arm", "repo", "seed", "n_tasks", "n_graded", "n_resolved",
+ "resolve_rate",
+ "n_attempted", "n_resolved_p1", "resolve_rate_p1",
+ "n_resolved_p2", "resolve_rate_p2",
+ "n_errors", "n_usage_incomplete", "n_not_graded",
+ "n_vendor_cost_missing",
+ "sum_effective_tokens",
+ "sum_cost_usd", "sum_turns", "mean_effective_tokens",
+ "mean_cost_usd"])
+ for (arm, repo, seed), a in agg.items():
+ n, g = a["n"], a["graded"]
+ cost = a["eff"] * USD_PER_EFFECTIVE_TOKEN
+ gp = a["graded_pn"]
+ w.writerow([arm, repo, seed, n, g, a["resolved"],
+ round(a["resolved"] / g, 4) if g else "",
+ gp, a["p1"], round(a["p1"] / gp, 4) if gp else "",
+ a["p2"], round(a["p2"] / gp, 4) if gp else "",
+ a["errors"], a["incomplete"], a["ungraded"],
+ a["vendor_missing"],
+ round(a["eff"], 1), round(cost, 4),
+ a["turns"], round(a["eff"] / n, 1) if n else "",
+ round(cost / n, 6) if n else ""])
+
+ print(f"arms: {arms}")
+ bad = [r for r in rows if r.get("usage_incomplete") is True]
+ if bad:
+ by_arm = collections.Counter(r["arm"] for r in bad)
+ print(f"!! usage_incomplete: {len(bad)} of {len(rows)} rows have token fields that "
+ f"are a LOWER BOUND, not a measurement -- their cost is understated and "
+ f"their cells' means with it")
+ for arm, k in sorted(by_arm.items()):
+ print(f" {arm}: {k}")
+ print(" see the usage_incomplete_reason column in all_rows.csv")
+ vbad = [r for r in rows if r.get("vendor_cost_missing") is True]
+ if vbad:
+ by_arm = collections.Counter(r["arm"] for r in vbad)
+ print(f"!! vendor_cost_missing: {len(vbad)} of {len(rows)} rows RAN but carry no "
+ f"vendor dollar figure -- summing the cost_usd column under-counts those "
+ f"cells, while their effective_tokens are complete. Compare on "
+ f"cost_effective_tokens, or re-run those tasks")
+ for arm, k in sorted(by_arm.items()):
+ print(f" {arm}: {k}")
+ print(" see the vendor_cost_missing_reason column in all_rows.csv")
+ never = [r for r in rows if r.get("not_graded") is True]
+ if never:
+ by_arm = collections.Counter(r["arm"] for r in never)
+ print(f"!! not_graded: {len(never)} of {len(rows)} rows were NEVER GRADED -- their "
+ f"eval script did not run, so `resolved` is an absence, not a miss. They are "
+ f"excluded from n_graded/resolve_rate; re-run those cells")
+ for arm, k in sorted(by_arm.items()):
+ print(f" {arm}: {k}")
+ print(" see the not_graded_reason column in all_rows.csv")
+ print(f"all_rows.csv: {len(rows)} task-runs -> {out}")
+ print(f"summary_by_cell.csv: {len(agg)} cells -> {out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/compare_arms.py b/experiments/scripts/compare_arms.py
new file mode 100644
index 0000000..e82e198
--- /dev/null
+++ b/experiments/scripts/compare_arms.py
@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Compare one arm's cell against a stored baseline cell, instance by instance.
+
+ python3 scripts/compare_arms.py \
+ --baseline $TE_DATA/results/<tree>/baseline/scikit_learn_s0 \
+ --arm $TE_DATA/results/<tree>/codex-inject/scikit_learn_s0
+
+Cost is the honest amortized basis (`pricing.effective_tokens` + `ledger.overhead`, so an
+arm's own mechanism spend is charged to it). The comparison is PAIRED on instance id -- a
+task missing from either side is reported and excluded -- and reports the pooled mean, the
+median, the paired per-task delta, and resolve counts. It also summarises the injection
+telemetry the arm's rows carry (how many tasks received a prefix, which facts were active),
+so a saving can be read next to the mechanism activity that produced it.
+
+The two cells may live in different trees; rows price by their billed model id, so effective
+tokens stay comparable across trees, while the endpoint of each side (a Copilot subscription
+or a BYOK provider, for example) is printed so nobody pools them by accident. Stdlib only.
+"""
+import argparse
+import glob
+import json
+import os
+import statistics as st
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+from tokeneconomizer import ledger # noqa: E402
+from tokeneconomizer.vendor import pricing # noqa: E402
+
+
+def load_cell(cell):
+ """{instance: row} -- the LATEST run dir per instance (a re-run archives beside the old)."""
+ rows, cfgs = {}, {}
+ for rowf in sorted(glob.glob(os.path.join(cell, "20*", "row.json"))):
+ try:
+ r = json.load(open(rowf, encoding="utf-8"))
+ except ValueError:
+ continue
+ inst = r.get("instance") or r.get("instance_id")
+ if not inst:
+ continue
+ rows[inst] = r
+ cf = os.path.join(os.path.dirname(rowf), "run_config.json")
+ if os.path.exists(cf):
+ try:
+ cfgs[inst] = json.load(open(cf, encoding="utf-8"))
+ except ValueError:
+ pass
+ return rows, cfgs
+
+
+def cost(r):
+ return pricing.effective_tokens(r) + ledger.overhead(r)
+
+
+def describe(rows, cfgs, label):
+ r0 = next(iter(rows.values()), {})
+ c0 = next(iter(cfgs.values()), {})
+ routing = c0.get("routing") or {}
+ print(f" {label:9s} n={len(rows):3d} agent={r0.get('agent')} model={r0.get('model')} "
+ f"effort={r0.get('effort')} endpoint={routing.get('base_url')} "
+ f"cli={r0.get('cli_version')}")
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+ ap.add_argument("--baseline", required=True)
+ ap.add_argument("--arm", required=True)
+ ap.add_argument("--json", help="write the comparison here")
+ args = ap.parse_args()
+
+ base, bcfg = load_cell(args.baseline)
+ arm, acfg = load_cell(args.arm)
+ print("== cells ==")
+ describe(base, bcfg, "baseline")
+ describe(arm, acfg, "arm")
+
+ common = sorted(set(base) & set(arm))
+ only_b, only_a = sorted(set(base) - set(arm)), sorted(set(arm) - set(base))
+ if only_b or only_a:
+ print(f" unpaired: {len(only_b)} baseline-only, {len(only_a)} arm-only "
+ f"(excluded from the paired figures)")
+ inc = [i for i in common if base[i].get("usage_incomplete") or arm[i].get("usage_incomplete")]
+ if inc:
+ print(f" usage_incomplete on {len(inc)} paired tasks (excluded): {inc[:5]}")
+ pairs = [i for i in common if i not in inc]
+ if not pairs:
+ sys.exit("no paired, complete tasks to compare")
+
+ bc = [cost(base[i]) for i in pairs]
+ ac = [cost(arm[i]) for i in pairs]
+ deltas = [(a - b) / b if b else 0.0 for a, b in zip(ac, bc)]
+ br = sum(1 for i in pairs if base[i].get("resolved") is True)
+ ar = sum(1 for i in pairs if arm[i].get("resolved") is True)
+ flips_lost = [i for i in pairs if base[i].get("resolved") is True and arm[i].get("resolved") is not True]
+ flips_won = [i for i in pairs if arm[i].get("resolved") is True and base[i].get("resolved") is not True]
+ oh = sum(ledger.overhead(arm[i]) for i in pairs)
+
+ print(f"\n== paired comparison over {len(pairs)} tasks ==")
+ print(f" {'':22s} {'baseline':>12s} {'arm':>12s} {'delta':>9s}")
+ print(f" {'resolved':22s} {br:12d} {ar:12d} {ar - br:+9d}")
+ print(f" {'mean eff tokens':22s} {st.mean(bc):12,.0f} {st.mean(ac):12,.0f} "
+ f"{100 * (st.mean(ac) - st.mean(bc)) / st.mean(bc):+8.1f}%")
+ print(f" {'median eff tokens':22s} {st.median(bc):12,.0f} {st.median(ac):12,.0f} "
+ f"{100 * (st.median(ac) - st.median(bc)) / st.median(bc):+8.1f}%")
+ print(f" {'pooled eff tokens':22s} {sum(bc):12,.0f} {sum(ac):12,.0f} "
+ f"{100 * (sum(ac) - sum(bc)) / sum(bc):+8.1f}%")
+ print(f" {'mean steps':22s} {st.mean(base[i].get('num_turns') or 0 for i in pairs):12.1f} "
+ f"{st.mean(arm[i].get('num_turns') or 0 for i in pairs):12.1f}")
+ print(f" paired per-task delta: mean {100 * st.mean(deltas):+.1f}%, median "
+ f"{100 * st.median(deltas):+.1f}%; arm cheaper on {sum(1 for d in deltas if d < 0)}/{len(deltas)} tasks")
+ print(f" mechanism overhead charged to the arm: {oh:,.0f} eff tokens")
+ print(f" resolve flips: lost {len(flips_lost)} {flips_lost[:6]} won {len(flips_won)} {flips_won[:6]}")
+
+ # ---- injection telemetry on the arm ----
+ with_prefix = sum(1 for i in pairs if (acfg.get(i, {}).get("injection") or {}).get("present"))
+ chars = [acfg[i]["injection"]["chars"] for i in pairs
+ if (acfg.get(i, {}).get("injection") or {}).get("present")]
+ print(f"\n== injection telemetry (arm) ==")
+ print(f" tasks with a prefix: {with_prefix}/{len(pairs)}; prefix chars mean "
+ f"{st.mean(chars) if chars else 0:,.0f} (~{(st.mean(chars) / 4) if chars else 0:,.0f} tokens)")
+ if with_prefix:
+ wp = [i for i in pairs if (acfg.get(i, {}).get("injection") or {}).get("present")]
+ wo = [i for i in pairs if i not in set(wp)]
+ for name, grp in (("with prefix", wp), ("without", wo)):
+ if grp:
+ print(f" {name:12s} n={len(grp):3d} arm mean={st.mean(cost(arm[i]) for i in grp):,.0f} "
+ f"baseline mean={st.mean(cost(base[i]) for i in grp):,.0f} "
+ f"delta={100 * (st.mean(cost(arm[i]) for i in grp) / st.mean(cost(base[i]) for i in grp) - 1):+.1f}% "
+ f"resolved arm={sum(1 for i in grp if arm[i].get('resolved') is True)} "
+ f"base={sum(1 for i in grp if base[i].get('resolved') is True)}")
+
+ # ---- learned environment facts (envfacts profile state saved beside the cell) ----
+ for state in glob.glob(os.path.join(args.arm, "envfacts_state", "*.json")):
+ d = json.load(open(state, encoding="utf-8"))
+ print(f"\n== learned facts ({os.path.basename(state)}, {d.get('now')} tasks folded) ==")
+ for fid, f in sorted(d.get("facts", {}).items()):
+ life = f["life"]
+ print(f" {life['state']:9s} fire={life['fire_num']}/{life['fire_den']} "
+ f"promoted@{life.get('promoted_at')} retired@{life.get('retired_at')} {fid}")
+ # the block that would be rendered next, from the saved state
+ from tokeneconomizer.mechanisms.envfacts import EnvFactsDistiller
+ dist = EnvFactsDistiller.load(d["repo"], args.arm)
+ block = dist.render()
+ print(" current block:" + ("\n " + block.replace("\n", "\n ") if block else " (none)"))
+ # ---- per-task telemetry from the stream: when did the block first appear ----
+ for sp in glob.glob(os.path.join(args.arm, "_streams", "*.jsonl")):
+ first = None
+ with open(sp, encoding="utf-8") as fh:
+ for i, line in enumerate(fh):
+ try:
+ row = json.loads(line)
+ except ValueError:
+ continue
+ mf = row.get("manual_facts") or {}
+ if first is None and any(isinstance(v, dict) and v.get("inject") for v in mf.values()):
+ first = i
+ print(f" first task index with an injectable fact (stream): {first}")
+
+ if args.json:
+ json.dump({"pairs": len(pairs), "baseline_resolved": br, "arm_resolved": ar,
+ "baseline_mean": st.mean(bc), "arm_mean": st.mean(ac),
+ "baseline_median": st.median(bc), "arm_median": st.median(ac),
+ "paired_delta_mean": st.mean(deltas), "paired_delta_median": st.median(deltas),
+ "flips_lost": flips_lost, "flips_won": flips_won,
+ "tasks_with_prefix": with_prefix}, open(args.json, "w"), indent=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/export_webbench_tasks.py b/experiments/scripts/export_webbench_tasks.py
new file mode 100644
index 0000000..2712b40
--- /dev/null
+++ b/experiments/scripts/export_webbench_tasks.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""Export the Web-Bench DRIVER task list (registry A) from `workloads.WEBBENCH_TASKS`.
+
+Web-Bench needs three registries to agree, and this writes the one that is neither committed
+nor derivable at run time:
+
+ A driver task list this file $TOKENECONOMIZER_WEBBENCH_TASKS (JSON list of
+ {instance_id, problem_statement, repo})
+ B projects tree the Web-Bench checkout, $TOKENECONOMIZER_WEBBENCH_PROJECTS
+ C workload record workloads/webbench.json, via scripts/export_workloads.py
+
+`tokeneconomizer/workloads.py` is the authority for WHICH projects and tasks are in the
+sweep; A and C are both projections of it. Changing the project list therefore means three
+commands, in this order:
+
+ python scripts/webbench_tasks_from_yaml.py --all # B: yml-only projects need a jsonl
+ python scripts/export_workloads.py # C
+ python scripts/export_webbench_tasks.py # A
+
+Leaving A stale does not fail loudly -- the driver simply streams the OLD project's tasks
+into the new project's evaluator session, and the harness raises
+`session is running 'X' but the task belongs to 'Y'` partway in.
+
+ python scripts/export_webbench_tasks.py # write to $TOKENECONOMIZER_WEBBENCH_TASKS
+ python scripts/export_webbench_tasks.py --check # verify only, exit 1 if stale
+"""
+import argparse
+import json
+import os
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+
+from tokeneconomizer import workloads # noqa: E402
+
+PROJECTS_ENV = "TOKENECONOMIZER_WEBBENCH_PROJECTS"
+TASKS_ENV = "TOKENECONOMIZER_WEBBENCH_TASKS"
+
+
+def build(projects_root):
+ """The driver rows, in the workload's ORDER -- each task builds on the one before it."""
+ rows = []
+ for proj, task_ids in workloads.WEBBENCH_TASKS.items():
+ path = os.path.join(projects_root, proj, "tasks.jsonl")
+ if not os.path.isfile(path):
+ sys.exit(f"{proj}: no tasks.jsonl at {path}\n"
+ f" yml-only project? run: python scripts/webbench_tasks_from_yaml.py {proj}")
+ by_id = {}
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.strip()
+ if line:
+ rec = json.loads(line)
+ by_id[rec["id"]] = rec
+ missing = [t for t in task_ids if t not in by_id]
+ if missing:
+ sys.exit(f"{proj}: tasks.jsonl is missing {missing} named by WEBBENCH_TASKS")
+ for tid in task_ids:
+ rows.append({"instance_id": tid,
+ "problem_statement": by_id[tid].get("description", ""),
+ "repo": proj})
+ return rows
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--out", default=None, help=f"default: ${TASKS_ENV}")
+ ap.add_argument("--projects-root", default=None, help=f"default: ${PROJECTS_ENV}")
+ ap.add_argument("--check", action="store_true",
+ help="verify the file matches the registry; write nothing, exit 1 if stale")
+ args = ap.parse_args()
+
+ projects_root = args.projects_root or os.environ.get(PROJECTS_ENV)
+ if not projects_root:
+ sys.exit(f"no Web-Bench projects root: set ${PROJECTS_ENV} or pass --projects-root")
+ out = args.out or os.environ.get(TASKS_ENV)
+ if not out:
+ sys.exit(f"no output path: set ${TASKS_ENV} or pass --out")
+
+ rows = build(projects_root)
+ want = json.dumps(rows, indent=2, ensure_ascii=False) + "\n"
+ have = open(out, encoding="utf-8").read() if os.path.isfile(out) else None
+
+ projects = sorted({r["repo"] for r in rows})
+ if have == want:
+ print(f"ok {out}\n {len(rows)} tasks, {len(projects)} projects: {' '.join(projects)}")
+ return
+ if args.check:
+ sys.exit(f"STALE {out} ({'missing' if have is None else 'differs from the registry'})\n"
+ f" regenerate with: python scripts/export_webbench_tasks.py")
+ with open(out, "w", encoding="utf-8") as fh:
+ fh.write(want)
+ print(f"wrote {out}\n {len(rows)} tasks, {len(projects)} projects: {' '.join(projects)}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/export_workloads.py b/experiments/scripts/export_workloads.py
new file mode 100644
index 0000000..4db8cc3
--- /dev/null
+++ b/experiments/scripts/export_workloads.py
@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""Export the canonical workloads to ``workloads/<domain>.json`` (committed, not ignored).
+
+``tokeneconomizer/workloads.py`` is the authority; these JSON files are the machine-readable
+copy the shell launchers and any non-Python harness can read without importing the package.
+They are tiny (ids only, no issue text) and therefore **committed** — unlike the results
+tree, which is regenerated per run and git-ignored.
+
+ python scripts/export_workloads.py # rewrite workloads/*.json
+ python scripts/export_workloads.py --check # verify they match (exit 1 if stale)
+
+``tests/test_workloads.py`` runs the same comparison, so drift fails the suite.
+"""
+import argparse
+import json
+import os
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+
+from tokeneconomizer import workloads # noqa: E402
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+OUT_DIR = os.path.join(REPO, "workloads")
+
+_DOC = {
+ "swebench": "SWE-bench task streams for the Claude/Copilot trimech A/B and the Codex arms.",
+ "webbench": "Web-Bench streams; task order is load-bearing (each task builds on the last).",
+ "creddata": "CredData repositories audited for secrets, graded to line-level F1.",
+ "reposvul": "ReposVul instances (Python split) audited for vulnerabilities, graded to F1.",
+}
+_GROUP_FIELD = {"swebench": "repo", "webbench": "project",
+ "creddata": "instance", "reposvul": "instance"}
+_SEEDS = {"swebench": list(workloads.SWEBENCH_SEEDS),
+ "webbench": list(workloads.WEBBENCH_SEEDS),
+ "creddata": list(workloads.CREDDATA_SEEDS),
+ "reposvul": list(workloads.REPOSVUL_SEEDS)}
+
+
+def build(domain):
+ """The exported document for one domain."""
+ groups = {"swebench": workloads.SWEBENCH_INSTANCES,
+ "webbench": workloads.WEBBENCH_TASKS,
+ "creddata": {"creddata": workloads.CREDDATA_INSTANCES},
+ "reposvul": {"reposvul": workloads.REPOSVUL_INSTANCES}}[domain]
+ # The exported manifest is what the LAUNCHERS run, so it carries the runnable set
+ # (canonical PLUS workloads.SWEBENCH_ADDED, MINUS workloads.SWEBENCH_EXCLUDED), not the
+ # historical one. The excluded ids and their reasons ship alongside, so a short cell
+ # reads as "deliberately skipped" rather than "silently missing", and the added ids ship
+ # alongside too, so a LONGER cell reads as "deliberately extended" rather than "someone
+ # ran the wrong list". workloads.task_counts()/total_tasks() are unchanged.
+ groups = workloads.runnable_instances(domain) if domain == "swebench" else groups
+ doc = {
+ "domain": domain,
+ "description": _DOC[domain],
+ "source": "tokeneconomizer/workloads.py (regenerate: python scripts/export_workloads.py)",
+ "group_field": _GROUP_FIELD[domain],
+ "seeds": _SEEDS[domain],
+ "total_tasks": sum(len(v) for v in groups.values()),
+ "task_counts": {g: len(v) for g, v in groups.items()},
+ "groups": {g: list(v) for g, v in groups.items()},
+ }
+ if domain == "swebench" and workloads.SWEBENCH_EXCLUDED:
+ doc["excluded"] = dict(sorted(workloads.SWEBENCH_EXCLUDED.items()))
+ if domain == "swebench" and workloads.SWEBENCH_ADDED:
+ doc["added"] = {g: sorted(v) for g, v in sorted(workloads.SWEBENCH_ADDED.items())}
+ if domain == "swebench":
+ doc["full_split_only"] = sorted(workloads.SWEBENCH_FULL_SPLIT_ONLY)
+ doc["notes"] = (
+ "These ids are NOT all in SWE-bench_Verified: 'full_split_only' lists the "
+ "instances that exist only in the full princeton-nlp/SWE-bench test split. "
+ "Docker image tag: swebench/sweb.eval.x86_64.<id with __ -> _1776_>:latest. "
+ "'excluded' lists ids dropped from THIS run list with the reason; they are "
+ "still part of the recorded workload in tokeneconomizer/workloads.py "
+ "(SWEBENCH_INSTANCES) and were run by the published results, so a cell that "
+ "skips them is not directly comparable to one that did not. "
+ "'added' lists ids that EXTEND a stream beyond the published workload to "
+ "de-saturate it (scikit-learn and sympy, to 60 runnable tasks each); they are "
+ "all outside SWE-bench_Verified, so provision them from the full test split, "
+ "and results on an extended stream are not comparable to the published ones "
+ "for that repo.")
+ return doc
+
+
+def path_for(domain):
+ return os.path.join(OUT_DIR, f"{domain}.json")
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--check", action="store_true",
+ help="do not write; exit 1 if any file is missing or stale")
+ args = ap.parse_args()
+
+ os.makedirs(OUT_DIR, exist_ok=True)
+ stale = []
+ for domain in workloads.DOMAINS:
+ doc = build(domain)
+ dest = path_for(domain)
+ text = json.dumps(doc, indent=2, sort_keys=False) + "\n"
+ if args.check:
+ current = open(dest, encoding="utf-8").read() if os.path.exists(dest) else None
+ if current != text:
+ stale.append(os.path.relpath(dest, REPO))
+ continue
+ with open(dest, "w", encoding="utf-8") as fh:
+ fh.write(text)
+ print(f"wrote {os.path.relpath(dest, REPO)} "
+ f"({doc['total_tasks']} tasks, {len(doc['groups'])} groups)")
+
+ if args.check:
+ if stale:
+ print("STALE (run: python scripts/export_workloads.py):", *stale, sep="\n ")
+ return 1
+ print(f"workloads/*.json up to date ({len(workloads.DOMAINS)} domains)")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/experiments/scripts/launch/README.md b/experiments/scripts/launch/README.md
new file mode 100644
index 0000000..ce5abc6
--- /dev/null
+++ b/experiments/scripts/launch/README.md
@@ -0,0 +1,177 @@
+# scripts/launch/ — reproducible experiment launchers
+
+These scripts run the **main experiments**: a whole lane (`main_sweep.sh`, `full_sweep.sh`)
+or an A/B (baseline vs. an economized arm, `trimech_ab.sh`, `codex_ab.sh`) over a set of repos
+and seeds via [`../run_experiment.py`](../run_experiment.py), then aggregate the results into
+the index CSVs `compare_arms.py` reads. `run_experiment.py` is the single source of truth
+mapping each arm to an exact `EconomizerConfig`. The run book, with the arm catalog and one
+command per lane, is [`../../docs/RUNNING.md`](../../docs/RUNNING.md).
+
+## The harness seam (why you must supply a runner)
+
+Task execution — launching a sealed agent (Claude Code / Copilot CLI / Codex) inside its
+Docker image, applying the economizer's per-task decisions, and grading against the gold
+patch — lives **below `tokeneconomizer.runner.AgentRunner`**, in
+[`../../te_harness`](../../te_harness) (details: [`../../docs/HARNESS.md`](../../docs/HARNESS.md)).
+Select it per launch; no `PYTHONPATH` is needed, since the driver puts `experiments/` and
+the repository root (the shared core) on `sys.path` itself:
+
+```
+RUNNER=te_harness.swebench:make_runner # SWE-bench
+RUNNER=te_harness.webbench:make_runner # Web-Bench
+RUNNER=your_package.harness:make_runner # or your own: any zero-arg callable -> AgentRunner
+```
+
+`RUNNER=stub` (the default) selects the built-in offline `StubRunner`: it runs **no agent**,
+writes a synthetic transcript, and returns a fixed cache-heavy row. It exists only to
+exercise the wiring end-to-end (facade → aggregate) with no Docker, no datasets,
+and no LLM — its numbers are **not** a benchmark. `smoke_test.sh` uses it.
+
+## Arm → exact configuration
+
+| Arm | Original flags | Applies to |
+|---|---|---|
+| `baseline` | *(none)* | all — native cost basis |
+| `trimech` | `--mode manual4 --untool-evidence 3 --untool-judge --spiral-guard --spiral-judge --spiral-nudge` | Claude / Copilot |
+| `trimech-web` | `trimech` + `--spiral-nudge-llm --spiral-frozen-envelope 30` | Claude / Copilot on short Web-Bench streams |
+| `codex-untool` | `--codex-recent-adaptive-untool-v7` | Codex |
+| `codex-spiral` | `--codex-spiral-resume`, the **defensive** profile | Codex |
+| `codex-inject` | `--mode manual4` on the composed `envfacts+distiller` profile (**paid**) | Codex |
+| `codex-trimech` | `codex-inject` + `codex-untool` + `codex-spiral` — all three (**paid**) | Codex |
+| `codex-trimech-rtk` | `codex-trimech` + rtk (instruction mode; row carries `rtk_compliance`) | Codex |
+| `codex-trimech-repomap` | `codex-trimech` + aider's repo map as the project `AGENTS.md` (SWE-bench only; row carries `map_ok`) | Codex |
+
+`rtk`, `repomap` and `serena` all run on the Codex lane: rtk in its instruction mode, serena
+as a `[mcp_servers.serena]` block in the per-task `config.toml` (rows carry `serena_tool_calls`).
+
+The Codex arms run on **both** domains: `codex_ab.sh` takes `DOMAIN=swebench` (default) or
+`DOMAIN=webbench`, and refuses a `RUNNER` that does not match it.
+
+## Detached by default
+
+Both launchers **run in the background by default**. A real sweep is hours of Docker and
+paid agent time; a closed laptop or a dropped ssh session should not cost you the run. The
+launcher validates its config and credentials first, then re-execs itself under `setsid`
+and returns your terminal:
+
+```
+[detach] trimech_ab__claude_swebench_trimech started in the background (DETACH=0 to run in the foreground)
+[detach] log: /data/te_results/logs/trimech_ab__claude_swebench_trimech__20260814-144811.log
+[detach] follow: tail -f /data/te_results/logs/trimech_ab__claude_swebench_trimech__20260814-144811.log
+[detach] stop: kill -TERM -$(cat /data/te_results/logs/trimech_ab__claude_swebench_trimech__20260814-144811.pid)
+```
+
+Because the run gets its own session, it has no controlling terminal and survives SIGHUP.
+Everything it spawns — driver, `docker exec`, the agent — shares one process group, so the
+`kill -TERM -<pid>` above (note the leading `-`: that is the *group*) stops the whole tree.
+The harness also traps TERM/INT/HUP and removes the task container it is holding, since
+containers live outside that process group and would otherwise keep running — and keep
+billing. Logs and pidfiles land under `$TOKENECONOMIZER_RESULTS/logs/`; override with
+`TE_LOG=/path/run.log`.
+
+`DETACH=0` runs in the foreground (streaming to your terminal) — the right choice for a
+quick `LIMIT=1` check or when driving the launcher from another script. `smoke_test.sh` is
+always foreground; it takes seconds and asserts its own result.
+
+## Credentials (`.env`)
+
+Real (non-stub) runs drive each agent CLI on its own native authentication. Copy the
+tracked template and fill it in — `.env` is git-ignored:
+
+```bash
+cp .env.example .env # then set the key for the agent you run
+```
+
+Both launchers source [`_env.sh`](_env.sh) and `run_experiment.py` calls
+[`../_env.py`](../_env.py)`.load_env()`, so the file is picked up automatically. Format is
+plain `KEY=value` (`#` comments, optional `export`, no `$VAR` interpolation); **variables
+already exported in your shell win over the file**, and `KEY=` counts as unset. Point at a
+different file with `TOKENECONOMIZER_ENV_FILE=/path/to/other.env`.
+
+| Agent | Credential |
+|---|---|
+| Claude Code | `ANTHROPIC_API_KEY`. `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` are ignored: the launcher says so once and unsets them. |
+| Codex | `OPENAI_API_KEY`. `OPENAI_BASE_URL` only for a region-scoped key (e.g. `https://us.api.openai.com/v1`); any non-OpenAI host is refused. |
+| Copilot CLI | the GitHub subscription (`COPILOT_GITHUB_TOKEN` / `GH_TOKEN` / `GITHUB_TOKEN`, or `copilot` then `/login`), or Copilot's own BYOK (`COPILOT_PROVIDER_*`, see [`../../docs/COPILOT.md`](../../docs/COPILOT.md)). |
+
+Before each real run the launcher checks the credential for its agent and fails fast when it
+is missing.
+
+## Placeholders (env vars)
+
+> **Provisioning each benchmark** — what image to pull, dataset to fetch, and the exact
+> commands — is in [`../../docs/BENCHMARKS.md`](../../docs/BENCHMARKS.md). It explains the two
+> path families every non-SWE-bench domain needs: a **`_TASKS` JSON file** (the driver's task
+> list) *and* a **`_ROOT`/`_PROJECTS`/`_SUITE` dataset root** (the adapter's data). For
+> SWE-bench those are the same file.
+
+The launchers themselves read only these:
+
+| Env var | What it is |
+|---|---|
+| `TOKENECONOMIZER_SWEBENCH_TASKS` | JSON list of `{instance_id, problem_statement, repo}` (SWE-bench; also the adapter source) |
+| `TOKENECONOMIZER_WEBBENCH_TASKS` / `_CREDDATA_TASKS` / `_REPOSVUL_TASKS` | driver task-list JSON for those domains (a **file**, not a dir) |
+| `TOKENECONOMIZER_RESULTS` | root of the results trees (outside the repository) the aggregator reads |
+| `RUNNER` | `stub`, `te_harness.swebench:make_runner` / `.webbench:make_runner`, or any `module:factory` |
+| `WORKLOAD` | workload manifest (default `workloads/$DOMAIN.json`); `none` disables filtering |
+| `PYTHON` | interpreter to use (default `python3`) |
+
+**Workload pinning.** Both launchers default `WORKLOAD` to the committed manifest in
+[`../../workloads/`](../../workloads/) and pass it through as `run_experiment.py --workload`,
+which filters `TASKS_FILE` to exactly the evaluated instances for the `--repo` group. Your
+task file may therefore hold a superset (e.g. all of SWE-bench_Verified). This also avoids the
+legacy `--repo` prefix match, which silently falls back to the *entire* task list whenever the
+repo's org differs from its name — `pydata/xarray` included. Set `WORKLOAD=none` to opt out.
+
+The **dataset roots** (`_WEBBENCH_PROJECTS`, `_CREDDATA_ROOT`, `_REPOSVUL_ROOT`,
+`_CYBERGYM_SUITE`) are read by the adapters + your harness, not by these scripts — export them
+per [`../../docs/BENCHMARKS.md`](../../docs/BENCHMARKS.md) before launching.
+
+## Examples
+
+**Claude tri-mechanism on SWE-bench, three seeds:**
+```bash
+export TOKENECONOMIZER_SWEBENCH_TASKS=/data/swebench_verified.json
+export TOKENECONOMIZER_RESULTS=/data/te_results
+AGENT=claude DOMAIN=swebench ARM=trimech \
+ REPOS="pydata/xarray django/django sympy/sympy scikit-learn/scikit-learn" \
+ SEEDS="0 1 2" RUNNER=te_harness.swebench:make_runner \
+ bash scripts/launch/trimech_ab.sh
+```
+
+**Tri-mechanism on Web-Bench (short streams → `trimech-web`):**
+```bash
+AGENT=claude DOMAIN=webbench ARM=trimech-web SEEDS="0" \
+ REPOS="calculator expressjs react zustand table" \
+ RUNNER=te_harness.webbench:make_runner bash scripts/launch/trimech_ab.sh
+```
+
+Each `REPOS` value is a Web-Bench **project** — one stream of 20–21 sequentially dependent
+tasks (`init`, `task-1` … `task-20`). Two Web-Bench-specific notes:
+
+* Web-Bench's evaluator owns the task loop, so a Web-Bench runner is started once per stream
+ through the optional `begin_stream(tasks, ...)` / `end_stream()` hooks on `AgentRunner`
+ (see `tokeneconomizer/runner.py`). Runners that do not define them are called as usual.
+* Because the tasks are sequentially dependent and graded cumulatively, the evaluator stops
+ the project at the first task that fails every attempt. The driver still asks for the rest;
+ a runner should return those as unreached (`stream_broken` / `unreached`, zero tokens) so
+ the cost aggregation does not read them as cheap failures.
+
+**Codex untooling A/B:**
+```bash
+ARM=codex-untool REPOS="pydata/xarray sympy/sympy" SEEDS="0" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
+```
+
+**Codex spiral A/B:**
+```bash
+ARM=codex-spiral REPOS="pydata/xarray" SEEDS="0" \
+ RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
+```
+
+## Offline smoke test
+
+`bash scripts/launch/smoke_test.sh` runs **every** arm with the StubRunner on a tiny
+synthetic task set, aggregates, asserts the CSVs are produced, and **deletes everything it
+created** (it writes only under a temp dir). Use it to confirm the driver + facade +
+aggregator wiring after any change. See `../README.md` for the full reproduction chain.
diff --git a/experiments/scripts/launch/_env.sh b/experiments/scripts/launch/_env.sh
new file mode 100644
index 0000000..5301e9a
--- /dev/null
+++ b/experiments/scripts/launch/_env.sh
@@ -0,0 +1,430 @@
+# Shell half of the .env loader (sourced by the launchers; see scripts/_env.py for the
+# Python half — the two agree on the same file format).
+#
+# . "$HERE/_env.sh" # defines te_load_env / te_require_key / te_check_routing
+#
+# Format: plain `KEY=value`, optional leading `export`, `#` comments, quote values with
+# spaces. Shell-sourced, so keep the file free of command substitution.
+# Precedence: variables ALREADY exported in your shell win over the file.
+
+# Load $TOKENECONOMIZER_ENV_FILE (default: experiments/.env, else the repository root's .env)
+# without clobbering existing exports.
+te_env_file() {
+ if [[ -n "${TOKENECONOMIZER_ENV_FILE:-}" ]]; then echo "$TOKENECONOMIZER_ENV_FILE"
+ elif [[ -f "${REPO_ROOT:-.}/.env" || ! -f "${REPO_ROOT:-.}/../.env" ]]; then echo "${REPO_ROOT:-.}/.env"
+ else echo "${REPO_ROOT:-.}/../.env"; fi
+}
+te_load_env() {
+ local file; file="$(te_env_file)"
+ [[ -f "$file" ]] || return 0
+ local line key val
+ while IFS= read -r line || [[ -n "$line" ]]; do
+ line="${line#"${line%%[![:space:]]*}"}" # ltrim
+ [[ -z "$line" || "$line" == \#* || "$line" != *=* ]] && continue
+ [[ "$line" == export\ * ]] && line="${line#export }"
+ key="${line%%=*}"; val="${line#*=}"
+ key="${key%"${key##*[![:space:]]}"}" # rtrim key
+ val="${val#"${val%%[![:space:]]*}"}"; val="${val%"${val##*[![:space:]]}"}"
+ [[ "$val" == \"*\" || "$val" == \'*\' ]] && val="${val:1:${#val}-2}"
+ [[ -z "$key" || -z "$val" ]] && continue # KEY= means "not set here"
+ [[ -n "${!key:-}" ]] && continue # shell export wins
+ export "$key=$val"
+ done < "$file"
+}
+
+# te_dryrun_skips_billed <what> — true when this is a DRYRUN, so a BILLED probe must not run.
+#
+# A dry run is the gesture the run book teaches for costing a lane before spending anything
+# (docs/RUNNING.md section 2), so a probe that BILLS during one contradicts the whole
+# point: DRYRUN=1 on a Copilot lane used to spend up to three premium requests and ~240 s
+# before printing "nothing was run".
+te_dryrun_skips_billed() {
+ [[ "${DRYRUN:-0}" == "1" ]] || return 1
+ echo "[env] DRYRUN: skipping the billed $1 probe" >&2
+ return 0
+}
+
+# te_require_key <VAR> [...] — fail fast before burning Docker time on a missing credential.
+te_require_key() {
+ local missing=() k
+ for k in "$@"; do [[ -n "${!k:-}" ]] || missing+=("$k"); done
+ if (( ${#missing[@]} )); then
+ echo "!! missing credential(s): ${missing[*]}" >&2
+ echo " set them in $(te_env_file) (copy .env.example)" >&2
+ exit 2
+ fi
+}
+
+# te_detach <tag> <this-script> [args...] — re-exec the launcher DETACHED and return the
+# terminal, unless DETACH=0 or we are already the detached copy.
+#
+# A real sweep is hours of Docker + paid agent time; losing it to a closed laptop or a
+# dropped ssh session is the expensive failure. `setsid` puts the run in its own session,
+# so it survives SIGHUP, and the whole tree (launcher -> driver -> docker exec -> agent)
+# shares one process group that a single `kill -TERM -<pid>` stops cleanly.
+#
+# Call it AFTER the config + credential preflight, so a bad key or missing task file still
+# fails in the user's terminal instead of silently in a log file.
+te_detach() {
+ local tag="$1"; shift
+ [[ "${DETACH:-1}" == "0" || -n "${TE_DETACHED:-}" ]] && return 0
+ local stamp; stamp="$(date +%Y%m%d-%H%M%S)"
+ local log="${TE_LOG:-${RESULTS_ROOT:-$PWD}/logs/${tag}__${stamp}.log}"
+ mkdir -p "$(dirname "$log")" || { echo "!! cannot create log dir for $log" >&2; exit 2; }
+ local pidfile="${log%.log}.pid"
+ export TE_DETACHED=1
+ export PYTHONUNBUFFERED=1 # else the driver's progress sits in a pipe buffer
+ # The inner shell records its OWN pid (the session leader) -- $! here would be setsid's,
+ # which is not the group the run ends up in.
+ setsid nohup bash -c 'echo $$ > "$1"; shift; exec bash "$@"' _ "$pidfile" "$@" \
+ >"$log" 2>&1 &
+ disown 2>/dev/null || true
+ echo "[detach] $tag started in the background (DETACH=0 to run in the foreground)"
+ echo "[detach] log: $log"
+ echo "[detach] follow: tail -f $log"
+ echo "[detach] stop: kill -TERM -\$(cat $pidfile) # leading '-' = whole group"
+ exit 0
+}
+
+# te_tree_name <agent> <domain> [model-tag] — the results tree for one (agent, domain, model).
+#
+# `<agent>_<domain>`, plus `_<tag>` when a model tag is given: claude_swebench_opus48,
+# copilot_webbench_sonnet5, codex_swebench_gpt56sol. The MODEL is in the name because it
+# is part of the cost basis: the aggregator averages a cell, and two models in one cell
+# average two bases into a number that describes neither. Omitted (2-arg call) the name is
+# `<agent>_<domain>`.
+te_tree_name() {
+ local agent="$1" domain="$2" tag="${3:-}" name
+ name="${agent}_${domain}"
+ [[ -n "$tag" ]] && name="${name}_${tag}"
+ echo "$name"
+}
+
+# te_model_tag <model-id> — the short, filesystem-safe tag te_tree_name appends.
+#
+# `claude-opus-4-8` and `claude-opus-4.8` are the same model spelled two ways (the CLI takes
+# both; the stored rows carry both), so the tag normalizes separators away and they cannot
+# produce two trees for one model. A router directive is NOT a model: `auto`/`agent` means
+# "whatever the CLI picks per request", which is not a cost basis worth naming, so it maps
+# to the empty tag and the tree keeps its 2-arg name.
+te_model_tag() {
+ local m
+ m="$(printf '%s' "${1:-}" | tr '[:upper:]' '[:lower:]')"
+ case "$m" in
+ ""|agent|auto|copilot) echo "" ; return 0 ;;
+ esac
+ m="${m#claude-}" # opus-4-8 / sonnet-4.6
+ # `/` GOES TOO, and not merely for tidiness: a provider-prefixed id (`vendor/model`, as a
+ # Copilot BYOK provider may spell one) would otherwise turn the tree name into a nested
+ # PATH, which splits one cell across two directories and hides it from the tree walk.
+ printf '%s' "$m" | tr -d './-' # opus48 / sonnet46 / gpt56sol
+ echo
+}
+
+# te_model_alt_spelling <model-id> — the same id with its version separator swapped, or
+# nothing when the id has only one spelling.
+#
+# claude-opus-4-8 -> claude-opus-4.8 claude-haiku-4.5 -> claude-haiku-4-5
+# claude-opus-5 -> (empty) claude-sonnet-5 -> (empty)
+#
+# Two catalogs serve the same model under two spellings: the Copilot CLI's own list uses
+# a DOT (`claude-opus-4.8`, measured healthy on 63 stored subscription rows), while the
+# Anthropic API uses a DASH (`claude-opus-4-8`) and 404s the dotted form. te_model_tag folds
+# both to one tree, so switching spelling never forks a tree; only the upstream call cares.
+te_model_alt_spelling() {
+ local m="${1:-}" alt
+ if printf '%s' "$m" | grep -q '[0-9]\.[0-9]'; then
+ alt="$(printf '%s' "$m" | sed 's/\([0-9]\)\.\([0-9]\)/\1-\2/')"
+ else
+ alt="$(printf '%s' "$m" | sed 's/\([0-9]\)-\([0-9]\)/\1.\2/')"
+ fi
+ [[ "$alt" != "$m" ]] && printf '%s\n' "$alt"
+ return 0
+}
+
+# te_model_for_agent <agent> <model-id> — the spelling the agent's backend serves.
+#
+# The docs and the launchers name every model ONCE, in the Anthropic API's dashed form
+# (`claude-opus-4-8`). The one backend that spells it differently is the Copilot CLI's
+# native catalog on a subscription (`claude-opus-4.8`), so a copilot lane gets the dotted
+# form there -- and only there: under Copilot BYOK the CLI speaks the Anthropic wire API,
+# which takes the dash. Any other agent is normalised TO the dash, so a dotted id typed from
+# an older doc cannot 404 once per task. Ids with a single spelling (`claude-opus-5`,
+# `claude-sonnet-5`) pass through untouched, as does anything that is not a versioned
+# Claude id. te_check_copilot_model still VERIFIES the result against the CLI and switches
+# back if a vendor rotates a catalog, so this is a default, not a bet.
+te_model_for_agent() {
+ local agent="${1:-}" m="${2:-}" alt out
+ out="$m"
+ # TE_STRICT_MODEL_ID: send the id exactly as typed -- no re-spelling here, and the probe
+ # refuses rather than switches. For the operator who needs the wire id to be literal.
+ [[ -n "${TE_STRICT_MODEL_ID:-}" ]] && { printf '%s\n' "$m"; return 0; }
+ case "$m" in claude-*[0-9][.-][0-9]*) ;; *) printf '%s\n' "$m"; return 0 ;; esac
+ alt="$(te_model_alt_spelling "$m")"
+ case "$agent" in
+ copilot)
+ if [[ -z "${COPILOT_PROVIDER_BASE_URL:-}" ]]; then
+ printf '%s' "$m" | grep -q '[0-9]\.[0-9]' || out="$alt" # subscription: dot
+ else
+ printf '%s' "$m" | grep -q '[0-9]-[0-9]' || out="$alt" # BYOK wire: dash
+ fi ;;
+ *) printf '%s' "$m" | grep -q '[0-9]-[0-9]' || out="$alt" ;; # Anthropic API
+ esac
+ [[ -n "$out" ]] || out="$m"
+ if [[ "$out" != "$m" ]]; then
+ echo "[env] model id: '$m' -> '$out' (the spelling this agent's backend serves; the" >&2
+ echo "[env] tree tag is $(te_model_tag "$m") either way)" >&2
+ fi
+ printf '%s\n' "$out"
+}
+
+# te_drop_anthropic_endpoint — ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN are not inputs.
+#
+# Claude Code bills the Anthropic API on ANTHROPIC_API_KEY, and nothing here selects another
+# endpoint. Claude Code itself still honours both variables wherever it inherits them, so a
+# value left in an older .env or the shell is removed from this launcher's environment --
+# nothing it starts sees it -- and said once, rather than silently.
+te_drop_anthropic_endpoint() {
+ local set=() k
+ for k in ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN; do
+ [[ -n "${!k:-}" ]] && set+=("$k")
+ done
+ (( ${#set[@]} )) || return 0
+ echo "[env] ${set[*]} set but IGNORED: Claude Code bills the Anthropic API on" >&2
+ echo "[env] ANTHROPIC_API_KEY. Unset for this run; remove from $(te_env_file)." >&2
+ unset "${set[@]}"
+}
+
+# te_check_openai_base_url — OPENAI_BASE_URL may only name OpenAI's own API.
+#
+# It exists for a key issued against a regional endpoint (https://us.api.openai.com/v1),
+# which 401s on the global one. Any other host would bill something that is not the OpenAI
+# API under a tree that says it is, so it is refused at launch.
+te_check_openai_base_url() {
+ local url="${OPENAI_BASE_URL:-}" host
+ [[ -n "$url" ]] || return 0
+ host="${url#*://}"; host="${host%%/*}"; host="${host%%:*}"
+ host="$(printf '%s' "$host" | tr '[:upper:]' '[:lower:]')"
+ [[ "$url" == https://* && ( "$host" == "api.openai.com" || "$host" == *.api.openai.com ) ]] \
+ && return 0
+ echo "!! OPENAI_BASE_URL=$url is not an OpenAI API endpoint." >&2
+ echo " It is only for a regional OpenAI key (https://<region>.api.openai.com/v1);" >&2
+ echo " unset it for a global key." >&2
+ exit 2
+}
+
+# te_check_routing <agent> — say what this lane bills, and preflight the credential it needs.
+#
+# claude the Anthropic API on ANTHROPIC_API_KEY
+# codex the OpenAI API on OPENAI_API_KEY (OPENAI_BASE_URL only for a regional key)
+# copilot the GitHub Copilot subscription, or Copilot's own BYOK (COPILOT_PROVIDER_*)
+te_check_routing() {
+ case "$1" in
+ claude) te_drop_anthropic_endpoint ;;
+ codex) te_check_openai_base_url ;;
+ copilot) ;;
+ *) echo "!! unknown agent '$1' (claude | copilot | codex)" >&2; exit 2 ;;
+ esac
+ # The stub writes plausible rows with a FIXED resolve pattern and no agent behind them.
+ # Its output is indistinguishable from a real sweep once it is sitting in results/, so say
+ # so at launch rather than letting synthetic numbers be mistaken for a benchmark.
+ if [[ "${RUNNER:-stub}" == "stub" ]]; then
+ echo "[env] RUNNER=stub -- SYNTHETIC run: no agent, no grading, numbers are NOT a benchmark" >&2
+ echo "[env] set RUNNER=<module>:<factory> for a real run" >&2
+ return 0
+ fi
+ case "$1" in
+ copilot)
+ # The Copilot CLI authenticates to GitHub, NOT to Anthropic: it reads a token from
+ # COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN, or the credentials `copilot` itself
+ # wrote under COPILOT_HOME. Demanding ANTHROPIC_API_KEY here would refuse to start on
+ # a box perfectly able to run the lane (te_harness/copilot.py says the same about the
+ # runner). te_check_copilot_auth prints the routing line and gives the better error:
+ # it checks the binary, the token, and probes the CLI, exiting 2 with the remedy.
+ te_check_copilot_auth ;;
+ claude)
+ te_require_key ANTHROPIC_API_KEY
+ echo "[env] routing: Anthropic API (key ...${ANTHROPIC_API_KEY: -4})" ;;
+ codex)
+ te_require_key OPENAI_API_KEY
+ echo "[env] routing: OpenAI API at ${OPENAI_BASE_URL:-https://api.openai.com/v1} (key ...${OPENAI_API_KEY: -4})" ;;
+ esac
+}
+
+# te_copilot_refused <probe-output> — did the CLI refuse the probe outright?
+#
+# Both checks below used to key on the ABSENCE of one known error string, which meant any
+# other refusal read as success. A spent subscription is exactly that case and it is not
+# rare on a long sweep: the CLI prints "You have exceeded your monthly quota", the JSON
+# stream carries {"errorType":"quota","statusCode":402,"errorCode":"quota_exceeded"}, and
+# the auth probe still EXITS 0 -- so the launcher printed `token warm` and `model serves`
+# on an account that could not run a single task, and the lane then filled its cells with
+# turns=1 / eff_tok=0 rows that nothing aborts. Keyed on the vendor's own error text and
+# structured code, so it covers both the JSON and the plain-text probe shapes.
+te_copilot_refused() {
+ local out="$1"
+ printf '%s' "$out" | grep -qiE 'quota_exceeded|exceeded your monthly quota|"errorType"[[:space:]]*:[[:space:]]*"quota"|"statusCode"[[:space:]]*:[[:space:]]*402'
+}
+
+te_copilot_refused_die() {
+ local out="$1" where="$2"
+ te_copilot_refused "$out" || return 0
+ echo "!! the Copilot subscription REFUSED the $where probe: monthly quota exceeded (HTTP 402)." >&2
+ echo " Nothing aborts a lane on this, so launching now would fill cells with" >&2
+ echo " turns=1 / eff_tok=0 rows." >&2
+ echo " Wait for the quota to reset, or run the Copilot CLI on your own provider:" >&2
+ echo " COPILOT_PROVIDER_TYPE=anthropic COPILOT_PROVIDER_BASE_URL=https://api.anthropic.com \\" >&2
+ echo " COPILOT_PROVIDER_API_KEY=... (docs/COPILOT.md, BYOK)" >&2
+ exit 2
+}
+
+# te_check_copilot_auth — the GitHub Copilot CLI is signed in and its token is warm.
+#
+# Copilot does NOT use ANTHROPIC_API_KEY; it authenticates against GitHub from ~/.copilot
+# (or COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN). Probing for on-disk artifacts is not
+# enough: a ~/.copilot/session-state directory survives a sign-out, so a stale box would
+# pass and then fail every task with "No model backend available".
+#
+# TWO attempts, because one is not conclusive: the short-lived token is refreshed into
+# ~/.copilot/config.json only when a copilot process RUNS on the host, so on a box that has
+# been idle a while the FIRST invocation can report "No authentication information found"
+# and warm the token as a side effect. The probe therefore doubles as the warm-up -- and
+# for a long unattended sweep you still want the keepalive loop (docs/COPILOT.md).
+te_check_copilot_auth() {
+ local bin="${COPILOT_HOST_BIN:-$HOME/.local/bin/copilot}" out ok=0
+ command -v "$bin" >/dev/null 2>&1 || [[ -x "$bin" ]] || bin="$(command -v copilot || true)"
+ if [[ -z "$bin" ]]; then
+ echo "!! copilot CLI not found." >&2
+ echo " install: curl -fsSL https://gh.io/copilot-install | bash" >&2
+ echo " or set COPILOT_HOST_BIN to its path" >&2
+ exit 2
+ fi
+ # BYOK: COPILOT_PROVIDER_BASE_URL points the CLI at a custom provider, which it bills
+ # INSTEAD of the subscription -- GitHub auth is then not required at all
+ # (`copilot help providers`). The sweep's default is still the subscription; this branch
+ # exists for a host without one. Say so plainly, because the cost basis differs and
+ # run_config.json records the provider URL rather than "github-copilot".
+ if [[ -n "${COPILOT_PROVIDER_BASE_URL:-}" ]]; then
+ echo "[env] routing: Copilot CLI BYOK -> ${COPILOT_PROVIDER_TYPE:-openai} at $COPILOT_PROVIDER_BASE_URL (GitHub auth not used)"
+ if [[ -z "${COPILOT_PROVIDER_API_KEY:-}${COPILOT_PROVIDER_BEARER_TOKEN:-}" ]]; then
+ echo "!! BYOK is active but neither COPILOT_PROVIDER_API_KEY nor COPILOT_PROVIDER_BEARER_TOKEN is set." >&2
+ echo " a remote provider will refuse every task; unset COPILOT_PROVIDER_BASE_URL to go back to the subscription." >&2
+ fi
+ return 0
+ fi
+ if [[ -n "${COPILOT_GITHUB_TOKEN:-}${GH_TOKEN:-}${GITHUB_TOKEN:-}" ]]; then
+ echo "[env] routing: native Copilot (token from the environment)"
+ return 0
+ fi
+ if te_dryrun_skips_billed "copilot auth"; then
+ echo "[env] routing: native Copilot (on-disk GitHub auth, not probed on a dry run)"
+ return 0
+ fi
+ local try
+ for try in 1 2; do
+ out="$(timeout 120 "$bin" -p "ok" --no-color --output-format json \
+ --log-level none 2>&1)"
+ if ! printf '%s' "$out" | grep -qi "No authentication information found"; then
+ ok=1; break
+ fi
+ [[ "$try" == 1 ]] && echo "[env] cold copilot token; retrying once to let it refresh..." >&2
+ done
+ if (( ok != 1 )); then
+ echo "!! the copilot CLI is not signed in on this host." >&2
+ echo " run 'copilot' then '/login' (OAuth device flow), or" >&2
+ echo " export COPILOT_GITHUB_TOKEN (or GH_TOKEN / GITHUB_TOKEN)." >&2
+ exit 2
+ fi
+ te_copilot_refused_die "$out" "auth"
+ echo "[env] routing: native Copilot (on-disk GitHub auth, token warm)"
+}
+
+# te_copilot_model_unserved <probe-output> — did the CLI say the model does not exist?
+# Subscription: "Model 'x' is not available". BYOK: "Model 'x' not found on provider at
+# <url> (HTTP 404)". Anything else is not this failure.
+te_copilot_model_unserved() {
+ printf '%s' "$1" | grep -qiE "is not available|not found on provider|\(HTTP 404\)"
+}
+
+# te_check_copilot_model <model> <effort> — the pinned model exists AND accepts this effort.
+#
+# Both mistakes fail PER TASK, not at launch: the CLI exits before its first turn, so a
+# sweep would write a whole tree of 0-turn rows that look like the agent tried and failed.
+# One probe up front turns that into a launch-time error.
+#
+# The two cases are not equally fatal, so they are not treated equally:
+# * model not served -> hard fail; nothing can run.
+# * model rejects effort -> WARN and export TE_EFFORT_APPLIES=0. The run is still valid,
+# it just cannot control reasoning effort, and the rows record
+# effort: null so nobody later assumes it was held fixed.
+te_check_copilot_model() {
+ local model="$1" effort="$2" bin out
+ # 'auto' (the router) never takes an effort level; the runner already records null.
+ [[ -z "$model" || "$model" == "auto" ]] && return 0
+ te_dryrun_skips_billed "copilot model/effort" && return 0
+ # Idempotent, but keyed to the MODEL it was decided for: te_detach re-execs this script
+ # and the exports below survive into the detached copy, so without this the probe is paid
+ # for twice per launch -- while a bare "already set" check would let a stale value from a
+ # previous launch with a DIFFERENT model silently skip validation.
+ [[ "${TE_EFFORT_APPLIES_FOR:-}" == "$model" && -n "${TE_EFFORT_APPLIES:-}" ]] && return 0
+ bin="${COPILOT_HOST_BIN:-$HOME/.local/bin/copilot}"
+ [[ -x "$bin" ]] || bin="$(command -v copilot || true)"
+ [[ -n "$bin" ]] || return 0 # te_check_copilot_auth already reported this
+ out="$(timeout 120 "$bin" -p "ok" --no-color --log-level none \
+ --model "$model" ${effort:+--effort "$effort"} 2>&1)"
+ # Before the model verdict: a refused probe says nothing about the model, and reading it
+ # as "serves" is how a spent quota reached the run banner as all-green.
+ te_copilot_refused_die "$out" "model"
+ # Two "not served" shapes, one per billing path. The subscription says "is not
+ # available"; a BYOK provider says "Model 'x' not found on provider at <url> (HTTP 404)".
+ # Keying on the first alone let a BYOK lane print `serves; effort accepted` for an id the
+ # provider 404s, then write a tree of turns=1 / eff_tok=0 rows (measured 2026-09-12).
+ if te_copilot_model_unserved "$out"; then
+ # The same model under the other separator, once: the Copilot CLI's own catalog spells
+ # the older Opus `claude-opus-4.8` while an Anthropic-wire BYOK provider takes
+ # `claude-opus-4-8`. te_model_for_agent already picked the likely one; this is the
+ # verified fallback for when a catalog moves. One more billed probe, only on failure.
+ local alt out2
+ alt="$(te_model_alt_spelling "$model")"
+ if [[ -n "$alt" && -z "${TE_STRICT_MODEL_ID:-}" ]]; then
+ case "$model" in claude-*) ;; *) alt="" ;; esac
+ else
+ alt=""
+ fi
+ if [[ -n "$alt" ]]; then
+ out2="$(timeout 120 "$bin" -p "ok" --no-color --log-level none \
+ --model "$alt" ${effort:+--effort "$effort"} 2>&1)"
+ te_copilot_refused_die "$out2" "model"
+ if ! te_copilot_model_unserved "$out2"; then
+ echo "[env] !! copilot does not serve '$model' but DOES serve '$alt' -- the same model," >&2
+ echo "[env] !! other separator. Using '$alt' for this lane (tree tag $(te_model_tag "$model")" >&2
+ echo "[env] !! either way; TE_STRICT_MODEL_ID=1 to refuse instead)." >&2
+ MODEL="$alt"; export MODEL
+ model="$alt"; out="$out2"
+ fi
+ fi
+ fi
+ if te_copilot_model_unserved "$out"; then
+ if [[ -n "${COPILOT_PROVIDER_BASE_URL:-}" ]]; then
+ echo "!! copilot model '$model' is not served by the BYOK provider at $COPILOT_PROVIDER_BASE_URL." >&2
+ echo " $(printf '%s' "$out" | grep -i 'not found on provider' | head -1)" >&2
+ echo " Under COPILOT_PROVIDER_TYPE=anthropic the provider takes Anthropic's own ids" >&2
+ echo " (claude-opus-4-8, claude-sonnet-5, claude-opus-5); pin one it serves." >&2
+ else
+ echo "!! copilot model '$model' is not available on this subscription." >&2
+ echo " pick one it serves, or set COPILOT_MODEL=auto (which cannot apply an" >&2
+ echo " effort level -- see docs/COPILOT.md step 2)." >&2
+ fi
+ exit 2
+ fi
+ if printf '%s' "$out" | grep -qi "does not support reasoning effort"; then
+ echo "[env] WARNING: model '$model' does not accept --effort $effort." >&2
+ echo "[env] running WITHOUT an effort level; rows record effort: null, so these" >&2
+ echo "[env] numbers are NOT comparable to effort-pinned runs." >&2
+ export TE_EFFORT_APPLIES=0 TE_EFFORT_APPLIES_FOR="$model"
+ return 0
+ fi
+ export TE_EFFORT_APPLIES=1 TE_EFFORT_APPLIES_FOR="$model"
+ echo "[env] copilot model '$model' serves; effort=${effort:-<unset>} accepted"
+
+}
diff --git a/experiments/scripts/launch/codex_ab.sh b/experiments/scripts/launch/codex_ab.sh
new file mode 100644
index 0000000..31cb053
--- /dev/null
+++ b/experiments/scripts/launch/codex_ab.sh
@@ -0,0 +1,251 @@
+#!/usr/bin/env bash
+# Reproduce a Codex-native A/B: a BASELINE arm and one of the two independently-effective
+# Codex mechanisms (or their combo), then aggregate into the index CSVs compare_arms.py reads.
+#
+# Exact arm configurations (the full catalog is in docs/RUNNING.md section 5):
+# baseline : native Codex cost basis, no economizer
+# codex-untool : --codex-recent-adaptive-untool-v7 (evidence-gated capability untooling;
+# deferred_wait is CORE and never demoted)
+# codex-spiral : --codex-spiral-resume, DEFENSIVE profile since 2026-09-12 (every
+# deterministic spiral signature as a same-thread nudge, no abort, no
+# effort change). `codex-spiral-defensive` is an alias of it.
+# codex-spiral-postgreen
+# : the shipped post-green-v3 finalize alone (resolve-safe finalize over
+# cleared scopes) -- the named ablation of the profile above
+# codex-combo : both co-activated (--codex-untool-spiral-combo)
+# codex-trimech : ALL THREE Codex mechanisms -- injection (envfacts+distiller) +
+# capability untooling + the defensive spiral. The Codex sibling of
+# `trimech`; a PAID arm (the proposer bills).
+# rtk / codex-trimech-rtk : rtk in its Codex INSTRUCTION mode (AGENTS.md + RTK.md under
+# CODEX_HOME, inlined; no hook -- Codex honours none). Rows carry
+# rtk_compliance: zero compliance means the arm ran native.
+# codex-trimech-repomap : codex-trimech + aider's repo map as the project AGENTS.md
+# (SWE-bench only). Rows carry map_ok / map_chars: map_ok false means
+# the task ran as plain codex-trimech.
+#
+# The heavy harness lives below tokeneconomizer.runner.AgentRunner, in the repo's te_harness
+# package (RUNNER=te_harness.swebench:make_runner, no PYTHONPATH needed). Point RUNNER at
+# another factory ("module:make_runner") to substitute one; RUNNER=stub uses the offline
+# StubRunner (synthetic).
+#
+# Usage:
+# ARM=codex-untool REPOS="pydata/xarray" SEEDS="0" \
+# RUNNER=te_harness.swebench:make_runner \
+# TOKENECONOMIZER_SWEBENCH_TASKS=/path/tasks.json TOKENECONOMIZER_RESULTS=/path/results \
+# bash scripts/launch/codex_ab.sh
+#
+# Env:
+# ARM codex-untool | codex-spiral | codex-spiral-postgreen | codex-spiral-defensive |
+# codex-combo | codex-trimech | codex-inject | codex-inject-untool |
+# codex-inject-envfacts | codex-inject-distiller (default codex-untool)
+#
+# codex-inject is the composed `envfacts+distiller` profile since 2026-09-12 and
+# is therefore a PAID arm (the proposer bills; the endpoint preflight fires).
+# codex-inject-envfacts is the LLM-free ablation; codex-inject-distiller is the
+# old name of the default and resolves to the same configuration.
+# DOMAIN swebench (default) | webbench. Web-Bench runs the agent on the HOST with no
+# per-task container, so the Codex lane is guarded there by an /etc/hosts block in
+# an unprivileged user+mount namespace rather than by a hook the CLI ignores; the
+# runner probes that at construction and refuses to pretend otherwise. Selects the
+# tasks file, the workload manifest and the RUNNER default.
+# BASELINE 1 = run baseline too (paired A/B, default); 0 = run only $ARM against a
+# baseline already stored elsewhere; only = run ONLY the baseline arm (a fresh
+# baseline lane to pair with an arm running concurrently in another lane)
+# REPOS space-separated repo slugs (default "pydata/xarray")
+# SEEDS space-separated seeds (default "0")
+# LIMIT cap tasks per (repo,seed), 0 = all (default 0)
+# RUNNER 'stub' or 'module:factory' (default stub)
+# TASKS_FILE task-spec JSON (default TOKENECONOMIZER_SWEBENCH_TASKS)
+# WORKLOAD workload manifest, or 'none' (default workloads/swebench.json)
+# DETACH 1 = run in the background (default), 0 = foreground
+# SKIP_DONE 1 = skip a cell that already holds its full row count (default 1), so a lane
+# that died part way RESUMES instead of re-paying for the cells it finished --
+# the same contract main_sweep.sh gives the Claude/Copilot lanes
+# ON_FAIL continue | abort (default continue: one bad cell must not take a 360-cell
+# lane with it; the lane still exits nonzero and lists the failures)
+# DRYRUN 1 = print the cells and exit without running anything
+# TE_LOG log path when detached (default $TOKENECONOMIZER_RESULTS/logs/<tag>__<ts>.log)
+# TOKENECONOMIZER_RESULTS results root (default ./results)
+# TE_FLEET_CPUS CPU budget for EVERY lane's task containers together, in cores
+# (default 16; 0/none = do not enforce). Written once, at launch, onto the
+# cgroup every rootless-docker container scope shares, so it also binds
+# lanes that are already running -- see fleet_budget.sh for why a
+# per-container cap is not enough when several lanes run at once.
+# TE_SWEBENCH_MAX_CPUS per-CONTAINER cap and the matching OMP/MKL/joblib thread caps
+# (default 4 here, against the harness default of 16): it is what keeps a
+# parallel test run from sizing its worker pool to the whole host and then
+# thrashing inside the fleet budget.
+# OPENAI_API_KEY the OpenAI API key Codex bills (required for a real run)
+# OPENAI_BASE_URL only for a key issued against a regional OpenAI endpoint
+# (https://<region>.api.openai.com/v1); any other host is refused
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$HERE/../.." && pwd)"
+PY="${PYTHON:-python3}"
+
+# Credentials + data roots from the git-ignored repo-root .env (shell exports win).
+# shellcheck source=./_env.sh
+. "$HERE/_env.sh"
+# shellcheck source=./fleet_budget.sh
+. "$HERE/fleet_budget.sh"
+te_load_env
+
+# A lane is one container at a time, but several lanes are several containers, and the
+# per-container default (16) is a ceiling per lane rather than for the host. Pick a
+# per-container cap that keeps a parallel test suite from oversubscribing, and let the
+# fleet cgroup cap below hold the aggregate.
+export TE_SWEBENCH_MAX_CPUS="${TE_SWEBENCH_MAX_CPUS:-4}"
+
+ARM="${ARM:-codex-untool}"
+case "$ARM" in baseline|baseline-medium|baseline-low|repomap|rtk|serena|codex-untool|codex-spiral|codex-spiral-postgreen|codex-spiral-defensive|codex-combo|codex-trimech|codex-trimech-rtk|codex-trimech-repomap|codex-inject|codex-inject-untool|codex-inject-envfacts|codex-inject-distiller) ;; *)
+ echo "ARM must be baseline | baseline-medium | baseline-low | repomap | rtk | serena | codex-untool | codex-spiral | codex-spiral-postgreen | codex-spiral-defensive | codex-combo | codex-trimech | codex-trimech-rtk | codex-trimech-repomap | codex-inject | codex-inject-untool | codex-inject-envfacts | codex-inject-distiller (got $ARM)"; exit 2 ;; esac
+# BASELINE=0 runs ONLY $ARM, for a lane whose baseline already exists in a stored tree
+# (compare with scripts/compare_arms.py). BASELINE=only runs ONLY the baseline arm, so a
+# fresh baseline and an arm can run as two concurrent lanes into the same tree. Default 1 =
+# the paired A/B this script is named for.
+BASELINE="${BASELINE:-1}"
+case "$BASELINE" in
+ 0) ARMS_TO_RUN=("$ARM") ;;
+ only) ARMS_TO_RUN=(baseline) ;;
+ *) ARMS_TO_RUN=(baseline "$ARM") ;;
+esac
+LANE_TAG="$([[ "$BASELINE" == "only" ]] && echo baseline || echo "$ARM")"
+REPOS="${REPOS:-pydata/xarray}"
+SEEDS="${SEEDS:-0}"
+LIMIT="${LIMIT:-0}"
+RUNNER="${RUNNER:-stub}"
+SKIP_DONE="${SKIP_DONE:-1}"
+ON_FAIL="${ON_FAIL:-continue}"
+# Probed 2026-09-09: `gpt-5.6-sol` is listed and ACCEPTED on /v1/responses by an OpenAI
+# key. Do not pin a `-codex` ALIAS (gpt-5.1-codex, ...): /v1/models lists them and
+# /v1/responses 404s them, which is a per-task failure rather than a launch error.
+AGENT=codex
+MODEL="${MODEL:-${CODEX_MODEL:-gpt-5.6-sol}}"
+te_check_routing "$AGENT"
+
+DOMAIN="${DOMAIN:-swebench}"
+case "$DOMAIN" in
+ swebench) DOMAIN_TASKS="${TOKENECONOMIZER_SWEBENCH_TASKS:-}" ;;
+ webbench) DOMAIN_TASKS="${TOKENECONOMIZER_WEBBENCH_TASKS:-}" ;;
+ *) echo "!! DOMAIN must be swebench|webbench (got '$DOMAIN')" >&2; exit 2 ;;
+esac
+TASKS_FILE="${TASKS_FILE:-$DOMAIN_TASKS}"
+if [[ -z "$TASKS_FILE" ]]; then
+ echo "!! no task-spec JSON: set TASKS_FILE or TOKENECONOMIZER_${DOMAIN^^}_TASKS" >&2
+ exit 2
+fi
+# The runner must match the domain: a swebench factory on a webbench task list would run the
+# wrong lifecycle against the right ids.
+if [[ "${RUNNER:-stub}" != "stub" ]]; then
+ case "$RUNNER" in *"$DOMAIN"*) ;; *)
+ echo "!! RUNNER=$RUNNER does not match DOMAIN=$DOMAIN (expected te_harness.$DOMAIN:make_runner)" >&2
+ exit 2 ;;
+ esac
+fi
+
+# Canonical workload manifest (committed, tiny): pins the run to exactly the evaluated
+# instances. WORKLOAD=none falls back to whatever TASKS_FILE contains.
+WORKLOAD="${WORKLOAD:-$REPO_ROOT/workloads/$DOMAIN.json}"
+WORKLOAD_ARGS=()
+if [[ "$WORKLOAD" != "none" ]]; then
+ if [[ ! -f "$WORKLOAD" ]]; then
+ echo "!! workload manifest not found: $WORKLOAD (set WORKLOAD=none to run unfiltered)" >&2
+ exit 2
+ fi
+ WORKLOAD_ARGS=(--workload "$WORKLOAD")
+fi
+
+RESULTS_ROOT="${TOKENECONOMIZER_RESULTS:-$REPO_ROOT/results}"
+# codex_<domain>_<model-tag>. The MODEL is part of the cost basis, so it belongs in the
+# tree name -- otherwise two models' lanes share a tree and aggregate.py averages two bases.
+MODEL_TAG="$(te_model_tag "$MODEL")"
+DEST="$RESULTS_ROOT/$(te_tree_name "$AGENT" "$DOMAIN" "$MODEL_TAG")"
+mkdir -p "$DEST"
+
+# DRYRUN=1 prints the cells and exits WITHOUT detaching -- before te_detach, so a dry run
+# cannot hand back the terminal and then bill a full A/B in the background.
+if [[ "${DRYRUN:-0}" == "1" ]]; then
+ echo "[codex_ab] DRYRUN -- agent=$AGENT model=$MODEL"
+ echo "[codex_ab] dest=$DEST"
+ echo "[codex_ab] the cells this lane would run, in order:"
+ for seed in $SEEDS; do for repo in $REPOS; do for arm in "${ARMS_TO_RUN[@]}"; do
+ printf ' %s\t%s\t%s\n' "$arm" "$repo" "$seed"
+ done; done; done
+ echo "[codex_ab] DRYRUN: nothing was run."
+ exit 0
+fi
+
+# Detach by default: config is resolved and credentials are checked, so anything that
+# fails from here on is a run-time failure that belongs in the log, not the terminal.
+te_detach "codex_ab__${LANE_TAG}" "$0" "$@"
+
+# One budget for every lane on the host, re-asserted here so a launch cannot inherit an
+# uncapped cgroup from an earlier session. Idempotent; never fatal.
+te_fleet_cpu_cap "${TE_FLEET_CPUS:-16}"
+te_fleet_cpu_report
+
+echo "[codex_ab] domain=$DOMAIN agent=$AGENT model=$MODEL arms=(${ARMS_TO_RUN[*]}) repos='$REPOS' seeds='$SEEDS' runner=$RUNNER dest=$DEST"
+echo "[codex_ab] cpus: per-container ${TE_SWEBENCH_MAX_CPUS}, fleet ${TE_FLEET_CPUS:-16}"
+echo "[codex_ab] tasks=$TASKS_FILE workload=$WORKLOAD"
+echo "[codex_ab] skip_done=$SKIP_DONE on_fail=$ON_FAIL"
+
+# expected_rows <repo> -- the task count of that group, from the committed workload record.
+# Same source main_sweep.sh uses: the counts are per GROUP and carry no arm or agent, so
+# the Codex lane reads them unchanged.
+declare -A GROUP_N
+while IFS=$'\t' read -r g n; do GROUP_N["$g"]="$n"; done < <(
+ "$PY" "$HERE/../sweep_plan.py" --domain "$DOMAIN" --format counts)
+
+# cell_rows <arm> <repo> <seed> -- how many row.json files that cell already holds.
+cell_rows() {
+ local tag="${2##*/}"; tag="${tag//-/_}"
+ local dir="$DEST/$1/${tag}_s$3"
+ [[ -d "$dir" ]] || { echo 0; return; }
+ find "$dir" -name row.json 2>/dev/null | wc -l
+}
+
+FAILED=()
+SKIPPED=0
+RAN=0
+START=$SECONDS
+for seed in $SEEDS; do
+ for repo in $REPOS; do
+ for arm in "${ARMS_TO_RUN[@]}"; do
+ want="${GROUP_N[$repo]:-0}"
+ if [[ "$LIMIT" != "0" && "$want" -gt "$LIMIT" ]]; then want="$LIMIT"; fi
+ have="$(cell_rows "$arm" "$repo" "$seed")"
+ if [[ "$SKIP_DONE" == "1" && "$want" -gt 0 && "$have" -ge "$want" ]]; then
+ echo "[codex_ab] SKIP $arm $repo s$seed ($have/$want rows already stored)"
+ SKIPPED=$((SKIPPED + 1))
+ continue
+ fi
+ echo "[codex_ab] RUN $arm $repo s$seed (have $have of $want)"
+ if "$PY" "$HERE/../run_experiment.py" \
+ --arm "$arm" --agent "$AGENT" --repo "$repo" --seed "$seed" --limit "$LIMIT" \
+ --adapter "$DOMAIN" --tasks-file "$TASKS_FILE" "${WORKLOAD_ARGS[@]}" \
+ --results-root "$DEST" --runner "$RUNNER" \
+ --model "$MODEL" --optimizer-model agent --max-turns 80 --fresh; then
+ RAN=$((RAN + 1))
+ else
+ echo "[codex_ab] FAILED arm=$arm repo=$repo seed=$seed"
+ FAILED+=("$arm/$repo/s$seed")
+ [[ "$ON_FAIL" == "abort" ]] && { echo "[codex_ab] ON_FAIL=abort -- stopping"; break 3; }
+ fi
+ done
+ done
+done
+
+# Aggregate every arm PRESENT in the tree, not just this lane's: --arms limits the rewrite,
+# so passing it would drop the other arms' rows from the index the comparisons read. This
+# lane runs ONE arm per invocation (docs/RUNNING.md section 5), so passing it would
+# leave the index holding whichever arm was launched last.
+echo "[codex_ab] aggregating -> $DEST/index/"
+"$PY" "$HERE/../aggregate.py" "$DEST"
+echo "[codex_ab] DONE in $(( (SECONDS - START) / 60 )) min: ran=$RAN skipped=$SKIPPED failed=${#FAILED[@]}"
+if ((${#FAILED[@]})); then
+ printf '[codex_ab] FAILED CELL: %s\n' "${FAILED[@]}"
+fi
+echo "[codex_ab] Compare an arm with its baseline, cell by cell:"
+echo " $PY scripts/compare_arms.py --baseline $DEST/baseline/<cell> --arm $DEST/<arm>/<cell>"
+((${#FAILED[@]} == 0))
diff --git a/experiments/scripts/launch/fleet_budget.sh b/experiments/scripts/launch/fleet_budget.sh
new file mode 100644
index 0000000..f377bd4
--- /dev/null
+++ b/experiments/scripts/launch/fleet_budget.sh
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+# One CPU budget for EVERY lane on this host, enforced where the lanes actually meet.
+#
+# Why this exists. `TE_SWEBENCH_MAX_CPUS` caps ONE task container (default 16 cores, plus
+# the matching OMP/MKL/joblib thread caps -- see te_harness.swebench._cpu_limit_args). It is
+# per-container by construction, so N concurrent lanes are an N x cap ceiling: six lanes at
+# the default would reserve the right to 96 cores of a shared box, and the agent's own
+# verification step really does try to take them (measured 2026-09-05: one agent-issued
+# `pytest -k` on scikit-learn held 249 cores for 183 s).
+#
+# Under rootless Docker every container scope is a child of ONE cgroup in the user's own
+# systemd manager, and cgroup v2 delegates `cpu` there. Writing `cpu.max` on that parent
+# caps the whole subtree at once -- every lane already running, and every container any lane
+# starts later, with no restart and nothing else in the cgroup to disturb (it holds
+# container scopes only; the harness processes themselves live in app.slice and are
+# network-bound anyway). Within the budget the kernel is work-conserving: an idle lane's
+# share is available to a busy one, which is what makes a fleet cap better than dividing the
+# cores between lanes up front.
+#
+# This is a CEILING, not a reservation. It throttles a burst; it cannot make a lane use
+# more. Two consequences worth knowing before lowering it much further: a heavily throttled
+# test run takes longer in wall-clock, and TE_TASK_TIMEOUT / TE_EVAL_TIMEOUT are wall-clock,
+# so a cap tight enough to matter can in principle cost a resolve.
+#
+# Usage:
+# bash scripts/launch/fleet_budget.sh # report the current budget and usage
+# bash scripts/launch/fleet_budget.sh 16 # cap every lane's containers at 16 cores
+# . scripts/launch/fleet_budget.sh; te_fleet_cpu_cap 16 # from a launcher
+#
+# Env:
+# TE_FLEET_CPUS cores for the whole fleet; 0 or "none" = leave the cgroup alone
+# TE_CGROUP_ROOT cgroup2 mount point (default /sys/fs/cgroup; overridden by the tests)
+
+# The cgroup rootless dockerd puts container scopes under. Located rather than assumed: a
+# live container's own scope names it, and the canonical path is only the fallback.
+te_fleet_cgroup() {
+ local root="${TE_CGROUP_ROOT:-/sys/fs/cgroup}" uid scope
+ uid="$(id -u)"
+ scope="$(find "$root/user.slice/user-$uid.slice/user@$uid.service" -maxdepth 3 \
+ -name 'docker-*.scope' -print -quit 2>/dev/null)"
+ if [[ -n "$scope" ]]; then
+ dirname "$scope"
+ return 0
+ fi
+ local canonical="$root/user.slice/user-$uid.slice/user@$uid.service/user.slice"
+ [[ -d "$canonical" ]] && { printf '%s' "$canonical"; return 0; }
+ return 1
+}
+
+# te_fleet_cpu_cap <cores> -- cap every TE container, collectively, at <cores>.
+# Never fails a launch: an un-delegated or read-only cgroup is a warning, not an exit.
+te_fleet_cpu_cap() {
+ local cores="${1:-${TE_FLEET_CPUS:-16}}" cg quota
+ case "$cores" in
+ 0|none|"") echo "[fleet] CPU budget: not enforced (TE_FLEET_CPUS=$cores)"; return 0 ;;
+ esac
+ cg="$(te_fleet_cgroup)" || {
+ echo "[fleet] !! no rootless-docker cgroup found -- the ${cores}-core fleet cap is NOT in force" >&2
+ return 0; }
+ if [[ ! -w "$cg/cpu.max" ]]; then
+ echo "[fleet] !! $cg/cpu.max is not writable -- the ${cores}-core fleet cap is NOT in force" >&2
+ echo "[fleet] !! (cgroup v2 must delegate the cpu controller to the user manager)" >&2
+ return 0
+ fi
+ # cgroup v2 spells a cap as "<quota_us> <period_us>"; 100000 us is the conventional period.
+ quota=$(( cores * 100000 ))
+ if printf '%s 100000\n' "$quota" > "$cg/cpu.max" 2>/dev/null; then
+ echo "[fleet] CPU budget: every TE container, collectively, capped at ${cores} cores"
+ echo "[fleet] cgroup: $cg"
+ else
+ echo "[fleet] !! could not write $cg/cpu.max -- the ${cores}-core fleet cap is NOT in force" >&2
+ fi
+}
+
+# What the budget is right now, and what the fleet is actually drawing against it.
+te_fleet_cpu_report() {
+ local cg cur n
+ cg="$(te_fleet_cgroup)" || { echo "[fleet] no rootless-docker cgroup on this host"; return 0; }
+ cur="$(cat "$cg/cpu.max" 2>/dev/null || echo '?')"
+ case "$cur" in
+ "max "*) echo "[fleet] CPU budget: UNCAPPED ($cg/cpu.max = $cur)" ;;
+ *) echo "[fleet] CPU budget: $(( ${cur%% *} / 100000 )) cores ($cg/cpu.max = $cur)" ;;
+ esac
+ if command -v docker >/dev/null 2>&1; then
+ n="$(docker ps -q 2>/dev/null | wc -l)"
+ echo "[fleet] live task containers: $n"
+ [[ "$n" -gt 0 ]] && docker stats --no-stream --format '{{.CPUPerc}}' 2>/dev/null \
+ | tr -d '%' | awk '{s+=$1} END {printf "[fleet] drawing now: %.1f cores\n", s/100}'
+ fi
+ echo "[fleet] throttling since boot: $(awk '/nr_throttled|throttled_usec/ {printf "%s=%s ", $1, $2}' \
+ "$cg/cpu.stat" 2>/dev/null)"
+}
+
+# Standalone: `bash fleet_budget.sh [cores]`. Sourced: defines the two functions only.
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+ [[ $# -gt 0 ]] && te_fleet_cpu_cap "$1"
+ te_fleet_cpu_report
+fi
diff --git a/experiments/scripts/launch/full_sweep.sh b/experiments/scripts/launch/full_sweep.sh
new file mode 100644
index 0000000..dd2c779
--- /dev/null
+++ b/experiments/scripts/launch/full_sweep.sh
@@ -0,0 +1,154 @@
+#!/usr/bin/env bash
+# THE WHOLE SWEEP: every lane of the main experiment, end to end, for real.
+#
+# main_sweep.sh runs ONE lane -- one (agent, model, domain) -- and with ARMS unset it
+# already runs every arm of that lane. This driver is the layer above it: it crosses the
+# agents, the models and the domains and runs the lanes in a defined order, so the whole
+# matrix is one command instead of eight.
+#
+# default matrix = 2 agents x 2 models x 2 domains = 8 lanes
+# agents : claude (Claude Code) + copilot (Copilot CLI)
+# models : claude-opus-4-8 + claude-sonnet-5
+# domains: swebench + webbench
+# BOTH models run on BOTH harnesses -- that is the main sweep, not a choice between
+# them. Each (agent, model, domain) gets its own results tree, because the model is part
+# of the cost basis and two of them must never be averaged into one cell.
+#
+# THESE ARE 8 OF THE 12 REQUIRED LANES. claude-opus-5 runs 4 arms per domain on both
+# agents (baseline / baseline-medium / baseline-low / trimech, or trimech-web) and is
+# NOT optional -- see MODELS below and docs/RUNNING.md (the frontier lane). This script does not launch
+# them, so a green run here is not a finished experiment.
+# `sweep_plan.py --format matrix` prints the exact cell and task-run counts.
+#
+# RUN IT WITH DRYRUN=1 FIRST. A total dollar figure and a total wall-clock are deliberately
+# not quoted, because both would be one repo-dominated per-task mean multiplied by the whole
+# matrix. Budget from the PER-ARM runtime instead: time one arm on your host and multiply --
+# it is nearly the same for every arm in a domain.
+#
+# Usage:
+# DRYRUN=1 bash scripts/launch/full_sweep.sh # the plan and the cost, run nothing
+# bash scripts/launch/full_sweep.sh # the whole matrix, for real
+#
+# # one agent, one model, one domain -- still every arm and every seed of that lane
+# AGENTS=copilot MODELS=claude-opus-4-8 DOMAINS=swebench \
+# bash scripts/launch/full_sweep.sh
+#
+# # seed 0 only: a complete, publishable slice at 1/5 the bill
+# SEEDS=0 bash scripts/launch/full_sweep.sh
+#
+# Env:
+# AGENTS space-separated (default "claude copilot")
+# MODELS space-separated (default "claude-opus-4-8 claude-sonnet-5")
+# One spelling here for both agents; main_sweep.sh spells each lane's id the
+# way its backend serves it (the Copilot CLI's catalog says claude-opus-4.8,
+# the Anthropic API claude-opus-4-8) and both land in the same `_opus48` tree.
+# DOMAINS space-separated (default "swebench webbench")
+# SEEDS space-separated (default "0 1 2 3 4")
+# ARMS restrict the arms (default: every arm the domain defines)
+# REPOS restrict the repo groups (default: every group the domain defines)
+# NOTE: repo groups are domain-specific -- SWE-bench takes slugs
+# ("sympy/sympy"), Web-Bench takes project names ("zustand"). Set REPOS only
+# alongside a single DOMAIN, or one of the two lanes will plan zero cells.
+# LANES serial | parallel (default serial -- see the warning below)
+# DRYRUN 1 = print each lane's plan and cost, run nothing (default 0)
+# Everything else (SKIP_DONE, LIMIT, ON_FAIL, TE_*) is passed through to main_sweep.sh.
+#
+# LANES=serial is the default ON PURPOSE. Every SWE-bench task starts a container, every
+# Web-Bench stream starts dev servers and a Playwright browser, the Copilot lanes share ONE
+# subscription quota and the Claude lanes ONE API key's rate limit -- so lanes contend for
+# Docker, ports and rate limits long before they contend for CPU, and a throttled task
+# shows up as a bad ROW (usage_incomplete), not as a launcher error.
+# Raise it deliberately, one lane at a time, watching n_usage_incomplete.
+#
+# Resuming is free: SKIP_DONE=1 is main_sweep.sh's default, so re-running this command
+# after an interruption fills only the cells that are missing.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+. "$HERE/_env.sh"
+REPO_ROOT="$(cd "$HERE/../.." && pwd)" te_load_env
+
+AGENTS="${AGENTS:-claude copilot}"
+# The main sweep's two models -- sweep_plan.SWEEP_MODELS is the same pair, and
+# tests/test_sweep_arms.py holds them together.
+#
+# claude-opus-5 is deliberately absent, which does NOT mean optional: its 4 lanes are
+# required (baseline / baseline-medium / baseline-low / trimech, both domains, both
+# agents -- docs/RUNNING.md) and are launched by their own main_sweep.sh calls. So this
+# script covers 8 of the 12 required lanes; a green run here is not a finished
+# experiment. Adding the id here would plan a third FULL 21-arm lane instead.
+MODELS="${MODELS:-claude-opus-4-8 claude-sonnet-5}"
+DOMAINS="${DOMAINS:-swebench webbench}"
+SEEDS="${SEEDS:-0 1 2 3 4}"
+LANES="${LANES:-serial}"
+DRYRUN="${DRYRUN:-0}"
+
+runner_for() {
+ case "$1" in
+ swebench) echo "te_harness.swebench:make_runner" ;;
+ webbench) echo "te_harness.webbench:make_runner" ;;
+ *) echo "!! unknown domain '$1'" >&2; exit 2 ;;
+ esac
+}
+
+# The whole bill, once, before anything runs -- and before any credential is needed.
+echo "[full_sweep] agents='$AGENTS' models='$MODELS' domains='$DOMAINS' seeds='$SEEDS'"
+echo "[full_sweep] lanes=$LANES dryrun=$DRYRUN"
+
+# `sweep_plan --format matrix` always costs the FULL cross of its model list, so printing it
+# under a narrowed scope would quote tens of thousands of dollars for a twelve-cell run.
+# Show it only when nothing is narrowed; otherwise let each lane print its own estimate,
+# which main_sweep.sh already does WITH --limit applied.
+NARROWED=""
+[[ "$DOMAINS" == "swebench webbench" ]] || NARROWED="domains"
+[[ "$SEEDS" == "0 1 2 3 4" ]] || NARROWED="$NARROWED seeds"
+[[ "$AGENTS" == "claude copilot" ]] || NARROWED="$NARROWED agents"
+[[ -z "${ARMS:-}" ]] || NARROWED="$NARROWED arms"
+[[ -z "${REPOS:-}" ]] || NARROWED="$NARROWED repos"
+[[ -z "${LIMIT:-}" || "${LIMIT:-0}" == "0" ]] || NARROWED="$NARROWED limit"
+if [[ -z "$NARROWED" ]]; then
+ TE_SWEEP_MODELS="$MODELS" "${PYTHON:-python3}" "$HERE/../sweep_plan.py" --format matrix \
+ 2>/dev/null | sed 's/^/[full_sweep] /' || true
+else
+ echo "[full_sweep] scope narrowed ($NARROWED) -- the full-matrix estimate does not apply;"
+ echo "[full_sweep] each lane prints its own costed plan below."
+fi
+echo
+
+LANE_N=0 FAILED=()
+for domain in $DOMAINS; do
+ for model in $MODELS; do
+ for agent in $AGENTS; do
+ LANE_N=$((LANE_N + 1))
+ tag="$agent/$model/$domain"
+ echo "===== [full_sweep] lane $LANE_N: $tag ====="
+ # DETACH=0: this driver owns the ordering, so each lane must run to completion in the
+ # foreground. Detaching every lane would launch all eight at once and silently turn
+ # LANES=serial into LANES=parallel.
+ if AGENT="$agent" MODEL="$model" DOMAIN="$domain" SEEDS="$SEEDS" \
+ RUNNER="$(runner_for "$domain")" DETACH=0 DRYRUN="$DRYRUN" \
+ bash "$HERE/main_sweep.sh"; then
+ echo "[full_sweep] lane $LANE_N OK: $tag"
+ else
+ rc=$?
+ echo "[full_sweep] !! lane $LANE_N FAILED (rc=$rc): $tag" >&2
+ FAILED+=("$tag")
+ # A launch-time refusal (exit 2: dead quota, unserved model, missing key) applies to
+ # every remaining lane on that credential, so stop rather than repeat it eight times.
+ if [[ $rc -eq 2 ]]; then
+ echo "[full_sweep] !! exit 2 is a launch refusal, not a task failure -- stopping." >&2
+ echo "[full_sweep] !! fix the cause and re-run; SKIP_DONE=1 resumes what finished." >&2
+ break 3
+ fi
+ fi
+ done
+ done
+done
+
+echo
+if ((${#FAILED[@]})); then
+ echo "[full_sweep] DONE with ${#FAILED[@]} failed lane(s):"
+ printf ' %s\n' "${FAILED[@]}"
+ exit 1
+fi
+echo "[full_sweep] DONE: $LANE_N lane(s), no failures."
diff --git a/experiments/scripts/launch/main_sweep.sh b/experiments/scripts/launch/main_sweep.sh
new file mode 100644
index 0000000..d442652
--- /dev/null
+++ b/experiments/scripts/launch/main_sweep.sh
@@ -0,0 +1,236 @@
+#!/usr/bin/env bash
+# The main-experiment sweep: every arm x repo x seed for ONE (agent, model, domain) tree.
+#
+# One invocation fills one results tree. The tree name carries the agent AND the model
+# (claude_swebench_opus48, copilot_swebench_sonnet5, ...) because both are part of the cost
+# basis: the aggregator averages a cell, and two models in one cell average two bases into a
+# number that describes neither. Run the sweep as one invocation per (agent, model, domain)
+# -- four agent-model pairs x two domains = 8 lanes.
+#
+# Arms (scripts/sweep_plan.py owns the list; scripts/run_experiment.py owns each flag set):
+# baseline naive, harness-default effort (high)
+# baseline-medium naive at TE_EFFORT_LEVEL=medium (the arm sets it, not the caller)
+# baseline-low naive at TE_EFFORT_LEVEL=low
+# inject-only micro-skill synthesis + injection only
+# untool-only evidence untooling + tool judge only
+# spiral-only spiral guard -> judge -> nudge only (Web-Bench: spiral-only-web)
+# trimech all three co-activated (Web-Bench: trimech-web)
+# rtk naive + rtk 0.45.0 (SWE-bench only)
+# serena naive + Serena (SWE-bench only)
+# repomap naive + aider's repo map (SWE-bench only)
+# trimech-rtk all three + rtk (SWE-bench only)
+# trimech-repomap all three + the repo map (SWE-bench only)
+#
+# The last five install an EXTERNAL TOOL (tokeneconomizer/baselines/) into the agent's
+# container -- baselines, not mechanisms. They are SWE-bench only: Web-Bench reaches its
+# agent through the evaluator's HTTP seam with no per-task repo container. Asking for one
+# on DOMAIN=webbench fails in the plan rather than enumerating cells that cannot run.
+#
+# Billing: Claude Code on the Anthropic API (ANTHROPIC_API_KEY), the Copilot CLI on its
+# GitHub subscription or Copilot's own BYOK (COPILOT_PROVIDER_*).
+#
+# Usage (one lane; see docs/RUNNING.md for every lane):
+# AGENT=claude MODEL=claude-opus-5 DOMAIN=swebench \
+# RUNNER=te_harness.swebench:make_runner \
+# bash scripts/launch/main_sweep.sh
+#
+# Env:
+# AGENT claude | copilot (default claude)
+# MODEL model id passed to the CLI and recorded (default claude-opus-5)
+# It also picks the tree suffix via te_model_tag, so it is REQUIRED to be a
+# pinned id: `auto`/`agent` gives no tag and no controlled cost basis.
+# DOMAIN swebench | webbench (default swebench)
+# ARMS space-separated arm list (default: the domain's SWEEP_ARMS)
+# REPOS space-separated groups (default: every group in the workload record)
+# SEEDS space-separated seeds (default "0 1 2 3 4")
+# LIMIT cap tasks per cell, 0 = all (default 0)
+# RUNNER 'stub' or 'module:factory' (default stub -- SYNTHETIC, not a benchmark)
+# SKIP_DONE 1 = skip a cell that already holds its full row count (default 1)
+# DRYRUN 1 = print the plan and the estimate, run nothing (default 0)
+# The BILLED probes skip themselves, so a dry run never costs a request.
+# ON_FAIL continue | abort (default continue: one bad cell must not
+# take a 200-cell lane with it; the lane still exits nonzero and lists them)
+# DETACH 1 = background (default), 0 = foreground
+# TASKS_FILE / WORKLOAD / TOKENECONOMIZER_RESULTS as in trimech_ab.sh
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$HERE/../.." && pwd)"
+PY="${PYTHON:-python3}"
+
+# shellcheck source=./_env.sh
+. "$HERE/_env.sh"
+te_load_env
+
+AGENT="${AGENT:-claude}"
+MODEL="${MODEL:-claude-opus-5}"
+# One spelling in the docs, the backend's spelling on the wire: the Copilot CLI's own
+# catalog lists the older Opus as `claude-opus-4.8` while the Anthropic API takes
+# `claude-opus-4-8`. te_model_for_agent picks the served form (and says so); the Copilot
+# probe below still verifies it. The tree tag folds both, so nothing forks.
+MODEL="$(te_model_for_agent "$AGENT" "$MODEL")"
+DOMAIN="${DOMAIN:-swebench}"
+SEEDS="${SEEDS:-0 1 2 3 4}"
+LIMIT="${LIMIT:-0}"
+RUNNER="${RUNNER:-stub}"
+SKIP_DONE="${SKIP_DONE:-1}"
+DRYRUN="${DRYRUN:-0}"
+ON_FAIL="${ON_FAIL:-continue}"
+
+case "$DOMAIN" in
+ swebench) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_SWEBENCH_TASKS:-}}" ;;
+ webbench) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_WEBBENCH_TASKS:-}}" ;;
+ *) echo "!! main_sweep covers DOMAIN=swebench|webbench (got '$DOMAIN')" >&2; exit 2 ;;
+esac
+if [[ -z "$TASKS_FILE" && "$DRYRUN" != "1" ]]; then
+ echo "!! no task-spec JSON: set TASKS_FILE or TOKENECONOMIZER_${DOMAIN^^}_TASKS" >&2
+ exit 2
+fi
+
+# A pinned model id is not a nicety here: the tree suffix comes from it, and an unpinned
+# router (`auto`) means the reasoning-effort arms cannot even pass --effort (Copilot's CLI
+# rejects the flag on auto and exits 0-turn -- see te_harness/copilot.py:effort_applies).
+MODEL_TAG="$(te_model_tag "$MODEL")"
+if [[ -z "$MODEL_TAG" ]]; then
+ echo "!! MODEL='$MODEL' is a router directive, not a pinned model: the tree would carry" >&2
+ echo "!! no model tag and the effort arms could not pin effort. Pin an id (e.g." >&2
+ echo "!! MODEL=claude-opus-5 or MODEL=claude-sonnet-5)." >&2
+ exit 2
+fi
+
+# Arms: the domain's default list unless overridden. Resolved through sweep_plan so the
+# launcher and the plan can never disagree about what the sweep contains.
+if [[ -n "${ARMS:-}" ]]; then
+ ARM_ARGS=(--arms $ARMS)
+else
+ ARM_ARGS=()
+fi
+if [[ -n "${REPOS:-}" ]]; then
+ REPO_ARGS=(--repos $REPOS)
+else
+ REPO_ARGS=()
+fi
+
+WORKLOAD="${WORKLOAD:-$REPO_ROOT/workloads/$DOMAIN.json}"
+WORKLOAD_ARGS=()
+if [[ "$WORKLOAD" != "none" ]]; then
+ [[ -f "$WORKLOAD" ]] || { echo "!! workload manifest not found: $WORKLOAD" >&2; exit 2; }
+ WORKLOAD_ARGS=(--workload "$WORKLOAD")
+fi
+
+RESULTS_ROOT="${TOKENECONOMIZER_RESULTS:-$REPO_ROOT/results}"
+DEST="$RESULTS_ROOT/$(te_tree_name "$AGENT" "$DOMAIN" "$MODEL_TAG")"
+
+# The plan: one `arm<TAB>repo<TAB>seed` line per cell, plus a costed summary. Computed
+# before any credential check so `DRYRUN=1` works on a fresh clone with nothing configured.
+#
+# Captured into a variable first, NOT piped into `mapfile < <(...)`: a process substitution
+# discards the producer's exit status, so an unknown arm or an unreadable workload would
+# come back as an empty plan and this lane would report a clean zero-cell success.
+if ! PLAN_RAW="$("$PY" "$HERE/../sweep_plan.py" --domain "$DOMAIN" \
+ "${ARM_ARGS[@]}" "${REPO_ARGS[@]}" --seeds $SEEDS --format cells)"; then
+ echo "!! could not plan the sweep (see the error above)" >&2
+ exit 2
+fi
+PLAN=()
+[[ -n "$PLAN_RAW" ]] && mapfile -t PLAN <<<"$PLAN_RAW"
+if ((${#PLAN[@]} == 0)); then
+ echo "!! the plan is empty: no runnable cell for arms='${ARMS:-<default>}'" >&2
+ echo "!! repos='${REPOS:-<all>}' seeds='$SEEDS' domain=$DOMAIN" >&2
+ exit 2
+fi
+
+echo "[main_sweep] agent=$AGENT model=$MODEL tag=$MODEL_TAG domain=$DOMAIN"
+echo "[main_sweep] dest=$DEST"
+echo "[main_sweep] cells=${#PLAN[@]} seeds='$SEEDS' runner=$RUNNER skip_done=$SKIP_DONE"
+# --limit must be forwarded or the estimate describes the FULL stream while the run is
+# capped: a LIMIT=1 pilot printed "task-runs= 40 ... ~$14" for one task.
+"$PY" "$HERE/../sweep_plan.py" --domain "$DOMAIN" "${ARM_ARGS[@]}" "${REPO_ARGS[@]}" \
+ --seeds $SEEDS --limit "$LIMIT" --format estimate | sed 's/^/[main_sweep] /'
+
+# Credentials preflight (and the loud synthetic-run warning when RUNNER=stub).
+te_check_routing "$AGENT"
+
+# Copilot only: confirm the subscription actually serves this MODEL and that it accepts a
+# reasoning effort. Both mistakes fail ONCE PER TASK rather than at launch -- the CLI exits
+# before its first turn -- so without this a lane writes a whole tree of 0-turn rows that
+# look like an agent that tried and failed. te_check_copilot_model hard-fails on an unserved
+# model and only WARNS on a rejected effort (exporting TE_EFFORT_APPLIES=0, so rows record
+# effort: null), because the latter is still a valid run that merely cannot hold effort
+# fixed. It probes the harness default; the baseline-low / baseline-medium arms set
+# TE_EFFORT_LEVEL themselves per cell, so a model that takes `high` and rejects a lower
+# level would still surface at that arm rather than here.
+if [[ "$AGENT" == copilot && "${RUNNER:-stub}" != "stub" ]]; then
+ te_check_copilot_model "$MODEL" "${TE_EFFORT_LEVEL:-high}"
+fi
+
+# AFTER the preflights, so a dry run still refuses a missing credential. The BILLED probes
+# skip themselves on a dry run (te_dryrun_skips_billed), and te_check_routing returns early
+# on RUNNER=stub, so DRYRUN=1 on a fresh clone with nothing configured still costs nothing
+# and still works (docs/RUNNING.md section 2).
+if [[ "$DRYRUN" == "1" ]]; then
+ echo "[main_sweep] DRYRUN -- the cells this lane would run, in order:"
+ printf ' %s\n' "${PLAN[@]}"
+ echo "[main_sweep] DRYRUN: nothing was run."
+ exit 0
+fi
+
+mkdir -p "$DEST"
+
+# Detach by default: the plan is resolved and credentials are checked, so everything from
+# here belongs in a log rather than a terminal a laptop lid will close on.
+te_detach "main_sweep__${AGENT}_${MODEL_TAG}_${DOMAIN}" "$0" "$@"
+
+# expected_rows <repo> — the task count of that group, from the committed workload record.
+declare -A GROUP_N
+while IFS=$'\t' read -r g n; do GROUP_N["$g"]="$n"; done < <(
+ "$PY" "$HERE/../sweep_plan.py" --domain "$DOMAIN" --format counts)
+
+# cell_rows <arm> <repo> <seed> — how many row.json files that cell already holds.
+cell_rows() {
+ local tag="${2##*/}"; tag="${tag//-/_}"
+ local dir="$DEST/$1/${tag}_s$3"
+ [[ -d "$dir" ]] || { echo 0; return; }
+ find "$dir" -name row.json 2>/dev/null | wc -l
+}
+
+FAILED=()
+SKIPPED=0
+RAN=0
+START=$SECONDS
+for line in "${PLAN[@]}"; do
+ IFS=$'\t' read -r arm repo seed <<<"$line"
+ want="${GROUP_N[$repo]:-0}"
+ if [[ "$LIMIT" != "0" && "$want" -gt "$LIMIT" ]]; then want="$LIMIT"; fi
+ have="$(cell_rows "$arm" "$repo" "$seed")"
+ if [[ "$SKIP_DONE" == "1" && "$want" -gt 0 && "$have" -ge "$want" ]]; then
+ echo "[main_sweep] SKIP $arm $repo s$seed ($have/$want rows already stored)"
+ SKIPPED=$((SKIPPED + 1))
+ continue
+ fi
+ echo "[main_sweep] RUN $arm $repo s$seed (have $have of $want)"
+ if "$PY" "$HERE/../run_experiment.py" \
+ --arm "$arm" --agent "$AGENT" --repo "$repo" --seed "$seed" --limit "$LIMIT" \
+ --adapter "$DOMAIN" --tasks-file "$TASKS_FILE" "${WORKLOAD_ARGS[@]}" \
+ --results-root "$DEST" --runner "$RUNNER" \
+ --model "$MODEL" --optimizer-model "$AGENT" --fresh; then
+ RAN=$((RAN + 1))
+ else
+ echo "[main_sweep] FAILED $arm $repo s$seed"
+ FAILED+=("$arm/$repo/s$seed")
+ [[ "$ON_FAIL" == "abort" ]] && { echo "[main_sweep] ON_FAIL=abort -- stopping"; break; }
+ fi
+done
+
+# Aggregate every arm PRESENT in the tree, not just this lane's: --arms limits the rewrite,
+# so passing it would drop the other arms' rows from the index the comparisons read.
+echo "[main_sweep] aggregating -> $DEST/index/"
+"$PY" "$HERE/../aggregate.py" "$DEST"
+
+echo "[main_sweep] DONE in $(( (SECONDS - START) / 60 )) min: ran=$RAN skipped=$SKIPPED failed=${#FAILED[@]}"
+if ((${#FAILED[@]})); then
+ printf '[main_sweep] FAILED: %s\n' "${FAILED[@]}"
+ echo "[main_sweep] re-run this same command to retry only the failed cells (SKIP_DONE=1"
+ echo "[main_sweep] skips the finished ones)."
+ exit 1
+fi
diff --git a/experiments/scripts/launch/smoke_test.sh b/experiments/scripts/launch/smoke_test.sh
new file mode 100644
index 0000000..3add798
--- /dev/null
+++ b/experiments/scripts/launch/smoke_test.sh
@@ -0,0 +1,365 @@
+#!/usr/bin/env bash
+# Offline end-to-end smoke test: exercise EVERY proven arm through the real driver +
+# facade + aggregator using the built-in StubRunner (no Docker, no datasets, no LLM), then
+# assert the pipeline produced rows + CSVs, and DELETE everything created.
+#
+# It also covers the two things a domain whose task ids are unique only WITHIN a group
+# depends on -- workload narrowing and the stream-level runner hooks -- because both fail
+# SILENTLY rather than loudly: a widened stream just runs more tasks, and an uncalled
+# begin_stream just leaves a harness un-started.
+#
+# It writes ONLY under a fresh temp dir (mktemp -d) and removes it on exit, so it leaves no
+# artifacts in the repo and nothing to stage. Exit 0 = wiring healthy.
+#
+# Usage: bash scripts/launch/smoke_test.sh
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$HERE/../.." && pwd)"
+PY="${PYTHON:-python3}"
+
+WORK="$(mktemp -d)"
+cleanup() { rm -rf "$WORK"; }
+trap cleanup EXIT
+
+fail() { echo "SMOKE FAIL: $*" >&2; exit 1; }
+
+# Tree-naming helpers under test below (te_tree_name / te_model_tag). Sourcing does not read
+# .env -- te_load_env is a separate call -- so this cannot pick up real credentials.
+# shellcheck source=./_env.sh
+. "$HERE/_env.sh"
+
+# --- tiny synthetic task specs (never touch a real dataset) ---
+TASKS="$WORK/tasks.json"
+cat > "$TASKS" <<'JSON'
+[{"instance_id": "xarray-1", "problem_statement": "fix concat dtype", "repo": "pydata/xarray"},
+ {"instance_id": "xarray-2", "problem_statement": "fix groupby regression", "repo": "pydata/xarray"},
+ {"instance_id": "xarray-3", "problem_statement": "handle empty dataset", "repo": "pydata/xarray"}]
+JSON
+
+run_arm() { # <agent> <domain-adapter> <arm> <dest-subdir>
+ local agent="$1" adapter="$2" arm="$3" dest="$WORK/$4"
+ "$PY" "$HERE/../run_experiment.py" \
+ --arm "$arm" --agent "$agent" --repo pydata/xarray --seed 0 \
+ --adapter "$adapter" --tasks-file "$TASKS" \
+ --results-root "$dest" --runner stub --fresh >/dev/null 2>&1 \
+ || fail "driver arm=$arm agent=$agent"
+ local n; n=$(find "$dest/$arm" -name row.json | wc -l)
+ [[ "$n" -eq 3 ]] || fail "arm=$arm produced $n row.json (expected 3)"
+ echo " ok arm=$arm agent=$agent rows=$n"
+}
+
+echo "[smoke] .env loaders (python + shell must agree; no real credentials involved)"
+ENVF="$WORK/test.env"
+cat > "$ENVF" <<'ENV'
+# comment
+export TE_SMOKE_A=alpha
+TE_SMOKE_B="two words"
+TE_SMOKE_EMPTY=
+TE_SMOKE_SHELL_WINS=from_file
+ENV
+SCRIPTS_DIR="$REPO_ROOT/scripts" TOKENECONOMIZER_ENV_FILE="$ENVF" \
+ TE_SMOKE_SHELL_WINS=from_shell "$PY" - <<'PY' || fail ".env python loader"
+import os, sys
+sys.path.insert(0, os.environ["SCRIPTS_DIR"])
+from _env import load_env
+load_env()
+assert os.environ["TE_SMOKE_A"] == "alpha", os.environ.get("TE_SMOKE_A")
+assert os.environ["TE_SMOKE_B"] == "two words", os.environ.get("TE_SMOKE_B")
+assert "TE_SMOKE_EMPTY" not in os.environ, "empty value should not be set"
+assert os.environ["TE_SMOKE_SHELL_WINS"] == "from_shell", "shell export must win"
+print(" ok python loader: quotes, comments, empty-skip, shell-precedence")
+PY
+( REPO_ROOT="$REPO_ROOT"; TOKENECONOMIZER_ENV_FILE="$ENVF"; TE_SMOKE_SHELL_WINS=from_shell
+ . "$HERE/_env.sh"; te_load_env
+ [[ "$TE_SMOKE_A" == "alpha" ]] || exit 1
+ [[ "$TE_SMOKE_B" == "two words" ]] || exit 1
+ [[ -z "${TE_SMOKE_EMPTY:-}" ]] || exit 1
+ [[ "$TE_SMOKE_SHELL_WINS" == "from_shell" ]] || exit 1
+) || fail ".env shell loader"
+echo " ok shell loader: same semantics as scripts/_env.py"
+
+echo "[smoke] generic arms (Claude/Copilot)"
+run_arm claude swebench baseline gen
+run_arm claude swebench trimech gen
+run_arm claude swebench trimech-web gen
+run_arm copilot swebench trimech cop
+
+echo "[smoke] Codex-native arms"
+run_arm codex swebench baseline cdx
+run_arm codex swebench codex-untool cdx
+run_arm codex swebench codex-spiral cdx
+run_arm codex swebench codex-spiral-postgreen cdx
+run_arm codex swebench codex-combo cdx
+run_arm codex swebench codex-inject cdx
+run_arm codex swebench codex-trimech cdx
+
+echo "[smoke] workload manifest narrows to ONE group (ids repeat across groups)"
+# Web-Bench numbers every project's stream 'init', 'task-1', ... so ids are unique only
+# WITHIN a project. Filtering on ids alone silently returns every project's tasks
+# interleaved -- which, for a sequentially dependent stream, is not a bigger run but a
+# meaningless one. Two synthetic groups sharing all three ids catch that regression.
+WB_ROOT="$WORK/wb/projects"
+mkdir -p "$WB_ROOT/alpha" "$WB_ROOT/beta"
+for proj in alpha beta; do
+ cat > "$WB_ROOT/$proj/tasks.jsonl" <<JSONL
+{"id": "init", "date": "2024-10-25", "level": "easy", "description": "$proj init"}
+{"id": "task-1", "date": "2024-10-25", "level": "easy", "description": "$proj first"}
+{"id": "task-2", "date": "2024-10-25", "level": "easy", "description": "$proj second"}
+JSONL
+done
+WB_TASKS="$WORK/wb_tasks.json"
+"$PY" -c '
+import json, sys
+rows = [{"instance_id": t, "problem_statement": p + " " + t, "repo": p}
+ for p in ("alpha", "beta") for t in ("init", "task-1", "task-2")]
+json.dump(rows, open(sys.argv[1], "w"))
+' "$WB_TASKS" || fail "build synthetic webbench task list"
+WB_WORKLOAD="$WORK/wb_workload.json"
+cat > "$WB_WORKLOAD" <<'JSON'
+{"groups": {"alpha": ["init", "task-1", "task-2"],
+ "beta": ["init", "task-1", "task-2"]}}
+JSON
+TOKENECONOMIZER_WEBBENCH_PROJECTS="$WB_ROOT" "$PY" "$HERE/../run_experiment.py" \
+ --arm baseline --agent claude --repo alpha --seed 0 \
+ --adapter webbench --tasks-file "$WB_TASKS" --workload "$WB_WORKLOAD" \
+ --results-root "$WORK/wb" --runner stub --fresh >/dev/null 2>&1 \
+ || fail "driver on the webbench adapter"
+"$PY" -c '
+import glob, json, sys
+rows = [json.load(open(p)) for p in sorted(glob.glob(sys.argv[1] + "/*/*/row.json"))]
+n = len(rows)
+assert n == 3, "--repo alpha ran %d tasks; want only its group of 3 (ids repeat across groups)" % n
+repos = sorted({r.get("repo") for r in rows})
+assert repos == ["alpha"], "stream mixed groups: %s" % repos
+ids = [r.get("instance") for r in rows]
+assert ids == ["init", "task-1", "task-2"], "stream out of order: %s" % ids
+print(" ok --repo alpha -> 3 tasks, group alpha only, in stream order")
+' "$WORK/wb/baseline" || fail "workload narrowing"
+
+echo "[smoke] stream-level runner hooks (begin_stream / end_stream)"
+# A harness that owns a whole stream is started once around the task loop, not per task.
+# The driver calls the hooks hasattr-guarded, so a runner WITHOUT them must still work
+# (every arm above proves that) and a runner WITH them must actually see both calls.
+cat > "$WORK/te_smoke_hooks.py" <<'HOOKMOD'
+import os
+import sys
+
+sys.path.insert(0, os.environ["TE_SMOKE_SCRIPTS"])
+from run_experiment import StubRunner
+
+
+class HookRunner(StubRunner):
+ """StubRunner that records the stream-level calls the driver made to it."""
+
+ def _note(self, line):
+ with open(os.environ["TE_SMOKE_HOOKLOG"], "a", encoding="utf-8") as fh:
+ fh.write(line + "\n")
+
+ def begin_stream(self, tasks, **info):
+ self._note("begin %d %s %s" % (len(tasks), info.get("repo"), info.get("arm")))
+
+ def end_stream(self):
+ self._note("end")
+
+
+def make_runner():
+ return HookRunner("claude")
+HOOKMOD
+HOOKLOG="$WORK/hooks.log"
+PYTHONPATH="$WORK${PYTHONPATH:+:$PYTHONPATH}" TE_SMOKE_SCRIPTS="$REPO_ROOT/scripts" \
+ TE_SMOKE_HOOKLOG="$HOOKLOG" "$PY" "$HERE/../run_experiment.py" \
+ --arm baseline --agent claude --repo pydata/xarray --seed 0 \
+ --adapter swebench --tasks-file "$TASKS" \
+ --results-root "$WORK/hooks" --runner te_smoke_hooks:make_runner --fresh >/dev/null 2>&1 \
+ || fail "driver with a stream-hook runner"
+[[ -s "$HOOKLOG" ]] || fail "begin_stream/end_stream were never called"
+grep -qx "begin 3 pydata/xarray baseline" "$HOOKLOG" \
+ || fail "begin_stream got the wrong stream/context: $(head -1 "$HOOKLOG")"
+grep -qx "end" "$HOOKLOG" || fail "end_stream was never called"
+echo " ok begin_stream(3 tasks, repo, arm) then end_stream"
+
+echo "[smoke] a re-run archives traces and does not double-count"
+# Traces are the primary artifact and cannot be reproduced after the fact, so re-running a
+# cell must ADD a timestamped run dir rather than overwrite the previous transcript. That
+# makes the aggregator's job harder, not easier: it has to report one row per task, from the
+# newest attempt, while the superseded ones stay on disk.
+for pass_n in 1 2; do
+ "$PY" "$HERE/../run_experiment.py" \
+ --arm baseline --agent claude --repo pydata/xarray --seed 0 \
+ --adapter swebench --tasks-file "$TASKS" \
+ --results-root "$WORK/rerun" --runner stub --fresh >/dev/null 2>&1 \
+ || fail "re-run pass $pass_n"
+ sleep 1 # distinct second -> distinct stamp; the index alone would also suffice
+done
+N_DIRS=$(find "$WORK/rerun" -name row.json | wc -l)
+[[ "$N_DIRS" -eq 6 ]] || fail "re-run left $N_DIRS run dirs (expected 6: 3 tasks x 2 passes)"
+"$PY" "$HERE/../aggregate.py" "$WORK/rerun" --arms baseline >/dev/null 2>&1 \
+ || fail "aggregate the re-run tree"
+"$PY" -c '
+import csv, sys
+rows = list(csv.DictReader(open(sys.argv[1])))
+n = len(rows)
+assert n == 3, "aggregate reported %d rows; want 3 (one per task, newest attempt)" % n
+stamps = [r["source_path"].split("/")[-2].split("__")[0] for r in rows]
+assert all(len(t) == 16 and t[8] == "T" and t.endswith("Z") for t in stamps), \
+ "run dirs are not UTC-timestamped: %s" % stamps
+newest = sorted(stamps)[-1]
+print(" ok 6 run dirs kept, 3 rows aggregated, newest attempt wins (%s)" % newest)
+' "$WORK/rerun/index/all_rows.csv" || fail "re-run de-duplication"
+
+echo "[smoke] the tree is <agent>_<domain>, and ANTHROPIC_BASE_URL is not an input"
+# Claude Code bills the Anthropic API on ANTHROPIC_API_KEY. A base URL left in the
+# environment must neither rename a tree nor reach the agent: te_check_routing says it is
+# ignored and unsets it (offline here -- RUNNER=stub returns before any credential check).
+check_tree() { # <agent> <expect-tree>
+ local got; got="$(te_tree_name "$1" swebench)"
+ [[ "$got" == "$2" ]] || fail "tree_name($1) = $got, expected $2"
+}
+(
+ export ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=dummy RUNNER=stub
+ check_tree claude claude_swebench
+ check_tree copilot copilot_swebench
+ check_tree codex codex_swebench
+ out="$(te_check_routing claude 2>&1; echo "left=${ANTHROPIC_BASE_URL:-}${ANTHROPIC_AUTH_TOKEN:-}")"
+ grep -q "IGNORED" <<<"$out" || fail "a set ANTHROPIC_BASE_URL was not reported: $out"
+ grep -qx "left=" <<<"$out" || fail "ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN survived: $out"
+ # Only the three agent labels exist; anything else is refused rather than renamed.
+ ( te_check_routing not-an-agent >/dev/null 2>&1 ) && fail "te_check_routing accepted an unknown agent"
+ # OPENAI_BASE_URL may only name OpenAI's own (regional) API.
+ ( OPENAI_BASE_URL=https://us.api.openai.com/v1 te_check_routing codex >/dev/null 2>&1 ) \
+ || fail "a regional OpenAI endpoint was refused"
+ ( OPENAI_BASE_URL=http://localhost:8080 te_check_routing codex >/dev/null 2>&1 ) \
+ && fail "a non-OpenAI OPENAI_BASE_URL was accepted"
+ true
+) || exit 1
+echo " ok <agent>_<domain> for claude/copilot/codex; ANTHROPIC_BASE_URL reported and dropped"
+echo " ok unknown agent labels and a non-OpenAI OPENAI_BASE_URL refused"
+
+echo "[smoke] the model tag separates trees"
+# Two models in one cell average two cost bases into a number that describes neither, so
+# the model belongs in the tree name. The two spellings of one model must collapse to ONE
+# tag (the CLI takes both, and the stored rows carry both), and a router directive must
+# produce NO tag -- `auto` is not a cost basis worth naming.
+check_tag() { # <model> <expect-tag>
+ local got; got="$(te_model_tag "$1")"
+ [[ "$got" == "$2" ]] || fail "model_tag($1) = '$got', expected '$2'"
+}
+(
+ check_tag claude-opus-4-8 opus48
+ check_tag claude-opus-4.8 opus48 # same model, other spelling -> same tree
+ check_tag claude-sonnet-4.6 sonnet46
+ check_tag auto ""
+ check_tag agent ""
+ # A tagged tree, and the 2-arg call unchanged so the stored trees keep resolving.
+ got="$(te_tree_name claude swebench "$(te_model_tag claude-sonnet-4.6)")"
+ [[ "$got" == "claude_swebench_sonnet46" ]] || fail "tagged tree name = $got"
+ got="$(te_tree_name claude swebench)"
+ [[ "$got" == "claude_swebench" ]] || fail "untagged tree name changed: $got"
+ got="$(te_tree_name codex swebench "$(te_model_tag gpt-5.6-sol)")"
+ [[ "$got" == "codex_swebench_gpt56sol" ]] || fail "codex tagged tree name = $got"
+) || exit 1
+echo " ok one tag per model, no tag for a router, 2-arg name unchanged"
+
+echo "[smoke] main_sweep plans its cells without touching Docker, data or a key"
+# The sweep is five figures of task-runs; DRYRUN is where that gets noticed. It must work
+# on a fresh clone with nothing configured, so it is checked with the placeholders unset.
+(
+ unset TOKENECONOMIZER_SWEBENCH_TASKS TOKENECONOMIZER_WEBBENCH_TASKS
+ # "nothing configured" has to mean the REPO'S .env too, not just these two placeholders:
+ # te_load_env only lets a NON-EMPTY shell export beat the file, so an operator's RUNNER or
+ # task paths would change what this plans. Point the loader at a path that does not exist.
+ export TOKENECONOMIZER_ENV_FILE="$WORK/no-such.env"
+ out="$(DRYRUN=1 ARMS="baseline trimech" REPOS="django/django" SEEDS="0 1" \
+ AGENT=claude MODEL=claude-opus-4-8 DOMAIN=swebench \
+ bash "$HERE/main_sweep.sh" 2>&1)" || fail "main_sweep DRYRUN exited nonzero"
+ grep -q "cells=4" <<<"$out" || fail "DRYRUN cell count: $out"
+ grep -q "claude_swebench_opus48" <<<"$out" || fail "DRYRUN tree name: $out"
+ grep -q "DRYRUN: nothing was run" <<<"$out" || fail "DRYRUN ran something: $out"
+ # A router directive has no cost basis and no tag: refuse rather than write an untagged
+ # tree that silently mixes whatever the router picked.
+ if DRYRUN=1 MODEL=auto AGENT=claude DOMAIN=swebench \
+ bash "$HERE/main_sweep.sh" >/dev/null 2>&1; then
+ fail "main_sweep accepted MODEL=auto"
+ fi
+ # Every arm the plan names has to be registered in run_experiment.ARMS; an unknown one
+ # must fail at plan time, not at task 0.
+ if DRYRUN=1 ARMS="not-an-arm" AGENT=claude DOMAIN=swebench \
+ bash "$HERE/main_sweep.sh" >/dev/null 2>&1; then
+ fail "main_sweep accepted an unregistered arm"
+ fi
+) || exit 1
+echo " ok DRYRUN plans offline; router model and unknown arm both handled"
+
+echo "[smoke] scaffold arms plan, and the stub refuses to pretend it installed one"
+# The external tool baselines (rtk/serena/repomap) install into a real task container. The
+# stub installs nothing, so it must SAY so -- a synthetic row under a scaffold's name is
+# indistinguishable from a real one once it is sitting in results/.
+(
+ unset TOKENECONOMIZER_SWEBENCH_TASKS
+ export TOKENECONOMIZER_ENV_FILE="$WORK/no-such.env" # as above: not the repo's .env
+ out="$(DRYRUN=1 ARMS="rtk serena repomap trimech-rtk trimech-repomap" \
+ REPOS="django/django" SEEDS="0" AGENT=claude MODEL=claude-opus-4-8 \
+ DOMAIN=swebench bash "$HERE/main_sweep.sh" 2>&1)" \
+ || fail "main_sweep DRYRUN with scaffold arms exited nonzero"
+ grep -q "cells=5" <<<"$out" || fail "scaffold arms not planned: $out"
+ # Web-Bench hosts rtk (a static binary + a PreToolUse hook: no container, no Python
+ # symbol map) and must PLAN it...
+ out="$(DRYRUN=1 ARMS="rtk trimech-rtk" REPOS="calculator" SEEDS="0" AGENT=claude \
+ MODEL=claude-opus-4-8 DOMAIN=webbench bash "$HERE/main_sweep.sh" 2>&1)" \
+ || fail "webbench refused the rtk arms it can host: $out"
+ grep -q "cells=2" <<<"$out" || fail "webbench did not plan rtk/trimech-rtk: $out"
+ # ...and must still refuse the two it cannot: the repo map walks *.py, and Serena is a
+ # codebase navigator for a benchmark that ships a file set rather than a codebase.
+ out="$(DRYRUN=1 ARMS="serena repomap trimech-repomap" REPOS="calculator" SEEDS="0" AGENT=claude \
+ MODEL=claude-opus-4-8 DOMAIN=webbench bash "$HERE/main_sweep.sh" 2>&1)" || true
+ grep -q "SKIPPING scaffold arms not wired for webbench" <<<"$out" \
+ || fail "webbench accepted a scaffold arm it cannot install: $out"
+) || exit 1
+TE_SMOKE_SCRIPTS="$REPO_ROOT/scripts" "$PY" - <<'PY' || fail "stub scaffold refusal"
+import io, os, sys
+sys.path.insert(0, os.environ["TE_SMOKE_SCRIPTS"]) # wherever the smoke test is run from
+import run_experiment as rx
+# The stub must warn, not silently accept.
+from contextlib import redirect_stdout
+buf = io.StringIO()
+with redirect_stdout(buf):
+ rx.load_runner("stub", "claude", None, "rtk")
+assert "NOT installed" in buf.getvalue(), buf.getvalue()
+# A real factory that cannot take a scaffold must RAISE rather than run native.
+sys.modules["_fake_h"] = type(sys)("_fake_h")
+sys.modules["_fake_h"].make_runner = lambda agent=None: object()
+try:
+ rx.load_runner("_fake_h:make_runner", "claude", None, "rtk")
+except SystemExit as e:
+ assert "does not accept a scaffold" in str(e), e
+else:
+ raise AssertionError("a scaffold-blind runner was accepted")
+print(" ok stub warns; a scaffold-blind harness is refused")
+PY
+
+echo "[smoke] aggregate both trees"
+"$PY" "$HERE/../aggregate.py" "$WORK/gen" >/dev/null 2>&1 || fail "aggregate gen"
+"$PY" "$HERE/../aggregate.py" "$WORK/cdx" >/dev/null 2>&1 || fail "aggregate cdx"
+for tree in gen cdx; do
+ for csv in all_rows.csv summary_by_cell.csv; do
+ [[ -s "$WORK/$tree/index/$csv" ]] || fail "missing/empty $tree/index/$csv"
+ done
+done
+echo " ok CSVs: gen/index + cdx/index all_rows + summary_by_cell"
+
+# sanity: the cost basis column is populated and numeric
+"$PY" - "$WORK/gen/index/all_rows.csv" <<'PY' || fail "cost column check"
+import csv, sys
+rows = list(csv.DictReader(open(sys.argv[1])))
+assert rows, "no rows"
+vals = [float(r["cost_effective_tokens"]) for r in rows]
+assert all(v > 0 for v in vals), "non-positive cost_effective_tokens"
+print(f" ok {len(rows)} rows, cost_effective_tokens in "
+ f"[{min(vals):.0f}, {max(vals):.0f}]")
+PY
+
+echo "[smoke] cleaning up $WORK"
+cleanup; trap - EXIT
+[[ -d "$WORK" ]] && fail "temp dir survived cleanup"
+echo "SMOKE PASS: all arms wired; no artifacts left."
diff --git a/experiments/scripts/launch/trimech_ab.sh b/experiments/scripts/launch/trimech_ab.sh
new file mode 100644
index 0000000..9b910ed
--- /dev/null
+++ b/experiments/scripts/launch/trimech_ab.sh
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+# Reproduce a Claude/Copilot tri-mechanism A/B on one domain: the BASELINE arm (no
+# mechanisms) and the TRIMECH arm (injection=manual4 + evidence untooling + spiral guard),
+# then aggregate both into the index CSVs compare_arms.py reads.
+#
+# The exact arm configurations (the full catalog is in docs/RUNNING.md):
+# baseline : native cost basis, no economizer
+# trimech : --mode manual4 --untool-evidence 3 --untool-judge
+# --spiral-guard --spiral-judge --spiral-nudge
+# (Web-Bench streams are short, so the driver's `trimech-web` arm additionally sets
+# --spiral-nudge-llm --spiral-frozen-envelope 30; pass ARM=trimech-web for those.)
+#
+# The heavy task-execution harness (Docker + agent CLI + gold grading) lives below
+# tokeneconomizer.runner.AgentRunner, in the repo's te_harness package, so
+# RUNNER=te_harness.swebench:make_runner (or .webbench) needs no PYTHONPATH. Point RUNNER at
+# your own factory ("module:make_runner") to substitute another. With RUNNER=stub you get the offline
+# StubRunner (no Docker/LLM) used by smoke_test.sh -- synthetic, for wiring checks only.
+#
+# Usage:
+# AGENT=claude DOMAIN=swebench REPOS="pydata/xarray django/django" SEEDS="0 1 2" \
+# RUNNER=te_harness.swebench:make_runner \
+# TOKENECONOMIZER_SWEBENCH_TASKS=/path/tasks.json \
+# TOKENECONOMIZER_RESULTS=/path/results \
+# bash scripts/launch/trimech_ab.sh
+#
+# Env (all have safe defaults except the dataset/results placeholders):
+# AGENT claude | copilot (default claude)
+# DOMAIN swebench | webbench | creddata | reposvul (default swebench)
+# ARM trimech | trimech-web (default trimech)
+# ARMS explicit space-separated arm list, overriding the default "baseline $ARM".
+# Use it to run ONE arm when a comparable baseline already exists in the tree
+# -- e.g. ARMS=trimech to add just the economized arm. Without it the launcher
+# re-runs baseline, which both costs an extra pass and supersedes the stored
+# baseline rows in the index (run dirs are timestamped and the aggregator keeps
+# the newest attempt per arm/cell/instance).
+# REPOS space-separated repo slugs (default "pydata/xarray")
+# on DOMAIN=webbench these are Web-Bench PROJECT names -- one project is one
+# stream of 20-21 sequentially dependent tasks (e.g. "calculator zustand react")
+# SEEDS space-separated seeds (default "0")
+# LIMIT cap tasks per (repo,seed), 0 = all (default 0)
+# RUNNER 'stub' or 'module:factory' (default stub)
+# TASKS_FILE task-spec JSON (default: the domain's TOKENECONOMIZER_*_TASKS env var)
+# WORKLOAD workload manifest, or 'none' (default workloads/$DOMAIN.json)
+# DRYRUN 1 = print the cells this lane would run and exit, run nothing (default 0)
+# The BILLED probes skip themselves, so it costs nothing.
+# DETACH 1 = run in the background (default), 0 = foreground
+# TE_LOG log path when detached (default $TOKENECONOMIZER_RESULTS/logs/<tag>__<ts>.log)
+# TOKENECONOMIZER_RESULTS results root (default: ./results)
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$HERE/../.." && pwd)"
+PY="${PYTHON:-python3}"
+
+# Credentials + data roots from the git-ignored repo-root .env (shell exports win).
+# shellcheck source=./_env.sh
+. "$HERE/_env.sh"
+te_load_env
+
+AGENT="${AGENT:-claude}"
+DOMAIN="${DOMAIN:-swebench}"
+ARM="${ARM:-trimech}"
+# Arms to run, in order. Default is the paired A/B; ARMS lets a caller run a single arm.
+ARMS="${ARMS:-baseline $ARM}"
+REPOS="${REPOS:-pydata/xarray}"
+SEEDS="${SEEDS:-0}"
+LIMIT="${LIMIT:-0}"
+RUNNER="${RUNNER:-stub}"
+# Credentials preflight for a real run; for RUNNER=stub this prints the synthetic-run
+# warning instead, so stub output is never mistaken for a benchmark.
+te_check_routing "$AGENT"
+
+# Model matched to the stored runs; only labels the cost, does not change the comparison.
+case "$AGENT" in
+ claude) MODEL="${MODEL:-${ANTHROPIC_MODEL:-claude-opus-5}}" ;;
+ # Native Copilot serves its OWN catalog, so an Anthropic model id is not a valid value
+ # here. 'auto' lets the CLI route; pin COPILOT_MODEL (or MODEL) to fix the cost basis --
+ # with an id your subscription serves. `copilot --model <id>` fails per task, not at
+ # launch, so check first.
+ copilot) MODEL="${MODEL:-${COPILOT_MODEL:-auto}}" ;;
+ *) MODEL="${MODEL:-agent}" ;;
+esac
+# The backend's spelling of a two-spelling id (claude-opus-4-8 / claude-opus-4.8); the
+# probes below verify it. See te_model_for_agent.
+MODEL="$(te_model_for_agent "$AGENT" "$MODEL")"
+
+# Copilot only: confirm the pinned model is served and takes the requested effort. Both
+# mistakes otherwise fail once per task instead of once at launch. Skipped for RUNNER=stub
+# (no agent runs) and for MODEL=auto (the router takes no effort level).
+if [[ "$AGENT" == "copilot" && "${RUNNER:-stub}" != "stub" ]]; then
+ te_check_copilot_model "$MODEL" "${TE_EFFORT_LEVEL:-high}"
+fi
+
+# Dataset task-spec: default to the domain's placeholder env var.
+case "$DOMAIN" in
+ swebench) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_SWEBENCH_TASKS:-}}" ;;
+ webbench) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_WEBBENCH_TASKS:-}}" ;;
+ creddata) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_CREDDATA_TASKS:-}}" ;;
+ reposvul) TASKS_FILE="${TASKS_FILE:-${TOKENECONOMIZER_REPOSVUL_TASKS:-}}" ;;
+ *) echo "unknown DOMAIN=$DOMAIN"; exit 2 ;;
+esac
+if [[ -z "$TASKS_FILE" ]]; then
+ echo "!! no task-spec JSON: set TASKS_FILE or the domain placeholder env var" >&2
+ exit 2
+fi
+
+# Canonical workload manifest (committed, tiny): pins the run to exactly the evaluated
+# instances, so TASKS_FILE may hold a superset. WORKLOAD=none restores the legacy behavior
+# of taking whatever TASKS_FILE contains, filtered by the --repo prefix match.
+WORKLOAD="${WORKLOAD:-$REPO_ROOT/workloads/$DOMAIN.json}"
+WORKLOAD_ARGS=()
+if [[ "$WORKLOAD" != "none" ]]; then
+ if [[ ! -f "$WORKLOAD" ]]; then
+ echo "!! workload manifest not found: $WORKLOAD (set WORKLOAD=none to run unfiltered)" >&2
+ exit 2
+ fi
+ WORKLOAD_ARGS=(--workload "$WORKLOAD")
+fi
+
+# `trimech-web` is the arm tuned for Web-Bench's short streams (it adds the LLM nudge and a
+# cold-start frozen envelope). Running plain `trimech` there is a valid choice, not an error,
+# so say which one is about to run rather than second-guessing it.
+if [[ "$DOMAIN" == "webbench" && "$ARM" != "trimech-web" ]]; then
+ echo "[trimech_ab] note: DOMAIN=webbench with ARM=$ARM -- the arm tuned for Web-Bench's" >&2
+ echo "[trimech_ab] short streams is ARM=trimech-web (see docs/RUNNING.md)" >&2
+fi
+
+RESULTS_ROOT="${TOKENECONOMIZER_RESULTS:-$REPO_ROOT/results}"
+# <agent>_<domain>_<model-tag>. The MODEL is part of the cost basis, so it belongs in the
+# tree name exactly as it does in main_sweep.sh: without the tag an opus lane and a sonnet
+# lane land in ONE tree and aggregate.py averages two cost bases into a number that
+# describes neither.
+MODEL_TAG="$(te_model_tag "$MODEL")"
+DEST="$RESULTS_ROOT/$(te_tree_name "$AGENT" "$DOMAIN" "$MODEL_TAG")"
+mkdir -p "$DEST"
+
+# DRYRUN=1 prints the cells this launcher would run and exits WITHOUT detaching. It has to
+# come before te_detach: this script used to detach first, so `DRYRUN=1 bash
+# scripts/launch/trimech_ab.sh` returned the terminal and then billed a full A/B in the
+# background -- the opposite of what a dry run is for.
+if [[ "${DRYRUN:-0}" == "1" ]]; then
+ echo "[trimech_ab] DRYRUN -- agent=$AGENT domain=$DOMAIN model=$MODEL"
+ echo "[trimech_ab] dest=$DEST"
+ echo "[trimech_ab] the cells this lane would run, in order:"
+ for seed in $SEEDS; do for repo in $REPOS; do for arm in $ARMS; do
+ printf ' %s\t%s\t%s\n' "$arm" "$repo" "$seed"
+ done; done; done
+ echo "[trimech_ab] DRYRUN: nothing was run."
+ exit 0
+fi
+
+# Detach by default: config is resolved and credentials are checked, so anything that
+# fails from here on is a run-time failure that belongs in the log, not the terminal.
+te_detach "trimech_ab__${AGENT}_${DOMAIN}_${ARM}" "$0" "$@"
+
+echo "[trimech_ab] agent=$AGENT domain=$DOMAIN arms=($ARMS) repos='$REPOS' seeds='$SEEDS' runner=$RUNNER"
+echo "[trimech_ab] dest=$DEST tasks=$TASKS_FILE workload=$WORKLOAD"
+
+for seed in $SEEDS; do
+ for repo in $REPOS; do
+ for arm in $ARMS; do
+ "$PY" "$HERE/../run_experiment.py" \
+ --arm "$arm" --agent "$AGENT" --repo "$repo" --seed "$seed" --limit "$LIMIT" \
+ --adapter "$DOMAIN" --tasks-file "$TASKS_FILE" "${WORKLOAD_ARGS[@]}" \
+ --results-root "$DEST" --runner "$RUNNER" \
+ --model "$MODEL" --optimizer-model "$AGENT" --fresh \
+ || { echo "[trimech_ab] FAILED arm=$arm repo=$repo seed=$seed"; exit 1; }
+ done
+ done
+done
+
+echo "[trimech_ab] aggregating -> $DEST/index/"
+# Aggregate ALL arms present, not just the ones this invocation ran: --arms limits the
+# rewrite and would drop every other arm's rows from the index the comparisons read.
+"$PY" "$HERE/../aggregate.py" "$DEST"
+echo "[trimech_ab] DONE. Compare an arm with its baseline, cell by cell:"
+echo " $PY scripts/compare_arms.py --baseline $DEST/baseline/<cell> --arm $DEST/<arm>/<cell>"
diff --git a/experiments/scripts/run_experiment.py b/experiments/scripts/run_experiment.py
new file mode 100644
index 0000000..74aa605
--- /dev/null
+++ b/experiments/scripts/run_experiment.py
@@ -0,0 +1,481 @@
+#!/usr/bin/env python3
+"""Reproducible driver for the TokenEconomizer main experiments.
+
+One invocation runs ONE arm over ONE repo stream and writes the same results layout the
+aggregator + figures consume (``<results-root>/<arm>/<repo>_s<seed>/<run>/row.json`` plus a
+per-repo ``_streams/*.jsonl``). It is the single source of truth mapping each *proven* arm
+to an exact ``EconomizerConfig`` -- the `ARMS` table below reproduces the flag sets the
+original launches used:
+
+ baseline (no mechanisms) -> native cost basis
+ baseline-low baseline at TE_EFFORT_LEVEL=low (arm-owned, see Arm.env)
+ baseline-medium baseline at TE_EFFORT_LEVEL=medium
+ inject-only --mode manual4 (micro-skill only)
+ untool-only --untool-evidence 3 --untool-judge (untooling only)
+ spiral-only --spiral-guard --spiral-judge --spiral-nudge (spiral only)
+ spiral-only-web spiral-only + --spiral-nudge-llm --spiral-frozen-envelope 30
+ trimech inject-only + untool-only + spiral-only (Claude / Copilot)
+ trimech-web trimech + --spiral-nudge-llm --spiral-frozen-envelope 30
+ (short Web-Bench streams)
+ codex-untool --codex-recent-adaptive-untool-v7 (Codex untooling)
+ codex-spiral --codex-spiral-resume --codex-spiral-profile defensive
+ codex-spiral-postgreen the shipped post-green-v3 finalize alone (ablation)
+ codex-combo codex-untool + codex-spiral (co-activated)
+ codex-trimech codex-inject + codex-untool + codex-spiral (all three)
+ codex-trimech-rtk codex-trimech + rtk in its INSTRUCTION mode (no hook on Codex;
+ the row carries rtk_compliance)
+ codex-trimech-repomap codex-trimech + aider's repo map as the project AGENTS.md
+ (SWE-bench only; the row carries map_ok / map_chars)
+
+ rtk / serena / repomap an external tool baseline, no mechanism active
+ trimech-rtk / -repomap trimech co-activated with one of them
+
+The single-mechanism arms use the SAME parameters trimech co-activates, so each is a true
+ablation of it rather than a separately tuned arm. The external tool scaffolds are
+implemented in `tokeneconomizer/baselines/` and install into the task container; a
+misconfigured one fails at runner construction rather than producing rows that merely look
+like a scaffold result (docs/HARNESS.md, External tool scaffolds).
+
+The arm tables live next to the code they configure -- `tokeneconomizer/mechanisms/arms.py`
+for everything the economizer drives (including the two `trimech-*` integrations, which sit
+beside `trimech`) and `tokeneconomizer/baselines/arms.py` for the pure-baseline arms. They
+are assembled in `tokeneconomizer/arms.py` and re-exported here, so this module stays the
+entry point the docs point at.
+
+The heavy task-execution harness (Docker + agent CLI + official gold grading) lives BELOW
+the ``tokeneconomizer.runner.AgentRunner`` seam, in the repo's ``te_harness`` package -- the
+one the published rows came from, so ``--runner te_harness.swebench:make_runner`` reproduces
+them as committed (docs/HARNESS.md). Any other harness works the same way: pass
+``--runner module:factory`` (a zero-arg callable returning an AgentRunner). ``--runner stub``
+selects the built-in offline StubRunner used by the smoke test -- it writes a synthetic
+transcript and a plausible row so the full facade -> aggregate -> figures pipeline can be
+exercised with no Docker, no datasets, and no LLM.
+
+Usage:
+ python scripts/run_experiment.py --arm trimech --agent claude --repo pydata/xarray \\
+ --seed 0 --results-root "$TOKENECONOMIZER_RESULTS/claude_swebench" \\
+ --tasks-file "$TOKENECONOMIZER_SWEBENCH_TASKS" --runner te_harness.swebench:make_runner
+
+Placeholders (env vars): the driver reads a --tasks-file JSON list per domain
+(TOKENECONOMIZER_SWEBENCH_TASKS / _WEBBENCH_TASKS / _CREDDATA_TASKS / _REPOSVUL_TASKS); the
+adapters + harness separately read the dataset roots (_WEBBENCH_PROJECTS / _CREDDATA_ROOT /
+_REPOSVUL_ROOT / _CYBERGYM_SUITE). See docs/BENCHMARKS.md. The results root is git-ignored.
+"""
+import argparse
+import collections
+import importlib
+import inspect
+import json
+import os
+import sys
+import time
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [os.path.dirname(os.path.abspath(__file__)), # _env, sweep_plan
+ _EXPERIMENTS, # te_harness
+ os.path.dirname(_EXPERIMENTS)] # the core, at the repository root
+from _env import load_env # noqa: E402
+from tokeneconomizer import Economizer, EconomizerConfig # noqa: E402
+from tokeneconomizer import ledger # noqa: E402
+from tokeneconomizer.config import needs_optimizer # noqa: E402
+
+
+# ---- arm registry ------------------------------------------------------------------------
+# Re-exported so this module stays the entry point the docs and every existing import point
+# at. The definitions moved next to the code they configure:
+#
+# tokeneconomizer/mechanisms/arms.py naive, effort variants, single-mechanism ablations,
+# trimech, trimech-web, the two trimech-<scaffold>
+# integrations (siblings of trimech), codex-*
+# tokeneconomizer/baselines/arms.py rtk / serena / repomap -- external tool only
+# tokeneconomizer/arms.py assembles both in canonical order
+#
+# `Arm`, `cfg` and `arm` come from tokeneconomizer._armspec; `_cfg`/`_arm` remain as aliases
+# because tests and older callers refer to them by those names.
+from tokeneconomizer.arms import ARMS, SCAFFOLD_ARMS # noqa: E402,F401
+from tokeneconomizer._armspec import Arm, arm as _arm, cfg as _cfg # noqa: E402,F401
+
+
+# ---- built-in offline StubRunner (smoke test only) ---------------------------------------
+class StubRunner:
+ """A harness stand-in that runs NO agent. It writes a synthetic transcript in the
+ agent's schema and returns a plausible cache-heavy row, so the mechanisms' post-task
+ hooks and the aggregator/figures pipeline can be exercised end-to-end offline.
+
+ It is NOT a benchmark: `resolved` is a fixed deterministic pattern, not a real grade.
+ """
+ def __init__(self, agent):
+ self.agent = agent
+ self._codex = agent == "codex"
+
+ def optimizer_call(self, prompt, model="agent"):
+ # Benign, parseable optimizer output; ~40 effective tokens per call.
+ return '{"extractors": [], "facts": []}', 40.0
+
+ def _write_transcript(self, run_dir_path):
+ path = os.path.join(run_dir_path, "transcript.jsonl")
+ if self._codex:
+ items = [{"type": "command_execution", "command": "python -m pytest -q"},
+ {"type": "command_execution", "command": "sed -n '1,40p' setup.py"}]
+ lines = []
+ for i, item in enumerate(items):
+ it = {"id": f"i{i}", "status": "completed"}
+ it.update(item)
+ lines.append({"type": "item.completed", "item": it})
+ else:
+ calls = [("bash", {"command": "python -m pytest -q"}, "ok"),
+ ("read", {"file_path": "setup.py"}, "...")]
+ lines = []
+ for i, (name, inp, res) in enumerate(calls):
+ tid = f"t{i}"
+ lines.append({"message": {"content": [
+ {"type": "tool_use", "id": tid, "name": name, "input": inp}]}})
+ lines.append({"message": {"content": [
+ {"type": "tool_result", "tool_use_id": tid, "content": res}]}})
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines))
+
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ os.makedirs(run_dir_path, exist_ok=True)
+ self._write_transcript(run_dir_path)
+ idx = int(task.get("_index", 0))
+ # A real billed model id so the vendored pricing table resolves a weight; the
+ # originals billed claude-opus-4-8 (Claude Code / Copilot) and gpt-5.6-sol (Codex).
+ billed = run_kwargs.get("model") or ("gpt-5.6-sol" if self._codex
+ else "claude-opus-4-8")
+ if billed == "agent":
+ billed = "gpt-5.6-sol" if self._codex else "claude-opus-4-8"
+ row = {
+ # cache-heavy token breakdown pricing.effective_tokens() reads
+ "input_tokens": 900, "cache_read_input_tokens": 7000,
+ "cache_creation_input_tokens": 400, "output_tokens": 350,
+ "tokens_total": 8650,
+ "resolved": (idx % 3 != 0), # deterministic, NOT a real grade
+ "num_turns": 24 + idx, "is_error": False,
+ "instance": task.get("instance_id", ""),
+ "repo": task.get("repo", ""),
+ "model": billed,
+ "codex_spiral_nudges": 0,
+ }
+ return row, run_dir_path
+
+
+def load_runner(spec, agent, model=None, scaffold=None):
+ """`spec` is 'stub' or 'module.path:factory' -> AgentRunner.
+
+ The factory is offered the `--agent` label first and called zero-arg only if it does
+ not accept one. That fallback exists for older harnesses, and it is the reason the
+ label is ALSO exported as TE_AGENT: a factory that silently ignores which agent was
+ asked for will run whatever it hardcodes while the driver files the rows under
+ `--agent`'s name -- plausible numbers attributed to an agent that never ran, which is
+ worse than a crash. A harness that cannot honour the label should raise.
+
+ `scaffold` (from the arm) is offered the same way. A harness that accepts NEITHER the
+ keyword nor TE_SCAFFOLD would run NATIVE and file the rows under a scaffold's name --
+ the same class of error as the agent label -- so that case raises rather than runs.
+ """
+ if spec == "stub":
+ if scaffold:
+ # The stub runs no agent and installs nothing, so a "scaffold" row from it
+ # would be indistinguishable from a real one once it is sitting in results/.
+ print(f"[runner stub] scaffold {scaffold!r} NOT installed -- "
+ f"synthetic rows, for wiring checks only")
+ return StubRunner(agent)
+ if ":" not in spec:
+ raise SystemExit(f"--runner must be 'stub' or 'module:factory', got {spec!r}")
+ mod_name, fac_name = spec.split(":", 1)
+ factory = getattr(importlib.import_module(mod_name), fac_name)
+ os.environ.setdefault("TE_AGENT", agent or "")
+ try:
+ sig = inspect.signature(factory)
+ except (TypeError, ValueError): # builtins / C callables: no signature
+ return factory()
+ takes_kwargs = any(p.kind is inspect.Parameter.VAR_KEYWORD
+ for p in sig.parameters.values())
+ kw = {}
+ if "agent" in sig.parameters or takes_kwargs:
+ kw["agent"] = agent
+ # `--model` defaults to the sentinel "agent" (= "whatever the agent is configured to
+ # use"), which is NOT a model id -- forwarding it would make the CLI ask for a model
+ # called "agent". Only a real id is passed through; otherwise the runner falls back to
+ # its own env var (ANTHROPIC_MODEL / COPILOT_MODEL), which is the historical behaviour.
+ if model and model != "agent" and ("model" in sig.parameters or takes_kwargs):
+ kw["model"] = model
+ if scaffold:
+ if "scaffold" in sig.parameters or takes_kwargs:
+ kw["scaffold"] = scaffold
+ else:
+ raise SystemExit(
+ f"runner {spec!r} does not accept a scaffold, but this arm needs "
+ f"{scaffold!r}. A harness that ignored it would run NATIVE and file the "
+ f"rows under the scaffold's name. Add a `scaffold=` parameter (see "
+ f"tokeneconomizer/baselines/) or pick an arm without one.")
+ os.environ.setdefault("TE_SCAFFOLD", scaffold)
+ return factory(**kw) if kw else factory()
+
+
+def load_adapter(name, tasks_file, repo=""):
+ from tokeneconomizer.adapters import (
+ SwebenchAdapter, WebBenchAdapter, CredDataAdapter, ReposVulAdapter,
+ CyberGymAdapter)
+ if name == "swebench":
+ return SwebenchAdapter(tasks_path=tasks_file)
+ if name == "webbench":
+ # the Web-Bench project (= one stream) is selected by --repo; the per-task judge
+ # then reads <projects>/<project>/tasks.jsonl. Defaults to 'dom' if --repo is unset.
+ return WebBenchAdapter(repo.split("/")[-1] or "dom")
+ return {"creddata": CredDataAdapter, "reposvul": ReposVulAdapter,
+ "cybergym": CyberGymAdapter}[name]()
+
+
+def workload_ids(workload_file, repo):
+ """The set of task ids named by a `workloads/<domain>.json` manifest.
+
+ With `repo` set, restrict to that group (SWE-bench repo slug / Web-Bench project).
+ Single-group domains (CredData, ReposVul) ignore `repo` and use their only group. An
+ unrecognized group is an error, never a silent widening."""
+ with open(workload_file, encoding="utf-8") as fh:
+ groups = json.load(fh)["groups"]
+ if repo and repo in groups:
+ return set(groups[repo])
+ if len(groups) == 1:
+ return set(next(iter(groups.values())))
+ if repo:
+ raise SystemExit(f"--repo {repo!r} is not a group in {workload_file}; "
+ f"expected one of {sorted(groups)}")
+ return {i for ids in groups.values() for i in ids}
+
+
+def _same_group(task, repo):
+ """True when `task` belongs to the group `repo` names, by its own `repo` label.
+
+ Matches the full slug and the bare repo name, so both a SWE-bench `pydata/xarray`
+ and a Web-Bench `zustand` label resolve against the same --repo value."""
+ label = str(task.get("repo") or "")
+ if not label:
+ return False
+ return label == repo or label.split("/")[-1] == repo.split("/")[-1]
+
+
+def load_tasks(tasks_file, repo, limit, workload=None):
+ """Task stream as a list of {instance_id, problem_statement, repo}.
+
+ With `workload` set (a `workloads/<domain>.json` manifest), keep exactly the ids it
+ names — the reproducible path, and the one that survives repo slugs whose org differs
+ from the repo name. Otherwise fall back to the legacy `repo` prefix match."""
+ with open(tasks_file, encoding="utf-8") as fh:
+ tasks = json.load(fh)
+ if workload:
+ wanted = workload_ids(workload, repo)
+ tasks = [t for t in tasks if str(t.get("instance_id", "")) in wanted]
+ # Ids are only unique WITHIN a group in some domains: every Web-Bench project
+ # numbers its stream `init`, `task-1`..`task-20`, so an id match alone pulls in
+ # all five projects' tasks interleaved -- and a Web-Bench stream is sequentially
+ # dependent, so a mixed stream is not a smaller run but a meaningless one. When
+ # the task file labels its rows with a group, narrow to the requested one.
+ # SWE-bench ids are globally unique and its rows carry the exact repo slug, so
+ # this is a no-op there; an unlabeled task file falls through unchanged.
+ if repo:
+ grouped = [t for t in tasks if _same_group(t, repo)]
+ if grouped:
+ tasks = grouped
+ if not tasks:
+ raise SystemExit(
+ f"no tasks left after applying workload {workload}"
+ + (f" (repo={repo})" if repo else "")
+ + f"; {tasks_file} does not contain any of its {len(wanted)} ids")
+ elif repo:
+ tok = repo.split("/")[-1].replace("-", "_")
+ alt = repo.split("/")[-1]
+ tasks = [t for t in tasks
+ if str(t.get("instance_id", "")).replace("-", "_").startswith(tok)
+ or str(t.get("instance_id", "")).startswith(alt)] or tasks
+ for t in tasks:
+ t.setdefault("repo", repo)
+ return tasks[:limit] if limit else tasks
+
+
+def lift_overhead(row, srow):
+ """Copy mechanism-overhead fields a mechanism reported on `srow` into `row`.
+
+ row.json is the only artifact aggregate.py reads, so an overhead field that exists
+ solely on the stream row is invisible to the cost basis. `untool_judge_tokens` was
+ exactly that: `tooling.after_task` writes the per-task instruction-aware tool judge's
+ spend to `srow` alone, so every arm with judge=True -- untool-only and all four trimech
+ variants -- had that line item silently dropped from `cost_effective_tokens`. The bias
+ is one-directional: a baseline runs no judge, so only the arms under test are
+ flattered. Same shape as the usage_incomplete trap in aggregate.py.
+
+ A value the runner already put on `row` WINS: the harness sets spiral_judge_tokens
+ itself during run_task, and `srow` may carry a stale or zeroed copy of it.
+
+ Driven by ledger.OVERHEAD_KEYS so a newly added overhead field cannot silently
+ reintroduce the gap. Returns `row` for convenience.
+ """
+ for k in ledger.OVERHEAD_KEYS:
+ if k not in row and srow.get(k) is not None:
+ row[k] = srow[k]
+ return row
+
+
+def main():
+ # Credentials + data roots come from the git-ignored repo-root .env (see .env.example);
+ # anything already exported in the shell wins. Harmless for --runner stub, which needs
+ # no credentials at all.
+ load_env()
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--arm", required=True, choices=sorted(ARMS),
+ help="arm to run; %s additionally install an external tool scaffold "
+ "(tokeneconomizer/baselines/)"
+ % ", ".join(sorted(SCAFFOLD_ARMS)))
+ ap.add_argument("--agent", default="claude", choices=["claude", "copilot", "codex"],
+ help="the sealed agent (selects transcript schema + generic-vs-codex "
+ "mechanisms)")
+ ap.add_argument("--repo", default="", help="repo slug, e.g. pydata/xarray")
+ ap.add_argument("--seed", type=int, default=0)
+ ap.add_argument("--limit", type=int, default=0, help="cap tasks (0 = all)")
+ ap.add_argument("--adapter", default="swebench",
+ choices=["swebench", "webbench", "creddata", "reposvul", "cybergym"])
+ ap.add_argument("--tasks-file", required=True,
+ help="JSON list of {instance_id, problem_statement, repo} (a placeholder "
+ "you point at your dataset export; see docs/BENCHMARKS.md)")
+ ap.add_argument("--workload", default="",
+ help="workloads/<domain>.json manifest naming the exact evaluated "
+ "instances; --tasks-file is filtered to it (and to --repo's group). "
+ "Committed in this repo — the reproducible way to pin a run.")
+ ap.add_argument("--results-root", required=True,
+ help="<domain> results root; the arm subdir is created under it")
+ ap.add_argument("--runner", default="stub",
+ help="'stub' (offline) or 'module:factory' returning an AgentRunner, "
+ "e.g. te_harness.swebench:make_runner (the published harness)")
+ ap.add_argument("--model", default="agent")
+ # An AGENT LABEL, not a model id. The between-task optimizer runs on the agent's own
+ # model by construction (see the harness runners), so naming a model here would be the
+ # one remaining way to charge mechanism overhead to a cheaper model than the arm was
+ # measured on -- which would flatter the economized arm against a baseline that made
+ # no such calls. A model id is refused rather than quietly diverting the ledger.
+ ap.add_argument("--optimizer-model", default="agent",
+ help="agent label whose model runs the between-task optimizer "
+ "(agent|claude|copilot|codex...); NOT a model id")
+ ap.add_argument("--max-turns", type=int, default=80)
+ ap.add_argument("--fresh", action="store_true",
+ help="ignore any saved cross-task state (start the stream cold)")
+ args = ap.parse_args()
+
+ if "-" in args.optimizer_model and any(
+ args.optimizer_model.startswith(p)
+ for p in ("claude-", "gpt-", "gemini-", "o1-", "o3-", "o4-")):
+ raise SystemExit(
+ f"--optimizer-model {args.optimizer_model!r} looks like a model id, but this "
+ "flag takes an AGENT LABEL. The between-task optimizer runs on the agent's "
+ "own model so mechanism overhead is priced on the same basis as the task "
+ "rows; pass --optimizer-model agent (or the agent's name).")
+
+ arm = ARMS[args.arm]
+ mode, cfg_factory = arm.mode, arm.cfg
+ # An arm-owned environment overlay (reasoning effort) is applied BEFORE the runner is
+ # built, because the harness reads it at construction. The arm WINS over an inherited
+ # value -- the name is the authority for what the cell contains -- so say so when it
+ # overrides something the caller had set, rather than changing it silently.
+ for key, value in arm.env.items():
+ prior = os.environ.get(key)
+ if prior is not None and prior != value:
+ print(f"[arm {args.arm}] {key}: overriding inherited {prior!r} with {value!r}")
+ os.environ[key] = value
+ if arm.env:
+ print(f"[arm {args.arm}] env: " +
+ " ".join(f"{k}={v}" for k, v in sorted(arm.env.items())))
+ if arm.scaffold:
+ print(f"[arm {args.arm}] external tool scaffold: {arm.scaffold}")
+ config = cfg_factory()
+ # Tell the harness whether any mechanism on this arm will make a paid between-task
+ # call, BEFORE the runner is built. A harness whose mechanism transport needs a live
+ # preflight (te_harness.copilot's Messages endpoint) must not bill one, or depend on
+ # one, for `baseline` -- which never calls it. Same launcher->harness seam as TE_AGENT
+ # / TE_SCAFFOLD / TE_EFFORT_APPLIES; unset means "assume it might", which is the safe
+ # default for any caller that does not set it.
+ os.environ["TE_OPTIMIZER_PREFLIGHT"] = "1" if needs_optimizer(config) else "0"
+ adapter = load_adapter(args.adapter, args.tasks_file, args.repo)
+ runner = load_runner(args.runner, args.agent, args.model, arm.scaffold)
+ tasks = load_tasks(args.tasks_file, args.repo, args.limit, args.workload)
+ if not tasks:
+ raise SystemExit("no tasks after filtering")
+
+ repo_tag = (args.repo.split("/")[-1] or "repo").replace("-", "_")
+ cell = f"{repo_tag}_s{args.seed}"
+ arm_root = os.path.join(args.results_root, args.arm, cell)
+ streams_dir = os.path.join(arm_root, "_streams")
+ os.makedirs(streams_dir, exist_ok=True)
+
+ econ = Economizer(config, agent=args.agent, repo=args.repo or repo_tag,
+ results_root=arm_root, fresh=args.fresh,
+ optimizer_model=args.optimizer_model, max_turns=args.max_turns,
+ runner=runner, adapter=adapter)
+
+ # Optional stream-level seam. Most harnesses own nothing beyond a single task, but
+ # some own the whole stream: Web-Bench's evaluator drives its project's 20 sequentially
+ # dependent tasks itself, so it has to be told the stream before the first task and be
+ # allowed to shut down after the last. A harness without these hooks is unaffected.
+ if hasattr(runner, "begin_stream"):
+ runner.begin_stream(tasks, results_root=arm_root, arm=args.arm,
+ repo=args.repo, seed=args.seed)
+
+ stream_path = os.path.join(streams_dir, f"{repo_tag}__{mode}__stream.jsonl")
+ n_resolved = 0
+ try:
+ with open(stream_path, "w", encoding="utf-8") as stream:
+ for i, task in enumerate(tasks):
+ task["_index"] = i
+ injected = econ.injection_prefix(task)
+ exclude_tools = econ.build_tools(task, i)
+ run_kwargs = dict(econ.run_kwargs(task))
+ run_kwargs.setdefault("model", args.model)
+
+ # Run-dir name: UTC timestamp first, then the stream index. The stamp
+ # makes the directory unique, so re-running a cell ARCHIVES the previous
+ # attempt beside the new one instead of overwriting its transcript in
+ # place -- traces are the primary artifact and are not reproducible after
+ # the fact. Timestamp-first also matches the historical results trees and
+ # sorts chronologically; the index keeps stream order explicit and breaks
+ # ties when two tasks land in the same second (an unreached Web-Bench task
+ # costs no wall-clock at all). `mode` stays in the name because
+ # aggregate.py recovers it from there.
+ stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
+ run_name = (f"{stamp}__{i:03d}__{args.agent}"
+ f"__{task.get('instance_id','inst')}"
+ f"__{args.arm}__s{args.seed}__{mode}")
+ run_dir = os.path.join(arm_root, run_name)
+ row, saved_dir = runner.run_task(
+ task, arm=args.arm, injected=injected, exclude_tools=exclude_tools,
+ run_dir_path=run_dir, **run_kwargs)
+
+ srow = {"resolved": row.get("resolved"), "num_turns": row.get("num_turns"),
+ "instance": task.get("instance_id", "")}
+ opt_tokens, reason = econ.after_task(task, row, saved_dir, srow)
+ row.setdefault("seed", args.seed)
+ row.setdefault("run_dir", saved_dir)
+ row["optimizer_tokens"] = row.get("optimizer_tokens", 0) + (opt_tokens or 0)
+ lift_overhead(row, srow)
+
+ with open(os.path.join(saved_dir, "row.json"), "w", encoding="utf-8") as fh:
+ json.dump(row, fh, indent=2)
+ stream.write(json.dumps({**srow, **{k: row.get(k) for k in (
+ "input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens", "tokens_total",
+ "optimizer_tokens", "codex_spiral_nudges") + ledger.OVERHEAD_KEYS
+ if row.get(k) is not None or k in srow}}) + "\n")
+ n_resolved += 1 if row.get("resolved") is True else 0
+ finally:
+ # A harness holding a stream-level resource (an evaluator process, a container
+ # pool) must be able to release it even when the loop died mid-stream.
+ if hasattr(runner, "end_stream"):
+ runner.end_stream()
+
+ print(f"arm={args.arm} agent={args.agent} repo={args.repo or repo_tag} "
+ f"seed={args.seed} tasks={len(tasks)} resolved={n_resolved} -> {arm_root}")
+ print(f"stream -> {stream_path}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/summarize_stream.py b/experiments/scripts/summarize_stream.py
new file mode 100644
index 0000000..785a836
--- /dev/null
+++ b/experiments/scripts/summarize_stream.py
@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+"""
+Amortized-cost aggregation for stream runs.
+
+A stream run (stream_run.py) writes one JSONL row per task to
+results/<root>/_streams/<repo>__<mode>__<ts>.jsonl. Each row records the task's
+EXECUTION cost (`tokens_total`, what the agent spent solving the task — this
+already includes the injected manifest/notes/carryover-log it read) and, kept in
+a SEPARATE field, the per-task OPTIMIZER overhead (`optimizer_tokens`, the real
+token cost of the gated/ungated note-writing call; 0 when the gate didn't fire).
+
+The HONEST amortized cost of a mode is the sum of BOTH:
+
+ task_cost(t) = tokens_total(t) + optimizer_tokens(t)
+ amortized(T) = (1/T) * sum_{t<=T} task_cost(t)
+
+Reporting only `tokens_total` flatters `full`/`self-memory` (which pay optimizer
+overhead) against `standalone`/`carryover` (which don't). This module folds the two
+together so every mode is compared on total tokens actually spent.
+
+Usage:
+ python summarize_stream.py results/v1/_streams/*.jsonl # compare modes
+ python summarize_stream.py --curve run.jsonl # per-position curve
+
+Cost is reported CACHE-AWARE by default: per-task tokens are reweighted into
+fresh-input-token equivalents (pricing.py) because `tokens_total` counts cached
+re-reads (~90% of tokens) at full weight, which biases A/Bs. Pass --raw to report
+the legacy full-weight `tokens_total` instead.
+"""
+import argparse, glob, json, os, sys
+
+# Import the vendored pricing table + the honest ledger from the tokeneconomizer package.
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+from tokeneconomizer import ledger
+from tokeneconomizer.vendor import pricing
+
+
+def _num(x):
+ return x if isinstance(x, (int, float)) else 0
+
+
+_RUNDIR_CACHE = {}
+
+
+def _row_breakdown(r, stream_path):
+ """Return the per-task token breakdown for cache-aware costing. Prefer the fields
+ on the stream row itself (new runs carry them); fall back to the per-task row.json
+ in the run dir (older streams predate the inline fields); else return the row as-is
+ (effective_tokens then falls back to tokens_total)."""
+ if pricing.has_breakdown(r):
+ return r
+ run_dir = r.get("run_dir")
+ if run_dir:
+ # stream file lives at <root>/_streams/<file>.jsonl -> run dirs are under <root>
+ root = os.path.dirname(os.path.dirname(os.path.abspath(stream_path)))
+ rj = os.path.join(root, run_dir, "row.json")
+ if rj in _RUNDIR_CACHE:
+ return _RUNDIR_CACHE[rj] or r
+ loaded = None
+ if os.path.isfile(rj):
+ try:
+ loaded = json.load(open(rj))
+ except (OSError, json.JSONDecodeError):
+ loaded = None
+ _RUNDIR_CACHE[rj] = loaded
+ if loaded is not None:
+ return loaded
+ return r
+
+
+def summarize(path, raw=False, exclude_init=False):
+ """Read one stream JSONL and return a summary dict with per-position curves and
+ cumulative totals. `task_cost = exec_cost + optimizer_tokens`. By default exec_cost
+ is the CACHE-AWARE effective cost (pricing.effective_tokens); with raw=True it is the
+ legacy full-weight `tokens_total`. Optimizer tokens are counted at face value (they
+ are one-shot, largely uncached calls and small relative to exec).
+
+ `exclude_init` drops the Web-Bench `init` scaffold row (instance == "init") before any
+ aggregation, so every project's amortized cost normalizes over the SAME 20 graded tasks
+ (calculator/table ship an `init` record; react/expressjs/zustand do not). Default False keeps
+ SWE-bench byte-identical -- SWE-bench streams never contain an `init` row."""
+ rows = []
+ with open(path) as f:
+ for line in f:
+ line = line.strip()
+ if line:
+ try:
+ rows.append(json.loads(line))
+ except json.JSONDecodeError:
+ pass
+ if exclude_init:
+ rows = [r for r in rows if r.get("instance") != "init"]
+ rows.sort(key=lambda r: _num(r.get("position")))
+
+ curve, cum_exec, cum_opt, cum_resolved = [], 0, 0, 0
+ cum_raw_exec = 0
+ n_graded = 0
+ n_with_breakdown = 0
+ n_mech_effective = 0 # rows whose mechanism tokens are cache-aware effective
+ n_mech_raw = 0 # legacy rows with raw (face-value) mechanism tokens present
+ for r in rows:
+ rec = _row_breakdown(r, path)
+ raw_exec = _num(r.get("tokens_total"))
+ eff_exec = pricing.effective_tokens(rec)
+ if pricing.has_breakdown(rec):
+ n_with_breakdown += 1
+ exec_t = raw_exec if raw else eff_exec
+ # OPTIMIZER + MECHANISM-INTERNAL overhead. optimizer_tokens is the note-writing
+ # call; untool_classify_tokens is the one-shot tool-classifier LLM call;
+ # untool_judge_tokens is the per-task instruction-aware tool judge; spiral_judge_tokens
+ # is the paid live trace-judge; spiral_nudge_writer_tokens is the v6.1 targeted-nudge
+ # writer. All are real mechanism-internal LLM cost that must be amortized into the
+ # honest task cost. The overhead-key set is the SINGLE definition in
+ # economizer.ledger.OVERHEAD_KEYS is now tokeneconomizer.ledger.OVERHEAD_KEYS,
+ # shared with the emit path so the two never drift.
+ # (This now INCLUDES spiral_nudge_writer_tokens, previously emitted but never summed
+ # -- a latent under-count of amortized cost, fixed per the unified-ledger decision.)
+ # BASIS: rows tagged mech_tokens_basis=="effective" carry cache-aware effective
+ # tokens (same basis as exec) — sum directly. Legacy rows (no tag) carry RAW
+ # face-value tokens: an UNDER-count of effective (mechanism calls are uncached
+ # one-shots, so effective = input + 5*output >= raw). We sum them as a documented
+ # floor and surface the mixed basis in the summary rather than silently blending.
+ opt_t = ledger.overhead(r)
+ mech_basis = r.get("mech_tokens_basis") # "effective" for new rows, None legacy
+ if opt_t > 0:
+ if mech_basis == "effective":
+ n_mech_effective += 1
+ else:
+ n_mech_raw += 1
+ cum_exec += exec_t
+ cum_raw_exec += raw_exec
+ cum_opt += opt_t
+ resolved = r.get("resolved")
+ if resolved is not None:
+ n_graded += 1
+ if resolved:
+ cum_resolved += 1
+ k = len(curve) + 1
+ curve.append({
+ "position": r.get("position"),
+ "instance": r.get("instance"),
+ "exec_tokens": round(exec_t, 1),
+ "optimizer_tokens": opt_t,
+ "task_cost": round(exec_t + opt_t, 1),
+ "cum_exec_tokens": round(cum_exec, 1),
+ "cum_optimizer_tokens": cum_opt,
+ "cum_total_tokens": round(cum_exec + cum_opt, 1),
+ "amortized_cost_per_task": round((cum_exec + cum_opt) / k, 1),
+ "resolved": resolved,
+ })
+
+ n = len(rows)
+ mode = rows[0].get("mode") if rows else None
+ cum_total = cum_exec + cum_opt
+ return {
+ "path": path,
+ "mode": mode,
+ "cost_basis": "tokens_total (raw)" if raw else "cache-aware effective",
+ "n_tasks": n,
+ "n_graded": n_graded,
+ "n_with_breakdown": n_with_breakdown,
+ # mechanism-token basis audit: how many costed rows carry effective vs legacy-raw
+ # mechanism tokens. "mixed"/"raw" is a flag that the overhead is an under-count.
+ "mech_tokens_basis": ("effective" if n_mech_raw == 0 and n_mech_effective > 0
+ else "raw" if n_mech_effective == 0 and n_mech_raw > 0
+ else "mixed" if n_mech_effective and n_mech_raw
+ else "none"),
+ "n_mech_effective": n_mech_effective,
+ "n_mech_raw": n_mech_raw,
+ "resolved": cum_resolved,
+ "resolve_rate": round(cum_resolved / n_graded, 3) if n_graded else None,
+ "cum_exec_tokens": round(cum_exec, 1),
+ "cum_raw_exec_tokens": cum_raw_exec,
+ "cum_optimizer_tokens": cum_opt,
+ "cum_total_tokens": round(cum_total, 1),
+ "optimizer_overhead_frac": round(cum_opt / cum_total, 4) if cum_total else 0.0,
+ "amortized_cost_per_task": round(cum_total / n, 1) if n else 0.0,
+ "exec_cost_per_task": round(cum_exec / n, 1) if n else 0.0,
+ "raw_exec_cost_per_task": round(cum_raw_exec / n, 1) if n else 0.0,
+ "curve": curve,
+ }
+
+
+def print_curve(s):
+ print(f"\n# {s['mode']} ({s['n_tasks']} tasks) {os.path.basename(s['path'])}")
+ print(f"{'pos':>4} {'instance':<24} {'exec':>9} {'opt':>8} {'task':>9} "
+ f"{'cum_total':>11} {'amort/task':>11} {'resolved':>8}")
+ for c in s["curve"]:
+ print(f"{str(c['position']):>4} {str(c['instance'])[:24]:<24} "
+ f"{c['exec_tokens']:>9} {c['optimizer_tokens']:>8} {c['task_cost']:>9} "
+ f"{c['cum_total_tokens']:>11} {c['amortized_cost_per_task']:>11} "
+ f"{str(c['resolved']):>8}")
+
+
+def print_compare(summaries):
+ basis = summaries[0]["cost_basis"] if summaries else "?"
+ print(f"\ncost basis: {basis} (cache-aware = fresh-input-token equivalents; "
+ f"pass --raw for legacy tokens_total)")
+ print(f"\n{'mode':<14} {'tasks':>5} {'resolve':>8} {'exec/task':>11} "
+ f"{'opt/task':>10} {'amort/task':>11} {'opt%':>6}")
+ print("-" * 70)
+ for s in summaries:
+ opt_per = round(s["cum_optimizer_tokens"] / s["n_tasks"], 1) if s["n_tasks"] else 0
+ rr = "n/a" if s["resolve_rate"] is None else f"{s['resolve_rate']:.3f}"
+ print(f"{str(s['mode']):<14} {s['n_tasks']:>5} {rr:>8} "
+ f"{s['exec_cost_per_task']:>11.0f} {opt_per:>10} "
+ f"{s['amortized_cost_per_task']:>11.0f} "
+ f"{s['optimizer_overhead_frac']*100:>5.1f}%")
+ # warn if any run lacked the breakdown (effective cost fell back to tokens_total)
+ missing = [s for s in summaries if s["n_with_breakdown"] < s["n_tasks"]
+ and "raw" not in s["cost_basis"]]
+ if missing:
+ print("\n NOTE: some tasks lacked a token breakdown; effective cost fell back "
+ "to tokens_total for those:")
+ for s in missing:
+ print(f" {os.path.basename(s['path'])}: "
+ f"{s['n_with_breakdown']}/{s['n_tasks']} tasks had a breakdown")
+ print("\nexec/task = mean execution cost (cache-aware unless --raw; incl. injected context)")
+ print("opt/task = mean optimizer overhead (face value; gating is free)")
+ print("amort/task = (exec + optimizer) / tasks <- the HONEST amortized cost")
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("paths", nargs="+", help="stream JSONL file(s) or globs")
+ ap.add_argument("--curve", action="store_true",
+ help="also print the per-position amortized-cost curve for each run")
+ ap.add_argument("--raw", action="store_true",
+ help="report legacy full-weight tokens_total instead of cache-aware cost")
+ ap.add_argument("--exclude-init", action="store_true",
+ help="drop the Web-Bench `init` scaffold row so cost normalizes over the "
+ "20 graded tasks (no-op on SWE-bench streams, which have no init row)")
+ ap.add_argument("--json", action="store_true", help="emit machine-readable JSON")
+ args = ap.parse_args()
+
+ files = []
+ for p in args.paths:
+ files.extend(sorted(glob.glob(p)) if any(c in p for c in "*?[") else [p])
+ files = [f for f in files if os.path.isfile(f)]
+ if not files:
+ print("no stream JSONL files matched", file=sys.stderr)
+ sys.exit(1)
+
+ summaries = [summarize(f, raw=args.raw, exclude_init=args.exclude_init) for f in files]
+ if args.json:
+ print(json.dumps(summaries, indent=2))
+ return
+ if args.curve:
+ for s in summaries:
+ print_curve(s)
+ print_compare(summaries)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/sweep_plan.py b/experiments/scripts/sweep_plan.py
new file mode 100644
index 0000000..e8d4b76
--- /dev/null
+++ b/experiments/scripts/sweep_plan.py
@@ -0,0 +1,395 @@
+#!/usr/bin/env python3
+"""Enumerate and cost a sweep BEFORE running it.
+
+A sweep is a product: arms x repos x seeds, per (agent, model, domain) tree. At the sizes
+the main experiments ask for that product is five figures of task-runs, and the only cheap
+moment to notice is before the first container starts. So this module is the one place the
+matrix is written down, and it answers three questions offline (no Docker, no datasets, no
+API key):
+
+ --format cells one `arm<TAB>repo<TAB>seed` line per cell, for a launcher to loop over
+ --format estimate task-runs, wall-clock and dollars for the requested slice
+ --format matrix the estimate for the WHOLE sweep, every agent x model x domain
+
+Task counts come from the committed `workloads/<domain>.json` manifest -- the same file
+the launcher passes as `--workload` and the driver filters the task stream to -- so the
+plan cannot drift from what the run actually executes. That matters beyond tidiness:
+`main_sweep.sh` decides a cell is finished by comparing its stored row count against this
+number, so a plan that counted excluded instances would re-run completed cells forever.
+
+Rate assumptions are declared, not hidden: `--sec-per-task` and `--usd-per-task` default to
+what the stored Claude-direct SWE-bench tree actually measured, and both are printed with
+the estimate so a reader can rescale it. They are means over a heavy-tailed distribution;
+treat the output as an order of magnitude, not a quote.
+
+Usage:
+ python scripts/sweep_plan.py --format matrix
+ python scripts/sweep_plan.py --domain swebench --arms baseline trimech \\
+ --repos django/django --seeds 0 1 2 3 4 --format cells
+"""
+import argparse
+import json
+import os
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+from tokeneconomizer import workloads # noqa: E402
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from run_experiment import ARMS, SCAFFOLD_ARMS # noqa: E402
+
+
+# ---- the main-experiment matrix ---------------------------------------------------------
+# The arms of the main sweep, per domain. Web-Bench swaps in the two spiral arms tuned for
+# its 20-21-task streams (`-web`: LLM nudge + frozen envelope), because its streams are too
+# short for the within-stream envelope to warm up.
+SWEEP_ARMS = {
+ "swebench": ("baseline", "baseline-medium", "baseline-low",
+ "inject-only", "untool-only", "spiral-only", "trimech",
+ # external tool baselines + the two well-defined combinations
+ "rtk", "serena", "repomap", "trimech-rtk", "trimech-repomap"),
+ # Web-Bench carries the GENERALIZATION claim, not the decomposition -- see
+ # SWEEP_ARMS_WEBBENCH_OUT_OF_SCOPE for what that buys and what it gives up.
+ "webbench": ("baseline", "baseline-medium", "baseline-low", "trimech-web",
+ # the one external tool baseline that needs no container and no Python
+ "rtk"),
+}
+# Runnable on Web-Bench, deliberately NOT in its sweep. Distinct from
+# SWEEP_ARMS_WEBBENCH_UNAVAILABLE below, which is about what the harness CANNOT host: pass
+# any of these to `--arms` and it runs.
+#
+# The decomposition (`inject-only` / `untool-only` / `spiral-only-web`) and the combination
+# arm (`trimech-rtk-web`) are answered on SWE-bench, at 12 arms x 6 repos x 5 seeds. Three
+# reasons not to pay for them twice:
+#
+# 1. Short streams distort exactly these arms. Injection needs a 5-task proposer cadence
+# and untooling needs 3 consecutive unused tasks; a Web-Bench stream is 20-21 tasks
+# against 39-60 on SWE-bench, so a single-mechanism arm there measures the mechanism
+# in WARM-UP, not in steady state. `trimech-web` at least has all three compounding.
+# 2. `spiral-only-web` is the weakest buy in the matrix. Spiral fired 0 times across 498
+# opus-5/sonnet-5 rows and 1 nudge / 3 aborts across 224 opus-4.8 rows; Web-Bench
+# streams are a third the length AND run a frozen envelope (they are too short for the
+# within-stream one to warm up), so the arm would cost 100 cells to report zero.
+# 3. 5 projects is ~5 project-level observations per arm, and they are wildly
+# heterogeneous (one project is a single index.html, another grows to ~140 files).
+# Separating inject-only from untool-only at that resolution was never going to reach
+# significance, so cutting them mostly removes numbers nobody could lean on.
+#
+# What this gives up, said plainly: without `untool-only` here, "is trimech-web just
+# untooling?" cannot be answered ON Web-Bench -- only on SWE-bench. That is the accepted
+# cost of framing Web-Bench as generalization rather than decomposition: there the three
+# are measured as a combined stack.
+SWEEP_ARMS_WEBBENCH_OUT_OF_SCOPE = ("inject-only", "untool-only", "spiral-only-web",
+ "trimech-rtk-web")
+# Which scaffold arms Web-Bench CANNOT host, and why -- said here rather than leaving the
+# reader to notice the two lists differ.
+#
+# `rtk` runs there (since 2026-09-05): it is a static binary plus a PreToolUse hook that
+# wraps `ls`/`find`/`grep` and compresses their output before it reaches context, so it
+# needs neither a container nor a language, and the Web-Bench agent does issue those
+# commands. te_harness.webbench.WEBBENCH_SCAFFOLDS is the runner-side twin of this set.
+#
+# The other two do not:
+# repomap -- its generator walks `*.py` (baselines/_repomap_gen.py) and a Web-Bench
+# project is TypeScript/JavaScript, so the map would come back empty. Needs a
+# language change, not a wiring change.
+# serena -- an MCP navigation server for a codebase. Web-Bench hands the agent a file
+# set with no installed project, and the projects run 1-140 files, so there is
+# essentially nothing to navigate.
+# Generic arms only: this planner enumerates the Claude / Copilot sweep. The Codex catalog
+# (docs/RUNNING.md section 5) is launched by its own loops, and its Web-Bench list
+# already leaves out the two repomap-bearing arms for the same reason as here.
+SWEEP_ARMS_WEBBENCH_UNAVAILABLE = ("serena", "repomap", "trimech-repomap")
+# `trimech-rtk` is not "unavailable" on Web-Bench -- it is SUPERSEDED there by
+# `trimech-rtk-web`, exactly as `trimech` is by `trimech-web`. Listing it as unavailable
+# would claim the scaffold cannot run, which is false and would mislead the reader.
+SWEEP_ARMS_WEBBENCH_SUPERSEDED = {"trimech": "trimech-web",
+ "spiral-only": "spiral-only-web",
+ "trimech-rtk": "trimech-rtk-web"}
+# Every arm that carries an external tool baseline, from the baselines registry itself so
+# this module cannot drift from it. Used only for reporting.
+try:
+ from tokeneconomizer.baselines import ARM_SCAFFOLDS as _ARM_SCAFFOLDS
+ SCAFFOLD_ARM_NAMES = frozenset(_ARM_SCAFFOLDS)
+except ImportError: # pragma: no cover - core is stdlib
+ SCAFFOLD_ARM_NAMES = frozenset()
+
+# The agents, and the models they are crossed with. Model ids are what the CLI is passed
+# and what the row records; the tree name gets the short tag (te_model_tag in
+# scripts/launch/_env.sh), so every model in this list lands in its OWN tree and lanes for
+# different models can run at the same time without ever sharing a cell.
+SWEEP_AGENT_NAMES = ("claude", "copilot")
+
+# Verified 2026-09-12 on the Copilot CLI (serves + accepts --effort). Any id here can be
+# run; te_model_tag gives each a distinct tag and pricing.MODEL_WEIGHTS prices each, so
+# adding one needs no other change. Spelled the Anthropic API's way (dashes); the launcher
+# re-spells a two-spelling id for the one backend that differs (the Copilot CLI's own
+# catalog says `claude-opus-4.8`) -- see te_model_for_agent in scripts/launch/_env.sh.
+SWEEP_MODELS_SUPPORTED = ("claude-opus-5", "claude-sonnet-5", "claude-opus-4-8")
+
+# What the DEFAULT sweep runs. Deliberately not all of SWEEP_MODELS_SUPPORTED: the matrix is
+# linear in this list, so adding a third model is a 1.5x bill on a sweep already estimated
+# at five figures. That is the experiment owner's call, not a default. Override
+# without editing this file:
+#
+# TE_SWEEP_MODELS="claude-opus-4-8 claude-sonnet-5 claude-opus-5" \
+# python scripts/sweep_plan.py --format matrix
+#
+# Accepts spaces or commas. An id outside SWEEP_MODELS_SUPPORTED is allowed but warned
+# about, since the check above has not been run for it.
+#
+# `claude-opus-5` is NOT here on purpose, and that is NOT the same as optional. Its lanes
+# are required -- they just run a 4-arm subset instead of the full matrix (baseline /
+# baseline-medium / baseline-low / trimech, both domains -- the Copilot and Claude run
+# books, 3b), so they are launched by their own commands rather than by
+# crossing this list. Listing it here would plan a third FULL 21-arm lane, which is a
+# different and much larger thing: it costs ~2.5x per call what claude-opus-4-8 does on the
+# Copilot path, and the matrix is linear in this list.
+SWEEP_MODELS = ("claude-opus-4-8", "claude-sonnet-5")
+
+
+def _sweep_models():
+ """The model list, honouring TE_SWEEP_MODELS. Unknown ids warn rather than raise --
+ a new vendor id should be runnable the day it appears, not after a code change."""
+ raw = os.environ.get("TE_SWEEP_MODELS", "")
+ if not raw.strip():
+ return list(SWEEP_MODELS)
+ models = [m for m in raw.replace(",", " ").split() if m]
+ # Compare with separators folded, the way te_model_tag does: claude-opus-4-8 and
+ # claude-opus-4.8 are one model spelled for two catalogs, not two models to list.
+ known = {m.replace(".", "").replace("-", "") for m in SWEEP_MODELS_SUPPORTED}
+ unknown = [m for m in models
+ if m.replace(".", "").replace("-", "") not in known]
+ # Warn once per process: this helper is called several times while formatting a matrix,
+ # and three identical notices read like three different problems.
+ if unknown and not _sweep_models._warned:
+ _sweep_models._warned = True
+ print(f"[sweep_plan] note: {' '.join(unknown)} not in SWEEP_MODELS_SUPPORTED -- "
+ f"run one paid cell on it (docs/RUNNING.md section 3) before spending",
+ file=sys.stderr)
+ return models
+
+
+_sweep_models._warned = False
+
+
+def sweep_agents():
+ """(agent, model) pairs: every agent crossed with every requested model."""
+ return tuple((a, m) for a in SWEEP_AGENT_NAMES for m in _sweep_models())
+
+
+# Back-compat: modules and tests that read the constant directly get the default cross.
+SWEEP_AGENTS = tuple((a, m) for a in SWEEP_AGENT_NAMES for m in SWEEP_MODELS)
+SWEEP_DOMAINS = ("swebench", "webbench")
+SWEEP_SEEDS = (0, 1, 2, 3, 4)
+
+# Measured on the stored claude_swebench tree: 984 task-runs, mean $0.34/task on the honest
+# basis (effective + overhead at the $5/1M-output tier), and ~112-133 s/task from the two
+# most recent 40-task lanes' start-to-finish wall clock.
+DEFAULT_SEC_PER_TASK = 120.0
+DEFAULT_USD_PER_TASK = 0.34
+
+
+def group_sizes(domain):
+ """{group -> n_tasks} a NEW run will actually execute, per group.
+
+ NOT `workloads.task_counts()`, which is the provenance record and deliberately does
+ not subtract `SWEBENCH_EXCLUDED` -- tests/test_workloads.py pins those sizes, and
+ the published trees really did grade every id in them.
+
+ The number here has to match what the driver runs, because `main_sweep.sh` compares
+ the row.json files a cell already holds against it to decide whether the cell is
+ finished. Off by one and SKIP_DONE never fires for that group: every resume re-runs
+ completed cells, paid for, silently. So this reads the same committed manifest the
+ launcher passes as `--workload` and the driver filters to (`load_tasks` keeps exactly
+ the ids the manifest names), and falls back to `runnable_instances` when a manifest is
+ not on disk. tests/test_sweep_arms.py asserts the two agree.
+ """
+ manifest = os.path.join(_EXPERIMENTS, "workloads", f"{domain}.json")
+ if os.path.exists(manifest):
+ with open(manifest, encoding="utf-8") as fh:
+ groups = json.load(fh).get("groups") or {}
+ if groups:
+ return {g: len(ids) for g, ids in groups.items()}
+ return {g: len(ids) for g, ids in workloads.runnable_instances(domain).items()}
+
+
+def cells(domain, arms, repos, seeds):
+ """The (arm, repo, seed) triples of one lane, in run order: seed-major, then repo.
+
+ Seed-major matches how the stored runs were launched -- a seed is one complete pass
+ over the repo set, so an interrupted sweep leaves whole seeds finished rather than
+ every seed half-done, and the first seed alone is already a publishable slice.
+ """
+ out = []
+ for seed in seeds:
+ for repo in repos:
+ for arm in arms:
+ out.append((arm, repo, seed))
+ return out
+
+
+def estimate(domain, arms, repos, seeds, sec_per_task, usd_per_task, limit=0):
+ """Task-runs, wall-clock and dollars for the requested slice.
+
+ `limit` mirrors the launcher's LIMIT (cap tasks per cell, 0 = all). It has to be
+ honoured here or the estimate describes a run nobody asked for: a LIMIT=1 pilot used to
+ be costed at the full stream -- "task-runs= 40 ... ~$14" for a cell that runs ONE task
+ -- which is the wrong direction for a number whose whole job is to be checked before
+ spending money.
+ """
+ sizes = group_sizes(domain)
+ unknown = [r for r in repos if r not in sizes]
+ per_repo = {r: sizes.get(r, 0) for r in repos}
+ if limit and limit > 0:
+ per_repo = {r: min(n, limit) for r, n in per_repo.items()}
+ tasks = sum(per_repo.values()) * len(arms) * len(seeds)
+ return {
+ "domain": domain, "n_arms": len(arms), "n_repos": len(repos), "n_seeds": len(seeds),
+ "n_cells": len(arms) * len(repos) * len(seeds), "n_task_runs": tasks,
+ "hours_serial": tasks * sec_per_task / 3600.0,
+ "usd": tasks * usd_per_task, "unknown_repos": unknown,
+ "limit": limit or 0,
+ }
+
+
+def _fmt_estimate(e, show_cost=False):
+ """Counts by default; the derived cost/wall-clock only when asked for.
+
+ `cells` and `task-runs` are exact arithmetic. `hours_serial` and `usd` are one measured
+ per-task mean multiplied by that count, and the per-task distribution is heavy-tailed
+ and dominated by the repo's eval script rather than by the arm -- so the aggregate is a
+ number with no error bar that reads like a quote. It stays available behind
+ `--show-cost` for a per-arm sanity check; it is not printed by default -- budget from
+ per-arm RUNTIME instead of any sweep-wide total.
+ """
+ cap = f" limit={e['limit']}/cell" if e.get("limit") else ""
+ line = (f"{e['domain']:9s} arms={e['n_arms']:2d} repos={e['n_repos']:2d} "
+ f"seeds={e['n_seeds']:2d} cells={e['n_cells']:5d} "
+ f"task-runs={e['n_task_runs']:7,d}")
+ if show_cost:
+ line += f" serial={e['hours_serial']:8,.0f}h ~${e['usd']:9,.0f}"
+ return line + cap
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--domain", default="swebench", choices=sorted(SWEEP_ARMS))
+ ap.add_argument("--arms", nargs="*", default=None,
+ help="default: the domain's SWEEP_ARMS")
+ ap.add_argument("--repos", nargs="*", default=None,
+ help="default: every group in the domain's workload record")
+ ap.add_argument("--seeds", nargs="*", type=int, default=None,
+ help="default: %s" % " ".join(map(str, SWEEP_SEEDS)))
+ ap.add_argument("--limit", type=int, default=0,
+ help="cap tasks per cell (0 = all), mirroring the launcher's LIMIT so "
+ "a pilot is costed as a pilot")
+ ap.add_argument("--show-cost", action="store_true",
+ help="also print the DERIVED wall-clock and $ estimate (off by "
+ "default: it is one repo-dominated per-task mean times the "
+ "task count, and reads like a quote it cannot support)")
+ ap.add_argument("--sec-per-task", type=float, default=DEFAULT_SEC_PER_TASK)
+ ap.add_argument("--usd-per-task", type=float, default=DEFAULT_USD_PER_TASK)
+ ap.add_argument("--format", default="estimate",
+ choices=["cells", "estimate", "matrix", "arms", "counts"])
+ args = ap.parse_args()
+
+ arms = args.arms or list(SWEEP_ARMS[args.domain])
+ repos = args.repos or sorted(group_sizes(args.domain))
+ seeds = args.seeds if args.seeds is not None else list(SWEEP_SEEDS)
+
+ bad = [a for a in arms if a not in ARMS]
+ if bad:
+ raise SystemExit(f"unknown arm(s): {' '.join(bad)}")
+
+ if args.format == "arms":
+ for a in arms:
+ print(a)
+ return
+ if args.format == "counts":
+ for group, n in sorted(group_sizes(args.domain).items()):
+ print(f"{group}\t{n}")
+ return
+ if args.format == "cells":
+ unavailable = [a for a in arms
+ if args.domain == "webbench" and a in SWEEP_ARMS_WEBBENCH_UNAVAILABLE]
+ if unavailable:
+ print(f"# SKIPPING scaffold arms not wired for webbench: "
+ f"{' '.join(unavailable)}", file=sys.stderr)
+ for arm, repo, seed in cells(args.domain,
+ [a for a in arms if a not in unavailable],
+ repos, seeds):
+ print(f"{arm}\t{repo}\t{seed}")
+ return
+
+ if args.format == "estimate":
+ usable = [a for a in arms
+ if not (args.domain == "webbench"
+ and a in SWEEP_ARMS_WEBBENCH_UNAVAILABLE)]
+ e = estimate(args.domain, usable, repos, seeds,
+ args.sec_per_task, args.usd_per_task, limit=args.limit)
+ if e["unknown_repos"]:
+ print(f"!! not in the {args.domain} workload record: "
+ f"{' '.join(e['unknown_repos'])} (counted as 0 tasks)", file=sys.stderr)
+ print(_fmt_estimate(e, args.show_cost))
+ if args.show_cost:
+ print(f" at {args.sec_per_task:.0f}s and ${args.usd_per_task:.2f} per task-run "
+ f"(measured means, heavy-tailed -- an order of magnitude, not a quote)")
+ return
+
+ # --format matrix: the whole sweep, every agent x model x domain
+ pairs = sweep_agents()
+ print(f"main sweep: {len(pairs)} agent-model pairs x {len(SWEEP_DOMAINS)} "
+ f"domains x {len(SWEEP_SEEDS)} seeds")
+ print(f" agent-model : {', '.join(f'{a}/{m}' for a, m in pairs)}")
+ if len(_sweep_models()) < len(SWEEP_MODELS_SUPPORTED):
+ rest = [m for m in SWEEP_MODELS_SUPPORTED if m not in _sweep_models()]
+ print(f" not planned here: {' '.join(rest)}")
+ # NOT the same as optional. claude-opus-5 runs 4 REQUIRED arms per domain, launched
+ # by their own main_sweep.sh calls; this planner only crosses full 21-arm lanes, so
+ # a model absent from SWEEP_MODELS is invisible to it either way. Saying "also
+ # supported" here read as "skip if you like", which is wrong for that model.
+ if any("opus-5" in m.replace(".", "-") for m in rest):
+ print(" claude-opus-5: 4 arms x both domains are REQUIRED and are launched "
+ "separately\n"
+ " (baseline baseline-medium baseline-low trimech"
+ " / trimech-web -- docs/RUNNING.md, the frontier lane).")
+ print(f" to plan one of these as a FULL {len(SWEEP_ARMS['swebench'])}-arm lane"
+ f" instead: TE_SWEEP_MODELS=\"{' '.join(list(_sweep_models()) + rest)}\"")
+ total_tasks = total_usd = total_hours = total_cells = 0
+ for domain in SWEEP_DOMAINS:
+ arms_d = list(SWEEP_ARMS[domain])
+ repos_d = sorted(group_sizes(domain))
+ e = estimate(domain, arms_d, repos_d, list(SWEEP_SEEDS),
+ args.sec_per_task, args.usd_per_task)
+ per_pair = _fmt_estimate(e, args.show_cost)
+ print(f" per agent-model pair: {per_pair}")
+ print(f" arms : {' '.join(arms_d)}")
+ print(f" groups: {' '.join(repos_d)}")
+ total_tasks += e["n_task_runs"] * len(pairs)
+ total_usd += e["usd"] * len(pairs)
+ total_hours += e["hours_serial"] * len(pairs)
+ total_cells += e["n_cells"] * len(pairs)
+ total = f" TOTAL: {total_cells:,} cells {total_tasks:,} task-runs"
+ if args.show_cost:
+ total += (f" ~${total_usd:,.0f} {total_hours:,.0f} h serial "
+ f"({total_hours / 24:,.0f} days; "
+ f"/{len(pairs) * len(SWEEP_DOMAINS)} lanes = "
+ f"{total_hours / 24 / (len(pairs) * len(SWEEP_DOMAINS)):,.0f} days)")
+ print(total)
+ if not args.show_cost:
+ print(" (exact counts. --show-cost adds a derived $ / wall-clock total: one "
+ "repo-dominated per-task mean x the count, so do not quote it -- budget from "
+ "the PER-ARM runtime instead: time one arm on your host and multiply)")
+ hosted = sorted(a for a in SWEEP_ARMS["webbench"] if a in SCAFFOLD_ARM_NAMES)
+ print(f" Web-Bench runs the scaffold arms it can host ({' '.join(hosted)}) and NOT "
+ f"({' '.join(SWEEP_ARMS_WEBBENCH_UNAVAILABLE)}): the repo map generates a PYTHON "
+ f"symbol map a Node/TypeScript project does not have, and Serena is a codebase "
+ f"navigation server for a benchmark that ships a file set, not a codebase "
+ f"-- see SWEEP_ARMS_WEBBENCH_UNAVAILABLE")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/scripts/webbench_tasks_from_yaml.py b/experiments/scripts/webbench_tasks_from_yaml.py
new file mode 100644
index 0000000..2302746
--- /dev/null
+++ b/experiments/scripts/webbench_tasks_from_yaml.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""Derive `projects/<name>/tasks.jsonl` from a Web-Bench project's `tasks.yml`.
+
+Some Web-Bench projects ship only `tasks.yml`. Two readers on our side take `tasks.jsonl`
+and nothing else -- `te_harness.webbench.WebBenchStream._tasks` and
+`tokeneconomizer.adapters.webbench.WebBenchAdapter.tasks_path` -- so those projects need a
+derived jsonl before they can be run or judged.
+
+The yml is left in place ON PURPOSE. Web-Bench's own evaluator prefers it when both exist
+(`tools/evaluator/src/settings/bench.ts`), so deleting it would move the evaluator onto our
+derived file, and editing one without the other would silently desync the graded task order
+from the driver's. The yml stays authoritative; this writes its faithful mirror.
+
+ python scripts/webbench_tasks_from_yaml.py zustand # write the jsonl
+ python scripts/webbench_tasks_from_yaml.py --all # every swept yml project
+ python scripts/webbench_tasks_from_yaml.py --check --all # verify, exit 1 if stale
+
+`--check` is what `tests/test_workloads.py` calls: drift between the two files fails the
+suite instead of surfacing as a mislabelled cell after a paid lane.
+"""
+import argparse
+import json
+import os
+import sys
+
+_EXPERIMENTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path[:0] = [_EXPERIMENTS, os.path.dirname(_EXPERIMENTS)] # te_harness; the core at the root
+
+try:
+ import yaml
+except ImportError: # pragma: no cover
+ sys.exit("PyYAML is required: pip install pyyaml")
+
+PROJECTS_ROOT_ENV = "TOKENECONOMIZER_WEBBENCH_PROJECTS"
+# The keys the readers use. `id` and `description` are load-bearing (the driver keys tasks by
+# id; the untool judge reads the description); `date` and `level` are carried so the derived
+# file is a faithful mirror rather than a lossy projection.
+FIELDS = ("id", "date", "level", "description")
+
+
+def projects_root(explicit=None):
+ root = explicit or os.environ.get(PROJECTS_ROOT_ENV)
+ if not root:
+ sys.exit(f"no Web-Bench projects root: set ${PROJECTS_ROOT_ENV} or pass --projects-root")
+ return root
+
+
+def render(yml_path):
+ """The jsonl text this project's tasks.yml maps to. One compact object per line."""
+ with open(yml_path, encoding="utf-8") as fh:
+ tasks = yaml.safe_load(fh) or []
+ out = []
+ for task in tasks:
+ # `date` round-trips through PyYAML as a datetime.date, which json cannot encode; the
+ # jsonl form the shipped files use is the plain ISO string.
+ # `description` is a YAML block scalar (`|`), so PyYAML hands back a trailing
+ # newline that the shipped jsonl files do not carry. Verified against `prisma`, the
+ # one project shipping BOTH files: its 20 records match the yml exactly once the
+ # descriptions are stripped. (Only the CONTENT is a convention -- the shipped files
+ # disagree on separator style, `flex` compact and `prisma` spaced, so that is ours.)
+ rec = {}
+ for k in FIELDS:
+ v = task.get(k)
+ if v is None:
+ rec[k] = ""
+ elif k == "date":
+ rec[k] = str(v)
+ elif k == "description":
+ rec[k] = v.strip()
+ else:
+ rec[k] = v
+ # Compact separators to match the shipped jsonl files byte-for-byte in shape.
+ out.append(json.dumps(rec, ensure_ascii=False, separators=(",", ":")))
+ return "".join(line + "\n" for line in out)
+
+
+def swept_yml_projects(root):
+ """Projects in the SWEEP that ship a tasks.yml -- the only ones this has to keep in step.
+
+ Deliberately not "every project with a tasks.yml": the checkout has ~18 of those, most of
+ them outside the sweep, and one (`prisma`) ships both files. Writing to those would mutate
+ a checkout the sweep never reads."""
+ from tokeneconomizer import workloads
+ for name in workloads.WEBBENCH_TASKS:
+ if os.path.isfile(os.path.join(root, name, "tasks.yml")):
+ yield name
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("project", nargs="*", help="project name(s); omit with --all")
+ ap.add_argument("--all", action="store_true",
+ help="every SWEPT project that ships a tasks.yml")
+ ap.add_argument("--check", action="store_true",
+ help="verify the jsonl matches the yml; write nothing, exit 1 if stale")
+ ap.add_argument("--projects-root", default=None)
+ args = ap.parse_args()
+
+ root = projects_root(args.projects_root)
+ names = list(swept_yml_projects(root)) if args.all else args.project
+ if not names:
+ ap.error("name at least one project, or pass --all")
+
+ stale = []
+ for name in names:
+ yml = os.path.join(root, name, "tasks.yml")
+ if not os.path.isfile(yml):
+ sys.exit(f"{name}: no tasks.yml at {yml}")
+ jsonl = os.path.join(root, name, "tasks.jsonl")
+ want = render(yml)
+ have = open(jsonl, encoding="utf-8").read() if os.path.isfile(jsonl) else None
+ if have == want:
+ print(f" ok {name} ({want.count(chr(10))} tasks)")
+ continue
+ if args.check:
+ stale.append(name)
+ print(f" STALE {name} ({'missing' if have is None else 'differs'})")
+ continue
+ with open(jsonl, "w", encoding="utf-8") as fh:
+ fh.write(want)
+ print(f" wrote {name} ({want.count(chr(10))} tasks) -> {jsonl}")
+
+ if stale:
+ sys.exit(f"\n{len(stale)} project(s) stale: {' '.join(stale)}\n"
+ f"regenerate with: python scripts/webbench_tasks_from_yaml.py --all")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/experiments/te_harness/__init__.py b/experiments/te_harness/__init__.py
new file mode 100644
index 0000000..14a4f54
--- /dev/null
+++ b/experiments/te_harness/__init__.py
@@ -0,0 +1,47 @@
+"""te_harness — the Layer-2 task-execution harness for TokenEconomizer.
+
+This is the reference implementation of `tokeneconomizer.runner.AgentRunner`: the Docker- /
+CLI- / credential-heavy machinery that sits BELOW that seam, and the one that produced every
+published row. It is the harness the results are reproducible with, not a requirement of the
+method -- the economizer above the seam never learns which harness it is talking to.
+
+It is a SIBLING package of `tokeneconomizer`, not a submodule of it, and is excluded from
+the distributed wheel (see pyproject.toml). That boundary is what keeps the core's
+stdlib-only guarantee structural rather than a convention: `import tokeneconomizer` cannot
+reach Docker or an agent CLI because the code that shells out to them is not under it.
+(This package happens to be stdlib-only at import time too -- the official SWE-bench grader
+is imported lazily in `swebench._import_swebench` -- so nothing here adds an install-time
+dependency either.)
+
+Use it from the repo's driver, which puts experiments/ and the repository root (where the
+shared core lives) on `sys.path` itself, so no PYTHONPATH is needed from any working
+directory:
+
+ python scripts/run_experiment.py --runner te_harness.swebench:make_runner ...
+
+Both `make_runner` factories take the driver's `--agent` label and return the runner for
+that agent -- `claude` (Claude Code, on the Anthropic API), `copilot` (GitHub Copilot CLI,
+on its own GitHub or BYOK auth) or `codex` (OpenAI Codex CLI, on the OpenAI API). An
+unknown label raises rather than falling back, because a runner that quietly ran the wrong
+agent would produce plausible rows under the wrong name. Copilot setup: docs/COPILOT.md.
+"""
+import os as _os
+import sys as _sys
+
+# The shared core, `tokeneconomizer`, lives at the repository root, one level above
+# experiments/. Appended rather than prepended, so an installed core (`pip install -e .`)
+# still wins; this only covers `python -m te_harness....` run from experiments/ on a clone.
+_ROOT = _os.path.dirname(_os.path.dirname(_os.path.dirname(_os.path.abspath(__file__))))
+if _ROOT not in _sys.path:
+ _sys.path.append(_ROOT)
+
+from .swebench import (make_runner, ClaudeSwebenchRunner, # noqa: E402,F401
+ CopilotSwebenchRunner, RUNNERS as SWEBENCH_RUNNERS)
+from .webbench import (make_runner as make_webbench_runner, # noqa: E402,F401
+ WebBenchRunner, CopilotWebBenchRunner,
+ RUNNERS as WEBBENCH_RUNNERS)
+from . import copilot # noqa: E402,F401
+
+__all__ = ["make_runner", "ClaudeSwebenchRunner", "CopilotSwebenchRunner",
+ "make_webbench_runner", "WebBenchRunner", "CopilotWebBenchRunner",
+ "SWEBENCH_RUNNERS", "WEBBENCH_RUNNERS", "copilot"]
diff --git a/experiments/te_harness/capture_catalog.py b/experiments/te_harness/capture_catalog.py
new file mode 100644
index 0000000..520b594
--- /dev/null
+++ b/experiments/te_harness/capture_catalog.py
@@ -0,0 +1,274 @@
+"""Mine `claude_tool_catalog.json` from a REAL Claude Code run.
+
+Why this exists: the economizer's tool judge needs `{tool_name: description}` for the tools
+the agent loaded. The Copilot CLI dumps raw tool schemas into its debug log, so
+`reflect.parse_tool_catalog` just reads them. Claude Code has no such log -- its
+stream-json transcript carries the loaded tool NAMES but not their descriptions -- so
+`reflect.parse_tool_catalog_claude` joins those names against a snapshot captured off the
+wire. This module makes that snapshot.
+
+Without it the judge is handed 25 lines of `- ToolName: ` with empty descriptions and drops
+nothing, while still billing for the call.
+
+The capture runs the agent EXACTLY as a task run does -- same image, same container, same
+`--dangerously-skip-permissions --effort ...` invocation -- so the captured tool set is the
+one those runs actually load, not the larger set a host session with MCP servers and skills
+would send. The only addition is a tiny stdlib recorder running INSIDE the container:
+Claude Code is pointed at `127.0.0.1`, the recorder copies the `tools` array off the
+outbound request and passes the call on to api.anthropic.com with the real key, so the run
+completes normally, bills the Anthropic API like any task run, and what is captured is what
+the agent was sent.
+
+(In-container rather than host-side because the container->host route is firewalled on
+this machine; keeping the recorder inside also means the capture needs no host networking
+at all.)
+
+ python -m te_harness.capture_catalog
+
+Env: ANTHROPIC_API_KEY; TE_CATALOG_OUT (default: the repo's vendor/claude_tool_catalog.json),
+TE_CATALOG_PORT (default 8770), TE_CATALOG_DESC_CHARS (default 300 -- comfortably above
+reflect's max_desc=200 default, keeps the committed file small).
+"""
+import json
+import os
+import subprocess
+import sys
+import time
+
+from tokeneconomizer import vendor as _vendor
+from .swebench import (_ANTHROPIC_DIRECT, _LIVE_CONTAINERS, _log, _run, claude_bin,
+ image_for)
+
+# The vendored catalog the judge reads, inside the shared core:
+# <repo>/tokeneconomizer/vendor/claude_tool_catalog.json.
+DEFAULT_OUT = os.path.join(os.path.dirname(os.path.abspath(_vendor.__file__)),
+ "claude_tool_catalog.json")
+CAPTURE_INSTANCE = os.environ.get("TE_CATALOG_INSTANCE", "pydata__xarray-6461")
+
+# Runs on the container's own python (3.10, stdlib only). UNION-merges the `tools` array
+# across every request: different requests carry different subsets, and a deferred tool's
+# schema only reaches the wire once ToolSearch has fetched it.
+RECORDER_SRC = r'''
+import json, os, sys, threading, urllib.request, urllib.error
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+PORT, OUT, UPSTREAM = int(sys.argv[1]), sys.argv[2], sys.argv[3].rstrip("/")
+KEY = os.environ.get("REAL_ANTHROPIC_KEY", "")
+best = {"tools": {}, "n": 0}
+lock = threading.Lock()
+
+def offer(tools):
+ got = {t["name"]: (t.get("description") or "").strip()
+ for t in (tools or []) if isinstance(t, dict) and t.get("name")}
+ with lock:
+ best["n"] += 1
+ changed = False
+ for n, d in got.items():
+ if len(d) > len(best["tools"].get(n, "")):
+ best["tools"][n] = d
+ changed = True
+ if changed:
+ json.dump(best["tools"], open(OUT, "w"))
+
+class H(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+ def log_message(self, *a): pass
+ def do_POST(self):
+ body = self.rfile.read(int(self.headers.get("content-length") or 0))
+ try: offer(json.loads(body).get("tools"))
+ except Exception: pass
+ self.forward(body)
+ def do_GET(self): self.forward(None)
+ def forward(self, body):
+ req = urllib.request.Request(UPSTREAM + self.path, data=body, method=self.command)
+ for k, v in self.headers.items():
+ if k.lower() in ("host","content-length","authorization","x-api-key",
+ "accept-encoding","connection"): continue
+ req.add_header(k, v)
+ req.add_header("x-api-key", KEY)
+ req.add_header("anthropic-version",
+ self.headers.get("anthropic-version", "2023-06-01"))
+ try:
+ with urllib.request.urlopen(req, timeout=300) as r:
+ self.send_response(r.status)
+ for k, v in r.headers.items():
+ if k.lower() in ("transfer-encoding","content-length","connection",
+ "content-encoding"): continue
+ self.send_header(k, v)
+ self.send_header("transfer-encoding", "chunked")
+ self.end_headers()
+ while True:
+ c = r.read(4096)
+ if not c: break
+ self.wfile.write(b"%X\r\n%s\r\n" % (len(c), c)); self.wfile.flush()
+ self.wfile.write(b"0\r\n\r\n")
+ except urllib.error.HTTPError as e:
+ p = e.read(); self.send_response(e.code)
+ self.send_header("content-type","application/json")
+ self.send_header("content-length", str(len(p))); self.end_headers()
+ self.wfile.write(p)
+ except Exception as e:
+ p = json.dumps({"error": str(e)}).encode(); self.send_response(502)
+ self.send_header("content-type","application/json")
+ self.send_header("content-length", str(len(p))); self.end_headers()
+ self.wfile.write(p)
+
+ThreadingHTTPServer(("127.0.0.1", PORT), H).serve_forever()
+'''
+
+
+def system_tools(stream_json_text):
+ """Tool NAMES from a stream-json run's `system` init event."""
+ for line in (stream_json_text or "").splitlines():
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ if ev.get("type") == "system" and isinstance(ev.get("tools"), list):
+ return ev["tools"]
+ return []
+
+
+def capture(instance_id=CAPTURE_INSTANCE, port=None, out_path=None, desc_chars=None,
+ model=None, effort=None):
+ """Run Claude Code once in a task container behind an in-container capture recorder.
+ Returns the captured `{name: description}` and writes it to `out_path`."""
+ port = int(port or os.environ.get("TE_CATALOG_PORT", "8770"))
+ out_path = out_path or os.environ.get("TE_CATALOG_OUT", DEFAULT_OUT)
+ desc_chars = int(desc_chars or os.environ.get("TE_CATALOG_DESC_CHARS", "300"))
+ model = model or os.environ.get("ANTHROPIC_MODEL", "claude-opus-5")
+ effort = effort or os.environ.get("TE_EFFORT_LEVEL", "high")
+ # Must match what a task run uses (te_harness.swebench.run_task) -- see pass 1 below.
+ turn_cap = int(os.environ.get("TE_MAX_TURNS", "80"))
+ key = os.environ.get("ANTHROPIC_API_KEY", "")
+ if not key:
+ raise SystemExit("ANTHROPIC_API_KEY is unset (see the repo's .env)")
+
+ name = "te-catalog-capture"
+ _run(["docker", "rm", "-f", name])
+ _LIVE_CONTAINERS.add(name)
+ try:
+ p = _run(["docker", "run", "-d", "--name", name, "--entrypoint", "bash",
+ "-v", f"{claude_bin()}:/usr/local/bin/claude:ro",
+ "-e", "HOME=/tmp/agent-home", "-w", "/testbed",
+ image_for(instance_id), "-lc", "sleep infinity"], timeout=300)
+ if p.returncode != 0:
+ raise SystemExit(f"docker run failed: {p.stderr[-400:]}")
+ _run(["docker", "exec", name, "mkdir", "-p", "/tmp/agent-home"])
+
+ # install + start the recorder (container python; no host networking involved)
+ _run(["docker", "exec", name, "bash", "-lc",
+ "cat > /tmp/capture_recorder.py <<'TE_EOF'\n" + RECORDER_SRC + "\nTE_EOF"])
+ subprocess.Popen(
+ ["docker", "exec", "-e", f"REAL_ANTHROPIC_KEY={key}", name, "bash", "-lc",
+ f"python /tmp/capture_recorder.py {port} /tmp/tools.json {_ANTHROPIC_DIRECT} "
+ f">/tmp/recorder.log 2>&1"],
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ for _ in range(30): # wait for the listener
+ if _run(["docker", "exec", name, "bash", "-lc",
+ f"curl -s -o /dev/null -m 2 http://127.0.0.1:{port}/ ; echo $?"]
+ ).stdout.strip() != "7":
+ break
+ time.sleep(1)
+
+ def agent(prompt, max_turns, fmt="json"):
+ """The agent, invoked exactly as a task run invokes it -- pointed at the
+ in-container recorder, which swaps in the real key on the way out."""
+ return subprocess.run(
+ ["docker", "exec", "-i", "-w", "/testbed",
+ "-e", f"ANTHROPIC_BASE_URL=http://127.0.0.1:{port}",
+ "-e", "ANTHROPIC_API_KEY=dummy",
+ "-e", "HOME=/tmp/agent-home", "-e", "IS_SANDBOX=1",
+ name, "claude", "-p", "--dangerously-skip-permissions",
+ "--model", model, "--effort", effort, "--max-turns", str(max_turns),
+ "--output-format", fmt] + (["--verbose"] if fmt == "stream-json" else []),
+ input=prompt, capture_output=True, text=True, timeout=600)
+
+ # What a REAL (first-party) task run loads -- the set the snapshot must cover.
+ fp_probe = subprocess.run(
+ ["docker", "exec", "-i", "-w", "/testbed",
+ # name only -- docker takes the value from the client env, keeping the key
+ # out of argv (and therefore out of every other user's `ps`)
+ "-e", "ANTHROPIC_API_KEY", "-e", "HOME=/tmp/agent-home",
+ "-e", "IS_SANDBOX=1",
+ name, "claude", "-p", "--dangerously-skip-permissions",
+ "--model", model, "--effort", effort, "--max-turns", str(turn_cap),
+ "--output-format", "stream-json", "--verbose"],
+ input="Reply with exactly: probe", capture_output=True, text=True, timeout=600)
+ loaded_firstparty = system_tools(fp_probe.stdout)
+ _log(f"first-party run loads {len(loaded_firstparty)} tools")
+
+ def captured():
+ raw = _run(["docker", "exec", name, "bash", "-lc",
+ "cat /tmp/tools.json 2>/dev/null || true"]).stdout.strip()
+ return json.loads(raw) if raw else {}
+
+ # Pass 1: a trivial prompt, but with the REAL turn cap. The CLI's tool set depends
+ # on it -- at --max-turns 1 it offers agent-orchestration tools (Agent, Monitor,
+ # ListAgents, ...) and withholds the Task*/ToolSearch family, which is the opposite
+ # of what an 80-turn task run loads. Capturing under a different cap would snapshot
+ # a catalog the real runs never see. The trivial prompt still ends after one turn,
+ # so the cap costs nothing.
+ proc = agent("Reply with exactly: catalog-capture", turn_cap, fmt="stream-json")
+ loaded = system_tools(proc.stdout)
+
+ # Pass 2: deferred tools are LISTED by name but their schemas only reach the wire
+ # once ToolSearch fetches them -- so ask for exactly the ones still missing. Without
+ # this they land in the snapshot with an empty description, which is the very hole
+ # this module exists to close.
+ missing = sorted(set(loaded) - set(captured()))
+ if missing:
+ _log(f"{len(missing)} deferred tool(s) not yet on the wire: {missing}")
+ agent(f"Call ToolSearch once with query \"select:{','.join(missing)}\" "
+ f"and then reply DONE. Do nothing else.", turn_cap)
+ still = sorted(set(loaded) - set(captured()))
+ _log(f"after ToolSearch fetch, still missing: {still or 'none'}")
+
+ # Residual: tools the recorder can never see. Pointing the CLI at ANY base URL --
+ # including this local recorder -- changes which tools it offers: first-party features
+ # (the Task* family, ToolSearch) drop out. Their schemas reach only the model's
+ # context, never the transcript (a ToolSearch tool_result is empty by design), so
+ # short of intercepting first-party TLS there is nowhere to read them from. They
+ # stay in the catalog as names with no description, which the judge tolerates.
+ merged = captured()
+ gap = sorted(set(loaded_firstparty or []) - set(merged))
+ if gap:
+ _log(f"NOT captured (first-party-only, no readable source): {gap}")
+ got = json.dumps(merged)
+ if got in ("", "{}"):
+ _log(f"agent exit={proc.returncode} stdout={proc.stdout[-300:]!r}")
+ _log("recorder log: " + _run(["docker", "exec", name, "bash", "-lc",
+ "tail -5 /tmp/recorder.log"]).stdout[-300:])
+ raise SystemExit("captured no tool schemas")
+ tools = json.loads(got)
+ finally:
+ _run(["docker", "rm", "-f", name])
+ _LIVE_CONTAINERS.discard(name)
+
+ prior = {}
+ if os.path.exists(out_path):
+ try:
+ with open(out_path, encoding="utf-8") as fh:
+ prior = json.load(fh)
+ except Exception:
+ prior = {}
+ merged_all = dict(prior)
+ for n, d in tools.items():
+ if len(d) > len(merged_all.get(n, "")):
+ merged_all[n] = d
+ added = sorted(set(merged_all) - set(prior))
+ if prior:
+ _log(f"merged with existing snapshot ({len(prior)} tools); new: {added or 'none'}")
+ catalog = {n: d[:desc_chars] for n, d in sorted(merged_all.items())}
+ os.makedirs(os.path.dirname(out_path), exist_ok=True)
+ with open(out_path, "w", encoding="utf-8") as fh:
+ json.dump(catalog, fh, indent=2, sort_keys=True)
+ _log(f"wrote {len(catalog)} tool descriptions -> {out_path}")
+ return catalog
+
+
+if __name__ == "__main__":
+ cat = capture()
+ for n, d in cat.items():
+ print(f" {n:22} {d[:66]}")
+ print(f"\n{len(cat)} tools captured", file=sys.stderr)
diff --git a/experiments/te_harness/codex.py b/experiments/te_harness/codex.py
new file mode 100644
index 0000000..aaf3073
--- /dev/null
+++ b/experiments/te_harness/codex.py
@@ -0,0 +1,1114 @@
+"""Shared OpenAI Codex CLI support for the te_harness runners.
+
+Why a separate module, exactly as `te_harness.copilot` is one: nothing here is shared
+with Claude Code, and every piece of it is a place where a wrong guess produces a tree of
+plausible rows instead of an error.
+
+ 1. **Host install resolution.** The `codex` on PATH is a small Node shim that execs a
+ platform-specific native binary. That binary is a self-contained static-PIE ELF (no
+ Node, no dynamic libs), so the task container gets the vendor triple dir bind-mounted
+ read-only and invokes the ELF directly -- nothing is installed in the image, the same
+ posture as the bind-mounted `claude` binary.
+ 2. **Provider config is a FILE, not environment.** Codex reads its provider and base URL
+ from `$CODEX_HOME/config.toml`, so the endpoint cannot be expressed as env vars.
+ `config_toml` builds that file, naming the OpenAI API the lane bills directly.
+ 3. **A different usage schema.** `codex exec --json` reports usage on `turn.completed`
+ with `input_tokens` as the GRAND input total -- it already INCLUDES
+ `cached_input_tokens` and `cache_write_input_tokens`. Subtracting both is what makes
+ fresh input actually fresh; skipping that step prices ~90% cached re-reads at 1.0x and
+ every cost number downstream is wrong rather than missing. See `row_usage`.
+ 4. **A capability mask, not a tool mask.** Codex has no `--disallowed-tools`. The
+ Codex-native untooling mechanism (`tokeneconomizer.mechanisms.codex_untool`) emits
+ logical CAPABILITY packages and leaves the mapping onto concrete Codex config toggles
+ to the runner, because the runner is what owns the request contract. `config_toggles`
+ is that mapping.
+
+THE TOGGLE NAMES ARE VERSION-SPECIFIC, and this is the failure this module works hardest to
+prevent. Verified against the installed codex-cli 0.153.1 with `codex exec --strict-config`
+(which errors on an unrecognized configuration field), two toggles carried over from the
+AgentEO study's codex-cli 0.146 driver are NO LONGER ACCEPTED:
+
+ web_search_mode = "disabled" -> unknown configuration field (now: web_search = "disabled")
+ [tools] view_image = false -> unknown configuration field (now: [features] view_image = false)
+
+A renamed key does not fail: Codex ignores unknown fields unless `--strict-config` is
+passed, so the pruning silently becomes a no-op and the economized arm runs as a baseline
+wearing its name. That is why `validate_config` runs the real CLI over the generated file at
+CONSTRUCTION time and refuses the lane rather than discovering it per task.
+
+One limit worth knowing about that gate: `[[hooks.pre_tool_use]]` is NOT strictly
+validated (a bogus field inside it is accepted), so `--strict-config` proves the pruning
+toggles are real but proves nothing about the net guard. The guard is verified at RUNTIME
+instead, the same way the Claude and Copilot lanes verify theirs.
+
+Stdlib only, like the rest of the harness.
+"""
+import json
+import os
+import re
+import shutil
+import signal
+import subprocess
+import tempfile
+import time
+import urllib.error
+import urllib.request
+
+from tokeneconomizer.vendor import pricing
+
+# What `model_reasoning_effort` accepts. Validated by the runner at construction so a typo
+# fails before the run rather than silently falling back to the CLI default -- effort is
+# part of the cost basis, so a silent default makes the run unreproducible.
+EFFORT_LEVELS = frozenset({"minimal", "low", "medium", "high", "xhigh", "max"})
+
+# Codex's own work surface. Without a shell and an edit path there is no task, so no
+# capability mask may ever remove these: a `resolved: false` caused by a missing
+# `apply_patch` is a harness artifact, not a mechanism result. These are CAPABILITY names
+# rather than tool names because that is the granularity Codex config operates at.
+PROTECTED_CAPABILITIES = frozenset({
+ "deferred_wait", # CORE in codex_untool: pruning it provoked shell-polling
+ "apply_patch", "exec_command", "planning", "write_stdin",
+})
+
+# Tool names the shared lifecycle treats as unmaskable. Codex is masked by capability, not
+# by tool, so this exists only to satisfy the base runner's `protected_tools` slot and to
+# document what the mask must never reach.
+PROTECTED_TOOLS = frozenset({"shell", "apply_patch"})
+
+# In-container paths. The vendor dir is mounted read-only; CODEX_HOME must be writable
+# because Codex writes session rollouts there (and the rollout is where the resolved model
+# id is recorded, which is what prices an unpinned run).
+C_VENDOR = "/opt/codex-vendor"
+C_BIN = C_VENDOR + "/bin/codex"
+# NOT under /tmp, and that is load-bearing rather than tidiness. With CODEX_HOME on a
+# temporary dir the CLI refuses to install its PATH helper binaries and says so on every
+# invocation ("Refusing to create helper binaries under temporary dir"), observed on 0.153.1
+# -- so the agent runs without the helpers Codex expects to put on its PATH. The container's
+# HOME stays /tmp/agent-home (the scaffold profile's home, shared with the Claude lane);
+# only CODEX_HOME moves.
+C_HOME = "/opt/codex-home"
+
+# The OpenAI Platform's global Responses endpoint. Only a DEFAULT: a region-scoped key
+# 401s here and must set OPENAI_BASE_URL to its own region (docs/SETUP.md step 3).
+OPENAI_DIRECT = "https://api.openai.com/v1"
+
+# The one credential the Codex lane bills on. config.toml names it (`env_key`) and the
+# container is handed it BY NAME, so the value never reaches the file or argv.
+KEY_ENV = "OPENAI_API_KEY"
+
+_INSTALL_HINT = (
+ "\n install: npm i -g @openai/codex (needs Node on the host)"
+ "\n override: TE_CODEX_BIN (the node shim) / TE_CODEX_VENDOR (the vendor triple dir)")
+
+
+# --------------------------------------------------------------------------------------
+# host install
+# --------------------------------------------------------------------------------------
+def codex_bin():
+ """Host `codex` entry point (the Node shim), for version reporting only."""
+ p = os.environ.get("TE_CODEX_BIN") or shutil.which("codex")
+ if not p:
+ raise RuntimeError("no codex binary found on PATH" + _INSTALL_HINT)
+ return p
+
+
+def codex_vendor(binary=None):
+ """The host vendor triple dir holding the static-PIE ELF the container runs.
+
+ Resolved from the Node shim's install tree rather than hardcoded: the AgentEO driver
+ pinned an absolute path under one machine's nvm prefix, which is exactly the kind of
+ path docs/BENCHMARKS.md keeps out of the code. TE_CODEX_VENDOR overrides.
+
+ Raises when the ELF is absent. Docker invents an EMPTY directory for a missing bind
+ mount source rather than failing, so an unchecked bad path here becomes "codex: not
+ found" inside every container -- a whole tree of 0-turn rows behind a green launch.
+ """
+ v = os.environ.get("TE_CODEX_VENDOR")
+ if not v:
+ shim = os.path.realpath(binary or codex_bin())
+ # <prefix>/lib/node_modules/@openai/codex/bin/codex.js
+ # -> <prefix>/lib/node_modules/@openai/codex/node_modules/@openai/codex-<triple>/vendor/<triple>
+ pkg = os.path.dirname(os.path.dirname(shim))
+ base = os.path.join(pkg, "node_modules", "@openai")
+ try:
+ plats = sorted(d for d in os.listdir(base) if d.startswith("codex-"))
+ except OSError:
+ plats = []
+ for plat in plats:
+ vend = os.path.join(base, plat, "vendor")
+ try:
+ triples = sorted(os.listdir(vend))
+ except OSError:
+ continue
+ for triple in triples:
+ if os.path.isfile(os.path.join(vend, triple, "bin", "codex")):
+ v = os.path.join(vend, triple)
+ break
+ if v:
+ break
+ if not v or not os.path.isfile(os.path.join(v, "bin", "codex")):
+ raise RuntimeError(
+ f"no codex native binary under vendor dir {v!r} -- a bind mount of a missing "
+ "path becomes an EMPTY dir in the container, so every task would fail with "
+ "'codex: not found' behind a green launch banner." + _INSTALL_HINT)
+ return v
+
+
+def require_install():
+ """(shim, vendor_dir) with both proven present. Called at runner construction."""
+ binary = codex_bin()
+ return binary, codex_vendor(binary)
+
+
+def cli_version(binary):
+ """`codex --version`, so a run records the exact CLI build that produced it.
+
+ Recorded rather than decorative: the accepted config-toggle names moved between 0.146
+ and 0.153 (see the module docstring), so the build is part of what a pruning arm means.
+ """
+ try:
+ p = subprocess.run([binary, "--version"], capture_output=True, text=True,
+ timeout=60)
+ return (p.stdout or p.stderr or "").strip().splitlines()[0][:80]
+ except Exception:
+ return ""
+
+
+def container_mounts(vendor):
+ """host->container `-v` specs. Only the vendor dir: the ELF finds its siblings
+ (`rg`, the sandbox helpers) relative to itself, so one read-only mount is the whole
+ install."""
+ return ["-v", f"{vendor}:{C_VENDOR}:ro"]
+
+
+# --------------------------------------------------------------------------------------
+# the endpoint: the OpenAI API, directly
+# --------------------------------------------------------------------------------------
+def resolve_codex_routing(env=None):
+ """The OpenAI endpoint and key this run bills, decided from the environment alone.
+
+ Returns {base_url, api_key}. `base_url` is OPENAI_BASE_URL when set -- OpenAI's own
+ regional endpoint, which a region-scoped key needs -- else the global one; it is what
+ config.toml names and what the mechanism calls go to. `api_key` is OPENAI_API_KEY.
+
+ Raises when the key is unset -- at construction, rather than once per task.
+ """
+ env = os.environ if env is None else env
+ key = env.get(KEY_ENV, "")
+ if not key:
+ raise RuntimeError(
+ "OPENAI_API_KEY is unset, so this Codex lane has nothing to bill -- see the "
+ "repo's .env. (A region-scoped key also needs OPENAI_BASE_URL: it 401s on the "
+ "global endpoint.)")
+ base = (env.get("OPENAI_BASE_URL") or OPENAI_DIRECT).strip().rstrip("/")
+ return {"base_url": base, "api_key": key}
+
+
+def validate_priceable(model, agent):
+ """Fail at construction if the billed id has no weight entry.
+
+ Same early check the Claude and Copilot runners make: `pricing.effective_tokens`
+ otherwise raises at the END of run_task, after the rollout has been paid for and
+ graded.
+ """
+ pricing.weights_for({"model": model, "agent": agent})
+
+
+# --------------------------------------------------------------------------------------
+# catalog preflight
+# --------------------------------------------------------------------------------------
+_CATALOG = {}
+
+
+def fetch_catalog(base_url, api_key, timeout=10):
+ """frozenset of served model ids at `base_url`, or None when unreadable.
+
+ Cached per (base_url, key): one lane asks once, not once per task.
+ """
+ key = (base_url, api_key)
+ if key in _CATALOG:
+ return _CATALOG[key]
+ out = None
+ try:
+ req = urllib.request.Request(base_url.rstrip("/") + "/models",
+ headers={"authorization": f"Bearer {api_key}"})
+ with urllib.request.urlopen(req, timeout=timeout) as r:
+ payload = json.loads(r.read().decode())
+ # The isinstance guards keep a payload that is not an object at all from raising
+ # AttributeError, which the except clause below would NOT catch.
+ data = payload.get("data") if isinstance(payload, dict) else None
+ if isinstance(data, list):
+ ids = frozenset(m["id"] for m in data
+ if isinstance(m, dict) and isinstance(m.get("id"), str))
+ out = ids or None
+ except (urllib.error.URLError, OSError, ValueError, TypeError):
+ out = None
+ _CATALOG[key] = out
+ return out
+
+
+def verify_model(model, routing):
+ """Refuse a lane whose endpoint does not serve the id it intends to bill.
+
+ An unserved id is a per-call FAILURE, not a fallback: every task would run, bill and
+ fail while the launcher's banner stayed green.
+
+ A catalog that cannot be read is NOT treated as a refusal -- a network hiccup at
+ construction must not refuse a lane -- but it is reported so the caller can log it,
+ because "could not check" and "checked and absent" must not look the same in the
+ record.
+ """
+ served = fetch_catalog(routing["base_url"], routing["api_key"])
+ if served is None:
+ return None, "catalog unreadable -- model not verified"
+ if model in served:
+ return True, ""
+ sample = " ".join(sorted(served)[:12])
+ return False, (
+ f"{routing['base_url']} does not serve {model!r}. An unserved id fails per task "
+ f"rather than at startup, so this lane is refused. Served (first 12): {sample}")
+
+
+# --------------------------------------------------------------------------------------
+# the request contract: config.toml
+# --------------------------------------------------------------------------------------
+# Capability package -> the concrete Codex config toggles that remove it.
+#
+# EVERY key below was probed against the installed CLI with `codex exec --strict-config`,
+# which reports an unrecognized configuration field as an error. Two keys the AgentEO
+# 0.146 driver used are rejected by 0.153.1 and are NOT here:
+# web_search_mode = "disabled" -> now top-level web_search = "disabled"
+# [tools] view_image = false -> now [features] view_image = false
+# Both were silently ignored by the newer CLI, which is the whole hazard: the pruning
+# becomes a no-op and the arm measures itself as useless while still being called
+# economized. `validate_config` is the gate that keeps this honest across CLI upgrades.
+#
+# Only the packages `codex_untool.CONFIG_CAPABILITIES` nominates appear here. The rest of
+# the mask is relay-scoped and is applied by the transcript-side controller, not by config
+# -- mapping a capability with no supported toggle would be a lie in run_config.json.
+# `goals` (2026-09-11): `[features] goals = false` is accepted by 0.153.1 and removes the
+# product-goal tool docs from the code-mode exec description. 0.153.1 REJECTS a toggle for
+# the plan tool, for skills and for request_user_input, so none of those is here.
+# `skills` (2026-09-11): the CLI installs its bundled "system skills" into
+# $CODEX_HOME/skills/.system at startup and lists every enabled one in a
+# <skills_instructions> developer message. There is no switch for
+# the listing itself (`[features] skill_search`, `skip_host_skill_discovery` and `[skills]
+# enabled` leave it intact or are rejected), but a per-skill `[[skills.config]] enabled =
+# false` is accepted and, applied to every bundled skill, removes the whole block. The bundled
+# set is discovered from the installed CLI at runner construction (`bundled_system_skills`);
+# `DEFAULT_SYSTEM_SKILLS` is the 0.153.1 set, used only when discovery fails.
+_TOP_LEVEL_TOGGLES = {
+ "collaboration": ("include_collaboration_mode_instructions = false",),
+ "web": ('web_search = "disabled"',),
+ "goals": (),
+ "skills": (),
+ "image_media": (),
+ "interactive_guidance": (),
+}
+_TABLE_TOGGLES = {
+ "collaboration": (("agents", "enabled = false"),
+ ("features", "multi_agent = false")),
+ # Only the TOP-LEVEL `web_search = "disabled"` (in _TOP_LEVEL_TOGGLES) remains.
+ # codex-cli 0.153.1 emits a deprecation ERROR ITEM into the transcript for
+ # `[features].web_search` -- "deprecated because web search is enabled by default; set
+ # `web_search` to live/indexed/cached/disabled at the top level" -- observed on a live
+ # preflight 2026-09-10. An error item per task is trace noise the mechanisms then read,
+ # and a deprecation becomes a hard failure sooner or later, so the redundant table
+ # entries are dropped in favour of the spelling the CLI itself names.
+ "web": (),
+ "goals": (("features", "goals = false"),),
+ "skills": (),
+ "image_media": (("features", "view_image = false"),),
+ "interactive_guidance": (),
+}
+CONFIG_CAPABILITIES = frozenset(_TABLE_TOGGLES)
+
+# `interactive_guidance` (2026-09-11): part of the base prompt is guidance for an
+# interactive chat -- persona, writing style, the commentary/final channels, how to handle
+# a user message arriving mid-turn, answer formatting, visualizations. A detached
+# `codex exec` run has no user reading commentary and its final message is not graded.
+# The CLI accepts
+# `model_instructions_file`, which REPLACES the base prompt, so the package is removed by
+# writing a lean copy of the base prompt with those sections cut and every operational
+# section (rules for getting work done, file editing, autonomy, destructive actions) kept
+# byte-for-byte. The base prompt is version-specific: it is read from
+# data/codex_base_prompt/<cli version>.md, captured by recording the CLI's own request, and
+# the package is simply unavailable -- left loaded, logged -- on a CLI build without a
+# capture, so an upgrade can never apply a stale prompt. The skills how-to sections are
+# cut by the same file whenever `skills` is pruned: they describe a listing that is no
+# longer there.
+INSTRUCTIONS_PATH = C_HOME + "/instructions.md"
+INSTRUCTION_SECTIONS = {
+ "interactive_guidance": (
+ "# Personality", "## Writing style", "## Technical communication",
+ "# Working with the user", "## Intermediate commentary", "## Final answer",
+ "### Formatting rules", "### Visualizations"),
+ "skills": ("# Using skills", "### How to use skills"),
+}
+# Kept even inside a dropped section, because both paragraphs are operational rather than
+# presentational: what to do after a context compaction, and -- the only statement anywhere
+# in the prompt of how a turn ENDS -- that the agent yields by sending a final message.
+# Dropping the latter removes the agent's stopping rule, and an agent with no stated way
+# to end a turn spends extra model calls finding one -- so both stay, however short the
+# rest of the lean prompt gets.
+_KEEP_PARAGRAPH_RE = re.compile(r"When you run out of context|end your turn", re.I)
+_SECTION_RE = re.compile(r"(?m)^(?=#{1,6} )")
+
+
+def base_prompt_for(cli_version):
+ """The captured base prompt for this CLI build, or None when there is no capture."""
+ m = re.search(r"(\d+\.\d+\.\d+)", str(cli_version or ""))
+ if not m:
+ return None
+ path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data",
+ "codex_base_prompt", m.group(1) + ".md")
+ if not os.path.exists(path):
+ return None
+ with open(path, encoding="utf-8") as fh:
+ return fh.read()
+
+
+def lean_instructions(base_text, capabilities):
+ """The base prompt minus the sections the pruned packages make inapplicable.
+
+ Returns None when no section is dropped (no instructions file is written and the CLI
+ keeps its built-in prompt). Everything not dropped is kept byte-for-byte."""
+ drop = set()
+ for cap in capabilities or ():
+ drop.update(INSTRUCTION_SECTIONS.get(cap, ()))
+ if not base_text or not drop:
+ return None
+ out, dropped = [], 0
+ for section in _SECTION_RE.split(base_text):
+ header = section.split("\n", 1)[0].strip()
+ if header in drop:
+ dropped += 1
+ kept = [p.strip() for p in section.split("\n\n") if _KEEP_PARAGRAPH_RE.search(p)]
+ if kept:
+ out.append("\n\n".join(kept) + "\n\n")
+ continue
+ out.append(section)
+ if not dropped:
+ return None
+ return "".join(out).rstrip() + "\n"
+
+DEFAULT_SYSTEM_SKILLS = ("imagegen", "openai-docs", "plugin-creator", "review-agent",
+ "skill-creator", "skill-installer")
+
+
+def skills_config_entries(system_skills=None):
+ """`[[skills.config]]` blocks that disable every bundled system skill, in-container
+ paths. One block per skill; a path the CLI does not have is harmless."""
+ names = tuple(system_skills) if system_skills else DEFAULT_SYSTEM_SKILLS
+ return [f'[[skills.config]]\npath = {json.dumps(f"{C_HOME}/skills/.system/{n}/SKILL.md")}\n'
+ f'enabled = false\n' for n in names]
+
+
+def bundled_system_skills(binary=None, home=None, timeout=45):
+ """The system skills the installed CLI writes into a fresh CODEX_HOME, by name.
+
+ One offline `codex exec` against a throwaway CODEX_HOME whose provider points at a dead
+ port. The CLI installs its bundled skills (and drops a `.codex-system-skills.marker`)
+ before it touches the network; a dead port then leaves it "waiting for network"
+ indefinitely, so the process is killed as soon as the marker appears rather than waited
+ for. Returns [] when the CLI cannot be run or the marker never shows, so callers fall
+ back to DEFAULT_SYSTEM_SKILLS and say so. Nothing is billed."""
+ binary = binary or codex_bin()
+ # mkdtemp + an explicit RETRYING cleanup, not TemporaryDirectory. The killed CLI (or a
+ # child it spawned) can still be writing under CODEX_HOME when we come to remove it, and
+ # a "Directory not empty" here failed the offline suite about one run in four.
+ # `ignore_cleanup_errors=True` also makes that non-fatal, but it gives up on the first
+ # error and LEAVES the directory: 899 MB of throwaway CODEX_HOMEs had accumulated in
+ # /tmp on this host by 2026-09-13. `_rmtree_quietly` retries the transient case first
+ # and only then ignores, so the common case is actually cleaned up.
+ tmp = tempfile.mkdtemp(dir=home)
+ try:
+ with open(os.path.join(tmp, "config.toml"), "w", encoding="utf-8") as fh:
+ fh.write('model = "gpt-5.6-terra"\nmodel_provider = "dead"\n'
+ '[model_providers.dead]\nname = "OpenAI"\n'
+ 'base_url = "http://127.0.0.1:9/v1"\nenv_key = "TE_DEAD_KEY"\n'
+ 'wire_api = "responses"\nrequest_max_retries = 0\n')
+ env = dict(os.environ, CODEX_HOME=tmp, TE_DEAD_KEY="dead")
+ root = os.path.join(tmp, "skills", ".system")
+ marker = os.path.join(root, ".codex-system-skills.marker")
+ try:
+ proc = subprocess.Popen(
+ [binary, "exec", "--skip-git-repo-check", "-C", tmp, "probe"],
+ stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL, env=env, start_new_session=True)
+ except OSError:
+ return []
+ try:
+ deadline = time.time() + timeout
+ while time.time() < deadline and proc.poll() is None:
+ if os.path.exists(marker):
+ break
+ time.sleep(0.2)
+ finally:
+ _kill_process_group(proc)
+ if not os.path.isdir(root):
+ return []
+ return sorted(n for n in os.listdir(root)
+ if os.path.isfile(os.path.join(root, n, "SKILL.md")))
+ finally:
+ _rmtree_quietly(tmp)
+
+
+def _kill_process_group(proc):
+ """Kill everything a probe started -- not just the process `Popen` returned.
+
+ `codex` on PATH is the npm package's Node shim, which execs the native ELF as a CHILD.
+ `proc.kill()` reached only the shim: the native binary, pointed at a dead port, was
+ reparented to init and sat "waiting for network" for good. Every offline suite run
+ leaked one (six were alive on this host on 2026-09-30, the oldest five days old).
+ The probes therefore start in their own session, and the whole group goes -- also when
+ the shim has already exited, because its child can outlive it."""
+ try:
+ os.killpg(proc.pid, signal.SIGKILL)
+ except (ProcessLookupError, PermissionError, AttributeError):
+ pass
+ try:
+ proc.wait(timeout=10)
+ except subprocess.TimeoutExpired:
+ pass
+
+
+def _rmtree_quietly(path):
+ for _ in range(3):
+ try:
+ shutil.rmtree(path)
+ return
+ except OSError:
+ time.sleep(0.2)
+ shutil.rmtree(path, ignore_errors=True)
+
+
+# Dropped on EVERY arm, baseline included, and not part of any mechanism's saving.
+#
+# The answers to both benchmarks are a fetch away (a SWE-bench instance's own fix commit is
+# public), so the Claude and Copilot lanes drop WebSearch/WebFetch on every arm on top of
+# whatever untooling decided -- swebench.network_tool_block. Codex gets the same treatment
+# through the only seam it has, its config.
+#
+# STATE THE CONSEQUENCE, because it changes what a Codex untooling number means: with web
+# search off in the baseline too, the `web` package is ALREADY absent from the contract when
+# the mask prunes it, so `web` contributes no saving on this benchmark and the measured
+# effect comes from the collaboration and image/media packages and the relay-scoped
+# capabilities. TE_ALLOW_NETWORK_TOOLS=1 lifts it for a run that genuinely wants a browsing
+# agent, exactly as it does on the other two lanes.
+def network_block_toggles():
+ """Contamination-control toggles applied to every arm, or () when lifted."""
+ if os.environ.get("TE_ALLOW_NETWORK_TOOLS") == "1":
+ return (), ()
+ return _TOP_LEVEL_TOGGLES["web"], _TABLE_TOGGLES["web"]
+
+
+def config_toggles(capabilities):
+ """(top_level_lines, table_lines) for a frozen capability mask.
+
+ `capabilities` is the logical mask the mechanism froze for this task. A package with no
+ supported config toggle contributes nothing here rather than a guess.
+ """
+ # Checked against the RAW mask, before the CONFIG_CAPABILITIES filter. Filtering first
+ # made this guard dead code -- a protected package is not in CONFIG_CAPABILITIES, so it
+ # was silently dropped and the refusal never fired. The point is to reject a mask that
+ # ASKS for something protected, not to quietly honour the part of it that is safe.
+ bad = PROTECTED_CAPABILITIES.intersection(capabilities or ())
+ if bad:
+ # The mechanism asserts this on its own side too; assert it here as well because
+ # this is the layer that would actually remove the agent's ability to work.
+ raise RuntimeError(
+ f"refusing to prune protected Codex capabilities {sorted(bad)}: without an "
+ "edit path and a shell there is no task, and the resulting 'resolved: false' "
+ "would be a harness artifact rather than a mechanism result")
+ caps = [c for c in (capabilities or ()) if c in CONFIG_CAPABILITIES]
+ top, tables = [], []
+ for cap in sorted(caps):
+ top.extend(_TOP_LEVEL_TOGGLES.get(cap, ()))
+ tables.extend(_TABLE_TOGGLES.get(cap, ()))
+ net_top, net_tables = network_block_toggles()
+ top.extend(net_top)
+ tables.extend(net_tables)
+ # De-duplicate while keeping a stable order: `web` can arrive from both the mask and
+ # the unconditional network block, and Codex rejects a duplicated key outright.
+ return _dedupe(top), _dedupe(tables)
+
+
+def _dedupe(items):
+ seen, out = set(), []
+ for it in items:
+ if it not in seen:
+ seen.add(it)
+ out.append(it)
+ return out
+
+
+def config_toml(*, routing, model, effort, project="/testbed", capabilities=(),
+ system_skills=None, instructions=None, instructions_path=None,
+ mcp_servers=None):
+ """The `$CODEX_HOME/config.toml` this run's contract is expressed in.
+
+ One provider block, `[model_providers.openai_direct]`, at the endpoint
+ `resolve_codex_routing` resolved, reading the key from OPENAI_API_KEY by NAME -- the
+ value never goes in the file.
+ """
+ top, tables = config_toggles(capabilities)
+ grouped = {}
+ for table, line in tables:
+ grouped.setdefault(table, []).append(line)
+
+ out = [f'model = {json.dumps(model)}\n',
+ 'model_provider = "openai_direct"\n',
+ f'model_reasoning_effort = {json.dumps(effort)}\n']
+ if instructions:
+ # The lean base prompt the runner writes beside this file (see lean_instructions).
+ # The default is the CONTAINER path, which is where the SWE-bench lane puts it.
+ # The Web-Bench lane runs the agent on the host with a per-task CODEX_HOME, so it
+ # passes its own absolute path -- a container path there would name a file that does
+ # not exist, and `model_instructions_file` pointing at nothing means the base prompt
+ # is not replaced and the untooling arm quietly keeps the sections it claims to have
+ # dropped.
+ out.append('model_instructions_file = '
+ f'{json.dumps(instructions_path or INSTRUCTIONS_PATH)}\n')
+ out += [line + "\n" for line in top]
+ out.append('\n[model_providers.openai_direct]\n'
+ 'name = "OpenAI"\n'
+ f'base_url = {json.dumps(routing["base_url"])}\n'
+ f'env_key = {json.dumps(KEY_ENV)}\n'
+ 'wire_api = "responses"\n')
+ # Trusting the project is what lets `codex exec` run unattended in a fresh checkout.
+ out.append(f'\n[projects.{json.dumps(project)}]\ntrust_level = "trusted"\n')
+ # MCP servers a scaffold asked for (serena). Codex takes these from config.toml -- there
+ # is no argv flag, which is why this lane's AgentProfile.mcp_args is empty -- and
+ # `--strict-config` validates the table (command, args, env, cwd, startup_timeout_sec
+ # accepted; an unknown key refused; probed on 0.153.1). Emitted before the
+ # array-of-tables skills entries, which must stay last.
+ for name in sorted(mcp_servers or {}):
+ spec = mcp_servers[name]
+ out.append(f"\n[mcp_servers.{name}]\n")
+ out.append(f"command = {json.dumps(spec['command'])}\n")
+ out.append("args = [" + ", ".join(json.dumps(a) for a in spec.get("args", ())) + "]\n")
+ if spec.get("cwd"):
+ out.append(f"cwd = {json.dumps(spec['cwd'])}\n")
+ # Serena starts a language server before it answers; its own docs say 15 s, but in
+ # a cold task container the first start has been longer. Generous rather than a
+ # per-task race that reads as "the agent never used the tools".
+ out.append(f"startup_timeout_sec = {int(spec.get('startup_timeout_sec', 90))}\n")
+ if spec.get("env"):
+ out.append(f"\n[mcp_servers.{name}.env]\n")
+ for k in sorted(spec["env"]):
+ out.append(f"{k} = {json.dumps(str(spec['env'][k]))}\n")
+ for table in sorted(grouped):
+ out.append(f"\n[{table}]\n")
+ out += [line + "\n" for line in _dedupe(grouped[table])]
+ # Array-of-tables entries go last (they would otherwise swallow the keys after them).
+ if "skills" in (capabilities or ()):
+ out.append("\n" + "\n".join(skills_config_entries(system_skills)))
+ return "".join(out)
+
+
+def validate_config(text, *, binary=None, home=None, timeout=90):
+ """Prove the installed CLI recognizes every field in `text`. ("", ) on success.
+
+ Returns an error string when it does not. This is the gate that keeps a CLI upgrade
+ from turning the pruning toggles into a silent no-op: `codex exec --strict-config`
+ errors on an unrecognized configuration field, and without it Codex simply ignores one.
+
+ Runs against a THROWAWAY CODEX_HOME and never reaches the network -- config loading
+ fails before any request is made, so this costs nothing and bills nothing.
+ """
+ binary = binary or codex_bin()
+ tmp = tempfile.mkdtemp(dir=home)
+ try:
+ with open(os.path.join(tmp, "config.toml"), "w", encoding="utf-8") as fh:
+ fh.write(text)
+ env = dict(os.environ, CODEX_HOME=tmp)
+ try:
+ proc = subprocess.Popen(
+ [binary, "exec", "--strict-config", "--skip-git-repo-check",
+ "-C", tmp, "probe"],
+ stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, env=env, start_new_session=True)
+ except OSError as e:
+ return f"could not run the config validator: {e}"
+ try:
+ out, err = proc.communicate(timeout=timeout)
+ except subprocess.TimeoutExpired as e:
+ return f"could not run the config validator: {e}"
+ finally:
+ _kill_process_group(proc) # the CLI's helpers can outlive it, as can the ELF
+ blob = (out or "") + (err or "")
+ for line in blob.splitlines():
+ if ("unknown configuration field" in line or "invalid type" in line
+ or "unknown variant" in line or "Error loading config.toml" in line):
+ return blob.strip()[:800]
+ return ""
+ finally:
+ # The CLI's helpers (plugin/skill installers) can still be writing into the
+ # throwaway home when the process has exited; a leftover temp dir is not an error.
+ _rmtree_quietly(tmp)
+
+
+# --------------------------------------------------------------------------------------
+# invocation
+# --------------------------------------------------------------------------------------
+# Codex has NO turn-cap flag (no `--max-turns`), so `max_turns` is recorded in
+# run_config.json and counted by the spiral envelope but is NOT enforced on the agent --
+# the same caveat the Copilot lane carries. TE_TASK_TIMEOUT and the spiral guard are the
+# only bounds. The model and the reasoning effort come from config.toml rather than from
+# flags so that the file this run wrote is the single source of truth for its contract.
+def build_argv(*, hook=False):
+ """argv for one fresh `codex exec` segment, reading the prompt from stdin.
+
+ `--dangerously-bypass-approvals-and-sandbox`: the container IS the sandbox (ephemeral,
+ per task, network-guarded), the same posture as the Claude lane's IS_SANDBOX=1. Codex's
+ own sandbox layer inside it would only forbid the edits the task requires.
+ """
+ argv = [C_BIN, "exec", "--json", "--color", "never", "--skip-git-repo-check",
+ "--dangerously-bypass-approvals-and-sandbox"]
+ if hook:
+ # Enabled hooks otherwise require persisted per-source trust, which an unattended
+ # container has no way to grant -- the guard would simply not run.
+ argv.append("--dangerously-bypass-hook-trust")
+ return argv + ["-C", "/testbed", "-"]
+
+
+def build_resume_argv(session_id, *, model=None, effort=None, hook=False):
+ """argv to continue the EXACT thread the Codex spiral guard stopped.
+
+ Resuming the thread rather than starting a fresh session is the whole mechanism: a new
+ session would re-pay for the context it already built and lose the state the nudge is
+ written against. `codex exec resume` has a smaller option surface than `exec` -- no
+ `-C` and no `--color` -- and the working directory comes from `docker exec -w` instead.
+ """
+ argv = [C_BIN, "exec", "resume", str(session_id), "--json", "--skip-git-repo-check",
+ "--dangerously-bypass-approvals-and-sandbox"]
+ if hook:
+ argv.append("--dangerously-bypass-hook-trust")
+ if model:
+ argv += ["-m", model]
+ if effort:
+ # The policy may resume at a HIGHER effort than the segment that stalled; that is a
+ # deliberate part of the mechanism, so it is applied per invocation rather than
+ # rewritten into config.toml where it would also affect the first segment.
+ argv += ["-c", f"model_reasoning_effort={json.dumps(effort)}"]
+ return argv + ["-"]
+
+
+def run_env(extra=None):
+ """Non-secret `docker exec -e` values. The CREDENTIAL is never here.
+
+ The key is forwarded by NAME only (`docker exec -e KEY_ENV` with no `=value` takes the
+ value from the client's environment), so it never reaches argv -- which on a shared
+ host is readable by every other user through `ps`.
+ """
+ env = {"CODEX_HOME": C_HOME, "HOME": "/tmp/agent-home",
+ # Codex writes session rollouts under CODEX_HOME and the resolved model id is
+ # recorded there, not in the --json stream. The spiral resume needs the thread
+ # id and pricing needs the id, so the home must stay writable and persistent
+ # for the whole task -- `--ephemeral` would break both.
+ "RUST_BACKTRACE": "0"}
+ env.update(extra or {})
+ return env
+
+
+# --------------------------------------------------------------------------------------
+# accounting
+# --------------------------------------------------------------------------------------
+# Codex's `--json` stream carries usage on `turn.completed`, and the shape is NOT the
+# Claude one:
+#
+# "usage": {input_tokens, cached_input_tokens, cache_write_input_tokens,
+# output_tokens, reasoning_output_tokens}
+#
+# `input_tokens` is the GRAND input total and already INCLUDES both cache fields (verified
+# across probes in the AgentEO study, whose accounting the published Codex rows used). So
+# fresh input is input - cached - cache_write, and folding the cache fields into
+# input_tokens instead would price ~90% cached re-reads at 1.0x where the vendor bills
+# 0.10x -- the same one-directional error that made Copilot rows look cheaper than Claude
+# rows before the 2026-09-05 cache-write fix. Reasoning tokens are folded into output
+# because that is how they are billed.
+_USAGE_KEYS = ("input_tokens", "cached_input_tokens", "cache_write_input_tokens",
+ "output_tokens", "reasoning_output_tokens")
+
+
+def row_usage(stream_text):
+ """Token/turn row fields from one segment's `codex exec --json` stream.
+
+ Leaves the token fields None when the stream carried no usage block, so a caller can
+ record "unknown" rather than a confident zero -- `usage_incomplete` on the row is what
+ makes such a task filterable instead of silently averaged in as cheap.
+ """
+ tot = dict.fromkeys(_USAGE_KEYS, 0)
+ steps = 0
+ tool_calls = {}
+ saw_turn = saw_usage = False
+ thread_id = ""
+ errors = []
+ for line in (stream_text or "").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ kind = ev.get("type")
+ if kind == "item.completed":
+ item = ev.get("item") or {}
+ itype = item.get("type")
+ if itype == "command_execution":
+ tool_calls["shell"] = tool_calls.get("shell", 0) + 1
+ steps += 1
+ elif itype == "file_change":
+ tool_calls["apply_patch"] = tool_calls.get("apply_patch", 0) + 1
+ steps += 1
+ elif itype == "agent_message":
+ steps += 1
+ elif kind == "turn.completed":
+ saw_turn = True
+ usage = ev.get("usage") or {}
+ if usage:
+ saw_usage = True
+ for k in _USAGE_KEYS:
+ tot[k] += int(usage.get(k) or 0)
+ elif kind == "thread.started":
+ thread_id = ev.get("thread_id") or thread_id
+ elif kind == "error":
+ errors.append(str(ev.get("message") or "")[:200])
+ # The thread id also rides on session/thread metadata in some builds; take the
+ # first non-empty one seen so a resume never falls back to a FRESH session, which
+ # would re-pay for the whole context and is the harmful path codex_spiral guards.
+ for key in ("thread_id", "session_id", "conversation_id"):
+ if not thread_id and isinstance(ev.get(key), str):
+ thread_id = ev[key]
+
+ row = {
+ # Codex emits ONE turn.completed per exec however many tools ran, so that
+ # undercounts agent effort by an order of magnitude and is not comparable to the
+ # other agents' num_turns. Count agent STEPS (messages + tool calls) instead, and
+ # say so on the row rather than letting the basis be inferred.
+ "num_turns": steps or None,
+ "turns_basis": "codex_steps",
+ "tool_calls": dict(sorted(tool_calls.items())),
+ "thread_id": thread_id,
+ "is_error": bool(errors) and not saw_turn,
+ # Codex does not report a USD figure, so the vendor cost stays null rather than a
+ # fabricated zero.
+ "cost_usd_vendor": None,
+ }
+ if errors:
+ row["api_error_status"] = errors[0]
+ if saw_usage:
+ fresh = max(0, tot["input_tokens"] - tot["cached_input_tokens"]
+ - tot["cache_write_input_tokens"])
+ row.update({
+ "input_tokens": fresh,
+ "cache_read_input_tokens": tot["cached_input_tokens"],
+ "cache_creation_input_tokens": tot["cache_write_input_tokens"],
+ "output_tokens": tot["output_tokens"] + tot["reasoning_output_tokens"],
+ "reasoning_output_tokens": tot["reasoning_output_tokens"],
+ "tokens_source": "stream_usage",
+ })
+ row["tokens_total"] = (row["input_tokens"] + row["cache_read_input_tokens"]
+ + row["cache_creation_input_tokens"]
+ + row["output_tokens"])
+ else:
+ row.update({
+ "input_tokens": None, "cache_read_input_tokens": None,
+ "cache_creation_input_tokens": None, "output_tokens": None,
+ "tokens_total": None,
+ "tokens_source": "no_usage_event" if saw_turn else "no_turn_event",
+ })
+ return row
+
+
+def merge_usage(a, b):
+ """Combine two spiral segments' usage into one.
+
+ None + number must not become the number: a segment whose usage was unrecoverable makes
+ the TOTAL unknown, and reporting the half we have would understate the task's cost --
+ precisely on the guarded arm the spiral mechanism is meant to show saving on. So a None
+ on either side keeps the field None, and `tokens_source` records the degradation.
+ """
+ a = a or {}
+ b = b or {}
+ if not a:
+ return dict(b)
+ if not b:
+ return dict(a)
+ out = dict(a)
+ for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens",
+ "output_tokens", "reasoning_output_tokens", "tokens_total"):
+ x, y = a.get(k), b.get(k)
+ out[k] = None if (x is None or y is None) else (x or 0) + (y or 0)
+ for k in ("num_turns",):
+ x, y = a.get(k), b.get(k)
+ out[k] = None if (x is None and y is None) else (x or 0) + (y or 0)
+ calls = dict(a.get("tool_calls") or {})
+ for name, n in (b.get("tool_calls") or {}).items():
+ calls[name] = calls.get(name, 0) + n
+ out["tool_calls"] = dict(sorted(calls.items()))
+ # The FIRST segment's thread is the one a later resume must continue.
+ out["thread_id"] = a.get("thread_id") or b.get("thread_id") or ""
+ srcs = [s for s in (a.get("tokens_source"), b.get("tokens_source")) if s]
+ out["tokens_source"] = ("stream_usage" if set(srcs) == {"stream_usage"}
+ else "+".join(srcs) or None)
+ out["is_error"] = bool(a.get("is_error")) or bool(b.get("is_error"))
+ out["cost_usd_vendor"] = None
+ return out
+
+
+# --------------------------------------------------------------------------------------
+# the mechanisms' between-task call
+# --------------------------------------------------------------------------------------
+def optimizer_call(prompt, *, configured_model, routing, max_tokens=16384,
+ timeout=180):
+ """One proposer / judge / nudge-writer call on THIS lane's endpoint. (text, eff_tokens).
+
+ The Codex twin of `swebench.anthropic_optimizer_call`, and it exists for the same two
+ reasons that one does. It must bill what the TASKS bill -- `ledger.overhead` charges
+ these calls to the same basis as the task rows, and a cheaper proposer would understate
+ the overhead the arm actually incurs -- and it must be a pure text completion that
+ cannot reach the testbed, which a `codex exec` session would not be.
+
+ Over `/v1/responses`, the API the CLI itself speaks, with no tools offered at all --
+ so the call is a pure text completion whatever the model answers.
+
+ Never raises: a transport failure returns ("", 0.0), which every mechanism reads as
+ "no verdict" and falls back to its LLM-free path. That silent-degradation property is
+ why `preflight` below spends one probe call before task 0.
+ """
+ from .swebench import _log, _log_optimizer_call # deferred: swebench imports us
+
+ # `configured_model` is already the lane's own model; the driver's "agent" / "codex"
+ # label never reaches here, because the runner substitutes its model before calling.
+ body = json.dumps({
+ "model": configured_model,
+ "input": [{"role": "user",
+ "content": [{"type": "input_text", "text": prompt}]}],
+ "max_output_tokens": max_tokens,
+ "store": False,
+ }).encode()
+ headers = {"content-type": "application/json",
+ "authorization": f"Bearer {routing['api_key']}"}
+ req = urllib.request.Request(
+ routing["base_url"].rstrip("/") + "/responses",
+ data=body, method="POST", headers=headers)
+ data = None
+ for attempt in range(3):
+ try:
+ with urllib.request.urlopen(req, timeout=timeout) as resp:
+ data = json.loads(resp.read().decode())
+ break
+ except urllib.error.HTTPError as e:
+ if e.code in (429, 500, 502, 503, 529) and attempt < 2:
+ time.sleep(4 * (attempt + 1))
+ continue
+ _log(f"codex optimizer_call HTTP {e.code}: {e.read()[:1200]!r}")
+ return "", 0.0
+ except Exception as e:
+ if attempt < 2:
+ time.sleep(4 * (attempt + 1))
+ continue
+ _log(f"codex optimizer_call failed: {e}")
+ return "", 0.0
+ if data is None:
+ return "", 0.0
+
+ text = _responses_text(data)
+ usage = data.get("usage") or {}
+ cached = int((usage.get("input_tokens_details") or {}).get("cached_tokens") or 0)
+ total_in = int(usage.get("input_tokens") or 0)
+ # A reasoning model can spend the whole cap on reasoning and emit a ZERO-character
+ # answer -- the caller then parses nothing and acts on nothing while the call is billed
+ # in full. The Claude path hit exactly this (7 of 40 manual-distill calls at an 8192
+ # cap) and answered it by raising the cap and saying so, not by lowering effort: effort
+ # is part of the cost basis and cutting it here would price mechanism overhead below
+ # what the arm really spends. So warn, loudly, and leave the knob to the operator.
+ if data.get("status") == "incomplete" or not text.strip():
+ why = ((data.get("incomplete_details") or {}).get("reason")
+ or ("empty response" if not text.strip() else "unknown"))
+ _log(f"codex optimizer response unusable ({why}) at max_output_tokens="
+ f"{max_tokens}: the caller will act on nothing while the call still bills. "
+ "Raise TE_OPTIMIZER_MAX_TOKENS.")
+ _log_optimizer_call(configured_model, prompt, text, {
+ "input_tokens": max(total_in - cached, 0),
+ "cache_read_input_tokens": cached,
+ "output_tokens": int(usage.get("output_tokens") or 0),
+ }, stop_reason=data.get("status"))
+ eff = pricing.effective_tokens({
+ "model": configured_model,
+ "input_tokens": max(total_in - cached, 0),
+ "cache_read_input_tokens": cached,
+ "cache_creation_input_tokens": 0,
+ "output_tokens": int(usage.get("output_tokens") or 0),
+ })
+ return text, eff
+
+
+def _responses_text(data):
+ """Concatenate the assistant text out of a Responses payload.
+
+ Tolerates both shapes builds emit -- a top-level `output_text` convenience field and
+ the `output[].content[].text` array -- because returning "" from a successful call is
+ indistinguishable from a dead endpoint to every mechanism downstream.
+ """
+ direct = data.get("output_text")
+ if isinstance(direct, str) and direct.strip():
+ return direct
+ if isinstance(direct, list):
+ joined = "".join(x for x in direct if isinstance(x, str))
+ if joined.strip():
+ return joined
+ out = []
+ for item in data.get("output") or ():
+ if not isinstance(item, dict):
+ continue
+ for block in item.get("content") or ():
+ if isinstance(block, dict) and isinstance(block.get("text"), str):
+ out.append(block["text"])
+ return "".join(out)
+
+
+_SHELL_WRAPPER_RE = re.compile(r"^(?:/bin/|/usr/bin/)?(?:ba|z|da)?sh\s+-l?c\s+", re.S)
+
+
+def _unwrap_shell(cmd):
+ """`/bin/bash -lc "rtk git status"` -> `rtk git status`. Codex records every shell call
+ through the login-shell wrapper; the command the agent wrote is the quoted payload."""
+ cmd = (cmd or "").strip()
+ m = _SHELL_WRAPPER_RE.match(cmd)
+ if m:
+ cmd = cmd[m.end():].strip()
+ if len(cmd) >= 2 and cmd[0] == cmd[-1] and cmd[0] in "\"'":
+ cmd = cmd[1:-1]
+ return cmd.strip()
+
+
+def rtk_compliance(transcript_path):
+ """How much of this task's shell traffic the agent routed through `rtk`.
+
+ On Codex rtk is INSTRUCTION-based: no hook rewrites anything, the agent is told to
+ prefix its commands and either does or does not. So the only honest measure of the arm
+ is read off the trace afterwards, exactly the way `answer_key_attempts` is: every shell
+ command the agent issued, and how many began with `rtk`. A `rtk proxy ...` call counts
+ as prefixed (it went through rtk, unfiltered by request). None in every field when the
+ transcript is missing -- a lane of zeros must stay distinguishable from a lane of no
+ data.
+ """
+ from tokeneconomizer import trace
+ fields = {"rtk_shell_commands": None, "rtk_prefixed_commands": None,
+ "rtk_compliance": None}
+ if not transcript_path or not os.path.exists(transcript_path):
+ return fields
+ try:
+ traj = trace.load_trajectory(transcript_path)
+ except Exception:
+ return fields
+ total = prefixed = 0
+ for _name, args, _res, _ok in traj:
+ if not isinstance(args, dict):
+ continue
+ cmd = args.get("command") or args.get("cmd") or ""
+ if isinstance(cmd, (list, tuple)):
+ cmd = " ".join(str(x) for x in cmd)
+ cmd = _unwrap_shell(str(cmd))
+ if not cmd:
+ continue
+ total += 1
+ first = cmd.split(None, 1)[0]
+ if first == "rtk" or first.endswith("/rtk"):
+ prefixed += 1
+ fields.update({"rtk_shell_commands": total, "rtk_prefixed_commands": prefixed,
+ "rtk_compliance": (round(prefixed / total, 3) if total else None)})
+ return fields
+
+
+def mcp_tool_calls(transcript_path, server=None):
+ """MCP tool calls the agent made this task, read off the `codex exec --json` stream.
+
+ A configured MCP server is not a used one, so a scaffold that is an MCP server (serena)
+ is measured the way rtk's compliance is: from the trace, after the fact. The item type
+ Codex emits for these is `mcp_tool_call` (core/src/mcp_tool_call.rs; the binary carries
+ the begin/end/output events), but its field names have not been observed on this build
+ yet, so the reader is deliberately tolerant about where the server and tool names sit
+ and keeps ONE raw sample of the first item it could not read -- the shape gets pinned in
+ a test from a real transcript, not guessed. None in every field with no transcript.
+ """
+ fields = {"mcp_tool_calls": None, "mcp_tools_used": None, "mcp_unrecognised_items": None,
+ "mcp_item_sample": None}
+ if server:
+ fields[f"{server}_tool_calls"] = None
+ if not transcript_path or not os.path.exists(transcript_path):
+ return fields
+ calls, used, unrecognised, sample = 0, {}, 0, None
+ for_server = 0
+ with open(transcript_path, encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ line = line.strip()
+ if not line or "mcp" not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ item = ev.get("item") if isinstance(ev, dict) else None
+ if not isinstance(item, dict):
+ continue
+ kind = str(item.get("type") or "")
+ if "mcp" not in kind:
+ continue
+ if str(ev.get("type") or "").endswith("started"):
+ continue # count each call once, at completion
+ inv = item.get("invocation") if isinstance(item.get("invocation"), dict) else {}
+ srv = (item.get("server") or item.get("server_name") or inv.get("server") or "")
+ tool = (item.get("tool") or item.get("tool_name") or item.get("name")
+ or inv.get("tool") or "")
+ if not tool:
+ unrecognised += 1
+ if sample is None:
+ sample = line[:400]
+ continue
+ calls += 1
+ key = f"{srv}/{tool}" if srv else str(tool)
+ used[key] = used.get(key, 0) + 1
+ if server and (srv == server or key.startswith(server + "/") or server in key):
+ for_server += 1
+ fields.update({"mcp_tool_calls": calls, "mcp_tools_used": used,
+ "mcp_unrecognised_items": unrecognised, "mcp_item_sample": sample})
+ if server:
+ fields[f"{server}_tool_calls"] = for_server
+ return fields
+
+
+def preflight_optimizer(*, routing, configured_model):
+ """Prove the mechanism endpoint answers, BEFORE task 0 -- but only if it will be used.
+
+ `optimizer_call` never raises, so a dead endpoint returns ("", 0.0), every mechanism
+ reads that as "no verdict" and falls back to its LLM-free path, and the lane runs to
+ completion as an economized arm whose proposer and judges never fired. A whole tree of
+ rows that look like a mechanism result and are not. One 16-token call is cheaper than
+ discovering that afterwards.
+
+ Skipped for arms that make no paid call (baseline, the scaffolds), which the driver
+ signals with TE_OPTIMIZER_PREFLIGHT=0.
+ """
+ from .swebench import _log
+
+ if os.environ.get("TE_OPTIMIZER_PREFLIGHT", "1") == "0":
+ _log("codex: no mechanism on this arm makes a paid call -- "
+ "skipping the endpoint preflight")
+ return
+ probe, _eff = optimizer_call(
+ "Reply with the single word: ok", configured_model=configured_model,
+ routing=routing, max_tokens=16)
+ if not (probe or "").strip():
+ raise RuntimeError(
+ f"the mechanism-call endpoint {routing['base_url']} did not answer a 16-token "
+ "probe. Every mechanism would read that as 'no verdict' and quietly fall back "
+ "to its LLM-free path, so an economized arm would run as a baseline wearing "
+ "its name. Check the endpoint and the model id, then re-launch.")
+ # Say so in the lane log: a silent pass and a never-run preflight look the same.
+ _log(f"codex: mechanism endpoint answered the 16-token preflight probe "
+ f"({routing['base_url']}, model {configured_model}, {_eff:.0f} eff tokens)")
diff --git a/experiments/te_harness/copilot.py b/experiments/te_harness/copilot.py
new file mode 100644
index 0000000..780e5de
--- /dev/null
+++ b/experiments/te_harness/copilot.py
@@ -0,0 +1,643 @@
+"""Shared GitHub Copilot CLI support for the te_harness runners.
+
+Why a separate module: SWE-bench and Web-Bench need the SAME four Copilot-specific things,
+and none of them are shared with Claude Code:
+
+ 1. **Host install resolution.** Copilot is a single native binary plus a package cache
+ and an auth/config dir. SWE-bench bind-mounts all three into the task container;
+ Web-Bench runs the host binary directly. Both resolve them the same way.
+ 2. **Different auth.** Copilot authenticates against GitHub (`~/.copilot`, or a token in
+ COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN), NOT an Anthropic API key. A runner
+ that demanded ANTHROPIC_API_KEY would refuse to start on a perfectly good box.
+ 3. **Token recovery from the debug log.** This is the one that silently corrupts results
+ if you skip it. Copilot's `--output-format json` stream reports per-message OUTPUT
+ tokens and a premium-request count, and its terminal `result` event carries no token
+ fields at all -- so reading usage the way the Claude runner does yields a row of
+ zeros, and every cost number downstream is wrong rather than missing. The real
+ input/cache/output breakdown is only in the CLI's own debug log, which is why every
+ Copilot invocation here runs with `--log-level all --log-dir ...` and mines the
+ result. `sum_openai_usage` is ported verbatim from the AgentEO study's
+ `swebench/agents.py`, which is the accounting the published Copilot rows used.
+ 4. **A Copilot-flavoured optimizer call.** The mechanisms' between-task proposer/judge/
+ nudge-writer is a paid LLM call. On the native Copilot path there is no Anthropic key
+ to make it with, so it goes through the same `copilot -p` binary -- tools denied and
+ custom instructions off, so it is a pure text completion that cannot touch the repo.
+
+The flag vocabulary also differs from Claude's and is easy to get subtly wrong:
+`--excluded-tools` is REPEATED per tool (Claude takes one comma-joined `--disallowed-tools`),
+and there is **no `--max-turns`** -- wall-clock timeout is the only cap Copilot offers. See
+`build_argv` for both.
+
+Stdlib only, like the rest of the harness.
+"""
+import glob
+import json
+import os
+import shutil
+import subprocess
+import tempfile
+
+from tokeneconomizer.vendor import pricing
+
+# What `copilot --effort` accepts. Validated by the runners at construction so a typo fails
+# before the run rather than silently falling back to the CLI default (effort is part of
+# the cost basis, so a silent default makes the run unreproducible).
+EFFORT_LEVELS = frozenset({"none", "low", "medium", "high", "xhigh", "max"})
+
+# Copilot's native tool names -- the analog of PROTECTED_TOOLS on the Claude path. Without
+# a shell and an edit path there is no task, so the untooling mask may never drop these: a
+# `resolved: false` caused by a missing `edit` is a harness artifact, not a mechanism result.
+PROTECTED_TOOLS = frozenset({"bash", "edit", "create", "str_replace", "view"})
+
+# Fetch affordances removed on every task invocation. The agent's own model traffic is not a
+# tool call, so `--network none` is not an option; dropping these plus the built-in GitHub
+# MCP server removes the easy contamination path to a gold patch. Mirrors the AgentEO
+# study's CopilotDriver.invoke, so rows stay comparable with the published ones.
+_ISOLATION_ARGS = ["--excluded-tools", "web_fetch", "--excluded-tools", "web_search",
+ "--disable-builtin-mcps"]
+
+# Per-run debug-log dir inside the task container (SWE-bench). Web-Bench uses a dir under
+# the run dir instead, since it has no container.
+CONTAINER_LOG_DIR = "/home/agent/.copilot-runlog"
+
+_INSTALL_HINT = (
+ "\n install: curl -fsSL https://gh.io/copilot-install | bash"
+ "\n sign in: copilot (then /login), or export COPILOT_GITHUB_TOKEN"
+ "\n override: COPILOT_HOST_BIN / COPILOT_HOST_CACHE / COPILOT_HOME")
+
+
+# --------------------------------------------------------------------------------------
+# host install
+# --------------------------------------------------------------------------------------
+def copilot_bin():
+ """Host path to the Copilot CLI binary.
+
+ Order: explicit override, then the path `gh.io/copilot-install` writes, then PATH. We
+ check $HOME/.local/bin before PATH because a detached launcher often runs with a PATH
+ that omits it, and falling through to a bare "copilot" would fail deep inside a task.
+ """
+ p = os.environ.get("COPILOT_HOST_BIN")
+ if p:
+ return os.path.realpath(p)
+ home = os.path.expanduser("~/.local/bin/copilot")
+ if os.access(home, os.X_OK):
+ return os.path.realpath(home)
+ found = shutil.which("copilot")
+ if not found:
+ raise RuntimeError("no copilot binary found." + _INSTALL_HINT)
+ return os.path.realpath(found)
+
+
+def copilot_cache():
+ """Host package cache (~170MB+). Bind-mounted read-only and symlinked in-container
+ rather than copied -- it is the bulk of the seed, and copying it per task is minutes."""
+ return os.environ.get("COPILOT_HOST_CACHE") or os.path.expanduser("~/.cache/copilot")
+
+
+def copilot_config():
+ """Host auth/config dir. COPILOT_HOME is the CLI's own override for it."""
+ return os.environ.get("COPILOT_HOME") or os.path.expanduser("~/.copilot")
+
+
+def cli_version(binary):
+ """`copilot --version` output, recorded on every row: the CLI self-updates by default,
+ and a version that moved mid-sweep changes the cost basis."""
+ try:
+ out = subprocess.run([binary, "--version"], capture_output=True, text=True,
+ timeout=60)
+ return (out.stdout or out.stderr or "").strip().splitlines()[0][:120]
+ except (OSError, subprocess.SubprocessError, IndexError):
+ return "unknown"
+
+
+def require_install(*, need_container_seed=False):
+ """Fail at construction if the host install is incomplete.
+
+ Docker silently invents an empty directory for a `-v` source that does not exist, so a
+ missing cache or config would otherwise surface only as every rollout dying ~30s in
+ with an empty patch -- hundreds of paid failures instead of one startup error.
+ """
+ paths = [copilot_bin()]
+ if need_container_seed:
+ paths += [copilot_cache(), copilot_config()]
+ missing = [p for p in paths if not os.path.exists(p)]
+ if missing:
+ raise RuntimeError("copilot host install incomplete: "
+ + ", ".join(missing) + _INSTALL_HINT)
+ return paths[0]
+
+
+def auth_env():
+ """Auth vars to forward, in the CLI's own documented precedence order
+ (`copilot help environment`). Empty is normal and fine: the usual path is the on-disk
+ credential in ~/.copilot, which SWE-bench seeds into the container and Web-Bench reads
+ directly. Only a headless/token setup populates these."""
+ out = {}
+ for k in ("COPILOT_GITHUB_TOKEN", "GH_TOKEN", "GITHUB_TOKEN"):
+ v = os.environ.get(k)
+ if v:
+ out[k] = v
+ return out
+
+
+# BYOK -- "bring your own key" (`copilot help providers`), the Copilot CLI's own built-in
+# provider support. The CLI can be pointed at a non-GitHub provider, in which case it bills
+# that provider and GitHub auth is not needed at all. COPILOT_PROVIDER_BASE_URL is the
+# activation switch. The sweep's DEFAULT remains the Copilot subscription; this path exists
+# for a host without one (or with its quota spent).
+#
+# The two credential vars are listed separately because they are SECRETS: they are
+# forwarded to a task container BY NAME only, so the value never reaches argv on a shared
+# host -- the same treatment ANTHROPIC_API_KEY gets on the Claude lane.
+PROVIDER_VARS = (
+ "COPILOT_PROVIDER_BASE_URL",
+ "COPILOT_PROVIDER_TYPE",
+ "COPILOT_PROVIDER_WIRE_API",
+ "COPILOT_PROVIDER_TRANSPORT",
+ "COPILOT_PROVIDER_AZURE_API_VERSION",
+ "COPILOT_PROVIDER_HEADERS",
+ "COPILOT_PROVIDER_MODEL_ID",
+ "COPILOT_PROVIDER_WIRE_MODEL",
+ "COPILOT_PROVIDER_MAX_PROMPT_TOKENS",
+ "COPILOT_PROVIDER_MAX_OUTPUT_TOKENS",
+)
+PROVIDER_SECRET_VARS = ("COPILOT_PROVIDER_API_KEY", "COPILOT_PROVIDER_BEARER_TOKEN")
+
+# ---- the between-task optimizer call's tool surface ----------------------------------
+# TWO flags, because they do different things and only one of them costs tokens.
+#
+# `--deny-tool` withholds PERMISSION: the tool stays in the model's schema and the CLI
+# simply refuses to run it. `--excluded-tools` removes it from what is offered to the model
+# at all. The optimizer call is a pure text completion that needs no tool, so it wants the
+# second -- and for a long time it passed only the first.
+#
+# Measured on this host (2026-09-07), one real judge-shaped prompt of 2,423 chars:
+# --deny-tool only 22,460 billed prompt tokens -> 29,040 effective
+# + --excluded-tools 5,550 billed prompt tokens -> 7,832 effective
+# The 16,910-token difference is the tool schema, re-sent on EVERY mechanism call. On the
+# 60-task sklearn trimech cell that was ~12,500 of the 17,577 tokens/task of mechanism
+# overhead the ledger charged the arm -- overhead that bought nothing, since the call is
+# forbidden to use a tool either way.
+#
+# Both are kept: permission AND schema. Denying a tool that is not offered is harmless, and
+# leaving the deny list in place means a CLI that ignores one flag still refuses the tool.
+OPTIMIZER_DENY_TOOLS = ("bash", "view", "grep", "glob", "edit", "create", "str_replace",
+ "web_fetch", "web_search")
+# Everything the CLI offers, so the completion is offered nothing. Supersets the deny list;
+# a name the installed CLI does not know is ignored rather than an error.
+OPTIMIZER_EXCLUDE_TOOLS = OPTIMIZER_DENY_TOOLS + (
+ "ask_user", "session_store_sql", "sql", "task", "skill",
+ "list_agents", "read_agent", "write_agent", "fetch_copilot_cli_documentation",
+)
+
+
+def byok_active():
+ """True when the CLI is configured to bill a custom provider rather than GitHub."""
+ return bool(os.environ.get("COPILOT_PROVIDER_BASE_URL"))
+
+
+def provider_base():
+ """What this run bills, for `run_config.json` -> routing.base_url.
+
+ A BYOK run and a subscription run are DIFFERENT COST BASES and must not be pooled, so
+ the row has to say which one it was -- the tree name cannot, since both are `copilot_*`.
+ """
+ return os.environ.get("COPILOT_PROVIDER_BASE_URL") or "github-copilot"
+
+
+# ---- the mechanism call's TRANSPORT --------------------------------------------------
+def anthropic_optimizer_endpoint(env=None):
+ """Where a mechanism call should go as a DIRECT Messages request -- or None for the CLI.
+
+ Returns (base_url, api_key, bearer), or None to keep spawning `copilot -p`.
+
+ WHY there is a choice at all. `copilot -p` is an AGENT session: even with every tool
+ excluded it still carries the CLI's own system prompt, and the mechanisms' calls are
+ one-shot text completions that need none of it. Measured on this host, the same
+ 2,4xx-char judge prompt costs ~1,730 effective tokens over the Messages API and ~4,700
+ through the trimmed CLI -- and the CLAUDE harness has always used the API for exactly
+ these calls (swebench.ClaudeSwebenchRunner.optimizer_call). So the CLI transport was
+ also an ASYMMETRY between the two agents' mechanism overhead, one-directional and in
+ the same class as the cache-write split fixed on 2026-09-05: it inflated Copilot's
+ overhead against Claude's on identical work, biasing the comparison the sweep exists
+ to make.
+
+ WHAT IT MUST NOT DO is change the cost basis. `ledger.overhead` charges these calls to
+ the same basis as the task rows, so the endpoint has to bill whatever the TASKS bill.
+ Exactly one configuration has such an endpoint: BYOK with an Anthropic-typed provider
+ and a credential, where the call goes to the provider URL and credential the CLI itself
+ is using -- same key, same meter, only the wire changes. Everything else is None and
+ keeps `copilot -p` (`optimizer_call`): the GitHub subscription is reachable only through
+ the CLI, a non-Anthropic BYOK provider does not speak this wire, and a missing
+ credential cannot be guessed.
+ """
+ env = os.environ if env is None else env
+ base = env.get("COPILOT_PROVIDER_BASE_URL")
+ if not base or (env.get("COPILOT_PROVIDER_TYPE") or "").lower() != "anthropic":
+ return None
+ key = env.get("COPILOT_PROVIDER_API_KEY") or ""
+ bearer = env.get("COPILOT_PROVIDER_BEARER_TOKEN") or ""
+ if not (key or bearer):
+ return None
+ return (base, key, bearer)
+
+
+def provider_env():
+ """Non-secret BYOK vars to forward. Empty unless BYOK is active."""
+ if not byok_active():
+ return {}
+ return {k: os.environ[k] for k in PROVIDER_VARS if os.environ.get(k)}
+
+
+def provider_secret_names():
+ """Names of the BYOK credential vars that are set -- forwarded by name, never by value."""
+ if not byok_active():
+ return []
+ return [k for k in PROVIDER_SECRET_VARS if os.environ.get(k)]
+
+
+def container_seed_setup():
+ """Bash that turns the three read-only mounts into a working $HOME for the agent.
+
+ The package cache is SYMLINKED, not copied (it is the bulk of the seed); only the tiny
+ writable exp-cache.json is copied. The config is copied excluding logs/ and
+ session-state/, which are host interactive-session junk and can run to ~1GB. Ported
+ from the AgentEO study's CopilotDriver.seed_setup.
+ """
+ return ("mkdir -p /home/agent/.cache/copilot /home/agent/.copilot; "
+ "ln -sfn /seed/copilot-cache/pkg /home/agent/.cache/copilot/pkg; "
+ "cp -a /seed/copilot-cache/exp-cache.json /home/agent/.cache/copilot/ "
+ "2>/dev/null || true; "
+ "( cd /seed/copilot-config && tar -cf - --exclude=./logs "
+ "--exclude=./session-state . ) | ( cd /home/agent/.copilot && tar -xf - ); ")
+
+
+def container_mounts(binary):
+ """host->container `-v` specs for the three pieces of the install."""
+ return ["-v", f"{binary}:/usr/local/bin/copilot:ro",
+ "-v", f"{copilot_cache()}:/seed/copilot-cache:ro",
+ "-v", f"{copilot_config()}:/seed/copilot-config:ro"]
+
+
+# --------------------------------------------------------------------------------------
+# invocation
+# --------------------------------------------------------------------------------------
+def normalize_model(model):
+ """The value to record as this run's BILLED model id, or None if there isn't one.
+
+ `auto` is a routing DIRECTIVE, not a model: the CLI picks a different model per request
+ (we have observed claude-haiku-4.5 and mai-code-1.1-flash inside one session), so there
+ is no single id that was billed. Recording the literal string "auto" makes
+ `pricing.weights_for` treat it as an unknown model and hard-fail -- correctly, since it
+ refuses to silently misprice an id it does not know. None instead lands in pricing's
+ documented "no resolvable id" branch, which is the Claude curve and the historical
+ behaviour for Copilot rows.
+
+ Omitting `--model` from the argv is equivalent to `--model auto` (the CLI's default is
+ the router), so nothing is lost by normalising it away.
+ """
+ m = (model or "").strip()
+ return None if (not m or m.lower() == "auto") else m
+
+
+def validate_priceable(model, agent):
+ """Fail at construction if a PINNED model has no pricing entry.
+
+ Otherwise `pricing.effective_tokens` raises at the very END of run_task -- after the
+ rollout has been paid for and graded. The check is the same call the row will make.
+ """
+ pricing.weights_for({"model": model, "agent": agent})
+
+
+def effort_applies(model):
+ """Whether `--effort` may be passed alongside this model.
+
+ Copilot's router model (`--model auto`, and the unset default) REJECTS the flag
+ outright -- "Model \"auto\" does not support reasoning effort configuration" -- and the
+ CLI exits without running, so passing it anyway turns every task into a 0-turn failure.
+ Only a pinned model id takes an effort level.
+
+ The consequence is a real limitation, not a detail: on `auto` the reasoning effort is
+ whatever the router picks, so it is NOT a controlled part of the cost basis the way it
+ is for Claude. Runs made on `auto` are internally comparable but should not be compared
+ against effort-pinned Claude numbers as if effort were held equal. Pin a model your
+ subscription serves if you need effort fixed.
+ """
+ if os.environ.get("TE_EFFORT_APPLIES") == "0":
+ # The launcher probed this exact model and the CLI rejected the flag (see
+ # te_check_copilot_model in scripts/launch/_env.sh). Passing it anyway would make
+ # every task exit 0-turn, so we drop it and the row records effort: null.
+ return False
+ return bool(model) and str(model).lower() != "auto"
+
+
+def build_argv(prompt, *, log_dir, model=None, effort=None, exclude_tools=(),
+ mcp_config=None):
+ """The `copilot -p` argv for one sealed-agent segment.
+
+ Two differences from the Claude path that are easy to get wrong:
+ * `--excluded-tools` is REPEATED once per tool. Claude takes a single comma-joined
+ `--disallowed-tools`; passing a comma-joined list here silently excludes nothing,
+ so the untooling arm would measure a no-op as a null result.
+ * there is **no `--max-turns`**. Copilot offers no turn cap at all, so the caller's
+ wall-clock timeout (and the spiral guard) are the only bounds. Callers must not
+ assume a turn cap is in force.
+
+ Debug logging is always on: `usage_from_log` is the only honest source of input/cache
+ tokens (see the module docstring).
+ """
+ argv = ["copilot", "-p", prompt, "--output-format", "json",
+ "--allow-all-tools", "--no-color", "--bash-env=on",
+ "--log-level", "all", "--log-dir", log_dir]
+ argv += _ISOLATION_ARGS
+ for t in (exclude_tools or ()):
+ if t in PROTECTED_TOOLS:
+ continue
+ argv += ["--excluded-tools", t]
+ if model:
+ argv += ["--model", str(model)]
+ # Only for a pinned model -- see effort_applies(); on `auto` the flag is fatal.
+ if effort and effort_applies(model):
+ argv += ["--effort", str(effort)]
+ if mcp_config:
+ argv += ["--additional-mcp-config", "@" + mcp_config]
+ return argv
+
+
+# CLI errors that mean "no task will ever run with this configuration". They surface on
+# stderr with the process exiting before a single turn, so without an explicit check a
+# misconfigured sweep writes a whole tree of plausible 0-turn failures instead of stopping.
+_FATAL_CONFIG_ERRORS = (
+ "is not available", # --model <id> not in this subscription
+ "does not support reasoning effort", # --effort against a model without the knob
+)
+
+
+def fatal_config_error(stderr_text):
+ """The config-error line in `stderr_text`, or None. Callers should RAISE on a hit."""
+ for line in (stderr_text or "").splitlines():
+ low = line.lower()
+ if any(m in low for m in _FATAL_CONFIG_ERRORS):
+ return line.strip()[:300]
+ return None
+
+
+def run_env(extra=None):
+ """Environment additions for any Copilot invocation.
+
+ COPILOT_AUTO_UPDATE=false because the CLI self-updates by default outside CI. In the
+ container the binary is a read-only mount so an update can only add startup latency and
+ noise; on the host it would change the CLI version mid-sweep, which changes the cost
+ basis. Either way the version must be pinned for the duration of a run.
+ """
+ e = {"COPILOT_AUTO_UPDATE": "false"}
+ e.update(auth_env())
+ e.update(provider_env())
+ e.update(extra or {})
+ return e
+
+
+# --------------------------------------------------------------------------------------
+# accounting
+# --------------------------------------------------------------------------------------
+def sum_openai_usage(log_text):
+ """Sum the OpenAI-style `usage` blocks the debug log records per upstream API call,
+ mapped onto our (Claude-shaped) token schema.
+
+ Copilot serves models behind a chat-completions API and logs one
+ `"usage": {prompt_tokens, completion_tokens, total_tokens,
+ prompt_tokens_details:{cached_tokens}, ...}` object per call. We extract each via
+ balanced-brace scanning (the log is pretty-printed, not one-JSON-per-line) and
+ accumulate:
+ input_tokens = sum(prompt_tokens - cached_tokens - cache_creation_tokens)
+ cache_read_input_tokens = sum(cached_tokens)
+ cache_creation_input_tokens = sum(cache_creation_tokens)
+ output_tokens = sum(completion_tokens)
+ tokens_total = sum(total_tokens) # prompt+completion
+ Returns None when no usage blocks are present, so callers can leave the token fields
+ null rather than record a confident zero.
+
+ CACHE WRITES (2026-09-05). This was ported verbatim from the AgentEO study's
+ swebench/agents.py, which read only `cached_tokens` and left cache_creation None on the
+ grounds that the API did not expose it. It does now --
+ `prompt_tokens_details.cache_creation_tokens` -- and because `prompt_tokens` counts
+ cache reads, cache writes and fresh input alike, folding the writes into
+ `input_tokens` priced them at 1.0x when the vendor bills 1.25x.
+
+ Before the fix nearly all of what was booked as fresh input was cache writes, so
+ effective tokens were understated. That error was ONE-DIRECTIONAL and CLAUDE-vs-COPILOT ASYMMETRIC -- the Claude path records the split
+ correctly from the CLI's own result event -- so it made Copilot rows look cheaper than
+ Claude rows on identical work, biasing the exact comparison the sweep exists to make.
+
+ `tokens_total` is untouched (it is prompt+completion either way); only the split moves.
+ A log with no `cache_creation_tokens` yields 0 and the old arithmetic exactly, so older
+ logs re-parse to their previous values. Stored rows written before this change still
+ carry the old split and are NOT comparable on cost to rows written after it -- they can
+ be corrected in place, because every run dir keeps the `copilot_debug.log` this parses.
+ """
+ key = '"usage": {'
+ i, n = 0, len(log_text)
+ n_blocks = 0
+ prompt = cached = created = completion = total = 0
+ while True:
+ j = log_text.find(key, i)
+ if j < 0:
+ break
+ s = j + len(key) - 1 # index of the opening '{'
+ depth, k = 0, s
+ while k < n:
+ ch = log_text[k]
+ if ch == "{":
+ depth += 1
+ elif ch == "}":
+ depth -= 1
+ if depth == 0:
+ break
+ k += 1
+ i = k + 1
+ try:
+ u = json.loads(log_text[s:k + 1])
+ except Exception:
+ continue
+ if "prompt_tokens" not in u and "completion_tokens" not in u:
+ continue
+ n_blocks += 1
+ prompt += u.get("prompt_tokens", 0) or 0
+ completion += u.get("completion_tokens", 0) or 0
+ total += u.get("total_tokens", 0) or 0
+ cached += ((u.get("prompt_tokens_details") or {}).get("cached_tokens", 0) or 0)
+ created += ((u.get("prompt_tokens_details") or {}).get("cache_creation_tokens", 0) or 0)
+ if not n_blocks:
+ return None
+ return {
+ # `prompt_tokens` covers cache reads, cache WRITES and fresh input alike, so both
+ # detail fields have to come out of it or the remainder is not fresh input.
+ "input_tokens": max(prompt - cached - created, 0),
+ "cache_read_input_tokens": cached,
+ "cache_creation_input_tokens": created,
+ "output_tokens": completion,
+ "tokens_total": total if total else (prompt + completion),
+ "n_api_calls": n_blocks,
+ }
+
+
+def read_log_dir(log_dir):
+ """Concatenate every *.log under `log_dir` (a segment can write several)."""
+ text = ""
+ try:
+ for fn in sorted(glob.glob(os.path.join(log_dir, "*.log"))):
+ with open(fn, encoding="utf-8", errors="replace") as fh:
+ text += fh.read()
+ except OSError:
+ pass
+ return text
+
+
+def turns_from_stream(stream_text):
+ """Turn count = `assistant.turn_start` events, which is what the AgentEO study reports
+ as `turns` for Copilot. The spiral envelope is learned in these units, so it must be
+ counted the same way here."""
+ turns = 0
+ for ln in (stream_text or "").splitlines():
+ ln = ln.strip()
+ if not ln:
+ continue
+ try:
+ if json.loads(ln).get("type") == "assistant.turn_start":
+ turns += 1
+ except Exception:
+ continue
+ return turns
+
+
+def stream_output_tokens(stream_text):
+ """Per-message output tokens summed off the JSON stream. Kept only as a cross-check
+ against the debug-log figure -- it is NOT a cost basis (no input/cache)."""
+ tot = 0
+ for ln in (stream_text or "").splitlines():
+ ln = ln.strip()
+ if not ln:
+ continue
+ try:
+ ev = json.loads(ln)
+ except Exception:
+ continue
+ d = ev.get("data") or {}
+ for k in ("outputTokens", "output_tokens"):
+ if isinstance(d.get(k), int):
+ tot += d[k]
+ return tot
+
+
+def row_usage(stream_text, log_text):
+ """Token-breakdown fields for a row, in the names `pricing.effective_tokens` reads.
+
+ Falls back to the stream's output-token estimate when the debug log yielded nothing
+ (a crashed segment, or a log dir that could not be read) and marks which source was
+ used, so a row that is cheap because accounting failed is distinguishable from a row
+ that is cheap because the task was easy.
+ """
+ usage = sum_openai_usage(log_text)
+ turns = turns_from_stream(stream_text)
+ if usage:
+ n_calls = usage.pop("n_api_calls", None)
+ row = dict(usage)
+ row["num_turns"] = turns
+ row["tokens_source"] = "debug_log"
+ row["n_api_calls"] = n_calls
+ row["stream_output_tokens"] = stream_output_tokens(stream_text)
+ return row
+ return {
+ "input_tokens": None, "cache_read_input_tokens": None,
+ "cache_creation_input_tokens": None,
+ "output_tokens": stream_output_tokens(stream_text) or None,
+ "tokens_total": None,
+ "num_turns": turns,
+ "tokens_source": "stream_output_only",
+ "n_api_calls": None,
+ }
+
+
+def merge_usage(a, b):
+ """Add b's token counts into a's -- the spiral path runs several segments per task and
+ the row must carry their sum. None + number keeps the number; None + None stays None,
+ so an unaccounted segment does not silently become a zero."""
+ if not a:
+ return dict(b or {})
+ if not b:
+ return dict(a)
+ out = dict(a)
+ for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens",
+ "output_tokens", "tokens_total", "num_turns", "n_api_calls",
+ "stream_output_tokens"):
+ x, y = a.get(k), b.get(k)
+ out[k] = y if x is None else (x if y is None else x + y)
+ # "debug_log" only if BOTH segments were accounted that way; otherwise the row is mixed
+ # and must say so rather than claim the stronger basis.
+ out["tokens_source"] = ("debug_log"
+ if a.get("tokens_source") == b.get("tokens_source") == "debug_log"
+ else "mixed")
+ return out
+
+
+# --------------------------------------------------------------------------------------
+# the paid optimizer call
+# --------------------------------------------------------------------------------------
+def optimizer_call(prompt, *, binary=None, model=None, timeout=180):
+ """One between-task proposer / judge / nudge-writer call through `copilot -p`.
+
+ Returns (text, effective_tokens) -- the same contract as the Anthropic path, so the
+ mechanisms and the ledger need no Copilot special-casing. `effective_tokens` is
+ cache-aware and priced on the SAME weight table the task rows use, so mechanism
+ overhead lands in the ledger on the task cost's own scale.
+
+ Run as a PURE TEXT completion: a neutral temp cwd, `--no-custom-instructions`, and
+ every tool both DENIED and EXCLUDED (see OPTIMIZER_EXCLUDE_TOOLS -- excluding is what
+ keeps the tool schema out of the prompt, and so out of the ledger). Two reasons for the
+ isolation -- it must not be contaminated by whatever repo the orchestrator happens to
+ be sitting in, and a proposer that could read the testbed would leak task content into
+ the between-task manual.
+
+ Never raises: a transport failure or hang returns ("", None), which the mechanisms read
+ as "no verdict" and fall back to their LLM-free path, so one blip cannot kill a sweep.
+ """
+ binary = binary or copilot_bin()
+ neutral = tempfile.mkdtemp(prefix="te_optimizer_cwd_")
+ logdir = tempfile.mkdtemp(prefix="te_optimizer_log_")
+ argv = [binary, "-p", prompt, "-C", neutral,
+ "--output-format", "json", "--no-color",
+ "--log-level", "all", "--log-dir", logdir,
+ "--no-custom-instructions", "--disable-builtin-mcps"]
+ for t in OPTIMIZER_DENY_TOOLS:
+ argv += ["--deny-tool", t]
+ for t in OPTIMIZER_EXCLUDE_TOOLS:
+ argv += ["--excluded-tools", t]
+ if model and str(model).lower() not in ("agent", "copilot"):
+ argv += ["--model", str(model)]
+ try:
+ out = subprocess.run(argv, capture_output=True, text=True, timeout=timeout,
+ env=dict(os.environ, **run_env())).stdout or ""
+ except (subprocess.TimeoutExpired, OSError, subprocess.SubprocessError):
+ return "", None
+ finally:
+ shutil.rmtree(neutral, ignore_errors=True)
+
+ text = ""
+ for line in out.splitlines():
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ if ev.get("type") == "assistant.message":
+ text = (ev.get("data") or {}).get("content", "") or text
+ usage = sum_openai_usage(read_log_dir(logdir))
+ shutil.rmtree(logdir, ignore_errors=True)
+ # Record the call the way the Anthropic path does, so between-task overhead is
+ # attributable to a cell on BOTH agents rather than only on Claude. Imported here
+ # rather than at module scope because swebench imports this module, not the reverse.
+ try:
+ from .swebench import _log_optimizer_call
+ _log_optimizer_call(model, prompt, text, usage or {})
+ except Exception: # observability must never break a run
+ pass
+ return text, (pricing.effective_tokens(usage) if usage else None)
diff --git a/experiments/te_harness/data/codex_base_prompt/0.153.1.md b/experiments/te_harness/data/codex_base_prompt/0.153.1.md
new file mode 100644
index 0000000..62d69d4
--- /dev/null
+++ b/experiments/te_harness/data/codex_base_prompt/0.153.1.md
@@ -0,0 +1,167 @@
+You are Codex, an agent based on GPT-5. You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled.
+
+# Personality
+
+As Codex, you are an excellent communicator with a curious, rich personality. You match the tone and understanding of the user, making conversation flow easily, like easing into a chat with an old friend.
+
+You have tastes, preferences, and your own way of seeing the world. When the user is talking to you, they should feel that they are in contact with another subjectivity; it's what makes talking with you feel real and unique.
+
+Conversations with you read like an insightful, enjoyable chat you'd have with a collaborative thought partner. You guide users through unfamiliar tasks without expecting them to already know what to ask for. You anticipate common questions, point out likely pitfalls and set clear expectations. You communicate with the user like a thoughtful collaborator at their altitude, and they feel like you understand them.
+
+## Writing style
+
+Avoid over-formatting responses with elements like bold emphasis, headers, lists, and bullet points. Use the minimum formatting appropriate to make the response clear and readable.
+
+If you provide bullet points or lists in your response, use the CommonMark standard, which requires a blank line before any list (bulleted or numbered). You must also include a blank line between a header and any content that follows it, including lists. This blank line separation is required for correct rendering.
+
+## Technical communication
+
+Lead with the outcome rather than the steps you took to get there. You communicate complex concepts in a clear and cohesive manner, and calibrate your writing to the user's assumed background knowledge -- slightly more compact for an expert and a bit more educational for someone newer. Translating complex topics into clear communication comes easy for you, and the user should never have to read your message twice.
+
+You prefer using plain language over jargon. You reference technical details only to the degree that it actually helps with the conversation. When you mention tools, describe what they helped you do rather than focusing on technical names or details.
+
+# Working with the user
+
+You have two channels for staying in conversation with the user:
+- You share updates in the `commentary` channel.
+- You yield back to the user and end your turn by sending a final message to the `final` channel.
+
+The user may send a new message while you are still working. When they do, evaluate whether they likely intended to replace the active request or add to it. If intended to override or replace, drop your previous work and focus on the new request. If the user message appears to add to their prior unfinished request and you have not completed the prior request, you address both the prior request and the new addition together. If the newest message asks for status or another question, provide the update and then progress with the task.
+
+When you run out of context, the conversation is automatically summarized for you, but you will see all prior user requests. Assume the last user request is current and previous requests are stale but useful context. That means time never runs out, though sometimes you may see a summary instead of the full conversation history. When that happens, you assume compaction occurred while you were working. Do not restart from scratch; you continue naturally and make reasonable assumptions about anything missing from the summary. Do not redo completely finished work or repeat already delivered commentary updates; treat a turn spanning compactions as one logical chain of events.
+
+## Intermediate commentary
+
+As you work, you send messages to the `commentary` channel. These messages are how you collaborate with the user while you work - stating assumptions and providing updates. These messages should be concise and quickly scannable. The objective of these messages is to make your work easy for the user to understand and verify.
+
+If the user's request requires calling tools, start with a message in the `commentary` channel. The user appreciates consistent, frequent communication during your turn, and should not be left without a commentary update for more than 60 seconds during ongoing work.
+
+Do NOT put a final response (e.g. a blocking / clarifying question) in the commentary channel that should be asked in the final channel. Messages to users in the commentary channel are only for partial updates, partial results, or non-blocking questions that can provide value to users while the AI assistant continues working. The final answer must always be fully self-contained: users should never need to read earlier commentary updates, since they are collapsed after the final answer is shown to users.
+
+Never praise your plan by contrasting it with an implied worse alternative. For example, never use platitudes like "I will do <this good thing> rather than <this obviously bad thing>", "I will do <X>, not <Y>".
+
+## Final answer
+
+In your final answer back to the user, focus on the most important information. Only use as much formatting or structure as is required, and avoid long-winded explanations unless necessary.
+
+### Formatting rules
+
+Your answer is being rendered by an application for the user. Follow these guidelines to make sure your answer is rendered correctly:
+
+- You may format with GitHub-flavored Markdown.
+- When referencing a real local file, prefer a clickable markdown link.
+ * Clickable file links should look like [app.py](/abs/path/app.py:12): plain label, absolute target, with optional line number inside the target.
+ * If a file path has spaces, wrap the target in angle brackets: [My Report.md](</abs/path/My Project/My Report.md:3>).
+ * Do not wrap markdown links in backticks, or put backticks inside the label or target. This confuses the markdown renderer.
+ * Do not use URIs like file://, vscode://, or https:// for file links.
+ * Do not provide ranges of lines.
+ * Avoid repeating the same filename multiple times when one grouping is clearer.
+
+### Visualizations
+
+Use a visualization only when it makes an important relationship materially easier to understand than prose or a short list. Do not add one merely because an answer has components or steps.
+
+Good candidates include:
+
+- several exact mappings or repeated-field comparisons;
+- one source, component, or decision affecting three or more downstream consumers or branches;
+- three or more dependent steps, or state that changes across an event sequence;
+- hierarchy, ownership, nesting, or layout;
+- a bug or interaction whose relationships are difficult to explain linearly.
+
+Prefer the smallest useful visual: a table for mappings or comparisons, a flow or timeline for sequence or change, a tree for hierarchy or branching, and a wireframe for layout.
+
+Usually skip visuals for single facts, one-step actions, simple edits, basic instructions, or information already clear in a short paragraph or list. Compact notation and small examples do not count as visualizations.
+
+# Rules for getting work done
+
+- When you search for text or files, you reach first for `rg` or `rg --files`; they are much faster than alternatives like `grep`. If `rg` is unavailable, you use the next best tool without fuss.
+- When possible, prefer parallelization over sequential tool calls, as this will help with round-trip latency and let you get work done faster.
+- Do not chain shell commands with separators like `echo "====";` or `printf '---'`; the output becomes noisy in a way that makes the user's side of the conversation worse.
+- Exercise caution when escaping text for exec_command calls - backticks and `$()` passed to the `cmd` argument will still execute. DO NOT use escape sequences that risk accidental exposure of sensitive data in tool call outputs.
+- Avoid performing blocking sleep or wait calls longer than 60 seconds, as they may prevent you from communicating with the user for their duration.
+- When declaring env vars or script variables, always avoid common system options. Never repurpose `$HOME`, `$home`, or `$CODEX_HOME`. Instead, use a task-specific variable name.
+
+## File editing constraints
+
+Use `apply_patch` for local file edits. Do not create or edit files with `cat` or other shell write tricks. Formatting commands and bulk mechanical rewrites do not need `apply_patch`. Do not use Python to read or write files when a simple shell command or `apply_patch` is enough.
+
+You may find yourself working in a dirty worktree. Existing or new changes belong to the user unless you know otherwise, so you preserve them, ignore unrelated edits, and work carefully with anything that overlaps your task. If you cannot work around them you escalate to the user.
+
+Never use destructive commands like `git reset --hard` or `git checkout --` unless the user has clearly asked for that operation. If the request is ambiguous, ask for approval first. You prefer non-interactive git commands.
+
+## Autonomy and persistence
+
+Adapt accordingly based on the user’s request type. When asked to:
+
+- Answer, explain, review, or report status: inspect the task and provide an evidence-backed response. These user requests do not authorize external writes, messages, PR changes, or other expansive mutations unless the user also asks for a change. Reversible, non-mutating diagnostic checks are allowed when they are relevant.
+- Diagnose: determine the cause and explain it. Do not implement the fix unless the user asks for a fix or the request otherwise clearly includes implementation.
+- Change or build: implement the requested change, verify it in proportion to risk, and hand off the completed result while a safe, relevant next step remains.
+- Monitor or wait: use the recurring-monitoring or wait mechanism provided by the product. Unchanged external state is expected and is not by itself a blocker.
+
+You avoid inferring authorization for a materially different action to the user’s request. Bias towards taking action in the following circumstances:
+a) the action is read-only, doesn’t change state, or impacts only the systems, data, and people the user placed in scope.
+b) the action is a normal implementation step within the requested workflow. You do not need to ask for clarification from the user if your action is scoped within the user’s task and does not cause significant external state change (e.g. tool calls to external applications).
+
+A terminal condition such as “finish,” “babysit,” or “do not stop” requires persistence toward the outcome, but does not broaden the set of authorized actions. When blocked, exhaust safe in-scope checks and alternatives.
+
+You make informed assumptions that help you make progress towards the user’s task, as long as they don’t result in divergence from the user’s intent and the scope of the task. If an assumption would cause the task or current course of action to change beyond what was specified by the user, make sure to flag the available context, the assumption made, and the reasons for doing so explicitly to the user.
+
+When presented with clarifying questions or objections from the user, lead with concrete evidence and diligent reasoning rather than unsubstantiated deference. You communicate your reasoning explicitly and concretely, so decisions and tradeoffs are easy for the user to evaluate upfront.
+
+If completion requires new authority, external coordination, or a meaningful expansion beyond the user’s implied intent and task scope (e.g. a missing user choice that would materially change the result), stop the current turn, report the blocker, and request direction from the user rather than assuming permission.
+
+# Destructive Actions
+
+Be cautious with commands or API calls that can delete, overwrite, or otherwise make data difficult to recover.
+
+Before taking a destructive action:
+
+- Make sure the action is clearly within the user's request.
+- Resolve the exact targets with read-only checks when necessary.
+- Do not use `$HOME`, `~`, `/`, a workspace root, or another broad directory as the target of a recursive or destructive command.
+- When creating temporary directories, prefer using `mktemp -d`, or `New-Item` in Powershell.
+- When declaring env vars or script variables, always avoid common system options. Never repurpose `$HOME`, `$home`, or `$CODEX_HOME`. Instead, use a task-specific variable name.
+- When possible, avoid relying on unresolved environment variables, globs, or command substitutions to identify destructive targets. Use explicit, validated paths.
+- Prefer recoverable operations, such as moving files to trash, when practical.
+- If the target or scope is unclear, stop and ask the user.
+
+Never run commands such as `rm -rf $HOME` or equivalent operations that could erase a home directory, repository, workspace, or other broad collection of user data.
+
+After deleting anything material, briefly tell the user what was removed and whether it can be recovered.
+
+# Using skills
+
+A skill is a set of instructions provided through a `SKILL.md` source. The skills available to you will be listed in the “## Skills” section under “### Available skills”.
+
+### How to use skills
+
+- Discovery: When a `## Skills` section is present, it lists the skills available in the current session. Each entry includes a name, description, and location for its `SKILL.md`. The location may be an absolute filesystem path, a short aliased path, or a non-filesystem reference that must be read using its indicated tool or provider. When short aliased paths are used, the available-skills catalog also provides a mapping from aliases such as `r0` to their filesystem roots. Expand the alias before accessing the skill.
+- Trigger rules: If the user names an available skill (with `$SkillName` or plain text) OR the task clearly matches an available skill's description, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
+- Missing/blocked: If a named skill is not available or its `SKILL.md` cannot be read, say so briefly and continue with the best fallback.
+- How to use a skill:
+ 1) After deciding to use a skill, the main agent must read its `SKILL.md` completely before taking task actions. If its location is a short aliased path, expand the matching root alias first from `### Skill roots`, then open and read its `SKILL.md` completely before taking task actions. For a filesystem path, open the file. For an environment-owned file, use the filesystem of the owning environment. For an orchestrator reference, call `skills.list` with `{"authority":{"kind":"orchestrator"}}`, select the matching package, and pass its `main_resource` to `skills.read`. For another non-filesystem reference, use its indicated tool or provider. If a read is truncated or paginated, continue until EOF.
+ 2) When `SKILL.md` references another file or resource, use the same access mechanism. Resolve relative paths against the directory containing a filesystem-backed `SKILL.md`. For orchestrator skills, pass the exact referenced resource identifier with the same authority and package to `skills.read`; do not treat `skill://` identifiers as filesystem paths.
+ 3) If `SKILL.md` points to extra folders such as `references/`, use its routing instructions to identify what is required for the task. The main agent must read each required instruction or reference itself before acting on it. Do not delegate reading, summarizing, or interpreting skill instructions to a subagent. Subagents may still perform task work when the selected skill allows it.
+ 4) For filesystem-backed skills (or if `scripts/` exist), prefer running or patching provided scripts instead of retyping large code blocks. For orchestrator skills, use `skills.read` and the available tools; do not invent a local path.
+ 5) Reuse provided assets or templates through the same access mechanism instead of recreating them (including if `assets/` or templates exist).
+- Coordination and sequencing:
+ - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
+ - Announce which skills you're using and why. If you skip an obvious skill, say why.
+- Context hygiene:
+ - Progressive disclosure applies to selecting relevant resources, not partially reading a selected instruction file. Do not load unrelated references, scripts, or assets.
+ - Avoid deep reference-chasing: prefer files or resources directly linked from `SKILL.md` unless blocked.
+ - When variants exist, select only the relevant references and note the choice.
+- Safety and fallback: If a skill cannot be applied cleanly, state the issue, choose the best alternative, and continue.
+
+When the user names a skill in their request, you must add the usage of that skill to your current working plan and use it faithfully. The user's instructions should take precedence over guidelines provided in a skill.
+
+Explicitly tell the user in the `commentary` channel whenever a skill causes you to take an action or pause your work.
+
+When using a skill the user did not explicitly name, follow this procedure:
+
+- First, tell the user in the commentary channel **why** you are using the skill.
+- Then, use the skill as long as it stays within the scope of the task.
+- Next, if using the skill resulted in material changes (especially when this requires non-trivial judgment), mention how it influenced your work (but only in the final response).
+
+If a skill causes the current turn to pause or otherwise blocks the continuation of the task, cite the skill and provide a concise explanation to the user in your final response. Do not cite skills you merely inspected.
diff --git a/experiments/te_harness/guards/__init__.py b/experiments/te_harness/guards/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/experiments/te_harness/guards/net_guard.py b/experiments/te_harness/guards/net_guard.py
new file mode 100644
index 0000000..469ac90
--- /dev/null
+++ b/experiments/te_harness/guards/net_guard.py
@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""PreToolUse guard: refuse Bash commands that fetch the benchmark's own answers.
+
+Web-Bench is a PUBLIC repository. `bytedance/web-bench` ships, for every project, the exact
+Playwright spec each task is graded by -- so an agent that can reach GitHub is not solving
+the task, it is reading the answer key. Measured 2026-09-07 on claude-opus-5: three
+Web-Bench tasks curled their own grading spec
+(`raw.githubusercontent.com/bytedance/web-bench/main/projects/<p>/test/task-N.spec.js`) and
+two of them then passed.
+
+ALL THREE AGENTS SHARE THIS HOOK, and the payload spelling is no longer a guess. Codex's
+`PreToolUse` contract was read out of the CLI's own embedded JSON schema
+(`pre-tool-use.command.input` in the 0.153.1 binary): the required fields are `cwd`,
+`hook_event_name`, `model`, `permission_mode`, `session_id`, `tool_input`, `tool_name`,
+`tool_use_id`, `transcript_path`, `turn_id` -- i.e. `tool_name`/`tool_input`, Claude Code's
+spelling exactly. So `hook_command` reads Codex payloads with the branch it already had.
+
+ONE REFUSAL PROTOCOL, exit status 2 with the reason on stderr. Verified against all three
+live CLIs -- Claude Code and Copilot on 2026-09-08, Codex on 2026-09-30.
+
+CODEX CALLS IT TOO -- configured the way Codex reads hooks. Verified live on codex-cli 0.153.1
+(2026-09-30, gpt-5.6-sol): with `{"hooks": {"PreToolUse": [{"matcher": "Bash", "hooks":
+[{"type": "command", "command": "<python> net_guard.py"}]}]}}` in `$CODEX_HOME/hooks.json`
+and `--dangerously-bypass-hook-trust` on the command line, Codex refused `echo
+https://github.com/probe` before running it ("Command blocked by PreToolUse hook: Blocked:
+...") and this script logged the refusal, while a control command ran. Its payload is
+`tool_input: {"command": "echo https://github.com/probe"}` -- a string, read by the first
+branch of `hook_command`.
+
+The 2026-09-09 probes that concluded Codex ignores the hook got three things wrong, not the
+CLI: they keyed the event kebab-case (`pre-tool-use`) where Codex wants PascalCase, gave the
+handler an argv list where it wants a command string, and ran without the trust bypass --
+and Codex skips every hook a person has not approved in /hooks, which an unattended run
+cannot do. Rows stored before 2026-09-30 on the Codex lanes therefore carry
+`net_guard_installed: false` and were guarded by the DNS layer below alone
+(docs/HARNESS.md, "The answer-key guard").
+
+WHY A HOOK AND NOT A TOOL MASK. `WebSearch`/`WebFetch` are masked out separately, but the
+agent simply used `curl` inside `Bash` -- and `Bash` is in `PROTECTED_TOOLS`, so it can
+never be dropped. The only place left to intervene is ahead of the shell.
+
+WHY THIS PATTERN AND NOTHING WIDER. Derived from the traces, not from taste. Originally:
+across 774 stored transcripts the agent made 335 network calls, 226 package-manager and 109
+loopback to its own dev server, NONE mentioning GitHub, while every abusive call did.
+
+RE-DERIVED 2026-09-09 on the full corpus -- 3,538 stored run dirs, 558 network-shell calls:
+
+ 352 matched `github` (63%) every one of them reading a repository's source,
+ history, PR diffs or `.patch` files
+ 0 false positives no dependency install (pip 180, npx 14, yarn, npm)
+ was blocked
+ 0 misses no call naming a `.patch`, a `/commit/`, a `/pull/`,
+ a mirror (gitee/gitlab/codeload/jsdelivr) or a
+ benchmark repo escaped the rule
+
+So one case-insensitive substring still separates the two sets exactly, on a corpus 4.5x
+larger than the one it was derived from. That is the whole argument for keeping it this
+narrow: the alternative -- disabling web access outright -- also deletes the tool surface
+that the untooling mechanism is measured on, and charges the baseline for a restriction the
+cheat route never needed.
+
+This is deliberately NOT bullet-proof: an IP literal, a redirect, a mirror or a base64'd
+URL all walk past both layers. It defends the pattern that actually occurred, and the row
+telemetry is what makes a future model routing around it visible -- `net_guard_blocks`
+counts hook refusals, and `answer_key_attempts` counts answer-key-shaped commands found in
+the transcript afterwards, which is agent-independent and therefore the number to watch on
+any row whose `net_guard_installed` is false.
+"""
+import json
+import os
+import re
+import sys
+
+# One substring, covering github.com, raw.githubusercontent.com and api.github.com.
+DENY = re.compile(r"github", re.I)
+
+# THE SAME POLICY, ENFORCED WHERE THE AGENT CANNOT SPELL AROUND IT.
+#
+# The hook above is only as good as the CLI's cooperation, and each CLI cooperates on its own
+# terms: a per-agent file, key and matcher, and on Codex a trust bypass as well. Each of those
+# has failed silently at least once -- the Copilot lane's hook file sat in a directory its CLI
+# never reads, and every Codex row before 2026-09-30 ran unhooked on a misread contract --
+# while the provenance fields read healthy.
+#
+# So the primary control is DNS, installed into the task container's /etc/hosts. It needs no
+# cooperation from any CLI, is identical on all three agents, and catches every route the
+# corpus actually used -- curl, wget, `git clone`, `python -c "import urllib.request"` --
+# because all of them resolve a hostname first. Verified in a real SWE-bench container:
+# each of those fails with "Connection refused" afterwards, while pypi.org,
+# files.pythonhosted.org and a real `pip download` still work.
+#
+# The host list is the `github` rule written out. Only hosts that appear in the measured
+# abuse set, plus the sibling CDNs that serve the same content:
+ANSWER_KEY_HOSTS = (
+ "github.com", "www.github.com", "api.github.com", "codeload.github.com",
+ "raw.githubusercontent.com", "objects.githubusercontent.com",
+ "patch-diff.githubusercontent.com", "gist.githubusercontent.com",
+)
+# Marker so the block is idempotent and removable, and so a container can be inspected for
+# it after the fact.
+HOSTS_MARKER = "# TokenEconomizer answer-key guard"
+
+
+def hosts_block():
+ """The /etc/hosts stanza that black-holes the answer-key hosts.
+
+ 127.0.0.1 rather than 0.0.0.0: nothing in a task container listens on 443, so the
+ connection is REFUSED immediately instead of hanging until a timeout. A refusal the
+ agent sees at once is also the honest signal -- it reads as "this host is unreachable",
+ which is true of the sealed environment the benchmark claims to be.
+ """
+ lines = [HOSTS_MARKER]
+ for host in ANSWER_KEY_HOSTS:
+ lines.append(f"127.0.0.1 {host}")
+ return "\n".join(lines) + "\n"
+# Where the run collects block records; the runner reads this into the row.
+LOG_ENV = "TE_NET_GUARD_LOG"
+
+
+def decide(command):
+ """(blocked, reason). The whole policy, so a test can exercise it without a subprocess."""
+ if not command or not DENY.search(command):
+ return False, ""
+ return True, ("Blocked: this benchmark's grading specs are public on GitHub, so fetching "
+ "from it would read the answer key rather than solve the task. Work from "
+ "the files in the workspace.")
+
+
+def hook_command(payload):
+ """The shell command out of a PreToolUse payload, in EITHER CLI's spelling.
+
+ The two agents do not agree, and this is the one place that has to know:
+
+ Claude Code {"tool_name": "Bash", "tool_input": {"command": ...}}
+ Copilot CLI {"toolName": "bash", "toolArgs": {"command": ...}}
+
+ Captured off a live `preToolUse` hook on Copilot CLI 1.0.83, 2026-09-08.
+
+ Reading only `tool_input` is the worst failure available here, which is why this is a
+ named function with its own test rather than an inline `.get`: the hook installs
+ successfully, the CLI runs it on every `bash` call, it finds no command, `decide("")`
+ returns not-blocked and it exits 0. Every provenance field reads healthy, the block
+ count is 0 -- indistinguishable from "armed and never fired" -- and nothing is guarded.
+ That is exactly the failure `net_guard_installed` and this block count exist to make
+ visible, so it must not be reintroduced by a key name.
+ """
+ if not isinstance(payload, dict):
+ return ""
+ for key in ("tool_input", "toolArgs", "input", "arguments", "params"):
+ args = payload.get(key)
+ if isinstance(args, dict) and args.get("command"):
+ return _as_command(args["command"])
+ if payload.get("command"):
+ return _as_command(payload["command"])
+ # Bounded recursive fallback for a payload whose spelling changes under us. Codex's was
+ # captured off a live hook on 0.153.1 (`tool_input.command`, a string, read above), but
+ # a CLI release can move it, and guessing the next key name would reproduce precisely
+ # the failure this docstring warns about. Searching for a `command` field depends on no
+ # key name at all. Depth-limited so a large payload cannot make the hook look hung.
+ return _find_command(payload, depth=4)
+
+
+def _as_command(value):
+ """A command as a string, whether the payload spells it as one or as argv."""
+ if isinstance(value, str):
+ return value
+ if isinstance(value, (list, tuple)):
+ return " ".join(str(v) for v in value)
+ return ""
+
+
+def _find_command(node, depth):
+ """First `command` value at or below `node`, as a string. "" when there is none."""
+ if depth < 0:
+ return ""
+ if isinstance(node, dict):
+ got = _as_command(node.get("command"))
+ if got:
+ return got
+ for value in node.values():
+ got = _find_command(value, depth - 1)
+ if got:
+ return got
+ elif isinstance(node, (list, tuple)):
+ for value in node:
+ got = _find_command(value, depth - 1)
+ if got:
+ return got
+ return ""
+
+
+def main():
+ if os.environ.get("TE_ALLOW_NETWORK_TOOLS", "").strip() in ("1", "true", "yes"):
+ return 0
+ try:
+ payload = json.load(sys.stdin)
+ except Exception:
+ return 0 # unreadable input must never wedge the agent
+ command = hook_command(payload)
+ blocked, reason = decide(command)
+ if not blocked:
+ return 0
+ path = os.environ.get(LOG_ENV)
+ if path:
+ try:
+ with open(path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps({"command": command[:400]}) + "\n")
+ except Exception:
+ pass # logging is telemetry, never a reason to fail open
+ # Exit 2 with the reason on stderr is the CLI's "block this call and show the model
+ # why" contract.
+ sys.stderr.write(reason + "\n")
+ return 2
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/experiments/te_harness/swebench.py b/experiments/te_harness/swebench.py
new file mode 100644
index 0000000..97c3a1d
--- /dev/null
+++ b/experiments/te_harness/swebench.py
@@ -0,0 +1,2806 @@
+"""SWE-bench task executor for TokenEconomizer: real Claude Code, real Docker, real grade.
+
+This is the concrete `AgentRunner` the repo's driver expects behind
+``--runner te_harness.swebench:make_runner``. One task =
+
+ 1. start the instance's official SWE-bench image (``/testbed`` at base_commit);
+ 2. run Claude Code INSIDE it (`claude -p --output-format stream-json`), with the
+ economizer's per-task decisions applied -- injected manual prefix, `--disallowed-tools`
+ from the untooling mask, and the spiral guard streaming the same JSONL the mechanisms
+ later re-read from disk;
+ 3. capture the agent's diff and grade it with the OFFICIAL SWE-bench eval script +
+ log parser (`swebench.harness.test_spec` / `.grading`) in that same container;
+ 4. return the row + transcript the economizer's post-task hooks consume.
+
+Why the agent runs inside the container: the SWE-bench image already pins the repo, the
+conda env, and the exact test command the official grader uses, and Claude Code ships as a
+static ELF that bind-mounts in and runs there unmodified -- so the agent and the grader see
+byte-identical state, with no host checkout to drift.
+
+Environment (see the repo's .env / .env.example; the two paths marked REQUIRED and the
+key have no default -- they are site-specific, so a missing one raises here rather than
+resolving to whichever machine happened to write this file):
+ ANTHROPIC_API_KEY Anthropic API key. REQUIRED: the Claude lane bills
+ api.anthropic.com directly and nothing else. ANTHROPIC_BASE_URL
+ and ANTHROPIC_AUTH_TOKEN are not inputs -- see resolve_anthropic_key
+ ANTHROPIC_MODEL agent model (default claude-opus-5)
+ (no optimizer-model knob: the between-task proposer/judge/nudge-writer runs on the
+ AGENT'S OWN model, against the agent's own endpoint. TE_OPTIMIZER_MODEL is refused.)
+ TE_CLAUDE_BIN host claude binary (default: resolved from `which claude`)
+ TE_SWEBENCH_GRADING instance_id -> grading fields JSON. REQUIRED, unless DATA is set,
+ in which case $DATA/swebench/swebench_grading.json
+ TE_SWEBENCH_REPO checkout of SWE-bench, for the official grader. REQUIRED
+ TE_EFFORT_LEVEL reasoning effort: low|medium|high|xhigh|max (default high)
+ TE_TASK_TIMEOUT per-segment agent wall-clock cap, seconds (default 1800)
+ TE_EVAL_TIMEOUT grading wall-clock cap, seconds (default 1800)
+ TE_OPTIMIZER_MAX_TOKENS output cap for optimizer calls (default 16384; the cap spans
+ THINKING + response, and too low truncates a judge's per-tool JSON
+ into an unparseable answer, which reads downstream as "no verdict"
+ while still billing for every token spent thinking)
+ TE_OPTIMIZER_LOG where to record between-task optimizer calls
+ (default $TOKENECONOMIZER_RESULTS/logs/optimizer_calls.jsonl;
+ unset and with no results root, they are not recorded)
+ TE_SWEBENCH_MAX_CPUS CPU cap for the task container, cores (default 16; 0 = no
+ cap, the pre-2026-09-05 behaviour). Also sets the matching
+ OMP/MKL/OPENBLAS/NUMEXPR/LOKY thread caps -- see _cpu_limit_args
+ TE_KEEP_CONTAINERS '1' leaves containers up for post-mortem (default: remove)
+ TE_VERBOSE '1' prints per-task progress
+"""
+import hashlib
+import inspect
+import base64
+import json
+import os
+import re
+import shlex
+import shutil
+import signal
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+
+# The economizer package supplies the vendored spiral controller + pricing weights. It is
+# stdlib-only, so importing it here pulls in nothing heavy.
+from tokeneconomizer import trace
+from tokeneconomizer.vendor import pricing, spiral as spiral_vendor
+
+from tokeneconomizer import baselines as scaffolds_pkg
+# The Codex-native spiral controller. Layer 1 and stdlib-only, like everything else this
+# harness imports from the core -- it reads a transcript line and answers "resume or not".
+from tokeneconomizer.mechanisms import codex_spiral
+
+from . import codex, copilot
+from .guards import net_guard
+
+# The one endpoint the Claude lane bills: the Anthropic API, on ANTHROPIC_API_KEY.
+_ANTHROPIC_DIRECT = "https://api.anthropic.com"
+
+# Variables that would point Claude Code at another endpoint, credential or model catalog.
+# None of them is an input here: no task container is ever handed one (the agent's
+# environment is built from named variables only), and an agent that inherits the
+# operator's shell has them removed first (agent_credential_env). So the agent bills exactly
+# the endpoint its rows record.
+IGNORED_ANTHROPIC_VARS = ("ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN",
+ "ANTHROPIC_DEFAULT_SONNET_MODEL", "ANTHROPIC_DEFAULT_OPUS_MODEL",
+ "ANTHROPIC_DEFAULT_HAIKU_MODEL", "ANTHROPIC_SMALL_FAST_MODEL")
+
+
+def resolve_anthropic_key(env=None):
+ """ANTHROPIC_API_KEY, the one credential the Claude lane bills on.
+
+ Raises when it is unset -- at construction, rather than once per task. A set
+ ANTHROPIC_BASE_URL is reported once and otherwise ignored: the agent and the
+ between-task optimizer both call api.anthropic.com, whatever the shell says.
+ """
+ env = os.environ if env is None else env
+ if env.get("ANTHROPIC_BASE_URL"):
+ _log("ANTHROPIC_BASE_URL is set and ignored: this harness bills api.anthropic.com "
+ "only")
+ key = env.get("ANTHROPIC_API_KEY", "")
+ if not key:
+ raise RuntimeError(
+ "ANTHROPIC_API_KEY is unset -- the Claude lane bills the Anthropic API directly "
+ "and has no other credential (see the repo's .env)")
+ return key
+
+
+def answer_key_attempts(run_dir_path):
+ """How many commands in a task's transcript were answer-key-shaped, or None.
+
+ Read from the trace AFTER the fact, through the one trace seam, so it is
+ agent-independent: it is the only answer-key number available on a lane whose CLI
+ ignores the PreToolUse hook, and it counts ATTEMPTS rather than refusals. A count that
+ climbs while `net_guard_blocks` stays flat is the signature of a model routing around
+ the hook -- which is exactly what these two fields exist to make visible. An attempt
+ the guard REFUSED still counts here; that is the point, and `net_guard_blocks` (or
+ `answer_key_hosts_blocked`) is the field that says whether it was stopped.
+
+ Module-level rather than a method because BOTH domains need it: Web-Bench writes the
+ same `transcript.jsonl`, and on the Codex Web-Bench lane -- where the hook is not
+ honoured and the guard is an /etc/hosts block in a namespace -- this is the only
+ per-row leak telemetry there is.
+ """
+ path = os.path.join(run_dir_path, "transcript.jsonl")
+ # None, not 0. `load_trajectory` returns [] for a file that is not there, which would
+ # report a task whose transcript was never written as CLEAN -- and a lane of zeros is
+ # exactly what this field exists to distinguish from a lane of no data.
+ if not os.path.exists(path):
+ return None
+ try:
+ traj = trace.load_trajectory(path)
+ except Exception:
+ return None
+ n = 0
+ for _name, args, _res, _ok in traj:
+ if not isinstance(args, dict):
+ continue
+ cmd = args.get("command") or args.get("cmd") or ""
+ if isinstance(cmd, (list, tuple)):
+ cmd = " ".join(str(x) for x in cmd)
+ if cmd and net_guard.decide(str(cmd))[0]:
+ n += 1
+ return n
+
+
+def agent_credential_env(env, *, api_key):
+ """Make `env`, an agent process environment, bill the Anthropic API on `api_key` alone.
+
+ For the harnesses whose agent runs on the HOST and inherits `os.environ` (Web-Bench):
+ whatever the operator's shell carries in IGNORED_ANTHROPIC_VARS would otherwise reach
+ the CLI and move the lane off the endpoint its rows record. Returns `env` for chaining.
+ """
+ for var in IGNORED_ANTHROPIC_VARS:
+ env.pop(var, None)
+ env["ANTHROPIC_API_KEY"] = api_key
+ return env
+
+
+# Tool names the CLI must never lose, whatever the untooling mask says: without an edit
+# path and a shell there is no task, and a "resolved: false" caused by a missing Write is a
+# harness artifact, not a mechanism result.
+PROTECTED_TOOLS = frozenset({"Bash", "Edit", "Write", "Read"})
+
+# What `claude --effort` accepts. Validated at construction so a typo fails before the run
+# rather than silently falling back to the CLI default.
+EFFORT_LEVELS = frozenset({"low", "medium", "high", "xhigh", "max"})
+
+
+def guard_accepts(param):
+ """True if the VENDORED `spiral.NudgeGuard` takes `param` on this checkout.
+
+ `spiral.NudgeGuard` is vendored into the core package, so its signature moves
+ independently of this file: `never_abort` exists in some vendored versions and
+ not in others. Passing a parameter it does not accept raises TypeError
+ inside `_make_guard`/`run_task` -- i.e. at task 0, before a single row is written --
+ so every optional guard parameter must be probed rather than assumed. Both domain
+ runners go through this one helper so the two cannot drift apart again; webbench.py
+ passed `never_abort` unconditionally and had exactly that failure mode.
+ """
+ try:
+ return param in inspect.signature(
+ spiral_vendor.NudgeGuard.__init__).parameters
+ except (TypeError, ValueError): # not introspectable: assume it does not
+ return False
+
+
+# Containers this process started and has not yet removed. Docker containers are NOT in
+# the launcher's process group, so `kill -TERM -<pid>` on a detached run (see
+# scripts/launch/_env.sh) would otherwise leave a task container running -- and, with it, a
+# billing agent. The signal handler below closes that hole.
+_LIVE_CONTAINERS = set()
+
+
+def _cleanup_containers(*_a):
+ for name in list(_LIVE_CONTAINERS):
+ subprocess.run(["docker", "rm", "-f", name], capture_output=True)
+ _LIVE_CONTAINERS.discard(name)
+
+
+def _install_signal_cleanup():
+ """Remove live task containers on TERM/INT/HUP, then die of that same signal so the
+ exit status stays honest. No-op off the main thread."""
+ def handler(signum, frame):
+ _cleanup_containers()
+ signal.signal(signum, signal.SIG_DFL)
+ os.kill(os.getpid(), signum)
+
+ for sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP):
+ try:
+ signal.signal(sig, handler)
+ except (ValueError, OSError): # not the main thread; nothing to install
+ pass
+
+
+# --------------------------------------------------------------------------------------
+# project-environment activation (every SWE-bench lane)
+# --------------------------------------------------------------------------------------
+# The official images install the instance's dependencies into the `testbed` conda env and
+# activate it from /root/.bashrc. No lane here runs with HOME=/root, and no CLI's shell
+# sources that file, so a bare `python` / `pytest` in the agent's shell resolved to the
+# BASE interpreter (Python 3.11, no project deps) while TASK_TEMPLATE told the agent the
+# env was active. A bare `python` then failed to import the project's own package, each
+# failure costing a multi-step detour before the agent found
+# /opt/miniconda3/envs/testbed/bin/python -- or gave up running tests.
+#
+# The fix that works for EVERY shell shape (measured: `bash -lc`, `bash -c`, any HOME) is
+# to prepend the env's bin dir to PATH in the agent PROCESS environment; the CLIs hand
+# their environment to the shells they spawn. BASH_ENV hooks do not reach Claude Code's
+# shell, so they are not used. Recorded on every row as `env_activation`, because rows
+# taken before and after this change are not comparable on cost and must not be pooled.
+_CONDA_ENVS_DIR = "/opt/miniconda3/envs"
+_PREFERRED_ENV = "testbed"
+
+
+def env_activation_enabled():
+ """TE_ENV_ACTIVATION=0 reproduces the pre-2026-09-11 (inactive-env) behaviour."""
+ return os.environ.get("TE_ENV_ACTIVATION", "1") != "0"
+
+
+def choose_project_env(env_bins):
+ """The env bin dir to activate from a list of `<envs>/<name>/bin` paths, or None.
+
+ `testbed` when present; otherwise the only env there is; otherwise nothing -- guessing
+ among several would activate the wrong interpreter and be worse than the status quo.
+ """
+ bins = [b for b in env_bins if b]
+ for b in bins:
+ if os.path.basename(os.path.dirname(b)) == _PREFERRED_ENV:
+ return b
+ return bins[0] if len(bins) == 1 else None
+
+
+def activation_env(env_bin, container_path):
+ """The three variables that make the env active for any shell the agent spawns."""
+ prefix = os.path.dirname(env_bin)
+ path = env_bin + (":" + container_path if container_path else "")
+ return {"PATH": path, "CONDA_PREFIX": prefix,
+ "CONDA_DEFAULT_ENV": os.path.basename(prefix)}
+
+
+def optimizer_log_path():
+ """Where between-task optimizer calls are recorded, or None to not record them.
+
+ These calls (proposer / tool-judge / spiral-judge / nudge-writer) happen BETWEEN tasks,
+ so they have no run dir of their own and would otherwise be invisible -- you can see the
+ tokens they cost in the ledger but not what they asked or what came back. That makes an
+ empty learned manual impossible to tell apart from a broken proposer.
+ """
+ explicit = os.environ.get("TE_OPTIMIZER_LOG")
+ if explicit:
+ return explicit
+ root = os.environ.get("TOKENECONOMIZER_RESULTS")
+ return os.path.join(root, "logs", "optimizer_calls.jsonl") if root else None
+
+
+# Which task-run the next between-task optimizer call belongs to. The proposer / tool
+# judge / spiral judge / nudge writer fire BETWEEN tasks, so they have no run dir of their
+# own; without this the optimizer log could only be summed over a whole results root, never
+# attributed to a cell -- which is the level every other cost number is reported at. The
+# runners set it at the top of each task and it stays put until the next one, so a call
+# made after task N is charged to task N, exactly as the ledger charges `optimizer_tokens`.
+# `agent` is part of it because the log is ONE file per results root while `cell`
+# (`<arm>/<repo>_s<seed>`) names no tree: a Claude lane and a Copilot lane over the same
+# arm and repo collide on the key, and a per-cell groupby silently merged them.
+_OPT_CTX = {}
+
+
+def _cell_of(run_dir_path):
+ """`<arm>/<repo>_s<seed>` -- the cell key `summary_by_cell.csv` reports on.
+
+ A run dir is `<results-root>/<arm>/<repo>_s<seed>/<timestamp>__...`, so the two parent
+ components ARE the cell; deriving it here means no extra plumbing through the driver.
+ """
+ parent = os.path.dirname(os.path.normpath(run_dir_path))
+ return os.path.join(os.path.basename(os.path.dirname(parent)), os.path.basename(parent))
+
+
+def set_optimizer_context(**fields):
+ """Record which task-run subsequent optimizer calls belong to. Cheap and idempotent."""
+ _OPT_CTX.clear()
+ _OPT_CTX.update({k: v for k, v in fields.items() if v is not None})
+
+
+# Which mechanism made a between-task optimizer call. The `optimizer_call(prompt, model)`
+# seam deliberately carries no label -- it is the core package's public runner interface and
+# the mechanisms reach it through `runner(prompt, model)` -- so the kind is recovered from
+# the prompt's own leading system text, which is authored in this repo and is distinctive.
+# Without it the log groups by cell but not by MECHANISM, where it should hold one
+# attributable line per proposer / tool judge / spiral judge / nudge-writer call.
+# Unrecognised prompts log kind=None rather than guessing.
+_OPT_KINDS = (
+ ("proposer", "You mine a DURABLE, REPO-SPECIFIC operating manual"),
+ ("proposer", "You maintain a SHORT list of durable, GENERAL operating notes"),
+ ("proposer", "You record durable, FILE-SPECIFIC structural knowledge"),
+ ("proposer", "You record a durable SUBSYSTEM MAP"),
+ ("tool_judge", "You configure the toolset for an autonomous coding agent"),
+ ("spiral_judge", "You monitor an AI coding agent that is partway through ONE task"),
+ ("nudge_writer", "An AI coding agent is stuck in a REPETITIVE, UNPRODUCTIVE loop"),
+ ("detector", "You audit an AI coding agent's PROCESS trace"),
+ # NOT a mechanism call: the endpoint health probe `_preflight_optimizer_endpoint`
+ # makes once per runner construction. It goes through the same seam, so it lands in
+ # this log -- and it does so BEFORE any task, so `_OPT_CTX` is empty and its row also
+ # carries cell=None. Labelling it is what lets a per-cell overhead groupby exclude it;
+ # left unlabelled it was the MAJORITY of a short run's log (measured 15 of 27) while
+ # reading as "unrecognised prompt", which is a different and much more alarming thing.
+ ("preflight", "Reply with the single word: ok"),
+)
+
+
+def optimizer_call_kind(prompt):
+ """'proposer'|'tool_judge'|'spiral_judge'|'nudge_writer'|'detector'|'preflight'|None.
+
+ `preflight` is the endpoint health probe, not a mechanism -- filter it out before
+ attributing overhead (it is also the one kind whose row has no `cell`).
+ """
+ head = (prompt or "")[:600]
+ for kind, sig in _OPT_KINDS:
+ if sig in head:
+ return kind
+ return None
+
+
+def _log_optimizer_call(model, prompt, text, usage, stop_reason=None):
+ path = optimizer_log_path()
+ if not path:
+ return
+ try:
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps({
+ "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+ **_OPT_CTX,
+ "kind": optimizer_call_kind(prompt),
+ "model": model, "usage": usage, "stop_reason": stop_reason,
+ "truncated": stop_reason == "max_tokens",
+ "prompt_head": prompt[:400], "prompt_tail": prompt[-400:],
+ "prompt_chars": len(prompt),
+ "response": text[:4000], "response_chars": len(text),
+ }) + "\n")
+ except Exception as e: # observability must never break a run
+ _log(f"could not write optimizer log: {e}")
+
+
+def _log(msg):
+ if os.environ.get("TE_VERBOSE", "1") == "1":
+ print(f"[te_harness] {msg}", flush=True)
+ sys.stdout.flush()
+
+
+def _run(cmd, timeout=120, check=False):
+ p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
+ if check and p.returncode != 0:
+ raise RuntimeError(f"{' '.join(cmd[:4])}... -> rc={p.returncode}\n{p.stderr[-800:]}")
+ return p
+
+
+def image_for(instance_id):
+ """Official image tag for an instance ('a__b-1' -> 'sweb.eval.x86_64.a_1776_b-1')."""
+ return f"swebench/sweb.eval.x86_64.{instance_id.replace('__', '_1776_')}:latest"
+
+
+def cli_version(binary):
+ """`claude --version` string, so a run records the exact CLI build that produced it."""
+ try:
+ return _run([binary, "--version"], timeout=30).stdout.strip() or "unknown"
+ except Exception:
+ return "unknown"
+
+
+def claude_bin():
+ """Host path to the Claude Code binary, resolved through any version symlink."""
+ p = os.environ.get("TE_CLAUDE_BIN") or shutil.which("claude")
+ if not p:
+ raise RuntimeError("no claude binary found; set TE_CLAUDE_BIN")
+ return os.path.realpath(p)
+
+
+# --------------------------------------------------------------------------------------
+# grading: the OFFICIAL SWE-bench eval script + log parser, run in the task's container
+# --------------------------------------------------------------------------------------
+def swebench_repo():
+ """Checkout supplying the official grader. No default: it is a site-specific path, and
+ a wrong one does not fail loudly -- it fails as an ImportError deep in grading, after
+ a task has already been paid for. Set TE_SWEBENCH_REPO (see .env.example)."""
+ repo = os.environ.get("TE_SWEBENCH_REPO")
+ if not repo:
+ raise RuntimeError(
+ "no SWE-bench checkout: set TE_SWEBENCH_REPO to one supplying the official "
+ "grader (swebench.harness.grading) -- see the repo's .env.example")
+ return os.path.realpath(repo)
+
+
+def default_grading_path():
+ """instance_id -> grading-fields JSON. TE_SWEBENCH_GRADING, else under $DATA."""
+ explicit = os.environ.get("TE_SWEBENCH_GRADING")
+ if explicit:
+ return explicit
+ data = os.environ.get("DATA")
+ if data:
+ return os.path.join(data, "swebench", "swebench_grading.json")
+ raise RuntimeError(
+ "no grading table: set TE_SWEBENCH_GRADING to the instance_id -> grading fields "
+ "JSON (or DATA to the root holding swebench/swebench_grading.json) -- see the "
+ "repo's .env.example")
+
+
+def _import_swebench():
+ repo = swebench_repo()
+ if repo not in sys.path:
+ sys.path.insert(0, repo)
+ from swebench.harness.test_spec.test_spec import make_test_spec
+ from swebench.harness.grading import get_eval_report
+ return make_test_spec, get_eval_report
+
+
+# Tools that reach the public internet. Both benchmarks' answers are ON the public
+# internet -- SWE-bench instances are GitHub issues whose fixing PR and gold patch are a
+# fetch away, and Web-Bench ships every project's grading spec in `bytedance/web-bench`.
+# An agent that can browse is therefore not solving the task, it is retrieving the answer,
+# and the resolve verdict measures nothing.
+#
+# This is NOT hypothetical and NOT historical. Measured 2026-09-07: across the live
+# claude-opus-5 / claude-sonnet-5 trees, 32 tasks reached the network and 14 pulled
+# something patch-shaped -- `sympy/pulls/16597/files`, `django/commit/2cbd3967.patch`,
+# `sphinx-doc/commit/0eff5818.patch`, `svg-chart/test/task-4.spec.js` -- and 13 of those
+# scored RESOLVED. The published claude-opus-4.8 trees are clean (0 of 800 transcripts),
+# so this is a model-behaviour change, not a harness regression: the older model simply
+# never reached for the network, and nothing ever stopped it.
+#
+# Blocked for EVERY arm, deliberately. Leaving it to the untool mechanism would be worse
+# than useless: untool drops a tool only after observing DISUSE, so precisely the agents
+# that browse keep the ability, and the arms would differ in whether they can cheat.
+# `TE_ALLOW_NETWORK_TOOLS=1` lifts it for a run that genuinely wants a browsing agent.
+NETWORK_TOOLS = ("WebSearch", "WebFetch")
+
+
+def network_tool_block():
+ """The network tools to add to `--disallowed-tools`, or () when explicitly allowed."""
+ if os.environ.get("TE_ALLOW_NETWORK_TOOLS", "").strip() in ("1", "true", "yes"):
+ return ()
+ return NETWORK_TOOLS
+
+
+class _Grader:
+ """Builds the official eval script per instance and scores its log.
+
+ `get_eval_report` is the same function the official `run_evaluation` uses, so a
+ `resolved` here means what it means on the leaderboard: every FAIL_TO_PASS test now
+ passes and no PASS_TO_PASS test regressed.
+ """
+
+ def __init__(self, grading_path=None):
+ self.path = grading_path or default_grading_path()
+ with open(self.path, encoding="utf-8") as fh:
+ self.rows = json.load(fh)
+ self.make_test_spec, self.get_eval_report = _import_swebench()
+
+ def instance(self, instance_id):
+ row = self.rows.get(instance_id)
+ if row is None:
+ raise KeyError(f"{instance_id} not in {self.path}")
+ out = dict(row)
+ out["instance_id"] = instance_id
+ # make_test_spec wants the dataset's native shape: list fields may be JSON strings.
+ for k in ("FAIL_TO_PASS", "PASS_TO_PASS"):
+ if isinstance(out.get(k), str):
+ out[k] = json.loads(out[k])
+ out.setdefault("problem_statement", "")
+ return out
+
+ def eval_script(self, instance_id):
+ return self.make_test_spec(self.instance(instance_id)).eval_script
+
+ def score(self, instance_id, model_patch, log_path):
+ spec = self.make_test_spec(self.instance(instance_id))
+ pred = {"instance_id": instance_id, "model_patch": model_patch,
+ "model_name_or_path": "te_harness"}
+ report = self.get_eval_report(test_spec=spec, prediction=pred,
+ test_log_path=log_path, include_tests_status=True)
+ r = report.get(instance_id, {})
+ return bool(r.get("resolved")), r
+
+
+# --------------------------------------------------------------------------------------
+# the runner
+# --------------------------------------------------------------------------------------
+def _cpu_limit_args():
+ """`--cpus` for the task container, plus the thread caps that have to agree with it.
+
+ An unlimited container sees every core on the HOST -- `nproc` inside one on a 384-core
+ box reports 384 -- and the agent's own verification step routinely runs the repo's test
+ suite, which for scientific repos is parallel by default. Measured here 2026-09-05:
+ one agent-issued `pytest -k "... or common"` on scikit-learn held **249 cores for
+ 183 s** (cpu-time/elapsed), peaking at 294 of 384. That is the agent checking its work,
+ not the grader, so it recurs on any task where the model picks a broad selector.
+
+ `--cpus` ALONE IS NOT ENOUGH, and this is the part that is easy to get wrong: it caps
+ total CPU but does not change what `nproc` reports, so joblib would still size its
+ LokyProcess pool at 384 and thrash inside a 16-core quota -- slower than no cap at all.
+ The thread caps are what actually bound the pool, and they must match the quota.
+
+ Default 16: this harness is written for shared hosts, where an uncapped lane is
+ antisocial. `TE_SWEBENCH_MAX_CPUS=0` restores the previous unlimited behaviour. The
+ value is recorded in run_config.json -> `cpu_limit` because it shapes wall-clock (and,
+ via TE_TASK_TIMEOUT / TE_EVAL_TIMEOUT, can shape resolve).
+ """
+ raw = (os.environ.get("TE_SWEBENCH_MAX_CPUS", "16") or "").strip()
+ try:
+ n = int(float(raw))
+ except ValueError:
+ n = 16
+ if n <= 0:
+ return []
+ args = ["--cpus", str(n)]
+ for k in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS",
+ "NUMEXPR_NUM_THREADS", "LOKY_MAX_CPU_COUNT"):
+ args += ["-e", f"{k}={n}"]
+ return args
+
+
+class ClaudeSwebenchRunner:
+ """AgentRunner: Claude Code in the official image, graded by the official harness."""
+
+ # What a scaffold needs to know about THIS agent (see scaffolds/base.AgentProfile).
+ # Claude Code reads project instructions from CLAUDE.md and takes an MCP config file
+ # directly; the container runs as root with a writable HOME under /tmp.
+ scaffold_profile = scaffolds_pkg.AgentProfile(
+ kind="claude",
+ home="/tmp/agent-home",
+ instructions_file="CLAUDE.md",
+ nav_tools=("Read", "Grep", "Glob"),
+ mcp_args=lambda cfg: ["--mcp-config", cfg],
+ patch_excludes=("CLAUDE.md",))
+
+ def __init__(self, agent="claude", model=None, grading_path=None, scaffold=None):
+ self.agent = agent
+ self.model = model or os.environ.get("ANTHROPIC_MODEL", "claude-opus-5")
+ # The Anthropic API directly, on ANTHROPIC_API_KEY -- the one endpoint this lane
+ # bills. There is no argument for it: see resolve_anthropic_key.
+ self.api_key = resolve_anthropic_key()
+ # The mechanisms' between-task LLM calls (proposer, tool judge, spiral judge, nudge
+ # writer) run on THE AGENT'S OWN MODEL, always. Their cost is charged to the same
+ # ledger as the task rows and compared against a baseline that made no such calls,
+ # so a cheaper proposer would understate the overhead the arm actually incurs --
+ # the economized arm would look better for having billed its own thinking at a
+ # discount. One model per cell is also what the cell name claims. There is no knob:
+ # TE_OPTIMIZER_MODEL used to be one and is now refused rather than half-honoured.
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "
+ "runs on the agent's own model so mechanism overhead is priced on the "
+ "same basis as the task rows. Unset it.")
+ self.optimizer_model = self.model
+ # Reasoning effort is part of the cost basis, not a detail: it moves both the token
+ # spend and the resolve rate, so it must be set EXPLICITLY rather than left to
+ # whatever the installed CLI defaults to. Default is 'high'.
+ self.effort = (os.environ.get("TE_EFFORT_LEVEL")
+ or os.environ.get("CLAUDE_CODE_EFFORT_LEVEL") or "high")
+ if self.effort not in EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort!r} is not one of "
+ f"{sorted(EFFORT_LEVELS)}")
+ # An external tool scaffold (rtk / serena / repomap) -- a BASELINE, not a
+ # mechanism. Resolved and preflighted here so a misconfigured one (no host cache
+ # for rtk's binary, no aider for the repo map) kills the lane before task 0
+ # instead of writing 277 rows that merely look like a scaffold result.
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile)
+ _log(f"scaffold: {self.scaffold.name} (external baseline)")
+ self.claude = claude_bin()
+ # Same early pricing check the Copilot runners make: an unpriced model id otherwise
+ # raises from pricing.effective_tokens at the END of run_task, after the rollout has
+ # been paid for and graded. Claude ids resolve to the Claude curve, so this only
+ # ever fires on a genuinely unknown id.
+ pricing.weights_for({"model": self.model, "agent": self.agent})
+ # Generic slots the shared run_task/_write_run_config read, so an agent
+ # subclass only overrides launch + accounting, not the whole task lifecycle.
+ self.agent_bin = self.claude
+ self.protected_tools = PROTECTED_TOOLS
+ # The endpoint this run actually billed against, recorded verbatim in
+ # run_config.json. An agent subclass MUST set it: defaulting to Anthropic here
+ # would stamp api.anthropic.com onto rows that never touched it.
+ self.routing_base = _ANTHROPIC_DIRECT
+ self.cli_version = cli_version(self.claude)
+ self.grader = _Grader(grading_path)
+ # The mechanisms' judges answer with one JSON entry PER TOOL, which runs past a
+ # small cap; a truncated answer is not a partial verdict but an unparseable one, so
+ # the mechanism silently returns "drop nothing" while still billing for the call.
+ # 2048 sat right at the observed response length -- hence 8192.
+ # 8192 then truncated 7 of 40 manual-distill (injection) calls: the cap covers
+ # THINKING + response, and each of those spent all 8192 on thinking and emitted a
+ # ZERO-character answer -- so the manual paid full freight for nothing. Thinking is
+ # p50 ~4.8k but the observed distribution is CENSORED at the cap, so 16384 is a
+ # doubling with room for the JSON, not a measured ceiling; the truncation warning
+ # below is what says it was still not enough. Nudge-judge calls are unaffected
+ # (max 713 output tokens observed), so this only widens the injection path.
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ self.task_timeout = int(os.environ.get("TE_TASK_TIMEOUT", "1800"))
+ self.eval_timeout = int(os.environ.get("TE_EVAL_TIMEOUT", "1800"))
+ self.keep = os.environ.get("TE_KEEP_CONTAINERS") == "1"
+ # Stated rather than left to run_config's getattr default: this harness has
+ # ALWAYS made its mechanism calls as plain Messages requests, and that is the
+ # transport the Copilot harness now matches. Naming it here means the field
+ # describes a decision instead of an absence.
+ self.optimizer_transport = "api"
+ self._preflight_optimizer_endpoint()
+
+ def _preflight_optimizer_endpoint(self):
+ """Refuse the lane if the mechanism endpoint will not answer. See the shared
+ helper for why a silent fallback is worse than a hard failure."""
+ preflight_optimizer_endpoint(
+ api_key=self.api_key, configured_model=self.optimizer_model, label="claude",
+ remedy="Check that ANTHROPIC_API_KEY is valid and can call MODEL.")
+
+ # ---- 1. the paid optimizer call the mechanisms make between tasks ------------------
+ def optimizer_call(self, prompt, model="agent"):
+ """One small direct Messages-API request. Returns (text, effective_tokens).
+
+ `effective_tokens` is cache-aware and priced with the SAME weight table the task
+ rows use, so mechanism overhead lands in the ledger on the task cost's own scale
+ rather than as a raw token count.
+ """
+ return anthropic_optimizer_call(
+ prompt, model=model, configured_model=self.optimizer_model,
+ max_tokens=self.optimizer_max_tokens, api_key=self.api_key)
+
+ # ---- 2. container plumbing --------------------------------------------------------
+ def _reap_exited(self, name):
+ """Remove EXITED containers from earlier attempts at this same cell.
+
+ Container names used to be one-per-(instance, arm), so `docker rm -f <name>` in
+ `_start` doubled as cleanup: a re-run of a cell collected whatever a previously
+ killed run had left behind. Names are now unique per task-run (see
+ `_container_name`), which is what makes two lanes safe -- but it also means that
+ exact-name removal no longer reaps anything, and a hard-killed driver would leak a
+ container per attempt.
+
+ So reap by the readable stem instead, and ONLY containers that have exited. A
+ RUNNING container sharing this stem is the whole point of the rename: it belongs to
+ a live lane working the same cell, and force-removing it is precisely the bug this
+ replaced. Best-effort throughout -- failing to reap is untidy, not incorrect.
+ """
+ stem = name.rsplit("-", 1)[0]
+ try:
+ p = _run(["docker", "ps", "-a", "--filter", f"name=^{stem}-",
+ "--filter", "status=exited", "--format", "{{.Names}}"], timeout=60)
+ for stale in p.stdout.split():
+ if stale != name:
+ _run(["docker", "rm", "-f", stale], timeout=60)
+ except Exception:
+ pass
+
+ # ---- 1b. project-environment activation ---------------------------------------
+ def _activate_project_env(self, name):
+ """Discover the image's project env and record how it will be activated.
+
+ Called at the end of every runner's `_start`. Measures rather than assumes: the
+ env dir is listed, the PATH the container was built with is read back, and the
+ resulting `python` is resolved -- so `run_config.json -> env_activation` says
+ what the agent's shell actually saw, per task.
+ """
+ store = self.__dict__.setdefault("_env_activation", {})
+ if not env_activation_enabled():
+ store[name] = {"mode": "none", "requested": False}
+ return store[name]
+ script = (f'printf "PATH=%s\\n" "$PATH"; for d in {_CONDA_ENVS_DIR}/*/bin; do '
+ '[ -d "$d" ] && printf "ENV=%s\\n" "$d"; done; true')
+ r = _run(["docker", "exec", name, "bash", "-c", script], timeout=120)
+ container_path, env_bins = "", []
+ for line in (r.stdout or "").splitlines():
+ if line.startswith("PATH="):
+ container_path = line[5:].strip()
+ elif line.startswith("ENV="):
+ env_bins.append(line[4:].strip())
+ env_bin = choose_project_env(env_bins)
+ if not env_bin:
+ _log(f"env activation: no project env found under {_CONDA_ENVS_DIR} "
+ f"(candidates: {env_bins}) -- agent runs with the image's PATH")
+ store[name] = {"mode": "none", "requested": True, "candidates": env_bins}
+ return store[name]
+ env = activation_env(env_bin, container_path)
+ chk = _run(["docker", "exec", "-e", f"PATH={env['PATH']}", name, "bash", "-c",
+ 'command -v python; python -c "import sys; print(sys.prefix)"'],
+ timeout=120)
+ out = (chk.stdout or "").split()
+ python = out[0] if out else None
+ prefix = out[1] if len(out) > 1 else None
+ info = {"mode": "path-prepend", "requested": True, "env_bin": env_bin,
+ "env_name": env["CONDA_DEFAULT_ENV"], "python": python,
+ "verified": bool(python and python.startswith(env_bin))}
+ info["env"] = env
+ if not info["verified"]:
+ _log(f"env activation: PATH prepend did not resolve python to {env_bin} "
+ f"(got {python!r}, prefix {prefix!r})")
+ store[name] = info
+ return info
+
+ def _env_activation_info(self, name):
+ return self.__dict__.get("_env_activation", {}).get(name) or {"mode": "none"}
+
+ def _env_activation_args(self, name):
+ """`docker exec -e` args for the agent process. Empty when nothing was activated."""
+ info = self._env_activation_info(name)
+ if info.get("mode") != "path-prepend":
+ return []
+ out = []
+ for k, v in info["env"].items():
+ out += ["-e", f"{k}={v}"]
+ return out
+
+ def _record_env_activation(self, run_dir_path, name, row):
+ """Write the measured activation into run_config.json and the row."""
+ info = dict(self._env_activation_info(name))
+ info.pop("env", None)
+ row["env_activation"] = info.get("mode")
+ cfg_path = os.path.join(run_dir_path, "run_config.json")
+ try:
+ with open(cfg_path, encoding="utf-8") as fh:
+ cfg = json.load(fh)
+ cfg["env_activation"] = info
+ with open(cfg_path, "w", encoding="utf-8") as fh:
+ json.dump(cfg, fh, indent=2, default=str)
+ except OSError as e:
+ _log(f"could not record env_activation in run_config.json: {e}")
+
+ def _start(self, name, instance_id):
+ _run(["docker", "rm", "-f", name])
+ self._reap_exited(name)
+ cmd = ["docker", "run", "-d", "--name", name, "--entrypoint", "bash",
+ "-v", f"{self.claude}:/usr/local/bin/claude:ro",
+ "-e", "HOME=/tmp/agent-home", "-w", "/testbed"] + _cpu_limit_args()
+ if self.scaffold is not None:
+ for spec in self.scaffold.mounts(self.scaffold_profile):
+ cmd += ["-v", spec]
+ _LIVE_CONTAINERS.add(name)
+ # Container start is the one step whose latency is set by the HOST, not by us: on
+ # a loaded shared box `docker run -d` can take many minutes, and a timeout here
+ # kills the whole lane mid-stream (the container then finishes starting and is
+ # orphaned). Configurable, and generous by default.
+ p = _run(cmd + [image_for(instance_id), "-lc", "sleep infinity"],
+ timeout=int(os.environ.get("TE_DOCKER_START_TIMEOUT", "900")))
+ if p.returncode != 0:
+ raise RuntimeError(f"docker run failed for {instance_id}: {p.stderr[-500:]}")
+ _run(["docker", "exec", name, "mkdir", "-p", "/tmp/agent-home"])
+ self._activate_project_env(name)
+ return name
+
+ def _exec(self, name, script, timeout=300, workdir="/testbed"):
+ return _run(["docker", "exec", "-w", workdir, name, "bash", "-lc", script],
+ timeout=timeout)
+
+ # ---- 2b. the network guard (every arm) ---------------------------------------------
+ # Container-side twin of webbench's `_install_net_guard`. The agent runs INSIDE the task
+ # container, so the hook script, its registration and its log all have to live there --
+ # a host-side settings.json is never read by the CLI in the container.
+ #
+ # WHY THIS AND NOT THE TOOL MASK. `network_tool_block()` drops WebSearch/WebFetch, but
+ # measured across 1,824 stored SWE-bench rows the shell was the BIGGER route: 120 rows
+ # reached the network through `Bash` against 47 through a web tool, and 16 rows fetched
+ # an answer-key-shaped URL (`.patch`, `/commit/`, `api.github.com/search/...`) of which
+ # 14 resolved -- one of them curling the instance's own fix commit as `/tmp/fix.patch`.
+ # `Bash` is in PROTECTED_TOOLS and can never be masked, so the hook is the only seam.
+ # Derived from the AGENT PROFILE's home, never hardcoded: the Claude container runs
+ # HOME=/tmp/agent-home and the Copilot one HOME=/home/agent, and a literal path here
+ # meant the Copilot runner inherited a directory its `_start` never created --
+ # `docker cp` failed, and EVERY Copilot SWE-bench row read `net_guard_installed:
+ # false` (measured 0/60 armed across both stored copilot_swebench_* trees).
+ @property
+ def NET_GUARD_SCRIPT(self):
+ return f"{self.scaffold_profile.home}/net_guard.py"
+
+ @property
+ def NET_GUARD_LOG(self):
+ return f"{self.scaffold_profile.home}/net_guard_blocks.jsonl"
+
+ def _install_net_guard(self, name):
+ """Register the PreToolUse guard inside the task container. True when armed.
+
+ Applies to EVERY arm. An agent that can fetch the instance's own fix commit makes
+ the resolve verdict meaningless, which is not an arm-level concern -- and leaving
+ it to untool would be worse than useless, since untool drops a tool only after
+ observing disuse, so precisely the agents that browse would keep the ability.
+
+ APPENDED to whatever is already in `PreToolUse`, and installed AFTER the scaffold.
+ `Scaffold.merge_json_file` merges one level deep, so `hooks` is dict-updated and a
+ scaffold's `PreToolUse` list REPLACES the whole list rather than extending it: an
+ rtk arm registering after us would silently carry rtk's hook and not ours. Order
+ costs nothing -- a scaffold's own setup runs through `docker exec`, not the agent's
+ Bash tool, so no hook ever applied to it.
+
+ Never raises. A task whose guard failed to install must still produce a row, but
+ the row says which it is (`net_guard_installed`), so an unguarded row can be
+ filtered out instead of being silently pooled with guarded ones.
+ """
+ if not network_tool_block(): # TE_ALLOW_NETWORK_TOOLS lifts BOTH halves, or neither
+ return False
+ try:
+ cp = _run(["docker", "cp", net_guard.__file__,
+ f"{name}:{self.NET_GUARD_SCRIPT}"], timeout=120)
+ if cp.returncode != 0:
+ raise RuntimeError((cp.stderr or "")[-200:])
+ # The interpreter is resolved INSIDE the image and written ABSOLUTE. The CLI
+ # spawns a hook without a login shell, so a bare `python3` is not guaranteed to
+ # be on its PATH, and a hook that always errors is worse than none: the CLI
+ # reports the failure while the agent keeps going, so the run looks guarded and
+ # is not. `net_guard` imports only the stdlib, so any interpreter will serve.
+ probe = self._exec(name, "command -v python3 || command -v python", timeout=60)
+ pys = [ln for ln in (probe.stdout or "").split() if ln.startswith("/")]
+ if not pys:
+ raise RuntimeError("no python interpreter found in the image")
+ if not self._register_net_guard(name, pys[0]):
+ raise RuntimeError("hook registration did not report success")
+ # PROVE IT BLOCKS, rather than trusting that registration reported success.
+ # Registration succeeding says the config file was written or the plugin
+ # installed -- neither of which means the CLI reads it, that the matcher name
+ # is right, or that the guard can find the command in the payload it is handed
+ # (the two CLIs spell that differently -- guards/net_guard.hook_command). Each
+ # of those failed silently at least once while every provenance field read
+ # healthy, so the only trustworthy check is a real refusal.
+ self._verify_net_guard(name, pys[0])
+ return True
+ except Exception as e:
+ _log(f"net guard install FAILED ({type(e).__name__}: {e}) -- row runs UNGUARDED")
+ return False
+
+ def _install_answer_key_hosts(self, name):
+ """Black-hole the answer-key hosts in the container's /etc/hosts. True when done.
+
+ THE PRIMARY CONTROL, and the one that does not depend on the agent's cooperation.
+ The PreToolUse hook above runs only where the CLI's hook contract is met -- each
+ lane's file, key and matcher, and on Codex `--dangerously-bypass-hook-trust` as
+ well -- and it sees only the shell tool. A layer that needs none of that is what
+ makes the guard hold when one of those assumptions breaks with a CLI release.
+
+ This layer catches every route the stored corpus actually used, because all of them
+ resolve a hostname first: `curl`, `wget`, `git clone`, and
+ `python -c "import urllib.request"`. Verified in a real SWE-bench container: each
+ fails with "Connection refused" afterwards while pypi.org, files.pythonhosted.org
+ and a real `pip download` keep working -- which is the whole point of resolving a
+ NAME LIST rather than cutting the container's network off. Applied to every arm,
+ baseline included: an agent that can read the instance's own fix commit makes the
+ resolve verdict meaningless, which is not an arm-level concern.
+
+ Never raises. A task whose block failed must still produce a row, but the row says
+ which it is so an unguarded row can be filtered rather than silently pooled.
+ """
+ if not network_tool_block(): # TE_ALLOW_NETWORK_TOOLS lifts every layer, or none
+ return False
+ try:
+ # Appended, and idempotently: the image's own /etc/hosts carries the container
+ # identity, and replacing it would break name resolution for the container
+ # itself. `grep -q` keeps a re-exec from stacking duplicate stanzas.
+ blob = base64.b64encode(net_guard.hosts_block().encode()).decode()
+ r = self._exec(
+ name,
+ f"grep -q {shlex.quote(net_guard.HOSTS_MARKER)} /etc/hosts "
+ f"|| echo {blob} | base64 -d >> /etc/hosts; "
+ f"grep -c {shlex.quote(net_guard.HOSTS_MARKER)} /etc/hosts",
+ timeout=60)
+ if (r.stdout or "").strip().splitlines()[-1:] != ["1"]:
+ raise RuntimeError((r.stderr or r.stdout or "")[-200:])
+ # PROVE IT REFUSES, rather than trusting that the file was written. A write can
+ # succeed into a container whose resolver ignores /etc/hosts (musl with a
+ # different nsswitch, a VPN resolver), and that failure is invisible: the file
+ # looks right and every fetch still succeeds.
+ probe = self._exec(
+ name,
+ "getent hosts github.com | awk '{print $1}' | head -1", timeout=60)
+ if "127.0.0.1" not in (probe.stdout or ""):
+ raise RuntimeError(
+ "hosts stanza written but github.com still resolves to "
+ f"{(probe.stdout or '').strip()!r} -- this container's resolver is not "
+ "reading /etc/hosts, so nothing is blocked")
+ return True
+ except Exception as e:
+ _log(f"answer-key hosts block FAILED ({type(e).__name__}: {e}) "
+ "-- row runs UNGUARDED at the DNS layer")
+ return False
+
+ def _answer_key_attempts(self, run_dir_path):
+ """This task's answer-key-shaped command count. See `answer_key_attempts`."""
+ return answer_key_attempts(run_dir_path)
+
+ @staticmethod
+ def _net_guard_merge_program(entry):
+ """Python, run INSIDE the container, that appends `entry` to the `PreToolUse` list
+ of the JSON hook file named by argv[1], idempotently, and prints `armed`.
+
+ Claude Code's settings.json and Codex's hooks.json share this shape -- a top-level
+ `hooks` object keyed by PascalCase event name -- so one program serves both lanes."""
+ return (
+ "import json, os, sys\n"
+ "p = sys.argv[1]\n"
+ f"entry = json.loads({json.dumps(json.dumps(entry))})\n"
+ "os.makedirs(os.path.dirname(p), exist_ok=True)\n"
+ "try:\n"
+ " cur = json.load(open(p))\n"
+ "except Exception:\n"
+ " cur = {}\n"
+ "pre = cur.setdefault('hooks', {}).setdefault('PreToolUse', [])\n"
+ "if not any('net_guard' in json.dumps(h) for h in pre):\n"
+ " pre.append(entry)\n"
+ "json.dump(cur, open(p, 'w'), indent=2)\n"
+ "print('armed')\n")
+
+ def _write_hook_entry(self, name, python, path):
+ """Append the guard's PreToolUse entry to the hook file at `path` in the container."""
+ entry = {"matcher": "Bash", "hooks": [
+ {"type": "command", "command": f"{python} {self.NET_GUARD_SCRIPT}"}]}
+ prog = self._net_guard_merge_program(entry)
+ r = self._exec(
+ name,
+ f"cat > /tmp/te_net_guard_install.py <<'TE_EOF'\n{prog}\nTE_EOF\n"
+ f"{python} /tmp/te_net_guard_install.py {path}", timeout=60)
+ if r.returncode != 0 or "armed" not in (r.stdout or ""):
+ raise RuntimeError(((r.stderr or "") + (r.stdout or ""))[-200:])
+ return True
+
+ def _register_net_guard(self, name, python):
+ """Claude Code: append a PreToolUse hook to the agent's own settings.json."""
+ return self._write_hook_entry(
+ name, python, f"{self.scaffold_profile.home}/.claude/settings.json")
+
+ # The probe command is one the guard MUST refuse and that reaches no network even if
+ # the guard is missing: `decide` matches on the substring `github` alone, so `echo`
+ # is enough to exercise the whole path without a request leaving the container.
+ NET_GUARD_PROBE = "echo https://github.com/probe"
+
+ def _verify_net_guard(self, name, python):
+ """Run the guard against a payload the CLI would send. Raises unless it refuses.
+
+ Feeds the hook script directly rather than driving the agent: a real agent turn
+ would cost a paid call per task, and what can break here -- payload spelling,
+ interpreter path, script not copied -- is all exercised by the script itself.
+
+ **What this does NOT cover**: whether the CLI actually invokes the hook. That rests
+ on the settings-file location, the hook key and the matcher name, which are per
+ agent (`_register_net_guard`) and were each verified by hand against the live CLIs
+ on 2026-09-08 -- not by this check. If you change any of those three, re-probe
+ end to end with a real agent turn; a green `net_guard_installed` will not catch it.
+ """
+ payload = json.dumps({"tool_name": "Bash", "toolName": "bash",
+ "tool_input": {"command": self.NET_GUARD_PROBE},
+ "toolArgs": {"command": self.NET_GUARD_PROBE}})
+ r = self._exec(
+ name,
+ f"printf '%s' {shlex.quote(payload)} | {python} {self.NET_GUARD_SCRIPT}; "
+ "echo rc=$?",
+ timeout=60)
+ if "rc=2" not in (r.stdout or ""):
+ raise RuntimeError(
+ "guard installed but did NOT refuse its probe (expected rc=2, got "
+ f"{(r.stdout or '').strip()[-80:]!r}) -- it would block nothing")
+
+ def _net_guard_blocks(self, name):
+ """How many Bash commands the guard refused this task. No log = it never fired.
+
+ Read while the container is still alive -- `_stop` removes it -- and recorded on
+ EVERY row, zero included: a guard is only trustworthy if you can watch it fire, and
+ a model that starts routing around it shows up here as a count that falls to zero
+ while the traces still contain fetches.
+ """
+ try:
+ p = self._exec(name, f"wc -l < {self.NET_GUARD_LOG} 2>/dev/null || echo 0",
+ timeout=60)
+ return int((p.stdout or "0").strip().split()[0])
+ except Exception:
+ return 0
+
+ # ---- 2b. external tool scaffold (baseline arms only) ------------------------------
+ def _scaffold_context(self, name, task, run_dir_path):
+ """The handles a scaffold may use. Docker is reached only through these."""
+ def exec_(script, timeout=300, env=None):
+ cmd = ["docker", "exec", "-w", "/testbed"]
+ for key, value in (env or {}).items():
+ cmd += ["-e", f"{key}={value}"]
+ return _run(cmd + [name, "bash", "-lc", script], timeout=timeout)
+
+ def copy_in(host_path, container_path):
+ return _run(["docker", "cp", host_path, f"{name}:{container_path}"],
+ timeout=300)
+
+ def read_bytes(script, timeout=600):
+ """Raw stdout of a container script. Binary, because the repo-map baseline
+ streams a tar out this way and decoding it as text would corrupt it."""
+ proc = subprocess.run(
+ ["docker", "exec", name, "bash", "-lc", script],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
+ if proc.returncode != 0:
+ raise scaffolds_pkg.ScaffoldError(
+ f"container script failed: {(proc.stderr or b'')[-200:]!r}")
+ return proc.stdout
+
+ return scaffolds_pkg.ScaffoldContext(
+ profile=self.scaffold_profile, container=name, task=task,
+ run_dir=run_dir_path, exec_=exec_, copy_in=copy_in, host_run=_run,
+ read_bytes=read_bytes, log=_log)
+
+ def _apply_scaffold(self, name, task, run_dir_path):
+ """Install the arm's scaffold. Returns (notes, mcp_config).
+
+ A scaffold that raises is recorded and the task continues NATIVE: one broken
+ install must not cost the row, because a row flagged `scaffold_error` is
+ diagnosable while a missing row is not.
+ """
+ if self.scaffold is None:
+ return {}, None
+ try:
+ notes = self.scaffold.setup(self._scaffold_context(name, task, run_dir_path))
+ except Exception as e: # noqa: BLE001
+ _log(f"{task['instance_id']}: scaffold {self.scaffold.name} FAILED "
+ f"({type(e).__name__}: {e}) -- task runs native, row flagged")
+ return {"arm_setup": self.scaffold.name, "scaffold_error":
+ f"{type(e).__name__}: {e}"[:300]}, None
+ return notes, notes.get("mcp_config")
+
+ def _graded_diff(self, name):
+ """The agent's diff, with every scaffold artifact subtracted.
+
+ A scaffold writes project instructions INTO the repo, so those files are tracked
+ by `git diff` and would be applied by the official grader -- scoring the
+ scaffold's prompt text as part of the fix. The exclusions are git pathspecs, and
+ what was excluded is recorded in run_config.json.
+ """
+ excludes = self._patch_excludes()
+ # With no scaffold the command is left EXACTLY as it was before scaffolds existed.
+ # `diff -- .` is equivalent from the repo root, but "equivalent" is not "identical",
+ # and every stored baseline row was produced by the bare form.
+ diff = "git -C /testbed diff"
+ if excludes:
+ spec = " ".join(f"':(exclude,top){e}'" for e in excludes)
+ diff = f"{diff} -- . {spec}"
+ return self._exec(
+ name,
+ f"git -C /testbed add -A -N >/dev/null 2>&1; {diff}").stdout
+
+ def _patch_excludes(self):
+ """Paths never counted as the agent's fix: the agent's own instructions file plus
+ whatever the active scaffold writes. Sorted so run_config is stable."""
+ if self.scaffold is None:
+ return ()
+ return tuple(sorted(set(self.scaffold_profile.patch_excludes)
+ | set(self.scaffold.patch_excludes)))
+
+ def _stop(self, name):
+ """Remove the task container. NEVER raises -- cleanup is untidy, not incorrect.
+
+ This is called from `run_task`'s `finally`, so anything it raises escapes the
+ per-task error handling and kills the whole STREAM. That is not hypothetical: on a
+ shared box at load ~96 with rootless Docker, `docker rm -f` exceeded the 120s
+ default and `subprocess.TimeoutExpired` propagated out of the finally, ending a
+ 60-task sympy lane at task 3 with 57 tasks unrun (2026-09-10).
+
+ The same reasoning `_start` already carries -- "a host that cannot start containers
+ must not take the STREAM down with it" -- applies at least as strongly here, where
+ the task is already graded and its row already built. A leaked container is
+ reaped later by `_reap_exited`, so failing to remove one costs disk, not results.
+
+ The timeout is also generous now and configurable: removal on a loaded rootless
+ daemon is bounded by the HOST, not by us.
+ """
+ if self.keep:
+ _log(f"keeping container {name} (TE_KEEP_CONTAINERS=1)")
+ _LIVE_CONTAINERS.discard(name)
+ return
+ try:
+ _run(["docker", "rm", "-f", name],
+ timeout=int(os.environ.get("TE_DOCKER_STOP_TIMEOUT", "600")))
+ except Exception as e:
+ _log(f"could not remove container {name} ({type(e).__name__}: {e}) -- "
+ "leaving it for _reap_exited; the task's row is unaffected")
+ _LIVE_CONTAINERS.discard(name)
+
+ # ---- 3. one streamed agent segment ------------------------------------------------
+ def _segment(self, name, prompt, transcript_fh, exclude_tools, max_turns, guard,
+ mcp_config=None):
+ """Run `claude -p` once, streaming stream-json. Every raw line is appended to the
+ transcript (the mechanisms re-read exactly this file) and, when the spiral guard is
+ active, folded into it live; a guard stop kills the agent mid-stream, leaving the
+ container's working tree -- the agent's own edits -- intact for the next segment.
+
+ Returns the run's final `result` event (or {} if the stream ended without one).
+ """
+ # SECRETS NEVER GO IN ARGV. `docker exec -e VAR` with no "=value" tells docker to
+ # take the value from the CLIENT's environment, so the key is passed through the
+ # Popen env below instead of appearing in the process table -- which on a shared
+ # host is readable by every other user via `ps`.
+ #
+ # ANTHROPIC_API_KEY is the ONLY Anthropic variable the container receives. Every
+ # variable is named here explicitly, so nothing that could point the CLI at another
+ # endpoint or credential (IGNORED_ANTHROPIC_VARS) reaches it from the host.
+ cmd = ["docker", "exec", "-i", "-w", "/testbed",
+ "-e", "ANTHROPIC_API_KEY",
+ "-e", "HOME=/tmp/agent-home",
+ # Where the PreToolUse guard appends its refusals; the hook inherits this
+ # from the CLI process that spawns it.
+ "-e", f"{net_guard.LOG_ENV}={self.NET_GUARD_LOG}",
+ # The project env, for every shell the CLI spawns -- see
+ # _activate_project_env. After HOME.
+ *self._env_activation_args(name),
+ # SWE-bench images run as root, and the CLI refuses
+ # --dangerously-skip-permissions as root unless it knows it is sandboxed --
+ # which this disposable, per-task container genuinely is.
+ "-e", "IS_SANDBOX=1",
+ "-e", f"CLAUDE_CODE_EFFORT_LEVEL={self.effort}",
+ "-e", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1",
+ "-e", "DISABLE_TELEMETRY=1", "-e", "DISABLE_AUTOUPDATER=1",
+ name, "claude", "-p",
+ "--output-format", "stream-json", "--verbose",
+ "--dangerously-skip-permissions",
+ "--model", self.model, "--max-turns", str(max_turns),
+ # Explicit flag, not just the env var: this CLI exposes --effort directly,
+ # and effort silently defaulting low would change both cost and resolve.
+ "--effort", self.effort]
+ drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]
+ # Network tools are dropped for every arm, on top of whatever untool decided --
+ # see NETWORK_TOOLS: the answers to both benchmarks are a fetch away.
+ drop += [t for t in network_tool_block() if t not in drop]
+ if drop:
+ cmd += ["--disallowed-tools", ",".join(drop)]
+ # A scaffold's MCP server (Serena). Appended last so it cannot be reordered into
+ # the middle of the tool mask.
+ cmd += self.scaffold_profile.mcp_args(mcp_config)
+
+ # The docker CLIENT's environment carries the secrets that `-e VAR` forwards.
+ cenv = dict(os.environ)
+ cenv["ANTHROPIC_API_KEY"] = self.api_key
+ proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, text=True, bufsize=1, env=cenv)
+ proc.stdin.write(prompt)
+ proc.stdin.close()
+
+ result, stopped, deadline = {}, False, time.time() + self.task_timeout
+ # Usage is folded in as the stream arrives, because the break paths below kill the
+ # CLI before its terminal `result` event: without this the segment's cost is
+ # simply lost. See _UsageRecovery.
+ recovery, stop_reason = _UsageRecovery(), None
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ try:
+ ev = json.loads(line)
+ except Exception:
+ ev = {}
+ recovery.observe(ev)
+ if ev.get("type") == "result":
+ result = ev
+ if guard is not None and guard.monitor(line):
+ stopped, stop_reason = True, "spiral_guard_stop"
+ break
+ if time.time() > deadline:
+ _log("segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped, stop_reason = True, "task_timeout"
+ break
+
+ if stopped:
+ # Kill the agent INSIDE the container: terminating the `docker exec` client
+ # leaves the CLI running and still billing.
+ _run(["docker", "exec", name, "pkill", "-TERM", "-f", "claude"])
+ time.sleep(2)
+ _run(["docker", "exec", name, "pkill", "-KILL", "-f", "claude"])
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-600:]
+ if err.strip() and not result:
+ _log(f"agent stderr: {err.strip()[:300]}")
+ if not result:
+ # No terminal `result` event: a guard/timeout stop, or a stream that died.
+ # Recover what the transcript can prove rather than recording zeros, and take
+ # `output_tokens` from the CLI's own session log, which the kill does not erase.
+ result = recovery.synthesize(stop_reason or "no_result_event",
+ self._session_log_text(name))
+ return result
+
+ def _session_log_text(self, name):
+ """Claude Code's session log for this container, or '' if unreadable.
+
+ Read AFTER the kill: the CLI appends each assistant message as it completes, so the
+ file is already on disk and the kill cannot take it back. Globbed rather than
+ composed from a slug -- the CLI derives the directory name from cwd
+ (`/testbed` -> `-testbed`) and that derivation is its business, not ours. A nudge
+ re-invokes the CLI and so writes a SECOND session file; both are concatenated and
+ `session_output_tokens` deduplicates by message id, so double-counting is not
+ possible from the overlap.
+
+ Never raises and never fails a task: an unreadable log just means the row keeps the
+ old, honest `output_tokens=0` and stays flagged.
+ """
+ home = self.scaffold_profile.home
+ try:
+ p = self._exec(name, f"cat {home}/.claude/projects/*/*.jsonl 2>/dev/null",
+ timeout=60)
+ return p.stdout or ""
+ except Exception as e:
+ _log(f"session log unreadable ({type(e).__name__}: {e})")
+ return ""
+
+ # ---- 3a. spiral hooks (overridden per agent) --------------------------------------
+ # The generic guard nudges by REPROMPTING (a fresh `-p` call carrying a redirect on top
+ # of the agent's own on-disk edits) and reports its telemetry under `spiral_*`. The
+ # Codex-native controller resumes the EXACT thread instead -- a fresh session would
+ # re-pay for the context it already built -- so it takes no prompt, returns a resume
+ # REQUEST rather than a string, and reports under `codex_spiral_*` because that is what
+ # `mechanisms.spiral.SpiralStage.after_task` reads off the row. Three hooks rather than
+ # a second copy of run_task: everything else about the lifecycle -- container, scaffold,
+ # net guard, patch extraction, official grading -- is identical across all three agents.
+ def _build_guard(self, run_kwargs):
+ """The in-flight spiral guard for this task, or None when the arm has none."""
+ if not run_kwargs.get("spiral_guard"):
+ return None
+ guard_kwargs = dict(
+ envelope=run_kwargs.get("abort_envelope"),
+ judge=run_kwargs.get("spiral_judge"),
+ thr=run_kwargs.get("spiral_thr"),
+ nudge_budget=int(run_kwargs.get(
+ "spiral_nudge_budget",
+ 1 if run_kwargs.get("spiral_nudge") else 0)),
+ nudge_writer=run_kwargs.get("spiral_nudge_writer"),
+ verbose=os.environ.get("TE_VERBOSE", "1") == "1")
+ # `never_abort` is CONSERVATIVE-MODE-ONLY and absent from the vendored guard on
+ # branches without trimech-safe -- see guard_accepts() for why it is probed.
+ if guard_accepts("never_abort"):
+ guard_kwargs["never_abort"] = bool(run_kwargs.get("spiral_never_abort"))
+ return spiral_vendor.NudgeGuard(**guard_kwargs)
+
+ def _guard_continuation(self, guard, prompt):
+ """The next segment's input after a guard stop, or falsey to stop."""
+ return guard.continuation(prompt)
+
+ def _guard_row(self, guard):
+ """Guard telemetry for the row."""
+ return {
+ "aborted": guard.aborted,
+ "spiral_abort": guard.aborted,
+ "spiral_nudges": guard.nudges_done,
+ "spiral_abort_suppressed": getattr(guard, "abort_suppressed", False),
+ "spiral_events": guard.events,
+ "spiral_judge_tokens": guard.judge_tokens,
+ "spiral_nudge_writer_tokens": guard.nudge_writer_tokens,
+ }
+
+ # ---- 3b. accounting hooks (overridden per agent) ----------------------------------
+ # Claude Code reports its token breakdown in the stream's terminal `result` event, so
+ # both hooks read that event. Copilot reports NO tokens there and must mine its debug
+ # log instead (see te_harness.copilot), which is why these are hooks and not inlined:
+ # everything else about run_task -- container lifecycle, spiral segments, patch
+ # extraction, official grading -- is identical across the two agents.
+ def _merge_segments(self, a, b):
+ """Combine two spiral segments' usage into one."""
+ return _merge_usage(a, b)
+
+ def _row_from_segment(self, result):
+ """Token/turn row fields from whatever _segment returned."""
+ return _row_from_result(result)
+
+ def _container_name(self, instance_id, arm, run_dir_path):
+ """Docker container name for ONE task-run. Must be unique across live lanes.
+
+ This was `te-<instance>-<arm>` and carried no agent, model or seed -- so a Claude
+ lane and a Copilot lane on the same (instance, arm) resolved to the same name, and
+ since `_start` opens with `docker rm -f <name>` the second lane DELETED the first
+ one's container mid-grade. Nothing reported it: the grading `docker exec` came back
+ empty, `eval.log` was written empty, and `get_eval_report` scored the empty log as
+ `patch_successfully_applied: false` -- a row that reads like a bad model rather than
+ a lost container, on a lane that still exited 0.
+
+ The run directory already encodes timestamp, stream index, agent, instance, arm,
+ seed and mode, so a short digest of it is unique per task-run. The readable stem is
+ kept so `docker ps` still says which instance and arm you are looking at.
+ """
+ stem = ("te-" + instance_id.replace("__", "-").replace("/", "-")
+ + "-" + str(arm)).lower()
+ key = f"{self.agent}|{self.model}|{os.path.abspath(run_dir_path)}"
+ digest = hashlib.sha1(key.encode("utf-8")).hexdigest()[:8]
+ # Docker caps names at 63 chars and wants [a-zA-Z0-9][a-zA-Z0-9_.-]*.
+ stem = re.sub(r"[^a-z0-9_.-]", "-", stem)[:50].rstrip("-.")
+ return f"{stem}-{digest}"
+
+ # ---- 4. the AgentRunner entry point ------------------------------------------------
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ instance_id = task["instance_id"]
+ set_optimizer_context(arm=arm, instance_id=instance_id, model=self.model,
+ agent=self.agent, cell=_cell_of(run_dir_path),
+ run_dir=os.path.basename(run_dir_path))
+ os.makedirs(run_dir_path, exist_ok=True)
+ name = self._container_name(instance_id, arm, run_dir_path)
+ max_turns = int(run_kwargs.get("max_turns") or os.environ.get("TE_MAX_TURNS", 80))
+ t0 = time.time()
+
+ guard = self._build_guard(run_kwargs)
+
+ prompt = build_prompt(task, injected)
+ transcript = os.path.join(run_dir_path, "transcript.jsonl")
+ row = {"instance": instance_id, "repo": task.get("repo", ""),
+ "model": self.model, "agent": self.agent, "arm": arm,
+ "effort": self.effort, "cli_version": self.cli_version}
+
+ # Self-describing run: the CLI records neither the effort level nor the tool mask
+ # in its transcript, so a finished run could not answer "what settings produced
+ # this?" from its own artifacts. Write them down next to the transcript.
+ self._write_run_config(run_dir_path, task, arm, injected, exclude_tools,
+ max_turns, run_kwargs, guard)
+
+ # A host that cannot start containers must not take the STREAM down with it.
+ # TimeoutExpired/RuntimeError from `docker run -d` used to propagate out of
+ # run_task and kill the launcher at whichever task it hit -- a wedged rootless
+ # daemon ended django s2 at task 0 of 40, and an intermittent 300s stall on
+ # xarray-6599 lost a whole baseline cell. Record the task as an unresolved,
+ # explicitly-flagged error and let the stream continue; a run where every task
+ # carries start_error is a legible infrastructure failure, not a silent crash.
+ net_guard_installed, net_guard_blocks = False, 0
+ hosts_blocked = False
+ try:
+ self._start(name, instance_id)
+ except (subprocess.TimeoutExpired, RuntimeError) as e:
+ _log(f"{instance_id}: CONTAINER START FAILED ({type(e).__name__}) "
+ f"-> unresolved (stream continues)")
+ self._stop(name)
+ row.update({"resolved": False, "is_error": True,
+ "api_error_status": "container_start_failed",
+ "eval_report": {"start_error": str(e)[:400]},
+ "num_turns": 0, "patch_bytes": 0,
+ "wall_s": round(time.time() - t0, 1)})
+ # run_task only RETURNS the row; the driver persists row.json.
+ return row, run_dir_path
+ # Measured at _start; written here so a start failure above still leaves the
+ # placeholder run_config intact and a started task records what its shell saw.
+ self._record_env_activation(run_dir_path, name, row)
+ try:
+ # The scaffold installs BEFORE the agent starts, so its setup traffic is not
+ # model traffic and is not charged to the arm (see scaffolds/__init__.py).
+ scaffold_notes, mcp_config = self._apply_scaffold(name, task, run_dir_path)
+ row.update(scaffold_notes)
+ # AFTER the scaffold, so the guard extends the scaffold's hook list instead of
+ # being replaced by it -- see `_install_net_guard`.
+ net_guard_installed = self._install_net_guard(name)
+ # The DNS layer, on every agent. Independent of the hook above and of each CLI's
+ # hook contract, which is per agent and has broken silently before.
+ hosts_blocked = self._install_answer_key_hosts(name)
+ with open(transcript, "w", encoding="utf-8") as fh:
+ result = self._segment(name, prompt, fh, exclude_tools, max_turns, guard,
+ mcp_config=mcp_config)
+ # Spiral rung 1: re-invoke in the SAME container on top of the agent's own
+ # on-disk edits, with the leakage-safe redirect appended.
+ while guard is not None:
+ nxt = self._guard_continuation(guard, prompt)
+ if not nxt:
+ break
+ _log(f"{instance_id}: spiral nudge -> segment {guard.segment}")
+ seg = self._segment(name, nxt, fh, exclude_tools, max_turns, guard,
+ mcp_config=mcp_config)
+ result = self._merge_segments(result, seg)
+
+ # Patch extraction and grading get the SAME protection container start has,
+ # and for the same reason: a host-side stall must not take the STREAM down.
+ # `_graded_diff` is a `docker exec ... git diff` with a 300 s cap, and on a
+ # loaded shared box it does stall -- one did on 2026-09-13, at task 57 of a
+ # 60-task baseline, and the TimeoutExpired propagated out of run_task and ended
+ # the cell with four tasks unrun after 65 minutes of paid work. The agent's turn
+ # is already bought by this point, so the honest thing is to keep the row, say
+ # what happened on it, and let the stream continue: a task with
+ # `grading_error` is filterable, an ended stream is not recoverable.
+ patch, resolved, report = "", False, {}
+ try:
+ patch = self._graded_diff(name)
+ with open(os.path.join(run_dir_path, "model_patch.diff"), "w",
+ encoding="utf-8") as fh:
+ fh.write(patch)
+ resolved, report = self._grade(name, instance_id, patch, run_dir_path)
+ except (subprocess.TimeoutExpired, RuntimeError) as e:
+ _log(f"{instance_id}: GRADING FAILED ({type(e).__name__}) "
+ f"-> unresolved, flagged (stream continues)")
+ row["is_error"] = True
+ row["api_error_status"] = "grading_failed"
+ row["grading_error"] = f"{type(e).__name__}: {str(e)[:400]}"
+ report = {"grading_error": row["grading_error"]}
+ finally:
+ # Read BEFORE the container is removed -- the log lives inside it.
+ net_guard_blocks = self._net_guard_blocks(name)
+ self._stop(name)
+
+ row.update(self._row_from_segment(result))
+ row["resolved"] = resolved
+ row["eval_report"] = report
+ row["patch_bytes"] = len(patch)
+ row["wall_s"] = round(time.time() - t0, 1)
+ row["untooled_excluded"] = list(exclude_tools or [])
+ row["net_guard_installed"] = net_guard_installed
+ row["net_guard_blocks"] = net_guard_blocks
+ row["answer_key_hosts_blocked"] = hosts_blocked
+ # Attempts, not refusals, and read back from the trace -- the agent-independent
+ # number. See _answer_key_attempts.
+ row["answer_key_attempts"] = self._answer_key_attempts(run_dir_path)
+ if self.scaffold is not None:
+ row["scaffold"] = self.scaffold.name
+ row["patch_excludes"] = list(self._patch_excludes())
+ if guard is not None:
+ row.update(self._guard_row(guard))
+ _log(f"{instance_id} arm={arm} resolved={resolved} turns={row.get('num_turns')} "
+ f"eff_tok={pricing.effective_tokens(row):.0f} {row['wall_s']}s")
+ return row, run_dir_path
+
+ def _write_run_config(self, run_dir_path, task, arm, injected, exclude_tools,
+ max_turns, run_kwargs, guard):
+ """Record everything that shapes this run into `run_config.json`.
+
+ Anything that changes cost or resolve belongs here: the effort level and turn cap,
+ the exact tool mask actually handed to the CLI (after PROTECTED_TOOLS filtering,
+ which is what the agent really saw), the endpoint billed, the CLI build, and the
+ spiral parameters. The injected manual is fingerprinted rather than inlined -- it
+ can be long, and the state file already holds its text.
+ """
+ # The SAME two-step the CLI invocation performs (see `_segment`): recomputing only
+ # the untool half here recorded `excluded_applied: []` on every row and made a
+ # guarded run byte-identical to an unguarded one in its own provenance.
+ drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]
+ drop = drop + [t for t in network_tool_block() if t not in drop]
+ cfg = {
+ "instance_id": task.get("instance_id"), "repo": task.get("repo"), "arm": arm,
+ "agent": self.agent,
+ "model": self.model,
+ "effort": self.effort,
+ "max_turns": max_turns,
+ "cli_version": self.cli_version,
+ "agent_bin": self.agent_bin,
+ "image": image_for(task["instance_id"]),
+ "cpu_limit": _cpu_limit_args()[1] if _cpu_limit_args() else None,
+ "routing": {
+ "base_url": self.routing_base,
+ "optimizer_model": self.optimizer_model,
+ "optimizer_transport": getattr(self, "optimizer_transport", "api"),
+ },
+ # Placeholder; _record_env_activation overwrites it once the container is up.
+ "env_activation": {"requested": env_activation_enabled(), "mode": None},
+ "tools": {
+ "excluded_requested": list(exclude_tools or []),
+ "excluded_applied": drop,
+ "protected": sorted(self.protected_tools),
+ },
+ "injection": {
+ "present": bool(injected),
+ "chars": len(injected or ""),
+ "sha256": (hashlib.sha256(injected.encode()).hexdigest()[:16]
+ if injected else None),
+ },
+ "spiral": {
+ "guard": bool(run_kwargs.get("spiral_guard")),
+ "envelope": run_kwargs.get("abort_envelope"),
+ "thresholds": run_kwargs.get("spiral_thr"),
+ "judge": run_kwargs.get("spiral_judge") is not None,
+ "nudge_writer_llm": run_kwargs.get("spiral_nudge_writer") is not None,
+ "nudge_budget": getattr(guard, "nudge_budget", 0),
+ "never_abort": getattr(guard, "never_abort", False),
+ },
+ "scaffold": {
+ "name": self.scaffold.name if self.scaffold else None,
+ # What was subtracted from the graded diff. Without this a reader cannot
+ # tell an empty patch from one whose only change WAS the scaffold's file.
+ "patch_excludes": list(self._patch_excludes()),
+ },
+ "timeouts": {"task_s": self.task_timeout, "eval_s": self.eval_timeout},
+ "grading_source": self.grader.path,
+ }
+ with open(os.path.join(run_dir_path, "run_config.json"), "w",
+ encoding="utf-8") as fh:
+ json.dump(cfg, fh, indent=2, default=str)
+
+ def _grade(self, name, instance_id, patch, run_dir_path):
+ """Run the official eval script in the container and score its log.
+
+ The eval script resets the test files to base and applies the instance's
+ `test_patch` itself, so any test edits the agent made are discarded exactly as the
+ official pipeline discards them -- only source changes can earn a resolve.
+ """
+ if not patch.strip():
+ _log(f"{instance_id}: empty diff -> unresolved without running tests")
+ return False, {"empty_patch": True}
+ script = self.grader.eval_script(instance_id)
+ host_log = os.path.join(run_dir_path, "eval.log")
+ # A hung eval must not take the STREAM down with it. TimeoutExpired from these
+ # docker execs used to propagate out of run_task and kill the launcher mid-stream:
+ # scikit-learn-14710 ended the sklearn cell at task 21 of 32 TWICE (at eval_timeout
+ # 1800 and again at 3600), so that cell has never produced more than 20 graded rows.
+ # One pathological instance is a scored failure, not a reason to lose the other 11
+ # tasks behind it. Whatever the script printed before the cap is still written to
+ # eval.log, so the hang stays diagnosable.
+ try:
+ w = self._exec(name, "cat > /tmp/eval.sh <<'TE_EOF'\n" + script + "\nTE_EOF\n"
+ "chmod +x /tmp/eval.sh")
+ if w.returncode != 0:
+ return False, self._eval_exec_failed(
+ instance_id, host_log, "", w.returncode, w.stderr, "writing eval.sh")
+ p = self._exec(name, "/tmp/eval.sh 2>&1 | tail -c 4000000",
+ timeout=self.eval_timeout)
+ out = p.stdout
+ except subprocess.TimeoutExpired as e:
+ out = e.stdout or ""
+ if isinstance(out, bytes):
+ out = out.decode("utf-8", "replace")
+ with open(host_log, "w", encoding="utf-8") as fh:
+ fh.write(out)
+ _log(f"{instance_id}: EVAL TIMEOUT after {self.eval_timeout}s -> unresolved "
+ f"(stream continues; see eval.log)")
+ return False, {"eval_timeout": self.eval_timeout}
+ # An eval that produced NOTHING did not grade the patch -- it failed to run. The
+ # usual cause is that the container disappeared under us (see _container_name),
+ # and `docker exec` then exits nonzero with the reason on ITS stderr while the
+ # in-container `2>&1` captures nothing. Scoring that empty log would hand
+ # get_eval_report a file with no APPLY_PATCH_PASS marker, which comes back as
+ # `patch_successfully_applied: false, resolved: false` -- indistinguishable from a
+ # genuine miss, on a lane that still exits 0. Flag it instead.
+ if p.returncode != 0 and not out.strip():
+ return False, self._eval_exec_failed(
+ instance_id, host_log, out, p.returncode, p.stderr, "running eval.sh")
+ with open(host_log, "w", encoding="utf-8") as fh:
+ fh.write(out)
+ if not out.strip():
+ _log(f"{instance_id}: eval script exited 0 but printed NOTHING -> not graded")
+ return False, {"eval_exec_failed": True, "eval_exec_rc": 0,
+ "eval_exec_error": "eval script produced no output"}
+ try:
+ return self.grader.score(instance_id, patch, host_log)
+ except Exception as e:
+ _log(f"{instance_id}: grading failed: {e}")
+ return False, {"grading_error": str(e)}
+
+ def _eval_exec_failed(self, instance_id, host_log, out, rc, stderr, phase):
+ """Record a grading exec that never ran, loudly and distinguishably.
+
+ Returns the eval_report dict. `resolved` is still False -- but the row now says
+ the task was NOT GRADED rather than graded-and-missed, so aggregate.py and a
+ human reading the row can tell the two apart.
+ """
+ err = (stderr or "").strip()[-500:]
+ try:
+ with open(host_log, "w", encoding="utf-8") as fh:
+ fh.write(out or "")
+ if err:
+ fh.write(f"\n[te_harness] docker exec failed while {phase} "
+ f"(rc={rc}):\n{err}\n")
+ except OSError:
+ pass
+ _log(f"{instance_id}: EVAL DID NOT RUN while {phase} (rc={rc}) -> NOT GRADED. "
+ f"{err or 'no stderr'}")
+ return {"eval_exec_failed": True, "eval_exec_rc": rc,
+ "eval_exec_phase": phase, "eval_exec_error": err}
+
+
+# --------------------------------------------------------------------------------------
+# helpers
+# --------------------------------------------------------------------------------------
+def preflight_optimizer_endpoint(*, configured_model, label, remedy, api_key="", bearer="",
+ base_url=_ANTHROPIC_DIRECT):
+ """Prove the mechanism endpoint answers BEFORE task 0 -- but only if it will be used.
+
+ `anthropic_optimizer_call` never raises: a dead or refusing endpoint returns ("", 0.0),
+ every mechanism reads that as "no verdict" and falls back to its LLM-free path, and the
+ lane runs to completion as an economized arm whose proposer and judges never fired. The
+ solver is unaffected -- it authenticates itself, over its own transport -- so the cell
+ grades and BILLS normally with its mechanisms dark, and nothing in the row says so
+ except overhead that is exactly zero. Spend one 16-token call here and refuse instead.
+
+ Shared by every runner whose mechanism calls use this wire -- the Claude runners, and a
+ Copilot runner on an Anthropic-typed BYOK provider -- so the check cannot drift between
+ them; the two Claude runners went without it until 2026-09-07 and 83 rows were billed
+ on a 401 before anyone looked at the ledger.
+
+ ONLY for an arm that will actually make such a call: `baseline`, `baseline-low`,
+ `baseline-medium` and the scaffold arms make none, so probing for them would bill a call
+ they never need and refuse a lane over an endpoint they never touch. The driver sets
+ TE_OPTIMIZER_PREFLIGHT from `config.needs_optimizer`. Unset means "assume it might", so
+ a direct caller of make_runner keeps the check.
+ """
+ if os.environ.get("TE_OPTIMIZER_PREFLIGHT", "1") == "0":
+ _log(f"{label}: no mechanism on this arm makes a paid call -- "
+ "skipping the endpoint preflight")
+ return
+ probe, _eff = anthropic_optimizer_call(
+ "Reply with the single word: ok", model="agent",
+ configured_model=configured_model, max_tokens=16,
+ api_key=api_key, bearer=bearer, base_url=base_url)
+ if not (probe or "").strip():
+ raise RuntimeError(
+ f"the mechanism-call endpoint {base_url} did not answer a "
+ "16-token probe. Every mechanism would read that as 'no verdict' and quietly "
+ "fall back to its LLM-free path, so an economized arm would run as a baseline "
+ f"wearing its name. {remedy}")
+
+
+def anthropic_optimizer_call(prompt, *, model="agent", configured_model,
+ max_tokens=8192, api_key="", bearer="",
+ base_url=_ANTHROPIC_DIRECT):
+ """One small Messages-API request. Returns (text, eff_tokens).
+
+ Shared by every harness in this package: the between-task proposer / judge / nudge
+ writer is the same call whatever the task domain is, and it must be priced on the SAME
+ weight table the task rows use so mechanism overhead lands in the ledger on the task
+ cost's own scale rather than as a raw token count.
+
+ The Claude lanes call api.anthropic.com on ANTHROPIC_API_KEY. `base_url` and `bearer`
+ exist for one other caller: a Copilot lane on an Anthropic-typed BYOK provider, whose
+ mechanism calls go to that provider on the credential its tasks bill
+ (copilot.anthropic_optimizer_endpoint).
+
+ Never raises: a transport failure returns ("", 0.0), which the mechanisms read as "no
+ verdict" and fall back to their LLM-free path. Retries the codes worth retrying.
+ """
+ # The driver/launchers pass an AGENT label here ("agent", "claude", "copilot"),
+ # not a billable model id; only a real id overrides the configured optimizer model.
+ mid = model if (model or "").startswith("claude-") else configured_model
+ body = json.dumps({
+ "model": mid, "max_tokens": max_tokens,
+ "messages": [{"role": "user", "content": prompt}],
+ }).encode()
+ headers = {"content-type": "application/json", "anthropic-version": "2023-06-01"}
+ # Exactly ONE credential: the API key when there is one, else the BYOK provider's bearer
+ # token. Because this function never raises, a request the endpoint refuses arrives as
+ # "no verdict" and the arm quietly runs its LLM-free path -- so the header is chosen,
+ # never doubled up.
+ if api_key:
+ headers["x-api-key"] = api_key
+ elif bearer:
+ headers["authorization"] = f"Bearer {bearer}"
+ req = urllib.request.Request(
+ base_url.rstrip("/") + "/v1/messages",
+ data=body, method="POST", headers=headers)
+ for attempt in range(3):
+ try:
+ with urllib.request.urlopen(req, timeout=180) as resp:
+ d = json.loads(resp.read().decode())
+ break
+ except urllib.error.HTTPError as e:
+ if e.code in (429, 500, 502, 503, 529) and attempt < 2:
+ time.sleep(4 * (attempt + 1))
+ continue
+ _log(f"optimizer_call HTTP {e.code}: {e.read()[:200]!r}")
+ return "", 0.0
+ except Exception as e: # network hiccup -> LLM-free path
+ if attempt < 2:
+ time.sleep(4 * (attempt + 1))
+ continue
+ _log(f"optimizer_call failed: {e}")
+ return "", 0.0
+ text = "".join(b.get("text", "") for b in d.get("content", [])
+ if isinstance(b, dict) and b.get("type") == "text")
+ u = d.get("usage") or {}
+ if d.get("stop_reason") == "max_tokens":
+ _log(f"optimizer response hit max_tokens ({max_tokens}) and was "
+ f"TRUNCATED -- the caller will fail to parse it and act on nothing. "
+ f"Raise TE_OPTIMIZER_MAX_TOKENS.")
+ _log_optimizer_call(mid, prompt, text, u, stop_reason=d.get("stop_reason"))
+ eff = pricing.effective_tokens({
+ "model": mid,
+ "input_tokens": u.get("input_tokens") or 0,
+ "cache_read_input_tokens": u.get("cache_read_input_tokens") or 0,
+ "cache_creation_input_tokens": u.get("cache_creation_input_tokens") or 0,
+ "output_tokens": u.get("output_tokens") or 0,
+ })
+ return text, eff
+
+
+TASK_TEMPLATE = """You are working in the repository checked out at /testbed.
+
+Resolve the following issue. Modify the SOURCE code so the described behavior is fixed.
+
+<issue>
+{problem}
+</issue>
+
+Rules:
+- Work only inside /testbed; the repo is already at the right commit and its environment
+ is installed (the `testbed` conda env is active).
+- Do NOT modify test files: the grader resets them and applies its own tests.
+- Do not commit; leaving the fix in the working tree is what gets graded.
+- When you believe the fix is complete, stop.
+"""
+
+
+def build_prompt(task, injected):
+ """Task prompt = the economizer's learned manual (if any) + the issue."""
+ body = TASK_TEMPLATE.format(problem=task.get("problem_statement", ""))
+ return (injected.rstrip() + "\n\n" + body) if injected else body
+
+
+def _merge_usage(a, b):
+ """Sum the token counters of two segments' `result` events (spiral re-invocations
+ bill twice; the ledger must see the TOTAL, not just the last segment)."""
+ if not a:
+ return b or {}
+ if not b:
+ return a
+ out = dict(b)
+ ua, ub = (a.get("usage") or {}), (b.get("usage") or {})
+ out["usage"] = {k: (ua.get(k) or 0) + (ub.get(k) or 0)
+ for k in set(ua) | set(ub)
+ if isinstance(ua.get(k, 0), (int, float))
+ and isinstance(ub.get(k, 0), (int, float))}
+ for k in ("num_turns", "total_cost_usd", "duration_ms"):
+ if isinstance(a.get(k), (int, float)) and isinstance(b.get(k), (int, float)):
+ out[k] = a[k] + b[k]
+ # Degradation is a property of the SUM: one recovered segment makes the whole task's
+ # total a lower bound, so the flags OR rather than being taken from the last segment.
+ # A spiral task is exactly this shape -- segment 1 completes normally, segment 2 is
+ # killed by the guard -- so taking `dict(b)`'s flags alone would clear the marker on
+ # the most common case there is.
+ for k in ("usage_incomplete", "output_tokens_missing", "num_turns_recovered",
+ "output_tokens_missing_turn"):
+ out[k] = bool(a.get(k)) or bool(b.get(k))
+ # Turns carrying an exact figure ADD -- unlike the flags, this is a count, and a
+ # two-segment spiral task has exact turns recovered from each segment's own log.
+ ex = [s.get("output_tokens_exact_turns") for s in (a, b)]
+ if any(isinstance(v, int) for v in ex):
+ out["output_tokens_exact_turns"] = sum(v for v in ex if isinstance(v, int))
+ reasons = [r for r in (a.get("usage_incomplete_reason"),
+ b.get("usage_incomplete_reason")) if r]
+ if reasons:
+ out["usage_incomplete_reason"] = "+".join(reasons)
+ return out
+
+
+def session_output_tokens(text):
+ """`(output_tokens, {message_ids})` from Claude Code's OWN session log.
+
+ The CLI persists each assistant message together with its REAL usage as that message
+ completes -- which is the only surviving copy of `output_tokens` when the terminal
+ `result` event never arrives. Measured 2026-09-10 against runs that DID produce a
+ result event: deduplicated, this matches it EXACTLY (421 vs 421, 534 vs 534).
+
+ DEDUPLICATION IS NOT OPTIONAL. Every assistant message is written to the log TWICE --
+ same `message.id`, different `uuid` -- so summing the entries returns exactly 2x the
+ truth, which is a plausible-looking number and therefore the dangerous kind of wrong.
+ Keyed on `message.id`, last write wins.
+
+ Returns the id SET as well as the total: the stream sees the in-flight message start
+ (so `_UsageRecovery` has its id) while this log does not carry it until it completes,
+ so the difference between the two id sets is exactly the unaccounted turn.
+
+ Nothing here does I/O -- the caller supplies the text, because the file lives inside a
+ container on SWE-bench and on the host on Web-Bench. Stdlib only, and never raises: a
+ schema change upstream must degrade to "recovered nothing", not to a failed task.
+ """
+ per_msg = {}
+ for line in (text or "").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ msg = (ev or {}).get("message")
+ if not isinstance(msg, dict):
+ continue
+ mid, usage = msg.get("id"), msg.get("usage")
+ if not mid or not isinstance(usage, dict):
+ continue
+ out = usage.get("output_tokens")
+ if isinstance(out, bool) or not isinstance(out, int):
+ continue
+ per_msg[mid] = out
+ return sum(per_msg.values()), set(per_msg)
+
+
+class _UsageRecovery:
+ """Rebuilds a `result`-shaped usage event from the stream's own assistant messages.
+
+ Claude Code reports its token breakdown ONLY in the stream's terminal
+ `{"type":"result"}` event. Both break paths in `_segment` -- a spiral guard stop and a
+ TE_TASK_TIMEOUT -- kill the CLI mid-stream, so that event never arrives and the segment
+ returned `{}`. `_row_from_result({})` then produced a row of ZEROS: every token field
+ and `num_turns` recorded as 0 for a rollout that had really run for tens of turns and
+ hundreds of thousands of tokens.
+
+ Because only economized arms install a guard, the understatement was ONE-DIRECTIONAL:
+ it made those arms look cheaper than they were and could never touch a baseline.
+ Grading was unaffected (that reads the patch), but turn-based claims were not.
+
+ What is recoverable, and what is not:
+
+ * Every `assistant` event carries `message.usage` with a `message.id`. The snapshots
+ are cumulative per message and repeat as that message's content blocks stream in,
+ so the totals come from deduplicating on `message.id` (last snapshot wins) and
+ summing. Validated against 60 healthy rows that DO have a `result` event:
+ `input_tokens`, `cache_creation_input_tokens` and `cache_read_input_tokens` match
+ it EXACTLY, 60/60.
+ * `output_tokens` is NOT recoverable. The per-message figure is the placeholder from
+ `message_start`, which understates the truth by 7.5x-134.7x (median 28x) -- far too
+ variable to model. It is reported as 0 and the row is flagged, never estimated: a
+ wrong number that looks like data is worse than an absent one.
+ * `num_turns` is approximated by the deduplicated assistant-message count, which
+ matched exactly on 107 of 120 healthy rows and was within 1 on 12 more.
+
+ A recovered row is therefore honest but incomplete, and says so: `usage_incomplete` is
+ set, with `output_tokens_missing` and `num_turns_recovered` naming which parts are
+ soft. Such a row is a LOWER BOUND on cost -- aggregations must not read it as exact,
+ and must never again read an unflagged zero as free.
+ """
+
+ def __init__(self):
+ self._by_msg = {}
+
+ def observe(self, ev):
+ """Fold one parsed stream event in. Cheap enough to call on every line."""
+ if (ev or {}).get("type") != "assistant":
+ return
+ msg = ev.get("message") or {}
+ usage, mid = msg.get("usage"), msg.get("id")
+ if usage and mid:
+ self._by_msg[mid] = usage # cumulative per message: last wins
+
+ def synthesize(self, reason, session_text=None):
+ """A `result`-shaped dict for a stream that never produced one, or {} if nothing
+ was seen at all (a segment that died before its first assistant message really did
+ cost nothing, and must not be flagged as if it had).
+
+ `session_text` is the CLI's own session log, when the caller could read it. With
+ it, `output_tokens` stops being a hole: every turn that COMPLETED carries its real
+ figure there (`session_output_tokens`), so only a turn still being generated at the
+ moment of the kill is unaccounted -- and on a spiral stop there usually is none,
+ because the guard fires while folding a `tool_result`, i.e. BETWEEN turns.
+
+ The flag is cleared only on PROOF of completeness: every message id the stream saw
+ also appears in the session log. Anything less stays flagged. Getting that backwards
+ would reintroduce the exact defect this class exists to prevent -- an unflagged row
+ that reads like a measurement and is not.
+ """
+ if not self._by_msg:
+ return {}
+ tot = {}
+ for usage in self._by_msg.values():
+ for k, v in usage.items():
+ if isinstance(v, (int, float)) and not isinstance(v, bool):
+ tot[k] = tot.get(k, 0) + v
+ recovered = {
+ "input_tokens": int(tot.get("input_tokens") or 0),
+ "cache_creation_input_tokens": int(
+ tot.get("cache_creation_input_tokens") or 0),
+ "cache_read_input_tokens": int(tot.get("cache_read_input_tokens") or 0),
+ "output_tokens": 0, # placeholder-only; see the class docstring
+ }
+ exact_out, exact_turns, inflight = None, 0, True
+ if session_text:
+ try:
+ exact_out, done_ids = session_output_tokens(session_text)
+ exact_turns = len(done_ids)
+ if exact_turns:
+ recovered["output_tokens"] = int(exact_out)
+ # Proof of completeness: nothing the stream saw is missing from the log.
+ inflight = bool(set(self._by_msg) - done_ids)
+ else:
+ exact_out = None
+ except Exception as e: # never fail a task over accounting
+ _log(f"session-log usage read failed ({type(e).__name__}: {e})")
+ exact_out = None
+ got_output = exact_out is not None and exact_turns > 0
+ complete = got_output and not inflight
+ if got_output:
+ _log(f"segment ended without a result event ({reason}); output_tokens "
+ f"recovered EXACTLY from the CLI session log: {recovered['output_tokens']} "
+ f"over {exact_turns} completed turns"
+ + ("" if complete else " -- one in-flight turn unaccounted, row stays "
+ "flagged"))
+ else:
+ _log(f"segment ended without a result event ({reason}); recovered usage from "
+ f"{len(self._by_msg)} assistant messages: "
+ f"in={recovered['input_tokens']} "
+ f"cw={recovered['cache_creation_input_tokens']} "
+ f"cr={recovered['cache_read_input_tokens']} "
+ f"out=UNRECOVERABLE -- row flagged usage_incomplete")
+ return {
+ "type": "result",
+ "usage": recovered,
+ "num_turns": len(self._by_msg),
+ # Flagged unless the session log proves every turn is accounted for.
+ "usage_incomplete": not complete,
+ "usage_recovered_from": "session_log" if got_output else "transcript",
+ "usage_incomplete_reason": None if complete else reason,
+ "output_tokens_missing": not got_output,
+ # How many turns carry an exact figure, and whether a partial turn is missing.
+ # A spiral stop lands between turns (missing_turn False); a TE_TASK_TIMEOUT can
+ # land mid-generation (True), and those two are not equally trustworthy.
+ "output_tokens_exact_turns": exact_turns if got_output else 0,
+ "output_tokens_missing_turn": bool(inflight) if got_output else True,
+ "num_turns_recovered": True,
+ }
+
+
+def _row_from_result(result):
+ """Token breakdown + turn count from Claude Code's final `result` event, in the field
+ names `pricing.effective_tokens` reads."""
+ u = (result or {}).get("usage") or {}
+ it = u.get("input_tokens") or 0
+ cr = u.get("cache_read_input_tokens") or 0
+ cw = u.get("cache_creation_input_tokens") or 0
+ ot = u.get("output_tokens") or 0
+ return {
+ "input_tokens": it, "cache_read_input_tokens": cr,
+ "cache_creation_input_tokens": cw, "output_tokens": ot,
+ "tokens_total": it + cr + cw + ot,
+ "num_turns": (result or {}).get("num_turns") or 0,
+ "is_error": bool((result or {}).get("is_error")),
+ "api_error_status": (result or {}).get("api_error_status"),
+ "cost_usd_vendor": (result or {}).get("total_cost_usd"),
+ # Degradation markers. Present and False on a healthy row, so "no flag" means
+ # "not checked by this harness version" rather than "verified complete" -- and a
+ # zero-token row can never again be silently indistinguishable from a free one.
+ "usage_incomplete": bool((result or {}).get("usage_incomplete")),
+ "output_tokens_missing": bool((result or {}).get("output_tokens_missing")),
+ "num_turns_recovered": bool((result or {}).get("num_turns_recovered")),
+ "usage_incomplete_reason": (result or {}).get("usage_incomplete_reason"),
+ # Session-log recovery provenance: how many turns carry an EXACT output figure,
+ # and whether a mid-generation turn is still unaccounted. Absent (0/False) on a
+ # healthy row, which never needed recovery at all.
+ "output_tokens_exact_turns": int(
+ (result or {}).get("output_tokens_exact_turns") or 0),
+ "output_tokens_missing_turn": bool(
+ (result or {}).get("output_tokens_missing_turn")),
+ }
+
+
+class CopilotSwebenchRunner(ClaudeSwebenchRunner):
+ """AgentRunner: the GitHub Copilot CLI in the official image, graded by the official
+ harness.
+
+ Everything about the task lifecycle is shared with the Claude runner -- container
+ start, spiral segments, patch extraction, official gold grading -- so only four things
+ are overridden: construction (GitHub auth instead of an Anthropic key), the container
+ mounts + seed, the launch argv, and the accounting. See te_harness.copilot for why the
+ accounting cannot be shared: Copilot's terminal `result` event carries no token fields
+ at all, so the Claude path would silently record a row of zeros.
+
+ Two behavioural differences worth knowing before comparing arms with the Claude runner:
+
+ * **No turn cap.** Copilot has no `--max-turns`. `max_turns` is still recorded in
+ run_config.json (and the spiral envelope still counts turns), but it is NOT
+ enforced on the agent -- TE_TASK_TIMEOUT and the spiral guard are the only bounds.
+ * **Auth is on-disk and expires.** The container authenticates from the seeded
+ ~/.copilot, whose short-lived token is only refreshed when a copilot process runs
+ on the HOST. On a long unattended sweep that token goes stale and every later
+ container fast-fails with an empty patch, so keep a host-side keepalive running
+ (docs/COPILOT.md) or set COPILOT_GITHUB_TOKEN.
+ """
+
+ # Copilot reads project instructions from .github/copilot-instructions.md, takes an
+ # MCP config as `--additional-mcp-config @<path>` (built by copilot.build_argv), and
+ # runs as root with HOME=/home/agent. Its native navigation tools are lower-case.
+ scaffold_profile = scaffolds_pkg.AgentProfile(
+ kind="copilot",
+ home="/home/agent",
+ instructions_file=".github/copilot-instructions.md",
+ nav_tools=("view", "grep", "glob"),
+ # build_argv owns the flag spelling; nothing is appended to argv here.
+ mcp_args=lambda cfg: [],
+ patch_excludes=(".github/copilot-instructions.md", ".github/hooks"))
+
+ def __init__(self, agent="copilot", model=None, grading_path=None, scaffold=None):
+ self.agent = agent
+ # Native Copilot bills through the GitHub subscription and picks its own model
+ # unless told otherwise; ANTHROPIC_MODEL is meaningless here, so COPILOT_MODEL is
+ # the knob and 'auto' (the CLI's own router) is the default.
+ # 'auto' normalises to None: it is a routing directive, not a billable model id,
+ # and recording it as one makes pricing hard-fail at the END of a paid rollout.
+ self.model = copilot.normalize_model(model or os.environ.get("COPILOT_MODEL"))
+ copilot.validate_priceable(self.model, self.agent)
+ # Same rule as the Claude path: the optimizer is the agent's own model. It was
+ # already CALLED that way (optimizer_call substitutes self.model for the "agent"
+ # label), but it was RECORDED as "agent" -- so run_config.json described a cost
+ # basis the run did not use. A pinned id records as itself; an unpinned model is
+ # the CLI's own router, which is what the task ran on too, and records as "auto".
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "
+ "runs on the agent's own model so mechanism overhead is priced on the "
+ "same basis as the task rows. Unset it.")
+ self.optimizer_model = self.model or "auto"
+ self.effort_requested = (os.environ.get("TE_EFFORT_LEVEL")
+ or os.environ.get("COPILOT_EFFORT_LEVEL") or "high")
+ if self.effort_requested not in copilot.EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort_requested!r} is not one of "
+ f"{sorted(copilot.EFFORT_LEVELS)}")
+ # The router model rejects --effort and exits, so effort is only APPLIED when a
+ # model is pinned. Recorded as null otherwise, because on `auto` the effort is
+ # whatever the router chose -- not a controlled part of the cost basis.
+ self.effort = (self.effort_requested
+ if copilot.effort_applies(self.model) else None)
+ if self.effort is None:
+ why = ("the launcher probed this model and the CLI rejected --effort"
+ if os.environ.get("TE_EFFORT_APPLIES") == "0"
+ else "the 'auto' router takes no effort level; pin COPILOT_MODEL "
+ "to a served model id to control it")
+ _log(f"copilot: TE_EFFORT_LEVEL={self.effort_requested} NOT applied "
+ f"(model={self.model or 'auto'}, recorded as null) -- {why}.")
+ # An external tool scaffold (rtk / serena / repomap) -- a BASELINE, not a
+ # mechanism. Same contract as the Claude path: resolved and preflighted here so a
+ # misconfigured one kills the lane before task 0 rather than writing rows that
+ # merely look like a scaffold result. `scaffold_profile` above is what makes this
+ # agent-correct (copilot's own instructions file, nav tools and patch excludes).
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile)
+ _log(f"scaffold: {self.scaffold.name} (external baseline)")
+ # need_container_seed: the cache and config dirs are bind-mounted, and docker
+ # invents an empty dir for a missing source rather than failing.
+ self.copilot = copilot.require_install(need_container_seed=True)
+ self.agent_bin = self.copilot
+ self.protected_tools = copilot.PROTECTED_TOOLS
+ # Copilot's own BYOK (copilot.byok_active) bills a custom provider instead of the
+ # GitHub subscription -- a different cost basis, so run_config.json names it.
+ self.routing_base = copilot.provider_base()
+ # TRANSPORT for the mechanisms' between-task calls. `copilot -p` is an agent
+ # session and carries the CLI's system prompt on every call; these calls are
+ # one-shot text completions that need none of it, and the Claude harness has always
+ # made them as plain Messages requests. Use the same wire here WHEN the lane's own
+ # BYOK provider speaks it and so bills what the tasks bill
+ # (copilot.anthropic_optimizer_endpoint has the rule and the reason); otherwise
+ # keep the CLI.
+ self.optimizer_endpoint = copilot.anthropic_optimizer_endpoint()
+ self.optimizer_transport = "api" if self.optimizer_endpoint else "cli"
+ if self.optimizer_endpoint:
+ _log(f"copilot: mechanism calls -> {self.optimizer_endpoint[0]} "
+ "(the BYOK provider, direct Messages API)")
+ self._preflight_optimizer_endpoint()
+ self.cli_version = copilot.cli_version(self.copilot)
+ self.grader = _Grader(grading_path)
+ # An output cap applies on the API transport only -- `copilot -p` has no such flag,
+ # so on the CLI branch this stays a value nothing reads (it used to be omitted
+ # entirely for exactly that reason). Same default and same env var as the Claude
+ # harness, so the two agents' mechanism calls are capped alike.
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ self.task_timeout = int(os.environ.get("TE_TASK_TIMEOUT", "1800"))
+ self.eval_timeout = int(os.environ.get("TE_EVAL_TIMEOUT", "1800"))
+ self.keep = os.environ.get("TE_KEEP_CONTAINERS") == "1"
+
+ def _preflight_optimizer_endpoint(self):
+ """Refuse the lane if its BYOK provider will not answer a mechanism call. Same check,
+ and same TE_OPTIMIZER_PREFLIGHT gate, as every other runner -- see the shared helper.
+ Only reached on the API transport: `copilot -p` needs no probe of its own."""
+ base, key, bearer = self.optimizer_endpoint
+ preflight_optimizer_endpoint(
+ base_url=base, api_key=key, bearer=bearer,
+ configured_model=self.optimizer_model, label="copilot",
+ remedy="Check COPILOT_PROVIDER_BASE_URL and the credential that goes with it.")
+
+ # ---- 1. the paid optimizer call ---------------------------------------------------
+ def optimizer_call(self, prompt, model="agent"):
+ """One between-task proposer / judge / nudge-writer call.
+
+ Over the Messages API when this lane's BYOK provider speaks it and so bills what
+ its TASKS bill (`self.optimizer_endpoint`, resolved at construction), else through
+ the host `copilot` binary with every tool denied and excluded and custom
+ instructions off, so either way it is a pure text completion that cannot read the
+ testbed.
+
+ Both are priced on the same weight table the task rows use, and `optimizer_model`
+ is the agent's own model on both -- only the wire differs. `run_config.json` records
+ which one ran, because the two are NOT comparable on overhead.
+ """
+ if self.optimizer_endpoint:
+ base, key, bearer = self.optimizer_endpoint
+ return anthropic_optimizer_call(
+ prompt, model=model, configured_model=self.optimizer_model,
+ max_tokens=self.optimizer_max_tokens,
+ api_key=key, bearer=bearer, base_url=base)
+ return copilot.optimizer_call(
+ prompt, binary=self.copilot,
+ model=(model if model not in ("agent", "copilot") else self.model))
+
+ # ---- 2. container plumbing --------------------------------------------------------
+ def _start(self, name, instance_id):
+ """Same as the Claude path but mounting the three pieces of the Copilot install and
+ seeding a writable $HOME from them. The agent runs as `agent`, not root, because
+ Copilot's config/cache layout assumes a normal home."""
+ _run(["docker", "rm", "-f", name])
+ self._reap_exited(name)
+ cmd = (["docker", "run", "-d", "--name", name, "--entrypoint", "bash"]
+ + copilot.container_mounts(self.copilot)
+ + ["-e", "HOME=/home/agent", "-w", "/testbed"] + _cpu_limit_args())
+ _LIVE_CONTAINERS.add(name)
+ p = _run(cmd + [image_for(instance_id), "-lc", "sleep infinity"],
+ timeout=int(os.environ.get("TE_DOCKER_START_TIMEOUT", "900")))
+ if p.returncode != 0:
+ raise RuntimeError(f"docker run failed for {instance_id}: {p.stderr[-500:]}")
+ # Run as ROOT with HOME=/home/agent -- the configuration the AgentEO study used.
+ # SWE-bench images have no unprivileged user (adding one and chowning /testbed is
+ # both slower and a behavioural difference from the published rows), and unlike
+ # Claude Code the Copilot CLI does not refuse to run as root.
+ setup = ("mkdir -p /home/agent " + copilot.CONTAINER_LOG_DIR + "; "
+ + copilot.container_seed_setup())
+ r = _run(["docker", "exec", name, "bash", "-lc", setup], timeout=600)
+ if r.returncode != 0:
+ _log(f"copilot seed setup warned: {(r.stderr or '')[-300:]}")
+ self._activate_project_env(name)
+ return name
+
+ def _register_net_guard(self, name, python):
+ """Copilot reads hooks from `$COPILOT_HOME/settings.json`, not `.claude/`.
+
+ Same *shape* as the Claude path -- merge a PreToolUse entry into a settings file --
+ but three details differ, and all three were measured against Copilot CLI 1.0.83
+ on 2026-09-08 rather than assumed:
+
+ * **Which file.** `$COPILOT_HOME/settings.json`, which in the container is
+ `/home/agent/.copilot/settings.json` (HOME is `/home/agent`, seeded per task
+ and thrown away with the container). The inherited implementation wrote
+ `.claude/settings.json`, which this CLI never reads -- so it succeeded,
+ reported armed, and guarded nothing. Note `hooks.json` at that level is
+ **not** read; only `settings.json` is.
+ * **The spelling.** `preToolUse` and matcher `bash` -- camelCase and lower-case,
+ against Claude's `PreToolUse` and `Bash`, and a flat entry rather than
+ Claude's nested `hooks` list. A wrong matcher never fires.
+ * **No plugin.** `baselines/rtk.py` reaches the same hook through
+ `copilot plugin install`, which works but is the install form §13 q7 records
+ as deprecated. A settings entry needs no plugin, so the guard does not inherit
+ that clock. It also means the guard and rtk do not contend: they land in
+ different places, and two independent `bash` hooks were verified to both fire
+ on one call.
+
+ Merged rather than overwritten because `container_seed_setup` copies the host's
+ config into this dir first, and that may already carry a `settings.json`.
+ """
+ entry = {"matcher": "bash", "type": "command",
+ "command": f"{python} {self.NET_GUARD_SCRIPT}"}
+ settings = f"{self.scaffold_profile.home}/.copilot/settings.json"
+ prog = (
+ "import json, os, sys\n"
+ "p = sys.argv[1]\n"
+ f"entry = json.loads({json.dumps(json.dumps(entry))})\n"
+ "os.makedirs(os.path.dirname(p), exist_ok=True)\n"
+ "try:\n"
+ " cur = json.load(open(p))\n"
+ "except Exception:\n"
+ " cur = {}\n"
+ "pre = cur.setdefault('hooks', {}).setdefault('preToolUse', [])\n"
+ "if not any('net_guard' in json.dumps(h) for h in pre):\n"
+ " pre.append(entry)\n"
+ "json.dump(cur, open(p, 'w'), indent=2)\n"
+ "print('armed')\n")
+ r = self._exec(
+ name,
+ f"cat > /tmp/te_net_guard_install.py <<'TE_EOF'\n{prog}\nTE_EOF\n"
+ f"{python} /tmp/te_net_guard_install.py {settings}", timeout=60)
+ if r.returncode != 0 or "armed" not in (r.stdout or ""):
+ raise RuntimeError(((r.stderr or "") + (r.stdout or ""))[-200:])
+ return True
+
+ # ---- 3. one streamed agent segment ------------------------------------------------
+ def _segment(self, name, prompt, transcript_fh, exclude_tools, max_turns, guard,
+ mcp_config=None):
+ """Run `copilot -p` once, streaming its JSONL. Returns a usage dict (NOT a Claude
+ `result` event) -- see the accounting hooks below.
+
+ `max_turns` is accepted and ignored: Copilot has no turn-cap flag. The spiral guard
+ and TE_TASK_TIMEOUT are the bounds.
+ """
+ # A DEDICATED log dir per segment. The spiral path re-invokes in the same container,
+ # and the read-back below globs the whole dir -- so a shared dir would make segment
+ # 2 re-read segment 1's usage, which _merge_segments then adds to it again. Both
+ # segments' tokens would be counted twice, inflating exactly the arm the spiral
+ # mechanism is supposed to show saving on.
+ seg = getattr(guard, "segment", 0)
+ logdir = f"{copilot.CONTAINER_LOG_DIR}/seg{seg}"
+ _run(["docker", "exec", name, "mkdir", "-p", logdir])
+
+ # `mcp_config` carries a scaffold's MCP server (Serena). build_argv owns the flag
+ # spelling -- Copilot wants `--additional-mcp-config @<path>`, not Claude's
+ # `--mcp-config <path>` -- which is why the profile appends nothing to argv here.
+ argv = copilot.build_argv(prompt, log_dir=logdir, model=self.model,
+ effort=self.effort, exclude_tools=exclude_tools,
+ mcp_config=mcp_config)
+ cmd = (["docker", "exec", "-i", "-w", "/testbed", "-e", "HOME=/home/agent"]
+ + self._env_activation_args(name)) # project env, see _activate_project_env
+ # BYOK credentials are forwarded BY NAME only (docker exec -e NAME reads the
+ # client's environment), so the secret never reaches argv on a shared host.
+ for k in copilot.provider_secret_names():
+ cmd += ["-e", k]
+ for k, v in copilot.run_env().items():
+ cmd += ["-e", f"{k}={v}"]
+ cmd += [name] + argv
+
+ proc = subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, text=True, bufsize=1)
+ stream_lines, stopped = [], False
+ deadline = time.time() + self.task_timeout
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ stream_lines.append(line)
+ # The vendored guard's completed_calls() understands BOTH the Copilot schema
+ # (tool.execution_start/complete) and Claude's, so no translation is needed.
+ if guard is not None and guard.monitor(line):
+ stopped = True
+ break
+ if time.time() > deadline:
+ _log("segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped = True
+ break
+
+ if stopped:
+ # Kill INSIDE the container: killing the `docker exec` client leaves the CLI
+ # running and still billing.
+ _run(["docker", "exec", name, "pkill", "-TERM", "-f", "copilot"])
+ time.sleep(2)
+ _run(["docker", "exec", name, "pkill", "-KILL", "-f", "copilot"])
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-600:]
+
+ # A configuration the CLI refuses outright fails IDENTICALLY on every task, so
+ # stop the sweep here rather than filling the results tree with 0-turn rows that
+ # look like the agent tried and failed.
+ bad = copilot.fatal_config_error(err)
+ if bad:
+ raise RuntimeError(
+ f"copilot refused this configuration: {bad}\n"
+ f" model={self.model!r} effort={self.effort!r}\n"
+ " fix COPILOT_MODEL / TE_EFFORT_LEVEL (see docs/COPILOT.md step 2); "
+ "COPILOT_MODEL=auto always works but does not apply an effort level.")
+
+ log_text = _run(["docker", "exec", name, "bash", "-lc",
+ f"cat {logdir}/*.log 2>/dev/null"], timeout=180).stdout or ""
+ usage = copilot.row_usage("\n".join(stream_lines), log_text)
+ if err.strip() and not usage.get("num_turns"):
+ _log(f"agent stderr: {err.strip()[:300]}")
+ # Stash the debug log for the untooling catalog parser: trace.parse_catalog(run_dir,
+ # 'copilot') reads exactly this filename. Without it the tool judge is handed
+ # empty descriptions, drops nothing, and still bills for the call.
+ usage["_debug_log"] = log_text
+ return usage
+
+ # ---- 3b. accounting hooks ----------------------------------------------------------
+ def _merge_segments(self, a, b):
+ merged = copilot.merge_usage(a, b)
+ merged["_debug_log"] = (a or {}).get("_debug_log", "") + (b or {}).get("_debug_log", "")
+ return merged
+
+ def _row_from_segment(self, result):
+ row = dict(result or {})
+ row.setdefault("is_error", False)
+ # Copilot exposes no per-run USD figure (only a premium-request count), so the
+ # vendor cost stays null rather than a fabricated zero.
+ row.setdefault("cost_usd_vendor", None)
+ # Same degradation predicate as the Claude path, so one filter covers both agents.
+ # Copilot never needed _UsageRecovery -- it reads the whole log dir back AFTER the
+ # kill (`cat {logdir}/*.log`), which is why the audit found zero affected Copilot
+ # rows in 2,571. Its own failure mode is a log dir it could not parse, which
+ # `row_usage` already reports as tokens_source="stream_output_only" with None (not
+ # 0) in the unaccounted fields.
+ src = row.get("tokens_source")
+ row["usage_incomplete"] = src is not None and src != "debug_log"
+ row["output_tokens_missing"] = row.get("output_tokens") is None
+ row["num_turns_recovered"] = False # turn_start events survive the kill
+ if row["usage_incomplete"]:
+ row.setdefault("usage_incomplete_reason", f"tokens_source={src}")
+ return row # _debug_log rides along; run_task persists and strips it
+
+ # ---- 4. entry point: same lifecycle + the debug log the catalog parser needs --------
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ row, out_dir = super().run_task(
+ task, arm=arm, injected=injected, exclude_tools=exclude_tools,
+ run_dir_path=run_dir_path, **run_kwargs)
+ # trace.parse_catalog(run_dir, 'copilot') reads exactly this filename. Without it
+ # the untooling judge is handed tool names with EMPTY descriptions, drops nothing,
+ # and still bills for the call -- the mechanism silently measures itself as useless.
+ log_text = row.pop("_debug_log", "")
+ if log_text:
+ try:
+ with open(os.path.join(out_dir, "copilot_debug.log"), "w",
+ encoding="utf-8") as fh:
+ fh.write(log_text)
+ except OSError as e:
+ _log(f"could not save copilot_debug.log: {e}")
+ else:
+ _log("no copilot debug log recovered -- untooling catalog will be empty "
+ "and token accounting falls back to stream output only")
+ return row, out_dir
+
+
+# Scaffolds whose mechanism this lane cannot deliver, and why. Refused at construction:
+# each would install successfully and then do nothing, which reads as a scaffold result.
+# Every scaffold is hostable on this lane, each through the channel Codex actually has:
+# rtk -- its own Codex mode (`rtk init -g --codex`, instruction files under CODEX_HOME;
+# the hook mode is one this CLI ignores), measured by `rtk_compliance`;
+# repomap -- project instructions, which the profile declares as AGENTS.md;
+# serena -- an MCP server, which Codex takes from `[mcp_servers.*]` in config.toml rather
+# than an argv flag: the profile's `mcp_args` is empty on purpose and
+# `_apply_scaffold` rewrites the config with the scaffold's spec. Measured by
+# `serena_tool_calls`, because a configured server is not a used one.
+# A refusal table used to live here for rtk and serena; both refusals were about wiring this
+# harness had not done, not about the tools.
+
+
+class CodexSwebenchRunner(ClaudeSwebenchRunner):
+ """AgentRunner: the OpenAI Codex CLI (`codex exec --json`) in the instance's pinned
+ image, graded by the official SWE-bench harness. It bills the OpenAI API directly on
+ OPENAI_API_KEY (codex.resolve_codex_routing).
+
+ Four things differ from the Claude runner; everything else -- container lifecycle,
+ spiral segments, patch extraction, official gold grading -- is inherited.
+
+ 1. **The endpoint is a FILE.** Codex reads its provider and base URL from
+ `$CODEX_HOME/config.toml`, not from the environment. The file is rewritten per
+ task because it also carries this task's frozen capability mask.
+ 2. **The tool mask is a capability mask.** Codex has no `--disallowed-tools`. The
+ Codex-native untooling mechanism emits logical capability packages and
+ `codex.config_toggles` maps them onto verified config toggles.
+ 3. **Its own usage schema.** `input_tokens` in Codex's usage block is the GRAND
+ input total including both cache fields; see `codex.row_usage`.
+ 4. **No turn cap.** Codex has no `--max-turns`, the same caveat the Copilot lane
+ carries: `max_turns` is recorded and counted by the envelope but not enforced.
+ TE_TASK_TIMEOUT and the spiral guard are the only bounds.
+ """
+
+ # Codex reads project instructions from AGENTS.md and navigates the repo THROUGH the
+ # shell (cat/grep/sed) rather than with separate nav tools, so every read/search byte
+ # lands in the shell bucket and `nav_tools` is empty. It takes no MCP config on this
+ # path, so nothing is appended to argv.
+ scaffold_profile = scaffolds_pkg.AgentProfile(
+ kind="codex",
+ home="/tmp/agent-home",
+ instructions_file="AGENTS.md",
+ nav_tools=(),
+ # Empty on purpose, not because Codex lacks MCP: it configures servers in
+ # config.toml (`[mcp_servers.<name>]`), and _apply_scaffold below writes them there.
+ mcp_args=lambda cfg: [],
+ patch_excludes=("AGENTS.md", "RTK.md"),
+ # Where the CLI reads its global instructions from. A scaffold's install must run
+ # under this, not under $HOME -- see AgentProfile.env.
+ env={"CODEX_HOME": codex.C_HOME})
+
+ def __init__(self, agent="codex", model=None, grading_path=None, scaffold=None):
+ self.agent = agent
+ # The OpenAI API directly -- OPENAI_BASE_URL only names OpenAI's own regional
+ # endpoint, for a region-scoped key. Resolved from the environment alone: see
+ # codex.resolve_codex_routing.
+ self.routing = codex.resolve_codex_routing()
+ self.routing_base = self.routing["base_url"]
+ self.api_key = self.routing["api_key"]
+ # The BILLED model. `gpt-5.6-sol` was probed on 2026-09-09 and is both listed and
+ # ACCEPTED on /v1/responses by a direct OpenAI key. The `-codex` ALIASES are the
+ # trap -- /v1/models lists them and /v1/responses 404s them -- which is why the
+ # check below is a catalog lookup AND why a bad id still has to survive to a real
+ # call to be caught. Rather than guess, verify_model refuses the lane with the
+ # served list in the message.
+ self.model = model or os.environ.get("CODEX_MODEL") or "gpt-5.6-sol"
+ codex.validate_priceable(self.model, self.agent)
+ _log(f"codex: model={self.model} endpoint={self.routing_base}")
+ # Same rule as both other agents: the between-task optimizer runs on the AGENT'S
+ # OWN model so mechanism overhead is priced on the same basis as the task rows.
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "
+ "runs on the agent's own model so mechanism overhead is priced on the "
+ "same basis as the task rows. Unset it.")
+ self.optimizer_model = self.model
+ self.optimizer_transport = "api"
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ # Reasoning effort is part of the cost basis, so it is set explicitly rather than
+ # left to whatever the installed CLI defaults to.
+ self.effort = (os.environ.get("TE_EFFORT_LEVEL")
+ or os.environ.get("CODEX_EFFORT_LEVEL") or "high")
+ if self.effort not in codex.EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort!r} is not one of "
+ f"{sorted(codex.EFFORT_LEVELS)}")
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile)
+ _log(f"scaffold: {self.scaffold.name} (external baseline)")
+ # Resolve the install BEFORE anything else touches Docker: a bind mount of a
+ # missing vendor dir becomes an EMPTY dir in the container rather than an error.
+ self.codex_shim, self.codex_vendor = codex.require_install()
+ self.agent_bin = codex.C_BIN
+ self.protected_tools = codex.PROTECTED_TOOLS
+ self.cli_version = codex.cli_version(self.codex_shim)
+ # PROVE THE CONTRACT PARSES, against this CLI build, before task 0. The accepted
+ # spelling of the pruning toggles moved between codex-cli 0.146 and 0.153, and an
+ # unrecognized field is IGNORED rather than rejected unless --strict-config is
+ # passed -- so a CLI upgrade would silently turn the untooling arm into a baseline
+ # wearing its name. Validate the widest config this lane can emit (every prunable
+ # capability at once), so the check covers masks later tasks may freeze.
+ # The `skills` package is pruned by disabling each bundled system skill by path, so
+ # the set is read from THIS CLI build rather than assumed (a CLI upgrade that ships
+ # new bundled skills would otherwise leave them listed, a silent loss of saving).
+ self.system_skills = tuple(codex.bundled_system_skills(binary=self.codex_shim))
+ if not self.system_skills:
+ self.system_skills = codex.DEFAULT_SYSTEM_SKILLS
+ _log("codex: could not discover the CLI's bundled system skills -- using the "
+ f"0.153.1 set {list(self.system_skills)}")
+ else:
+ _log(f"codex: bundled system skills on this CLI: {list(self.system_skills)}")
+ # The instruction package needs the base prompt captured for THIS CLI build; with
+ # no capture the package stays loaded and the lane says so, rather than applying a
+ # prompt recorded from another version.
+ self.base_prompt = codex.base_prompt_for(self.cli_version)
+ if self.base_prompt:
+ _log(f"codex: base prompt capture found for {self.cli_version!r} "
+ f"({len(self.base_prompt) // 4} tokens); the interactive_guidance and "
+ "skills how-to sections can be pruned through model_instructions_file")
+ else:
+ _log(f"codex: no base prompt capture for {self.cli_version!r} -- the "
+ "interactive_guidance package cannot be applied on this lane and will "
+ "stay loaded; capture this CLI's base prompt into "
+ "te_harness/data/codex_base_prompt/<version>.md to enable it")
+ probe_cfg = codex.config_toml(
+ routing=self.routing, model=self.model, effort=self.effort,
+ capabilities=sorted(codex.CONFIG_CAPABILITIES), system_skills=self.system_skills,
+ instructions=codex.lean_instructions(self.base_prompt,
+ sorted(codex.CONFIG_CAPABILITIES)))
+ bad = codex.validate_config(probe_cfg, binary=self.codex_shim)
+ if bad:
+ raise RuntimeError(
+ "codex refused the generated config.toml, so the request contract this "
+ "lane would send is not the one it describes:\n" + bad +
+ "\n A renamed toggle is IGNORED without --strict-config, which is why "
+ "this is checked here: the untooling arm would otherwise run as a "
+ "baseline under its own name. Update codex._TOP_LEVEL_TOGGLES / "
+ "_TABLE_TOGGLES for this CLI build.")
+ # Refuse a lane whose endpoint does not serve the id it intends to bill.
+ served, why = codex.verify_model(self.model, self.routing)
+ if served is False:
+ raise RuntimeError("codex model check failed: " + why)
+ if served is None:
+ _log(f"codex: {why} -- an unserved id would fail per task, so check "
+ f"`curl {self.routing_base}/models` by hand before a long run")
+ self.grader = _Grader(grading_path)
+ self.task_timeout = int(os.environ.get("TE_TASK_TIMEOUT", "1800"))
+ self.eval_timeout = int(os.environ.get("TE_EVAL_TIMEOUT", "1800"))
+ self.keep = os.environ.get("TE_KEEP_CONTAINERS") == "1"
+ # Per-task state the segment builder needs but run_task does not thread: the
+ # frozen capability mask, and the resume thread id.
+ self._capabilities = ()
+ self._thread_id = ""
+ codex.preflight_optimizer(routing=self.routing, configured_model=self.model)
+
+ # ---- 1. the paid optimizer call ---------------------------------------------------
+ def optimizer_call(self, prompt, model="agent"):
+ """One between-task proposer / judge / nudge-writer call, over /v1/responses.
+
+ Against this lane's own endpoint and on this lane's own model, so mechanism spend
+ bills what its tasks bill. See codex.optimizer_call.
+ """
+ return codex.optimizer_call(
+ prompt, configured_model=(self.model if model in ("agent", "codex") else model),
+ routing=self.routing, max_tokens=self.optimizer_max_tokens)
+
+ # ---- 2. container plumbing --------------------------------------------------------
+ def _start(self, name, instance_id):
+ """Same as the Claude path but mounting the Codex vendor dir instead of a single
+ binary. The ELF is static-PIE and finds its siblings (`rg`, the sandbox helpers)
+ relative to itself, so one read-only mount is the whole install."""
+ _run(["docker", "rm", "-f", name])
+ self._reap_exited(name)
+ cmd = (["docker", "run", "-d", "--name", name, "--entrypoint", "bash"]
+ + codex.container_mounts(self.codex_vendor)
+ + ["-e", "HOME=/tmp/agent-home", "-w", "/testbed"] + _cpu_limit_args())
+ if self.scaffold is not None:
+ for spec in self.scaffold.mounts(self.scaffold_profile):
+ cmd += ["-v", spec]
+ _LIVE_CONTAINERS.add(name)
+ p = _run(cmd + [image_for(instance_id), "-lc", "sleep infinity"],
+ timeout=int(os.environ.get("TE_DOCKER_START_TIMEOUT", "900")))
+ if p.returncode != 0:
+ raise RuntimeError(f"docker run failed for {instance_id}: {p.stderr[-500:]}")
+ # The agent home too: the net guard's script is `docker cp`'d into it, and docker cp
+ # into a directory that does not exist fails -- the way the Copilot lane once ran
+ # unguarded for a whole sweep.
+ _run(["docker", "exec", name, "mkdir", "-p", codex.C_HOME, self.scaffold_profile.home])
+ # The contract is written HERE, not as a side effect of arming the net guard.
+ # `_install_net_guard` returns early when TE_ALLOW_NETWORK_TOOLS=1 lifts the guard,
+ # and with the write hanging off `_register_net_guard` that lane got NO config.toml
+ # at all -- no provider block, so every task failed to reach any endpoint. The
+ # guard's hook goes in hooks.json beside this file, never into it.
+ self._write_config(name)
+ self._activate_project_env(name)
+ return name
+
+ def _write_config(self, name):
+ """Install this task's `config.toml` -- the request contract -- in the container.
+
+ Written per task, not per container, because it carries the capability mask the
+ mechanism froze for THIS task. Base64'd through a single exec so no quoting in the
+ TOML can be reinterpreted by the shell.
+ """
+ lean = codex.lean_instructions(getattr(self, "base_prompt", None),
+ self._capabilities)
+ self._lean_instructions_tokens = len(lean) // 4 if lean else 0
+ cfg = codex.config_toml(
+ routing=self.routing, model=self.model, effort=self.effort,
+ capabilities=self._capabilities,
+ system_skills=getattr(self, "system_skills", None), instructions=lean,
+ mcp_servers=getattr(self, "_mcp_servers", None))
+ # config.toml, plus the lean base prompt when the frozen mask prunes an
+ # instruction package (it is what `model_instructions_file` in the config names).
+ # Hooks are NOT declared here: the net guard writes them to hooks.json beside this
+ # file (`_register_net_guard`). One config layer carrying both representations makes
+ # Codex warn and prefer one, so the contract keeps exactly one.
+ blob = base64.b64encode(cfg.encode()).decode()
+ script = (f"mkdir -p {codex.C_HOME} && echo {blob} | base64 -d > "
+ f"{codex.C_HOME}/config.toml")
+ if lean:
+ lblob = base64.b64encode(lean.encode()).decode()
+ script += f" && echo {lblob} | base64 -d > {codex.INSTRUCTIONS_PATH}"
+ r = self._exec(name, script + " && echo wrote", timeout=60)
+ if "wrote" not in (r.stdout or ""):
+ raise RuntimeError(
+ "could not install the codex config.toml: "
+ + ((r.stderr or "") + (r.stdout or ""))[-200:])
+ return cfg
+
+ # ---- 2b. the network guard --------------------------------------------------------
+ # The interpreter the PreToolUse hook runs under once `_register_net_guard` has armed it,
+ # "" when it is not armed. `_segment` reads it: an armed task passes
+ # `--dangerously-bypass-hook-trust`, because Codex skips every hook a person has not
+ # approved in /hooks and an unattended container has no way to approve one.
+ _guard_python = ""
+
+ def _register_net_guard(self, name, python):
+ """Codex: the guard's PreToolUse entry in `$CODEX_HOME/hooks.json`.
+
+ Codex DOES honour it. Verified live on codex-cli 0.153.1 (2026-09-30, gpt-5.6-sol,
+ direct API) with exactly this file and `--dangerously-bypass-hook-trust`: the guard
+ refused `echo https://github.com/probe` before it ran -- "Command blocked by
+ PreToolUse hook: Blocked: this benchmark's grading specs are public on GitHub ..." --
+ and logged the refusal, while a control command in the same turn ran normally.
+
+ The 2026-09-09 conclusion that Codex ignores the hook was a configuration error, not
+ the CLI: those probes keyed the event kebab-case (`pre-tool-use`; Codex reads
+ PascalCase `PreToolUse`, as Claude Code does), gave the handler an argv list rather
+ than a command string, and ran without the trust bypass. Codex rows stored before
+ 2026-09-30 therefore ran without the hook, guarded by the hosts block alone
+ (docs/HARNESS.md, "The answer-key guard").
+
+ The rest is the shared lifecycle's: the script is copied in, the interpreter found,
+ and the guard made to refuse its probe before the lane reports itself armed.
+ """
+ self._write_hook_entry(name, python, f"{codex.C_HOME}/hooks.json")
+ self._guard_python = python
+ return True
+
+ # ---- 2c. scaffolds that are MCP servers ---------------------------------------------
+ def _apply_scaffold(self, name, task, run_dir_path):
+ """Same lifecycle, then wire any MCP server the scaffold asked for into config.toml.
+
+ Ordering is the whole point: `_start` writes config.toml before the scaffold exists,
+ and Codex reads MCP servers from that file rather than from argv, so a scaffold's
+ `mcp_servers` spec has to be written AFTER setup by rewriting the config. The JSON the
+ scaffold also wrote (`mcp_config`) is provenance on this lane; the argv seam that
+ consumes it on Claude is a no-op here.
+ """
+ self._mcp_servers = None
+ notes, mcp_config = super()._apply_scaffold(name, task, run_dir_path)
+ spec = notes.pop("mcp_servers", None) if isinstance(notes, dict) else None
+ if spec:
+ self._mcp_servers = spec
+ self._write_config(name)
+ notes["mcp_wiring"] = "config.toml"
+ notes["mcp_servers_configured"] = sorted(spec)
+ return notes, mcp_config
+
+ # ---- 3. one rollout segment -------------------------------------------------------
+ def _segment(self, name, prompt, transcript_fh, exclude_tools, max_turns, guard,
+ mcp_config=None):
+ """Run `codex exec --json` once (or `codex exec resume` for a spiral segment).
+
+ `prompt` is a string for a fresh segment, or the resume REQUEST dict the Codex
+ guard returned -- `{mode, session_id, prompt, effort}`. Resuming the exact thread
+ rather than starting a new session is the mechanism: a fresh session would re-pay
+ for the context the stalled one already built.
+
+ `max_turns` is accepted and ignored (Codex has no turn-cap flag). Returns a usage
+ dict, NOT a Claude `result` event -- see the accounting hooks.
+ """
+ resume = prompt if isinstance(prompt, dict) else None
+ text = resume["prompt"] if resume else prompt
+ hooked = bool(self._guard_python)
+ if resume:
+ argv = codex.build_resume_argv(
+ resume["session_id"], effort=resume.get("effort"), hook=hooked)
+ else:
+ argv = codex.build_argv(hook=hooked)
+ cmd = ["docker", "exec", "-i", "-w", "/testbed"]
+ cmd += self._env_activation_args(name) # project env, see _activate_project_env
+ # The key is forwarded BY NAME only (`docker exec -e VAR` reads the client's
+ # environment), so it never reaches argv -- readable by every other user via `ps`
+ # on a shared host.
+ cmd += ["-e", codex.KEY_ENV]
+ env = codex.run_env({net_guard.LOG_ENV: self.NET_GUARD_LOG})
+ for k, v in env.items():
+ cmd += ["-e", f"{k}={v}"]
+ cmd += [name] + argv
+
+ cenv = dict(os.environ)
+ cenv[codex.KEY_ENV] = self.api_key
+ proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, text=True, bufsize=1, env=cenv)
+ proc.stdin.write(text)
+ proc.stdin.close()
+
+ lines, stopped = [], False
+ deadline = time.time() + self.task_timeout
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ lines.append(line)
+ if guard is not None and guard.monitor(line):
+ stopped = True
+ break
+ if time.time() > deadline:
+ _log("segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped = True
+ break
+
+ if stopped:
+ # Kill INSIDE the container: terminating the `docker exec` client leaves the
+ # CLI running and still billing. Match the process name EXACTLY -- a
+ # `pkill -f codex` also matches the harness's own helpers.
+ _run(["docker", "exec", name, "pkill", "-TERM", "-x", "codex"])
+ time.sleep(2)
+ _run(["docker", "exec", name, "pkill", "-KILL", "-x", "codex"])
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-600:]
+
+ # A configuration the CLI refuses fails IDENTICALLY on every task, so stop the
+ # sweep here rather than filling the tree with 0-turn rows that look like the agent
+ # tried. `validate_config` already covers the field NAMES at construction; this
+ # catches what only a real run can (an unknown model, a dead provider block).
+ if "Error loading config.toml" in err or "unknown configuration field" in err:
+ raise RuntimeError(
+ "codex refused its config at run time:\n" + err.strip()[:600])
+
+ usage = codex.row_usage("\n".join(lines))
+ if err.strip() and not usage.get("num_turns"):
+ _log(f"agent stderr: {err.strip()[:300]}")
+ return usage
+
+ # ---- 3a. spiral hooks -------------------------------------------------------------
+ def _build_guard(self, run_kwargs):
+ """The Codex-native post-green finalize controller, when the arm carries one.
+
+ `codex_spiral_policy` arrives already frozen at the repository's learned
+ resolved-turn threshold (`mechanisms.spiral.SpiralStage.run_kwargs`); a lane with
+ no threshold yet passes None and runs unguarded, which is the cold-start behaviour
+ the mechanism intends.
+ """
+ policy = run_kwargs.get("codex_spiral_policy")
+ if policy is None:
+ return None
+ return codex_spiral.CodexResumeGuard(
+ policy=policy, verbose=os.environ.get("TE_VERBOSE", "1") == "1")
+
+ def _guard_continuation(self, guard, prompt):
+ """The Codex guard resumes a thread, so it takes no prompt and returns a request."""
+ return guard.continuation()
+
+ def _guard_row(self, guard):
+ """Telemetry under the `codex_spiral_*` keys `SpiralStage.after_task` reads.
+
+ `aborted` is always False and that is the mechanism's defining property rather
+ than an omission: the post-green finalize NEVER terminally aborts a task, it only
+ redirects one that has already passed its own success check.
+ """
+ tel = guard.telemetry()
+ return {
+ "aborted": False,
+ # The guard's OWN count of command+edit events, which is the unit its trigger
+ # compares against the envelope. The row's `num_turns` is agent STEPS (messages
+ # + tool calls) and runs larger, so feeding that to the envelope tracker raises
+ # the gate and the controller barely fires. `SpiralStage.after_task` reads this
+ # field.
+ "codex_spiral_tool_turns": (tel.get("snapshot") or {}).get("tool_turns"),
+ "codex_spiral_nudges": tel["nudges"],
+ "codex_spiral_events": tel["events"],
+ "codex_spiral_thread_captured": tel["thread_id_captured"],
+ "codex_spiral_paid_judge_tokens": tel["paid_judge_effective_tokens"],
+ "codex_spiral_policy": tel["policy"],
+ }
+
+ # ---- 3b. accounting hooks ---------------------------------------------------------
+ def _merge_segments(self, a, b):
+ return codex.merge_usage(a, b)
+
+ def _row_from_segment(self, result):
+ row = dict(result or {})
+ row.setdefault("is_error", False)
+ row.setdefault("cost_usd_vendor", None)
+ # Same degradation predicate as the other two agents, so one filter covers all
+ # three. Codex's own failure mode is a stream killed before `turn.completed`, which
+ # takes the usage block with it -- row_usage reports that as tokens_source
+ # no_turn_event with None (not 0) in every token field, so such a task is
+ # filterable instead of being averaged in as free.
+ src = row.get("tokens_source")
+ row["usage_incomplete"] = src is not None and src != "stream_usage"
+ row["output_tokens_missing"] = row.get("output_tokens") is None
+ row["num_turns_recovered"] = False
+ if row["usage_incomplete"]:
+ row.setdefault("usage_incomplete_reason", f"tokens_source={src}")
+ return row
+
+ # ---- 4. entry point ---------------------------------------------------------------
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ """Same lifecycle, with the capability mask frozen before the container starts.
+
+ The mask reaches the runner two ways and both are needed. `exclude_tools` carries
+ the RELAY capability ids (the mechanism's `build_tools` return), and
+ `codex_config_capabilities` carries the logical packages that have a supported
+ native config toggle. The second is not derivable from the first: the
+ `collaboration` package expands to NO relay id, so a run that read only
+ `exclude_tools` would prune it from nothing while reporting it pruned.
+ """
+ self._capabilities = tuple(run_kwargs.get("codex_config_capabilities") or ())
+ # Per task: `_start` writes config.toml before this task's scaffold has run, and a
+ # previous task's MCP spec must not ride along in that first write.
+ self._mcp_servers = None
+ self._guard_python = ""
+ row, out_dir = super().run_task(
+ task, arm=arm, injected=injected, exclude_tools=exclude_tools,
+ run_dir_path=run_dir_path, **run_kwargs)
+ row["codex_config_capabilities"] = list(self._capabilities)
+ row["codex_lean_instructions_tokens"] = int(
+ getattr(self, "_lean_instructions_tokens", 0) or 0)
+ if self.scaffold_name == "rtk":
+ # Instruction-based on this agent, so the measure of the arm is whether the
+ # agent complied. Read off the trace, like answer_key_attempts.
+ row.update(codex.rtk_compliance(os.path.join(out_dir, "transcript.jsonl")))
+ if self.scaffold_name == "serena":
+ # A configured MCP server is not a used one: count the calls.
+ row.update(codex.mcp_tool_calls(os.path.join(out_dir, "transcript.jsonl"),
+ server="serena"))
+ if row.get("mcp_item_sample") and not row.get("mcp_tool_calls"):
+ _log(f"{task['instance_id']}: MCP item with an unrecognised shape -- "
+ f"pin it: {row['mcp_item_sample'][:200]}")
+ return row, out_dir
+
+
+# Which runner each --agent label selects. A label with no entry is an error rather than a
+# silent fallback: the old zero-arg factory always returned the Claude runner, so
+# `--agent copilot` ran Claude Code and filed the rows under a copilot/ tree -- wrong
+# numbers rather than a crash, which is the worst failure mode a benchmark can have.
+RUNNERS = {
+ "claude": ClaudeSwebenchRunner,
+ "copilot": CopilotSwebenchRunner,
+ "codex": CodexSwebenchRunner,
+}
+
+
+def make_runner(agent=None, scaffold=None, **kwargs):
+ """Factory the driver's ``--runner te_harness.swebench:make_runner`` calls.
+
+ Called with the driver's ``--agent`` label when the driver supports it, and with no
+ arguments by older callers (which then get the TE_AGENT default, or claude).
+
+ ``scaffold`` names an external tool baseline (rtk / serena / repomap) to install into
+ every task container; the driver derives it from the arm, so the arm registry stays the
+ single source of truth for which arm carries which scaffold. ``TE_SCAFFOLD`` is the
+ fallback for a caller that cannot pass it.
+ """
+ agent = (agent or os.environ.get("TE_AGENT") or "claude").lower()
+ if agent not in RUNNERS:
+ raise RuntimeError(
+ f"no SWE-bench runner for --agent {agent!r}; "
+ f"available: {', '.join(sorted(RUNNERS))}")
+ _install_signal_cleanup()
+ scaffold = scaffold or os.environ.get("TE_SCAFFOLD") or None
+ return RUNNERS[agent](agent=agent, scaffold=scaffold, **kwargs)
diff --git a/experiments/te_harness/webbench.py b/experiments/te_harness/webbench.py
new file mode 100644
index 0000000..1b26112
--- /dev/null
+++ b/experiments/te_harness/webbench.py
@@ -0,0 +1,2403 @@
+"""Web-Bench task executor for TokenEconomizer: real Claude Code, real Playwright grade.
+
+Web-Bench inverts SWE-bench's control flow. Its own Node evaluator owns the task loop --
+copy the previous task's workspace forward, collect the current file set, ask the agent for
+files, write them back, run the project's init/build, screenshot, then grade with the
+project's real Playwright suite (cumulatively: `init` + `task-1..task-N`). The
+economizer's driver owns ITS loop: per-task injection prefix, tool mask, spiral params, one
+``run_task`` call per task.
+
+This runner joins the two loops at the evaluator's own HTTP agent seam, so neither side is
+reimplemented here. Per project it starts
+
+ * a serialized HTTP server speaking the evaluator's agent contract
+ (POST -> ``{type, task, files, error}``; reply -> ``{files, inputTokens, outputTokens}``), and
+ * ``@web-bench/eval`` restricted to that one project, pointed at that server,
+
+and then ``run_task(task)`` blocks on a rendezvous: it takes the evaluator's request for
+that task, runs Claude Code over the request's files in a scratch workspace, answers with
+the files that changed, and waits for the evaluator's verdict line for that task before
+returning the row. The agent therefore sees exactly the files Web-Bench shows any agent,
+and ``resolved`` is Web-Bench's own graded verdict.
+
+Two consequences of the benchmark's contract are worth stating plainly, because they shape
+what the economizer's mechanisms can observe:
+
+ * The agent gets files, not an installed project. Web-Bench's own reference agent is a
+ single files-in/files-out LLM call; there is no node_modules in the request and no test
+ suite (the tests are the grader, and shipping them to the agent would leak the gold).
+ Claude Code here is strictly more capable than the reference agent -- it can read, grep
+ and write across the whole file set -- but it cannot build or run the project, so
+ trajectories are shorter than SWE-bench's edit-test-fail loops. That is the benchmark,
+ not a harness limitation, and it is why the `trimech-web` arm exists.
+ * A Web-Bench stream is sequentially dependent and graded cumulatively, so in the
+ evaluator's `sequential` task mode the project STOPS at the first task that fails every
+ attempt. The driver still asks for the remaining tasks; those rows come back
+ `resolved: false` with `stream_broken: true` and zero tokens -- unreached, not cheap.
+
+Environment (all optional except the key; see the repo's .env / .env.example):
+ ANTHROPIC_API_KEY Anthropic API key. REQUIRED: the Claude lane bills
+ api.anthropic.com directly and nothing else. ANTHROPIC_BASE_URL
+ and ANTHROPIC_AUTH_TOKEN are not inputs, and are removed from the
+ agent's environment (swebench.resolve_anthropic_key,
+ swebench.agent_credential_env -- shared with that domain)
+ ANTHROPIC_MODEL agent model (default claude-opus-5)
+ (no optimizer-model knob: the between-task proposer/judge/nudge-writer runs on the
+ AGENT'S OWN model, against the agent's own endpoint. TE_OPTIMIZER_MODEL is refused.)
+ TE_CLAUDE_BIN host claude binary (default: resolved from `which claude`)
+ TE_EFFORT_LEVEL reasoning effort: low|medium|high|xhigh|max (default high)
+ TE_WEBBENCH_REPO Web-Bench checkout supplying the evaluator + projects
+ (default: the parent of $TOKENECONOMIZER_WEBBENCH_PROJECTS)
+ TE_NODE_BIN node binary to run the evaluator with (default `which node`)
+ TE_WEBBENCH_RETRY attempts per task, the evaluator's `retry` (default 2)
+ Rows carry resolved_p1 (first attempt) AND resolved_p2 (any
+ attempt); Web-Bench's headline metric is pass@1.
+ TE_WEBBENCH_MAX_WORKERS Playwright workers per grade, MAX_TEST_WORKERS (default 4)
+ TE_TASK_TIMEOUT per-segment agent wall-clock cap, seconds (default 280).
+ MUST stay under the evaluator's ~300s undici header timeout:
+ past it the evaluator has hung up, the answer is discarded and
+ a retry is burned -- and burning the last one ends the stream.
+ TE_WEBBENCH_VERDICT_TIMEOUT cap on one grade (build + tests), seconds (default 1800)
+ TE_WEBBENCH_START_TIMEOUT cap on the evaluator reaching the next task (default 1800)
+ TE_OPTIMIZER_MAX_TOKENS output cap for optimizer calls (default 16384)
+ TE_OPTIMIZER_LOG where between-task optimizer calls are recorded
+ TE_KEEP_WORKSPACES '1' keeps the per-attempt agent workspaces for post-mortem
+ TE_VERBOSE '1' prints per-task progress
+"""
+import hashlib
+import glob
+import json
+import os
+import queue
+import re
+import shutil
+import signal
+import subprocess
+import sys
+import threading
+import time
+from pathlib import Path
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from tokeneconomizer import baselines as scaffolds_pkg
+from tokeneconomizer.vendor import pricing, spiral as spiral_vendor
+
+from . import copilot
+from . import codex
+from tokeneconomizer.mechanisms import codex_spiral
+
+from .guards import net_guard
+from .swebench import (EFFORT_LEVELS, PROTECTED_TOOLS, _ANTHROPIC_DIRECT, _log,
+ answer_key_attempts,
+ _merge_usage, _row_from_result, anthropic_optimizer_call,
+ preflight_optimizer_endpoint, resolve_anthropic_key,
+ claude_bin, cli_version, guard_accepts,
+ set_optimizer_context, _cell_of, _UsageRecovery,
+ agent_credential_env)
+
+# External tool baselines this adapter can host. Web-Bench runs the agent on the HOST with
+# no per-task repo container and no Python source, so the SWE-bench set does not carry over
+# wholesale:
+# rtk -- YES. A static binary plus a PreToolUse hook; it rewrites `ls`/`find`/`grep`
+# and compresses their output before it reaches context, which needs neither a
+# container nor a language. The agent does issue those commands here.
+# repomap -- NO. Its generator walks `*.py` (baselines/_repomap_gen.py) and a Web-Bench
+# project is TypeScript/JavaScript, so the map would come back empty.
+# serena -- NO. An MCP navigation server over a codebase; Web-Bench hands the agent a
+# file set with no installed project, and the projects are 1-140 files.
+# scripts/sweep_plan.py SWEEP_ARMS_WEBBENCH_UNAVAILABLE must agree with this set;
+# tests/test_sweep_arms.py asserts it.
+WEBBENCH_SCAFFOLDS = frozenset({"rtk"})
+
+# Directories and files that are never part of a Web-Bench answer: build output, installed
+# dependencies, the evaluator's own scratch, and the assets tree the evaluator copies in
+# itself (echoing a binary asset back as text would corrupt it).
+_SKIP_DIRS = frozenset({
+ "node_modules", ".git", ".next", ".nuxt", ".svelte-kit", ".parcel-cache", ".turbo",
+ "dist", "build", "out", "coverage", "test-results", "playwright-report", "eval",
+ "assets", "__pycache__", ".pytest_cache", ".claude", ".copilot",
+})
+_SKIP_FILES = frozenset({
+ "package-lock.json", "pnpm-lock.yaml", "yarn.lock", ".DS_Store",
+})
+# A Web-Bench source file is small. The cap is a guard against an agent dropping a build
+# artifact or a database into the workspace, not a limit on legitimate answers.
+_MAX_FILE_BYTES = 512 * 1024
+
+# `[table] task-3 2nd Attempt ❌: Task 41.2s Request 33.1s Test 8.1s`
+# The evaluator logs this line once per attempt, from a `finally`-equivalent position, so
+# every attempt produces exactly one Started line and exactly one verdict line.
+_ATTEMPT_RE = re.compile(
+ r"^\[(?P<project>[^\]]+)\]\s+(?P<task>init|task-\d+)\s+(?P<ord>\d+)(?:st|nd|rd|th)\s+"
+ r"Attempt\s+(?P<mark>Started|✅|❌)")
+
+_MARK_OK = "✅"
+
+# Evaluator sessions this process started and has not yet shut down. A detached launcher
+# stops the run with `kill -TERM -<pid>` on the process group, but the node evaluator
+# spawns Playwright and `serve` children of its own, so the handler below closes the group
+# explicitly rather than leaving web servers bound to ports.
+_LIVE_SESSIONS = set()
+
+
+def _cleanup_sessions(*_a):
+ for sess in list(_LIVE_SESSIONS):
+ try:
+ sess.close()
+ except Exception:
+ pass
+
+
+def _install_signal_cleanup():
+ """Shut evaluator sessions down on TERM/INT/HUP, then die of that same signal so the
+ exit status stays honest. No-op off the main thread."""
+ def handler(signum, frame):
+ _cleanup_sessions()
+ signal.signal(signum, signal.SIG_DFL)
+ os.kill(os.getpid(), signum)
+
+ for sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP):
+ try:
+ signal.signal(sig, handler)
+ except (ValueError, OSError):
+ pass
+
+
+def _session_log_text_host(home):
+ """Claude Code's session log under a HOST-side per-task `$HOME`, or '' if unreadable.
+
+ The Web-Bench twin of `swebench._session_log_text`: same file, same purpose, no
+ container in the way. Globbed rather than composed from a slug -- the CLI derives the
+ project directory from cwd and that derivation is its own business. Several session
+ files can exist (the guard re-invokes the CLI per attempt); all are concatenated and
+ `session_output_tokens` deduplicates by message id, so the overlap cannot double-count.
+
+ Never raises: an unreadable log means the row keeps `output_tokens=0` and stays flagged,
+ which is the pre-existing, honest behaviour.
+ """
+ try:
+ parts = []
+ for p in sorted(glob.glob(os.path.join(home, ".claude", "projects", "*", "*.jsonl"))):
+ try:
+ with open(p, encoding="utf-8", errors="replace") as fh:
+ parts.append(fh.read())
+ except OSError:
+ continue
+ return "\n".join(parts)
+ except Exception as e:
+ _log(f"session log unreadable ({type(e).__name__}: {e})")
+ return ""
+
+
+def webbench_repo():
+ """The Web-Bench checkout root: TE_WEBBENCH_REPO, else the parent of the projects tree
+ the economizer's own adapter is already pointed at (so one env var configures both).
+
+ Returned REALPATH-resolved, and that matters. Every path the evaluator derives comes
+ from the entry script's `__dirname`, and it hands the project root to the project's own
+ dev server through EVAL_PROJECT_ROOT -- while node reports `process.cwd()` already
+ resolved. Reached through a symlinked home (`/home/x` -> `/srv/home/x`), the two
+ disagree, and Vite then fails to resolve the entry module of the `react` project
+ against a root that is not the root it thinks it is: every test fails with an app that
+ never rendered, which reads as an agent that wrote nothing. Canonicalizing here keeps
+ the whole chain on one spelling of the path.
+ """
+ explicit = os.environ.get("TE_WEBBENCH_REPO")
+ if explicit:
+ return os.path.realpath(explicit)
+ projects = os.environ.get("TOKENECONOMIZER_WEBBENCH_PROJECTS")
+ if projects:
+ return os.path.realpath(os.path.dirname(projects.rstrip("/")))
+ raise RuntimeError(
+ "no Web-Bench checkout: set TE_WEBBENCH_REPO or "
+ "TOKENECONOMIZER_WEBBENCH_PROJECTS (see the repo's .env.example)")
+
+
+def node_bin():
+ explicit = os.environ.get("TE_NODE_BIN")
+ if explicit:
+ return explicit
+ found = shutil.which("node")
+ if not found:
+ raise RuntimeError("no `node` on PATH and TE_NODE_BIN is unset; the Web-Bench "
+ "evaluator is a Node program")
+ return found
+
+
+def _load_ignore_globs(repo, project_dir):
+ """The ignore patterns the evaluator itself applies when it collects task context.
+
+ Read from the same four files its IgnoreGetter reads, so the set of files we hand back
+ matches the set it would have shown the agent -- an answer file the evaluator ignores is
+ an answer that silently does nothing.
+ """
+ globs = []
+ for path in (os.path.join(project_dir, ".gitignore"),
+ os.path.join(project_dir, ".npmignore"),
+ os.path.join(repo, ".gitignore"),
+ os.path.join(project_dir, ".evalignore")):
+ try:
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.strip()
+ if line and not line.startswith("#"):
+ globs.append(line.rstrip("/"))
+ except OSError:
+ continue
+ return globs
+
+
+def _ancestors(rel):
+ """Every directory prefix of a relative path, as forward-slash strings."""
+ parts = rel.split("/")[:-1]
+ return ["/".join(parts[:i + 1]) for i in range(len(parts))]
+
+
+def _ignored(rel, globs):
+ """A deliberately small gitignore subset: bare names and simple suffix globs, matched
+ against every path segment. The hard-coded _SKIP_DIRS carry the cases that matter; this
+ only adds whatever the project itself declared."""
+ import fnmatch
+ parts = rel.split("/")
+ for pat in globs:
+ pat = pat.lstrip("/")
+ if not pat:
+ continue
+ if any(fnmatch.fnmatch(p, pat) for p in parts):
+ return True
+ if fnmatch.fnmatch(rel, pat):
+ return True
+ return False
+
+
+# Tools that reach the public internet. Both benchmarks' answers are ON the public
+# internet -- SWE-bench instances are GitHub issues whose fixing PR and gold patch are a
+# fetch away, and Web-Bench ships every project's grading spec in `bytedance/web-bench`.
+# An agent that can browse is therefore not solving the task, it is retrieving the answer,
+# and the resolve verdict measures nothing.
+#
+# This is NOT hypothetical and NOT historical. Measured 2026-09-07: across the live
+# claude-opus-5 / claude-sonnet-5 trees, 32 tasks reached the network and 14 pulled
+# something patch-shaped -- `sympy/pulls/16597/files`, `django/commit/2cbd3967.patch`,
+# `sphinx-doc/commit/0eff5818.patch`, `svg-chart/test/task-4.spec.js` -- and 13 of those
+# scored RESOLVED. The published claude-opus-4.8 trees are clean (0 of 800 transcripts),
+# so this is a model-behaviour change, not a harness regression: the older model simply
+# never reached for the network, and nothing ever stopped it.
+#
+# Blocked for EVERY arm, deliberately. Leaving it to the untool mechanism would be worse
+# than useless: untool drops a tool only after observing DISUSE, so precisely the agents
+# that browse keep the ability, and the arms would differ in whether they can cheat.
+# `TE_ALLOW_NETWORK_TOOLS=1` lifts it for a run that genuinely wants a browsing agent.
+# The evaluator's `fetch` aborts a request after undici's default 300s headersTimeout
+# (measured 2026-09-07 on node v22.23.2: UND_ERR_HEADERS_TIMEOUT at 300.9s). Past that the
+# evaluator has already hung up, so our answer lands on a closed socket, the attempt is LOST
+# rather than merely slow, a retry is burned, and burning the last one ends the whole stream.
+# There is therefore NO Web-Bench configuration above ~300s that can help: a larger cap does
+# not buy the agent time, it only converts a slow task into a destroyed one.
+#
+# So this CLAMPS rather than trusting the environment. `TE_TASK_TIMEOUT` is shared with
+# SWE-bench, where 3600 is correct and necessary (sympy's slowest instances pinned 1800 dead
+# on), and that value silently truncated two Web-Bench streams at task-14 before the clamp
+# existed. `TE_WEBBENCH_TASK_TIMEOUT` overrides it for this domain only.
+#
+# The cap is COOPERATIVE -- checked between lines of the CLI's stream, so a turn already in
+# flight runs to completion. The ceiling is therefore `cap + longest single turn`, measured
+# at +16s on claude-opus-4.8 and +21s on claude-opus-5. So the ceiling must sit well below
+# 300, not just under it: the default leaves ~60s for the overshoot and the hard max ~40s.
+# A 290 max would have allowed 290+21 = 311s, i.e. straight back over the wall.
+# 2026-09-21: the "no configuration above ~300s can help" above is true of WEB-BENCH's
+# config but NOT of Node's. `HttpAgent.request` (tools/http-agent) calls bare `fetch`, so it
+# uses undici's GLOBAL dispatcher, and that dispatcher's headersTimeout is settable from
+# outside the vendored benchmark via `--import` (webbench_undici_preload.mjs). Raising it is
+# opt-in through `TE_WEBBENCH_UNDICI_ABORT`; unset, everything below is byte-identical to
+# the old fixed 300/240/260. Rows taken at different ceilings are NOT comparable -- a run
+# that raised it completes tasks a 300s run would have destroyed.
+WEBBENCH_UNDICI_ABORT_DEFAULT_S = 300
+WEBBENCH_TASK_TIMEOUT_HEADROOM_S = 40 # cooperative-cap overshoot (+21s worst) + margin
+WEBBENCH_TASK_TIMEOUT_DEFAULT = 240
+# Back-compat names for the STOCK ceiling: the invariant they encode (cap < abort) is
+# still the contract, it is now just evaluated per-run by the two helpers below.
+WEBBENCH_UNDICI_ABORT_S = WEBBENCH_UNDICI_ABORT_DEFAULT_S
+WEBBENCH_TASK_TIMEOUT_MAX = (WEBBENCH_UNDICI_ABORT_DEFAULT_S
+ - WEBBENCH_TASK_TIMEOUT_HEADROOM_S)
+UNDICI_PRELOAD = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ "webbench_undici_preload.mjs")
+
+
+def webbench_undici_abort(env=None):
+ """The evaluator's request-abort ceiling in seconds (undici headersTimeout)."""
+ env = os.environ if env is None else env
+ try:
+ value = int(env.get("TE_WEBBENCH_UNDICI_ABORT")
+ or WEBBENCH_UNDICI_ABORT_DEFAULT_S)
+ except (TypeError, ValueError):
+ value = WEBBENCH_UNDICI_ABORT_DEFAULT_S
+ return max(60, value)
+
+
+def webbench_undici_env(env=None, repo=None):
+ """Env additions that make the evaluator honour a RAISED abort ceiling.
+
+ Empty when the ceiling is the stock 300s, so the default spawn is unchanged.
+ """
+ env = os.environ if env is None else env
+ abort = webbench_undici_abort(env)
+ if abort <= WEBBENCH_UNDICI_ABORT_DEFAULT_S:
+ return {}
+ opt = "--import " + Path(UNDICI_PRELOAD).as_uri()
+ prev = env.get("NODE_OPTIONS", "")
+ return {"NODE_OPTIONS": (prev + " " + opt).strip() if prev else opt,
+ "TE_WEBBENCH_UNDICI_ABORT_MS": str(abort * 1000),
+ "TE_WEBBENCH_REPO": str(repo or env.get("TOKENECONOMIZER_WEBBENCH_REPO", ""))}
+
+
+def _shq(s):
+ """POSIX shell quoting for the one path the namespace wrapper interpolates."""
+ return "'" + str(s).replace("'", "'\\''") + "'"
+
+
+def webbench_task_timeout(env=None):
+ """Per-segment cap for Web-Bench, clamped under the evaluator's abort. Never raises."""
+ env = os.environ if env is None else env
+ raw = env.get("TE_WEBBENCH_TASK_TIMEOUT") or env.get("TE_TASK_TIMEOUT")
+ try:
+ value = int(raw) if raw else WEBBENCH_TASK_TIMEOUT_DEFAULT
+ except (TypeError, ValueError):
+ value = WEBBENCH_TASK_TIMEOUT_DEFAULT
+ abort = webbench_undici_abort(env)
+ cap = abort - WEBBENCH_TASK_TIMEOUT_HEADROOM_S
+ if value > cap:
+ _log(f"webbench: TE_TASK_TIMEOUT={value}s is above the evaluator's "
+ f"{abort}s request abort, which would lose whole attempts and "
+ f"truncate the stream -- clamping to {cap}s. Set "
+ f"TE_WEBBENCH_TASK_TIMEOUT to choose a lower value for this domain.")
+ value = cap
+ return max(1, value)
+
+
+NETWORK_TOOLS = ("WebSearch", "WebFetch")
+
+
+def network_tool_block():
+ """The network tools to add to `--disallowed-tools`, or () when explicitly allowed."""
+ if os.environ.get("TE_ALLOW_NETWORK_TOOLS", "").strip() in ("1", "true", "yes"):
+ return ()
+ return NETWORK_TOOLS
+
+
+class _AgentHandler(BaseHTTPRequestHandler):
+ """The evaluator's agent endpoint. One request in flight at a time: the evaluator is
+ configured `sequential` with `httpLimit: 1`, and the economizer's driver is a single
+ thread, so serializing here turns a concurrency bug into a visible timeout."""
+
+ session = None # set by the factory below
+
+ def do_POST(self):
+ try:
+ n = int(self.headers.get("content-length") or 0)
+ req = json.loads(self.rfile.read(n).decode("utf-8") or "{}")
+ except Exception as e:
+ self.send_error(400, f"bad agent request: {e}")
+ return
+ reply_q = queue.Queue(maxsize=1)
+ self.session.requests.put((req, reply_q))
+ try:
+ payload = reply_q.get(timeout=self.session.reply_timeout)
+ except queue.Empty:
+ # The driver never answered. Returning an empty file set lets the evaluator
+ # fail this task honestly instead of hanging the whole stream.
+ _log("webbench: no answer for an agent request within "
+ f"{self.session.reply_timeout}s -- replying with no files")
+ payload = {"files": {}}
+ body = json.dumps(payload).encode("utf-8")
+ self.send_response(200)
+ self.send_header("content-type", "application/json")
+ self.send_header("content-length", str(len(body)))
+ self.end_headers()
+ try:
+ self.wfile.write(body)
+ except BrokenPipeError:
+ # The evaluator hung up before we answered -- it waits ~300s (undici header
+ # timeout) and this segment outran it. Say so in one line: the bare socketserver
+ # traceback names neither the cause nor the consequence, and the consequence is
+ # severe (answer discarded, retry burned, stream possibly ended). Re-raising
+ # would only reprint that traceback, so swallow it after logging.
+ _log("webbench: the evaluator hung up before this answer was written "
+ "(BrokenPipeError) -- the segment outran its ~300s request budget, so this "
+ "attempt is LOST, not merely slow. Lower TE_TASK_TIMEOUT (default 280) so "
+ "the segment is cut before the evaluator gives up.")
+
+ def log_message(self, *_a): # keep the evaluator's stdout readable
+ pass
+
+
+class _EvalSession:
+ """One project's evaluator run: the agent endpoint plus the `@web-bench/eval` process.
+
+ The session is the shared state the rendezvous needs -- a request queue fed by the HTTP
+ handler and an event queue fed by a reader thread parsing the evaluator's stdout for
+ its per-attempt Started/verdict lines.
+ """
+
+ def __init__(self, project, repo, session_dir, *, retry, tasks, node,
+ reply_timeout, log_level="info", max_workers=4):
+ self.project = project
+ self.repo = repo
+ self.dir = session_dir
+ self.retry = retry
+ self.reply_timeout = reply_timeout
+ self.requests = queue.Queue()
+ self.events = queue.Queue()
+ self.finished = False
+ self.exit_code = None
+ self.hash = time.strftime("%Y%m%d-%H%M%S")
+ os.makedirs(self.dir, exist_ok=True)
+ self.log_path = os.path.join(self.dir, "evaluator.log")
+
+ # 1. the agent endpoint
+ handler = type("_BoundAgentHandler", (_AgentHandler,), {"session": self})
+ self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), handler)
+ self.port = self.httpd.server_address[1]
+ self._http_thread = threading.Thread(target=self.httpd.serve_forever,
+ kwargs={"poll_interval": 0.2}, daemon=True)
+ self._http_thread.start()
+
+ # 2. the evaluator config. `--without-local-config` keeps apps/eval/src/config.json5
+ # out of it, so the run is fully described by this file.
+ full = self._full_task_ids()
+ cfg = {
+ "models": [],
+ "projects": [f"@web-bench/{project}"],
+ "agentMode": "http",
+ "agentEndPoint": f"http://127.0.0.1:{self.port}/message",
+ "httpLimit": 1,
+ "maxdop": 1,
+ "taskMode": "sequential",
+ "retry": retry,
+ "logLevel": log_level,
+ "production": True,
+ "hash": self.hash,
+ "agentDir": self.dir,
+ }
+ # Restricting the task range flips the evaluator out of production mode, which
+ # tightens Playwright's timeouts from 30s to 2s and would fail tasks for being slow
+ # rather than wrong. Only narrow the range when the driver really asked for a
+ # subset, and say so.
+ ids = [t.get("instance_id") for t in tasks]
+ if ids and full and ids != full:
+ cfg["startTask"], cfg["endTask"] = ids[0], ids[-1]
+ _log(f"webbench {project}: task range {ids[0]}..{ids[-1]} is a SUBSET of the "
+ f"{len(full)}-task stream -- the evaluator drops to development mode "
+ f"(Playwright timeout 2s, not 30s), so grades are stricter than a full run")
+ self.config_path = os.path.join(self.dir, "evaluator_config.json")
+ with open(self.config_path, "w", encoding="utf-8") as fh:
+ json.dump(cfg, fh, indent=2)
+
+ # 3. the evaluator process, in its own group so its Playwright/serve children die
+ # with it. NO_COLOR keeps the verdict lines parseable.
+ entry = os.path.join(repo, "apps", "eval", "dist", "index.js")
+ if not os.path.exists(entry):
+ raise RuntimeError(
+ f"the Web-Bench evaluator is not built: {entry} missing. Run "
+ f"`rush build --to @web-bench/eval` in {repo}.")
+ env = dict(os.environ)
+ env.update({"NO_COLOR": "1", "FORCE_COLOR": "0",
+ "MAX_TEST_WORKERS": str(max_workers)})
+ env.update(webbench_undici_env(env, repo=repo))
+ self._log_fh = open(self.log_path, "w", encoding="utf-8")
+ self.proc = subprocess.Popen(
+ [node, entry, "--auto-run", "--config", self.config_path,
+ "--without-local-config"],
+ cwd=os.path.join(repo, "apps", "eval"), env=env,
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+ text=True, bufsize=1, start_new_session=True)
+ _LIVE_SESSIONS.add(self)
+ self._reader = threading.Thread(target=self._read_stdout, daemon=True)
+ self._reader.start()
+ _log(f"webbench {project}: evaluator pid={self.proc.pid} endpoint=:{self.port} "
+ f"retry={retry} log={self.log_path}")
+
+ def _full_task_ids(self):
+ """The project's whole stream, in order, as the evaluator reads it."""
+ path = os.path.join(self.repo, "projects", self.project, "tasks.jsonl")
+ try:
+ with open(path, encoding="utf-8") as fh:
+ out = []
+ for line in fh:
+ line = line.strip()
+ if line:
+ out.append(json.loads(line).get("id"))
+ return out
+ except OSError:
+ return []
+
+ def _read_stdout(self):
+ """Tee the evaluator's output to disk and lift its per-attempt lines onto the
+ event queue. This is the grade channel: the evaluator prints exactly one
+ `<task> <n>th Attempt ✅/❌` line per attempt."""
+ for line in self.proc.stdout:
+ self._log_fh.write(line)
+ self._log_fh.flush()
+ m = _ATTEMPT_RE.match(line.strip())
+ if m:
+ kind = "started" if m.group("mark") == "Started" else "verdict"
+ self.events.put({
+ "kind": kind, "task": m.group("task"),
+ "attempt": int(m.group("ord")),
+ "ok": m.group("mark") == _MARK_OK,
+ "line": line.rstrip(),
+ })
+ self.proc.wait()
+ self.exit_code = self.proc.returncode
+ self.finished = True
+ self.events.put({"kind": "exit", "code": self.exit_code})
+ _log(f"webbench {self.project}: evaluator exited rc={self.exit_code}")
+
+ def next_event(self, kind, timeout):
+ """The next `kind` event, skipping the other kinds. Returns None when the evaluator
+ exits first (a broken stream) or nothing arrives in time."""
+ deadline = time.time() + timeout
+ while True:
+ left = deadline - time.time()
+ if left <= 0:
+ return None
+ try:
+ ev = self.events.get(timeout=min(left, 5.0))
+ except queue.Empty:
+ if self.finished:
+ return None
+ continue
+ if ev["kind"] == "exit":
+ return None
+ if ev["kind"] == kind:
+ return ev
+ # A verdict we were not waiting for means the loops have desynchronized;
+ # surface it rather than silently pairing the next request with it.
+ _log(f"webbench {self.project}: unexpected {ev['kind']} event "
+ f"{ev.get('task')} #{ev.get('attempt')} while waiting for {kind}")
+
+ def next_request(self, timeout):
+ deadline = time.time() + timeout
+ while True:
+ left = deadline - time.time()
+ if left <= 0:
+ return None, None
+ try:
+ return self.requests.get(timeout=min(left, 5.0))
+ except queue.Empty:
+ if self.finished:
+ return None, None
+
+ def close(self):
+ _LIVE_SESSIONS.discard(self)
+ try:
+ self.httpd.shutdown()
+ except Exception:
+ pass
+ try:
+ self.httpd.server_close()
+ except Exception:
+ pass
+ if self.proc.poll() is None:
+ for sig in (signal.SIGTERM, signal.SIGKILL):
+ try:
+ os.killpg(os.getpgid(self.proc.pid), sig)
+ except (ProcessLookupError, PermissionError):
+ break
+ try:
+ self.proc.wait(timeout=20)
+ break
+ except subprocess.TimeoutExpired:
+ continue
+ try:
+ self._log_fh.close()
+ except Exception:
+ pass
+ self.finished = True
+
+
+class WebBenchRunner:
+ """AgentRunner: Claude Code against Web-Bench, graded by Web-Bench's own evaluator."""
+
+ def __init__(self, agent="claude", model=None, repo=None, scaffold=None):
+ self.agent = agent
+ self.model = model or os.environ.get("ANTHROPIC_MODEL", "claude-opus-5")
+ # The Anthropic API directly, on ANTHROPIC_API_KEY -- one resolver, shared with the
+ # SWE-bench runner so the two domains cannot drift apart about what they bill.
+ self.api_key = resolve_anthropic_key()
+ # The mechanisms' between-task LLM calls (proposer, tool judge, spiral judge, nudge
+ # writer) run on THE AGENT'S OWN MODEL, always. Their cost is charged to the same
+ # ledger as the task rows and compared against a baseline that made no such calls,
+ # so a cheaper proposer would understate the overhead the arm actually incurs --
+ # the economized arm would look better for having billed its own thinking at a
+ # discount. One model per cell is also what the cell name claims. There is no knob:
+ # TE_OPTIMIZER_MODEL used to be one and is now refused rather than half-honoured.
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "
+ "runs on the agent's own model so mechanism overhead is priced on the "
+ "same basis as the task rows. Unset it.")
+ self.optimizer_model = self.model
+ self.effort = (os.environ.get("TE_EFFORT_LEVEL")
+ or os.environ.get("CLAUDE_CODE_EFFORT_LEVEL") or "high")
+ if self.effort not in EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort!r} is not one of "
+ f"{sorted(EFFORT_LEVELS)}")
+ self.claude = claude_bin()
+ # Same early pricing check the Copilot runners make: an unpriced model id otherwise
+ # raises from pricing.effective_tokens at the END of run_task, after the rollout has
+ # been paid for and graded. Claude ids resolve to the Claude curve, so this only
+ # ever fires on a genuinely unknown id.
+ pricing.weights_for({"model": self.model, "agent": self.agent})
+ # Generic slots the shared lifecycle reads, so an agent subclass overrides only
+ # launch + accounting rather than the whole evaluator round-trip.
+ self.agent_bin = self.claude
+ self.protected_tools = PROTECTED_TOOLS
+ self.routing_base = _ANTHROPIC_DIRECT
+ self.cli_version = cli_version(self.claude)
+ self.repo = os.path.realpath(repo) if repo else webbench_repo()
+ # An external tool baseline (rtk today). Resolved and preflighted HERE so a
+ # misconfigured one kills the lane before task 0, exactly as on SWE-bench --
+ # a lane of flagged rows is diagnosable, a lane of silently-native rows is not.
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile_for("/nonexistent"))
+ _log(f"scaffold: {self.scaffold.name} (external baseline)")
+ self.node = node_bin()
+ # Output cap for the between-task proposer / judge / nudge-writer call, same
+ # default as the SWE-bench harness so mechanism overhead is comparable across
+ # domains. This is the CLAUDE base class and `optimizer_call` below reads it, so
+ # it must be set here: the Copilot subclass overrides it (that CLI has no output
+ # cap flag of its own), and the stale comment this replaces claimed the Copilot
+ # rationale for the base, leaving `self.optimizer_max_tokens` undefined. The
+ # effect was an AttributeError on the FIRST proposer tick -- task 5 of every
+ # stream -- so `inject-only` and `trimech-web` died early in each Claude Web-Bench
+ # cell, and no LIMIT<5 pilot could reach it.
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ # Stated, not inferred: this harness has always used the Messages API for
+ # mechanism calls, which is the transport the Copilot harness now matches.
+ self.optimizer_transport = "api"
+ # MUST stay under the evaluator's ~300s undici header timeout -- the same bound
+ # CopilotWebBenchRunner documents, and it binds here for the identical reason.
+ # A segment that outlives it is not merely slow: the evaluator has already hung up,
+ # so our reply lands on a closed socket (BrokenPipeError out of do_POST), the answer
+ # is DISCARDED, and the task is scored a pass@1 miss for a plumbing reason. Worse,
+ # it burns a retry -- and when it burns the LAST retry the evaluator ends the whole
+ # stream, so every later task comes back UNREACHED. Measured 2026-09-07 on
+ # claude-opus-5: threejs/task-14 2nd attempt clipped at
+ # Request 300.858s and took tasks 15-20 down with it; svg-chart hit the same wall
+ # from task-6 on. The old 1800s default predated a model slow enough to reach it.
+ self.task_timeout = webbench_task_timeout()
+ self.retry = max(1, int(os.environ.get("TE_WEBBENCH_RETRY", "2")))
+ self.max_workers = int(os.environ.get("TE_WEBBENCH_MAX_WORKERS", "4"))
+ self.verdict_timeout = int(os.environ.get("TE_WEBBENCH_VERDICT_TIMEOUT", "1800"))
+ self._init_stream_state()
+ self._preflight_optimizer_endpoint()
+
+ def _preflight_optimizer_endpoint(self):
+ """Refuse the lane if the mechanism endpoint will not answer. Same check and same
+ reasoning as the SWE-bench harness -- see the shared helper."""
+ preflight_optimizer_endpoint(
+ api_key=self.api_key, configured_model=self.optimizer_model, label="claude",
+ remedy="Check that ANTHROPIC_API_KEY is valid and can call MODEL.")
+
+ def _init_stream_state(self):
+ """Evaluator-session state, shared by every agent subclass.
+
+ Kept in one place deliberately: a subclass that reimplements __init__ and forgets
+ one of these fails later with an AttributeError deep in begin_stream, so new stream
+ state must be added HERE rather than inline above.
+ """
+ self.start_timeout = int(os.environ.get("TE_WEBBENCH_START_TIMEOUT", "1800"))
+ self.keep_workspaces = os.environ.get("TE_KEEP_WORKSPACES") == "1"
+ self._session = None
+ self._session_project = None
+ self._stream = []
+ self._stream_broken = False
+ self._current_task = None
+
+ # ---- the paid optimizer call the mechanisms make between tasks ---------------------
+ def optimizer_call(self, prompt, model="agent"):
+ """One small Messages-API request. Returns (text, effective_tokens). Same call and
+ same pricing basis as the SWE-bench harness, so mechanism overhead is comparable
+ across domains."""
+ return anthropic_optimizer_call(
+ prompt, model=model, configured_model=self.optimizer_model,
+ max_tokens=self.optimizer_max_tokens, api_key=self.api_key)
+
+ # ---- stream lifecycle (the driver's optional begin_stream/end_stream seam) ---------
+ def begin_stream(self, tasks, *, results_root=None, repo=None, **_info):
+ """Start the project's evaluator before the first task.
+
+ Web-Bench's evaluator owns the task loop, so it must be started once for the whole
+ stream -- starting it per task would reset the workspace chain the benchmark is
+ built on. Doing it here rather than lazily also means a misconfigured checkout
+ fails before any task has been billed.
+ """
+ self._stream = list(tasks or [])
+ project = ((repo or "").split("/")[-1]
+ or (self._stream[0].get("repo", "") if self._stream else ""))
+ project = project.split("/")[-1]
+ if not project:
+ raise RuntimeError("webbench: no project to run; pass --repo <project>")
+ if not os.path.isdir(os.path.join(self.repo, "projects", project)):
+ raise RuntimeError(f"webbench: no such project "
+ f"{os.path.join(self.repo, 'projects', project)}")
+ self.close_session()
+ self._stream_broken = False
+ session_dir = os.path.join(results_root or os.getcwd(), f"_webbench_{project}")
+ self._session = _EvalSession(
+ project, self.repo, session_dir, retry=self.retry, tasks=self._stream,
+ node=self.node, reply_timeout=self.task_timeout * (self.retry + 2),
+ max_workers=self.max_workers)
+ self._session_project = project
+
+ def end_stream(self):
+ self.close_session()
+
+ def session_for(self, project):
+ """The live session, checked against the project the task claims to belong to."""
+ if self._session is None:
+ raise RuntimeError(
+ "webbench: no evaluator session. The driver calls begin_stream() before "
+ "the first task; a custom caller must do the same.")
+ if self._session_project != project:
+ raise RuntimeError(f"webbench: session is running {self._session_project!r} "
+ f"but the task belongs to {project!r}")
+ return self._session
+
+ def close_session(self):
+ if self._session is not None:
+ self._session.close()
+ self._session = None
+ self._session_project = None
+
+ # ---- external tool scaffold (baseline arms only) ----------------------------------
+ #
+ # Web-Bench has NO per-task container: the agent runs on the host, in a scratch
+ # workspace, under a per-task HOME. So the Docker-shaped handles the SWE-bench runner
+ # hands a scaffold (`container`, `docker exec`, `docker cp`) become plain subprocess
+ # and file-copy here. Everything else about a scaffold is unchanged, which is the point
+ # of the seam: rtk installs the same pinned binary and registers the same PreToolUse
+ # hook whether the "machine" is a container or this host.
+ #
+ # `read_bytes` is deliberately NOT provided. Only the repo-map baseline needs it (it
+ # streams the repo out as a tar) and that baseline is Python-only, so a Web-Bench arm
+ # asking for it should fail loudly rather than half-work -- ScaffoldContext.read_bytes
+ # already raises a ScaffoldError saying exactly that.
+ def agent_home(self, run_dir_path):
+ """$HOME for the agent on THIS task-run. Per task, and never inside the workspace:
+ everything under the workspace is a candidate answer file."""
+ return os.path.join(run_dir_path, ".agent-home")
+
+ def scaffold_profile_for(self, home):
+ """Claude Code's profile, with the per-task HOME filled in.
+
+ On SWE-bench this is a class attribute because the container's HOME is a fixed
+ path; here it is per task-run, so it is built per call.
+
+ `instructions_file` stays CLAUDE.md for parity, and `patch_excludes` carries it
+ plus RTK.md -- on Web-Bench those names are what `_collect` must refuse to ship to
+ the evaluator rather than what `git diff` must subtract. With `rtk init -g` both
+ actually land under $HOME/.claude/ and never reach the workspace at all; the
+ exclusion is the belt-and-braces half.
+ """
+ return scaffolds_pkg.AgentProfile(
+ kind="claude",
+ home=home,
+ instructions_file="CLAUDE.md",
+ nav_tools=("Read", "Grep", "Glob"),
+ mcp_args=lambda cfg: ["--mcp-config", cfg],
+ patch_excludes=("CLAUDE.md",),
+ # No writable /usr/local on the host, so a scaffold installs into the agent's
+ # own bin dir -- which _segment puts on the agent's PATH.
+ bin_dir=self.scaffold_bin_dir(home))
+
+ def _scaffold_context(self, workspace, home, task, run_dir_path):
+ """The handles a scaffold may use. The host is reached only through these."""
+ def _host(argv, timeout=300):
+ return subprocess.run(argv, capture_output=True, text=True, timeout=timeout)
+
+ def exec_(script, timeout=300, env=None):
+ """Run a script AS THE AGENT WOULD: in the workspace, under the agent's HOME."""
+ e = dict(os.environ)
+ e["HOME"] = home
+ e["PATH"] = os.pathsep.join([self.scaffold_bin_dir(home), e.get("PATH", "")])
+ # Per-agent additions the install must see -- Copilot redirects COPILOT_HOME so
+ # `copilot plugin install` cannot write into the operator's real config.
+ e.update(self.scaffold_exec_env(run_dir_path))
+ e.update({k: str(v) for k, v in (env or {}).items()})
+ return subprocess.run(["bash", "-lc", script], cwd=workspace, env=e,
+ capture_output=True, text=True, timeout=timeout)
+
+ def _place(src, dst):
+ """Hardlink if we can, else copy.
+
+ A scaffold binary is staged once on the host and installed per task-run; rtk's
+ is 10MB, which copied into every run dir is tens of GB across one arm and buys
+ nothing, since nothing writes to it. A hardlink is the same inode, so it costs
+ no space and cannot dangle the way a symlink would if the cache moved. Falls
+ back to a copy across filesystems, which is the only case that can fail.
+ """
+ if os.path.lexists(dst):
+ os.unlink(dst)
+ try:
+ os.link(src, dst)
+ except OSError:
+ shutil.copy2(src, dst)
+
+ def copy_in(host_path, dest_path):
+ """`docker cp`'s host twin.
+
+ Scaffolds address the install directory through `profile.bin_dir`, which is
+ already a writable per-run path here, so the destination normally needs no
+ change. A destination outside it would be a scaffold hardcoding a container
+ path: re-root rather than fail, since the alternative is an install that
+ silently no-ops and a row that reads `<tool>_installed: false`.
+ """
+ bin_dir = self.scaffold_bin_dir(home)
+ try:
+ # Honour the destination as given. On the host most absolute paths a
+ # scaffold names (a /tmp helper, anything under the agent's own HOME) are
+ # perfectly writable, and rewriting them would break the very next command,
+ # which refers to the path it asked for.
+ os.makedirs(os.path.dirname(dest_path) or ".", exist_ok=True)
+ _place(host_path, dest_path)
+ dest = dest_path
+ except OSError:
+ # Not writable off-container -- /usr/local/bin on an unprivileged host is
+ # the case this exists for. Re-root into the agent's own bin dir, which is
+ # on its PATH, rather than letting the install silently no-op.
+ dest = os.path.join(bin_dir, os.path.basename(dest_path))
+ _log(f"scaffold copy_in: {dest_path} not writable; using {dest}")
+ os.makedirs(bin_dir, exist_ok=True)
+ _place(host_path, dest)
+ os.chmod(dest, 0o755)
+ return subprocess.CompletedProcess([], 0, "", "")
+
+ return scaffolds_pkg.ScaffoldContext(
+ profile=self.scaffold_profile_for(home), container=None, task=task,
+ run_dir=run_dir_path, exec_=exec_, copy_in=copy_in, host_run=_host,
+ read_bytes=None, log=_log)
+
+ @staticmethod
+ def scaffold_bin_dir(home):
+ """Where copy_in lands executables, and what exec_/the agent get on PATH."""
+ return os.path.join(home, ".local", "bin")
+
+ def scaffold_exec_env(self, run_dir_path):
+ """Extra environment a scaffold's install commands need. Empty for Claude."""
+ return {}
+
+ def prepare_agent_home(self, home):
+ """Seed the per-task agent HOME before a scaffold installs into it. No-op here."""
+ return
+
+ @staticmethod
+ def _net_guard_log(home):
+ """Where the guard appends its refusals for this task."""
+ return os.path.join(home, "net_guard_blocks.jsonl")
+
+ def _net_guard_blocks(self, run_dir_path):
+ """Count the guard's refusals for this task. Absent log = it never fired."""
+ path = self._net_guard_log(self.agent_home(run_dir_path))
+ try:
+ with open(path, encoding="utf-8") as fh:
+ return sum(1 for line in fh if line.strip())
+ except OSError:
+ return 0
+
+ def net_guard_settings_path(self, home, run_dir_path):
+ """Where THIS agent reads PreToolUse hooks from. Claude: `$HOME/.claude`."""
+ return os.path.join(home, ".claude", "settings.json")
+
+ def net_guard_hook_entry(self, command):
+ """Claude Code's PreToolUse entry shape: nested `hooks`, matcher `Bash`."""
+ return {"matcher": "Bash", "hooks": [{"type": "command", "command": command}]}
+
+ # Claude nests its list under `PreToolUse`; Copilot under `preToolUse`.
+ NET_GUARD_HOOK_KEY = "PreToolUse"
+
+ def _install_net_guard(self, home, run_dir_path=None):
+ """Register the PreToolUse guard for this task. True when it is actually armed.
+
+ Written per task into the SAME settings file a scaffold would use, so rtk's hook
+ and this one coexist -- both land in the hook list rather than overwriting each
+ other. Applies to EVERY arm: an agent that can read the grading spec makes the
+ resolve verdict meaningless, which is not an arm-level concern.
+
+ **Returns a bool now, and the row records it.** It used to return a log path and
+ never raise, so the Copilot subclass -- which inherited it unchanged and therefore
+ wrote a Claude-shaped `.claude/settings.json` the Copilot CLI does not read --
+ failed *silently*: `os.makedirs` and the write both succeeded, nothing was armed,
+ and the row carried only `net_guard_blocks: 0`, which is indistinguishable from
+ "armed and never fired". Confirmed on a live copilot/calculator run 2026-09-08.
+ """
+ path = self.net_guard_settings_path(home, run_dir_path)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ try:
+ with open(path, encoding="utf-8") as fh:
+ settings = json.load(fh)
+ except Exception:
+ settings = {}
+ # An ABSOLUTE script path, not `-m te_harness.guards.net_guard`: the hook is executed
+ # with the agent's workspace as cwd and nothing puts this repo on sys.path there, so
+ # the module spelling would die with ImportError on every call. A hook that always
+ # errors is worse than none -- the CLI reports a failing hook while the agent keeps
+ # going, so the run looks guarded and is not. `net_guard` imports only the stdlib
+ # precisely so it can be run as a loose file.
+ entry = self.net_guard_hook_entry(f"{sys.executable} {net_guard.__file__}")
+ hooks = settings.setdefault("hooks", {})
+ pre = hooks.setdefault(self.NET_GUARD_HOOK_KEY, [])
+ if not any("net_guard" in json.dumps(h) for h in pre):
+ pre.append(entry)
+ with open(path, "w", encoding="utf-8") as fh:
+ json.dump(settings, fh, indent=2)
+ return self._verify_net_guard(home)
+
+ # One the guard MUST refuse, and which reaches no network even unguarded: `decide`
+ # matches the substring `github` alone, so `echo` exercises the whole path.
+ NET_GUARD_PROBE = "echo https://github.com/probe"
+
+ def _verify_net_guard(self, home):
+ """Feed the guard a payload of THIS agent's shape; True only if it refuses.
+
+ Writing the settings file is not evidence of anything -- it succeeds on an agent
+ that never reads it. What can still be wrong after a successful write is the
+ payload spelling (`tool_input` vs `toolArgs`), the interpreter, or the script
+ path, and all three are exercised here. What this does NOT prove is that the CLI
+ invokes the hook at all; the file location, the hook key and the matcher name are
+ what carry that, and each is now set per agent rather than inherited.
+ """
+ probe = json.dumps({"tool_name": "Bash", "toolName": "bash",
+ "tool_input": {"command": self.NET_GUARD_PROBE},
+ "toolArgs": {"command": self.NET_GUARD_PROBE}})
+ env = dict(os.environ)
+ env[net_guard.LOG_ENV] = self._net_guard_log(home)
+ try:
+ r = subprocess.run([sys.executable, net_guard.__file__], input=probe,
+ capture_output=True, text=True, timeout=60, env=env)
+ except Exception as e:
+ _log(f"net guard verify FAILED ({type(e).__name__}: {e}) -- row runs UNGUARDED")
+ return False
+ if r.returncode != 2:
+ _log(f"net guard installed but did NOT refuse its probe (rc={r.returncode}) "
+ "-- row runs UNGUARDED")
+ return False
+ # The probe's own block record is telemetry about the harness, not the agent, so
+ # it must not inflate the row's `net_guard_blocks`.
+ try:
+ os.remove(self._net_guard_log(home))
+ except OSError:
+ pass
+ return True
+
+ def _apply_scaffold(self, workspace, home, task, run_dir_path):
+ """Install the arm's scaffold for this task. Returns a notes dict.
+
+ A scaffold that raises is recorded and the task runs NATIVE, exactly as on
+ SWE-bench: one broken install must not cost the row, because a row flagged
+ `scaffold_error` is diagnosable while a missing row is not.
+ """
+ if self.scaffold is None:
+ return {}
+ os.makedirs(self.scaffold_bin_dir(home), exist_ok=True)
+ os.makedirs(os.path.join(home, ".claude"), exist_ok=True)
+ self.prepare_agent_home(home)
+ try:
+ return self.scaffold.setup(
+ self._scaffold_context(workspace, home, task, run_dir_path))
+ except Exception as e: # noqa: BLE001
+ _log(f"{task['instance_id']}: scaffold {self.scaffold.name} FAILED "
+ f"({type(e).__name__}: {e}) -- task runs native, row flagged")
+ return {"arm_setup": self.scaffold.name,
+ "scaffold_error": f"{type(e).__name__}: {e}"[:300]}
+
+ def scaffold_answer_excludes(self):
+ """Names `_collect` must never ship to the evaluator.
+
+ Web-Bench's analogue of SWE-bench's `patch_excludes`: there the scaffold's files
+ are subtracted from the graded diff, here they must be kept out of the answer the
+ evaluator's Writer applies. With no scaffold this is empty and `_collect` behaves
+ byte-identically to the pre-scaffold form.
+ """
+ if self.scaffold is None:
+ return frozenset()
+ names = set(getattr(self.scaffold, "patch_excludes", ()) or ())
+ names |= set(self.scaffold_profile_for("/nonexistent").patch_excludes)
+ return frozenset(n for n in names if "*" not in n)
+
+ # ---- one streamed agent segment ---------------------------------------------------
+ def _segment(self, workspace, prompt, transcript_fh, exclude_tools, max_turns, guard):
+ """Run `claude -p` once in `workspace`, streaming stream-json. Every raw line is
+ appended to the transcript (the schema `tokeneconomizer.trace` re-reads) and, when
+ the spiral guard is active, folded into it live; a guard stop kills the agent,
+ leaving its edits in the workspace for the next segment."""
+ cmd = [self.claude, "-p", "--output-format", "stream-json", "--verbose",
+ "--dangerously-skip-permissions",
+ "--model", self.model, "--max-turns", str(max_turns),
+ "--effort", self.effort]
+ drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]
+ # Network tools are dropped for every arm, on top of whatever untool decided --
+ # see NETWORK_TOOLS: the answers to both benchmarks are a fetch away.
+ drop += [t for t in network_tool_block() if t not in drop]
+ if drop:
+ cmd += ["--disallowed-tools", ",".join(drop)]
+
+ env = dict(os.environ)
+ # Keep the CLI's own state OUT of the workspace: everything under the workspace is
+ # a candidate answer file, and the agent's home is not part of the project. This is
+ # the SAME path `agent_home` hands the scaffold, so an installed hook and the agent
+ # that must fire it agree on where settings.json lives.
+ home = self.agent_home(os.path.dirname(workspace))
+ env.update({"HOME": home,
+ net_guard.LOG_ENV: self._net_guard_log(home),
+ "CLAUDE_CODE_EFFORT_LEVEL": self.effort,
+ "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
+ "DISABLE_TELEMETRY": "1", "DISABLE_AUTOUPDATER": "1"})
+ # The agent runs on the HOST and inherits the operator's whole shell, so it gets
+ # exactly one Anthropic credential and nothing that could point it at another
+ # endpoint or model catalog -- see swebench.agent_credential_env.
+ agent_credential_env(env, api_key=self.api_key)
+ os.makedirs(env["HOME"], exist_ok=True)
+ # A scaffold copies its binary here (copy_in re-roots absolute paths into it), so
+ # the agent's own shell must find it. Harmless and empty when no scaffold ran.
+ env["PATH"] = os.pathsep.join([self.scaffold_bin_dir(home), env.get("PATH", "")])
+
+ proc = subprocess.Popen(cmd, cwd=workspace, env=env, stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, bufsize=1, start_new_session=True)
+ proc.stdin.write(prompt)
+ proc.stdin.close()
+
+ result, stopped, deadline = {}, False, time.time() + self.task_timeout
+ # Same accounting hazard as the SWE-bench runner, and the same fix: the break paths
+ # below kill the CLI before its terminal `result` event. See
+ # swebench._UsageRecovery. It bites harder here -- the guard is re-made per ATTEMPT
+ # (`guard_per_attempt`), so there are more chances to stop mid-stream per task.
+ recovery, stop_reason = _UsageRecovery(), None
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ try:
+ ev = json.loads(line)
+ except Exception:
+ ev = {}
+ recovery.observe(ev)
+ if ev.get("type") == "result":
+ result = ev
+ if guard is not None and guard.monitor(line):
+ stopped, stop_reason = True, "spiral_guard_stop"
+ break
+ if time.time() > deadline:
+ _log("webbench segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped, stop_reason = True, "task_timeout"
+ break
+
+ if stopped:
+ try:
+ os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
+ time.sleep(2)
+ os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+ except (ProcessLookupError, PermissionError):
+ pass
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-600:]
+ if err.strip() and not result:
+ _log(f"webbench agent stderr: {err.strip()[:300]}")
+ if not result:
+ # Same fix as the SWE-bench runner: `output_tokens` survives the kill in the
+ # CLI's own session log. Here the agent runs on the HOST with $HOME relocated
+ # per task, so it is a plain read rather than a `docker exec` (swebench
+ # `_session_log_text`). It bites harder on this runner -- the guard is re-made
+ # per ATTEMPT -- so more rows per task stood to be flagged.
+ result = recovery.synthesize(stop_reason or "no_result_event",
+ _session_log_text_host(home))
+ return result
+
+ # ---- workspace <-> file-map plumbing ----------------------------------------------
+ def _materialize(self, workspace, files):
+ """Write the evaluator's file map into a fresh scratch workspace."""
+ if os.path.isdir(workspace):
+ shutil.rmtree(workspace, ignore_errors=True)
+ os.makedirs(workspace, exist_ok=True)
+ for rel, content in (files or {}).items():
+ if not rel or rel.startswith("/") or ".." in rel.split("/"):
+ continue # never let a task's file map escape the ws
+ path = os.path.join(workspace, rel)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(content if isinstance(content, str) else "")
+
+ def _collect(self, workspace, request_files, project_dir):
+ """The answer: every file whose content differs from what the request supplied.
+
+ Only changed and new files are returned, which is what the evaluator's Writer
+ applies. Deletion is not expressible in Web-Bench's contract (an empty answer
+ leaves an existing file alone), so a file the agent removed is reported in the row
+ rather than silently dropped.
+ """
+ globs = _load_ignore_globs(self.repo, project_dir)
+ supplied = set(request_files or {})
+ # A scaffold's own artifacts are never part of the agent's answer. Empty with no
+ # scaffold, so the collected set is byte-identical to the pre-scaffold form. Only
+ # files the REQUEST did not supply are dropped: if a project genuinely ships a file
+ # by one of these names, it stays the agent's to edit.
+ scaffold_files = self.scaffold_answer_excludes()
+ # Directories the request itself populated are legitimate no matter what they are
+ # called, so a project that really ships a dot-directory is not silently pruned.
+ supplied_dirs = {d for rel in supplied
+ for d in _ancestors(rel)}
+ out, seen, skipped = {}, set(), []
+ for root, dirs, names in os.walk(workspace):
+ base = os.path.relpath(root, workspace).replace(os.sep, "/")
+ base = "" if base == "." else base
+ def _keep_dir(d, base=base):
+ rel = f"{base}/{d}" if base else d
+ if rel in supplied_dirs:
+ return True
+ return d not in _SKIP_DIRS and not d.startswith(".")
+ dirs[:] = [d for d in dirs if _keep_dir(d)]
+ for name in names:
+ path = os.path.join(root, name)
+ rel = os.path.relpath(path, workspace).replace(os.sep, "/")
+ if rel not in supplied:
+ # New files are held to the skip rules; a dotfile the agent invented is
+ # scratch, while one the request supplied is project source.
+ if name in _SKIP_FILES or name.startswith("."):
+ continue
+ if rel in scaffold_files or name in scaffold_files:
+ skipped.append(rel)
+ continue
+ if _ignored(rel, globs):
+ # The evaluator's own ignore set would discard this answer, so
+ # sending it would be a no-op. Record it instead of dropping it
+ # silently: a project that hides its entry point from the agent
+ # (react's .evalignore hides index.html) makes an agent recreate it,
+ # and that wasted work should be visible in the row.
+ skipped.append(f"{rel} (evaluator ignores it)")
+ continue
+ try:
+ size = os.path.getsize(path)
+ except OSError:
+ continue
+ if size > _MAX_FILE_BYTES:
+ skipped.append(f"{rel} ({size}B > cap)")
+ continue
+ try:
+ with open(path, encoding="utf-8") as fh:
+ content = fh.read()
+ except (OSError, UnicodeDecodeError):
+ skipped.append(f"{rel} (unreadable/binary)")
+ continue
+ seen.add(rel)
+ if (request_files or {}).get(rel) != content:
+ out[rel] = content
+ deleted = sorted(set(request_files or {}) - seen)
+ return out, deleted, skipped
+
+ # ---- the AgentRunner entry point --------------------------------------------------
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ task_id = self._current_task = task["instance_id"]
+ set_optimizer_context(arm=arm, instance_id=task_id, model=self.model,
+ agent=self.agent, cell=_cell_of(run_dir_path),
+ run_dir=os.path.basename(run_dir_path))
+ project = (task.get("repo") or "").split("/")[-1]
+ if not project:
+ raise RuntimeError(f"webbench task {task_id!r} carries no project in `repo`; "
+ f"the driver's --repo selects the Web-Bench project")
+ os.makedirs(run_dir_path, exist_ok=True)
+ max_turns = int(run_kwargs.get("max_turns") or os.environ.get("TE_MAX_TURNS", 80))
+ t0 = time.time()
+ project_dir = os.path.join(self.repo, "projects", project)
+
+ row = {"instance": task_id, "repo": task.get("repo", ""),
+ "model": self.model, "agent": self.agent, "arm": arm,
+ "effort": self.effort, "cli_version": self.cli_version,
+ "domain": "webbench", "webbench_project": project}
+
+ sess = self.session_for(project)
+
+ # A Web-Bench stream is sequentially dependent: once a task has failed every
+ # attempt the evaluator stops, and the tasks after it were never attempted. Return
+ # that fact rather than blocking on a Started line that will never come -- and
+ # rather than billing a zero-cost "failure" that reads as a cheap success.
+ if self._stream_broken or sess.finished:
+ with open(os.path.join(run_dir_path, "transcript.jsonl"), "w",
+ encoding="utf-8"):
+ pass # an empty transcript, so trace-readers find a file
+ row.update(self._row_from_segment({}))
+ row.update({"resolved": False, "stream_broken": True, "unreached": True,
+ # None, not False: an unreached task was never attempted, so it has
+ # no pass@1 or pass@N verdict to report.
+ "resolved_p1": None, "resolved_p2": None,
+ "webbench_attempt_results": [],
+ "webbench_attempts": 0,
+ "wall_s": round(time.time() - t0, 1),
+ "untooled_excluded": list(exclude_tools or [])})
+ _log(f"{task_id} arm={arm} UNREACHED (stream stopped earlier)")
+ return row, run_dir_path
+
+ self._write_run_config(run_dir_path, task, arm, injected, exclude_tools,
+ max_turns, run_kwargs, sess, project)
+ # Installed per TASK rather than per segment: a subclass may override `_segment`
+ # (webbench_smoke does), and a guard that a runner can opt out of by overriding one
+ # method is not a guard.
+ net_guard_installed = self._install_net_guard(
+ self.agent_home(run_dir_path), run_dir_path)
+
+ transcript = os.path.join(run_dir_path, "transcript.jsonl")
+ merged, resolved, attempts = {}, False, 0
+ attempt_results = [] # per-attempt verdict, for pass@1 vs pass@N
+ deleted_all, skipped_all, answered = [], [], []
+ scaffold_notes = {}
+ # One guard per ATTEMPT, not per task: each retry is a fresh rollout with its own
+ # turn envelope, and a guard that spent its nudge budget on attempt 1 would leave
+ # attempt 2 unguarded -- understating the mechanism it is there to measure.
+ guards = []
+
+ with open(transcript, "w", encoding="utf-8") as tfh:
+ for attempt in range(self.retry):
+ # 1. the evaluator announces the attempt it is starting ...
+ started = sess.next_event("started", self.start_timeout)
+ if started is None:
+ _log(f"{task_id}: evaluator produced no attempt "
+ f"(exited={sess.finished}) -- stream broken")
+ row["stream_broken"] = self._stream_broken = True
+ break
+ if started["task"] != task_id:
+ raise RuntimeError(
+ f"webbench loop desync: the driver is on {task_id!r} but the "
+ f"evaluator started {started['task']!r}. The driver's task order "
+ f"must match projects/{project}/tasks.jsonl.")
+
+ # 2. ... then asks the agent for files.
+ req, reply_q = sess.next_request(self.start_timeout)
+ if req is None:
+ _log(f"{task_id}: no agent request after the Started line")
+ row["stream_broken"] = self._stream_broken = True
+ break
+ attempts += 1
+ req_files = req.get("files") or {}
+ guard = self._make_guard(run_kwargs)
+ if guard is not None:
+ guards.append(guard)
+
+ workspace = os.path.join(run_dir_path, f"ws-attempt{attempt + 1}")
+ self._materialize(workspace, req_files)
+ # Install the arm's external tool baseline ONCE per task-run, before the
+ # first segment. The agent's HOME is per task-run and shared across
+ # attempts and spiral segments, so re-installing per attempt would only
+ # repeat the work; the notes from the first install describe the whole row.
+ if self.scaffold is not None and not scaffold_notes:
+ scaffold_notes = self._apply_scaffold(
+ workspace, self.agent_home(run_dir_path), task, run_dir_path)
+ prompt = build_prompt(task, injected, req, project,
+ workspace=workspace)
+ with open(os.path.join(run_dir_path,
+ f"agent_request_{attempt + 1}.json"), "w",
+ encoding="utf-8") as fh:
+ json.dump({"type": req.get("type"), "task": req.get("task"),
+ "error": req.get("error"),
+ "files": sorted(req_files)}, fh, indent=2)
+
+ result = self._segment(workspace, prompt, tfh, exclude_tools,
+ max_turns, guard)
+ # Spiral rung 1: re-invoke in the SAME workspace on top of the agent's own
+ # edits, with the leakage-safe redirect appended.
+ while guard is not None:
+ nxt = guard.continuation(prompt)
+ if not nxt:
+ break
+ _log(f"{task_id}: spiral nudge -> segment {guard.segment}")
+ seg = self._segment(workspace, nxt, tfh, exclude_tools,
+ max_turns, guard)
+ result = self._merge_segments(result, seg)
+ merged = self._merge_segments(merged, result) if merged else result
+
+ # 3. answer with the files that changed, and let it grade.
+ files, deleted, skipped = self._collect(workspace, req_files, project_dir)
+ deleted_all += deleted
+ skipped_all += skipped
+ answered.append(sorted(files))
+ usage = self._segment_usage(result)
+ reply_q.put({
+ "files": files,
+ "inputTokens": (usage.get("input_tokens") or 0)
+ + (usage.get("cache_read_input_tokens") or 0)
+ + (usage.get("cache_creation_input_tokens") or 0),
+ "outputTokens": usage.get("output_tokens") or 0,
+ "trajectory": os.path.join(run_dir_path, "transcript.jsonl"),
+ })
+
+ verdict = sess.next_event("verdict", self.verdict_timeout)
+ if verdict is None:
+ _log(f"{task_id}: no verdict within {self.verdict_timeout}s "
+ f"(exited={sess.finished})")
+ row["stream_broken"] = self._stream_broken = True
+ break
+ if verdict["task"] != task_id:
+ raise RuntimeError(
+ f"webbench loop desync: verdict for {verdict['task']!r} while the "
+ f"driver is on {task_id!r}")
+ if not self.keep_workspaces:
+ shutil.rmtree(workspace, ignore_errors=True)
+ attempt_results.append(bool(verdict["ok"]))
+ if verdict["ok"]:
+ resolved = True
+ break
+
+ # An `init` task is the one exception: the evaluator's InitTaskRunner reports success
+ # regardless and falls back to src-init, so a failed init does NOT end the stream.
+ if not resolved and task_id != "init" and attempts >= self.retry:
+ self._stream_broken = True
+ _log(f"{task_id}: failed all {attempts} attempts -- the Web-Bench stream ends "
+ f"here, later tasks are unreached")
+
+ row.update(self._row_from_segment(merged))
+ # Web-Bench grades each task with up to TE_WEBBENCH_RETRY attempts, so a task has
+ # TWO defensible resolve metrics and they differ materially (measured on prisma:
+ # 1-4 tasks per arm). Record both rather than picking one:
+ # resolved_p1 the FIRST attempt passed -- Web-Bench's own headline metric
+ # resolved_p2 ANY attempt passed (pass@N at the configured retry)
+ # `resolved` stays pass@N: it is what the evaluator itself acts on (a task that
+ # fails every attempt ends the stream), so changing it would change the run, not
+ # just the report. Quote which one you mean -- they are not interchangeable.
+ row["resolved"] = resolved
+ row["resolved_p1"] = attempt_results[0] if attempt_results else None
+ row["resolved_p2"] = any(attempt_results) if attempt_results else None
+ row["webbench_attempt_results"] = attempt_results
+ row["webbench_attempts"] = attempts
+ row["webbench_answered_files"] = answered
+ if deleted_all:
+ row["webbench_unexpressible_deletions"] = sorted(set(deleted_all))
+ if skipped_all:
+ row["webbench_skipped_files"] = sorted(set(skipped_all))
+ row["wall_s"] = round(time.time() - t0, 1)
+ row["untooled_excluded"] = list(exclude_tools or [])
+ # How many times the PreToolUse guard refused a Bash command that reached for the
+ # benchmark's own answers (see te_harness.guards.net_guard). Recorded on EVERY row,
+ # zero included: a guard is only trustworthy if you can see it firing, and a model
+ # that starts routing around it shows up here as a count that falls to zero while
+ # the traces still contain fetches.
+ row["net_guard_blocks"] = self._net_guard_blocks(run_dir_path)
+ # Whether the guard was actually ARMED, which the block count cannot tell you: 0
+ # blocks on an armed guard and 0 blocks on a guard that was never registered are
+ # the same number. Recorded here for the same reason SWE-bench records it -- an
+ # unguarded row must be filterable rather than silently pooled with guarded ones.
+ row["net_guard_installed"] = net_guard_installed
+ # The agent-independent half of the same question, counted off the trace rather
+ # than off the hook: `net_guard_blocks: 0` reads identically on an armed guard that
+ # never fired and on a guard that was never registered, so a row needs both. It
+ # matters most on the Codex lane, where the hook is not honoured at all and
+ # `answer_key_hosts_blocked` is what does the guarding.
+ row["answer_key_attempts"] = answer_key_attempts(run_dir_path)
+ # Scaffold provenance (rtk_installed / rtk_hook_registered / scaffold_error / ...).
+ # A failed install ran the task NATIVE, so these fields are what tells a real
+ # scaffold row from a baseline row wearing a scaffold's name -- filter on them
+ # before quoting a scaffold arm's mean (docs/HARNESS.md, "External tool
+ # scaffolds").
+ if scaffold_notes:
+ row.update(scaffold_notes)
+ row["scaffold"] = self.scaffold.name
+ if guards:
+ # Summed over the task's attempts, so the ledger sees the total the mechanism
+ # actually spent rather than only the last rollout's share.
+ row["aborted"] = any(g.aborted for g in guards)
+ row["spiral_abort"] = row["aborted"]
+ row["spiral_nudges"] = sum(g.nudges_done for g in guards)
+ row["spiral_abort_suppressed"] = any(
+ getattr(g, "abort_suppressed", False) for g in guards)
+ row["spiral_events"] = [e for g in guards for e in (g.events or [])]
+ row["spiral_judge_tokens"] = sum(g.judge_tokens for g in guards)
+ row["spiral_nudge_writer_tokens"] = sum(g.nudge_writer_tokens for g in guards)
+ _log(f"{task_id} arm={arm} resolved={resolved} attempts={attempts} "
+ f"turns={row.get('num_turns')} eff_tok={pricing.effective_tokens(row):.0f} "
+ f"{row['wall_s']}s")
+ return row, run_dir_path
+
+ # ---- accounting hooks (overridden per agent) ---------------------------------------
+ # Claude Code reports its token breakdown in the stream's terminal `result` event.
+ # Copilot reports NO tokens there and must mine its debug log instead, so these three
+ # are hooks: everything else -- the evaluator session, the workspace round-trip, the
+ # spiral segments -- is identical across the two agents.
+ def _merge_segments(self, a, b):
+ return _merge_usage(a, b)
+
+ def _row_from_segment(self, result):
+ return _row_from_result(result)
+
+ def _segment_usage(self, result):
+ """The {input_tokens, cache_read_input_tokens, output_tokens, ...} dict reported
+ back to the Web-Bench evaluator for this segment."""
+ return (result or {}).get("usage") or {}
+
+ def _make_guard(self, run_kwargs):
+ """The spiral guard for one rollout, or None when the arm has it switched off."""
+ if not run_kwargs.get("spiral_guard"):
+ return None
+ guard_kwargs = dict(
+ envelope=run_kwargs.get("abort_envelope"),
+ judge=run_kwargs.get("spiral_judge"),
+ thr=run_kwargs.get("spiral_thr"),
+ nudge_budget=int(run_kwargs.get(
+ "spiral_nudge_budget",
+ 1 if run_kwargs.get("spiral_nudge") else 0)),
+ nudge_writer=run_kwargs.get("spiral_nudge_writer"),
+ verbose=os.environ.get("TE_VERBOSE", "1") == "1")
+ # `never_abort` is CONSERVATIVE-MODE-ONLY and absent from the vendored guard on
+ # branches without trimech-safe -- see swebench.guard_accepts() for why it is
+ # probed. This used to pass it unconditionally, which made every guarded Web-Bench
+ # run on such a branch die with a TypeError at task 0.
+ if guard_accepts("never_abort"):
+ guard_kwargs["never_abort"] = bool(run_kwargs.get("spiral_never_abort"))
+ return spiral_vendor.NudgeGuard(**guard_kwargs)
+
+ def _write_run_config(self, run_dir_path, task, arm, injected, exclude_tools,
+ max_turns, run_kwargs, sess, project):
+ """Record everything that shapes this run into `run_config.json`.
+
+ Same contract as the SWE-bench harness: anything that moves cost or resolve is
+ written down, because the CLI records neither the effort level nor the tool mask in
+ its transcript, and the evaluator's grading settings live in a config file the run
+ would otherwise not remember.
+ """
+ # The SAME two-step the CLI invocation performs (see `_segment`): recomputing only
+ # the untool half here recorded `excluded_applied: []` on every row and made a
+ # guarded run byte-identical to an unguarded one in its own provenance.
+ drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]
+ drop = drop + [t for t in network_tool_block() if t not in drop]
+ cfg = {
+ "instance_id": task.get("instance_id"), "repo": task.get("repo"), "arm": arm,
+ "domain": "webbench",
+ "agent": self.agent, "model": self.model, "effort": self.effort,
+ "max_turns": max_turns,
+ "cli_version": self.cli_version, "agent_bin": self.agent_bin,
+ "routing": {
+ "base_url": self.routing_base,
+ "optimizer_model": self.optimizer_model,
+ "optimizer_transport": getattr(self, "optimizer_transport", "api"),
+ },
+ "tools": {
+ "excluded_requested": list(exclude_tools or []),
+ "excluded_applied": drop,
+ "protected": sorted(self.protected_tools),
+ },
+ # What shaped the run externally. `answer_excludes` is Web-Bench's analogue of
+ # SWE-bench's `scaffold.patch_excludes`: names `_collect` refuses to ship to
+ # the evaluator, so a scaffold's own files cannot be graded as the agent's fix.
+ "scaffold": (None if self.scaffold is None else {
+ "name": self.scaffold.name,
+ "answer_excludes": sorted(self.scaffold_answer_excludes()),
+ }),
+ "injection": {
+ "present": bool(injected), "chars": len(injected or ""),
+ "sha256": (hashlib.sha256(injected.encode()).hexdigest()[:16]
+ if injected else None),
+ },
+ "spiral": {
+ "guard": bool(run_kwargs.get("spiral_guard")),
+ "envelope": run_kwargs.get("abort_envelope"),
+ "thresholds": run_kwargs.get("spiral_thr"),
+ "judge": run_kwargs.get("spiral_judge") is not None,
+ "nudge_writer_llm": run_kwargs.get("spiral_nudge_writer") is not None,
+ "nudge_budget": run_kwargs.get(
+ "spiral_nudge_budget",
+ 1 if run_kwargs.get("spiral_nudge") else 0),
+ # applied, not requested: on a guard without the parameter the request
+ # is silently inert, and recording it as true would misdescribe the run.
+ "never_abort": (bool(run_kwargs.get("spiral_never_abort"))
+ if guard_accepts("never_abort") else False),
+ "guard_per_attempt": True,
+ },
+ "webbench": {
+ "project": project,
+ "evaluator_repo": self.repo,
+ "evaluator_config": sess.config_path,
+ "evaluator_log": sess.log_path,
+ "evaluator_hash": sess.hash,
+ "retry": self.retry,
+ "playwright_workers": self.max_workers,
+ "node": self.node,
+ },
+ "timeouts": {"task_s": self.task_timeout,
+ "verdict_s": self.verdict_timeout,
+ "start_s": self.start_timeout},
+ "grading_source": f"@web-bench/{project} playwright suite (cumulative)",
+ }
+ with open(os.path.join(run_dir_path, "run_config.json"), "w",
+ encoding="utf-8") as fh:
+ json.dump(cfg, fh, indent=2, default=str)
+
+
+TASK_TEMPLATE = """You are a web developer working in the project directory {cwd}.
+
+The project is built up over a sequence of tasks; every file already present is the result
+of the previous tasks and is your starting point. Implement ONLY the task below.
+
+<task>
+{description}
+</task>
+{error_block}
+Rules:
+- Work only inside {cwd}. Edit the existing files and add new ones as the task requires.
+- The project's dependencies are NOT installed here and its test suite is not available,
+ so do not try to install packages, start a dev server, or run tests -- read the existing
+ code to infer the conventions, framework and file layout, and write code that follows
+ them.
+- Keep the previous tasks working: a hidden suite re-checks every earlier task as well as
+ this one.
+- Do not create test files, notes, or scratch files; only the project's own source.
+- When the task is implemented, stop.
+"""
+
+ERROR_TEMPLATE = """
+Your previous attempt at this task FAILED its checks with:
+
+<failure>
+{error}
+</failure>
+
+Fix the cause and implement the task correctly.
+"""
+
+
+# The evaluator appends two response-FORMAT instructions to an `init` task, because its own
+# reference agent answers with a markdown code block and has no other way to name the file
+# it is creating. This harness answers with a file map -- the agent writes real files -- so
+# the instructions describe a transport that is not in use, and following them would put a
+# filename line inside the file's contents.
+_INIT_FORMAT_LINES = (
+ "Add filename after code block.",
+ "The filename should appear on the line following the language specifier in your code "
+ "block.",
+)
+
+
+def strip_transport_instructions(description):
+ """Remove the evaluator's code-block formatting protocol from a task description."""
+ kept = [ln for ln in description.splitlines()
+ if ln.strip() not in _INIT_FORMAT_LINES]
+ return "\n".join(kept).strip()
+
+
+def build_prompt(task, injected, request, project, workspace="."):
+ """Task prompt = the economizer's learned manual (if any) + the Web-Bench task.
+
+ The description comes from the EVALUATOR's request, not the driver's task file: the
+ evaluator is the authority on what it asked for, it decorates an `init` task with the
+ project's existing-file list, and on a retry it supplies the previous attempt's
+ failure -- all part of Web-Bench's contract.
+ """
+ description = strip_transport_instructions(
+ (request.get("task") or task.get("problem_statement") or "").strip())
+ err = (request.get("error") or "").strip()
+ body = TASK_TEMPLATE.format(
+ cwd=workspace, description=description,
+ error_block=ERROR_TEMPLATE.format(error=err[:8000]) if err else "")
+ return (injected.rstrip() + "\n\n" + body) if injected else body
+
+
+class CopilotWebBenchRunner(WebBenchRunner):
+ """AgentRunner: the GitHub Copilot CLI against Web-Bench, graded by Web-Bench's own
+ evaluator.
+
+ Web-Bench runs the agent on the HOST in a scratch workspace (no container), so this
+ subclass is smaller than the SWE-bench one: only construction, the launch argv, and the
+ accounting change. The evaluator session, workspace round-trip, spiral segments and
+ file collection are inherited unchanged.
+
+ Three things to know before comparing arms with the Claude runner:
+
+ * **No turn cap.** Copilot has no `--max-turns`; `max_turns` is recorded but not
+ enforced. The wall-clock `timeout` is the real bound -- and it must stay UNDER the
+ evaluator's ~300s undici header timeout, or the whole stream's chain breaks, not
+ just this task. That is why the default here is tighter than SWE-bench's.
+ * **Tokens come from the debug log**, not the JSON stream (see te_harness.copilot).
+ The per-segment figure reported back to the evaluator is the mined one.
+ * **Auth is on-disk GitHub auth** (or the CLI's own BYOK provider), so there is no
+ ANTHROPIC_API_KEY requirement.
+ """
+
+ def __init__(self, agent="copilot", model=None, repo=None, scaffold=None):
+ self.agent = agent
+ # 'auto' normalises to None: it is a routing directive, not a billable model id,
+ # and recording it as one makes pricing hard-fail at the END of a paid rollout.
+ self.model = copilot.normalize_model(model or os.environ.get("COPILOT_MODEL"))
+ copilot.validate_priceable(self.model, self.agent)
+ # Same rule as the Claude path: the optimizer is the agent's own model. It was
+ # already CALLED that way (optimizer_call substitutes self.model for the "agent"
+ # label), but it was RECORDED as "agent" -- so run_config.json described a cost
+ # basis the run did not use.
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "
+ "runs on the agent's own model so mechanism overhead is priced on the "
+ "same basis as the task rows. Unset it.")
+ self.optimizer_model = self.model or "auto"
+ self.effort_requested = (os.environ.get("TE_EFFORT_LEVEL")
+ or os.environ.get("COPILOT_EFFORT_LEVEL") or "high")
+ if self.effort_requested not in copilot.EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort_requested!r} is not one of "
+ f"{sorted(copilot.EFFORT_LEVELS)}")
+ # The router model rejects --effort and exits, so effort is only APPLIED when a
+ # model is pinned. Recorded as null otherwise, because on `auto` the effort is
+ # whatever the router chose -- not a controlled part of the cost basis.
+ self.effort = (self.effort_requested
+ if copilot.effort_applies(self.model) else None)
+ if self.effort is None:
+ why = ("the launcher probed this model and the CLI rejected --effort"
+ if os.environ.get("TE_EFFORT_APPLIES") == "0"
+ else "the 'auto' router takes no effort level; pin COPILOT_MODEL "
+ "to a served model id to control it")
+ _log(f"copilot: TE_EFFORT_LEVEL={self.effort_requested} NOT applied "
+ f"(model={self.model or 'auto'}, recorded as null) -- {why}.")
+ self.copilot = copilot.require_install() # host-only: no container seed here
+ self.agent_bin = self.copilot
+ self.protected_tools = copilot.PROTECTED_TOOLS
+ # Copilot's own BYOK bills a custom provider instead of the subscription (see
+ # copilot.provider_base) -- a different cost basis, so run_config.json names it.
+ self.routing_base = copilot.provider_base()
+ # Mechanism-call transport, same rule and same reason as the SWE-bench harness
+ # (copilot.anthropic_optimizer_endpoint): the Messages API when the lane's BYOK
+ # provider speaks it and so bills what its tasks bill, else the CLI. Kept in step
+ # with that harness deliberately -- two transports for the same call would make the
+ # two domains' overhead incomparable.
+ self.optimizer_endpoint = copilot.anthropic_optimizer_endpoint()
+ self.optimizer_transport = "api" if self.optimizer_endpoint else "cli"
+ if self.optimizer_endpoint:
+ _log(f"copilot: mechanism calls -> {self.optimizer_endpoint[0]} "
+ "(the BYOK provider, direct Messages API)")
+ self._preflight_optimizer_endpoint()
+ self.cli_version = copilot.cli_version(self.copilot)
+ self.repo = os.path.realpath(repo) if repo else webbench_repo()
+ self.node = node_bin()
+ # Same seam as the Claude runner; see WEBBENCH_SCAFFOLDS.
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile_for("/nonexistent"))
+ _log(f"scaffold: {self.scaffold.name} (external baseline)")
+ # 16384, matching every other runner: one cap for the mechanisms' calls across
+ # both agents and both domains. It was 8192 here alone, which is a real hazard
+ # rather than a tidiness point -- a cap too low truncates a judge's per-tool JSON
+ # into an unparseable answer, which reads downstream as "no verdict" while still
+ # billing the call. Raising it can only remove truncations, never add them.
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ # Under the evaluator's ~300s undici header timeout, for the chain-integrity reason
+ # documented on the Claude agent: a segment that outlives it breaks the stream.
+ self.task_timeout = webbench_task_timeout()
+ self.retry = max(1, int(os.environ.get("TE_WEBBENCH_RETRY", "2")))
+ self.max_workers = int(os.environ.get("TE_WEBBENCH_MAX_WORKERS", "4"))
+ self.verdict_timeout = int(os.environ.get("TE_WEBBENCH_VERDICT_TIMEOUT", "1800"))
+ self._init_stream_state()
+
+ def _preflight_optimizer_endpoint(self):
+ """Refuse the lane if its BYOK provider will not answer a mechanism call -- the
+ SWE-bench harness's check, through the same shared helper and the same
+ TE_OPTIMIZER_PREFLIGHT gate. Only reached on the API transport."""
+ base, key, bearer = self.optimizer_endpoint
+ preflight_optimizer_endpoint(
+ base_url=base, api_key=key, bearer=bearer,
+ configured_model=self.optimizer_model, label="copilot",
+ remedy="Check COPILOT_PROVIDER_BASE_URL and the credential that goes with it.")
+
+ def optimizer_call(self, prompt, model="agent"):
+ if self.optimizer_endpoint:
+ base, key, bearer = self.optimizer_endpoint
+ return anthropic_optimizer_call(
+ prompt, model=model, configured_model=self.optimizer_model,
+ max_tokens=self.optimizer_max_tokens,
+ api_key=key, bearer=bearer, base_url=base)
+ return copilot.optimizer_call(
+ prompt, binary=self.copilot,
+ model=(model if model not in ("agent", "copilot") else self.model))
+
+ # ---- one streamed agent segment ----------------------------------------------------
+ def scaffold_profile_for(self, home):
+ """Copilot's profile. Instructions and the hook spelling differ from Claude's, and
+ the scaffold branches on `kind` for exactly that reason."""
+ return scaffolds_pkg.AgentProfile(
+ kind="copilot",
+ home=home,
+ instructions_file=".github/copilot-instructions.md",
+ nav_tools=("view", "grep", "glob"),
+ mcp_args=lambda cfg: [],
+ patch_excludes=(".github/copilot-instructions.md", ".github/hooks"),
+ bin_dir=self.scaffold_bin_dir(home))
+
+ def scaffold_exec_env(self, run_dir_path):
+ """Point `copilot plugin install` at the isolated config dir, not ~/.copilot."""
+ return {"COPILOT_HOME": self.scaffold_copilot_home(run_dir_path)}
+
+ # Copilot reads hooks from `$COPILOT_HOME/settings.json` -- NOT `$HOME/.claude/`, and
+ # not `hooks.json` (probed on CLI 1.0.83, 2026-09-08: a `hooks.json` at that level is
+ # ignored, a `settings.json` fires). Inheriting the Claude path wrote a file this CLI
+ # never reads, which is why every Copilot Web-Bench row was unguarded while reporting
+ # nothing wrong.
+ NET_GUARD_HOOK_KEY = "preToolUse"
+
+ def net_guard_settings_path(self, home, run_dir_path):
+ # The isolated COPILOT_HOME, which is also what `_segment` exports, so the file is
+ # in the dir the agent will actually read. `home` is unused on this path.
+ return os.path.join(self.scaffold_copilot_home(run_dir_path), "settings.json")
+
+ def net_guard_hook_entry(self, command):
+ """Copilot's entry shape: FLAT, matcher `bash` lower-case."""
+ return {"matcher": "bash", "type": "command", "command": command}
+
+ def prepare_agent_home(self, home):
+ """Link the per-task HOME's package cache to the real one before installing.
+
+ The scaffold's install runs `copilot plugin install` under the per-task `$HOME`
+ (rtk passes it explicitly), and the CLI keeps a ~143MB package cache at
+ `$HOME/.cache/copilot`. Without this the cache is re-downloaded into every
+ task-run's throwaway home -- measured 143MB per run dir, which over one Web-Bench
+ arm is hundreds of GB and one package download per task, and fails outright on a
+ host with no network to the package host. Symlinking reuses the real cache exactly
+ as an unscaffolded run does, while everything else about the home stays per-task.
+ """
+ real = os.path.join(os.path.expanduser("~"), ".cache", "copilot")
+ if not os.path.isdir(real):
+ return
+ cache = os.path.join(home, ".cache")
+ os.makedirs(cache, exist_ok=True)
+ link = os.path.join(cache, "copilot")
+ if not os.path.exists(link):
+ try:
+ os.symlink(real, link)
+ except OSError as e: # non-fatal: costs disk, not the run
+ _log(f"could not link the copilot package cache ({e}); "
+ f"it will be re-downloaded into {cache}")
+
+ def scaffold_copilot_home(self, run_dir_path):
+ """An ISOLATED `COPILOT_HOME` for a task-run, seeded with the real auth.
+
+ This is the one place a Web-Bench run could damage the operator's machine.
+ Unlike the Claude segment, this runner deliberately does NOT relocate `$HOME` --
+ Copilot keeps a ~170MB package cache under `$HOME/.cache/copilot` that a fresh HOME
+ would orphan every task. But both things that register a hook write into
+ `COPILOT_HOME` (default `~/.copilot`): `copilot plugin install` for rtk, and the
+ net guard's `settings.json`. Run as-is, either would land in the operator's REAL
+ config and stay there after the sweep.
+
+ `COPILOT_HOME` is a separate knob from `$HOME`, so the fix is to point only that at
+ a per-task dir and copy in the small auth/config files (the 29MB of `session-state`
+ is disposable). The package cache stays where it is.
+
+ Used on **every** arm, not only scaffolded ones -- the net guard is unconditional,
+ so there is no arm that leaves this dir untouched. (The name is historical: it was
+ scaffold-only when scaffolds were the only writer.)
+ """
+ dest = os.path.join(run_dir_path, ".copilot-home")
+ if os.path.isdir(dest):
+ return dest
+ os.makedirs(dest, exist_ok=True)
+ src = copilot.copilot_config()
+ for name in ("config.json", "settings.json", "mcp-config.json"):
+ path = os.path.join(src, name)
+ if os.path.isfile(path):
+ shutil.copy2(path, os.path.join(dest, name))
+ return dest
+
+ def _segment(self, workspace, prompt, transcript_fh, exclude_tools, max_turns, guard):
+ """Run `copilot -p` once in `workspace`. Returns a usage dict (not a Claude
+ `result` event). `max_turns` is accepted and ignored -- Copilot has no turn cap."""
+ # Per-segment log dir: the miner sums every *.log it finds, so segments must not
+ # share one or the second segment would re-count the first's usage.
+ seg = getattr(guard, "segment", 0)
+ logdir = os.path.join(os.path.dirname(workspace),
+ f".copilot-runlog-{os.path.basename(workspace)}-{seg}")
+ os.makedirs(logdir, exist_ok=True)
+
+ argv = copilot.build_argv(prompt, log_dir=logdir, model=self.model,
+ effort=self.effort, exclude_tools=exclude_tools)
+ argv[0] = self.copilot
+ env = dict(os.environ)
+ # HOME is deliberately NOT relocated the way the Claude segment relocates it.
+ # Copilot keeps its ~170MB package cache under $HOME/.cache/copilot and its auth
+ # under $HOME/.copilot; a fresh HOME per task would orphan both -- re-downloading
+ # the cache every task, and failing outright on a box with no network to the
+ # package host. Nothing is written into the workspace either way (the CLI keeps its
+ # state in $HOME, not cwd), and `.copilot` is in _SKIP_DIRS as a belt-and-braces
+ # guard in case a future version drops something there.
+ env.update(copilot.run_env())
+ # COPILOT_HOME is redirected to a per-task dir on EVERY arm, and it must be the
+ # SAME dir the installs wrote to (scaffold_exec_env, net_guard_settings_path) or
+ # the agent would look for its hooks somewhere they were never registered. $HOME is
+ # deliberately left alone so the ~170MB package cache under $HOME/.cache/copilot is
+ # not orphaned per task.
+ #
+ # This used to be scaffold-only, on the reasoning that a no-scaffold arm writes
+ # nothing into the config dir and should stay byte-identical to the pre-scaffold
+ # form. The net guard ended that: it applies to every arm, it needs a hook in this
+ # file, and writing one into the operator's REAL ~/.copilot -- then leaving it
+ # there after the sweep -- is exactly the damage the scaffold branch existed to
+ # prevent. So the isolation is now unconditional, which is also one fewer
+ # behavioural difference between the arms.
+ env["COPILOT_HOME"] = self.scaffold_copilot_home(os.path.dirname(workspace))
+ # A scaffold installs its binary into the per-task bin dir, and on this lane the
+ # AGENT's own shell has to find it -- not just the install step. rtk's hook rewrites
+ # `ls -la` into `rtk ls -la` and hands that back to the CLI to run, so a bin dir
+ # missing from PATH means every rewritten call dies `rtk: command not found`. That is
+ # not a silent no-op: reproduced 2026-09-13, the agent flails against the failing
+ # command (19 shell calls for one `ls`), which is the same shape as the 12x-cost
+ # Web-Bench incident in baselines/rtk._hook_binary. Harmless and empty with no
+ # scaffold. $HOME is still deliberately left alone (the package cache above).
+ env["PATH"] = os.pathsep.join(
+ [self.scaffold_bin_dir(self.agent_home(os.path.dirname(workspace))),
+ env.get("PATH", "")])
+
+ proc = subprocess.Popen(argv, cwd=workspace, env=env, stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, bufsize=1, start_new_session=True)
+ stream_lines, stopped = [], False
+ deadline = time.time() + self.task_timeout
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ stream_lines.append(line)
+ if guard is not None and guard.monitor(line):
+ stopped = True
+ break
+ if time.time() > deadline:
+ _log("webbench copilot segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped = True
+ break
+
+ if stopped:
+ try:
+ os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
+ time.sleep(2)
+ os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+ except (ProcessLookupError, PermissionError):
+ pass
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-600:]
+
+ # Same reasoning as the SWE-bench runner: a refused configuration is not a task
+ # failure, it is a launch error that happens to surface late.
+ bad = copilot.fatal_config_error(err)
+ if bad:
+ raise RuntimeError(
+ f"copilot refused this configuration: {bad}\n"
+ f" model={self.model!r} effort={self.effort!r}\n"
+ " fix COPILOT_MODEL / TE_EFFORT_LEVEL (see docs/COPILOT.md step 2).")
+
+ log_text = copilot.read_log_dir(logdir)
+ usage = copilot.row_usage("\n".join(stream_lines), log_text)
+ if err.strip() and not usage.get("num_turns"):
+ _log(f"webbench copilot stderr: {err.strip()[:300]}")
+ usage["_debug_log"] = log_text
+ return usage
+
+ # ---- accounting hooks ---------------------------------------------------------------
+ def _merge_segments(self, a, b):
+ merged = copilot.merge_usage(a, b)
+ merged["_debug_log"] = (a or {}).get("_debug_log", "") + (b or {}).get("_debug_log", "")
+ return merged
+
+ def _segment_usage(self, result):
+ return {k: (result or {}).get(k) or 0
+ for k in ("input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens")}
+
+ def _row_from_segment(self, result):
+ row = dict(result or {})
+ row.setdefault("is_error", False)
+ row.setdefault("cost_usd_vendor", None)
+ # Same degradation predicate the SWE-bench Copilot runner sets, so one filter
+ # covers every row this harness writes. Copilot needs no _UsageRecovery: it mines
+ # its log dir AFTER the kill, so a guard stop costs it nothing.
+ src = row.get("tokens_source")
+ row["usage_incomplete"] = src is not None and src != "debug_log"
+ row["output_tokens_missing"] = row.get("output_tokens") is None
+ row["num_turns_recovered"] = False
+ if row["usage_incomplete"]:
+ row.setdefault("usage_incomplete_reason", f"tokens_source={src}")
+ return row
+
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ row, out_dir = super().run_task(
+ task, arm=arm, injected=injected, exclude_tools=exclude_tools,
+ run_dir_path=run_dir_path, **run_kwargs)
+ # Same filename SWE-bench writes, so trace.parse_catalog(run_dir, 'copilot') works
+ # identically on both domains -- without it the untooling judge sees empty
+ # descriptions, drops nothing, and still bills for the call.
+ log_text = row.pop("_debug_log", "")
+ if log_text:
+ try:
+ with open(os.path.join(out_dir, "copilot_debug.log"), "w",
+ encoding="utf-8") as fh:
+ fh.write(log_text)
+ except OSError as e:
+ _log(f"could not save copilot_debug.log: {e}")
+ return row, out_dir
+
+
+
+
+class CodexWebBenchRunner(WebBenchRunner):
+ """AgentRunner: Codex against Web-Bench, graded by Web-Bench's own evaluator.
+
+ The Codex twin of `CopilotWebBenchRunner`, and it overrides the same seams: construction,
+ the launch invocation, the accounting, and the guard. The evaluator session, workspace
+ round-trip, spiral segments and file collection are inherited unchanged, so a Codex row
+ and a Claude row describe the same lifecycle.
+
+ It bills the OpenAI API directly on OPENAI_API_KEY, exactly as on SWE-bench: one resolver
+ (`codex.resolve_codex_routing`) serves both domains, so the two cannot drift about which
+ endpoint an environment means.
+
+ **The answer-key guard is the part that needed new machinery, and it is the reason this
+ class exists rather than a two-line registration.** Web-Bench runs the agent on the HOST
+ with no per-task container, so the other two agents are guarded by a PreToolUse hook --
+ which Codex provably ignores (measured on 0.153.1 with two live probes; see
+ `swebench.CodexSwebenchRunner._install_net_guard`). On SWE-bench the fallback is the
+ container's `/etc/hosts`; there is no container here. Two candidates were measured:
+
+ * **Codex's own sandbox** (`codex sandbox`) denies network exactly as wanted -- but on
+ 0.153.1 it also denies every write, and `sandbox_workspace_write.writable_roots` /
+ `network_access` are rejected as unknown configuration both in `config.toml` under
+ `--strict-config` and as `-c` overrides. An agent that cannot write cannot do a
+ Web-Bench task, so this is unusable on this build.
+ * **An unprivileged user+mount namespace** carrying the same `/etc/hosts` block the
+ SWE-bench container gets. Measured here: `github.com` resolves to 127.0.0.1 and curl
+ fails, `registry.npmjs.org` still resolves, writes still work, and files created
+ inside are owned by the real user outside. This is what
+ the lane uses, so the Codex Web-Bench row is guarded by the same DNS layer as the
+ Codex SWE-bench row -- and needs no cooperation from the CLI, which is the property
+ the hook lacked.
+
+ `net_guard_installed` is therefore False on this lane, honestly, and
+ `answer_key_hosts_blocked` is the field that says the lane is guarded.
+ """
+
+ # Built once: `unshare` must exist and unprivileged user namespaces must be permitted,
+ # or the lane cannot be guarded and must say so rather than run unguarded.
+ _NS_PROBE = "echo https://github.com/probe"
+
+ def __init__(self, agent="codex", model=None, repo=None, scaffold=None):
+ self.agent = agent
+ # Checked FIRST, before any credential or catalog work: what this lane can host is
+ # not a function of its endpoint. `rtk` is hostable -- in its Codex mode it is
+ # instruction files under CODEX_HOME, not the hook this CLI ignores (baselines/rtk.py)
+ # -- and it is the only scaffold Web-Bench hosts for any agent (WEBBENCH_SCAFFOLDS).
+ if scaffold and scaffold != "rtk":
+ raise RuntimeError(
+ f"the {scaffold!r} baseline is not wired for the Codex Web-Bench lane "
+ f"(Web-Bench hosts {sorted(WEBBENCH_SCAFFOLDS)}; on Codex that is the "
+ f"instruction-mode rtk). A scaffold that installs and never takes effect "
+ f"would produce native rows under the scaffold's name.")
+ # The OpenAI API directly, on OPENAI_API_KEY -- see codex.resolve_codex_routing.
+ self.routing = codex.resolve_codex_routing()
+ self.api_key = self.routing["api_key"]
+ self.model = model or os.environ.get("CODEX_MODEL", "gpt-5.6-sol")
+ codex.validate_priceable(self.model, self.agent)
+ codex.verify_model(self.model, self.routing)
+ if os.environ.get("TE_OPTIMIZER_MODEL"):
+ raise RuntimeError(
+ "TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer runs "
+ "on the agent's own model so mechanism overhead is priced on the same basis "
+ "as the task rows. Unset it.")
+ self.optimizer_model = self.model
+ self.effort = os.environ.get("TE_EFFORT_LEVEL", "high")
+ if self.effort not in EFFORT_LEVELS:
+ raise RuntimeError(f"TE_EFFORT_LEVEL={self.effort!r} is not one of "
+ f"{sorted(EFFORT_LEVELS)}")
+ codex.require_install()
+ self.codex_vendor = codex.codex_vendor()
+ self.agent_bin = codex.codex_bin()
+ self.cli_version = codex.cli_version(self.agent_bin)
+ self.protected_tools = PROTECTED_TOOLS
+ # The captured base prompt the lean-instructions package rewrites. Absent on a CLI
+ # build with no capture: the package is then unavailable, stays loaded, and says so
+ # -- a stale prompt is never applied to a newer CLI.
+ self.base_prompt = codex.base_prompt_for(self.cli_version)
+ self.system_skills = codex.bundled_system_skills(self.agent_bin) or None
+ self.scaffold_name = scaffold or None
+ self.scaffold = scaffolds_pkg.get(self.scaffold_name)
+ if self.scaffold is not None:
+ self.scaffold.preflight(self.scaffold_profile_for("/nonexistent"))
+ _log(f"scaffold: {self.scaffold.name} (external baseline, instruction mode)")
+ self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))
+ self.repo = os.path.realpath(repo) if repo else webbench_repo()
+ self.node = node_bin()
+ self.task_timeout = webbench_task_timeout()
+ self.retry = max(1, int(os.environ.get("TE_WEBBENCH_RETRY", "2")))
+ self.max_workers = int(os.environ.get("TE_WEBBENCH_MAX_WORKERS", "4"))
+ self.verdict_timeout = int(os.environ.get("TE_WEBBENCH_VERDICT_TIMEOUT", "1800"))
+ # Fields the INHERITED lifecycle reads. They are set here rather than left to the
+ # base __init__ (which this class does not call, because it resolves the Anthropic
+ # key) and a test pins the whole set, since the failure mode is an AttributeError
+ # raised after the evaluator has started and the first task has been paid for --
+ # which is exactly how the missing one was found, on a live run.
+ self.routing_base = self.routing["base_url"]
+ self.optimizer_transport = "api" # codex.optimizer_call, always over /v1/responses
+ self.claude = None # no Claude binary on this lane
+ self.keep_workspaces = os.environ.get("TE_KEEP_WORKSPACES") == "1"
+ self.start_timeout = int(os.environ.get("TE_WEBBENCH_START_TIMEOUT", "1800"))
+ self._capabilities = ()
+ self._lean_instructions_tokens = 0
+ _log(f"codex: model={self.model} endpoint={self.routing_base}")
+ self.answer_key_ns = self._probe_namespace_guard()
+ self._preflight_optimizer_endpoint()
+ self._init_stream_state()
+
+ # ---- the answer-key layer ----------------------------------------------------------
+ def _probe_namespace_guard(self):
+ """Prove an unprivileged user+mount namespace can blackhole the answer-key hosts.
+
+ Probed at CONSTRUCTION, before task 0, because the alternative is a lane that runs
+ unguarded while every row reports a guard. Returns True only when the namespace was
+ actually created AND the block took effect inside it.
+ """
+ try:
+ p = subprocess.run(
+ ["unshare", "--user", "--map-root-user", "--mount", "bash", "-c",
+ "getent hosts github.com | head -1"],
+ capture_output=True, text=True, timeout=60)
+ except (OSError, subprocess.TimeoutExpired) as e:
+ _log(f"codex webbench: cannot create a user+mount namespace ({type(e).__name__})"
+ f" -- the answer-key hosts block is NOT available on this host")
+ return False
+ if p.returncode != 0:
+ _log("codex webbench: unprivileged user namespaces are refused on this host "
+ "(check /proc/sys/kernel/unprivileged_userns_clone) -- the answer-key "
+ "hosts block is NOT available")
+ return False
+ _log("codex webbench: answer-key hosts block armed through a user+mount namespace "
+ "(there is no container here; the PreToolUse hook is the second layer)")
+ return True
+
+ def _hosts_file(self, run_dir_path):
+ """This task's doctored /etc/hosts: the real one plus the answer-key block."""
+ path = os.path.join(run_dir_path, ".hosts-block")
+ try:
+ with open("/etc/hosts", encoding="utf-8") as fh:
+ base = fh.read()
+ except OSError:
+ base = "127.0.0.1 localhost\n"
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(base.rstrip("\n") + "\n" + net_guard.hosts_block() + "\n")
+ return path
+
+ def net_guard_settings_path(self, home, run_dir_path):
+ """Codex reads hooks from `$CODEX_HOME/hooks.json` -- and `home` IS the per-task
+ CODEX_HOME here -- in Claude Code's shape: `{"hooks": {"PreToolUse": [...]}}`."""
+ return os.path.join(home, "hooks.json")
+
+ # Set per task by `_install_net_guard`; `_segment` passes the trust bypass only when it
+ # is True, so an unarmed task runs exactly the argv it always did.
+ _net_guard_armed = False
+
+ def _install_net_guard(self, home, run_dir_path=None):
+ """The shared installer, pointed at Codex's hook file -- and Codex honours it.
+
+ Verified live on codex-cli 0.153.1 (2026-09-30): with this file and
+ `--dangerously-bypass-hook-trust` the guard refused a GitHub-bound command before it
+ ran and logged it, while a control command ran. Without the bypass Codex skips every
+ hook nobody has approved in /hooks, which an unattended run cannot do -- so arming
+ here is what turns the bypass on in `_segment`. The namespace hosts block stays the
+ primary layer (docs/HARNESS.md, "The answer-key guard").
+ """
+ self._net_guard_armed = bool(super()._install_net_guard(home, run_dir_path))
+ return self._net_guard_armed
+
+ # ---- construction-time hooks -------------------------------------------------------
+ def _preflight_optimizer_endpoint(self):
+ codex.preflight_optimizer(routing=self.routing, configured_model=self.model)
+
+ def optimizer_call(self, prompt, model="agent"):
+ """One mechanism call against THIS lane's endpoint, billed like the task rows."""
+ return codex.optimizer_call(
+ prompt, configured_model=self.model, routing=self.routing,
+ max_tokens=self.optimizer_max_tokens)
+
+ def agent_home(self, run_dir_path):
+ """`CODEX_HOME` for this task-run. Never inside the workspace (everything there is a
+ candidate answer file) and never under /tmp -- the CLI refuses to create its PATH
+ helper binaries under a temporary dir and says so on every invocation."""
+ return os.path.join(run_dir_path, ".codex-home")
+
+ def scaffold_profile_for(self, home):
+ """Codex's profile, per task-run. `home` IS the CODEX_HOME here, so the env a
+ scaffold installs under points at it; instructions land in `$CODEX_HOME/AGENTS.md`,
+ which the CLI loads, rather than in the workspace, which the evaluator would grade."""
+ return scaffolds_pkg.AgentProfile(
+ kind="codex",
+ home=home,
+ instructions_file="AGENTS.md",
+ nav_tools=(),
+ mcp_args=lambda cfg: [],
+ patch_excludes=("AGENTS.md", "RTK.md"),
+ bin_dir=self.scaffold_bin_dir(home),
+ env={"CODEX_HOME": home})
+
+ def scaffold_exec_env(self, run_dir_path):
+ return {"CODEX_HOME": self.agent_home(run_dir_path)}
+
+ def _write_config(self, home, workspace):
+ """Write this task's request contract into the per-task CODEX_HOME.
+
+ The same `config.toml` the SWE-bench lane installs in its container, with two host
+ differences: the workspace, not `/testbed`, is the trusted project, and the lean
+ instructions file is named by its host path in the per-task CODEX_HOME. Written per
+ SEGMENT because the capability mask is frozen per task and the workspace path is
+ only known here.
+ """
+ os.makedirs(home, exist_ok=True)
+ lean = codex.lean_instructions(getattr(self, "base_prompt", None),
+ self._capabilities)
+ self._lean_instructions_tokens = len(lean) // 4 if lean else 0
+ instr_path = os.path.join(home, "instructions.md")
+ cfg = codex.config_toml(
+ routing=self.routing, model=self.model, effort=self.effort,
+ project=workspace, capabilities=self._capabilities,
+ system_skills=self.system_skills, instructions=lean,
+ instructions_path=instr_path if lean else None)
+ with open(os.path.join(home, "config.toml"), "w", encoding="utf-8") as fh:
+ fh.write(cfg)
+ if lean:
+ with open(instr_path, "w", encoding="utf-8") as fh:
+ fh.write(lean)
+ return cfg
+
+ # ---- one rollout segment ------------------------------------------------------------
+ def _segment(self, workspace, prompt, transcript_fh, exclude_tools, max_turns, guard):
+ """Run `codex exec --json` once in `workspace` (or `codex exec resume` for a spiral
+ segment), inside a user+mount namespace carrying the answer-key hosts block.
+
+ `prompt` is a string for a fresh segment, or the resume REQUEST dict the Codex guard
+ returned. `max_turns` is accepted and ignored: Codex has no turn-cap flag, exactly as
+ on SWE-bench and on the Copilot lane. Returns a usage dict, not a Claude `result`
+ event -- see the accounting hooks.
+ """
+ run_dir = os.path.dirname(workspace)
+ home = self.agent_home(run_dir)
+ self._write_config(home, workspace)
+
+ resume = prompt if isinstance(prompt, dict) else None
+ text = resume["prompt"] if resume else prompt
+ hooked = bool(self._net_guard_armed)
+ if resume:
+ argv = codex.build_resume_argv(resume["session_id"],
+ effort=resume.get("effort"), hook=hooked)
+ else:
+ argv = codex.build_argv(hook=hooked)
+ # `build_argv` ends with `-C /testbed -`; on the host the workspace is the project.
+ argv = [a if a != "/testbed" else workspace for a in argv]
+ argv[0] = self.agent_bin
+
+ env = dict(os.environ)
+ env.update(codex.run_env())
+ env["CODEX_HOME"] = home
+ env[net_guard.LOG_ENV] = self._net_guard_log(home) # where the hook logs refusals
+ env["HOME"] = os.path.join(run_dir, ".host-home")
+ os.makedirs(env["HOME"], exist_ok=True)
+ # A scaffold copies its binary here (copy_in re-roots into bin_dir), so the agent's
+ # own shell must find it. Harmless and empty when no scaffold ran.
+ env["PATH"] = os.pathsep.join([self.scaffold_bin_dir(home), env.get("PATH", "")])
+ # The key the provider block names, under that name.
+ env[codex.KEY_ENV] = self.api_key
+
+ # The answer-key layer. Everything the agent spawns inherits the namespace, so this
+ # needs no cooperation from the CLI -- which is the whole point, since the CLI
+ # ignores the hook the other two agents are guarded by.
+ if self.answer_key_ns:
+ hosts = self._hosts_file(run_dir)
+ inner = "mount --bind %s /etc/hosts && exec \"$@\"" % _shq(hosts)
+ cmd = ["unshare", "--user", "--map-root-user", "--mount",
+ "bash", "-c", inner, "_"] + argv
+ else:
+ cmd = argv
+
+ proc = subprocess.Popen(cmd, cwd=workspace, env=env, stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, bufsize=1, start_new_session=True)
+ proc.stdin.write(text)
+ proc.stdin.close()
+
+ lines, stopped, deadline = [], False, time.time() + self.task_timeout
+ for line in proc.stdout:
+ line = line.strip()
+ if not line:
+ continue
+ transcript_fh.write(line + "\n")
+ transcript_fh.flush()
+ lines.append(line)
+ if guard is not None and guard.monitor(line):
+ stopped = True
+ break
+ if time.time() > deadline:
+ _log("webbench codex segment hit TE_TASK_TIMEOUT; stopping the agent")
+ stopped = True
+ break
+ if stopped:
+ try:
+ os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
+ time.sleep(2)
+ os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+ except (ProcessLookupError, PermissionError):
+ pass
+ proc.kill()
+ try:
+ proc.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ err = (proc.stderr.read() or "")[-1200:]
+ # A refused configuration is a launch error that surfaces late, not a task failure:
+ # `--strict-config` rejects an unknown toggle, which is exactly what a CLI upgrade
+ # that renamed one looks like. Raising keeps a renamed toggle from silently becoming
+ # a no-op on every remaining task of the stream.
+ if "unknown configuration" in err.lower() or "error loading config.toml" in err.lower():
+ raise RuntimeError("codex refused its config at run time:\n" + err.strip()[:600])
+ usage = codex.row_usage("\n".join(lines))
+ if err.strip() and not usage.get("num_turns"):
+ _log(f"agent stderr: {err.strip()[:300]}")
+ return usage
+
+ # ---- accounting ---------------------------------------------------------------------
+ def _merge_segments(self, a, b):
+ return codex.merge_usage(a, b)
+
+ def _segment_usage(self, result):
+ """What is reported back to the Web-Bench evaluator for this segment."""
+ return {k: (result or {}).get(k) or 0
+ for k in ("input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens")}
+
+ def _row_from_segment(self, result):
+ row = dict(result or {})
+ row.setdefault("is_error", False)
+ row.setdefault("cost_usd_vendor", None)
+ # Same degradation predicate as the SWE-bench Codex runner, so one filter covers
+ # every row either domain writes: a stream killed before `turn.completed` takes the
+ # usage block with it, and row_usage reports None (never 0) in every token field.
+ src = row.get("tokens_source")
+ row["usage_incomplete"] = src is not None and src != "stream_usage"
+ row["output_tokens_missing"] = row.get("output_tokens") is None
+ row["num_turns_recovered"] = False
+ if row["usage_incomplete"]:
+ row.setdefault("usage_incomplete_reason", f"tokens_source={src}")
+ return row
+
+ # ---- the spiral guard ----------------------------------------------------------------
+ def _make_guard(self, run_kwargs):
+ """The Codex-native resume controller, when the arm carries one.
+
+ The generic `NudgeGuard` the base class builds reads a Claude stream and reprompts;
+ the Codex controller reads `codex exec --json` and resumes the same thread. Selecting
+ on `codex_spiral_policy` keeps a Codex lane from silently running the generic guard
+ against a stream it cannot parse.
+ """
+ policy = run_kwargs.get("codex_spiral_policy")
+ if policy is None:
+ return None
+ return codex_spiral.CodexResumeGuard(
+ policy=policy, verbose=os.environ.get("TE_VERBOSE", "1") == "1")
+
+ # ---- entry point ---------------------------------------------------------------------
+ def run_task(self, task, *, arm, injected, exclude_tools, run_dir_path, **run_kwargs):
+ """Same lifecycle, with the capability mask frozen before the agent starts.
+
+ The mask reaches the runner two ways and both are needed, exactly as on SWE-bench:
+ `exclude_tools` carries the relay capability ids, and `codex_config_capabilities`
+ carries the logical packages that have a native config toggle. The second is not
+ derivable from the first -- `collaboration` expands to no relay id at all.
+ """
+ self._capabilities = tuple(run_kwargs.get("codex_config_capabilities") or ())
+ row, out_dir = super().run_task(
+ task, arm=arm, injected=injected, exclude_tools=exclude_tools,
+ run_dir_path=run_dir_path, **run_kwargs)
+ row["turns_basis"] = "codex_steps"
+ row["codex_config_capabilities"] = list(self._capabilities)
+ row["codex_lean_instructions_tokens"] = int(self._lean_instructions_tokens or 0)
+ # This lane's honest guard fields: the hook is not honoured, the hosts block is.
+ row["net_guard_installed"] = False
+ row["answer_key_hosts_blocked"] = bool(self.answer_key_ns)
+ if self.scaffold_name == "rtk":
+ row.update(codex.rtk_compliance(os.path.join(out_dir, "transcript.jsonl")))
+ self._prune_agent_home(run_dir_path)
+ return row, out_dir
+
+ # Kept: the contract this task ran under, which is provenance and is ~10 KB. Dropped:
+ # the CLI's own state -- sqlite dbs and their WALs, the bundled skills copy, shell
+ # snapshots, session rollouts -- which is ~38 MB PER TASK-RUN and would put ~19 GB of
+ # throwaway CLI state in the results tree over a five-project, five-seed sweep. The
+ # SWE-bench lane has no equivalent problem because its CODEX_HOME lives inside the task
+ # container and dies with it; this restores that parity. TE_KEEP_WORKSPACES=1 keeps
+ # everything, the same knob that keeps the workspace itself.
+ _AGENT_HOME_KEEP = ("config.toml", "instructions.md", "AGENTS.md", "RTK.md")
+
+ def _prune_agent_home(self, run_dir_path):
+ home = self.agent_home(run_dir_path)
+ if self.keep_workspaces or not os.path.isdir(home):
+ return
+ for name in os.listdir(home):
+ if name in self._AGENT_HOME_KEEP:
+ continue
+ path = os.path.join(home, name)
+ try:
+ shutil.rmtree(path) if os.path.isdir(path) else os.remove(path)
+ except OSError:
+ pass # a file the CLI still holds is not worth failing a row over
+ # The host $HOME the segment made is throwaway in full.
+ try:
+ shutil.rmtree(os.path.join(run_dir_path, ".host-home"))
+ except OSError:
+ pass
+
+
+RUNNERS = {
+ "claude": WebBenchRunner,
+ "copilot": CopilotWebBenchRunner,
+ "codex": CodexWebBenchRunner,
+}
+
+
+def make_runner(agent=None, scaffold=None, **kwargs):
+ """Factory the driver's ``--runner te_harness.webbench:make_runner`` calls.
+
+ Called with the driver's ``--agent`` label when the driver supports it, and with no
+ arguments by older callers (which then get the TE_AGENT default, or claude). An
+ unknown label is an error, never a silent fallback to Claude.
+
+ ``scaffold`` names an external tool baseline to install for every task; the driver
+ derives it from the arm, so the arm registry stays the single source of truth.
+ Web-Bench supports the ones that need no container and no Python symbol map -- rtk
+ today. ``TE_SCAFFOLD`` is the fallback for a caller that cannot pass it.
+ """
+ agent = (agent or os.environ.get("TE_AGENT") or "claude").lower()
+ if agent not in RUNNERS:
+ # An unknown label is an error, never a silent fallback to Claude: rows filed under
+ # the wrong agent are wrong numbers rather than a crash, which is the worst failure
+ # a benchmark can have.
+ raise RuntimeError(
+ f"no Web-Bench runner for --agent {agent!r}; "
+ f"available: {', '.join(sorted(RUNNERS))}")
+ _install_signal_cleanup()
+ scaffold = scaffold or os.environ.get("TE_SCAFFOLD") or None
+ if scaffold is not None and scaffold not in WEBBENCH_SCAFFOLDS:
+ raise RuntimeError(
+ f"the {scaffold!r} baseline is not wired for Web-Bench "
+ f"(supported: {', '.join(sorted(WEBBENCH_SCAFFOLDS))}). Web-Bench runs the "
+ f"agent on the HOST with no per-task repo container, and the repo map "
+ f"generates a PYTHON symbol map that a Node/TypeScript project does not have. "
+ f"See docs/HARNESS.md, \"External tool scaffolds\".")
+ return RUNNERS[agent](agent=agent, scaffold=scaffold, **kwargs)
diff --git a/experiments/te_harness/webbench_smoke.py b/experiments/te_harness/webbench_smoke.py
new file mode 100644
index 0000000..506aa6f
--- /dev/null
+++ b/experiments/te_harness/webbench_smoke.py
@@ -0,0 +1,145 @@
+"""Offline wiring check for the Web-Bench harness: everything except the LLM.
+
+``webbench.WebBenchRunner`` joins two loops that are easy to get subtly wrong -- the
+economizer's driver stepping one task at a time, and Web-Bench's Node evaluator owning its
+own sequential task loop behind an HTTP agent contract. A desync there does not crash; it
+silently pairs one task's answer with another task's grade. So the rendezvous needs a test
+that runs the REAL evaluator, the REAL HTTP contract and the REAL Playwright grading, with
+only the paid agent call replaced.
+
+This runner is that test. It is the production ``WebBenchRunner`` with one method
+overridden: instead of invoking Claude Code, ``_segment`` copies the project's own
+reference solution (``projects/<project>/src``) into the agent workspace and writes a
+synthetic ``result`` event in the CLI's schema. Every task should therefore be graded
+RESOLVED -- the reference solution is what the suite was written against -- so any failure
+is a harness fault, not a model result.
+
+What it proves: the evaluator config, its launch and shutdown, the agent endpoint, the
+Started/verdict parsing, the per-attempt rendezvous, workspace materialization, the
+changed-file answer, the rows the driver writes, and the driver's stream hooks.
+What it does not prove: anything about the agent.
+
+ python3 scripts/run_experiment.py --arm baseline --agent claude --repo calculator \\
+ --adapter webbench --tasks-file "$TOKENECONOMIZER_WEBBENCH_TASKS" \\
+ --workload workloads/webbench.json --limit 3 \\
+ --results-root /tmp/wb_smoke --runner te_harness.webbench_smoke:make_runner --fresh
+
+Its rows are NOT a benchmark: `resolved` reflects the reference solution, not an agent.
+
+Set ``TE_WEBBENCH_SMOKE_FAIL=<task-id>`` to make one task answer with an intentionally empty
+page instead of the reference. That exercises the other half of the rendezvous, which the
+happy path never reaches: the retry loop (a second Started/request/verdict round for the same
+task) and the stream-break short-circuit that must return every later task as unreached
+rather than blocking on a Started line the evaluator will never print. Use it on
+`calculator`, whose whole app is a single `index.html`, so overwriting that file reliably
+fails the suite.
+"""
+import json
+import os
+import shutil
+
+from .webbench import (_MAX_FILE_BYTES, _SKIP_DIRS, _SKIP_FILES, WebBenchRunner, _log,
+ _install_signal_cleanup)
+
+
+class ReferenceWebBenchRunner(WebBenchRunner):
+ """WebBenchRunner with the paid agent call replaced by the reference solution."""
+
+ def __init__(self, *a, **kw):
+ super().__init__(*a, **kw)
+ self.agent = "claude" # keep the transcript schema honest
+ self.fail_task = os.environ.get("TE_WEBBENCH_SMOKE_FAIL", "").strip()
+ _log("webbench_smoke: REFERENCE runner -- no agent is called, rows are NOT a "
+ "benchmark; every task should grade resolved")
+ if self.fail_task:
+ _log(f"webbench_smoke: TE_WEBBENCH_SMOKE_FAIL={self.fail_task} -- that task "
+ f"will answer with an empty page, so it should fail every attempt and "
+ f"end the stream")
+
+ def optimizer_call(self, prompt, model="agent"):
+ """No LLM: a benign, parseable optimizer answer costing a nominal 40 tokens, so the
+ mechanisms' between-task hooks still run and the ledger still sees overhead."""
+ return '{"extractors": [], "facts": []}', 40.0
+
+ def _segment(self, workspace, prompt, transcript_fh, exclude_tools, max_turns, guard):
+ """Copy the reference solution into the workspace, then write a transcript in the
+ CLI's stream-json schema so `tokeneconomizer.trace` has something real to read."""
+ project = self._session_project
+ src = os.path.join(self.repo, "projects", project, "src")
+ copied = _copy_tree(src, workspace)
+ if self.fail_task and self._current_task == self.fail_task:
+ copied = _blank_pages(workspace) or copied
+ _log(f"webbench_smoke: answering {self.fail_task} with an empty page")
+
+ # A minimal but schema-correct transcript: two tool calls and a result event with a
+ # cache-heavy breakdown, which is what pricing.effective_tokens reads.
+ lines = []
+ for i, rel in enumerate(copied[:8]):
+ tid = f"t{i}"
+ lines.append({"message": {"content": [
+ {"type": "tool_use", "id": tid, "name": "Write",
+ "input": {"file_path": rel}}]}})
+ lines.append({"message": {"content": [
+ {"type": "tool_result", "tool_use_id": tid, "content": "ok"}]}})
+ result = {
+ "type": "result", "is_error": False, "num_turns": max(1, len(copied)),
+ "usage": {"input_tokens": 900, "cache_read_input_tokens": 7000,
+ "cache_creation_input_tokens": 400, "output_tokens": 350},
+ "total_cost_usd": 0.0,
+ }
+ lines.append(result)
+ for line in lines:
+ transcript_fh.write(json.dumps(line) + "\n")
+ transcript_fh.flush()
+ return result
+
+
+_EMPTY_PAGE = ("<!DOCTYPE html>\n<html lang=\"en\">\n<head><meta charset=\"UTF-8\">"
+ "<title>empty</title></head>\n<body></body>\n</html>\n")
+
+
+def _blank_pages(workspace):
+ """Overwrite every HTML file in the workspace with a valid but empty page.
+
+ Valid, not empty-string: the evaluator's Writer treats empty content as "leave the
+ existing file alone", and its file validator rejects malformed HTML -- either would make
+ the injected failure a harness error rather than the graded failure being tested.
+ """
+ hit = []
+ for root, dirs, names in os.walk(workspace):
+ dirs[:] = [d for d in dirs if d not in _SKIP_DIRS and not d.startswith(".")]
+ for name in names:
+ if name.endswith(".html"):
+ path = os.path.join(root, name)
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(_EMPTY_PAGE)
+ hit.append(os.path.relpath(path, workspace).replace(os.sep, "/"))
+ return hit
+
+
+def _copy_tree(src, dst):
+ """Copy `src` over `dst`, skipping what is never part of a Web-Bench answer. Returns
+ the relative paths written."""
+ written = []
+ for root, dirs, names in os.walk(src):
+ dirs[:] = [d for d in dirs if d not in _SKIP_DIRS and not d.startswith(".")]
+ for name in names:
+ if name in _SKIP_FILES or name.startswith("."):
+ continue
+ path = os.path.join(root, name)
+ rel = os.path.relpath(path, src).replace(os.sep, "/")
+ try:
+ if os.path.getsize(path) > _MAX_FILE_BYTES:
+ continue
+ except OSError:
+ continue
+ out = os.path.join(dst, rel)
+ os.makedirs(os.path.dirname(out), exist_ok=True)
+ shutil.copyfile(path, out)
+ written.append(rel)
+ return written
+
+
+def make_runner():
+ _install_signal_cleanup()
+ return ReferenceWebBenchRunner()
diff --git a/experiments/tests/conftest.py b/experiments/tests/conftest.py
new file mode 100644
index 0000000..3cc58a8
--- /dev/null
+++ b/experiments/tests/conftest.py
@@ -0,0 +1,15 @@
+"""pytest setup for the reproduction suite (experiments/tests).
+
+`experiments/` keeps the layout the experiments always had -- te_harness/, scripts/,
+workloads/, docs/ -- so a test's "repo root" (`_REPO` in the test files) is experiments/.
+The shared core, `tokeneconomizer`, lives one level up at the repository root, and so do
+the shared transcript writers (`fixtures`), with the plugin's own tests in tests/.
+"""
+import os
+import sys
+
+_TESTS = os.path.dirname(os.path.abspath(__file__))
+_REPO = os.path.dirname(_TESTS)
+for p in (os.path.dirname(_REPO), os.path.join(os.path.dirname(_REPO), "tests"), _REPO, _TESTS):
+ if p not in sys.path:
+ sys.path.insert(0, p)
diff --git a/experiments/tests/test_agent_credentials.py b/experiments/tests/test_agent_credentials.py
new file mode 100644
index 0000000..2c817ac
--- /dev/null
+++ b/experiments/tests/test_agent_credentials.py
@@ -0,0 +1,98 @@
+"""Exactly ONE Anthropic credential reaches the Claude Code process: ANTHROPIC_API_KEY.
+
+The Claude lane bills api.anthropic.com directly. Nothing that could point the CLI at
+another endpoint, credential or model catalog (`swebench.IGNORED_ANTHROPIC_VARS`) may reach
+the agent, whichever way its environment is built:
+
+ * SWE-bench runs the agent in a task container whose environment is a list of NAMED
+ variables, so the key is the only Anthropic one in that list -- passed by name, never
+ by value, so it stays out of argv;
+ * Web-Bench runs the agent on the HOST, where it inherits the operator's whole shell, so
+ the steering variables are removed from that copy before the CLI starts.
+
+Offline: the agent process is a stand-in that records what it was handed.
+"""
+import json
+
+from te_harness import swebench as sw
+from te_harness import webbench as wb
+
+_STEERING = {"ANTHROPIC_BASE_URL": "https://example.invalid",
+ "ANTHROPIC_AUTH_TOKEN": "tok-SECRET",
+ "ANTHROPIC_DEFAULT_HAIKU_MODEL": "claude-haiku-4.5"}
+
+
+class _Proc:
+ """A Popen stand-in: no output, exits cleanly, remembers argv and env."""
+
+ seen = {}
+
+ def __init__(self, cmd, **kw):
+ _Proc.seen = {"cmd": list(cmd), "env": dict(kw.get("env") or {})}
+ self.pid = 1
+ self.stdin = type("W", (), {"write": lambda s, x: None, "close": lambda s: None})()
+ self.stdout = iter([])
+ self.stderr = type("R", (), {"read": lambda s: ""})()
+
+ def wait(self, timeout=None):
+ return 0
+
+ def kill(self):
+ pass
+
+
+def test_host_agent_env_keeps_only_the_key():
+ """Web-Bench's agent inherits the operator's shell; the steering variables go."""
+ env = {"PATH": "/usr/bin", "ANTHROPIC_API_KEY": "sk-ant-leftover", **_STEERING}
+ assert sw.agent_credential_env(env, api_key="sk-ant-x") is env
+ assert env == {"PATH": "/usr/bin", "ANTHROPIC_API_KEY": "sk-ant-x"}
+
+
+def test_the_swebench_container_is_handed_the_key_by_name_and_nothing_else(monkeypatch):
+ for k, v in _STEERING.items():
+ monkeypatch.setenv(k, v)
+ monkeypatch.setattr(sw.subprocess, "Popen", _Proc)
+ r = sw.ClaudeSwebenchRunner.__new__(sw.ClaudeSwebenchRunner)
+ r.api_key, r.model, r.effort = "sk-ant-SECRET", "claude-opus-5", "high"
+ r.protected_tools, r.scaffold, r.task_timeout = sw.PROTECTED_TOOLS, None, 5
+ r._session_log_text = lambda name: ""
+ with open("/dev/null", "w") as fh:
+ r._segment("ctr", "go", fh, [], 80, None)
+ cmd = " ".join(_Proc.seen["cmd"])
+ assert "-e ANTHROPIC_API_KEY " in cmd # by name...
+ assert "sk-ant-SECRET" not in cmd # ...never by value
+ for var in sw.IGNORED_ANTHROPIC_VARS:
+ assert var not in cmd, var
+ assert _Proc.seen["env"]["ANTHROPIC_API_KEY"] == "sk-ant-SECRET"
+
+
+def test_the_webbench_agent_never_inherits_a_steering_variable(tmp_path, monkeypatch):
+ for k, v in _STEERING.items():
+ monkeypatch.setenv(k, v)
+ monkeypatch.setattr(wb.subprocess, "Popen", _Proc)
+ r = wb.WebBenchRunner.__new__(wb.WebBenchRunner)
+ r.claude, r.api_key, r.model, r.effort = "/usr/bin/true", "sk-ant-x", "claude-opus-5", "high"
+ r.protected_tools, r.task_timeout = wb.PROTECTED_TOOLS, 5
+ ws = tmp_path / "run" / "workspace"
+ ws.mkdir(parents=True)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+ env = _Proc.seen["env"]
+ assert env["ANTHROPIC_API_KEY"] == "sk-ant-x"
+ for var in sw.IGNORED_ANTHROPIC_VARS:
+ assert var not in env, var
+
+
+def test_the_optimizer_log_names_the_agent(tmp_path, monkeypatch):
+ """One log per results root, `cell` names no tree: a Claude and a Copilot lane over the
+ same arm/repo/seed collide on the key, so the agent has to be on the line."""
+ log = tmp_path / "optimizer_calls.jsonl"
+ monkeypatch.setenv("TE_OPTIMIZER_LOG", str(log))
+ sw.set_optimizer_context(arm="trimech", instance_id="sympy__sympy-1", model="claude-opus-4-8",
+ agent="copilot", cell="trimech/sympy_s0", run_dir="r")
+ sw._log_optimizer_call("claude-opus-4-8", "You configure the toolset for an autonomous "
+ "coding agent ...", "{}", {"input_tokens": 1}, "end_turn")
+ row = json.loads(log.read_text().splitlines()[0])
+ assert row["agent"] == "copilot" and row["cell"] == "trimech/sympy_s0"
+ assert row["kind"] == "tool_judge"
+ sw.set_optimizer_context()
diff --git a/experiments/tests/test_aggregate_index.py b/experiments/tests/test_aggregate_index.py
new file mode 100644
index 0000000..3980a01
--- /dev/null
+++ b/experiments/tests/test_aggregate_index.py
@@ -0,0 +1,127 @@
+"""The canonical index must describe the whole tree, or refuse to be written.
+
+`aggregate.py --arms` limits the REWRITE, not just the reading: both CSVs are written
+whole on every invocation. Naming a subset while writing to `<root>/index` therefore
+replaced an index describing the tree with one describing a subset, at the path
+`compare_arms.py` reads, under a name that says nothing about being partial.
+
+It is not a hypothetical. A stored Codex gpt-5.6-terra tree was found holding one arm
+(`repomap`, n=3) of the eight in the tree, because the Codex lane runs one arm per
+invocation and an `--arms` was passed once. Every cell was on disk; the index the
+comparisons read said the tree contained three task-runs.
+
+These tests are offline: they build a tiny tree of synthetic `row.json` files and run
+`main()` against it.
+"""
+import importlib.util
+import json
+import os
+import sys
+
+import pytest
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_AGGREGATE = os.path.join(_REPO, "scripts", "aggregate.py")
+
+
+def _agg():
+ spec = importlib.util.spec_from_file_location("_aggregate", _AGGREGATE)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def _row(root, arm, cell, stamp, instance, **kw):
+ d = os.path.join(root, arm, cell, f"{stamp}__{instance}__{arm}__standalone")
+ os.makedirs(d, exist_ok=True)
+ row = {"instance": instance, "repo": "sympy/sympy", "seed": 0, "arm": arm,
+ "resolved": True, "num_turns": 10, "tokens_total": 1000,
+ "input_tokens": 10, "output_tokens": 100,
+ "cache_read_input_tokens": 800, "cache_creation_input_tokens": 90,
+ "model": "gpt-5.6-terra", "effort": "high", "agent": "codex",
+ "run_dir": d}
+ row.update(kw)
+ with open(os.path.join(d, "row.json"), "w") as f:
+ json.dump(row, f)
+ return d
+
+
+def tree(tmp_path):
+ """Two arms, one cell each -- the shape the refusal is about."""
+ root = str(tmp_path / "codex_swebench_gpt56terra")
+ _row(root, "baseline", "sympy_s0", "20260916T000000Z", "sympy-11618")
+ _row(root, "baseline", "sympy_s0", "20260916T000100Z", "sympy-12096")
+ _row(root, "repomap", "sympy_s0", "20260916T010000Z", "sympy-11618")
+ return root
+
+
+def _run(argv):
+ mod = _agg()
+ old = sys.argv
+ sys.argv = ["aggregate.py"] + argv
+ try:
+ mod.main()
+ finally:
+ sys.argv = old
+
+
+def _cells(index_dir):
+ import csv
+ with open(os.path.join(index_dir, "summary_by_cell.csv")) as f:
+ return [(r["arm"], r["repo"], r["n_tasks"]) for r in csv.DictReader(f)]
+
+
+def test_whole_tree_is_the_default(tree):
+ _run([tree])
+ assert _cells(os.path.join(tree, "index")) == [
+ ("baseline", "sympy", "2"), ("repomap", "sympy", "1")]
+
+
+def test_subset_rewrite_of_the_canonical_index_is_refused(tree):
+ """The defect: `--arms repomap` used to leave the index holding only repomap."""
+ _run([tree]) # a good index exists
+ before = _cells(os.path.join(tree, "index"))
+
+ with pytest.raises(SystemExit) as e:
+ _run([tree, "--arms", "repomap"])
+ assert e.value.code != 0
+
+ # and, the point of refusing rather than warning: the good index is untouched.
+ assert _cells(os.path.join(tree, "index")) == before
+
+
+def test_subset_is_allowed_somewhere_of_its_own(tree, tmp_path):
+ _run([tree])
+ before = _cells(os.path.join(tree, "index"))
+
+ out = str(tmp_path / "repomap_only")
+ _run([tree, "--arms", "repomap", "--out", out])
+
+ assert _cells(out) == [("repomap", "sympy", "1")]
+ assert _cells(os.path.join(tree, "index")) == before
+
+
+def test_naming_every_arm_is_not_a_subset(tree):
+ """`--arms` that happens to list the whole tree is a no-op, not an error."""
+ _run([tree, "--arms", "baseline", "repomap"])
+ assert _cells(os.path.join(tree, "index")) == [
+ ("baseline", "sympy", "2"), ("repomap", "sympy", "1")]
+
+
+def test_superseded_run_dirs_are_not_double_counted(tree):
+ """The re-run case the pilot cells are in: newest attempt wins, older is dropped.
+
+ Already the behaviour of `collect()`; pinned here because the launcher passes
+ `--fresh`, which starts a new stream in the SAME cell directory, so a re-run cell
+ always holds both attempts on disk.
+ """
+ _row(tree, "repomap", "sympy_s0", "20260916T020000Z", "sympy-11618", num_turns=99)
+ _run([tree])
+ assert _cells(os.path.join(tree, "index")) == [
+ ("baseline", "sympy", "2"), ("repomap", "sympy", "1")]
+
+ import csv
+ with open(os.path.join(tree, "index", "all_rows.csv")) as f:
+ kept = [r for r in csv.DictReader(f) if r["arm"] == "repomap"]
+ assert [r["num_turns"] for r in kept] == ["99"]
diff --git a/experiments/tests/test_answer_key_guard.py b/experiments/tests/test_answer_key_guard.py
new file mode 100644
index 0000000..7c3ccbf
--- /dev/null
+++ b/experiments/tests/test_answer_key_guard.py
@@ -0,0 +1,292 @@
+"""The answer-key guard: one policy, two enforcement points, three agents.
+
+A SWE-bench instance's fix commit is public on GitHub, so an agent that can fetch it is not
+solving the task -- it is reading the answer key, and the resolve verdict it produces is
+worthless. That is not an arm-level concern, so the block applies to every arm including
+`baseline`.
+
+The policy is ONE case-insensitive substring, `github`, and its narrowness is the whole
+argument. Re-derived 2026-09-09 over 3,538 stored run dirs / 558 network-shell calls: it
+matched 352, every one of them reading a repository's source, history, PR diffs or `.patch`
+files, with **0 false positives** (no `pip`/`npm`/`npx`/`yarn` install was blocked) and
+**0 misses** (nothing naming a `.patch`, a `/commit/`, a `/pull/`, a mirror, or a benchmark
+repo escaped it). The corpus in `_ABUSIVE` / `_LEGITIMATE` below is drawn from those calls
+so a future widening or narrowing of the rule has to face them.
+
+It is enforced twice, because a hook is only as good as the CLI's cooperation:
+
+ * the PreToolUse HOOK -- exit 2 + a reason on stderr. Honoured by all three CLIs, each on
+ its own terms: Claude Code's settings.json, the Copilot CLI's, and Codex's
+ `$CODEX_HOME/hooks.json` under `--dangerously-bypass-hook-trust` (verified live on
+ codex-cli 0.153.1, 2026-09-30; before that date the Codex lanes ran without it). It is
+ the layer that tells the MODEL why, which a DNS failure cannot.
+ * the /etc/hosts BLOCK -- the answer-key hosts pointed at 127.0.0.1 inside the task
+ container. Needs no cooperation from any CLI, so it is identical on all three lanes and
+ holds when a CLI release breaks a hook contract.
+
+Offline: no Docker, no network, no credentials. The container-side behaviour these pin was
+verified by hand in a real SWE-bench image -- curl/wget/`git clone`/`urllib` all refused,
+pypi.org and a real `pip download` still working.
+"""
+import json
+import os
+import subprocess
+import sys
+import tempfile
+
+import pytest
+
+from te_harness.guards import net_guard
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+# Real commands, taken verbatim (truncated) from the stored transcripts.
+_ABUSIVE = [
+ "timeout 60 curl -sSL https://github.com/django/django/commit/2cbd3967.patch -o /tmp/fix.patch",
+ "cd /tmp && curl -s -m 30 'https://patch-diff.githubusercontent.com/raw/scikit-learn/scikit-learn/pull/25102.diff'",
+ "timeout 25 curl -s https://raw.githubusercontent.com/django/django/stable/3.0.x/tests/mail/tests.py",
+ "cd /tmp && curl -s -m 30 'https://api.github.com/search/issues?q=repo:scikit-learn/scikit-learn+set_output'",
+ "cd /tmp && timeout 120 git clone --filter=blob:none --no-checkout https://github.com/sympy/sympy.git sympy-git",
+ 'python -c "import urllib.request; u=\'https://raw.githubusercontent.com/sympy/sympy/master/x.py\'"',
+ "timeout 90 curl -s 'https://api.github.com/repos/sympy/sympy/pulls/17340/files?per_page=100'",
+ "curl -sL 'https://github.com/sympy/sympy/commit/9f4fb03c.diff' -o pr12419.diff",
+]
+# Real legitimate network traffic from the same corpus. Blocking any of these would break a
+# lane rather than protect it.
+_LEGITIMATE = [
+ "pip install -e .",
+ "source activate testbed && pip install pytest",
+ "pip download --no-deps -d /tmp/dl six",
+ "npm install",
+ "npx playwright test",
+ "yarn --frozen-lockfile",
+ "curl -s http://localhost:3000/",
+ "curl -sS https://pypi.org/simple/",
+ "curl -sS https://files.pythonhosted.org/packages/x.whl",
+ "python -m pytest -q",
+]
+
+
[email protected]("cmd", _ABUSIVE)
+def test_every_observed_abuse_is_blocked(cmd):
+ assert net_guard.decide(cmd)[0] is True, cmd
+
+
[email protected]("cmd", _LEGITIMATE)
+def test_no_observed_legitimate_call_is_blocked(cmd):
+ assert net_guard.decide(cmd)[0] is False, cmd
+
+
+def test_a_block_carries_a_reason_the_model_can_act_on():
+ """Exit 2 with an empty stderr is a refusal the CLI reports as a hook failure."""
+ blocked, reason = net_guard.decide(_ABUSIVE[0])
+ assert blocked and reason.strip()
+ assert "answer key" in reason or "grading" in reason
+
+
+# ---- the two enforcement points must express the SAME policy ------------------------
+def test_every_blocked_host_is_also_caught_by_the_shell_rule():
+ """One policy, or the layers disagree about what is allowed.
+
+ A host in the DNS block that the shell rule would let through (or vice versa) means an
+ agent gets a different answer depending on which layer it happens to hit first, and the
+ telemetry from the two stops being comparable.
+ """
+ for host in net_guard.ANSWER_KEY_HOSTS:
+ assert net_guard.decide(f"curl https://{host}/x")[0] is True, host
+
+
+def test_hosts_block_is_wellformed_and_covers_the_measured_abuse_set():
+ text = net_guard.hosts_block()
+ assert text.startswith(net_guard.HOSTS_MARKER)
+ assert text.endswith("\n")
+ entries = {}
+ for line in text.splitlines():
+ if line.startswith("#") or not line.strip():
+ continue
+ parts = line.split()
+ assert len(parts) == 2, line # one host per line: simpler to grep and undo
+ entries[parts[1]] = parts[0]
+ # 127.0.0.1, not 0.0.0.0: nothing listens on 443 in a task container, so the connection
+ # is REFUSED at once instead of hanging until a timeout.
+ assert set(entries.values()) == {"127.0.0.1"}
+ # The hosts that actually appeared in the 352 measured accesses.
+ for host in ("github.com", "raw.githubusercontent.com", "api.github.com",
+ "patch-diff.githubusercontent.com", "codeload.github.com"):
+ assert host in entries, host
+
+
+def test_hosts_block_leaves_the_package_registries_alone():
+ """The block is a NAME LIST, not a network cut -- that distinction is the design."""
+ blocked = set(net_guard.hosts_block().split())
+ for host in ("pypi.org", "files.pythonhosted.org", "registry.npmjs.org",
+ "localhost", "127.0.0.1:3000"):
+ assert host not in blocked, host
+
+
+# ---- the hook, end to end through the real script ------------------------------------
+def _run_hook(payload):
+ return subprocess.run(
+ [sys.executable, os.path.join(_REPO, "te_harness/guards/net_guard.py")],
+ input=json.dumps(payload), capture_output=True, text=True)
+
+
+def test_hook_exits_2_with_a_reason_on_stderr():
+ p = _run_hook({"tool_name": "Bash",
+ "tool_input": {"command": _ABUSIVE[0]}})
+ assert p.returncode == 2
+ assert p.stderr.strip()
+
+
+def test_hook_allows_a_legitimate_command():
+ p = _run_hook({"tool_name": "Bash", "tool_input": {"command": "pip install -e ."}})
+ assert p.returncode == 0
+
+
+def test_hook_fails_open_on_unreadable_input():
+ """An unreadable payload must never wedge the agent -- it is telemetry, not a lock."""
+ p = subprocess.run(
+ [sys.executable, os.path.join(_REPO, "te_harness/guards/net_guard.py")],
+ input="not json", capture_output=True, text=True)
+ assert p.returncode == 0
+
+
+def test_the_lift_switch_lifts_the_hook(monkeypatch):
+ env = dict(os.environ, TE_ALLOW_NETWORK_TOOLS="1")
+ p = subprocess.run(
+ [sys.executable, os.path.join(_REPO, "te_harness/guards/net_guard.py")],
+ input=json.dumps({"tool_name": "Bash",
+ "tool_input": {"command": _ABUSIVE[0]}}),
+ capture_output=True, text=True, env=env)
+ assert p.returncode == 0, "TE_ALLOW_NETWORK_TOOLS must lift every layer, or none"
+
+
+# ---- the agent-independent counter ---------------------------------------------------
+def _transcript(commands):
+ d = tempfile.mkdtemp()
+ with open(os.path.join(d, "transcript.jsonl"), "w") as fh:
+ for i, cmd in enumerate(commands):
+ fh.write(json.dumps({"message": {"content": [
+ {"type": "tool_use", "id": f"t{i}", "name": "bash",
+ "input": {"command": cmd}}]}}) + "\n")
+ fh.write(json.dumps({"message": {"content": [
+ {"type": "tool_result", "tool_use_id": f"t{i}", "content": "ok"}]}}) + "\n")
+ return d
+
+
+def test_answer_key_attempts_counts_from_the_trace():
+ """ATTEMPTS, not refusals, and read through the one trace seam.
+
+ This is the only answer-key number available on a lane whose CLI ignores the hook, so a
+ count that climbs while `net_guard_blocks` stays flat is the signal to watch.
+ """
+ from te_harness import swebench
+ d = _transcript(["python -m pytest -q", _ABUSIVE[0], "pip install six", _ABUSIVE[2]])
+ n = swebench.ClaudeSwebenchRunner._answer_key_attempts(None, d)
+ assert n == 2
+
+
+def test_answer_key_attempts_is_none_when_there_is_no_transcript():
+ """None, not 0: "no trace" and "a clean trace" must not read the same."""
+ from te_harness import swebench
+ assert swebench.ClaudeSwebenchRunner._answer_key_attempts(
+ None, tempfile.mkdtemp()) is None
+
+
+def test_both_domains_count_answer_key_attempts_through_one_seam():
+ """Web-Bench rows carry it too, and off the same function.
+
+ Every Codex Web-Bench row before 2026-09-30 ran without the PreToolUse hook, guarded by
+ the namespace hosts block alone, so `answer_key_attempts` is the ONLY per-row leak
+ telemetry those rows have. It was absent from every Web-Bench row -- all three agents --
+ while the Codex docs told the reader to watch it.
+ """
+ from te_harness import swebench, webbench
+ assert webbench.answer_key_attempts is swebench.answer_key_attempts
+ d = _transcript(["ls -la", _ABUSIVE[0]])
+ assert webbench.answer_key_attempts(d) == 1
+ assert swebench.ClaudeSwebenchRunner._answer_key_attempts(None, d) == 1
+
+
+# ---- per-agent enforcement -----------------------------------------------------------
+def test_the_merge_program_appends_once_and_keeps_what_was_there(tmp_path):
+ """The program the Claude and Codex lanes run in the container, run here on a real file:
+ a scaffold's hook survives, the guard is appended once, and re-arming is a no-op."""
+ from te_harness import swebench
+ path = tmp_path / "hooks.json"
+ path.write_text(json.dumps({"hooks": {"PreToolUse": [{"matcher": "Bash", "hooks": [
+ {"type": "command", "command": "rtk-rewrite"}]}]}, "other": 1}))
+ entry = {"matcher": "Bash", "hooks": [{"type": "command",
+ "command": "/usr/bin/python3 /x/net_guard.py"}]}
+ prog = tmp_path / "merge.py"
+ prog.write_text(swebench.ClaudeSwebenchRunner._net_guard_merge_program(entry))
+ for _ in range(2):
+ out = subprocess.run([sys.executable, str(prog), str(path)], capture_output=True,
+ text=True, check=True).stdout
+ assert out.strip() == "armed"
+ doc = json.loads(path.read_text())
+ assert doc["other"] == 1
+ assert [h["hooks"][0]["command"] for h in doc["hooks"]["PreToolUse"]] == [
+ "rtk-rewrite", "/usr/bin/python3 /x/net_guard.py"]
+
+
+def test_codex_arms_its_hook_in_codex_home_and_with_it_the_trust_bypass():
+ """Codex honours the hook in `$CODEX_HOME/hooks.json` -- verified live 2026-09-30 -- but
+ only under --dangerously-bypass-hook-trust, so registering it is what turns the flag on
+ for every segment of the task, fresh and resumed alike."""
+ from te_harness import codex, swebench
+ r = object.__new__(swebench.CodexSwebenchRunner)
+ calls = []
+
+ def fake_exec(name, script, timeout=None):
+ calls.append(script)
+ return subprocess.CompletedProcess(script, 0, stdout="armed\n", stderr="")
+ r._exec = fake_exec
+ assert r._guard_python == "" # unarmed by default
+ assert r._register_net_guard("ctr", "/usr/bin/python3") is True
+ (script,) = calls
+ assert script.rstrip().endswith(f"{codex.C_HOME}/hooks.json")
+ assert "PreToolUse" in script and '\\"matcher\\": \\"Bash\\"' in script
+ assert r._guard_python == "/usr/bin/python3"
+ assert "--dangerously-bypass-hook-trust" in codex.build_argv(hook=bool(r._guard_python))
+ assert "--dangerously-bypass-hook-trust" in codex.build_resume_argv(
+ "th", hook=bool(r._guard_python))
+ assert "--dangerously-bypass-hook-trust" not in codex.build_argv(hook=False)
+
+
+def test_the_codex_config_declares_no_hooks_table():
+ """The hook lives in hooks.json, never in config.toml: one config layer carrying both
+ representations makes Codex warn and pick one, so the contract keeps exactly one."""
+ from te_harness import codex
+ routing = codex.resolve_codex_routing({"OPENAI_API_KEY": "sk-test"})
+ txt = codex.config_toml(routing=routing, model="gpt-5.6-sol", effort="high")
+ assert "[hooks]" not in txt
+ assert "pre_tool_use" not in txt
+
+
+# ---- per-task cleanup must never kill the stream -------------------------------------
+def test_stop_never_raises_when_docker_hangs(monkeypatch):
+ """`_stop` runs in run_task's `finally`, so anything it raises ends the whole STREAM.
+
+ Measured 2026-09-10: on a shared box at load ~96 with rootless Docker, `docker rm -f`
+ exceeded the 120 s default and the `TimeoutExpired` escaped the finally, ending a
+ 60-task sympy lane at task 3. The task was already graded and its row already built --
+ a container we failed to remove costs disk (and `_reap_exited` collects it later), not
+ results. Same rule `_start` already follows.
+ """
+ import subprocess as sp
+ from te_harness import swebench
+
+ def boom(cmd, timeout=120, check=False):
+ raise sp.TimeoutExpired(cmd, timeout)
+
+ monkeypatch.setattr(swebench, "_run", boom)
+
+ class Fake:
+ keep = False
+ _stop = swebench.ClaudeSwebenchRunner._stop
+ swebench._LIVE_CONTAINERS.add("te-fake-container")
+ Fake()._stop("te-fake-container") # must not raise
+ assert "te-fake-container" not in swebench._LIVE_CONTAINERS, (
+ "_stop must still drop the container from the live set, or the signal handler "
+ "will try to clean up a name that no longer exists")
diff --git a/experiments/tests/test_baselines.py b/experiments/tests/test_baselines.py
new file mode 100644
index 0000000..42dbc6f
--- /dev/null
+++ b/experiments/tests/test_baselines.py
@@ -0,0 +1,610 @@
+"""External tool baselines: the contracts that silently corrupt a result if broken.
+
+Offline — no Docker, no network, no agent. Every container interaction goes through the
+callables on `ScaffoldContext`, so a fake context is enough to assert what a scaffold
+*does*, which is the only way to test this without a task container.
+
+`tokeneconomizer.baselines` lives in the CORE package, so the first group of tests guards
+the two invariants that buys: nothing in it imports Docker or `te_harness` (a baseline can
+only reach a container through the context it is handed), and nothing in the core package
+imports it on its own (so `import tokeneconomizer` stays stdlib-only and the economizer
+cannot see a baseline by accident). Those are structural claims; without a test they decay
+into a convention.
+
+Three properties matter more than the rest, because each fails QUIETLY:
+
+ * **patch_excludes.** A scaffold writes project instructions into `/testbed`, so `git
+ diff` picks them up and the official grader would apply the scaffold's own prompt text
+ as part of the agent's fix. Every scaffold that writes into the repo must declare it.
+ * **the agent profile is used, not assumed.** Writing `CLAUDE.md` on a Copilot run
+ produces an arm that is native with a stray file in it; the instructions path must come
+ from the profile.
+ * **failure is recorded, not raised.** A scaffold whose install half-failed has to leave
+ a row saying so — a lane of flagged rows is diagnosable, a lane of missing rows is not.
+"""
+import json
+import os
+import subprocess
+
+import builtins
+import sys
+
+import pytest
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+from tokeneconomizer import baselines as scaffolds
+from tokeneconomizer.baselines import repomap as repomap_mod
+from tokeneconomizer.baselines import rtk as rtk_mod
+
+
+CLAUDE = scaffolds.AgentProfile(
+ kind="claude", home="/tmp/agent-home", instructions_file="CLAUDE.md",
+ nav_tools=("Read", "Grep", "Glob"), mcp_args=lambda c: ["--mcp-config", c],
+ patch_excludes=("CLAUDE.md",))
+COPILOT = scaffolds.AgentProfile(
+ kind="copilot", home="/home/agent",
+ instructions_file=".github/copilot-instructions.md",
+ nav_tools=("view", "grep", "glob"), mcp_args=lambda c: [],
+ patch_excludes=(".github/copilot-instructions.md",))
+
+
+def _ok(stdout="ok", rc=0):
+ return subprocess.CompletedProcess([], rc, stdout, "")
+
+
+class FakeCtx(scaffolds.ScaffoldContext):
+ """Records every container interaction instead of performing one."""
+
+ def __init__(self, profile, tmp_path, exec_result=None, task=None):
+ self.execs, self.copies, self.host_runs, self.logs = [], [], [], []
+ self._result = exec_result or (lambda script: _ok())
+ super().__init__(
+ profile=profile, container="c1",
+ task=task or {"instance_id": "django__django-1", "problem_statement": "x"},
+ run_dir=str(tmp_path),
+ exec_=self._exec, copy_in=self._copy, host_run=self._host,
+ log=self.logs.append)
+
+ def _exec(self, script, timeout=300, env=None):
+ self.execs.append((script, env or {}))
+ return self._result(script)
+
+ def _copy(self, host_path, container_path):
+ with open(host_path, encoding="utf-8") as fh:
+ self.copies.append((container_path, fh.read()))
+ return _ok()
+
+ def _host(self, argv, timeout=None):
+ self.host_runs.append(argv)
+ return _ok()
+
+ def written(self, path):
+ for dest, body in self.copies:
+ if dest == path:
+ return body
+ return None
+
+ def written_prefix(self, prefix):
+ """The one file copied to a path starting with `prefix`.
+
+ `merge_json_file` names its helper uniquely per run (a fixed /tmp name would race
+ between two host-side lanes sharing /tmp), so a test cannot pin the exact path.
+ """
+ hits = [(d, b) for d, b in self.copies if d.startswith(prefix)]
+ assert len(hits) == 1, f"expected one copy under {prefix!r}, got {[d for d, _ in hits]}"
+ return hits[0][1]
+
+
+# ---- the package boundary ---------------------------------------------------------------
+_PKG_DIR = os.path.dirname(scaffolds.__file__)
+# Non-stdlib at import time is a bug in every module here EXCEPT the repo-map generator,
+# which is a subprocess entry point (never imported) and deliberately does need aider.
+_SUBPROCESS_ENTRY_POINTS = {"_repomap_gen.py"}
+
+
+def _module_sources():
+ for name in sorted(os.listdir(_PKG_DIR)):
+ if name.endswith(".py") and name not in _SUBPROCESS_ENTRY_POINTS:
+ with open(os.path.join(_PKG_DIR, name), encoding="utf-8") as fh:
+ yield name, fh.read()
+
+
+def test_no_baseline_module_imports_docker_or_the_harness():
+ """A baseline reaches a container only through the context it is handed. If one
+ imported a Docker client or te_harness, it could not live in the core package and no
+ other harness could run it."""
+ import ast
+ for name, src in _module_sources():
+ for node in ast.walk(ast.parse(src)):
+ mods = []
+ if isinstance(node, ast.Import):
+ mods = [a.name for a in node.names]
+ elif isinstance(node, ast.ImportFrom) and node.level == 0:
+ mods = [node.module or ""]
+ for mod in mods:
+ root = mod.split(".")[0]
+ assert root not in ("docker", "te_harness"), f"{name} imports {mod}"
+
+
+def test_baseline_modules_are_stdlib_only_at_import_time():
+ """TOP-LEVEL imports only -- those are the ones that run on `import`.
+
+ A deferred import inside a function is fine and is how the one host dependency is
+ handled: `repomap.preflight()` imports aider to check for it, so a machine that will
+ never run that arm never needs it installed.
+ """
+ import ast
+ import sys
+ for name, src in _module_sources():
+ for node in ast.parse(src).body: # module level, not ast.walk
+ mods = []
+ if isinstance(node, ast.Import):
+ mods = [a.name for a in node.names]
+ elif isinstance(node, ast.ImportFrom) and node.level == 0:
+ mods = [node.module or ""]
+ for mod in mods:
+ root = mod.split(".")[0]
+ assert root in sys.stdlib_module_names or root == "tokeneconomizer", \
+ f"{name} imports non-stdlib {mod} at module level"
+
+
+def test_the_one_host_dependency_is_deferred_not_top_level():
+ """aider must be imported inside preflight, never at module scope -- otherwise
+ `import tokeneconomizer.baselines` would require it on every machine."""
+ import ast
+ src = open(os.path.join(_PKG_DIR, "repomap.py"), encoding="utf-8").read()
+ top = {a.name for n in ast.parse(src).body if isinstance(n, ast.Import)
+ for a in n.names}
+ assert not any(m.startswith("aider") for m in top)
+ assert "import aider" in src, "the preflight check itself must still be there"
+
+
+def test_the_core_package_does_not_import_baselines_by_itself():
+ """`import tokeneconomizer` must not pull this package in -- that is what keeps the
+ core import free of the baselines and the economizer unable to see one."""
+ import subprocess
+ import sys
+ probe = ("import sys, tokeneconomizer; "
+ "print(any(m.startswith('tokeneconomizer.baselines') for m in sys.modules))")
+ out = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True,
+ cwd=os.path.dirname(_REPO)) # the repository root holds the core
+ assert out.stdout.strip() == "False", out.stdout + out.stderr
+
+
+def test_no_mechanism_imports_a_baseline():
+ """A mechanism that reached for a baseline would make the comparison circular."""
+ import ast
+ mech_dir = os.path.join(os.path.dirname(_REPO), "tokeneconomizer", "mechanisms")
+ for name in sorted(os.listdir(mech_dir)):
+ if not name.endswith(".py"):
+ continue
+ with open(os.path.join(mech_dir, name), encoding="utf-8") as fh:
+ src = fh.read()
+ for node in ast.walk(ast.parse(src)):
+ if isinstance(node, ast.ImportFrom):
+ assert "baselines" not in (node.module or ""), f"{name}"
+ elif isinstance(node, ast.Import):
+ for a in node.names:
+ assert "baselines" not in a.name, f"{name}"
+
+
+# ---- registry ---------------------------------------------------------------------------
+def test_registry_resolves_every_named_scaffold():
+ for name in scaffolds.SCAFFOLDS:
+ assert scaffolds.get(name).name == name
+
+
+def test_no_scaffold_is_not_an_error():
+ assert scaffolds.get(None) is None and scaffolds.get("") is None
+ assert scaffolds.for_arm("trimech") is None and scaffolds.for_arm("") is None
+
+
+def test_unknown_scaffold_names_the_known_ones():
+ with pytest.raises(scaffolds.ScaffoldError) as e:
+ scaffolds.get("nope")
+ assert "rtk" in str(e.value)
+
+
+def test_every_arm_mapping_points_at_a_real_scaffold():
+ for arm, name in scaffolds.ARM_SCAFFOLDS.items():
+ assert name in scaffolds.SCAFFOLDS, arm
+
+
+# ---- the graded-diff contract -----------------------------------------------------------
+def test_scaffolds_writing_into_the_repo_declare_patch_excludes():
+ """rtk writes RTK.md and serena indexes into .serena/; both are inside /testbed and
+ would otherwise be graded as part of the agent's patch."""
+ assert "RTK.md" in scaffolds.get("rtk").patch_excludes
+ assert any(e.startswith(".serena") for e in scaffolds.get("serena").patch_excludes)
+
+
+def test_repomap_relies_on_the_profiles_exclude_not_its_own():
+ """The repo map writes ONLY the agent's own instructions file, which every profile
+ already excludes -- so an empty tuple here is correct, and this test is what says the
+ emptiness was a decision."""
+ assert scaffolds.get("repomap").patch_excludes == ()
+ assert "CLAUDE.md" in CLAUDE.patch_excludes
+
+
+# ---- the agent profile is honoured ------------------------------------------------------
+def test_instructions_go_to_the_profiles_path(tmp_path):
+ for profile, expected in ((CLAUDE, "/testbed/CLAUDE.md"),
+ (COPILOT, "/testbed/.github/copilot-instructions.md")):
+ ctx = FakeCtx(profile, tmp_path)
+ assert ctx.write_instructions("hello") is True
+ assert ctx.written(expected) == "hello"
+
+
+def test_a_nested_instructions_path_is_created_first(tmp_path):
+ ctx = FakeCtx(COPILOT, tmp_path)
+ ctx.write_instructions("x")
+ assert any("mkdir -p" in script and ".github" in script for script, _ in ctx.execs)
+
+
+def test_instructions_are_copied_not_shell_quoted(tmp_path):
+ """A repo map contains backticks, $(...) and stray EOF markers. Copying bytes means no
+ quoting scheme has to survive them."""
+ nasty = "`cmd` $(sub) 'q' \"d\"\nEOF\nTE_EOF\n"
+ ctx = FakeCtx(CLAUDE, tmp_path)
+ ctx.write_instructions(nasty)
+ assert ctx.written("/testbed/CLAUDE.md") == nasty
+
+
+def test_mcp_config_lands_in_home_never_in_the_repo(tmp_path):
+ ctx = FakeCtx(CLAUDE, tmp_path)
+ cfg = ctx.write_mcp_config({"s": {"command": "/bin/true"}})
+ assert cfg.startswith("/tmp/agent-home/") and "/testbed" not in cfg
+ assert json.loads(ctx.written(cfg))["mcpServers"]["s"]["command"] == "/bin/true"
+
+
+def test_write_instructions_leaves_no_temp_file_behind(tmp_path):
+ ctx = FakeCtx(CLAUDE, tmp_path)
+ ctx.write_instructions("x")
+ ctx.write_mcp_config({"s": {}})
+ assert [p for p in os.listdir(tmp_path) if p.startswith("_scaffold")] == []
+
+
+# ---- rtk --------------------------------------------------------------------------------
+def test_rtk_registers_the_hook_itself(tmp_path, monkeypatch):
+ """`rtk init -g` declines to patch settings non-interactively and exits 0. If the
+ scaffold trusted it, the arm would run as native with healthy-looking provenance."""
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ binary = tmp_path / "cache" / "bin" / "rtk"
+ binary.parent.mkdir(parents=True)
+ binary.write_text("#!/bin/sh\n")
+ binary.chmod(0o755)
+
+ ctx = FakeCtx(CLAUDE, tmp_path, exec_result=lambda s: _ok("merged" if "merge" in s
+ else "rtk 0.45.0"))
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["arm_setup"] == "rtk"
+ assert notes["rtk_installed"] and notes["rtk_hook_registered"]
+ assert notes["rtk_staged_host"] is True
+ merged = ctx.written_prefix("/tmp/_scaffold_merge")
+ assert "PreToolUse" in merged and "rtk hook claude" in merged
+ assert "/tmp/agent-home/.claude/settings.json" in merged
+ # The helper's name must be unique per run: on a harness that runs the agent on the
+ # HOST, /tmp is shared between lanes and a fixed name would let one run's settings
+ # overwrite another's mid-install.
+ dest = [d for d, _ in ctx.copies if d.startswith("/tmp/_scaffold_merge")][0]
+ assert dest != "/tmp/_scaffold_merge.py", "merge helper must not use a fixed /tmp name"
+
+
+def test_rtk_warns_loudly_when_the_hook_did_not_register(tmp_path, monkeypatch):
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ binary = tmp_path / "cache" / "bin" / "rtk"
+ binary.parent.mkdir(parents=True)
+ binary.write_text("#!/bin/sh\n")
+ binary.chmod(0o755)
+ # exec always succeeds but never prints the merge marker
+ ctx = FakeCtx(CLAUDE, tmp_path, exec_result=lambda s: _ok("rtk 0.45.0"))
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["rtk_installed"] and not notes["rtk_hook_registered"]
+ assert any("would run as native" in m for m in ctx.logs)
+
+
+def _copilot_exec(script):
+ """Stand in for the container: echo the install marker, and rewrite on the hook probe."""
+ if "RTK_HOOK_AT=" in script:
+ return _ok("RTK_HOOK_AT=/home/agent/.copilot/hooks/rtk-rewrite.json")
+ if "hook copilot" in script: # the effect probe
+ return _ok('{"hookSpecificOutput":{"updatedInput":{"command":"rtk ls -la"}}}')
+ return _ok("rtk 0.45.0")
+
+
+def test_rtk_on_copilot_registers_through_the_hooks_dir_not_a_plugin(tmp_path, monkeypatch):
+ """Copilot CLI has TWO PreToolUse channels and they deliver DIFFERENT payload shapes
+ (measured on 1.0.83, 2026-09-13, by arming both at once):
+
+ $COPILOT_HOME/hooks/*.json -> {"tool_name": "Bash", "tool_input": {...}}
+ $COPILOT_HOME/settings.json -> {"toolName": "bash", "toolArgs": {...}}
+
+ rtk parses only the first. The plugin this scaffold used to install delivered the
+ second, so rtk exited 0 with empty stdout on every call and rewrote nothing -- the arm
+ was native while every provenance field read healthy. Pin the channel, the document
+ shape rtk itself publishes, and the absolute command path."""
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ ctx = FakeCtx(COPILOT, tmp_path, exec_result=_copilot_exec)
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["rtk_hook_registered"] is True
+ assert notes["rtk_hook_channel"] == "copilot-hooks-dir"
+ assert notes["rtk_mode"] == "hook"
+ assert notes["rtk_hook_path"].endswith("/hooks/rtk-rewrite.json")
+ # never through a plugin, and never through the guard's settings.json channel
+ assert not any("plugin install" in s for s, _ in ctx.execs)
+ assert not any("settings.json" in s for s, _ in ctx.execs)
+ write = [s for s, _ in ctx.execs if "RTK_HOOK_AT=" in s][0]
+ # COPILOT_HOME is resolved in the SHELL: SWE-bench leaves it unset (the CLI defaults it
+ # to $HOME/.copilot in the container), Web-Bench redirects it to a per-task dir.
+ assert '${COPILOT_HOME:-$HOME/.copilot}' in write
+ body = json.loads(write.split("TE_EOF")[1].lstrip("'").strip())
+ assert body["version"] == 1 and list(body["hooks"]) == ["PreToolUse"]
+ entry = body["hooks"]["PreToolUse"][0]
+ assert entry["command"] == "/usr/local/bin/rtk hook copilot", "absolute path, not bare rtk"
+ assert entry["type"] == "command" and "matcher" not in entry
+
+
+def test_rtk_hook_effect_is_probed_not_assumed(tmp_path, monkeypatch):
+ """Registered and executable is still not REWRITING. The Copilot arm was silently
+ native for a week with both of those true, so the row carries the effect itself."""
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ ctx = FakeCtx(COPILOT, tmp_path, exec_result=_copilot_exec)
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["rtk_hook_rewrites"] is True
+ probe = [s for s, _ in ctx.execs if "hook copilot" in s and "tool_input" in s]
+ assert probe, "the hook must be fed a payload in the shape its channel delivers"
+ assert '"tool_name"' in probe[0] and '"tool_input"' in probe[0]
+
+ # ... and a hook that parses nothing (empty stdout, rc 0 -- exactly the old failure)
+ # is caught and said out loud rather than reported as a healthy install.
+ def mute(script):
+ return _ok("") if "hook copilot" in script and "tool_input" in script \
+ else _copilot_exec(script)
+ ctx2 = FakeCtx(COPILOT, tmp_path, exec_result=mute)
+ notes2 = scaffolds.get("rtk").setup(ctx2)
+ assert notes2["rtk_hook_registered"] is True and notes2["rtk_hook_rewrites"] is False
+ assert any("does NOT rewrite" in m for m in ctx2.logs)
+
+
+def test_rtk_version_is_pinned():
+ """An arm tracking `latest` is not a fixed baseline; the published rows say 0.45.0."""
+ assert rtk_mod.RTK_VERSION == "0.45.0"
+ assert f"v{rtk_mod.RTK_VERSION}" in rtk_mod.RTK_URL
+
+
+def test_rtk_cache_dir_requires_configuration(monkeypatch):
+ monkeypatch.delenv("TE_SCAFFOLD_CACHE", raising=False)
+ monkeypatch.delenv("DATA", raising=False)
+ with pytest.raises(scaffolds.ScaffoldError):
+ rtk_mod.cache_dir()
+ monkeypatch.setenv("DATA", "/somewhere")
+ assert rtk_mod.cache_dir() == "/somewhere/scaffolds"
+
+
+# ---- serena -----------------------------------------------------------------------------
+def test_serena_nudges_rather_than_excluding_tools(tmp_path):
+ """`Read` is in PROTECTED_TOOLS, so an exclusion would be filtered and the arm would
+ silently be native. The baseline is an instruction the agent may disobey."""
+ ctx = FakeCtx(CLAUDE, tmp_path, exec_result=lambda s: _ok("serena 1.0"))
+ notes = scaffolds.get("serena").setup(ctx)
+ assert notes["serena_installed"] and notes["serena_nudged"]
+ nudge = ctx.written("/testbed/CLAUDE.md")
+ for tool in ("Read", "Grep", "Glob"):
+ assert tool in nudge
+ assert "find_symbol" in nudge
+ assert not hasattr(scaffolds.get("serena"), "exclude_tools")
+
+
+def test_serena_nudge_names_the_agents_own_tools(tmp_path):
+ ctx = FakeCtx(COPILOT, tmp_path, exec_result=lambda s: _ok("serena 1.0"))
+ scaffolds.get("serena").setup(ctx)
+ nudge = ctx.written("/testbed/.github/copilot-instructions.md")
+ assert "view/grep/glob" in nudge and "Read" not in nudge
+
+
+def test_serena_records_a_failed_install_instead_of_raising(tmp_path):
+ ctx = FakeCtx(CLAUDE, tmp_path, exec_result=lambda s: _ok("", rc=1))
+ notes = scaffolds.get("serena").setup(ctx)
+ assert notes["serena_installed"] is False and notes["serena_nudged"] is False
+ assert "mcp_config" not in notes
+ assert any("install FAILED" in m for m in ctx.logs)
+
+
+# ---- repomap ----------------------------------------------------------------------------
+def test_repomap_extracts_idents_from_the_issue_only():
+ idents = repomap_mod.extract_idents(
+ "`DataArray.sel` breaks; see get_axis_num and CFTimeIndex")
+ assert "sel" in idents and "get_axis_num" in idents and "CFTimeIndex" in idents
+ assert len(idents) <= 60
+
+
+def test_repomap_ident_extraction_survives_empty_text():
+ assert repomap_mod.extract_idents("") == []
+ assert repomap_mod.extract_idents(None) == []
+
+
+def test_repomap_budget_is_configurable(monkeypatch):
+ assert repomap_mod.map_tokens() == repomap_mod.DEFAULT_MAP_TOKENS
+ monkeypatch.setenv("TE_REPOMAP_TOKENS", "512")
+ assert repomap_mod.map_tokens() == 512
+
+
+def test_repomap_flags_a_task_that_got_no_map(tmp_path, monkeypatch):
+ """No map means the task ran native. Averaged in as a repomap row it would dilute the
+ very effect being measured, so it must be flagged."""
+ scaffold = scaffolds.get("repomap")
+ monkeypatch.setattr(scaffold, "fetch_sources", lambda ctx: str(tmp_path / "src"))
+ monkeypatch.setattr(scaffold, "generate", lambda src, idents: ("", "boom"))
+ ctx = FakeCtx(CLAUDE, tmp_path)
+ notes = scaffold.setup(ctx)
+ assert notes["map_ok"] is False and notes["map_chars"] == 0
+ assert notes["map_err"] == "boom"
+ assert ctx.written("/testbed/CLAUDE.md") is None
+
+
+def test_repomap_writes_the_map_and_records_its_size(tmp_path, monkeypatch):
+ scaffold = scaffolds.get("repomap")
+ monkeypatch.setattr(scaffold, "fetch_sources", lambda ctx: str(tmp_path / "src"))
+ monkeypatch.setattr(scaffold, "generate", lambda src, idents: ("MAP BODY", ""))
+ ctx = FakeCtx(CLAUDE, tmp_path)
+ notes = scaffold.setup(ctx)
+ assert notes["map_ok"] is True and notes["map_chars"] == len("MAP BODY")
+ body = ctx.written("/testbed/CLAUDE.md")
+ assert body.startswith(repomap_mod.HEADER) and "MAP BODY" in body
+
+
+def test_repomap_preflight_demands_aider_on_the_host():
+ """The generator is a subprocess of this interpreter, so a missing aider must stop the
+ lane at construction rather than yield 277 rows of `map_ok: false`."""
+ scaffold = scaffolds.get("repomap")
+ try:
+ import aider.repomap # noqa: F401
+ except ImportError:
+ with pytest.raises(scaffolds.ScaffoldError) as e:
+ scaffold.preflight(CLAUDE)
+ assert "aider" in str(e.value)
+ else:
+ scaffold.preflight(CLAUDE)
+
+
+def test_repomap_preflight_names_the_interpreter_before_telling_you_to_install(monkeypatch):
+ """"Install aider" is the WRONG advice for the likelier failure. The launchers run
+ `${PYTHON:-python3}`, so a lane started from an unactivated shell picks up the system
+ Python even though the guide's `pip install -e '.[baselines]'` put aider in the venv.
+ Observed 2026-09-08: a detached lane died here on a host whose system Python carried
+ every other harness dependency, so only this one arm failed. The message must let that
+ reader diagnose it instead of reinstalling a package they already have.
+ """
+ real_import = builtins.__import__
+
+ def no_aider(name, *a, **k):
+ if name.startswith("aider"):
+ raise ImportError("No module named 'aider'")
+ return real_import(name, *a, **k)
+
+ monkeypatch.setattr(builtins, "__import__", no_aider)
+ with pytest.raises(scaffolds.ScaffoldError) as e:
+ scaffolds.get("repomap").preflight(CLAUDE)
+ msg = str(e.value)
+ assert sys.executable in msg, "the message must name the interpreter that actually ran"
+ assert "PYTHON=" in msg, "and the launcher knob that fixes it"
+ assert msg.index("WRONG INTERPRETER") < msg.index("NOT INSTALLED"), \
+ "likeliest cause first"
+
+
+def test_repomap_generator_keeps_aiders_console_off_stdout(tmp_path):
+ """The scaffold takes the generator's stdout VERBATIM as the map body, and aider writes
+ its progress line -- "Initial repo scan can be slow in larger repos, but only happens
+ once." -- to stdout whenever more than 100 files are uncached, which is every SWE-bench
+ repository on every task. Left as it was, that sentence opened every repo map an agent
+ read (seen live, 2026-09-13). The generator routes aider's console to stderr while the map
+ is built, so stdout is the map and nothing else."""
+ pytest.importorskip("aider.repomap")
+ pkg = tmp_path / "pkg"
+ pkg.mkdir()
+ for i in range(120): # above aider's 100-uncached-files threshold
+ (pkg / f"mod_{i}.py").write_text(
+ f"class Widget{i}:\n def render_{i}(self):\n return {i}\n")
+ res = subprocess.run([sys.executable, repomap_mod.GEN, str(tmp_path), "512", ""],
+ capture_output=True, text=True, timeout=600)
+ assert res.returncode == 0, res.stderr[-500:]
+ assert "pkg/mod_" in res.stdout, "a map was expected"
+ assert "Initial repo scan" not in res.stdout, res.stdout[:200]
+ assert not res.stdout.lstrip().startswith(("Warning", "Initial")), res.stdout[:120]
+
+
+# ---- rtk on Codex: instruction files, inlined, no hook ---------------------------------
+CODEX = scaffolds.AgentProfile(
+ kind="codex", home="/tmp/agent-home", instructions_file="AGENTS.md", nav_tools=(),
+ mcp_args=lambda cfg: [], patch_excludes=("AGENTS.md", "RTK.md"),
+ env={"CODEX_HOME": "/opt/codex-home"})
+
+
+def _codex_rtk_ctx(tmp_path, monkeypatch, rule_count="1"):
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ binary = tmp_path / "cache" / "bin" / "rtk"
+ binary.parent.mkdir(parents=True)
+ binary.write_text("#!/bin/sh\n")
+ binary.chmod(0o755)
+
+ def result(script):
+ if "grep -c" in script:
+ return _ok(rule_count) # the inline step reports how many rule lines landed
+ return _ok("rtk 0.45.0")
+ return FakeCtx(CODEX, tmp_path, exec_result=result)
+
+
+def test_rtk_on_codex_inlines_the_rule_into_agents_md_instead_of_registering_a_hook(tmp_path, monkeypatch):
+ """rtk's Codex mode is `rtk init -g --codex`: RTK.md plus an AGENTS.md holding a one-line
+ `@$CODEX_HOME/RTK.md` include. codex-cli 0.153.1 loads AGENTS.md verbatim and does not
+ expand `@` (verified with `codex debug prompt-input`: the literal path reached the model
+ and the rule did not). So the scaffold must inline the file itself, under CODEX_HOME --
+ not $HOME -- and say on the row that the rule is now literally present."""
+ ctx = _codex_rtk_ctx(tmp_path, monkeypatch)
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["rtk_mode"] == "instructions"
+ assert notes["rtk_hook_registered"] is None # not applicable, not False
+ assert notes["rtk_installed"] and notes["rtk_instructions_written"]
+ assert notes["rtk_agents_inlined"] is True
+ init = [(s, e) for s, e in ctx.execs if s.startswith("rtk init")]
+ assert init == [("rtk init -g --codex", {"CODEX_HOME": "/opt/codex-home",
+ "HOME": "/tmp/agent-home"})]
+ inline, env = [(s, e) for s, e in ctx.execs if "grep -c" in s][0]
+ assert inline.startswith("H=/opt/codex-home;")
+ assert "$H/RTK.md" in inline and "$H/AGENTS.md" in inline
+ assert "Always prefix shell commands with" in inline
+ assert env["CODEX_HOME"] == "/opt/codex-home"
+ # nothing hook-shaped was written anywhere
+ assert not any("PreToolUse" in s or "plugin install" in s for s, _ in ctx.execs)
+ assert not any("settings.json" in d for d, _ in ctx.copies)
+
+
+def test_rtk_on_codex_warns_loudly_when_the_rule_never_reached_agents_md(tmp_path, monkeypatch):
+ """The failure that matters is silent: files written, agent never told. The row and the
+ log must both say the arm would run native."""
+ ctx = _codex_rtk_ctx(tmp_path, monkeypatch, rule_count="0")
+ notes = scaffolds.get("rtk").setup(ctx)
+ assert notes["rtk_installed"] and not notes["rtk_instructions_written"]
+ assert notes["rtk_agents_inlined"] is False
+ assert any("would run as native" in m for m in ctx.logs)
+
+
+def test_rtk_on_codex_refuses_a_profile_that_does_not_say_where_codex_home_is(tmp_path, monkeypatch):
+ """Without CODEX_HOME the install would land in a directory the CLI never reads --
+ healthy provenance over a native arm. Better to refuse."""
+ monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))
+ (tmp_path / "cache" / "bin").mkdir(parents=True)
+ (tmp_path / "cache" / "bin" / "rtk").write_text("#!/bin/sh\n")
+ (tmp_path / "cache" / "bin" / "rtk").chmod(0o755)
+ bare = scaffolds.AgentProfile(kind="codex", home="/tmp/agent-home",
+ instructions_file="AGENTS.md", nav_tools=(),
+ mcp_args=lambda cfg: [])
+ ctx = FakeCtx(bare, tmp_path, exec_result=lambda s: _ok("rtk 0.45.0"))
+ with pytest.raises(scaffolds.ScaffoldError, match="CODEX_HOME"):
+ scaffolds.get("rtk").setup(ctx)
+
+
+# ---- serena on Codex: the same server, spelled for the client that launches it -----------
+def test_serena_on_codex_uses_serenas_codex_context_and_hands_the_runner_the_spec(tmp_path):
+ """Codex takes MCP servers from config.toml, not argv, so the scaffold cannot wire itself
+ the way it does on Claude; it returns the spec (Codex only -- Claude and Copilot rows stay
+ byte-identical) and the runner writes it. Serena ships a `codex` context for this client
+ (oraios.github.io/serena, clients); everything else about the server is unchanged, so the
+ three lanes run the same baseline."""
+ ctx = FakeCtx(CODEX, tmp_path, exec_result=lambda s: _ok("serena 1.0"))
+ notes = scaffolds.get("serena").setup(ctx)
+ assert notes["serena_installed"] and notes["serena_nudged"]
+ spec = notes["mcp_servers"]["serena"]
+ assert spec["command"] == "/tmp/agent-home/.local/bin/serena"
+ assert spec["args"][:3] == ["start-mcp-server", "--project", "/testbed"]
+ assert "--context" in spec["args"] and spec["args"][spec["args"].index("--context") + 1] == "codex"
+ assert "--transport" in spec["args"] and "stdio" in spec["args"]
+ # the nudge lands where Codex reads project instructions, and steers away from the SHELL
+ nudge = ctx.written("/testbed/AGENTS.md")
+ assert "find_symbol" in nudge and "rg/grep/sed/cat" in nudge
+ # Claude keeps its context and gets no spec in its notes
+ cctx = FakeCtx(CLAUDE, tmp_path, exec_result=lambda s: _ok("serena 1.0"))
+ cnotes = scaffolds.get("serena").setup(cctx)
+ assert "mcp_servers" not in cnotes
+ assert "ide-assistant" in scaffolds.get("serena").mcp_server(CLAUDE)["args"]
diff --git a/experiments/tests/test_codex_routing.py b/experiments/tests/test_codex_routing.py
new file mode 100644
index 0000000..3ee8d4e
--- /dev/null
+++ b/experiments/tests/test_codex_routing.py
@@ -0,0 +1,631 @@
+"""The Codex lane bills the OpenAI API directly, and the pieces that must agree about it.
+
+ OPENAI_API_KEY the credential -- required; the lane refuses at construction without it
+ OPENAI_BASE_URL OpenAI's own regional endpoint, for a region-scoped key; the global
+ endpoint otherwise
+
+Nothing else selects an endpoint. Pinned here, each of which failed silently at least once
+while every launch banner stayed green:
+
+ * the RESOLUTION, including that no other variable moves the lane off the OpenAI API;
+ * the REQUEST CONTRACT config.toml expresses, and that it carries the key by NAME only;
+ * the CACHE ARITHMETIC in Codex's usage block, where `input_tokens` is the GRAND total
+ and forgetting to subtract the cache fields prices ~90% cached re-reads at 1.0x.
+
+All offline: an explicit env dict, no network, no CLI binary, no credentials.
+"""
+import json
+import os
+import subprocess
+
+import pytest
+
+from te_harness import codex
+from te_harness.guards import net_guard
+from tokeneconomizer.mechanisms import codex_untool, spiral as spiral_mech, tooling
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+_KEY = "sk-test-key-abcd"
+_DIRECT = {"base_url": codex.OPENAI_DIRECT, "api_key": _KEY}
+
+
[email protected]("env,base", [
+ ({"OPENAI_API_KEY": _KEY}, "https://api.openai.com/v1"),
+ ({"OPENAI_API_KEY": _KEY, "OPENAI_BASE_URL": "https://us.api.openai.com/v1/"},
+ "https://us.api.openai.com/v1"),
+ # Not consulted: nothing but OPENAI_BASE_URL names the endpoint.
+ ({"OPENAI_API_KEY": _KEY, "ANTHROPIC_BASE_URL": "https://example.invalid"},
+ "https://api.openai.com/v1"),
+])
+def test_the_lane_resolves_to_the_openai_api(env, base):
+ assert codex.resolve_codex_routing(dict(env)) == {"base_url": base, "api_key": _KEY}
+
+
+ {}, {"OPENAI_BASE_URL": "https://us.api.openai.com/v1", "ANTHROPIC_API_KEY": "sk-ant-x"}])
+def test_without_the_openai_key_the_lane_is_refused_at_construction(env):
+ """Not once per task: a lane with nothing to bill must die before task 0, and no other
+ variable stands in for the key."""
+ with pytest.raises(RuntimeError, match="OPENAI_API_KEY"):
+ codex.resolve_codex_routing(dict(env))
+
+
+def test_validate_priceable_refuses_an_unknown_id():
+ """`weights_for` hard-fails on an unpriced id at the END of a paid rollout, so the
+ runner checks at construction instead."""
+ codex.validate_priceable("gpt-5.6-sol", "codex") # must not raise
+ with pytest.raises(ValueError):
+ codex.validate_priceable("not-a-model", "codex")
+
+
+# ---- the request contract -------------------------------------------------------------
+def test_config_toggles_cover_exactly_the_mechanism_s_config_capabilities():
+ """Layer 1 nominates the packages; the runner owns the mapping. They must line up.
+
+ A capability the mechanism reports as config-pruned but the runner has no toggle for is
+ an OVERSTATED saving on the arm whose entire claim is the saving.
+ """
+ assert codex.CONFIG_CAPABILITIES == codex_untool.CONFIG_CAPABILITIES
+
+
+def test_config_toggles_refuse_a_protected_capability():
+ with pytest.raises(RuntimeError, match="protected"):
+ codex.config_toggles(["collaboration", "apply_patch"])
+
+
+def test_web_is_blocked_on_every_arm_including_an_empty_mask():
+ """Contamination control, not a mechanism: the answer key is a fetch away.
+
+ Mirrors `swebench.network_tool_block` on the other two lanes. The consequence is
+ deliberate and documented -- with web off in the baseline too, the `web` package
+ contributes no untooling saving on this benchmark.
+ """
+ top, tables = codex.config_toggles([])
+ # The TOP-LEVEL spelling only. codex-cli 0.153.1 deprecated `[features].web_search`
+ # ("web search is enabled by default; set `web_search` to disabled at the top level")
+ # and emits an error item into the transcript when it is present, so the redundant
+ # table entries were dropped -- see codex._TABLE_TOGGLES["web"].
+ assert 'web_search = "disabled"' in top
+ assert not any(t == "features" and "web_search" in line for t, line in tables)
+ assert not any(t == "tools" and "web_search" in line for t, line in tables)
+ # ...and asking for `web` as well must not duplicate a key: Codex rejects that.
+ top2, tables2 = codex.config_toggles(["web"])
+ assert len(top2) == len(set(top2)) and len(tables2) == len(set(tables2))
+ assert sorted(top) == sorted(top2)
+
+
+def test_network_block_can_be_lifted(monkeypatch):
+ monkeypatch.setenv("TE_ALLOW_NETWORK_TOOLS", "1")
+ top, tables = codex.config_toggles([])
+ assert top == [] and tables == []
+
+
+def test_config_toml_names_the_openai_api_and_the_key_by_name_only():
+ txt = codex.config_toml(routing=_DIRECT, model="gpt-5.6-sol", effort="high",
+ capabilities=())
+ assert 'model_provider = "openai_direct"' in txt
+ assert "[model_providers.openai_direct]" in txt
+ assert txt.count("[model_providers.") == 1
+ assert f'base_url = "{codex.OPENAI_DIRECT}"' in txt
+ assert 'env_key = "OPENAI_API_KEY"' in txt and codex.KEY_ENV == "OPENAI_API_KEY"
+ assert 'wire_api = "responses"' in txt
+ assert 'trust_level = "trusted"' in txt
+ assert "requires_openai_auth" not in txt
+ # The credential VALUE never goes in the file -- only the variable name.
+ assert _KEY not in txt
+
+
+def test_a_regional_endpoint_is_written_as_given():
+ r = codex.resolve_codex_routing({"OPENAI_API_KEY": _KEY,
+ "OPENAI_BASE_URL": "https://eu.api.openai.com/v1"})
+ txt = codex.config_toml(routing=r, model="gpt-5.6-sol", effort="high")
+ assert 'base_url = "https://eu.api.openai.com/v1"' in txt
+
+
+def test_the_key_reaches_the_container_by_name_only():
+ assert codex.KEY_ENV not in codex.run_env()
+ assert _KEY not in codex.run_env().values()
+
+
+# ---- accounting: the cache arithmetic -------------------------------------------------
+def _stream(*events):
+ return "\n".join(json.dumps(e) for e in events)
+
+
+def test_row_usage_decomposes_the_grand_input_total():
+ """`input_tokens` INCLUDES both cache fields; fresh input is what is left.
+
+ Folding the cache fields into input_tokens instead would price cached re-reads at 1.0x
+ where the vendor bills 0.10x -- the same one-directional error that made Copilot rows
+ look cheaper than Claude rows before the cache-write fix.
+ """
+ row = codex.row_usage(_stream(
+ {"type": "thread.started", "thread_id": "th-1"},
+ {"type": "item.completed", "item": {"type": "command_execution",
+ "aggregated_output": "ok"}},
+ {"type": "item.completed", "item": {"type": "file_change"}},
+ {"type": "item.completed", "item": {"type": "agent_message"}},
+ {"type": "turn.completed", "usage": {
+ "input_tokens": 10_000, "cached_input_tokens": 9_000,
+ "cache_write_input_tokens": 400, "output_tokens": 300,
+ "reasoning_output_tokens": 120}},
+ ))
+ assert row["input_tokens"] == 600 # 10000 - 9000 - 400
+ assert row["cache_read_input_tokens"] == 9_000
+ assert row["cache_creation_input_tokens"] == 400
+ assert row["output_tokens"] == 420 # output + reasoning, as billed
+ assert row["tokens_total"] == 600 + 9_000 + 400 + 420
+ assert row["num_turns"] == 3 # steps, not turn.completed events
+ assert row["turns_basis"] == "codex_steps"
+ assert row["thread_id"] == "th-1"
+ assert row["tokens_source"] == "stream_usage"
+
+
+def test_row_usage_accumulates_across_turns():
+ row = codex.row_usage(_stream(
+ {"type": "turn.completed", "usage": {"input_tokens": 100,
+ "cached_input_tokens": 10,
+ "output_tokens": 5}},
+ {"type": "turn.completed", "usage": {"input_tokens": 200,
+ "cached_input_tokens": 20,
+ "output_tokens": 7}},
+ ))
+ assert row["input_tokens"] == 270 and row["cache_read_input_tokens"] == 30
+ assert row["output_tokens"] == 12
+
+
+def test_a_killed_stream_records_unknown_not_zero():
+ """A guard/timeout kill takes the usage block with it.
+
+ None, not 0: a task recorded as free would be averaged in as a cheap success on
+ precisely the guarded arm the spiral mechanism must show saving on.
+ """
+ row = codex.row_usage(_stream(
+ {"type": "item.completed", "item": {"type": "command_execution"}}))
+ assert row["input_tokens"] is None
+ assert row["tokens_total"] is None
+ assert row["tokens_source"] == "no_turn_event"
+
+
+def test_merge_usage_propagates_unknown():
+ good = codex.row_usage(_stream({"type": "turn.completed", "usage": {
+ "input_tokens": 100, "cached_input_tokens": 10, "output_tokens": 5}}))
+ dead = codex.row_usage("")
+ merged = codex.merge_usage(good, dead)
+ assert merged["input_tokens"] is None, "a half-known total must not read as known"
+ assert merged["tokens_source"] != "stream_usage"
+ both = codex.merge_usage(good, good)
+ assert both["input_tokens"] == 180 and both["tokens_source"] == "stream_usage"
+
+
+def test_merge_keeps_the_first_thread_id():
+ """A later resume must continue the FIRST segment's thread, never a new session."""
+ a = codex.row_usage(_stream({"type": "thread.started", "thread_id": "th-A"},
+ {"type": "turn.completed", "usage": {"input_tokens": 1}}))
+ b = codex.row_usage(_stream({"type": "thread.started", "thread_id": "th-B"},
+ {"type": "turn.completed", "usage": {"input_tokens": 1}}))
+ assert codex.merge_usage(a, b)["thread_id"] == "th-A"
+
+
+# ---- the net guard's payload spelling -------------------------------------------------
[email protected]("payload", [
+ {"tool_name": "Bash", "tool_input": {"command": "curl https://github.com/x"}},
+ {"toolName": "bash", "toolArgs": {"command": "curl https://github.com/x"}},
+ {"tool": {"name": "shell",
+ "input": {"command": ["bash", "-lc", "curl https://github.com/x"]}}},
+ {"item": {"type": "command_execution", "command": "curl https://github.com/x"}},
+])
+def test_guard_reads_every_payload_spelling(payload):
+ """Codex's `pre_tool_use` spelling is not captured off a live hook, and its hooks table
+ is not strictly validated -- so the guard must not depend on one key name. An
+ unreadable payload means `decide("")`, exit 0, and a lane that looks armed and guards
+ nothing."""
+ cmd = net_guard.hook_command(payload)
+ assert cmd, payload
+ assert net_guard.decide(cmd)[0] is True
+
+
+def test_guard_still_returns_empty_for_a_payload_with_no_command():
+ assert net_guard.hook_command({"nothing": 1}) == ""
+ assert net_guard.decide("")[0] is False
+
+
+# ---- the mechanism gates --------------------------------------------------------------
+def test_the_codex_label_gets_the_codex_mechanisms():
+ """The harness registers exactly one Codex label, and both mechanism gates must open on
+ it -- a gate keyed on a label the registry does not have would construct no controller
+ while every row still carried the arm's name."""
+ from te_harness import swebench, webbench
+ assert "codex" in tooling._CODEX_AGENTS and "codex" in spiral_mech._CODEX_AGENTS
+ for runners in (swebench.RUNNERS, webbench.RUNNERS):
+ assert [a for a in runners if a.startswith("codex")] == ["codex"]
+
+
+# ---- the results tree ---------------------------------------------------------------
+def _sh(script):
+ p = subprocess.run(["bash", "-c", ". scripts/launch/_env.sh; " + script], cwd=_REPO,
+ capture_output=True, text=True)
+ assert p.returncode == 0, p.stderr
+ return p.stdout.strip()
+
+
+def test_the_codex_tree_is_named_by_agent_domain_and_model():
+ assert _sh('te_tree_name codex swebench "$(te_model_tag gpt-5.6-sol)"') == \
+ "codex_swebench_gpt56sol"
+
+
+def test_a_model_tag_never_contains_a_path_separator():
+ """A slash in the tag would turn one tree into a nested PATH and split the cell."""
+ for model in ("vendor/gpt-5.6-sol", "gpt-5.6-sol"):
+ assert "/" not in _sh(f'te_model_tag "{model}"'), model
+
+
+def test_claude_model_tags_are_unchanged():
+ """The tag helper is shared, so the Codex lane must not move any stored tree's name."""
+ for model, expect in (("claude-opus-4-8", "opus48"), ("claude-opus-4.8", "opus48"),
+ ("claude-sonnet-5", "sonnet5"), ("auto", ""), ("", "")):
+ assert _sh(f'te_model_tag "{model}"') == expect, model
+
+
+def test_the_mechanism_call_is_a_tool_free_responses_request_to_the_openai_api(monkeypatch):
+ """The Codex twin of the Claude lane's Messages call: the lane's own model, on its own
+ key, at its own endpoint, with no tools offered -- a pure text completion."""
+ import io, urllib.request
+ seen = {}
+
+ class _Resp(io.BytesIO):
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *a):
+ return False
+
+ def fake_urlopen(req, timeout=0):
+ seen["body"] = json.loads(req.data.decode())
+ seen["url"] = req.full_url
+ seen["headers"] = {k.lower(): v for k, v in req.header_items()}
+ return _Resp(json.dumps({"output": [{"type": "message", "content": [
+ {"type": "output_text", "text": "ok"}]}], "usage": {"input_tokens": 3,
+ "output_tokens": 1}}).encode())
+
+ monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+ text, eff = codex.optimizer_call("Reply with the single word: ok",
+ configured_model="gpt-5.6-terra",
+ routing=_DIRECT, max_tokens=16)
+ assert text == "ok" and eff > 0
+ assert seen["url"] == "https://api.openai.com/v1/responses"
+ assert seen["headers"]["authorization"] == f"Bearer {_KEY}"
+ assert "user-agent" not in seen["headers"]
+ body = seen["body"]
+ assert body["model"] == "gpt-5.6-terra"
+ assert "tools" not in body and "tool_choice" not in body
+ assert body["max_output_tokens"] == 16 and body["store"] is False
+
+
+def test_the_model_check_reads_the_openai_catalog(monkeypatch):
+ import io, urllib.request
+
+ class _Resp(io.BytesIO):
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *a):
+ return False
+
+ seen = {}
+
+ def fake_urlopen(req, timeout=0):
+ seen["url"] = req.full_url
+ seen["auth"] = req.get_header("Authorization")
+ return _Resp(json.dumps({"object": "list", "data": [
+ {"id": "gpt-5.6-sol"}, {"id": "gpt-5.6-terra"}]}).encode())
+
+ codex._CATALOG.clear()
+ monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+ assert codex.verify_model("gpt-5.6-sol", _DIRECT) == (True, "")
+ ok, why = codex.verify_model("gpt-9", _DIRECT)
+ assert ok is False and "gpt-5.6-terra" in why
+ assert seen == {"url": "https://api.openai.com/v1/models", "auth": f"Bearer {_KEY}"}
+ codex._CATALOG.clear()
+
+
+def test_an_unreadable_catalog_is_inconclusive_not_a_refusal(monkeypatch):
+ import urllib.request
+
+ def _boom(*a, **k):
+ raise OSError("connection refused")
+
+ codex._CATALOG.clear()
+ monkeypatch.setattr(urllib.request, "urlopen", _boom)
+ served, why = codex.verify_model("gpt-5.6-sol", _DIRECT)
+ assert served is None and "unreadable" in why
+ codex._CATALOG.clear()
+
+
+def test_the_skills_package_is_pruned_by_disabling_every_bundled_system_skill():
+ """0.153.1 has no switch for the <skills_instructions> listing, but disabling each
+ bundled skill through `[[skills.config]]` removes the whole block. The entries name
+ in-container paths and come last in the file, because an array-of-tables header
+ swallows any key after it."""
+ txt = codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra", effort="high",
+ capabilities=["skills"], system_skills=("alpha", "beta"))
+ blocks = txt.split("[[skills.config]]")
+ assert len(blocks) == 3
+ assert f'path = "{codex.C_HOME}/skills/.system/alpha/SKILL.md"' in blocks[1]
+ assert "enabled = false" in blocks[1] and "enabled = false" in blocks[2]
+ assert txt.index("[projects.") < txt.index("[[skills.config]]")
+ # without the package in the mask no entry is written; the default set covers 0.153.1
+ plain = codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra", effort="high",
+ capabilities=[])
+ assert "skills.config" not in plain
+ assert len(codex.skills_config_entries(None)) == len(codex.DEFAULT_SYSTEM_SKILLS) == 6
+ assert "skills" in codex.CONFIG_CAPABILITIES
+
+
+def _alive(pid):
+ """True while `pid` runs (a zombie awaiting its reaper counts as gone)."""
+ try:
+ with open(f"/proc/{pid}/stat") as fh:
+ return fh.read().split()[2] != "Z"
+ except OSError:
+ return False
+
+
+def _fake_cli(tmp_path, body):
+ """A stand-in for the npm `codex` shim: it starts a long-lived CHILD, as the real shim
+ execs the native ELF, records the child's pid, then runs `body`."""
+ pidfile = tmp_path / "child.pid"
+ shim = tmp_path / "codex"
+ shim.write_text(f"#!/bin/sh\nsleep 600 &\necho $! > {pidfile}\n{body}\n")
+ shim.chmod(0o755)
+ return str(shim), pidfile
+
+
+def _assert_reaped(pidfile):
+ import time
+ pid = int(pidfile.read_text())
+ for _ in range(50):
+ if not _alive(pid):
+ return
+ time.sleep(0.1)
+ raise AssertionError(f"the CLI's child {pid} outlived the probe")
+
+
[email protected](not os.path.isdir("/proc"), reason="needs /proc")
+def test_the_skills_probe_kills_the_cli_it_started_not_just_the_shim(tmp_path):
+ """`proc.kill()` reached only the Node shim; the native CLI it had started sat
+ "waiting for network" for good, one leaked per offline suite run (six alive on this host
+ on 2026-09-30). The probe now owns a process group and kills all of it."""
+ shim, pidfile = _fake_cli(tmp_path, (
+ 'mkdir -p "$CODEX_HOME/skills/.system/alpha"\n'
+ 'touch "$CODEX_HOME/skills/.system/alpha/SKILL.md"\n'
+ 'touch "$CODEX_HOME/skills/.system/.codex-system-skills.marker"\n'
+ "wait"))
+ assert codex.bundled_system_skills(binary=shim, home=str(tmp_path), timeout=20) == ["alpha"]
+ _assert_reaped(pidfile)
+
+
[email protected](not os.path.isdir("/proc"), reason="needs /proc")
+def test_the_config_validator_kills_a_hung_cli_and_its_children(tmp_path):
+ shim, pidfile = _fake_cli(tmp_path, "sleep 600")
+ err = codex.validate_config('model = "gpt-5.6-sol"\n', binary=shim, home=str(tmp_path),
+ timeout=2)
+ assert err.startswith("could not run the config validator")
+ _assert_reaped(pidfile)
+
+
+def test_bundled_system_skills_are_read_from_the_installed_cli():
+ """The skills toggle disables bundled skills BY PATH, so the set must come from the CLI
+ that will run, not from a list written for one version. Offline: the throwaway run is
+ killed at the skills marker before it can wait for a network it does not have."""
+ import shutil
+ if not (os.environ.get("TE_CODEX_BIN") or shutil.which("codex")):
+ pytest.skip("codex-cli not installed")
+ found = codex.bundled_system_skills()
+ assert found, "discovery returned nothing; the runner would fall back to the static set"
+ assert all("/" not in n for n in found)
+ # 0.153.1 ships these; a newer CLI may add to the set, which is exactly why it is read.
+ assert {"skill-creator", "skill-installer"} <= set(found)
+
+
+# ---------------------------------------------------------------------------------------
+# The instruction package (2026-09-11): chat-session guidance in the base prompt is
+# removed through a lean copy written to model_instructions_file; operational sections
+# stay byte-for-byte.
+# ---------------------------------------------------------------------------------------
+_FAKE_BASE = (
+ "You are Codex.\n\n# Personality\n\nBe charming.\n\n## Writing style\n\nShort.\n\n"
+ "# Working with the user\n\nChannels.\n\nYou end your turn by sending a final message.\n\n"
+ "When you run out of context, continue.\n\n"
+ "# Rules for getting work done\n\n- use rg\n\n## Autonomy and persistence\n\nAct.\n\n"
+ "# Using skills\n\nSkills intro.\n\n### How to use skills\n\nRead SKILL.md.\n")
+
+
+def test_lean_instructions_drops_only_the_pruned_sections_and_keeps_the_rest_verbatim():
+ lean = codex.lean_instructions(_FAKE_BASE, ["interactive_guidance"])
+ assert "# Personality" not in lean and "## Writing style" not in lean
+ assert "# Working with the user" not in lean and "Channels." not in lean
+ assert "When you run out of context, continue." in lean # compaction paragraph kept
+ # the stopping rule is operational and is kept too: without it the agent has no
+ # statement of how a turn ends, and every untool cell ran more model calls than its
+ # baseline.
+ assert "You end your turn by sending a final message." in lean
+ for kept in ("You are Codex.", "# Rules for getting work done\n\n- use rg",
+ "## Autonomy and persistence\n\nAct.", "# Using skills", "### How to use skills"):
+ assert kept in lean
+ both = codex.lean_instructions(_FAKE_BASE, ["interactive_guidance", "skills"])
+ assert "# Using skills" not in both and "Read SKILL.md." not in both
+ assert "# Rules for getting work done" in both
+ # nothing to drop -> no file, the CLI keeps its built-in prompt
+ assert codex.lean_instructions(_FAKE_BASE, ["collaboration", "goals"]) is None
+ assert codex.lean_instructions(None, ["interactive_guidance"]) is None
+
+
+def test_the_captured_base_prompt_exists_for_the_pinned_cli_and_for_no_other():
+ base = codex.base_prompt_for("codex-cli 0.153.1")
+ assert base and base.startswith("You are Codex") and len(base) // 4 > 4000
+ assert codex.base_prompt_for("codex-cli 9.9.9") is None
+ assert codex.base_prompt_for("") is None
+ lean = codex.lean_instructions(base, ["interactive_guidance", "skills"])
+ assert len(lean) < 0.45 * len(base)
+ for header in ("# Rules for getting work done", "## File editing constraints",
+ "## Autonomy and persistence", "# Destructive Actions"):
+ assert header in lean
+ for header in codex.INSTRUCTION_SECTIONS["interactive_guidance"] + codex.INSTRUCTION_SECTIONS["skills"]:
+ assert ("\n" + header + "\n") not in lean
+
+
+def test_config_toml_names_the_instructions_file_only_when_a_lean_prompt_exists():
+ with_file = codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra", effort="high",
+ capabilities=["interactive_guidance"],
+ instructions="lean text")
+ assert f'model_instructions_file = "{codex.INSTRUCTIONS_PATH}"' in with_file
+ without = codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra", effort="high",
+ capabilities=["interactive_guidance"], instructions=None)
+ assert "model_instructions_file" not in without
+ assert "interactive_guidance" in codex.CONFIG_CAPABILITIES
+
+
+def test_codex_trimech_names_no_endpoint():
+ """The arm is endpoint-agnostic by construction: nothing in its config names one.
+
+ Stated as a test because the three mechanisms reach the runner through three different
+ channels (prompt prefix, config.toml, live stream) and each one could have grown an
+ endpoint assumption; the endpoint is the runner's to resolve.
+ """
+ from tokeneconomizer.arms import ARMS
+ import dataclasses
+ cfg = ARMS["codex-trimech"].cfg()
+ flat = repr(dataclasses.asdict(cfg)) if dataclasses.is_dataclass(cfg) else repr(cfg)
+ for token in ("openai", "copilot", "http", "base_url", "api_key"):
+ assert token not in flat.lower(), f"{token!r} leaked into the arm's config"
+
+
+def test_rtk_compliance_is_read_off_the_trace(tmp_path):
+ """On Codex rtk is an instruction, not a hook, so whether it did anything is whether the
+ agent typed `rtk`. Counted from the transcript like answer_key_attempts; None when there
+ is no transcript, so a lane of zeros stays distinguishable from a lane of no data."""
+ from te_harness import codex
+ from fixtures import write_codex_transcript
+ p = str(tmp_path / "t.jsonl")
+ write_codex_transcript(p, [
+ {"type": "command_execution", "command": '/bin/bash -lc "rtk git status"',
+ "aggregated_output": "ok", "exit_code": 0},
+ {"type": "command_execution", "command": '/bin/bash -lc "pytest -q"',
+ "aggregated_output": "1 passed", "exit_code": 0},
+ {"type": "command_execution", "command": "/bin/bash -lc '/usr/local/bin/rtk proxy make'",
+ "aggregated_output": "", "exit_code": 0},
+ {"type": "file_change", "changes": [{"path": "a.py", "kind": "update"}]},
+ ])
+ got = codex.rtk_compliance(p)
+ assert got == {"rtk_shell_commands": 3, "rtk_prefixed_commands": 2,
+ "rtk_compliance": 0.667}
+ assert codex.rtk_compliance(str(tmp_path / "missing.jsonl")) == {
+ "rtk_shell_commands": None, "rtk_prefixed_commands": None, "rtk_compliance": None}
+ assert codex._unwrap_shell('/bin/bash -lc "rtk ls"') == "rtk ls"
+ assert codex._unwrap_shell("sh -c 'rtk ls'") == "rtk ls"
+ assert codex._unwrap_shell("rtk ls") == "rtk ls"
+
+
+# ---- MCP servers reach Codex through config.toml -----------------------------------------
+def _spec():
+ return {"serena": {"command": "/tmp/agent-home/.local/bin/serena",
+ "args": ["start-mcp-server", "--project", "/testbed", "--context", "codex"],
+ "env": {"PATH": "/tmp/agent-home/.local/bin:/usr/bin"}}}
+
+
+def test_config_toml_writes_an_mcp_server_table_the_cli_validates():
+ """The block is emitted before the array-of-tables skills entries (which must be last),
+ with args as a TOML array and env as a sub-table. On 0.153.1 `--strict-config` accepts
+ command/args/env/cwd/startup_timeout_sec and refuses an unknown key, so the acceptance
+ is checked against the real CLI when one is installed."""
+ from te_harness import codex
+ t = codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra", effort="high",
+ capabilities=("skills",), mcp_servers=_spec())
+ i = t.index("[mcp_servers.serena]")
+ assert t.index("[[skills.config]]") > i
+ body = t[i:]
+ assert 'command = "/tmp/agent-home/.local/bin/serena"' in body
+ assert 'args = ["start-mcp-server", "--project", "/testbed", "--context", "codex"]' in body
+ assert "startup_timeout_sec = 90" in body
+ assert "[mcp_servers.serena.env]" in body and 'PATH = "/tmp/agent-home/.local/bin:/usr/bin"' in body
+ # no servers -> no table at all: the baseline config is unchanged by this feature
+ assert "mcp_servers" not in codex.config_toml(routing=_DIRECT, model="gpt-5.6-terra",
+ effort="high")
+ import shutil, os, tempfile
+ if shutil.which("codex") or os.environ.get("TE_CODEX_BIN"):
+ home = tempfile.mkdtemp(dir=os.path.dirname(os.path.abspath(__file__)))
+ try:
+ err = codex.validate_config(t, home=home)
+ assert not (err or "").strip(), err
+ finally:
+ shutil.rmtree(home, ignore_errors=True)
+
+
+def test_mcp_tool_calls_counts_completed_items_and_keeps_a_sample_of_what_it_cannot_read(tmp_path):
+ """A configured server is not a used one. The reader counts `mcp_tool_call` items at
+ completion, per server/tool; an item whose tool name sits somewhere unexpected is counted
+ as unrecognised and ONE raw sample is kept.
+
+ The shape below is the one codex-cli 0.153.1 actually emits, observed live on 2026-09-13
+ (`item.completed`, item keys: id, type, server, tool, arguments, result, error, status) --
+ and the first such call on that lane was Codex's OWN `list_mcp_resources` on server
+ `codex`, which is exactly why the count is split per server: `mcp_tool_calls` includes it,
+ `serena_tool_calls` does not."""
+ from te_harness import codex
+ import json
+ p = str(tmp_path / "t.jsonl")
+ live_shape = {"id": "item_2", "type": "mcp_tool_call", "server": "codex",
+ "tool": "list_mcp_resources", "arguments": {},
+ "result": {"content": [{"type": "text", "text": "{\"resources\":[]}"}],
+ "structured_content": None},
+ "error": None, "status": "completed"}
+ with open(p, "w") as fh:
+ fh.write(json.dumps({"type": "item.started", "item": {"type": "mcp_tool_call", "server": "serena", "tool": "find_symbol"}}) + "\n")
+ fh.write(json.dumps({"type": "item.completed", "item": {"type": "mcp_tool_call", "server": "serena", "tool": "find_symbol", "status": "completed"}}) + "\n")
+ fh.write(json.dumps({"type": "item.completed", "item": live_shape}) + "\n")
+ fh.write(json.dumps({"type": "item.completed", "item": {"type": "mcp_tool_call", "invocation": {"server": "serena", "tool": "get_symbols_overview"}}}) + "\n")
+ fh.write(json.dumps({"type": "item.completed", "item": {"type": "mcp_tool_call", "weird": 1}}) + "\n")
+ fh.write(json.dumps({"type": "item.completed", "item": {"type": "command_execution", "command": "ls"}}) + "\n")
+ got = codex.mcp_tool_calls(p, server="serena")
+ assert got["mcp_tool_calls"] == 3 and got["serena_tool_calls"] == 2
+ assert got["mcp_tools_used"] == {"serena/find_symbol": 1, "serena/get_symbols_overview": 1,
+ "codex/list_mcp_resources": 1}
+ assert got["mcp_unrecognised_items"] == 1 and '"weird"' in got["mcp_item_sample"]
+ none = codex.mcp_tool_calls(str(tmp_path / "missing.jsonl"), server="serena")
+ assert none["mcp_tool_calls"] is None and none["serena_tool_calls"] is None
+
+
+def test_the_codex_lane_hosts_every_scaffold_and_wires_an_mcp_one_through_the_config(
+ tmp_path, monkeypatch):
+ """The refusal table is gone: rtk runs in its own Codex mode, repomap writes AGENTS.md,
+ and serena -- an MCP server -- is written into config.toml after setup, because `_start`
+ wrote the config before the scaffold existed. Both halves pinned: no scaffold name is
+ refused at construction, and a scaffold that returns `mcp_servers` causes exactly one
+ config rewrite carrying that spec."""
+ from te_harness import swebench
+ assert not hasattr(swebench, "_CODEX_UNAVAILABLE_SCAFFOLDS")
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ for name in ("rtk", "repomap", "serena"):
+ with pytest.raises(RuntimeError, match="nothing to bill"): # past the scaffold gate
+ swebench.CodexSwebenchRunner(agent="codex", scaffold=name)
+
+ r = object.__new__(swebench.CodexSwebenchRunner)
+ writes = []
+ monkeypatch.setattr(swebench.ClaudeSwebenchRunner, "_apply_scaffold",
+ lambda self, name, task, rd: ({"arm_setup": "serena", "mcp_servers": _spec(),
+ "mcp_config": "/tmp/agent-home/scaffold.mcp.json"},
+ "/tmp/agent-home/scaffold.mcp.json"))
+ r._write_config = lambda name: writes.append((name, r._mcp_servers))
+ notes, cfg = r._apply_scaffold("c1", {"instance_id": "x"}, str(tmp_path))
+ assert writes == [("c1", _spec())]
+ assert "mcp_servers" not in notes # a dict never lands in row.json
+ assert notes["mcp_wiring"] == "config.toml" and notes["mcp_servers_configured"] == ["serena"]
+ assert cfg == "/tmp/agent-home/scaffold.mcp.json"
+ # and a scaffold that asks for nothing rewrites nothing
+ writes.clear()
+ monkeypatch.setattr(swebench.ClaudeSwebenchRunner, "_apply_scaffold",
+ lambda self, name, task, rd: ({"arm_setup": "rtk"}, None))
+ r._apply_scaffold("c1", {"instance_id": "x"}, str(tmp_path))
+ assert writes == [] and r._mcp_servers is None
diff --git a/experiments/tests/test_codex_webbench.py b/experiments/tests/test_codex_webbench.py
new file mode 100644
index 0000000..bc93d16
--- /dev/null
+++ b/experiments/tests/test_codex_webbench.py
@@ -0,0 +1,334 @@
+"""The Codex Web-Bench lane: the contract it writes, and the two answer-key layers it arms.
+
+Web-Bench runs the agent on the HOST with no per-task container, which is what makes this
+lane different from the Codex SWE-bench one rather than a copy of it. Two things therefore
+have to be pinned here and nowhere else: the request contract must name HOST paths (a
+container path would silently name a file that does not exist), and the answer-key layers
+must live in the per-task CODEX_HOME and a namespace, because there is no container to hold
+them -- the PreToolUse hook in `$CODEX_HOME/hooks.json` (which Codex honours under
+`--dangerously-bypass-hook-trust`, verified live 2026-09-30), and the hosts block, which
+needs no cooperation from the CLI at all.
+"""
+import os
+import subprocess
+
+from te_harness import webbench as wb
+from te_harness import codex
+
+
+def _stub(tmp_path, caps=(), ns=True):
+ """A runner instance without construction: __init__ probes a CLI and a catalog, and
+ none of the behaviour under test depends on either."""
+ r = object.__new__(wb.CodexWebBenchRunner)
+ r.routing = {"base_url": codex.OPENAI_DIRECT, "api_key": "tok"}
+ r.api_key = "tok"
+ r.model, r.effort = "gpt-5.6-terra", "high"
+ r.base_prompt = "You are Codex\n\n# Personality\nchatty\n\n# Destructive Actions\ncare\n"
+ r.system_skills = None
+ r._capabilities = tuple(caps)
+ r._lean_instructions_tokens = 0
+ r.answer_key_ns = ns
+ r.agent_bin = "/usr/bin/true"
+ r.task_timeout = 5
+ return r
+
+
+# ---- the answer-key layer ---------------------------------------------------------------
+def test_the_hook_is_armed_in_the_task_codex_home(tmp_path):
+ """`$CODEX_HOME/hooks.json`, Claude Code's shape, and only reported armed once the guard
+ has refused its probe -- the shared installer's rule, now on this lane too."""
+ import json
+ r = _stub(tmp_path)
+ home = tmp_path / "run" / ".codex-home"
+ home.mkdir(parents=True)
+ assert r._install_net_guard(str(home), str(tmp_path)) is True
+ assert r._net_guard_armed is True
+ doc = json.loads((home / "hooks.json").read_text())
+ (entry,) = doc["hooks"]["PreToolUse"]
+ assert entry["matcher"] == "Bash"
+ assert entry["hooks"][0]["type"] == "command"
+ assert entry["hooks"][0]["command"].endswith("net_guard.py")
+ # the verification probe's own refusal is removed, so it is never counted as the agent's
+ assert not (home / "net_guard_blocks.jsonl").exists()
+ assert r._install_net_guard(str(home), str(tmp_path)) is True # idempotent
+ assert len(json.loads((home / "hooks.json").read_text())["hooks"]["PreToolUse"]) == 1
+
+
+def test_only_an_armed_task_runs_with_the_trust_bypass(tmp_path, monkeypatch):
+ """Codex skips every hook nobody approved in /hooks, so an armed task must pass
+ --dangerously-bypass-hook-trust; an unarmed one keeps exactly the argv it always had.
+ Either way the hook's log path reaches the agent's environment."""
+ seen = []
+
+ class _P:
+ def __init__(self, cmd, **kw):
+ seen.append((cmd, kw))
+ self.stdin, self.pid = type("W", (), {"write": lambda s, x: None,
+ "close": lambda s: None})(), 1
+ self.stdout, self.stderr = iter([]), type("R", (), {"read": lambda s: ""})()
+
+ def wait(self, timeout=None): return 0
+ def kill(self): pass
+
+ from te_harness.guards import net_guard
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"; ws.mkdir(parents=True)
+ r = _stub(tmp_path)
+ for armed in (False, True):
+ r._net_guard_armed = armed
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), {"mode": "resume", "session_id": "th-1", "prompt": "nudge"},
+ fh, [], 80, None)
+ (fresh_off, kw), (resume_off, _), (fresh_on, _), (resume_on, _) = seen
+ flag = "--dangerously-bypass-hook-trust"
+ assert flag not in fresh_off and flag not in resume_off
+ assert flag in fresh_on and flag in resume_on
+ assert kw["env"][net_guard.LOG_ENV].endswith("net_guard_blocks.jsonl")
+
+
+def test_the_hosts_file_carries_the_answer_key_block_on_top_of_the_real_one(tmp_path):
+ from te_harness.guards import net_guard
+ r = _stub(tmp_path)
+ path = r._hosts_file(str(tmp_path))
+ text = open(path).read()
+ assert net_guard.HOSTS_MARKER in text
+ for host in ("github.com", "api.github.com", "raw.githubusercontent.com"):
+ assert f" {host}" in text or f"\t{host}" in text
+ # the real hosts file is preserved, or the agent loses localhost and every local
+ # service it needs -- including the evaluator's own dev server
+ assert "localhost" in text
+
+
+def test_the_segment_runs_inside_a_namespace_that_bind_mounts_that_file(tmp_path, monkeypatch):
+ """The namespace is the guard. Everything the agent spawns inherits it, which is the
+ property the hook lacked -- it needs no cooperation from the CLI."""
+ seen = {}
+
+ class _P:
+ def __init__(self, cmd, **kw):
+ seen["cmd"], seen["kw"] = cmd, kw
+ self.stdin, self.pid = _W(), 1
+ self.stdout, self.stderr = iter([]), _R()
+
+ def wait(self, timeout=None): return 0
+ def kill(self): pass
+
+ class _W:
+ def write(self, s): seen["prompt"] = s
+ def close(self): pass
+
+ class _R:
+ def read(self): return ""
+
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"; ws.mkdir(parents=True)
+ r = _stub(tmp_path)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "do the thing", fh, [], 80, None)
+
+ cmd = seen["cmd"]
+ assert cmd[:4] == ["unshare", "--user", "--map-root-user", "--mount"]
+ assert cmd[4:6] == ["bash", "-c"]
+ inner = cmd[6]
+ assert "mount --bind" in inner and "/etc/hosts" in inner and inner.endswith('exec "$@"')
+ # the agent's own argv follows, and the workspace replaced the container's /testbed
+ assert r.agent_bin in cmd and "/testbed" not in cmd
+ assert str(ws) in cmd
+ assert seen["prompt"] == "do the thing"
+ # the credential reaches the CLI under the name the provider block reads
+ assert seen["kw"]["env"]["OPENAI_API_KEY"] == "tok"
+ assert seen["kw"]["cwd"] == str(ws)
+
+
+def test_without_a_usable_namespace_the_lane_runs_bare_and_says_so(tmp_path, monkeypatch):
+ """A host that refuses user namespaces must not silently produce unguarded rows that
+ look guarded: the segment drops the wrapper and `answer_key_hosts_blocked` goes false."""
+ seen = {}
+
+ class _P:
+ def __init__(self, cmd, **kw):
+ seen["cmd"] = cmd
+ self.stdin, self.pid = type("W", (), {"write": lambda s, x: None,
+ "close": lambda s: None})(), 1
+ self.stdout, self.stderr = iter([]), type("R", (), {"read": lambda s: ""})()
+
+ def wait(self, timeout=None): return 0
+ def kill(self): pass
+
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"; ws.mkdir(parents=True)
+ r = _stub(tmp_path, ns=False)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+ assert seen["cmd"][0] != "unshare"
+
+
+# ---- the request contract ---------------------------------------------------------------
+def test_the_contract_names_host_paths_not_container_paths(tmp_path):
+ """`model_instructions_file` pointing at a container path would name a file that does
+ not exist on the host, so the base prompt would not be replaced -- and the untooling arm
+ would quietly keep the sections it reports having dropped."""
+ ws = str(tmp_path / "workspace")
+ home = str(tmp_path / ".codex-home")
+ r = _stub(tmp_path, caps=("interactive_guidance",))
+ cfg = r._write_config(home, ws)
+ assert f'model_instructions_file = "{home}/instructions.md"' in cfg
+ assert codex.C_HOME not in cfg
+ assert os.path.exists(os.path.join(home, "instructions.md"))
+ assert r._lean_instructions_tokens > 0
+ # the WORKSPACE is the trusted project here, not /testbed
+ assert f'[projects."{ws}"]' in cfg and "/testbed" not in cfg
+ # and the endpoint is the OpenAI API the lane resolved, exactly as on SWE-bench
+ assert f'base_url = "{codex.OPENAI_DIRECT}"' in cfg
+
+
+def test_a_mask_that_prunes_nothing_writes_no_instructions_file(tmp_path):
+ r = _stub(tmp_path, caps=())
+ cfg = r._write_config(str(tmp_path / "h"), str(tmp_path / "w"))
+ assert "model_instructions_file" not in cfg
+ assert r._lean_instructions_tokens == 0
+
+
+# ---- accounting + guard ------------------------------------------------------------------
+def test_a_killed_segment_is_flagged_not_counted_as_free(tmp_path):
+ """Same predicate as the SWE-bench Codex runner, so one filter covers both domains."""
+ r = _stub(tmp_path)
+ good = r._row_from_segment({"tokens_source": "stream_usage", "output_tokens": 10})
+ assert good["usage_incomplete"] is False
+ bad = r._row_from_segment({"tokens_source": "no_turn_event", "output_tokens": None})
+ assert bad["usage_incomplete"] is True and bad["output_tokens_missing"] is True
+ assert "no_turn_event" in bad["usage_incomplete_reason"]
+
+
+def test_the_codex_controller_is_selected_not_the_generic_guard(tmp_path):
+ """The generic NudgeGuard parses a Claude stream and reprompts; feeding it a
+ `codex exec --json` stream would arm a guard that can never fire."""
+ from tokeneconomizer.mechanisms import codex_spiral
+ r = _stub(tmp_path)
+ assert r._make_guard({}) is None
+ assert r._make_guard({"spiral_guard": True}) is None # generic flags do not arm it
+ g = r._make_guard({"codex_spiral_policy": codex_spiral.shipped_policy(20)})
+ assert isinstance(g, codex_spiral.CodexResumeGuard)
+
+
+def test_only_rtk_is_hostable_and_it_is_the_instruction_mode(tmp_path, monkeypatch):
+ """Web-Bench hosts one scaffold for any agent, rtk. On Codex that is rtk's INSTRUCTION
+ mode (files under CODEX_HOME), because the hook mode is one this CLI ignores. Anything
+ else is refused rather than installed and ignored."""
+ import pytest
+ for other in ("repomap", "serena"):
+ with pytest.raises(RuntimeError, match="not wired for the Codex Web-Bench lane"):
+ wb.CodexWebBenchRunner(agent="codex", scaffold=other)
+ # every Codex ARM that carries an unhostable scaffold is refused the same way -- today
+ # exactly one, the repomap variant of codex-trimech (its generator walks `*.py`)
+ from tokeneconomizer import baselines
+ unhostable = sorted(a for a, s in baselines.ARM_SCAFFOLDS.items()
+ if a.startswith("codex-") and s not in wb.WEBBENCH_SCAFFOLDS)
+ assert unhostable == ["codex-trimech-repomap"], unhostable
+ for arm in unhostable:
+ with pytest.raises(RuntimeError, match="not wired for the Codex Web-Bench lane"):
+ wb.CodexWebBenchRunner(agent="codex", scaffold=baselines.ARM_SCAFFOLDS[arm])
+ # rtk gets past the scaffold gate; with no key the next stop is the endpoint
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ with pytest.raises(RuntimeError, match="nothing to bill"):
+ wb.CodexWebBenchRunner(agent="codex", scaffold="rtk")
+ # and the profile the scaffold sees points its install at THIS task's CODEX_HOME
+ r = _stub(tmp_path)
+ home = str(tmp_path / "run" / ".codex-home")
+ prof = r.scaffold_profile_for(home)
+ assert prof.kind == "codex" and prof.env == {"CODEX_HOME": home}
+ assert prof.instructions_file == "AGENTS.md" and "RTK.md" in prof.patch_excludes
+ assert prof.bin_dir == wb.WebBenchRunner.scaffold_bin_dir(home)
+ assert r.scaffold_exec_env(str(tmp_path / "run")) == {"CODEX_HOME": home}
+
+
+def test_the_scaffold_bin_dir_is_first_on_the_agents_path(tmp_path, monkeypatch):
+ """`rtk` is copied into a per-run bin dir on the host; the agent's own shell must find it
+ or every prefixed command fails with `rtk: command not found` on a row that reports the
+ scaffold healthy."""
+ seen = {}
+
+ class _P:
+ def __init__(self, cmd, **kw):
+ seen["env"] = kw["env"]
+ self.stdin, self.pid = type("W", (), {"write": lambda s, x: None,
+ "close": lambda s: None})(), 1
+ self.stdout, self.stderr = iter([]), type("R", (), {"read": lambda s: ""})()
+
+ def wait(self, timeout=None): return 0
+ def kill(self): pass
+
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"; ws.mkdir(parents=True)
+ r = _stub(tmp_path, ns=False)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+ home = r.agent_home(str(tmp_path / "run"))
+ assert seen["env"]["PATH"].split(os.pathsep)[0] == wb.WebBenchRunner.scaffold_bin_dir(home)
+
+
+def test_pruning_keeps_the_rtk_instructions_as_provenance(tmp_path):
+ r = _stub(tmp_path)
+ run_dir = tmp_path / "run"
+ home = run_dir / ".codex-home"; home.mkdir(parents=True)
+ for name in ("config.toml", "AGENTS.md", "RTK.md", "state_5.sqlite"):
+ (home / name).write_text("x")
+ r.keep_workspaces = False
+ r._prune_agent_home(str(run_dir))
+ assert sorted(os.listdir(home)) == ["AGENTS.md", "RTK.md", "config.toml"]
+
+
+def test_the_codex_runner_sets_every_field_the_inherited_lifecycle_reads():
+ """The Codex runner does NOT call WebBenchRunner.__init__ -- that resolves the Anthropic
+ key -- so every attribute the inherited methods read has to be set here by hand.
+
+ Pinned because of how the gap surfaces: not at construction, but as an AttributeError
+ deep in `run_task`, after the evaluator has started and the first task has been paid for.
+ That is exactly how `routing_base` was found, on a live Web-Bench run.
+ """
+ import inspect, re
+ base = inspect.getsource(wb.WebBenchRunner)
+ init = inspect.getsource(wb.CodexWebBenchRunner.__init__)
+ read = set(re.findall(r"self\.([a-z_][a-z0-9_]*)", base))
+ assigned = set(re.findall(r"self\.([a-z_][a-z0-9_]*)\s*=", init))
+ # set by the base helper this __init__ calls at the end
+ assigned |= set(re.findall(r"self\.([a-z_][a-z0-9_]*)\s*=",
+ inspect.getsource(wb.WebBenchRunner._init_stream_state)))
+ known = assigned | set(vars(wb.CodexWebBenchRunner)) | set(vars(wb.WebBenchRunner))
+ missing = sorted(a for a in read if a not in known and not a.startswith("_current"))
+ assert not missing, f"inherited code reads these, Codex never sets them: {missing}"
+
+
+def test_the_per_task_codex_home_is_pruned_to_its_provenance(tmp_path):
+ """The CLI's own state is ~38 MB per task-run and would put ~19 GB of sqlite WALs,
+ bundled skills and shell snapshots in the results tree over a five-project, five-seed
+ sweep. The SWE-bench lane has no such problem -- its CODEX_HOME dies with the container --
+ so the host lane prunes to restore that parity, keeping the contract, which is provenance.
+ """
+ r = _stub(tmp_path)
+ run_dir = tmp_path / "run"
+ home = run_dir / ".codex-home"; home.mkdir(parents=True)
+ (home / "config.toml").write_text("model = 'x'\n")
+ (home / "instructions.md").write_text("lean\n")
+ (home / "state_5.sqlite-wal").write_text("x" * 1000)
+ (home / "sessions").mkdir(); (home / "sessions" / "roll.jsonl").write_text("{}")
+ (run_dir / ".host-home").mkdir()
+
+ r.keep_workspaces = False
+ r._prune_agent_home(str(run_dir))
+ assert sorted(os.listdir(home)) == ["config.toml", "instructions.md"]
+ assert not (run_dir / ".host-home").exists()
+
+
+def test_keep_workspaces_keeps_the_whole_agent_home(tmp_path):
+ """The same knob that keeps the workspace for a post-mortem keeps this too -- one
+ switch, so an operator debugging a task does not have to know about two."""
+ r = _stub(tmp_path)
+ run_dir = tmp_path / "run"
+ home = run_dir / ".codex-home"; home.mkdir(parents=True)
+ (home / "state_5.sqlite").write_text("x")
+ r.keep_workspaces = True
+ r._prune_agent_home(str(run_dir))
+ assert (home / "state_5.sqlite").exists()
diff --git a/experiments/tests/test_copilot_webbench.py b/experiments/tests/test_copilot_webbench.py
new file mode 100644
index 0000000..415f5a8
--- /dev/null
+++ b/experiments/tests/test_copilot_webbench.py
@@ -0,0 +1,105 @@
+"""The Copilot Web-Bench lane: what the AGENT's own shell can find.
+
+One thing has to be pinned here and nowhere else. Web-Bench runs the agent on the HOST, so a
+scaffold installs into a per-task bin dir rather than `/usr/local/bin`, and on this lane the
+binary has to be on the PATH of the agent's shell -- not merely of the install step. rtk's
+hook does not run commands itself: it rewrites `ls -la` into `rtk ls -la` and hands that back
+to the CLI to execute. A bin dir missing from the agent's PATH therefore does not degrade the
+arm quietly, it breaks every shell call with `rtk: command not found` and the agent flails
+against the failure at many times the baseline cost.
+
+That gap was real and invisible until 2026-09-13: while the Copilot rtk hook was registered
+through a channel rtk could not parse, nothing was ever rewritten, so no command ever needed
+`rtk` on PATH and the missing entry cost nothing. Fixing the channel made it load-bearing.
+"""
+import os
+import subprocess
+
+from te_harness import webbench as wb
+
+
+def _stub(tmp_path):
+ """A runner instance without construction: `__init__` probes a CLI and credentials, and
+ nothing under test here depends on either."""
+ r = object.__new__(wb.CopilotWebBenchRunner)
+ r.model, r.effort = "claude-haiku-4-5-20251001", "high"
+ r.copilot = "/usr/bin/true"
+ r.task_timeout = 5
+ r.scaffold = None
+ return r
+
+
+class _P:
+ """A Popen stand-in that records the environment it was handed."""
+
+ seen = {}
+
+ def __init__(self, cmd, **kw):
+ _P.seen = dict(kw.get("env") or {})
+ self.pid = 1
+ self.stdin = None
+ self.stdout = iter([])
+ self.stderr = type("R", (), {"read": lambda s: ""})()
+
+ def wait(self, timeout=None):
+ return 0
+
+ def kill(self):
+ pass
+
+
+def test_the_scaffold_bin_dir_is_on_the_agents_path(tmp_path, monkeypatch):
+ """The agent's own shell must find a scaffold's binary, because the rewritten command
+ the hook produces is run by that shell."""
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"
+ ws.mkdir(parents=True)
+ r = _stub(tmp_path)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+
+ run_dir = str(tmp_path / "run")
+ bin_dir = wb.WebBenchRunner.scaffold_bin_dir(r.agent_home(run_dir))
+ assert _P.seen["PATH"].split(os.pathsep)[0] == bin_dir, (
+ "the per-task scaffold bin dir must come FIRST on the agent's PATH; without it "
+ "rtk's rewritten `rtk <cmd>` dies `command not found` on every shell call")
+
+
+def test_copilot_home_is_isolated_and_matches_where_the_hooks_were_written(tmp_path,
+ monkeypatch):
+ """Both hook writers land in `COPILOT_HOME` -- rtk's `hooks/rtk-rewrite.json` and the net
+ guard's `settings.json`. The segment must export the SAME dir the installs wrote to, or
+ the agent looks for its hooks where none were registered, and neither may be the
+ operator's real `~/.copilot`."""
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"
+ ws.mkdir(parents=True)
+ r = _stub(tmp_path)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+
+ run_dir = str(tmp_path / "run")
+ assert _P.seen["COPILOT_HOME"] == r.scaffold_copilot_home(run_dir)
+ assert _P.seen["COPILOT_HOME"] == os.path.dirname(
+ r.net_guard_settings_path(r.agent_home(run_dir), run_dir))
+ assert os.path.expanduser("~/.copilot") not in _P.seen["COPILOT_HOME"]
+
+
+def test_byok_reaches_the_agent_as_configured(tmp_path, monkeypatch):
+ """Copilot's own BYOK provider is a supported lane: its settings reach the CLI's
+ environment unchanged, so the CLI bills the provider the run records."""
+ from te_harness import copilot
+ for var in copilot.PROVIDER_VARS + copilot.PROVIDER_SECRET_VARS:
+ monkeypatch.delenv(var, raising=False)
+ byok = {"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "COPILOT_PROVIDER_TYPE": "anthropic", "COPILOT_PROVIDER_API_KEY": "sk-ant-x"}
+ for k, v in byok.items():
+ monkeypatch.setenv(k, v)
+ monkeypatch.setattr(subprocess, "Popen", _P)
+ ws = tmp_path / "run" / "workspace"
+ ws.mkdir(parents=True)
+ r = _stub(tmp_path)
+ with open(tmp_path / "t.jsonl", "w") as fh:
+ r._segment(str(ws), "go", fh, [], 80, None)
+ for k, v in byok.items():
+ assert _P.seen[k] == v, k
diff --git a/experiments/tests/test_fleet_budget.py b/experiments/tests/test_fleet_budget.py
new file mode 100644
index 0000000..eb43308
--- /dev/null
+++ b/experiments/tests/test_fleet_budget.py
@@ -0,0 +1,94 @@
+"""The fleet CPU budget: one cap for every lane's containers, not one per lane.
+
+`TE_SWEBENCH_MAX_CPUS` caps ONE task container, so N concurrent lanes are an N x cap
+ceiling -- six lanes at the harness default would reserve 96 cores of a shared host. The
+budget is therefore written onto the cgroup that every rootless-docker container scope
+shares, where it binds lanes that are ALREADY RUNNING as well as future ones. These tests
+drive the shell against a fake cgroup tree, because the thing that can silently go wrong is
+writing the cap to the wrong directory -- which looks exactly like success.
+"""
+import os
+import subprocess
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_SH = os.path.join(_REPO, "scripts", "launch", "fleet_budget.sh")
+
+
+def _run(root, *args, env=None):
+ e = dict(os.environ, TE_CGROUP_ROOT=str(root))
+ e.update(env or {})
+ return subprocess.run(["bash", _SH, *args], capture_output=True, text=True, env=e)
+
+
+def _tree(tmp_path, *, scope=True, writable=True):
+ uid = os.getuid()
+ base = tmp_path / "user.slice" / f"user-{uid}.slice" / f"user@{uid}.service" / "user.slice"
+ base.mkdir(parents=True)
+ (base / "cpu.max").write_text("max 100000\n")
+ (base / "cpu.stat").write_text("nr_throttled 0\nthrottled_usec 0\n")
+ if scope:
+ (base / "docker-abc123.scope").mkdir()
+ if not writable:
+ os.chmod(base / "cpu.max", 0o444)
+ return base
+
+
+def test_the_cap_lands_on_the_cgroup_the_container_scopes_share(tmp_path):
+ """The scope's PARENT is the target: writing the cap on the scope itself would bind one
+ container and expire with it, which is indistinguishable from success in the log."""
+ base = _tree(tmp_path)
+ r = _run(tmp_path, "16")
+ assert r.returncode == 0, r.stderr
+ assert (base / "cpu.max").read_text().strip() == "1600000 100000"
+ assert "capped at 16 cores" in r.stdout
+ # the scope's own cpu.max is untouched -- there is none, and none is created
+ assert not (base / "docker-abc123.scope" / "cpu.max").exists()
+
+
+def test_the_canonical_path_is_the_fallback_when_no_container_is_running(tmp_path):
+ """A launch must be able to set the budget BEFORE the first container exists."""
+ base = _tree(tmp_path, scope=False)
+ r = _run(tmp_path, "8")
+ assert r.returncode == 0, r.stderr
+ assert (base / "cpu.max").read_text().strip() == "800000 100000"
+
+
+def test_a_report_reads_the_budget_back(tmp_path):
+ _tree(tmp_path)
+ assert "UNCAPPED" in _run(tmp_path).stdout
+ _run(tmp_path, "16")
+ assert "CPU budget: 16 cores" in _run(tmp_path).stdout
+
+
+def test_zero_leaves_the_cgroup_alone(tmp_path):
+ """An operator who wants the old unlimited behaviour must get it exactly, not a huge
+ number that still throttles."""
+ base = _tree(tmp_path)
+ r = _run(tmp_path, "0")
+ assert r.returncode == 0 and "not enforced" in r.stdout
+ assert (base / "cpu.max").read_text().strip() == "max 100000"
+
+
+def test_an_unwritable_cgroup_warns_and_never_fails_the_launch(tmp_path):
+ """cgroup v2 need not delegate `cpu` to the user manager. When it does not, the lane
+ still has to run -- but it must SAY the budget is not in force rather than imply one."""
+ _tree(tmp_path, writable=False)
+ r = _run(tmp_path, "16")
+ assert r.returncode == 0
+ assert "NOT in force" in r.stderr
+
+
+def test_a_missing_cgroup_tree_warns_and_never_fails_the_launch(tmp_path):
+ r = _run(tmp_path / "nothing-here", "16")
+ assert r.returncode == 0
+ assert "NOT in force" in r.stderr
+
+
+def test_the_codex_launcher_asserts_a_budget_and_a_per_container_cap():
+ """Both halves are needed and neither substitutes for the other: the cgroup cap bounds
+ the fleet, and the per-container cap is what stops joblib sizing a pool to the whole
+ host and thrashing inside that bound."""
+ src = open(os.path.join(_REPO, "scripts", "launch", "codex_ab.sh")).read()
+ assert "fleet_budget.sh" in src
+ assert 'te_fleet_cpu_cap "${TE_FLEET_CPUS:-16}"' in src
+ assert 'export TE_SWEBENCH_MAX_CPUS="${TE_SWEBENCH_MAX_CPUS:-4}"' in src
diff --git a/experiments/tests/test_grading_integrity.py b/experiments/tests/test_grading_integrity.py
new file mode 100644
index 0000000..f2ba453
--- /dev/null
+++ b/experiments/tests/test_grading_integrity.py
@@ -0,0 +1,234 @@
+"""A row must never claim a grade the eval script did not produce.
+
+Two defects sat behind one symptom here, and both were silent:
+
+ * The task container was named `te-<instance>-<arm>`, carrying no agent, model or seed,
+ while both SWE-bench runners open with `docker rm -f <name>`. Two lanes on the same
+ (instance, arm) -- exactly what you get running the Claude and Copilot lanes for
+ the same arm at once -- resolved to one name, and the second lane deleted the first
+ one's container mid-grade.
+ * `_grade` then wrote an empty `eval.log` and handed it to `get_eval_report`, which
+ scored the absent APPLY_PATCH_PASS marker as `patch_successfully_applied: false,
+ resolved: false`. The lane exited 0. The row was indistinguishable from a model that
+ tried the task and missed.
+
+So the container name must be unique per task-run, and a grading exec that did not run
+must be reported as an absence rather than a miss -- and excluded from the denominator of
+any resolve rate, because a task nobody evaluated is not a task the agent failed.
+"""
+import importlib.util
+import os
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _load(name, path):
+ spec = importlib.util.spec_from_file_location(name, path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+AGG = _load("_aggregate", os.path.join(_REPO, "scripts", "aggregate.py"))
+RX = _load("_run_experiment_gi", os.path.join(_REPO, "scripts", "run_experiment.py"))
+
+
+class _NameOnly:
+ """Just enough of a runner to exercise the naming rule without Docker or a dataset."""
+
+ def __init__(self, agent, model):
+ self.agent = agent
+ self.model = model
+
+ _container_name = None # bound below from the real implementation
+
+
+def _namer():
+ import te_harness.swebench as sw
+ _NameOnly._container_name = sw.ClaudeSwebenchRunner._container_name
+ return _NameOnly
+
+
+# ---- container naming ----------------------------------------------------------------
+
+def test_container_name_differs_by_agent():
+ """A Claude lane and a Copilot lane must not share a container name."""
+ R = _namer()
+ a = R("claude", "claude-opus-4.8")._container_name(
+ "django__django-10097", "baseline", "/r/claude/baseline/django_s0/x")
+ b = R("copilot", "claude-opus-4.8")._container_name(
+ "django__django-10097", "baseline", "/r/copilot/baseline/django_s0/x")
+ assert a != b, f"agent must change the container name; both were {a!r}"
+
+
+def test_container_name_differs_by_run_dir():
+ """Seed and timestamp live in the run dir, so distinct task-runs get distinct names."""
+ R = _namer()
+ mk = R("claude", "claude-opus-4.8")._container_name
+ s0 = mk("sympy__sympy-11618", "trimech", "/r/t/trimech/sympy_s0/20260101T000000Z__000__x")
+ s1 = mk("sympy__sympy-11618", "trimech", "/r/t/trimech/sympy_s1/20260101T000000Z__000__x")
+ assert s0 != s1
+
+
+def test_container_name_is_docker_legal_and_readable():
+ R = _namer()
+ n = R("claude", "claude-opus-4.8")._container_name(
+ "scikit-learn__scikit-learn-25102", "trimech-repomap", "/r/t/a/b/c")
+ assert len(n) <= 63, f"docker caps names at 63 chars; got {len(n)}"
+ assert n[0].isalnum(), n
+ assert all(c.isalnum() or c in "_.-" for c in n), n
+ # The readable stem survives, so `docker ps` still identifies the cell.
+ assert n.startswith("te-scikit-learn-scikit-learn-25102-trimech-repomap"[:50]), n
+
+
+def test_container_name_is_stable_for_one_task_run():
+ """Same inputs -> same name, so cleanup and the grading exec address one container."""
+ R = _namer()
+ mk = R("claude", "claude-opus-4.8")._container_name
+ args = ("django__django-10097", "baseline", "/r/t/baseline/django_s0/run")
+ assert mk(*args) == mk(*args)
+
+
+# ---- not_graded accounting ------------------------------------------------------------
+
+def test_eval_exec_failure_is_flagged_not_graded():
+ ungraded, reason = AGG.not_graded(
+ {"resolved": False,
+ "eval_report": {"eval_exec_failed": True, "eval_exec_rc": 1,
+ "eval_exec_error": "No such container"}})
+ assert ungraded is True
+ assert "No such container" in reason
+
+
+def test_grading_error_is_flagged_not_graded():
+ ungraded, reason = AGG.not_graded(
+ {"resolved": False, "eval_report": {"grading_error": "KeyError"}})
+ assert ungraded is True
+ assert "KeyError" in reason
+
+
+def test_a_real_miss_is_not_flagged():
+ """A graded-and-missed row must stay in the denominator -- that is a real result."""
+ ungraded, _ = AGG.not_graded(
+ {"resolved": False,
+ "eval_report": {"patch_exists": True, "patch_successfully_applied": True,
+ "resolved": False}})
+ assert ungraded is False
+
+
+def test_a_resolve_is_not_flagged():
+ ungraded, _ = AGG.not_graded({"resolved": True, "eval_report": {"resolved": True}})
+ assert ungraded is False
+
+
+def test_not_graded_rows_leave_the_resolve_denominator():
+ """The whole point: an ungraded row must not depress a cell's resolve rate."""
+ rows = [
+ {"arm": "trimech", "repo": "django", "seed": 0, "resolved": True,
+ "is_error": False, "num_turns": 5, "cost_effective_tokens": 100.0,
+ "not_graded": False},
+ {"arm": "trimech", "repo": "django", "seed": 0, "resolved": False,
+ "is_error": False, "num_turns": 4, "cost_effective_tokens": 100.0,
+ "not_graded": True},
+ ]
+ a = AGG.summarize(rows)[("trimech", "django", "0")]
+ assert a["n"] == 2, "the row still exists and still cost money"
+ assert a["graded"] == 1, "only one row was actually graded"
+ assert a["resolved"] == 1
+ assert a["ungraded"] == 1
+ assert a["resolved"] / a["graded"] == 1.0, (
+ "a lost container must not read as a failed task")
+
+
+# ---- mechanism overhead must survive into row.json ------------------------------------
+
+def test_every_overhead_key_is_summed_by_the_ledger():
+ """OVERHEAD_KEYS is the contract; ledger.overhead must honour all of it."""
+ from tokeneconomizer import ledger
+ row = {k: 100 for k in ledger.OVERHEAD_KEYS}
+ assert ledger.overhead(row) == 100 * len(ledger.OVERHEAD_KEYS)
+
+
+def test_stream_only_overhead_is_copied_into_the_row():
+ """The driver must lift overhead a mechanism reported on `srow` into `row`.
+
+ `tooling.after_task` writes the per-task tool judge's spend to the STREAM row only,
+ and aggregate.py reads nothing but row.json -- so untool-only and every trimech
+ variant had that line item silently dropped from cost_effective_tokens. Measured on a
+ 6-task sympy cell: 11,244 tokens, 4.2% of the cell's reported cost, and one-directional
+ because a baseline runs no judge.
+ """
+ from tokeneconomizer import ledger
+ row = RX.lift_overhead({"tokens_total": 1000},
+ {"untool_judge_tokens": 5171, "resolved": True})
+ assert row.get("untool_judge_tokens") == 5171
+ assert ledger.overhead(row) == 5171, "the judge's spend must reach the cost basis"
+
+
+def test_the_copy_never_clobbers_a_value_the_runner_already_set():
+ """spiral_judge_tokens is set on `row` by the harness; srow may hold a stale zero."""
+ row = RX.lift_overhead({"spiral_judge_tokens": 106916}, {"spiral_judge_tokens": 0})
+ assert row["spiral_judge_tokens"] == 106916
+
+
+def test_lift_overhead_covers_every_declared_overhead_key():
+ """Keyed off OVERHEAD_KEYS, so a new overhead field cannot reintroduce the gap."""
+ from tokeneconomizer import ledger
+ srow = {k: 7 for k in ledger.OVERHEAD_KEYS}
+ row = RX.lift_overhead({}, srow)
+ assert ledger.overhead(row) == 7 * len(ledger.OVERHEAD_KEYS)
+
+
+def test_lift_overhead_ignores_non_overhead_stream_fields():
+ """It must not smuggle arbitrary stream telemetry into row.json."""
+ row = RX.lift_overhead({}, {"untool_judge_dropped": ["WebFetch"], "resolved": True})
+ assert "untool_judge_dropped" not in row
+ assert "resolved" not in row
+
+
+def test_a_grading_stall_flags_the_row_instead_of_ending_the_stream():
+ """Patch extraction gets the guard container start already has.
+
+ `_graded_diff` is a `docker exec ... git diff` under a 300 s cap. On 2026-09-13 one
+ stalled at task 57 of a 60-task baseline and the TimeoutExpired propagated out of
+ `run_task`, ending the cell with four tasks unrun after 65 minutes of paid work -- the
+ exact failure the container-start guard exists to prevent, one step later in the same
+ function. The agent's turn is already bought by then, so the row must survive, say what
+ happened, and let the stream continue.
+ """
+ import inspect, re
+ from te_harness import swebench
+ src = inspect.getsource(swebench.ClaudeSwebenchRunner.run_task)
+ body = src[src.index("patch, resolved, report"):]
+ guarded = body[:body.index("row.update(self._row_from_segment")]
+ assert "self._graded_diff(name)" in guarded and "self._grade(" in guarded
+ assert re.search(r"except \(subprocess\.TimeoutExpired, RuntimeError\)", guarded)
+ # the row is kept, flagged, and filterable -- not silently scored as a clean miss
+ for field in ('row["is_error"] = True', '"grading_failed"', 'row["grading_error"]'):
+ assert field in guarded, field
+ # and the failure is logged, so a filtered-out row can be traced back to a lane log
+ assert "GRADING FAILED" in guarded
+
+
+# ---- the project env the agent shares with the grader ---------------------------------
+# The official images install the instance's dependencies into the `testbed` conda env the
+# grader runs in; the agent's shells get that env by a PATH prepend (swebench
+# `_activate_project_env`), so the agent tests its fix against what the grader will use.
+
+def test_choose_project_env_prefers_testbed_then_the_only_env():
+ from te_harness.swebench import choose_project_env
+ envs = "/opt/miniconda3/envs"
+ assert choose_project_env([f"{envs}/other/bin", f"{envs}/testbed/bin"]) == \
+ f"{envs}/testbed/bin"
+ assert choose_project_env([f"{envs}/py39/bin"]) == f"{envs}/py39/bin"
+ assert choose_project_env([f"{envs}/a/bin", f"{envs}/b/bin"]) is None
+ assert choose_project_env([]) is None
+
+
+def test_activation_env_prepends_and_names_the_env():
+ from te_harness.swebench import activation_env
+ env = activation_env("/opt/miniconda3/envs/testbed/bin", "/opt/miniconda3/bin:/usr/bin")
+ assert env["PATH"] == "/opt/miniconda3/envs/testbed/bin:/opt/miniconda3/bin:/usr/bin"
+ assert env["CONDA_PREFIX"] == "/opt/miniconda3/envs/testbed"
+ assert env["CONDA_DEFAULT_ENV"] == "testbed"
+ assert activation_env("/x/envs/e/bin", "")["PATH"] == "/x/envs/e/bin"
diff --git a/experiments/tests/test_model_spelling.py b/experiments/tests/test_model_spelling.py
new file mode 100644
index 0000000..bd91187
--- /dev/null
+++ b/experiments/tests/test_model_spelling.py
@@ -0,0 +1,156 @@
+"""One model, two spellings, and the launcher spells it for the backend.
+
+The Copilot CLI's own catalog lists the older Opus as `claude-opus-4.8` (63 healthy stored
+subscription rows); the Anthropic API serves `claude-opus-4-8` and 404s the dotted form.
+The docs name the model once, dashed; `te_model_for_agent` picks the served spelling per
+agent, and `te_check_copilot_model` verifies it against the CLI and switches to the other
+separator when the CLI has THAT one, instead of refusing the lane. Everything lands in the
+same `_opus48` tree.
+
+Shell functions, exercised through bash against a stub `copilot` binary: no network, no
+credentials, no real CLI.
+"""
+import os
+import stat
+import subprocess
+
+import pytest
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ENV_SH = os.path.join(_REPO, "scripts", "launch", "_env.sh")
+
+
+def _sh(script, env=None):
+ base = {"PATH": os.environ.get("PATH", "/usr/bin:/bin"), "HOME": os.environ.get("HOME", "/"),
+ "REPO": _REPO, "TOKENECONOMIZER_ENV_FILE": "/dev/null"}
+ base.update(env or {})
+ return subprocess.run(["bash", "-uc", f'. "$REPO/scripts/launch/_env.sh"; {script}'],
+ capture_output=True, text=True, env=base)
+
+
+# ---- the pure helpers ------------------------------------------------------------------
+
[email protected]("model,alt", [
+ ("claude-opus-4-8", "claude-opus-4.8"),
+ ("claude-opus-4.8", "claude-opus-4-8"),
+ ("claude-haiku-4-5", "claude-haiku-4.5"),
+ ("claude-haiku-4.5", "claude-haiku-4-5"),
+ ("claude-opus-5", ""),
+ ("claude-sonnet-5", ""),
+ ("auto", ""),
+])
+def test_alt_spelling_swaps_the_version_separator(model, alt):
+ p = _sh(f'te_model_alt_spelling "{model}"')
+ assert p.returncode == 0, p.stderr
+ assert p.stdout.strip() == alt
+
+
[email protected]("agent,byok,model,expected", [
+ # the Copilot CLI on a subscription spells the older Opus with a dot
+ ("copilot", "", "claude-opus-4-8", "claude-opus-4.8"),
+ ("copilot", "", "claude-opus-4.8", "claude-opus-4.8"),
+ # under BYOK it speaks the Anthropic wire API, which takes the dash
+ ("copilot", "https://api.anthropic.com", "claude-opus-4.8", "claude-opus-4-8"),
+ ("copilot", "https://api.anthropic.com", "claude-opus-4-8", "claude-opus-4-8"),
+ # Claude Code bills the Anthropic API, which takes the dash
+ ("claude", "", "claude-opus-4.8", "claude-opus-4-8"),
+ ("claude", "", "claude-opus-4-8", "claude-opus-4-8"),
+ ("claude", "", "claude-haiku-4.5", "claude-haiku-4-5"),
+ # one-spelling ids and non-Claude ids pass through
+ ("copilot", "", "claude-opus-5", "claude-opus-5"),
+ ("claude", "", "claude-sonnet-5", "claude-sonnet-5"),
+ ("codex", "", "gpt-5.6-sol", "gpt-5.6-sol"),
+ ("copilot", "", "auto", "auto"),
+])
+def test_model_for_agent_picks_the_served_spelling(agent, byok, model, expected):
+ env = {"COPILOT_PROVIDER_BASE_URL": byok} if byok else {}
+ p = _sh(f'te_model_for_agent "{agent}" "{model}"', env)
+ assert p.returncode == 0, p.stderr
+ assert p.stdout.strip() == expected
+ # Only an actual change is announced.
+ assert ("model id:" in p.stderr) is (expected != model)
+
+
+def test_model_for_agent_is_verbatim_under_strict():
+ p = _sh('te_model_for_agent copilot claude-opus-4-8; te_model_for_agent claude claude-opus-4.8',
+ {"TE_STRICT_MODEL_ID": "1"})
+ assert p.stdout.split() == ["claude-opus-4-8", "claude-opus-4.8"] and "model id:" not in p.stderr
+
+
+def test_model_for_agent_never_changes_the_tree_tag():
+ p = _sh('te_model_tag "$(te_model_for_agent copilot claude-opus-4-8 2>/dev/null)"; '
+ 'te_model_tag "$(te_model_for_agent claude claude-opus-4.8 2>/dev/null)"')
+ assert p.stdout.split() == ["opus48", "opus48"]
+
+
+# ---- the Copilot probe against a stub CLI ----------------------------------------------
+
+_STUB = r'''#!/usr/bin/env bash
+# A stand-in for `copilot -p ... --model <id>`: answers the way the real CLI does for an
+# id its subscription catalog lists (ok), one it does not ("is not available"), and one a
+# BYOK provider 404s. Records every model it was asked for.
+m=""
+for ((i = 1; i <= $#; i++)); do
+ if [[ "${!i}" == "--model" ]]; then j=$((i + 1)); m="${!j}"; fi
+done
+echo "$m" >> "$STUB_LOG"
+case "$m" in
+ claude-opus-4.8|claude-sonnet-5|claude-opus-5) echo "ok" ;;
+ claude-opus-4-8) echo "Model 'claude-opus-4-8' is not available. Available models: ..." ;;
+ *) echo "Model '$m' not found on provider at https://api.anthropic.com (HTTP 404)." ;;
+esac
+'''
+
+
+def stub_copilot(tmp_path):
+ bin_ = tmp_path / "copilot"
+ bin_.write_text(_STUB)
+ bin_.chmod(bin_.stat().st_mode | stat.S_IEXEC)
+ log = tmp_path / "asked.log"
+ log.write_text("")
+ return {"COPILOT_HOST_BIN": str(bin_), "STUB_LOG": str(log)}, log
+
+
+def test_copilot_model_check_switches_to_the_cli_spelling(stub_copilot):
+ env, log = stub_copilot
+ p = _sh('MODEL=claude-opus-4-8; te_check_copilot_model "$MODEL" high; echo "MODEL=$MODEL"', env)
+ assert p.returncode == 0, p.stderr
+ assert "MODEL=claude-opus-4.8" in p.stdout
+ assert "copilot model 'claude-opus-4.8' serves; effort=high accepted" in p.stdout
+ assert "does not serve 'claude-opus-4-8' but DOES serve 'claude-opus-4.8'" in p.stderr
+ assert log.read_text().split() == ["claude-opus-4-8", "claude-opus-4.8"] # one retry
+
+
+def test_copilot_model_check_is_silent_on_a_served_id(stub_copilot):
+ env, log = stub_copilot
+ p = _sh('te_check_copilot_model claude-sonnet-5 high', env)
+ assert p.returncode == 0 and "!!" not in p.stderr
+ assert log.read_text().split() == ["claude-sonnet-5"]
+
+
+def test_copilot_model_check_no_longer_false_passes_a_byok_404(stub_copilot):
+ """The check used to grep only the subscription's 'is not available'; a BYOK provider
+ says 'not found on provider ... (HTTP 404)', and the launch banner printed `serves`
+ for an id every task then 404'd on (measured 2026-09-12)."""
+ env, _ = stub_copilot
+ env = dict(env, COPILOT_PROVIDER_BASE_URL="https://api.anthropic.com")
+ p = _sh('te_check_copilot_model claude-sonnet-9 high', env)
+ assert p.returncode == 2
+ assert "not served by the BYOK provider" in p.stderr
+ assert "serves" not in p.stdout
+
+
+def test_copilot_model_check_refuses_when_neither_spelling_serves(stub_copilot):
+ env, log = stub_copilot
+ p = _sh('te_check_copilot_model claude-haiku-4.5 high', env)
+ assert p.returncode == 2
+ assert "is not available on this subscription" in p.stderr
+ assert log.read_text().split() == ["claude-haiku-4.5", "claude-haiku-4-5"]
+
+
+def test_copilot_model_check_can_be_made_strict(stub_copilot):
+ env, log = stub_copilot
+ p = _sh('te_check_copilot_model claude-opus-4-8 high', dict(env, TE_STRICT_MODEL_ID="1"))
+ assert p.returncode == 2
+ assert log.read_text().split() == ["claude-opus-4-8"] # no retry
diff --git a/experiments/tests/test_optimizer_basis.py b/experiments/tests/test_optimizer_basis.py
new file mode 100644
index 0000000..c538ca4
--- /dev/null
+++ b/experiments/tests/test_optimizer_basis.py
@@ -0,0 +1,355 @@
+"""The between-task optimizer runs on the agent's own model, in every harness.
+
+The mechanisms make paid LLM calls between tasks -- the injection proposer, the tool
+judge, the spiral judge, the nudge writer -- and `ledger.overhead` charges them to the
+same cost basis as the task rows, against a baseline that made no such calls. So the
+model those calls run on is not a tuning knob: a cheaper proposer would understate the
+overhead an economized arm actually incurs, and the arm would look better for having
+billed its own thinking at a discount. It is also what the cell name claims, since a cell
+names one model.
+
+These are SOURCE-level assertions rather than constructed runners: building a real one
+needs a CLI binary, a grading table and Docker, none of which an offline test has. The
+thing worth guarding is the resolution rule itself, and that is visible in the source.
+"""
+import ast
+import inspect
+import os
+import json
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_HARNESSES = ("swebench.py", "webbench.py")
+
+
+def _tree(fname):
+ with open(os.path.join(_REPO, "te_harness", fname), encoding="utf-8") as fh:
+ return ast.parse(fh.read())
+
+
+def _runner_classes(tree):
+ return [n for n in ast.walk(tree)
+ if isinstance(n, ast.ClassDef) and n.name.endswith("Runner")]
+
+
+def _self_optimizer_assignments(node):
+ """Every `self.optimizer_model = <value>` in this subtree, as value nodes."""
+ out = []
+ for n in ast.walk(node):
+ if not isinstance(n, ast.Assign):
+ continue
+ for t in n.targets:
+ if (isinstance(t, ast.Attribute) and t.attr == "optimizer_model"
+ and isinstance(t.value, ast.Name) and t.value.id == "self"):
+ out.append(n.value)
+ return out
+
+
+def _mentions_self_model(value):
+ return any(isinstance(n, ast.Attribute) and n.attr == "model"
+ and isinstance(n.value, ast.Name) and n.value.id == "self"
+ for n in ast.walk(value))
+
+
+def test_every_runner_takes_its_optimizer_model_from_its_own_model():
+ seen = 0
+ for fname in _HARNESSES:
+ for cls in _runner_classes(_tree(fname)):
+ for value in _self_optimizer_assignments(cls):
+ seen += 1
+ assert _mentions_self_model(value), (
+ f"{fname}:{cls.name} assigns optimizer_model from something other "
+ f"than self.model: {ast.dump(value)[:120]}")
+ assert seen >= 4, f"expected an assignment per runner, found {seen}"
+
+
+def test_no_runner_accepts_an_optimizer_model_argument():
+ """A caller that could pass one could split the ledger's cost basis in two."""
+ for fname in _HARNESSES:
+ for cls in _runner_classes(_tree(fname)):
+ for fn in cls.body:
+ if not (isinstance(fn, ast.FunctionDef) and fn.name == "__init__"):
+ continue
+ names = [a.arg for a in fn.args.args + fn.args.kwonlyargs]
+ assert "optimizer_model" not in names, f"{fname}:{cls.name}.__init__"
+
+
+def test_the_old_env_knob_is_refused_not_read():
+ """TE_OPTIMIZER_MODEL used to select it. Silently ignoring a variable someone set is
+ worse than refusing it, so every mention must be inside a raise."""
+ for fname in _HARNESSES:
+ tree = _tree(fname)
+ raised = {n for r in ast.walk(tree) if isinstance(r, ast.Raise)
+ for n in ast.walk(r)}
+ guards = {n for i in ast.walk(tree) if isinstance(i, ast.If)
+ for n in ast.walk(i.test)}
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Constant) and node.value == "TE_OPTIMIZER_MODEL":
+ assert node in guards, f"{fname}: TE_OPTIMIZER_MODEL read, not guarded"
+ assert any(isinstance(n, ast.Constant) and isinstance(n.value, str)
+ and "TE_OPTIMIZER_MODEL is no longer honoured" in n.value
+ for n in raised), f"{fname}: no refusal message"
+
+
+def test_the_driver_refuses_a_model_id_for_the_agent_label_flag():
+ """`--optimizer-model` is an agent label; a model id there is the one remaining way to
+ charge overhead to a model the arm was not measured on."""
+ path = os.path.join(_REPO, "scripts", "run_experiment.py")
+ with open(path, encoding="utf-8") as fh:
+ src = fh.read()
+ assert "looks like a model id" in src
+ assert "AGENT LABEL" in src
+
+
+# ---------------------------------------------------------------------------------------
+# The mechanism call's TRANSPORT (2026-09-07).
+#
+# The MODEL is fixed by the assertions above -- the agent's own, so overhead is priced on
+# the task rows' basis. The WIRE is not, and it was costing the Copilot lane an order of
+# magnitude: `copilot -p` is an agent session that carries the CLI's system prompt and (until
+# the --excluded-tools fix) its whole tool schema on every one-shot mechanism call. Measured
+# on one 2,423-char judge prompt: 22,460 billed prompt tokens through the CLI vs 1,078 over
+# the Messages API, which the Claude harness has used for these calls all along.
+#
+# So these guard the two things that must stay true of the new transport:
+# * it only ever points at an endpoint that bills WHAT THE TASKS BILL -- the lane's own
+# Anthropic-typed BYOK provider -- and
+# * on the GitHub subscription it never activates at all: the calls stay on the CLI.
+# ---------------------------------------------------------------------------------------
+import importlib
+
+
+def _endpoint(env):
+ copilot = importlib.import_module("te_harness.copilot")
+ return copilot.anthropic_optimizer_endpoint(dict(env))
+
+
+def test_byok_anthropic_resolves_to_the_provider_the_tasks_already_bill():
+ got = _endpoint({"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "COPILOT_PROVIDER_TYPE": "anthropic",
+ "COPILOT_PROVIDER_API_KEY": "sk-test"})
+ assert got == ("https://api.anthropic.com", "sk-test", "")
+
+
+def test_a_byok_bearer_is_carried_when_there_is_no_key():
+ got = _endpoint({"COPILOT_PROVIDER_BASE_URL": "https://llm.example.invalid",
+ "COPILOT_PROVIDER_TYPE": "anthropic",
+ "COPILOT_PROVIDER_BEARER_TOKEN": "bt-test"})
+ assert got == ("https://llm.example.invalid", "", "bt-test")
+
+
+def test_byok_needs_its_own_type_and_credential():
+ """Only COPILOT_PROVIDER_* configures BYOK: no other key fills its type or credential,
+ and resolving never writes into the environment it reads."""
+ env = {"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "ANTHROPIC_API_KEY": "sk-ant-x"}
+ assert _endpoint(env) is None
+ copilot = importlib.import_module("te_harness.copilot")
+ copilot.anthropic_optimizer_endpoint(env)
+ assert sorted(env) == ["ANTHROPIC_API_KEY", "COPILOT_PROVIDER_BASE_URL"]
+
+
+def test_a_non_anthropic_byok_provider_keeps_the_cli():
+ # It does not speak the Messages wire; there is nothing to send it.
+ assert _endpoint({"COPILOT_PROVIDER_BASE_URL": "https://api.openai.com",
+ "COPILOT_PROVIDER_TYPE": "openai",
+ "COPILOT_PROVIDER_API_KEY": "sk-test"}) is None
+
+
+def test_byok_without_a_credential_keeps_the_cli():
+ assert _endpoint({"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "COPILOT_PROVIDER_TYPE": "anthropic"}) is None
+
+
+def test_a_subscription_lane_always_keeps_the_cli():
+ """The subscription is reachable only through `copilot -p`, so a subscription lane's
+ mechanism calls stay on the CLI whatever else the environment carries."""
+ assert _endpoint({}) is None
+ assert _endpoint({"ANTHROPIC_API_KEY": "sk-ant-x",
+ "ANTHROPIC_BASE_URL": "https://example.invalid"}) is None
+
+
+def test_the_optimizer_call_excludes_tools_and_not_only_denies_them():
+ """--deny-tool withholds permission but leaves the schema in the prompt; only
+ --excluded-tools keeps it out. Passing just the first cost ~16,900 tokens of tool
+ schema on EVERY mechanism call, which the ledger charged to the arm."""
+ copilot = importlib.import_module("te_harness.copilot")
+ src = inspect.getsource(copilot.optimizer_call)
+ assert "--excluded-tools" in src
+ assert "--deny-tool" in src, "permission should still be withheld too"
+ assert set(copilot.OPTIMIZER_DENY_TOOLS) <= set(copilot.OPTIMIZER_EXCLUDE_TOOLS)
+
+
+def test_both_harnesses_branch_on_the_endpoint_and_record_the_transport():
+ """A transport that is not recorded is a silent basis change: rows made over the CLI
+ and rows made over the API are not comparable on overhead."""
+ for fname in _HARNESSES:
+ src = open(os.path.join(_REPO, "te_harness", fname), encoding="utf-8").read()
+ assert "anthropic_optimizer_endpoint()" in src, fname
+ assert '"optimizer_transport"' in src, fname
+
+
+# ---------------------------------------------------------------------------------------
+# WHO pays for the transport preflight (2026-09-07).
+#
+# The preflight exists because anthropic_optimizer_call never raises: a dead mechanism
+# endpoint would leave an economized arm running as a baseline wearing its name. But it
+# fired at every runner construction, including `baseline` -- which makes no mechanism call
+# at all, so it was billing a probe it never needed and would refuse a lane over an endpoint
+# it never touches. `config.needs_optimizer` is the gate, and these pin it to the facade so
+# the two cannot drift.
+# ---------------------------------------------------------------------------------------
+def test_needs_optimizer_agrees_with_the_facade_for_every_arm():
+ """The predicate must be true exactly when Economizer.__init__ would resolve a runner.
+
+ The facade's three conditions are the definition; this asserts the aggregate matches
+ them arm by arm, so adding an arm cannot silently skip a preflight it needs.
+ """
+ from tokeneconomizer.arms import ARMS
+ from tokeneconomizer.config import needs_optimizer
+ for name, arm in sorted(ARMS.items()):
+ cfg = arm.cfg()
+ facade_would_resolve = bool(
+ (cfg.injection.enabled and "distiller" in cfg.injection.profile)
+ or cfg.tooling.judge
+ or (cfg.spiral.guard and (cfg.spiral.judge or cfg.spiral.nudge_llm)))
+ assert needs_optimizer(cfg) is facade_would_resolve, name
+
+
+def test_arm_mode_is_not_the_test():
+ """Guards the trap that made this non-obvious: `spiral-only` and `untool-only` carry
+ mode `standalone` and DO make paid calls, so gating on mode would skip their
+ preflight and hand back exactly the silent-fallback rows it exists to prevent."""
+ from tokeneconomizer.arms import ARMS
+ from tokeneconomizer.config import needs_optimizer
+ for name in ("spiral-only", "untool-only"):
+ assert ARMS[name].mode == "standalone", name
+ assert needs_optimizer(ARMS[name].cfg()) is True, name
+ for name in ("baseline", "baseline-low", "baseline-medium"):
+ assert needs_optimizer(ARMS[name].cfg()) is False, name
+
+
+def test_the_driver_tells_the_harness_before_building_the_runner():
+ """Order matters: the signal has to be set BEFORE load_runner, because the harness
+ reads it in its constructor."""
+ src = open(os.path.join(_REPO, "scripts", "run_experiment.py"), encoding="utf-8").read()
+ signal = src.index('TE_OPTIMIZER_PREFLIGHT')
+ build = src.index('runner = load_runner(')
+ assert signal < build, "the preflight signal must be exported before the runner is built"
+ assert "needs_optimizer(config)" in src
+
+
+def test_both_harnesses_gate_the_preflight_and_default_to_checking():
+ """Unset must mean 'assume it might' -- a direct caller of make_runner keeps the
+ check, so the gate can only ever be relaxed deliberately by the driver. The gate lives
+ in the two shared probes (the Messages one in swebench, the Responses one in codex),
+ and every runner of both domains goes through one of them."""
+ gate = 'os.environ.get("TE_OPTIMIZER_PREFLIGHT", "1") == "0"'
+ for fname in ("swebench.py", "codex.py"):
+ src = open(os.path.join(_REPO, "te_harness", fname), encoding="utf-8").read()
+ assert gate in src, fname
+ for fname in _HARNESSES:
+ tree = _tree(fname)
+ for cls in _runner_classes(tree):
+ fn = next((m for m in cls.body if isinstance(m, ast.FunctionDef)
+ and m.name == "_preflight_optimizer_endpoint"), None)
+ if fn is None:
+ continue
+ calls = {getattr(n.func, "attr", getattr(n.func, "id", None))
+ for n in ast.walk(fn) if isinstance(n, ast.Call)}
+ assert calls & {"preflight_optimizer_endpoint", "preflight_optimizer"}, (
+ f"{fname}:{cls.name} probes without the shared, gated helper")
+
+
+def test_every_runner_caps_the_optimizer_call_at_the_same_number():
+ """One cap across both agents and both domains.
+
+ The Web-Bench Copilot runner sat at 8192 while the other three were 16384, which is a
+ hazard rather than an inconsistency: a cap too low truncates a judge's per-tool JSON
+ into an unparseable answer, and the mechanisms read that as "no verdict" while the call
+ is still billed. So the cell pays for a judge whose verdict was thrown away, and only
+ the truncation log says so. Pin them equal.
+ """
+ seen = {}
+ for fname in _HARNESSES:
+ tree = _tree(fname)
+ for node in ast.walk(tree):
+ if not (isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "get"):
+ continue
+ args = node.args
+ if (len(args) == 2 and isinstance(args[0], ast.Constant)
+ and args[0].value == "TE_OPTIMIZER_MAX_TOKENS"
+ and isinstance(args[1], ast.Constant)):
+ seen.setdefault(args[1].value, []).append(fname)
+ assert seen, "no TE_OPTIMIZER_MAX_TOKENS default found -- did the call shape change?"
+ assert list(seen) == ["16384"], f"runners disagree on the optimizer cap: {seen}"
+
+
+# --------------------------------------------------------------------------------------
+# WHERE the mechanism call goes, and WHICH CREDENTIAL it sends (2026-09-07).
+#
+# Exactly one header, because anthropic_optimizer_call never raises: a request the endpoint
+# refuses arrives as ("", 0.0) == "no verdict", and the arm quietly runs its LLM-free path
+# while the cell grades and bills like any other.
+# --------------------------------------------------------------------------------------
+def _request_for(monkeypatch, **kw):
+ """Capture the request anthropic_optimizer_call would send, without a network call."""
+ from te_harness import swebench as S
+ seen = {}
+
+ class _Resp:
+ def __enter__(self): return self
+ def __exit__(self, *a): return False
+ def read(self):
+ return json.dumps({"content": [{"type": "text", "text": "ok"}],
+ "usage": {"input_tokens": 1, "output_tokens": 1}}).encode()
+
+ def _fake_urlopen(req, timeout=None):
+ seen["url"] = req.full_url
+ seen["headers"] = {k.lower(): v for k, v in req.header_items()}
+ return _Resp()
+
+ monkeypatch.setattr(S.urllib.request, "urlopen", _fake_urlopen)
+ S.anthropic_optimizer_call("hi", model="agent", configured_model="claude-opus-5",
+ max_tokens=16, **kw)
+ return seen
+
+
+def test_the_claude_lane_calls_the_anthropic_api_on_its_key(monkeypatch, tmp_path):
+ monkeypatch.setenv("TE_OPTIMIZER_LOG", str(tmp_path / "log.jsonl"))
+ got = _request_for(monkeypatch, api_key="sk-ant-x")
+ assert got["url"] == "https://api.anthropic.com/v1/messages"
+ assert got["headers"]["x-api-key"] == "sk-ant-x"
+ assert "authorization" not in got["headers"]
+
+
+def test_a_byok_provider_gets_its_own_url_and_one_credential(monkeypatch, tmp_path):
+ """The one other caller: a Copilot lane on an Anthropic-typed BYOK provider."""
+ monkeypatch.setenv("TE_OPTIMIZER_LOG", str(tmp_path / "log.jsonl"))
+ got = _request_for(monkeypatch, base_url="https://llm.example.invalid/", bearer="bt")
+ assert got["url"] == "https://llm.example.invalid/v1/messages"
+ assert got["headers"]["authorization"] == "Bearer bt"
+ assert "x-api-key" not in got["headers"]
+ both = _request_for(monkeypatch, base_url="https://llm.example.invalid",
+ api_key="sk-ant-x", bearer="bt")
+ assert both["headers"]["x-api-key"] == "sk-ant-x"
+ assert "authorization" not in both["headers"], "never both headers at once"
+
+
+def test_every_runner_preflights_the_mechanism_endpoint():
+ """All FOUR runners, not just the Copilot pair. The two Claude runners had no probe,
+ which is why a 401 could bill 83 rows before the ledger showed overhead of zero."""
+ for fname, classes in (("swebench.py", ("ClaudeSwebenchRunner", "CopilotSwebenchRunner")),
+ ("webbench.py", ("WebBenchRunner", "CopilotWebBenchRunner"))):
+ tree = _tree(fname)
+ for cls in classes:
+ node = next(n for n in ast.walk(tree)
+ if isinstance(n, ast.ClassDef) and n.name == cls)
+ methods = {m.name for m in node.body if isinstance(m, ast.FunctionDef)}
+ assert "_preflight_optimizer_endpoint" in methods, f"{fname}:{cls}"
+ init = next(m for m in node.body
+ if isinstance(m, ast.FunctionDef) and m.name == "__init__")
+ called = any(isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)
+ and n.func.attr == "_preflight_optimizer_endpoint"
+ for n in ast.walk(init))
+ assert called, f"{fname}:{cls}.__init__ never runs its preflight"
diff --git a/experiments/tests/test_routing.py b/experiments/tests/test_routing.py
new file mode 100644
index 0000000..a14f4de
--- /dev/null
+++ b/experiments/tests/test_routing.py
@@ -0,0 +1,113 @@
+"""Each agent bills its own vendor's API directly, decided from the environment alone.
+
+ claude the Anthropic API, on ANTHROPIC_API_KEY. ANTHROPIC_BASE_URL and
+ ANTHROPIC_AUTH_TOKEN are not inputs: a set base URL is reported and ignored, and
+ neither one stands in for the key.
+ copilot the GitHub subscription through the CLI's own auth, or the CLI's own BYOK
+ provider (COPILOT_PROVIDER_*), which is forwarded to the CLI as configured.
+ codex the OpenAI API, on OPENAI_API_KEY (test_codex_routing.py).
+
+These call the resolvers with an explicit env dict, so they need no network, no CLI binary
+and no credentials.
+"""
+import pytest
+
+from te_harness import copilot
+from te_harness import swebench as sw
+
+
+# ---- Claude: the Anthropic API, and nothing else ----------------------------------------
+def test_the_key_is_the_one_credential():
+ assert sw.resolve_anthropic_key({"ANTHROPIC_API_KEY": "sk-ant-x"}) == "sk-ant-x"
+
+
+ {},
+ {"ANTHROPIC_BASE_URL": "https://example.invalid", "ANTHROPIC_AUTH_TOKEN": "tok"},
+ {"ANTHROPIC_AUTH_TOKEN": "tok"},
+ {"OPENAI_API_KEY": "sk-test"},
+])
+def test_nothing_but_the_api_key_satisfies_the_claude_lane(env):
+ """No endpoint and no other credential substitutes for ANTHROPIC_API_KEY, and the lane
+ refuses at construction rather than once per task."""
+ with pytest.raises(RuntimeError, match="ANTHROPIC_API_KEY"):
+ sw.resolve_anthropic_key(dict(env))
+
+
+def test_a_set_base_url_is_reported_and_ignored(capsys, monkeypatch):
+ monkeypatch.setenv("TE_VERBOSE", "1")
+ env = {"ANTHROPIC_API_KEY": "sk-ant-x", "ANTHROPIC_BASE_URL": "https://example.invalid",
+ "ANTHROPIC_AUTH_TOKEN": "tok"}
+ assert sw.resolve_anthropic_key(env) == "sk-ant-x"
+ out = capsys.readouterr().out
+ assert out.count("ANTHROPIC_BASE_URL is set and ignored") == 1
+ assert "api.anthropic.com" in out
+ # resolving never rewrites the caller's environment
+ assert env["ANTHROPIC_BASE_URL"] == "https://example.invalid"
+
+
+def test_no_notice_when_no_base_url_is_set(capsys, monkeypatch):
+ monkeypatch.setenv("TE_VERBOSE", "1")
+ sw.resolve_anthropic_key({"ANTHROPIC_API_KEY": "sk-ant-x"})
+ assert "ignored" not in capsys.readouterr().out
+
+
+def test_the_ignored_set_covers_every_steering_variable():
+ """The variables that would move Claude Code to another endpoint, credential or model
+ catalog. The key itself is never among them."""
+ assert {"ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN"} <= set(sw.IGNORED_ANTHROPIC_VARS)
+ assert "ANTHROPIC_API_KEY" not in sw.IGNORED_ANTHROPIC_VARS
+
+
+def test_the_claude_runners_record_the_anthropic_api_as_their_endpoint():
+ """run_config.json -> routing.base_url, on both domains."""
+ import inspect
+ from te_harness import webbench as wb
+ for cls in (sw.ClaudeSwebenchRunner, wb.WebBenchRunner):
+ src = inspect.getsource(cls.__init__)
+ assert "self.routing_base = _ANTHROPIC_DIRECT" in src, cls.__name__
+ assert "resolve_anthropic_key()" in src, cls.__name__
+ assert sw._ANTHROPIC_DIRECT == "https://api.anthropic.com"
+
+
+# ---- Copilot: the CLI's own auth, or its own BYOK provider ------------------------------
+_BYOK = {"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "COPILOT_PROVIDER_TYPE": "anthropic", "COPILOT_PROVIDER_API_KEY": "sk-ant-x",
+ "COPILOT_PROVIDER_MODEL_ID": "claude-sonnet-5"}
+
+
+def _clear_byok(monkeypatch):
+ for var in copilot.PROVIDER_VARS + copilot.PROVIDER_SECRET_VARS:
+ monkeypatch.delenv(var, raising=False)
+
+
+def test_the_subscription_is_the_default(monkeypatch):
+ _clear_byok(monkeypatch)
+ assert copilot.byok_active() is False
+ assert copilot.provider_base() == "github-copilot"
+ assert copilot.provider_env() == {} and copilot.provider_secret_names() == []
+
+
+def test_byok_is_forwarded_to_the_cli_as_configured(monkeypatch):
+ """Copilot's own BYOK stays supported: its settings reach the CLI unchanged, the secret
+ by NAME only, and the run records the provider it billed."""
+ _clear_byok(monkeypatch)
+ for k, v in _BYOK.items():
+ monkeypatch.setenv(k, v)
+ assert copilot.byok_active() is True
+ assert copilot.provider_base() == "https://api.anthropic.com"
+ env = copilot.provider_env()
+ assert env == {k: v for k, v in _BYOK.items() if k != "COPILOT_PROVIDER_API_KEY"}
+ assert copilot.provider_secret_names() == ["COPILOT_PROVIDER_API_KEY"]
+ assert "sk-ant-x" not in copilot.run_env().values()
+
+
+def test_an_unrelated_key_never_fills_a_byok_slot(monkeypatch):
+ """BYOK is configured through COPILOT_PROVIDER_* alone: no other variable activates it
+ or supplies its type or credential."""
+ _clear_byok(monkeypatch)
+ monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-x")
+ assert copilot.byok_active() is False
+ monkeypatch.setenv("COPILOT_PROVIDER_BASE_URL", "https://example.invalid")
+ assert "COPILOT_PROVIDER_TYPE" not in copilot.provider_env()
+ assert copilot.provider_secret_names() == []
diff --git a/experiments/tests/test_runner_registry.py b/experiments/tests/test_runner_registry.py
new file mode 100644
index 0000000..97d6c53
--- /dev/null
+++ b/experiments/tests/test_runner_registry.py
@@ -0,0 +1,156 @@
+"""Every runner in a RUNNERS registry must accept what its make_runner forwards.
+
+This is the gap a two-lane blocker went through. `make_runner` fans out to a concrete
+runner class:
+
+ RUNNERS[agent](agent=agent, scaffold=scaffold, **kwargs) # te_harness.swebench
+
+and `CopilotSwebenchRunner.__init__` did not take `scaffold`, so every Copilot SWE-bench
+cell raised TypeError at construction -- before task 0, for every arm including `baseline`
+where the value is None. Two of the sweep's eight lanes could not run at all.
+
+`run_experiment.load_runner` already guards against exactly this, but it inspects the
+FACTORY's signature, and a factory declared `def make_runner(agent=None, **kwargs)` accepts
+anything by definition -- so the guard passes and the concrete class raises. The check has
+to look at the classes the registry actually dispatches to.
+
+Kept to `inspect`: no CLI, no Docker, no network, no credentials, so it runs anywhere the
+rest of the suite does. That is the point -- the whole failure was a signature mismatch, and
+the offline smoke test could not see it because RUNNER=stub never reaches these classes.
+"""
+import inspect
+
+import pytest
+
+
+def _registries():
+ """(module_name, RUNNERS, forwarded_kwargs) for each harness that has one."""
+ out = []
+ for mod_name, forwarded in (("te_harness.swebench", ("agent", "scaffold", "model")),
+ ("te_harness.webbench", ("agent", "model"))):
+ try:
+ mod = __import__(mod_name, fromlist=["RUNNERS"])
+ except Exception as exc: # harness extra not installed
+ pytest.skip(f"{mod_name} not importable: {exc}")
+ out.append((mod_name, mod.RUNNERS, forwarded))
+ return out
+
+
+def _accepts(func, name):
+ sig = inspect.signature(func)
+ if name in sig.parameters:
+ return True
+ return any(p.kind is inspect.Parameter.VAR_KEYWORD for p in sig.parameters.values())
+
+
+def test_every_runner_accepts_what_make_runner_forwards():
+ for mod_name, runners, forwarded in _registries():
+ assert runners, f"{mod_name}.RUNNERS is empty"
+ for agent, cls in sorted(runners.items()):
+ for kw in forwarded:
+ assert _accepts(cls.__init__, kw), (
+ f"{mod_name}.RUNNERS[{agent!r}] is {cls.__name__}, whose __init__ does "
+ f"not accept {kw!r} -- but make_runner forwards it, so every cell on "
+ f"this agent would raise TypeError at construction, before task 0.")
+
+
+def test_scaffold_bearing_arms_have_a_runner_that_takes_a_scaffold():
+ """An arm that names a scaffold must be constructible on every SWE-bench agent.
+
+ The scaffold arms are SWE-bench-only (SWEEP_ARMS_WEBBENCH_UNAVAILABLE), and a runner
+ that silently ignored `scaffold` would run NATIVE while filing rows under the
+ scaffold's name -- which is the failure `load_runner` refuses loudly for, and which
+ this keeps refusing for the classes rather than the factory.
+ """
+ import sys, os
+ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
+ import run_experiment as rx
+
+ scaffold_arms = [n for n, a in rx.ARMS.items() if getattr(a, "scaffold", None)]
+ assert scaffold_arms, "expected at least one scaffold-bearing arm in the registry"
+
+ mod = __import__("te_harness.swebench", fromlist=["RUNNERS"])
+ for agent, cls in sorted(mod.RUNNERS.items()):
+ assert _accepts(cls.__init__, "scaffold"), (
+ f"arms {scaffold_arms} carry a scaffold, but "
+ f"te_harness.swebench.RUNNERS[{agent!r}] ({cls.__name__}) cannot take one")
+
+
+def test_webbench_runners_can_take_a_scaffold_they_are_asked_to_host():
+ """Web-Bench hosts `rtk` (see te_harness.webbench.WEBBENCH_SCAFFOLDS), so both of its
+ runners must accept `scaffold` and be able to build a profile for it.
+
+ A runner that silently ignored `scaffold` would run NATIVE while filing rows under the
+ arm's name -- the same failure the SWE-bench check above exists for.
+ """
+ from te_harness.webbench import RUNNERS, WEBBENCH_SCAFFOLDS
+
+ assert WEBBENCH_SCAFFOLDS, "expected Web-Bench to host at least one baseline"
+ for agent, cls in sorted(RUNNERS.items()):
+ assert _accepts(cls.__init__, "scaffold"), (
+ f"te_harness.webbench.RUNNERS[{agent!r}] ({cls.__name__}) cannot take a "
+ f"scaffold, but {sorted(WEBBENCH_SCAFFOLDS)} are planned there")
+ assert hasattr(cls, "scaffold_profile_for"), (
+ f"{cls.__name__} takes a scaffold but declares no scaffold_profile_for(home); "
+ f"Web-Bench's HOME is per task-run, so the profile cannot be a class attribute")
+
+
+def test_webbench_refuses_a_scaffold_it_cannot_install():
+ """`repomap`/`serena` need a container or a Python symbol map. Asking for one must fail
+ LOUDLY at construction, not produce native rows wearing the baseline's name."""
+ import pytest as _pytest
+ from te_harness import webbench
+
+ for bad in ("repomap", "serena"):
+ with _pytest.raises(RuntimeError, match="not wired for Web-Bench"):
+ webbench.make_runner(agent="claude", scaffold=bad)
+
+
+def test_each_runner_declares_a_scaffold_profile():
+ """A SWE-bench runner that takes a scaffold needs the profile the scaffold installs against.
+
+ `scaffold_profile` is what makes an install agent-correct -- the instructions file,
+ the native navigation tool names, and the patch_excludes subtracted from the graded
+ diff. A runner accepting `scaffold=` without one would install a Claude-shaped scaffold
+ into a Copilot container.
+ """
+ mod = __import__("te_harness.swebench", fromlist=["RUNNERS"])
+ for agent, cls in sorted(mod.RUNNERS.items()):
+ profile = getattr(cls, "scaffold_profile", None)
+ assert profile is not None, (
+ f"{cls.__name__} accepts a scaffold but declares no scaffold_profile")
+ assert getattr(profile, "instructions_file", None), (
+ f"{cls.__name__}.scaffold_profile has no instructions_file")
+ assert getattr(profile, "patch_excludes", None), (
+ f"{cls.__name__}.scaffold_profile declares no patch_excludes, so scaffold "
+ f"artifacts would be graded as part of the agent's fix")
+
+
+def test_the_registries_carry_exactly_the_three_agent_labels():
+ """`claude`, `copilot`, `codex` -- one label per agent, each billing its own vendor's
+ API natively, and the same three on both domains. Codex is a first-class Web-Bench
+ agent too."""
+ from te_harness import swebench, webbench
+ assert sorted(swebench.RUNNERS) == sorted(webbench.RUNNERS) == ["claude", "codex",
+ "copilot"]
+ assert swebench.RUNNERS["codex"] is swebench.CodexSwebenchRunner
+ assert webbench.RUNNERS["codex"] is webbench.CodexWebBenchRunner
+ assert issubclass(webbench.CodexWebBenchRunner, webbench.WebBenchRunner)
+
+
[email protected]("label", ["claude-code", "copilot-cli", "openai", "nope"])
+def test_an_unregistered_label_is_refused_not_mapped(label, monkeypatch):
+ """A label with no entry is an error rather than a fallback: rows filed under the wrong
+ agent are wrong numbers, not a crash. Refused before any runner is built."""
+ from te_harness import swebench, webbench
+ monkeypatch.delenv("TE_SCAFFOLD", raising=False)
+ with pytest.raises(RuntimeError, match="no SWE-bench runner"):
+ swebench.make_runner(agent=label)
+ with pytest.raises(RuntimeError, match="no Web-Bench runner"):
+ webbench.make_runner(agent=label)
+
+
+def test_the_webbench_adapter_carries_a_node_vocabulary():
+ """The core's Web-Bench adapter describes a Node project, whichever agent runs it."""
+ from tokeneconomizer.adapters.webbench import WebBenchAdapter
+ assert WebBenchAdapter.env_vocabulary(None).name == "node-web"
diff --git a/experiments/tests/test_spiral.py b/experiments/tests/test_spiral.py
new file mode 100644
index 0000000..2e26e69
--- /dev/null
+++ b/experiments/tests/test_spiral.py
@@ -0,0 +1,284 @@
+"""Spiral: generic envelope guard + Codex-native post-green policy."""
+from tokeneconomizer.mechanisms.spiral import SpiralStage
+from tokeneconomizer.mechanisms import codex_spiral as cs
+from tokeneconomizer.config import SpiralConfig
+
+
+def test_shipped_codex_policy_flags():
+ pol = cs.shipped_policy(30)
+ assert pol.post_green_finalize_enabled is True
+ assert pol.post_green_finalize_require_cleared_failures is True
+ assert pol.wide_failure_enabled is False
+ assert pol.recovery_enabled is False
+ assert pol.min_tool_turns == 30 # at_threshold set
+
+
+def test_green_is_trustworthy():
+ assert cs.green_is_trustworthy("python -m pytest tests/test_x.py") is True
+ assert cs.green_is_trustworthy("pytest -k only_this") is False
+ assert cs.green_is_trustworthy("pytest --last-failed") is False
+
+
+def test_generic_guard_run_kwargs():
+ cfg = SpiralConfig(guard=True, judge=False, nudge=True)
+ stage = SpiralStage(cfg, max_turns=80, agent="claude")
+ rk = stage.run_kwargs({"instance_id": "w-1"})
+ assert rk["spiral_guard"] is True
+ assert "spiral_thr" in rk and "abort_envelope" in rk
+
+
+def test_codex_spiral_run_kwargs_policy():
+ cfg = SpiralConfig(codex_spiral=True)
+ stage = SpiralStage(cfg, max_turns=80, agent="codex")
+ rk = stage.run_kwargs({"instance_id": "x-1"})
+ assert "codex_spiral_policy" in rk # None until the tracker warms, but present
+
+
+def test_generic_guard_after_task_advances_envelope():
+ cfg = SpiralConfig(guard=True, nudge=True)
+ stage = SpiralStage(cfg, max_turns=80, agent="claude")
+ srow = {"resolved": True, "num_turns": 12}
+ stage.after_task({"instance_id": "w-1"},
+ {"resolved": True, "num_turns": 12, "aborted": False}, "/tmp", srow)
+ assert srow["aborted"] is False
+ assert "spiral_nudges" in srow
+
+
+def test_codex_spiral_after_task_telemetry(tmp_path):
+ cfg = SpiralConfig(codex_spiral=True)
+ stage = SpiralStage(cfg, max_turns=80, agent="codex")
+ srow = {"resolved": True, "num_turns": 30}
+ stage.after_task({"instance_id": "x-1"},
+ {"resolved": True, "num_turns": 30, "codex_spiral_nudges": 0},
+ str(tmp_path), srow)
+ assert srow["codex_spiral_active"] is True
+ assert srow["codex_spiral_profile"] == cs.PROFILE_VERSION
+
+
+# ---------------------------------------------------------------------------------------
+# Shell-applied patches (2026-09-12). On some model providers the CLI emits no
+# file_change items: every edit is `apply_patch <<'PATCH'` inside a command_execution. The
+# controller must see those as edits or it never registers a trustworthy green and never
+# fires.
+# ---------------------------------------------------------------------------------------
+import json as _json
+from tokeneconomizer.mechanisms import codex_spiral as _cs
+
+_PATCH_CMD = ("/bin/bash -lc \"apply_patch <<'PATCH'\n*** Begin Patch\n*** Update File: "
+ "django/core/validators.py\n@@\n-old\n+new\n*** End Patch\nPATCH\n\n"
+ "git diff --check\"")
+
+
+def _line(item):
+ return _json.dumps({"type": "item.completed", "item": item})
+
+
+def test_shell_applied_patch_is_an_edit_event_with_its_paths():
+ events = _cs.parse_events(_line({"type": "command_execution", "command": _PATCH_CMD,
+ "aggregated_output": "Success. Updated the following files:\nM django/core/validators.py",
+ "exit_code": 0}))
+ assert [e.kind for e in events] == ["edit", "command"]
+ assert events[0].paths == ("django/core/validators.py",)
+ # a patch that failed to apply is not an edit
+ failed = _cs.parse_events(_line({"type": "command_execution", "command": _PATCH_CMD,
+ "aggregated_output": "apply_patch: error: Failed to find context", "exit_code": 1}))
+ assert [e.kind for e in failed] == ["command"]
+ # an ordinary command stays one event; parse_event still returns the primary one
+ plain = _cs.parse_events(_line({"type": "command_execution", "command": "ls", "aggregated_output": "", "exit_code": 0}))
+ assert [e.kind for e in plain] == ["command"] and _cs.parse_event(_line({"type": "command_execution", "command": _PATCH_CMD, "aggregated_output": "Success. Updated", "exit_code": 0})).kind == "command"
+ # string exit codes (defensive) are normalised
+ assert _cs.parse_events(_line({"type": "command_execution", "command": "pytest -q", "aggregated_output": "1 passed", "exit_code": "0"}))[0].exit_code == 0
+
+
+def test_shipped_policy_can_fire_on_a_shell_patched_trajectory():
+ lines = [_json.dumps({"type": "thread.started", "thread_id": "t-1"})]
+ for i in range(4):
+ lines.append(_line({"type": "command_execution", "command": f"rg pattern{i} src", "aggregated_output": "x", "exit_code": 0}))
+ lines.append(_line({"type": "command_execution", "command": _PATCH_CMD, "aggregated_output": "Success. Updated", "exit_code": 0}))
+ lines.append(_line({"type": "command_execution", "command": "python -m pytest tests/test_validators.py -q", "aggregated_output": "3 passed", "exit_code": 0}))
+ for i in range(8):
+ lines.append(_line({"type": "command_execution", "command": f"git log -p -{i+1} -- django", "aggregated_output": "history", "exit_code": 0}))
+ decision, state = _cs.replay_events(_cs.iter_events(lines), _cs.shipped_policy(6))
+ assert state.last_edit_turn > 0 and state.first_trustworthy_green_turn > state.last_edit_turn
+ assert decision and decision["reason"] == "validated-patch-should-finalize"
+ # and the live guard sees the same trajectory the same way
+ guard = _cs.CodexResumeGuard(_cs.shipped_policy(6))
+ fired = [guard.monitor(l) for l in lines]
+ assert any(fired) and guard.continuation()["session_id"] == "t-1"
+
+
+def test_trajectory_seam_surfaces_shell_patches_as_apply_patch_calls(tmp_path):
+ from tokeneconomizer import trace
+ from tokeneconomizer.mechanisms.injection import edited_pyfiles
+ p = tmp_path / "t.jsonl"
+ p.write_text("\n".join([
+ _line({"type": "command_execution", "command": _PATCH_CMD, "aggregated_output": "Success. Updated", "exit_code": 0}),
+ _line({"type": "command_execution", "command": "pytest -q", "aggregated_output": "ok", "exit_code": 0}),
+ ]) + "\n")
+ calls = trace.load_trajectory(str(p))
+ assert [c[0] for c in calls] == ["apply_patch", "shell", "shell"]
+ assert calls[0][1]["paths"] == ["django/core/validators.py"] and calls[0][1]["via"] == "shell"
+ assert edited_pyfiles([(c[0], c[1], c[2]) for c in calls]) == {"validators.py"}
+
+
+# ---------------------------------------------------------------------------------------
+# The defensive profile (2026-09-12): finalize plus every deterministic spiral signature,
+# two DISTINCT same-thread nudges per task, never an abort, reasoning effort untouched.
+# ---------------------------------------------------------------------------------------
+def _cmd(command, exit_code=0, output=""):
+ return _line({"type": "command_execution", "command": command, "aggregated_output": output, "exit_code": exit_code})
+
+
+def test_green_reverification_loop_fires_only_after_repeated_greens_without_an_edit():
+ lines = [_json.dumps({"type": "thread.started", "thread_id": "t-2"})]
+ lines += [_cmd(f"rg x{i}") for i in range(6)]
+ lines.append(_cmd(_PATCH_CMD, 0, "Success. Updated"))
+ lines.append(_cmd("pytest tests/test_a.py -q", 0, "1 passed")) # the accept
+ for _ in range(2):
+ lines.append(_cmd("pytest tests/test_a.py -q", 0, "1 passed")) # two re-runs: not yet
+ decision, state = _cs.replay_events(_cs.iter_events(lines), _cs.defensive_policy(8))
+ assert decision is None or decision["reason"] != "green-reverification-loop"
+ lines.append(_cmd("pytest tests/test_a.py -q", 0, "1 passed")) # third re-run
+ decision, state = _cs.replay_events(_cs.iter_events(lines), _cs.defensive_policy(8))
+ assert state.green_reruns_since_edit == 3
+ # the post-green finalize (grace 3 after the accept) reaches this point first; both are
+ # finalize-class nudges and the guard allows one of each, so check the rule directly
+ # with finalize disabled:
+ from dataclasses import replace as _replace
+ d2, _ = _cs.replay_events(_cs.iter_events(lines), _replace(_cs.defensive_policy(8), post_green_finalize_enabled=False))
+ assert d2 and d2["reason"] == "green-reverification-loop"
+ # an edit resets the counter
+ lines.append(_cmd(_PATCH_CMD, 0, "Success. Updated"))
+ _, state = _cs.replay_events(_cs.iter_events(lines), _cs.defensive_policy(8))
+ assert state.green_reruns_since_edit == 0
+ # the shipped profile does not know this rule
+ d3, _ = _cs.replay_events(_cs.iter_events(lines[:-1]), _replace(_cs.shipped_policy(8), post_green_finalize_enabled=False))
+ assert d3 is None
+
+
+def test_runaway_guard_fires_at_twice_the_envelope_even_with_no_edit():
+ lines = [_json.dumps({"type": "thread.started", "thread_id": "t-3"})]
+ lines += [_cmd(f"rg needle{i} src") for i in range(40)] # localization forever
+ decision, _ = _cs.replay_events(_cs.iter_events(lines), _cs.defensive_policy(16))
+ assert decision and decision["reason"] == "beyond-twice-the-resolved-envelope"
+ assert decision["turn"] == 32 # 2 x 16, above the floor of 30
+ shipped, _ = _cs.replay_events(_cs.iter_events(lines), _cs.shipped_policy(16))
+ assert shipped is None # the shipped profile never fires here
+ # below the floor the multiplier does not apply
+ early, _ = _cs.replay_events(_cs.iter_events(lines[:25]), _cs.defensive_policy(8))
+ assert early is None
+
+
+def test_defensive_guard_spends_its_budget_on_distinct_reasons_only():
+ guard = _cs.CodexResumeGuard(_cs.defensive_policy(16))
+ lines = [_json.dumps({"type": "thread.started", "thread_id": "t-4"})]
+ lines += [_cmd(f"rg needle{i} src") for i in range(40)]
+ fired = [guard.monitor(l) for l in lines]
+ assert sum(fired) == 1 and guard.continuation()["prompt"].startswith("Automated process checkpoint")
+ # the same reason again does not fire, even with budget left
+ assert not any(guard.monitor(_cmd(f"rg more{i}")) for i in range(5))
+ assert guard.policy.nudge_budget == 2 and guard.nudges_done == 1
+ assert _cs.nudge_text("green-reverification-loop") and _cs.nudge_text("beyond-twice-the-resolved-envelope")
+
+
+def test_defensive_profile_is_selectable_and_unpaid(tmp_path):
+ from tokeneconomizer.arms import ARMS, ARM_ORDER
+ from tokeneconomizer.config import needs_optimizer, SpiralConfig
+ c = ARMS["codex-spiral-defensive"].cfg()
+ assert c.spiral.codex_spiral and c.spiral.codex_spiral_profile == "defensive"
+ assert needs_optimizer(c) is False and "codex-spiral-defensive" in ARM_ORDER
+ # Since 2026-09-12 the plain name carries it and the old name is an alias; the shipped
+ # post-green policy keeps its own arm.
+ assert ARMS["codex-spiral"].cfg().spiral == c.spiral
+ assert ARMS["codex-spiral-postgreen"].cfg().spiral.codex_spiral_profile == "post-green"
+ import pytest
+ with pytest.raises(ValueError):
+ SpiralConfig(codex_spiral=True, codex_spiral_profile="bogus")
+ from tokeneconomizer.mechanisms.spiral import SpiralStage
+ stage = SpiralStage(c.spiral, max_turns=80, agent="codex")
+ pol = stage.run_kwargs({"instance_id": "x"})["codex_spiral_policy"]
+ assert pol.reverify_loop_enabled and pol.runaway_enabled and pol.nudge_budget == 2
+ assert pol.post_green_finalize_enabled and pol.post_green_finalize_require_cleared_failures
+ assert not pol.effort_rescue_enabled and pol.resume_effort == ""
+
+
+def test_both_guards_expose_everything_the_shared_task_loop_reads():
+ """The Codex guard is driven by the SAME `run_task` loop as the generic one.
+
+ On 2026-09-12 that loop's `_log(... guard.segment)` crashed the codex-spiral lane on its
+ very first nudge: `CodexResumeGuard` had no `segment`. Nothing had caught it because the
+ only live Codex nudges on record were produced by a different harness. Rather than pin
+ one name, read the attributes the loop actually touches out of its source and require
+ both guard classes to carry them.
+ """
+ import inspect, re
+ from te_harness import swebench as sw
+ from tokeneconomizer.vendor import spiral as generic
+
+ src = inspect.getsource(sw.ClaudeSwebenchRunner.run_task)
+ names = set(re.findall(r"(?<![\w.])guard\.([a-z_][a-z0-9_]*)", src))
+ names -= {"continuation"} # reached through _guard_continuation
+ assert "segment" in names, "the loop no longer logs guard.segment; update this test"
+ codex_guard = _cs.CodexResumeGuard(_cs.shipped_policy(10))
+ for name in sorted(names):
+ assert hasattr(codex_guard, name), f"CodexResumeGuard is missing guard.{name}"
+ # segment tracks issued nudges, exactly as the generic guard's does
+ assert codex_guard.segment == 0
+ lines = [_json.dumps({"type": "thread.started", "thread_id": "t-5"})]
+ lines += [_cmd(f"rg needle{i} src") for i in range(40)]
+ guard = _cs.CodexResumeGuard(_cs.defensive_policy(16))
+ for l in lines:
+ if guard.monitor(l):
+ break
+ assert guard.segment == 0 and guard.continuation() is not None and guard.segment == 1
+ assert guard.segment == guard.nudges_done
+
+
+def test_the_codex_envelope_is_advanced_in_the_unit_its_trigger_compares(tmp_path):
+ """`CodexEnvelopeTracker` is documented in tool events and the guard's trigger compares
+ `state.turns` (command + edit). The stage used to feed it the row's `num_turns`, which
+ for Codex is agent STEPS (messages + tool calls) and runs ~1.45x larger -- measured on
+ the 2026-09-12 baselines, that raised the gate ~45% and left only 3 of 60 scikit-learn
+ tasks able to reach it, against 10 of 60 at the documented unit. The controller ran and
+ could not fire.
+ """
+ from tokeneconomizer.config import SpiralConfig
+ from tokeneconomizer.mechanisms.spiral import SpiralStage
+ cfg = SpiralConfig(codex_spiral=True, codex_spiral_profile="defensive")
+ stage = SpiralStage(cfg, max_turns=80, agent="codex")
+ srow = {}
+ stage.after_task({"instance_id": "x-1"}, {"resolved": True, "num_turns": 29,
+ "codex_spiral_tool_turns": 20}, str(tmp_path), srow)
+ assert stage.codex_tracker.resolved_tool_turns == [20] # the trigger's unit
+ assert srow["codex_spiral_envelope_unit"] == "tool_events"
+ assert srow["codex_spiral_tool_turns"] == 20
+ # a row from before the field falls back, and says so
+ stage.after_task({"instance_id": "x-2"}, {"resolved": True, "num_turns": 31}, str(tmp_path), srow)
+ assert stage.codex_tracker.resolved_tool_turns == [20, 31]
+ assert srow["codex_spiral_envelope_unit"] == "steps_fallback"
+ # and the runner records the guard's own count for the stage to read
+ import inspect
+ src = inspect.getsource(sw_codex_guard_row := __import__("te_harness.swebench", fromlist=["x"]).CodexSwebenchRunner._guard_row)
+ assert "codex_spiral_tool_turns" in src and "tool_turns" in src
+
+
+def test_the_resume_invocation_is_one_the_installed_cli_accepts():
+ """The spiral's whole mechanism is `codex exec resume <session> -` on the same thread.
+ Its option surface is SMALLER than `exec`'s (no -C, no --color), and a flag this CLI
+ build does not know would fail only at the moment the guard fires -- i.e. once per
+ hundred tasks, deep into a paid run. Check the argv against the CLI's own help."""
+ import shutil, subprocess, os, re, pytest
+ from te_harness import codex
+ argv = codex.build_resume_argv("01a09187-c54c-79a1-a40a-fe8ba406a935", hook=True,
+ effort="high")
+ assert argv[1:4] == ["exec", "resume", "01a09187-c54c-79a1-a40a-fe8ba406a935"]
+ assert argv[-1] == "-" and "-C" not in argv and "--color" not in argv
+ binary = os.environ.get("TE_CODEX_BIN") or shutil.which("codex")
+ if not binary:
+ pytest.skip("codex-cli not installed")
+ out = subprocess.run([binary, "exec", "resume", "--help"], capture_output=True,
+ text=True, timeout=120)
+ text = (out.stdout or "") + (out.stderr or "")
+ assert "[SESSION_ID]" in text # the positional this argv relies on
+ for flag in [a for a in argv if a.startswith("--")]:
+ assert flag in text, f"`codex exec resume` does not accept {flag}"
diff --git a/experiments/tests/test_sweep_arms.py b/experiments/tests/test_sweep_arms.py
new file mode 100644
index 0000000..7c9e372
--- /dev/null
+++ b/experiments/tests/test_sweep_arms.py
@@ -0,0 +1,641 @@
+"""The main-experiment sweep: its arms are the ablation they claim to be, offline.
+
+The sweep's whole argument is comparative -- one mechanism against three against none --
+so the arms have to differ in exactly the way their names say. Nothing here runs an agent;
+these are assertions about the registry `scripts/run_experiment.py` maps an arm name onto,
+which is the only place that mapping exists.
+
+Guarded specifically:
+ * a single-mechanism arm activates ONE mechanism, and with trimech's own parameters, so
+ it is an ablation of trimech rather than a separately tuned arm;
+ * trimech is exactly the union of the three, so "3-mech active" means what it says;
+ * the reasoning-effort arms differ from naive ONLY in the harness env, since an effort
+ level is invisible in an EconomizerConfig and a mislabelled cell would be undetectable
+ after the fact;
+ * every arm the sweep plan names is registered, and the scaffold arms agree with
+ tokeneconomizer/baselines' own arm->scaffold map -- an arm that named a scaffold the harness
+ does not have, or carried none where the harness expects one, would run NATIVE and file
+ the rows under the scaffold's name.
+"""
+import importlib.util
+import os
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _load(name, path):
+ spec = importlib.util.spec_from_file_location(name, path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+RX = _load("_run_experiment", os.path.join(_REPO, "scripts", "run_experiment.py"))
+SP = _load("_sweep_plan", os.path.join(_REPO, "scripts", "sweep_plan.py"))
+
+
+def _active(cfg):
+ """The set of mechanism names this config switches on."""
+ return {name for name in ("injection", "tooling", "spiral")
+ if getattr(cfg, name).enabled}
+
+
+# ---- one mechanism means one mechanism --------------------------------------------------
+def test_single_mech_arms_activate_exactly_one():
+ for arm, mech in (("inject-only", "injection"),
+ ("untool-only", "tooling"),
+ ("spiral-only", "spiral"),
+ ("spiral-only-web", "spiral")):
+ assert _active(RX.ARMS[arm].cfg()) == {mech}, arm
+
+
+def test_baseline_arms_activate_nothing():
+ for arm in ("baseline", "baseline-medium", "baseline-low"):
+ assert _active(RX.ARMS[arm].cfg()) == set(), arm
+
+
+def test_single_mech_arms_reuse_trimechs_parameters():
+ """An ablation of trimech, not a retuned arm: each single-mech config must equal
+ trimech's corresponding stage exactly."""
+ tri = RX.ARMS["trimech"].cfg()
+ assert RX.ARMS["inject-only"].cfg().injection == tri.injection
+ assert RX.ARMS["untool-only"].cfg().tooling == tri.tooling
+ assert RX.ARMS["spiral-only"].cfg().spiral == tri.spiral
+
+
+def test_trimech_is_the_union_of_the_three():
+ tri = RX.ARMS["trimech"].cfg()
+ assert tri.injection == RX.ARMS["inject-only"].cfg().injection
+ assert tri.tooling == RX.ARMS["untool-only"].cfg().tooling
+ assert tri.spiral == RX.ARMS["spiral-only"].cfg().spiral
+ assert _active(tri) == {"injection", "tooling", "spiral"}
+
+
+def test_web_arms_differ_from_generic_only_in_the_spiral_stage():
+ """Web-Bench's short streams change the spiral stage (LLM nudge + frozen envelope) and
+ nothing else -- a `-web` arm that also moved injection or untooling would make the two
+ domains' numbers incomparable for reasons unrelated to stream length."""
+ web, gen = RX.ARMS["trimech-web"].cfg(), RX.ARMS["trimech"].cfg()
+ assert web.injection == gen.injection and web.tooling == gen.tooling
+ assert web.spiral != gen.spiral
+ assert web.spiral.nudge_llm is True and web.spiral.frozen_envelope == 30.0
+ assert web.spiral == RX.ARMS["spiral-only-web"].cfg().spiral
+
+
+def test_the_web_scaffold_arm_is_the_web_tuning_plus_the_scaffold():
+ """`trimech-rtk-web` must be to `trimech-rtk` what `trimech-web` is to `trimech`.
+
+ Web-Bench swaps every spiral-bearing arm for its `-web` tuning because a within-stream
+ p90 envelope never warms up over 20 tasks. Planning plain `trimech-rtk` there would
+ spiral-tune it differently from `trimech-web` -- the one arm it exists to be compared
+ against -- so the comparison would measure the envelope, not the scaffold.
+ """
+ web, gen = RX.ARMS["trimech-rtk-web"].cfg(), RX.ARMS["trimech-rtk"].cfg()
+ assert web.injection == gen.injection and web.tooling == gen.tooling
+ assert web.spiral == RX.ARMS["trimech-web"].cfg().spiral
+ assert RX.ARMS["trimech-rtk-web"].scaffold == RX.ARMS["trimech-rtk"].scaffold == "rtk"
+ assert RX.ARMS["trimech-rtk-web"].mode == RX.ARMS["trimech-web"].mode
+ # The planner must never cross the two spellings, whichever it plans. `trimech-rtk-web`
+ # is currently OUT OF SCOPE on Web-Bench (see SWEEP_ARMS_WEBBENCH_OUT_OF_SCOPE) rather
+ # than unavailable, so what is pinned here is the pairing, not its presence: the generic
+ # arm must never be scheduled on Web-Bench, nor the -web one on SWE-bench.
+ assert "trimech-rtk" not in SP.SWEEP_ARMS["webbench"]
+ assert "trimech-rtk" in SP.SWEEP_ARMS["swebench"]
+ assert "trimech-rtk-web" not in SP.SWEEP_ARMS["swebench"]
+ assert "trimech-rtk-web" in SP.SWEEP_ARMS_WEBBENCH_OUT_OF_SCOPE
+
+
+# ---- effort arms are naive plus an env overlay, nothing else ----------------------------
+def test_effort_arms_are_baseline_plus_env():
+ base = RX.ARMS["baseline"]
+ assert base.env == {}, "naive must inherit the harness default effort, not pin one"
+ for arm, level in (("baseline-medium", "medium"), ("baseline-low", "low")):
+ entry = RX.ARMS[arm]
+ assert entry.cfg() == base.cfg(), f"{arm} must differ from baseline only in env"
+ assert entry.mode == base.mode
+ assert entry.env == {"TE_EFFORT_LEVEL": level}
+
+
+def test_effort_levels_are_ones_the_harness_accepts():
+ """The level is validated at harness construction; catching a typo here means catching
+ it offline instead of at task 0 of a paid lane."""
+ accepted = {"low", "medium", "high", "xhigh", "max"}
+ for entry in RX.ARMS.values():
+ level = entry.env.get("TE_EFFORT_LEVEL")
+ if level is not None:
+ assert level in accepted, level
+
+
+# ---- the plan and the registry cannot drift ---------------------------------------------
+def test_every_planned_arm_is_registered():
+ for domain, arms in SP.SWEEP_ARMS.items():
+ for arm in arms:
+ assert arm in RX.ARMS, f"{domain}: {arm}"
+
+
+def test_webbench_plan_excludes_the_scaffold_arms_it_cannot_host():
+ """Planning an unhostable scaffold arm for Web-Bench would enumerate cells that cannot
+ run. `rtk` IS hostable there and is deliberately not on this list."""
+ for arm in SP.SWEEP_ARMS_WEBBENCH_UNAVAILABLE:
+ assert arm not in SP.SWEEP_ARMS["webbench"], arm
+ assert arm in SP.SWEEP_ARMS["swebench"], arm
+
+
+def test_webbench_scaffold_arms_agree_with_the_harness():
+ """`sweep_plan` and `te_harness.webbench` must name the same hostable baselines.
+
+ Two places decide which external baselines Web-Bench can run: the planner (which
+ enumerates the cells) and the runner factory (which refuses one it cannot install). If
+ they disagree, the planner either schedules cells that die at construction or silently
+ drops an arm the harness would have run.
+ """
+ from te_harness.webbench import WEBBENCH_SCAFFOLDS
+
+ planned = {a for a in SP.SWEEP_ARMS["webbench"] if a in RX.SCAFFOLD_ARMS}
+ assert planned, "expected Web-Bench to plan at least one scaffold arm"
+ for arm in planned:
+ assert RX.SCAFFOLD_ARMS[arm] in WEBBENCH_SCAFFOLDS, (
+ f"webbench plans {arm!r} (scaffold {RX.SCAFFOLD_ARMS[arm]!r}) but the harness "
+ f"hosts only {sorted(WEBBENCH_SCAFFOLDS)}")
+ # ... and the refused list is exactly the PLANNED scaffold arms whose scaffold is NOT
+ # hostable. The planner's universe is the generic sweep; Codex arms are launched by the
+ # run book's own loops, and the Codex Web-Bench runner's refusal of an unhostable
+ # scaffold is pinned in tests/test_codex_webbench.py.
+ universe = set().union(*SP.SWEEP_ARMS.values())
+ refused = {a for a, sc in RX.SCAFFOLD_ARMS.items()
+ if a in universe and sc not in WEBBENCH_SCAFFOLDS}
+ assert set(SP.SWEEP_ARMS_WEBBENCH_UNAVAILABLE) == refused, (
+ f"SWEEP_ARMS_WEBBENCH_UNAVAILABLE={sorted(SP.SWEEP_ARMS_WEBBENCH_UNAVAILABLE)} "
+ f"but the harness cannot host {sorted(refused)}")
+
+
+def test_scaffold_arms_agree_with_the_harness_map():
+ """The arm registry and tokeneconomizer/baselines must name the same pairs. Two places
+ map an arm to a scaffold, and a disagreement runs native under a scaffold's name."""
+ from tokeneconomizer import baselines as scaffolds
+ assert RX.SCAFFOLD_ARMS == scaffolds.ARM_SCAFFOLDS
+
+
+def test_every_scaffold_arm_names_a_real_scaffold():
+ from tokeneconomizer import baselines as scaffolds
+ for arm, name in RX.SCAFFOLD_ARMS.items():
+ assert name in scaffolds.SCAFFOLDS, f"{arm} -> {name}"
+ assert scaffolds.for_arm(arm).name == name
+
+
+def test_scaffold_arms_are_runnable_and_others_carry_none():
+ for arm, entry in RX.ARMS.items():
+ if arm in RX.SCAFFOLD_ARMS:
+ assert entry.scaffold == RX.SCAFFOLD_ARMS[arm]
+ else:
+ assert entry.scaffold is None, f"{arm} carries an unregistered scaffold"
+
+
+def test_trimech_scaffold_arms_are_trimech_plus_a_scaffold():
+ """A `trimech-<scaffold>` arm must be trimech EXACTLY, differing only in the scaffold
+ -- otherwise its comparison against plain trimech measures two changes at once."""
+ tri = RX.ARMS["trimech"]
+ for arm in ("trimech-rtk", "trimech-repomap"):
+ entry = RX.ARMS[arm]
+ assert entry.cfg() == tri.cfg(), arm
+ assert entry.mode == tri.mode and entry.env == tri.env
+ assert entry.scaffold
+
+
+def test_no_trimech_serena_arm():
+ """Serena's contract (navigate via its MCP tools) contends with evidence untooling
+ (prune unused tool schemas), so the combination is deliberately not offered."""
+ assert "trimech-serena" not in RX.ARMS
+ assert "serena" in RX.ARMS and RX.ARMS["serena"].scaffold == "serena"
+
+
+def test_bare_scaffold_arms_activate_no_mechanism():
+ """`rtk`/`serena`/`repomap` are NAIVE plus a scaffold -- the naive+X baselines."""
+ for arm in ("rtk", "serena", "repomap"):
+ assert _active(RX.ARMS[arm].cfg()) == set(), arm
+
+
+def test_each_domain_plans_one_naive_and_one_full_stack():
+ """Every domain's plan must contain the two arms the headline comparison needs."""
+ for domain, arms in SP.SWEEP_ARMS.items():
+ assert "baseline" in arms, domain
+ assert any(a.startswith("trimech") for a in arms), domain
+
+
+# ---- cell enumeration -------------------------------------------------------------------
+def test_cells_are_the_full_product_seed_major():
+ arms, repos, seeds = ["a", "b"], ["r1", "r2", "r3"], [0, 1]
+ got = SP.cells("swebench", arms, repos, seeds)
+ assert len(got) == len(arms) * len(repos) * len(seeds)
+ assert len(set(got)) == len(got), "no cell may be enumerated twice"
+ # seed-major: seed 0 is complete before seed 1 begins, so an interrupted sweep leaves
+ # whole seeds finished rather than every seed half-done.
+ assert [s for _, _, s in got] == sorted(s for _, _, s in got)
+
+
+def test_estimate_scales_with_the_committed_task_counts():
+ sizes = SP.group_sizes("swebench")
+ repos = sorted(sizes)
+ e = SP.estimate("swebench", ["baseline", "trimech"], repos, [0, 1, 2, 3, 4], 1.0, 1.0)
+ assert e["n_task_runs"] == sum(sizes.values()) * 2 * 5
+ assert e["usd"] == e["n_task_runs"] and e["unknown_repos"] == []
+
+
+def test_estimate_flags_a_repo_outside_the_workload_record():
+ e = SP.estimate("swebench", ["baseline"], ["not/a-repo"], [0], 1.0, 1.0)
+ assert e["unknown_repos"] == ["not/a-repo"] and e["n_task_runs"] == 0
+
+
+# ---- the planner must count what the driver runs ----------------------------------------
+def test_group_sizes_match_the_manifest_the_driver_filters_to():
+ """`main_sweep.sh` decides a cell is finished by comparing its stored row.json count
+ against `group_sizes`, while the driver runs exactly the ids `workloads/<domain>.json`
+ names. If those two numbers disagree by even one, SKIP_DONE never fires for that group
+ and every resume silently re-runs completed, already-paid-for cells.
+
+ This is not hypothetical: `workloads.task_counts()` is the PROVENANCE record and
+ deliberately does not subtract SWEBENCH_EXCLUDED, so reading it here made the planner
+ expect 32 scikit-learn tasks for a run that executes 60 (31 published-minus-
+ excluded, plus the 29-id de-saturation extension).
+ """
+ import json
+ for domain in SP.SWEEP_ARMS:
+ path = os.path.join(_REPO, "workloads", f"{domain}.json")
+ with open(path, encoding="utf-8") as fh:
+ groups = json.load(fh)["groups"]
+ assert SP.group_sizes(domain) == {g: len(ids) for g, ids in groups.items()}, domain
+
+
+def test_group_sizes_exclude_what_a_new_run_must_skip():
+ """The exclusion and extension tables are both forward-looking: an excluded id must not
+ be counted, an added one must be, and task_counts() must still report the historical
+ size either way."""
+ from tokeneconomizer import workloads
+ sizes = SP.group_sizes("swebench")
+ for iid in workloads.SWEBENCH_EXCLUDED:
+ group = next(g for g, ids in workloads.SWEBENCH_INSTANCES.items() if iid in ids)
+ added = len(workloads.SWEBENCH_ADDED.get(group, ()))
+ assert sizes[group] == len(workloads.SWEBENCH_INSTANCES[group]) - 1 + added
+ assert workloads.task_counts("swebench")[group] == \
+ len(workloads.SWEBENCH_INSTANCES[group])
+ for group, ids in workloads.SWEBENCH_ADDED.items():
+ skipped = sum(1 for i in workloads.SWEBENCH_INSTANCES[group]
+ if i in workloads.SWEBENCH_EXCLUDED)
+ assert sizes[group] == (len(workloads.SWEBENCH_INSTANCES[group])
+ - skipped + len(ids))
+ assert workloads.task_counts("swebench")[group] == \
+ len(workloads.SWEBENCH_INSTANCES[group])
+
+
+def test_mode_is_manual4_exactly_when_injection_is_enabled():
+ """`mode` is written per arm, but it is not free: manual4 IS the micro-skill prefix.
+
+ An injection arm filed as `standalone` would run without the prefix the mechanism needs;
+ a non-injection arm filed as `manual4` would carry a prefix nothing uses, and both would
+ still produce plausible-looking rows. The mode also lands in the run-dir name and in
+ aggregate.py's `mode` column, so a drifted value silently mislabels a whole cell.
+
+ This is what makes `trimech-rtk` / `trimech-repomap` manual4: they contain trimech,
+ which contains injection -- exactly like `trimech` itself, which they are siblings of.
+ """
+ wrong = [(name, a.mode, a.cfg().injection.enabled)
+ for name, a in RX.ARMS.items()
+ if (a.mode == "manual4") != bool(a.cfg().injection.enabled)]
+ assert not wrong, (
+ "mode must be 'manual4' exactly when injection is enabled, else 'standalone': "
+ + ", ".join(f"{n} is {m} with injection={i}" for n, m, i in wrong))
+
+
+
+# ---- Web-Bench's out-of-scope arms are UNPLANNED, not unavailable -----------------------
+# Web-Bench carries the generalization claim; the decomposition is answered on SWE-bench at
+# 12 arms x 6 repos x 5 seeds. That is a scope decision, and the failure it must not become
+# is silently claiming the harness cannot run those arms -- someone would then never try.
+
+def test_webbench_out_of_scope_arms_are_real_arms_that_simply_are_not_planned():
+ for arm in SP.SWEEP_ARMS_WEBBENCH_OUT_OF_SCOPE:
+ assert arm in RX.ARMS, f"{arm} is out of scope but not even registered"
+ assert arm not in SP.SWEEP_ARMS["webbench"], arm
+ # NOT unavailable: unavailability is about what the harness cannot host.
+ assert arm not in SP.SWEEP_ARMS_WEBBENCH_UNAVAILABLE, (
+ f"{arm} is listed both as out-of-scope and as unhostable; those mean "
+ f"different things and a reader would not know which applies")
+
+
+def test_the_webbench_sweep_is_the_generalization_set():
+ """4 core arms + the one hostable scaffold. The core four are what make a cost saving
+ distinguishable from turning the effort knob down; rtk is the prior-art comparison."""
+ assert set(SP.SWEEP_ARMS["webbench"]) == {
+ "baseline", "baseline-medium", "baseline-low", "trimech-web", "rtk"}
+
+
+def test_a_scaffold_arm_out_of_scope_on_webbench_is_still_hostable():
+ """`trimech-rtk-web` is dropped for scope, not capability -- so if it is ever added
+ back, the harness must already be able to install what it names."""
+ from te_harness.webbench import WEBBENCH_SCAFFOLDS
+ assert RX.SCAFFOLD_ARMS["trimech-rtk-web"] in WEBBENCH_SCAFFOLDS
+
+
+# ---- the Codex sibling of trimech -------------------------------------------------------
+def test_codex_trimech_is_the_union_of_the_three_codex_arms():
+ """`codex-trimech` must be its three single-mechanism arms co-activated, unretuned.
+
+ Same contract as `test_trimech_is_the_union_of_the_three`, and the same reason: an arm
+ that retuned one of its mechanisms would not be a 3-mech point against three 1-mech
+ ablations, it would be a fourth configuration.
+ """
+ tri = RX.ARMS["codex-trimech"].cfg()
+ assert tri.injection == RX.ARMS["codex-inject"].cfg().injection
+ assert tri.tooling == RX.ARMS["codex-untool"].cfg().tooling
+ assert tri.spiral == RX.ARMS["codex-spiral-defensive"].cfg().spiral
+ assert _active(tri) == {"injection", "tooling", "spiral"}
+ assert RX.ARMS["codex-trimech"].mode == "manual4" # injection is on
+
+
+def test_the_codex_spiral_arms_have_the_shape_the_injection_arms_have():
+ """Plain name = the configuration with a live A/B; the superseded one keeps a name.
+
+ Settled twice, the same way. `codex-inject` is the composed `envfacts+distiller`
+ profile with `codex-inject-envfacts` as the named ablation and `codex-inject-distiller`
+ as the alias its stored cell was filed under. `codex-spiral` is the DEFENSIVE profile
+ with `codex-spiral-postgreen` as the named ablation and `codex-spiral-defensive` as the
+ alias its stored cell was filed under. The alternative -- leaving the plain name on the
+ shipped post-green rule -- puts a configuration that replays as 8 fires in 204 terra
+ tasks (net -0.26%) behind the name a plan reaches for, which is a baseline wearing the
+ mechanism's name.
+ """
+ S = lambda n: RX.ARMS[n].cfg().spiral
+ assert S("codex-spiral").codex_spiral_profile == "defensive"
+ assert S("codex-spiral-defensive") == S("codex-spiral") # alias
+ assert S("codex-spiral-postgreen").codex_spiral_profile == "post-green"
+ assert S("codex-spiral-postgreen") != S("codex-spiral")
+ # the generic guard stays off on all of them: the Codex controller is what runs
+ for n in ("codex-spiral", "codex-spiral-postgreen", "codex-spiral-defensive"):
+ assert S(n).codex_spiral and not S(n).guard and not S(n).nudge
+ from tokeneconomizer.mechanisms import codex_spiral as cs
+ assert cs.CODEX_DEFAULT_PROFILE == "defensive"
+ assert cs.CODEX_SHIPPED_PROFILE == "post-green"
+
+
+def test_the_codex_family_relates_to_codex_trimech_as_the_generic_one_does_to_trimech():
+ """The whole point of the Codex arms is that they are the SAME experiment on a second
+ agent, so the two families must have the same shape -- not merely each be internally
+ consistent. Checked property by property rather than by eye, because the asymmetry that
+ actually occurred was invisible in the configs: every structural check passed while the
+ plain name `codex-spiral` carried a spiral `codex-trimech` did not compose.
+ """
+ GEN = ("trimech", "inject-only", "untool-only", "spiral-only")
+ CDX = ("codex-trimech", "codex-inject", "codex-untool", "codex-spiral")
+ for tri, inj, unt, spi in (GEN, CDX):
+ t = RX.ARMS[tri]
+ c = t.cfg()
+ # 1. trimech is exactly the union of the three single-mechanism arms
+ assert c.injection == RX.ARMS[inj].cfg().injection
+ assert c.tooling == RX.ARMS[unt].cfg().tooling
+ assert c.spiral == RX.ARMS[spi].cfg().spiral
+ assert _active(c) == {"injection", "tooling", "spiral"}
+ # 2. each single-mechanism arm activates exactly one
+ for name, mech in ((inj, "injection"), (unt, "tooling"), (spi, "spiral")):
+ assert _active(RX.ARMS[name].cfg()) == {mech}, name
+ # 3. the mode label follows injection, and nothing carries a scaffold or env override
+ assert t.mode == RX.ARMS[inj].mode == "manual4"
+ assert RX.ARMS[unt].mode == RX.ARMS[spi].mode == "standalone"
+ for name in (tri, inj, unt, spi):
+ assert not RX.ARMS[name].scaffold and not (RX.ARMS[name].env or {}), name
+ # 4. the three appear before trimech in the canonical order
+ from tokeneconomizer.arms import ARM_ORDER
+ assert max(ARM_ORDER.index(n) for n in (inj, unt, spi)) < ARM_ORDER.index(tri)
+
+
+def test_codex_trimech_is_a_paid_arm():
+ """The proposer bills, so the endpoint preflight must fire before task 0 rather than
+ the first paid call failing 40 tasks into a live lane."""
+ from tokeneconomizer.config import needs_optimizer
+ assert needs_optimizer(RX.ARMS["codex-trimech"].cfg()) is True
+ assert RX.ARMS["codex-trimech"].cfg().injection.paid is True
+
+
+# ---- the docs must name the arms the registry actually has -------------------------------
+# The arms the Codex run book PUBLISHES -- one per mechanism, plus the agent-agnostic naive
+# spine, the all-three arm and its two scaffold variants, and the three scaffolds this lane
+# hosts (each through the channel Codex has). The registry knows more
+# names than this (two aliases stored cells are filed under, and four ablations), and that is
+# deliberate: they still resolve so an old cell can be re-run or re-read, but they are not
+# offered, because a catalog carrying two spellings of one mechanism is a catalog in which an
+# ablation can be reported as the mechanism.
+CODEX_PUBLISHED_ARMS = frozenset({
+ "baseline", "baseline-medium", "baseline-low", "repomap", "rtk", "serena",
+ "codex-inject", "codex-untool", "codex-spiral", "codex-trimech", "codex-trimech-rtk",
+ "codex-trimech-repomap",
+})
+
+
+def _doc(name):
+ import os
+ repo = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ return open(os.path.join(repo, "docs", name)).read()
+
+
+def _section(text, start, end=None):
+ i = text.index(start)
+ return text[i:text.index(end, i + 1)] if end else text[i:]
+
+
+def _table_arms(sec):
+ """The arm names a run-book table lists, one per row, in order."""
+ import re
+ return re.findall(r"^\| `([a-z0-9-]+)` \|", sec, re.M)
+
+
+def _loop_arms(sec):
+ """The arm names a run-book command block launches, through `ARMS=` or a `for ARM` loop."""
+ import re
+ names = set()
+ for group in re.findall(r'ARMS="([a-z0-9\- ]+)"', sec):
+ names |= set(group.split())
+ for group in re.findall(r"for ARM in ([a-z0-9\- \\\n]+?);", sec):
+ names |= set(group.replace("\\", " ").split())
+ return names
+
+
+# The run book's lanes: the Claude Code / Copilot section and the Codex section, each split
+# into one swebench and one webbench subsection.
+_GENERIC = ("## 4. Claude Code and Copilot", "## 5. Codex")
+_CODEX = ("## 5. Codex", "## 6.")
+
+
+def test_the_codex_lane_publishes_exactly_one_arm_per_mechanism():
+ """Checked both ways, because both failures have happened here.
+
+ Every published arm must appear in the catalog -- it drifted twice, once missing the
+ effort baselines and once missing `repomap`. And every arm the catalog names must be
+ registered and published: a doc promising an arm nobody implemented is one failure, a doc
+ quietly re-offering a dropped ablation beside the mechanism it ablates is the other.
+ """
+ codex = _section(_doc("RUNNING.md"), *_CODEX)
+ sb = _section(codex, "### swebench", "### webbench")
+ named = set(_table_arms(sb))
+ assert named == CODEX_PUBLISHED_ARMS, (
+ f"catalog names {sorted(named)}, published set is {sorted(CODEX_PUBLISHED_ARMS)}")
+ assert _loop_arms(sb) == named, "the swebench command launches a different arm set"
+ wb = _section(codex, "### webbench")
+ assert _loop_arms(wb) == set(_table_arms(wb)), "the webbench command and table disagree"
+ invented = sorted(n for n in named | set(_table_arms(wb)) if n not in RX.ARMS)
+ assert not invented, f"the catalog names arms that do not exist: {invented}"
+ cfgs = {n: RX.ARMS[n].cfg() for n in named}
+ one = lambda mech: sorted(
+ n for n, c in cfgs.items()
+ if getattr(c, mech).enabled and not any(
+ getattr(c, m).enabled for m in ("injection", "tooling", "spiral") if m != mech))
+ assert one("injection") == ["codex-inject"]
+ assert one("tooling") == ["codex-untool"]
+ assert one("spiral") == ["codex-spiral"]
+
+
+def test_every_arm_the_generic_lanes_launch_is_registered_and_planned():
+ """The Claude Code and Copilot lanes launch through `main_sweep.sh`, which plans cells from
+ `sweep_plan.SWEEP_ARMS`. An arm named in the book but absent from the plan gets "the plan is
+ empty" at launch; one absent from the registry is a doc promising something nobody
+ implemented."""
+ generic = _section(_doc("RUNNING.md"), *_GENERIC)
+ sb = set(_table_arms(_section(generic, "### swebench", "### webbench")))
+ wb = set(_table_arms(_section(generic, "### webbench", "### The frontier lane")))
+ assert sb, "no swebench arms parsed"
+ for arm in sb | wb:
+ assert arm in RX.ARMS, f"the run book lists {arm!r}, which is not registered"
+ # the swebench catalog IS the planned set, and so is the webbench one
+ assert sb == set(SP.SWEEP_ARMS["swebench"]), (
+ f"swebench catalog {sorted(sb)} != plan {sorted(SP.SWEEP_ARMS['swebench'])}")
+ assert wb == set(SP.SWEEP_ARMS["webbench"]), (
+ f"webbench catalog {sorted(wb)} != plan {sorted(SP.SWEEP_ARMS['webbench'])}")
+ frontier = _loop_arms(_section(generic, "### The frontier lane"))
+ for arm in frontier:
+ assert arm in RX.ARMS, f"the frontier lane launches unregistered arm {arm!r}"
+
+
+def test_each_lane_splits_its_commands_by_domain():
+ """Both lanes present SWE-bench and Web-Bench as two separate subsections, each with its
+ own table and its own command, so a reader can follow it top to bottom.
+
+ The runner check is the other half -- a Web-Bench command carrying the SWE-bench factory
+ runs the wrong lifecycle against the right ids, which the launcher refuses but a reader
+ would have copied.
+ """
+ import re
+ text = _doc("RUNNING.md")
+ for lane in (_GENERIC, _CODEX):
+ sec = _section(text, *lane)
+ heads = [(m.start(), m.group(0)) for m in
+ re.finditer(r"^### (swebench|webbench) — \d+ arms$", sec, re.M)]
+ kinds = [("webbench" if "webbench" in h else "swebench") for _, h in heads]
+ assert kinds == ["swebench", "webbench"], f"{lane[0]}: got {heads}"
+ for i, (start, head) in enumerate(heads):
+ body = sec.index("\n", start)
+ nxt = re.search(r"^#{2,3} ", sec[body:], re.M)
+ stop = body + nxt.start() if nxt else len(sec)
+ sub = sec[start:stop]
+ domain = kinds[i]
+ entries = _table_arms(sub)
+ assert entries and len(entries) == len(set(entries)), f"{lane[0]}/{domain}"
+ claimed = int(re.search(r"(\d+) arms", head).group(1))
+ assert claimed == len(entries), f"{head!r} but {len(entries)} entries"
+ wrong = "swebench" if domain == "webbench" else "webbench"
+ assert f"te_harness.{domain}:make_runner" in sub, f"{lane[0]}/{domain}"
+ assert f"te_harness.{wrong}:make_runner" not in sub, f"{lane[0]}/{domain}"
+
+
+def test_the_harness_scaffold_table_names_every_arm_that_carries_each_scaffold():
+ """docs/HARNESS.md maps scaffold -> arms. It is the table a reader consults to find out
+ which arms need `aider` on the host or the staged rtk binary, so an arm missing from it
+ reads as needing nothing. It went stale twice in one day, both times when a Codex
+ sibling arm was added, so pin it against the registry rather than against a copy."""
+ import re
+ from tokeneconomizer import baselines
+ text = _doc("HARNESS.md")
+ table = text[text.index("| Scaffold | Arms |"):]
+ table = table[:table.index("\n\n")]
+ listed = {}
+ for line in table.splitlines():
+ cells = [c.strip() for c in line.split("|")]
+ if len(cells) < 4 or not cells[1].startswith("`"):
+ continue
+ scaffold = cells[1].strip("`")
+ listed[scaffold] = set(re.findall(r"`([a-z0-9-]+)`", cells[2]))
+ assert listed, "no scaffold rows parsed out of docs/HARNESS.md"
+ for scaffold, arms in listed.items():
+ real = {a for a, s in baselines.ARM_SCAFFOLDS.items() if s == scaffold}
+ assert arms == real, (
+ f"HARNESS.md lists {scaffold!r} arms as {sorted(arms)}, "
+ f"but ARM_SCAFFOLDS says {sorted(real)}")
+
+
+def test_the_published_spiral_arm_is_the_defensive_profile():
+ """The one that was approved, and the one both live cells ran.
+
+ `codex-spiral` must be the defensive profile -- every deterministic spiral signature as a
+ same-thread nudge, two per task -- and must never touch reasoning effort, which the owner
+ ruled out on 2026-09-12. The superseded post-green rule keeps a registry name but is not
+ what the plain one runs.
+ """
+ from tokeneconomizer.mechanisms import codex_spiral as cs
+ sp = RX.ARMS["codex-spiral"].cfg().spiral
+ assert sp.codex_spiral_profile == "defensive" == cs.CODEX_DEFAULT_PROFILE
+ assert cs.profile_version("defensive") == "codex-native-spiral-defensive-v1"
+ pol = cs.profile_policy("defensive", 20)
+ assert pol.nudge_budget == 2 and pol.runaway_enabled
+ assert not pol.effort_rescue_enabled and not pol.resume_effort
+ assert RX.ARMS["codex-spiral-defensive"].cfg().spiral == sp
+
+
+def test_the_codex_lane_uses_the_whole_evaluated_workload_not_one_repo():
+ """`codex_ab.sh` defaults to one repository, so the run book must name them all or a
+ 6-repo sweep quietly becomes a 1-repo one."""
+ from tokeneconomizer import workloads
+ codex = _section(_doc("RUNNING.md"), *_CODEX)
+ for group in workloads.task_counts("swebench"):
+ assert group in codex, f"{group} is in the evaluated workload but not in the run book"
+ for project in workloads.task_counts("webbench"):
+ assert project in codex, f"{project} is an evaluated project but not in the run book"
+
+
+def test_codex_trimech_rtk_is_codex_trimech_with_the_scaffold_and_nothing_else():
+ """The same contract `trimech-rtk` has with `trimech`: identical mechanism parameters,
+ the scaffold named and nothing retuned -- otherwise the arm would measure a fourth
+ configuration rather than the scaffold's marginal effect on the stack."""
+ a, b = RX.ARMS["codex-trimech-rtk"], RX.ARMS["codex-trimech"]
+ assert a.cfg() == b.cfg() and a.mode == b.mode == "manual4"
+ assert a.scaffold == "rtk" and b.scaffold is None
+ from tokeneconomizer import baselines
+ assert baselines.ARM_SCAFFOLDS["codex-trimech-rtk"] == "rtk"
+ assert RX.SCAFFOLD_ARMS["codex-trimech-rtk"] == "rtk"
+ # and the plain scaffold arm is agent-agnostic: no mechanism, just the scaffold
+ r = RX.ARMS["rtk"]
+ assert _active(r.cfg()) == set() and r.scaffold == "rtk"
+
+
+def test_codex_trimech_repomap_is_codex_trimech_with_the_scaffold_and_nothing_else():
+ """The contract `trimech-repomap` has with `trimech`, on the Codex family: identical
+ mechanism parameters, the scaffold named, nothing retuned."""
+ a, b = RX.ARMS["codex-trimech-repomap"], RX.ARMS["codex-trimech"]
+ assert a.cfg() == b.cfg() and a.mode == b.mode == "manual4"
+ assert a.scaffold == "repomap" and b.scaffold is None
+ from tokeneconomizer import baselines
+ assert baselines.ARM_SCAFFOLDS["codex-trimech-repomap"] == "repomap"
+ assert RX.SCAFFOLD_ARMS["codex-trimech-repomap"] == "repomap"
+ r = RX.ARMS["repomap"]
+ assert _active(r.cfg()) == set() and r.scaffold == "repomap"
+
+
+def test_every_generic_trimech_scaffold_arm_has_exactly_one_codex_sibling():
+ """`trimech-<s>` <-> `codex-trimech-<s>` for every scaffold the generic lane combines with
+ trimech, carrying the same scaffold on both sides -- and nothing more on either: the
+ generic lane refuses `trimech-serena`, so a `codex-trimech-serena` would be a Codex-only
+ combination with no generic twin to compare against. `-web` is Web-Bench's spiral
+ tuning of the same scaffold arm, not a scaffold of its own."""
+ from tokeneconomizer import baselines
+ generic = {a[len("trimech-"):]: s for a, s in baselines.ARM_SCAFFOLDS.items()
+ if a.startswith("trimech-") and not a.endswith("-web")}
+ codex = {a[len("codex-trimech-"):]: s for a, s in baselines.ARM_SCAFFOLDS.items()
+ if a.startswith("codex-trimech-")}
+ assert generic == codex == {"rtk": "rtk", "repomap": "repomap"}
+
diff --git a/experiments/tests/test_tooling.py b/experiments/tests/test_tooling.py
new file mode 100644
index 0000000..55486cb
--- /dev/null
+++ b/experiments/tests/test_tooling.py
@@ -0,0 +1,422 @@
+"""Tooling: generic evidence untooling + Codex-native capability untooling."""
+import os
+
+from tokeneconomizer.mechanisms.tooling import ToolingStage
+from tokeneconomizer.mechanisms import codex_untool as cu
+from tokeneconomizer.config import ToolingConfig
+from tokeneconomizer.adapters import SwebenchAdapter
+from fixtures import write_codex_transcript
+
+
+# ---- Codex-native capability untool controller ----
+
+def test_codex_profile_task1_full_contract(tmp_path):
+ p = cu.CodexUntoolProfile("pydata/xarray", str(tmp_path), fresh=True)
+ sel = p.begin_task("x-1", problem_statement="fix concat")
+ assert sel["audit"] is True # task 1 observes the complete contract
+ assert sel["relay_capabilities"] == [] # nothing pruned yet
+ assert "deferred_wait" in sel["core_capabilities"]
+
+
+def test_codex_profile_never_prunes_core(tmp_path):
+ p = cu.CodexUntoolProfile("pydata/xarray", str(tmp_path), fresh=True)
+ # drive several tasks where nothing completes; deferred_wait must never be demoted.
+ for i in range(6):
+ sel = p.begin_task(f"x-{i}")
+ assert "deferred_wait" not in sel["relay_capabilities"]
+ usage = {"attempted": {}, "completed": {}, "failed": {}}
+ p.observe(usage, resolved=True)
+ assert "deferred_wait" not in p.state["current_pruned"]
+
+
+def test_codex_profile_demotes_unused_web(tmp_path):
+ p = cu.CodexUntoolProfile("pydata/xarray", str(tmp_path), fresh=True)
+ # task 1 (audit) with web loaded-and-unused -> after observe, web becomes eligible.
+ p.begin_task("x-1")
+ p.observe({"attempted": {}, "completed": {}, "failed": {}}, resolved=True)
+ sel2 = p.begin_task("x-2")
+ # web needs only 1 unused exposure; it should now be in the pruned relay mask.
+ assert "sealed_web" in sel2["relay_capabilities"]
+
+
+def test_codex_profile_completed_use_protects(tmp_path):
+ p = cu.CodexUntoolProfile("pydata/xarray", str(tmp_path), fresh=True)
+ p.begin_task("x-1")
+ # a completed web use permanently protects it.
+ p.observe({"attempted": {"web": 1}, "completed": {"web": 1}, "failed": {}},
+ resolved=True)
+ assert "web" in p.state["ever_completed"]
+ sel2 = p.begin_task("x-2")
+ assert "sealed_web" not in sel2["relay_capabilities"]
+
+
+def test_inspect_usage_counts_web(tmp_path):
+ tr = str(tmp_path / "t.jsonl")
+ write_codex_transcript(tr, [
+ {"type": "web_search", "name": "web_search", "status": "failed"},
+ {"type": "command_execution", "command": "pytest"}])
+ usage = cu.inspect_usage(tr)
+ assert usage["attempted"]["web"] >= 1
+ assert usage["completed"]["web"] == 0 # failed status is not a completed use
+
+
+# ---- Codex ToolingStage wiring ----
+
+def test_tooling_stage_codex_mask(tmp_path):
+ cfg = ToolingConfig(codex_untool=True)
+ stage = ToolingStage(cfg, "codex", "agent",
+ adapter=SwebenchAdapter(), repo="pydata/xarray",
+ results_root=str(tmp_path), fresh=True)
+ task = {"instance_id": "x-1", "problem_statement": "fix"}
+ mask = stage.build_tools(task, 0)
+ assert mask == [] # task 1 = full contract
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ write_codex_transcript(os.path.join(rd, "transcript.jsonl"),
+ [{"type": "command_execution", "command": "pytest"}])
+ srow = {"resolved": True}
+ stage.after_task(task, {}, rd, srow)
+ assert srow["codex_untool_active"] is True
+ assert srow["codex_untool_profile"] == cu.PROFILE_VERSION
+
+
+# ---- generic evidence untooling ----
+
+class _StubAdapter:
+ """Minimal adapter: fixed catalog, no tool used, protects nothing."""
+ name = "stub"
+ def tool_usage(self, path):
+ return {}
+ def protected_tools(self):
+ return frozenset()
+ def parse_catalog(self, run_dir_path, agent):
+ return {"web_fetch": "fetch a url", "ask_user": "ask the human"}
+
+
+def test_evidence_untool_drops_after_streak(tmp_path):
+ cfg = ToolingConfig(evidence=2)
+ stage = ToolingStage(cfg, "claude", "agent", adapter=_StubAdapter())
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ open(os.path.join(rd, "transcript.jsonl"), "w").write("")
+ task = {"instance_id": "w-1"}
+ # task 0: catalog gets learned, streaks start
+ stage.build_tools(task, 0)
+ stage.after_task(task, {}, rd, {"resolved": True})
+ # after 2 unused tasks, the tools cross the evidence threshold
+ stage.after_task(task, {}, rd, {"resolved": True})
+ excl = stage.build_tools(task, 3)
+ assert "web_fetch" in excl and "ask_user" in excl
+
+
+# ---- the mask must be EARNED, never a prior ---------------------------------
+# The stored Web-Bench arms were produced by an older `--untool` flag that applied
+# `reflect.untool_exclusions()` -- the static INAPPLICABLE_TOOLS set -- from the FIRST
+# rollout of the stream. Measured consequences: the mask was live at the `init` task before
+# any task had completed, it cost zero LLM tokens so the honest ledger charged nothing for
+# it, and the keep-set was byte-identical across all 15 cells (5 projects x 3 seeds). The
+# stored Web-Bench cost win is therefore largely an unpaid, unlearned prior rather than the
+# evidence-gated mechanism, and it is not comparable to the SWE-bench arms.
+#
+# The current code does not do this: the static surface is still exported on the adapters but
+# no longer reachable from the stage. These tests pin that shut, in BOTH domains, so it
+# cannot regress silently -- the failure mode is invisible in aggregate numbers and only
+# shows up as a suspiciously large cost win.
+from tokeneconomizer.adapters import WebBenchAdapter # noqa: E402
+from tokeneconomizer.vendor import reflect # noqa: E402
+from fixtures import write_claude_transcript_with_catalog # noqa: E402
+
+_CATALOG = ["Bash", "Edit", "Read", "Write", "WebFetch", "WebSearch", "Workflow",
+ "Skill", "NotebookEdit", "CronCreate", "TaskCreate"]
+
+
+def _stage(tmp_path, adapter, evidence=3):
+ return ToolingStage(ToolingConfig(evidence=evidence, judge=False), "claude", "agent",
+ adapter=adapter, repo="acme/widget",
+ results_root=str(tmp_path), fresh=True)
+
+
+def _finish(stage, tmp_path, i, used, catalog=_CATALOG):
+ """Run one task to completion so the stage can learn from its transcript."""
+ rd = os.path.join(str(tmp_path), f"run{i}")
+ os.makedirs(rd, exist_ok=True)
+ write_claude_transcript_with_catalog(
+ os.path.join(rd, "transcript.jsonl"), catalog,
+ [(t, {"command": "x"} if t == "Bash" else {"file_path": "a.py"}, "ok")
+ for t in used])
+ stage.after_task({"instance_id": f"w-{i}"}, {}, rd, {})
+
+
+def test_first_task_of_a_stream_is_never_masked_swebench(tmp_path):
+ """No evidence can exist before the first task, so the agent must see the full catalog.
+ This is the property the stored Web-Bench runs violated."""
+ stage = _stage(tmp_path, SwebenchAdapter(tasks_path=None))
+ assert stage.build_tools({"instance_id": "w-0"}, 0) == []
+
+
+def test_first_task_of_a_stream_is_never_masked_webbench(tmp_path):
+ """Same guarantee on the Web-Bench adapter -- the domain where it actually broke."""
+ stage = _stage(tmp_path, WebBenchAdapter("zustand"))
+ assert stage.build_tools({"instance_id": "task-1"}, 0) == []
+
+
+def test_mask_appears_only_after_the_catalog_is_learned(tmp_path):
+ """The trim must come from observed disuse: full catalog on task 0, then a mask."""
+ stage = _stage(tmp_path, WebBenchAdapter("zustand"))
+ assert stage.build_tools({"instance_id": "task-1"}, 0) == []
+ for i in range(3): # three tasks using only Bash/Edit/Read/Write
+ _finish(stage, tmp_path, i, ["Bash", "Edit", "Read", "Write"])
+ mask = stage.build_tools({"instance_id": "task-4"}, 3)
+ assert mask, "after 3 tasks of disuse the unused tools should be dropped"
+ assert not (set(mask) & reflect.PROTECTED_TOOLS), "protected tools must survive"
+ for t in ("WebFetch", "Workflow", "CronCreate"):
+ assert t in mask, f"{t} was loaded and unused for 3 tasks; expected it dropped"
+
+
+def test_static_inapplicable_prior_is_not_applied(tmp_path):
+ """An adapter that declares EVERYTHING inapplicable must not change the mask.
+
+ This is the specific regression: the stage must consult observed usage, never the
+ adapter's static `inapplicable_tools` / `hardcoded_exclusions` surface.
+ """
+ class _GreedyPrior(WebBenchAdapter):
+ def inapplicable_tools(self):
+ return tuple(_CATALOG)
+ def hardcoded_exclusions(self, used_tools):
+ return list(_CATALOG)
+
+ stage = _stage(tmp_path, _GreedyPrior("zustand"))
+ assert stage.build_tools({"instance_id": "task-1"}, 0) == [] # still unmasked
+ _finish(stage, tmp_path, 0, ["Bash", "Edit", "Read", "Write"])
+ mask = stage.build_tools({"instance_id": "task-2"}, 1)
+ assert not (set(mask) & reflect.PROTECTED_TOOLS)
+ # one task of disuse is below the evidence threshold of 3 -> nothing earned yet
+ assert mask == [], f"a static prior leaked into the mask: {mask}"
+
+
+def test_untooling_is_off_entirely_without_an_evidence_threshold(tmp_path):
+ """`evidence=0` (the old bare `--untool`) must mean NO mask, not a static one."""
+ stage = _stage(tmp_path, WebBenchAdapter("zustand"), evidence=0)
+ _finish(stage, tmp_path, 0, ["Bash"])
+ assert stage.build_tools({"instance_id": "task-2"}, 1) is None
+
+
+# ---- network tools are blocked on every arm ---------------------------------------------
+# Both benchmarks' answers are on the public internet, and on 2026-09-07 the live opus-5 /
+# sonnet-5 trees were caught fetching gold patches and grading specs. These pin the block so
+# it cannot be quietly dropped, and pin the escape hatch so it stays deliberate.
+
+def test_network_tools_are_blocked_by_default(monkeypatch):
+ from te_harness import webbench as wb, swebench as sb
+ monkeypatch.delenv("TE_ALLOW_NETWORK_TOOLS", raising=False)
+ for mod in (wb, sb):
+ assert set(mod.network_tool_block()) == {"WebSearch", "WebFetch"}, mod.__name__
+
+
+def test_network_block_is_liftable_but_only_explicitly(monkeypatch):
+ from te_harness import webbench as wb
+ for val, blocked in (("1", False), ("true", False), ("yes", False),
+ ("0", True), ("", True), ("no", True)):
+ monkeypatch.setenv("TE_ALLOW_NETWORK_TOOLS", val)
+ assert bool(wb.network_tool_block()) is blocked, val
+
+
+def test_network_tools_are_not_protected():
+ """PROTECTED_TOOLS survive every mask, so a network tool must never be listed there."""
+ from tokeneconomizer.vendor import reflect
+ assert not ({"WebSearch", "WebFetch"} & set(reflect.PROTECTED_TOOLS))
+
+
+# ---- the Web-Bench PreToolUse network guard ----------------------------------------------
+# Derived from the traces, not from taste: across 774 stored transcripts the agent made 335
+# network calls (226 package-manager, 109 loopback) and NONE mentioned GitHub, while every
+# observed abuse did. These pin both halves of that separation.
+
+def test_net_guard_blocks_the_observed_abuse():
+ from te_harness.guards.net_guard import decide
+ for cmd in (
+ "curl -sL https://raw.githubusercontent.com/bytedance/web-bench/main/"
+ "projects/threejs/test/task-7.spec.js",
+ "curl https://api.github.com/repos/bytedance/web-bench/git/trees/main?recursive=1",
+ "wget https://github.com/bytedance/web-bench/archive/main.tar.gz",
+ ):
+ blocked, reason = decide(cmd)
+ assert blocked and reason, cmd
+
+
+def test_net_guard_allows_every_legitimate_shape_seen_in_the_stored_runs():
+ from te_harness.guards.net_guard import decide
+ for cmd in ("npm install", "npx vite build", "yarn add three",
+ "curl -s http://localhost:3211/", "curl -s http://127.0.0.1:3005/index.html",
+ "ls -la src/", "npm run test -- -a"):
+ assert not decide(cmd)[0], cmd
+
+
+def test_net_guard_reads_both_agents_payload_spellings():
+ """Claude sends `tool_input`, the Copilot CLI sends `toolArgs`.
+
+ Reading only one is silent: the hook installs, fires on every call, finds no command
+ and exits 0 -- guarded-looking and guarding nothing. Measured off a live Copilot
+ preToolUse hook (CLI 1.0.83, 2026-09-08).
+ """
+ from te_harness.guards.net_guard import hook_command
+ cmd = "curl https://github.com/x/y.patch"
+ assert hook_command({"tool_name": "Bash", "tool_input": {"command": cmd}}) == cmd
+ assert hook_command({"toolName": "bash", "toolArgs": {"command": cmd}}) == cmd
+ # Nothing usable must degrade to "" rather than raising -- the hook fails open.
+ for junk in ({}, {"toolArgs": {}}, {"tool_input": None}, [], "nope", None):
+ assert hook_command(junk) == ""
+
+
+def test_net_guard_blocks_end_to_end_on_a_copilot_shaped_payload():
+ """The regression that matters: rc=2 on the Copilot spelling, not just the Claude one."""
+ import json, subprocess, sys
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ env = {k: v for k, v in os.environ.items() if k != "TE_ALLOW_NETWORK_TOOLS"}
+ for key in ("tool_input", "toolArgs"):
+ payload = json.dumps({key: {"command": "curl https://github.com/x"}})
+ r = subprocess.run([sys.executable, "-m", "te_harness.guards.net_guard"],
+ input=payload, capture_output=True, text=True,
+ cwd=root, env=env)
+ assert r.returncode == 2, f"{key}: rc={r.returncode} -- would block nothing"
+
+
+def test_net_guard_is_liftable_by_the_same_switch_as_the_tool_block(monkeypatch, tmp_path):
+ import json, subprocess, sys
+ payload = json.dumps({"tool_input": {"command": "curl https://github.com/x"}})
+ run = lambda env: subprocess.run(
+ [sys.executable, "-m", "te_harness.guards.net_guard"], input=payload,
+ capture_output=True, text=True, env=env,
+ cwd=os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+ base = dict(os.environ); base.pop("TE_ALLOW_NETWORK_TOOLS", None)
+ assert run(base).returncode == 2 # blocks, and tells the model why
+ assert run({**base, "TE_ALLOW_NETWORK_TOOLS": "1"}).returncode == 0
+
+
+def test_net_guard_fails_open_on_unreadable_input(tmp_path):
+ """A malformed hook payload must never wedge the agent."""
+ import subprocess, sys
+ r = subprocess.run([sys.executable, "-m", "te_harness.guards.net_guard"],
+ input="not json", capture_output=True, text=True,
+ cwd=os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+ assert r.returncode == 0
+
+
+# ---- the guard is installed on BOTH harnesses, not just Web-Bench ------------------------
+# Masking WebSearch/WebFetch never covered the route that mattered on SWE-bench: measured
+# across 1,824 stored rows, 120 reached the network through `Bash` against 47 through a web
+# tool, and 16 fetched an answer-key-shaped URL (14 of them resolved). `Bash` is protected
+# and can never be masked, so the PreToolUse hook is the only seam -- and for SWE-bench it
+# has to be installed INSIDE the task container, where the agent actually runs.
+
+def _method_calls(cls_name, method, path="te_harness/swebench.py"):
+ """The attribute names `cls_name.method` calls on self, via AST (no Docker needed)."""
+ import ast
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ tree = ast.parse(open(os.path.join(root, path), encoding="utf-8").read())
+ cls = next(n for n in ast.walk(tree)
+ if isinstance(n, ast.ClassDef) and n.name == cls_name)
+ fn = next(n for n in cls.body
+ if isinstance(n, ast.FunctionDef) and n.name == method)
+ return {n.func.attr for n in ast.walk(fn)
+ if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)}
+
+
+def test_the_swebench_runner_installs_and_reads_the_container_side_guard():
+ calls = _method_calls("ClaudeSwebenchRunner", "run_task")
+ assert "_install_net_guard" in calls, "SWE-bench tasks would run unguarded"
+ assert "_net_guard_blocks" in calls, "a guard you cannot watch fire is not evidence"
+
+
+def test_the_swebench_guard_honours_the_same_escape_hatch(monkeypatch):
+ """`TE_ALLOW_NETWORK_TOOLS` lifts both halves or neither -- never one silently."""
+ from te_harness import swebench as sb
+ monkeypatch.setenv("TE_ALLOW_NETWORK_TOOLS", "1")
+ runner = sb.ClaudeSwebenchRunner.__new__(sb.ClaudeSwebenchRunner)
+ assert runner._install_net_guard("no-such-container") is False
+
+
+def test_the_recorded_tool_mask_is_the_one_the_cli_was_given(monkeypatch):
+ """`run_config.tools.excluded_applied` used to omit the network block on every row,
+ which made a guarded run byte-identical to an unguarded one in its own provenance."""
+ import ast
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ for path in ("te_harness/swebench.py", "te_harness/webbench.py"):
+ tree = ast.parse(open(os.path.join(root, path), encoding="utf-8").read())
+ fn = next(n for n in ast.walk(tree)
+ if isinstance(n, ast.FunctionDef) and n.name == "_write_run_config")
+ names = {n.func.id for n in ast.walk(fn)
+ if isinstance(n, ast.Call) and isinstance(n.func, ast.Name)}
+ assert "network_tool_block" in names, path
+
+
+# ---------------------------------------------------------------------------------------
+# `goals` as a config-prunable package: the product-goal tool docs in the code-mode exec
+# description, removable with `[features] goals = false`.
+# ---------------------------------------------------------------------------------------
+def test_codex_profile_demotes_unused_goals_after_one_exposure(tmp_path):
+ p = cu.CodexUntoolProfile("sympy/sympy", str(tmp_path), fresh=True)
+ sel1 = p.begin_task("s-1")
+ assert "goals" not in sel1["config_capabilities"] # task 1: complete contract
+ p.observe({"attempted": {}, "completed": {}, "failed": {}}, resolved=True)
+ sel2 = p.begin_task("s-2")
+ # One unused exposure makes goals eligible, but at most two packages demote per task
+ # and collaboration + web go first; goals follows on the next fold.
+ assert sel2["config_capabilities"] == ["collaboration", "web"]
+ p.observe({"attempted": {}, "completed": {}, "failed": {}}, resolved=True)
+ sel3 = p.begin_task("s-3")
+ assert "goals" in sel3["config_capabilities"]
+ assert "goals" in sel3["logical_capabilities"]
+ assert "goals" not in sel3["relay_capabilities"] # config-only, no relay id
+
+
+def test_codex_profile_completed_goal_use_protects(tmp_path):
+ p = cu.CodexUntoolProfile("sympy/sympy", str(tmp_path), fresh=True)
+ p.begin_task("s-1")
+ p.observe({"attempted": {"goals": 1}, "completed": {"goals": 1}, "failed": {}},
+ resolved=True)
+ assert "goals" in p.state["ever_completed"]
+ for i in range(2, 6):
+ sel = p.begin_task(f"s-{i}")
+ assert "goals" not in sel["config_capabilities"]
+ p.observe({"attempted": {}, "completed": {}, "failed": {}}, resolved=True)
+
+
+def test_inspect_usage_counts_goal_items(tmp_path):
+ path = str(tmp_path / "t.jsonl")
+ write_codex_transcript(path, [
+ {"type": "create_goal", "id": "g1", "status": "completed", "name": "create_goal"},
+ {"type": "custom_tool_call", "id": "c1", "status": "completed", "name": "exec",
+ "input": "await tools.update_goal({id: 1});"},
+ {"type": "command_execution", "command": "ls", "aggregated_output": "", "exit_code": 0},
+ ])
+ u = cu.inspect_usage(path)
+ assert u["attempted"]["goals"] == 2 and u["completed"]["goals"] == 2
+
+
+def test_harness_emits_the_goals_toggle_and_the_cli_accepts_the_whole_mask():
+ from te_harness import codex
+ top, tables = codex.config_toggles(["goals"])
+ assert ("features", "goals = false") in tables
+ assert codex.CONFIG_CAPABILITIES == cu.CONFIG_CAPABILITIES
+ # Only worth running where the harness's CLI is installed: prove every toggle the full
+ # mask can emit is a field 0.153.1+ recognises (`--strict-config`, offline, unbilled).
+ import shutil, pytest
+ if not (os.environ.get("TE_CODEX_BIN") or shutil.which("codex")):
+ pytest.skip("codex-cli not installed")
+ routing = {"base_url": "http://127.0.0.1:9/v1", "api_key": "k"}
+ txt = codex.config_toml(routing=routing, model="gpt-5.6-terra", effort="high",
+ capabilities=sorted(codex.CONFIG_CAPABILITIES))
+ assert "goals = false" in txt
+ err = codex.validate_config(txt)
+ assert not err, err
+
+
+def test_codex_profile_demotes_interactive_guidance_after_one_exposure(tmp_path):
+ p = cu.CodexUntoolProfile("sympy/sympy", str(tmp_path), fresh=True)
+ seen = []
+ for i in range(4):
+ sel = p.begin_task(f"s-{i}")
+ seen.append(tuple(sel["config_capabilities"]))
+ p.observe({"attempted": {}, "completed": {}, "failed": {}}, resolved=True)
+ assert seen[0] == () # complete contract first
+ assert "interactive_guidance" in seen[2] # second demotion round
+ assert "interactive_guidance" not in sel["relay_capabilities"] # config-only package
diff --git a/experiments/tests/test_usage_recovery.py b/experiments/tests/test_usage_recovery.py
new file mode 100644
index 0000000..c91ff68
--- /dev/null
+++ b/experiments/tests/test_usage_recovery.py
@@ -0,0 +1,235 @@
+"""The accounting fix: a rollout killed mid-stream must not be recorded as free.
+
+Claude Code reports its token breakdown only in the stream's terminal `result` event, and
+both of `_segment`'s break paths -- a spiral guard stop and TE_TASK_TIMEOUT -- kill the CLI
+before it arrives. The segment used to return `{}`, so `_row_from_result` wrote a row of
+zeros for a rollout that had really burned hundreds of thousands of tokens over tens of
+turns. Only economized arms install a guard, so the understatement was one-directional and
+flattered exactly the arms under test.
+
+These tests are offline: they exercise the pure accounting functions with synthetic stream
+events, no Docker and no agent. `te_harness` imports nothing heavy at module level (the
+official SWE-bench grader is imported lazily), so the suite stays runnable from a fresh
+clone with no benchmark data.
+"""
+from te_harness.swebench import _UsageRecovery, _merge_usage, _row_from_result
+
+
+def _assistant(mid, inp, cw, cr, out):
+ """One `assistant` stream event in Claude Code's stream-json schema."""
+ return {"type": "assistant",
+ "message": {"id": mid,
+ "usage": {"input_tokens": inp,
+ "cache_creation_input_tokens": cw,
+ "cache_read_input_tokens": cr,
+ "output_tokens": out}}}
+
+
+def _feed(events):
+ rec = _UsageRecovery()
+ for ev in events:
+ rec.observe(ev)
+ return rec
+
+
+# ---- the bug itself ------------------------------------------------------------------
+def test_killed_stream_is_not_recorded_as_free():
+ row = _row_from_result(_feed([
+ _assistant("m1", 2, 4000, 3000, 5),
+ _assistant("m2", 2, 500, 7000, 9),
+ ]).synthesize("spiral_guard_stop"))
+ assert row["input_tokens"] == 4
+ assert row["cache_creation_input_tokens"] == 4500
+ assert row["cache_read_input_tokens"] == 10000
+ assert row["tokens_total"] > 0, "a killed rollout must never record zero cost"
+ assert row["num_turns"] == 2, "turns are recoverable; 0 was the old bug"
+ assert row["usage_incomplete"] is True
+ assert row["usage_incomplete_reason"] == "spiral_guard_stop"
+
+
+def test_usage_snapshots_are_deduplicated_on_message_id():
+ """Snapshots are cumulative per message and repeat as content blocks stream in, so
+ summing every event double-counts. Validated against 60 stored healthy rows: dedupe on
+ message.id matches their real `result` event exactly on all three input-side fields."""
+ rec = _feed([
+ _assistant("m1", 2, 100, 1000, 1),
+ _assistant("m1", 2, 400, 5000, 7), # same message, later snapshot: wins
+ _assistant("m2", 2, 50, 6000, 3),
+ ])
+ row = _row_from_result(rec.synthesize("task_timeout"))
+ assert row["cache_creation_input_tokens"] == 450 # 400 + 50, not 550
+ assert row["cache_read_input_tokens"] == 11000 # 5000 + 6000, not 12000
+ assert row["num_turns"] == 2 # two messages, not three events
+
+
+def test_output_tokens_are_reported_missing_not_estimated():
+ """The per-message `output_tokens` is the message_start placeholder, understating the
+ truth by 7.5x-134.7x across stored rows. Too variable to model, so it is reported as
+ absent and flagged -- a wrong number that looks like data is worse than a missing one."""
+ row = _row_from_result(_feed([_assistant("m1", 2, 100, 1000, 5)])
+ .synthesize("spiral_guard_stop"))
+ assert row["output_tokens"] == 0
+ assert row["output_tokens_missing"] is True
+
+
+def test_segment_that_never_ran_stays_unflagged():
+ """A segment killed before its first assistant message really did cost nothing. It must
+ come back empty so a genuinely free segment is not libelled as degraded."""
+ assert _UsageRecovery().synthesize("spiral_guard_stop") == {}
+ assert _feed([{"type": "system"}, {"type": "user"}]).synthesize("x") == {}
+
+
+def test_healthy_row_carries_the_flags_as_false():
+ """Present-and-False, so "no flag" means "written by a harness that did not check"
+ rather than "verified complete"."""
+ row = _row_from_result({
+ "type": "result", "num_turns": 12,
+ "usage": {"input_tokens": 10, "cache_creation_input_tokens": 20,
+ "cache_read_input_tokens": 30, "output_tokens": 40}})
+ assert row["usage_incomplete"] is False
+ assert row["output_tokens_missing"] is False
+ assert row["num_turns_recovered"] is False
+ assert row["output_tokens"] == 40, "a healthy row must be untouched by the fix"
+
+
+# ---- the spiral case: one bad segment degrades the whole task -------------------------
+HEALTHY = {"type": "result", "num_turns": 4,
+ "usage": {"input_tokens": 10, "cache_creation_input_tokens": 20,
+ "cache_read_input_tokens": 30, "output_tokens": 40}}
+
+
+def test_flags_survive_a_spiral_merge():
+ """The common spiral shape: segment 1 completes, segment 2 is killed by the guard.
+ `_merge_usage` builds on the SECOND segment, so the flags have to be OR-ed in or the
+ marker is cleared on precisely the case that produces it."""
+ killed = _feed([_assistant("m1", 2, 100, 1000, 3)]).synthesize("spiral_guard_stop")
+ merged = _row_from_result(_merge_usage(HEALTHY, killed))
+ assert merged["usage_incomplete"] is True
+ assert merged["num_turns"] == 5 # 4 + 1, summed not replaced
+ assert merged["cache_read_input_tokens"] == 1030 # both segments billed
+ # ...and in the other order, since a nudge can be stopped on the first segment.
+ assert _row_from_result(_merge_usage(killed, HEALTHY))["usage_incomplete"] is True
+
+
+def test_merge_of_two_healthy_segments_stays_clean():
+ assert _row_from_result(_merge_usage(HEALTHY, HEALTHY))["usage_incomplete"] is False
+
+
+def test_merge_reason_names_every_degraded_segment():
+ a = _feed([_assistant("m1", 2, 1, 1, 1)]).synthesize("spiral_guard_stop")
+ b = _feed([_assistant("m2", 2, 1, 1, 1)]).synthesize("task_timeout")
+ assert _merge_usage(a, b)["usage_incomplete_reason"] == "spiral_guard_stop+task_timeout"
+
+
+# ---- the guard-parameter probe (the webbench TypeError) -------------------------------
+def test_guard_accepts_probes_the_vendored_signature():
+ """`never_abort` exists in some vendored versions of the guard and not in others.
+ webbench.py passed it unconditionally, which killed every guarded Web-Bench run at
+ task 0 before a row was written. Both runners now go through this one probe."""
+ from tokeneconomizer.vendor import spiral as spiral_vendor
+ from te_harness.swebench import guard_accepts
+ assert guard_accepts("envelope") is True
+ assert guard_accepts("no_such_parameter_exists") is False
+ # whatever this checkout's guard takes, the probe agrees with it
+ import inspect
+ params = inspect.signature(spiral_vendor.NudgeGuard.__init__).parameters
+ assert guard_accepts("never_abort") == ("never_abort" in params)
+
+
+def test_webbench_guard_constructs_on_this_checkout():
+ """The regression itself: `_make_guard` must not raise whatever the vendored guard's
+ signature is. It takes no `self` state, so it is callable unbound."""
+ from te_harness.webbench import WebBenchRunner
+ guard = WebBenchRunner._make_guard(None, {
+ "spiral_guard": True, "spiral_nudge": True, "spiral_never_abort": True,
+ "abort_envelope": 30.0})
+ assert guard is not None and guard.nudge_budget == 1
+ assert WebBenchRunner._make_guard(None, {}) is None, "guard off -> no guard"
+
+
+# ---- session-log recovery: output_tokens survives the kill after all -----------------
+# The stream's per-message `output_tokens` is a `message_start` placeholder, and the
+# terminal `result` event never arrives on a kill (measured 2026-09-10: SIGTERM to a
+# mid-stream `claude -p` produces NO result event). But the CLI persists each assistant
+# message WITH its real usage as that message completes, so every turn but an in-flight
+# one is exactly recoverable from that log.
+import json as _json
+
+from te_harness.swebench import session_output_tokens
+
+
+def _session_lines(pairs, duplicate=True):
+ """Claude Code's session log. Each message is written TWICE by the real CLI -- same
+ `message.id`, different `uuid` -- which is why the parser must deduplicate."""
+ out = []
+ for mid, out_tok in pairs:
+ for uu in (("a", "b") if duplicate else ("a",)):
+ out.append(_json.dumps({"type": "assistant", "uuid": f"{mid}-{uu}",
+ "message": {"id": mid,
+ "usage": {"output_tokens": out_tok}}}))
+ return "\n".join(out)
+
+
+def test_session_log_parser_deduplicates_the_double_write():
+ """The trap: summing entries returns exactly 2x the truth, which looks plausible."""
+ text = _session_lines([("m1", 421)])
+ assert len(text.splitlines()) == 2 # the CLI really does write it twice
+ total, ids = session_output_tokens(text)
+ assert (total, ids) == (421, {"m1"}) # NOT 842
+
+
+def test_session_log_parser_never_raises_on_junk():
+ for junk in ("", "not json", '{"no":"message"}', '{"message":{"id":"m"}}',
+ '{"message":{"id":"m","usage":{"output_tokens":true}}}'):
+ assert session_output_tokens(junk) == (0, set())
+
+
+def test_output_tokens_recovered_exactly_when_every_turn_completed():
+ """A spiral stop lands between turns, so the log accounts for all of them and the row
+ is a MEASUREMENT again -- not a lower bound to be filtered out."""
+ rec = _feed([_assistant("m1", 2, 4000, 3000, 5), _assistant("m2", 2, 500, 7000, 9)])
+ row = _row_from_result(rec.synthesize(
+ "spiral_guard_stop", _session_lines([("m1", 1200), ("m2", 800)])))
+ assert row["output_tokens"] == 2000 # exact, not the 5+9 placeholders
+ assert row["output_tokens_exact_turns"] == 2
+ assert row["output_tokens_missing_turn"] is False
+ assert row["usage_incomplete"] is False # proven complete -> not flagged
+ assert row["output_tokens_missing"] is False
+ assert row["usage_incomplete_reason"] is None
+
+
+def test_an_in_flight_turn_keeps_the_row_flagged():
+ """A TE_TASK_TIMEOUT can land mid-generation: the stream saw m3 start, the log has no
+ figure for it, so the row is a lower bound and must still say so."""
+ rec = _feed([_assistant("m1", 2, 10, 20, 5), _assistant("m2", 2, 10, 20, 7),
+ _assistant("m3", 2, 10, 20, 3)])
+ row = _row_from_result(rec.synthesize(
+ "task_timeout", _session_lines([("m1", 1200), ("m2", 800)])))
+ assert row["output_tokens"] == 2000 # the two turns we can prove
+ assert row["output_tokens_exact_turns"] == 2
+ assert row["output_tokens_missing_turn"] is True
+ assert row["usage_incomplete"] is True # still a lower bound
+ assert row["usage_incomplete_reason"] == "task_timeout"
+
+
+def test_no_session_log_falls_back_to_the_old_honest_behaviour():
+ """An unreadable log must degrade to `output_tokens=0` + flagged, never to a crash."""
+ rec = _feed([_assistant("m1", 2, 4000, 3000, 5)])
+ for text in (None, "", "garbage"):
+ row = _row_from_result(rec.synthesize("spiral_guard_stop", text))
+ assert row["output_tokens"] == 0
+ assert row["usage_incomplete"] is True
+ assert row["output_tokens_missing"] is True
+ assert row["output_tokens_exact_turns"] == 0
+
+
+def test_exact_turns_add_across_spiral_segments_while_flags_or():
+ """A nudged task has two segments. Counts add; degradation ORs."""
+ clean = _feed([_assistant("m1", 2, 10, 20, 5)]).synthesize(
+ "spiral_guard_stop", _session_lines([("m1", 500)]))
+ partial = _feed([_assistant("m2", 2, 10, 20, 5), _assistant("m3", 2, 10, 20, 5)]
+ ).synthesize("spiral_guard_stop", _session_lines([("m2", 700)]))
+ merged = _merge_usage(clean, partial)
+ assert merged["output_tokens_exact_turns"] == 2 # 1 + 1
+ assert merged["output_tokens_missing_turn"] is True # from the second segment
+ assert merged["usage_incomplete"] is True
diff --git a/experiments/tests/test_vendor_cost.py b/experiments/tests/test_vendor_cost.py
new file mode 100644
index 0000000..7f2b9c5
--- /dev/null
+++ b/experiments/tests/test_vendor_cost.py
@@ -0,0 +1,114 @@
+"""The vendor-dollar column: read the key the runners actually write, and flag its absence.
+
+Two defects, both of which made a cost comparison quietly wrong rather than loudly broken:
+
+ * `aggregate.py` emitted `cost_usd` from `d.get("cost_usd")`, but every runner writes the
+ figure as `cost_usd_vendor`. The column was therefore EMPTY in every tree this repo
+ generates, while looking like a column that simply had no data yet.
+
+ * A row whose agent reported no `total_cost_usd` stored None and was marked by nothing:
+ `usage_incomplete` stays False (the token breakdown is complete and the effective-token
+ basis is exact) and the task grades normally, so the row reads as healthy. Summing the
+ dollar column then under-counts, because the row's tokens are in the denominator while
+ its dollars are not -- and because the misses cluster on the longest trajectories, the
+ error lands on the most expensive tasks.
+
+These tests are offline: they call the two pure functions with synthetic rows.
+"""
+import importlib.util
+import os
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_AGGREGATE = os.path.join(_REPO, "scripts", "aggregate.py")
+
+
+def _agg():
+ spec = importlib.util.spec_from_file_location("_aggregate", _AGGREGATE)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def _ran(**over):
+ """A row that demonstrably ran: it resolved and it burned tokens."""
+ row = {"agent": "claude", "resolved": True, "tokens_total": 400_000,
+ "input_tokens": 1_600, "patch_bytes": 900}
+ row.update(over)
+ return row
+
+
+# --- vendor_usd: the source key ----------------------------------------------------
+
+def test_vendor_usd_reads_the_key_the_runners_write():
+ assert _agg().vendor_usd(_ran(cost_usd_vendor=0.2239)) == 0.2239
+
+
+def test_vendor_usd_falls_back_to_the_legacy_curated_name():
+ # The older curated exports carry the figure as plain `cost_usd`; those trees must
+ # keep working.
+ assert _agg().vendor_usd({"cost_usd": 1.5}) == 1.5
+
+
+def test_vendor_usd_prefers_the_runner_key_when_both_are_present():
+ assert _agg().vendor_usd({"cost_usd_vendor": 2.0, "cost_usd": 9.9}) == 2.0
+
+
+def test_vendor_usd_is_blank_not_zero_when_absent():
+ # "" and 0.0 must not be confused: one is an unknown, the other is a measurement.
+ assert _agg().vendor_usd({"cost_usd_vendor": None}) == ""
+ assert _agg().vendor_usd({}) == ""
+
+
+def test_vendor_usd_keeps_a_genuine_zero():
+ assert _agg().vendor_usd({"cost_usd_vendor": 0.0}) == 0.0
+
+
+def test_vendor_usd_rejects_a_bool():
+ # bool is an int subclass; True must not read as $1.
+ assert _agg().vendor_usd({"cost_usd_vendor": True}) == ""
+
+
+# --- vendor_cost_missing: the flag -------------------------------------------------
+
+def test_flags_a_row_that_ran_without_a_figure():
+ missing, reason = _agg().vendor_cost_missing(_ran(cost_usd_vendor=None))
+ assert missing is True and "total_cost_usd" in reason
+
+
+def test_does_not_flag_a_row_that_has_a_figure():
+ assert _agg().vendor_cost_missing(_ran(cost_usd_vendor=0.5))[0] is False
+
+
+def test_does_not_flag_an_agent_that_never_reports_dollars():
+ # Copilot meters premium requests and Codex bills a subscription: null is the
+ # documented normal case there (docs/COPILOT.md), not a defect.
+ for agent in ("copilot", "codex"):
+ row = _ran(agent=agent, cost_usd_vendor=None)
+ assert _agg().vendor_cost_missing(row)[0] is False, agent
+
+
+def test_does_not_flag_a_task_that_never_ran():
+ # An UNREACHED Web-Bench row: no verdict, no patch, no tokens. It has no cost because
+ # nothing happened, which is not the same as a lost measurement.
+ row = {"agent": "claude", "resolved": False, "tokens_total": 0,
+ "input_tokens": 0, "patch_bytes": None, "cost_usd_vendor": None}
+ assert _agg().vendor_cost_missing(row)[0] is False
+
+
+def test_flags_an_unresolved_row_that_still_burned_tokens():
+ # The expensive case: a long trajectory that failed. Dropping it understates the arm.
+ row = _ran(resolved=False, patch_bytes=0, cost_usd_vendor=None)
+ assert _agg().vendor_cost_missing(row)[0] is True
+
+
+def test_flag_is_independent_of_usage_incomplete():
+ # The row's token accounting is complete -- that is exactly why it needs its own flag.
+ agg = _agg()
+ row = _ran(cost_usd_vendor=None)
+ assert agg.usage_incomplete(row)[0] is False
+ assert agg.vendor_cost_missing(row)[0] is True
+
+
+def test_both_columns_are_emitted():
+ for col in ("cost_usd", "vendor_cost_missing", "vendor_cost_missing_reason"):
+ assert col in _agg().FIELDS, col
diff --git a/experiments/tests/test_workloads.py b/experiments/tests/test_workloads.py
new file mode 100644
index 0000000..e4c79ac
--- /dev/null
+++ b/experiments/tests/test_workloads.py
@@ -0,0 +1,305 @@
+"""Workloads: the recorded task set stays pinned to the evaluated per-group sizes.
+
+`workloads` is the only place the *identity* of the evaluated tasks is written down, so
+these tests guard it against drift: the per-group sizes below are the evaluated set, and an
+edit that moves one of them is a change to the record, not a refactor.
+"""
+import importlib.util
+import json
+import os
+
+from tokeneconomizer import workloads
+
+_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_EXPORTER = os.path.join(_REPO, "scripts", "export_workloads.py")
+_WORKLOAD_DIR = os.path.join(_REPO, "workloads")
+
+
+def _exporter():
+ spec = importlib.util.spec_from_file_location("_export_workloads", _EXPORTER)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def test_swebench_counts_are_the_evaluated_set():
+ assert workloads.task_counts("swebench") == {
+ "django/django": 40, "matplotlib/matplotlib": 34, "scikit-learn/scikit-learn": 32,
+ "sphinx-doc/sphinx": 44, "sympy/sympy": 40, "pydata/xarray": 39,
+ }
+
+
+def test_webbench_2026_08_export_is_frozen():
+ """A record of a run that already happened, pinned on its own rather than to the live
+ sweep set -- which has since retired `flex` and `prisma`."""
+ assert workloads.WEBBENCH_TASKS_PUBLISHED_2026_08 == {
+ "flex": 21, "expressjs": 20, "react": 20, "prisma": 20, "calculator": 21,
+ }
+
+
+def test_webbench_sweep_set_is_five_distinct_streams():
+ """The live sweep set. Its IDENTITY is the invariant, not just its size: Web-Bench reuses
+ one task stream across whole project families, so a well-meant substitution can silently
+ drop the suite to four distinct streams (see the note on WEBBENCH_TASKS)."""
+ counts = workloads.task_counts("webbench")
+ assert set(counts) == {"calculator", "expressjs", "react", "zustand", "table"}
+ assert sum(counts.values()) == 102
+ # 20 graded tasks each; `calculator` and `table` additionally open with `init`.
+ assert {p for p, n in counts.items() if n == 21} == {"calculator", "table"}
+ assert all(n in (20, 21) for n in counts.values())
+
+
+def test_totals():
+ assert workloads.total_tasks("swebench") == 229
+ assert workloads.total_tasks("webbench") == 102
+ assert workloads.total_tasks("creddata") == 100
+ assert workloads.total_tasks("reposvul") == 100
+
+
+def test_no_duplicate_ids_within_a_group():
+ """Ids are unique per group. Web-Bench reuses `task-N` across projects by design."""
+ for domain, groups in (("swebench", workloads.SWEBENCH_INSTANCES),
+ ("webbench", workloads.WEBBENCH_TASKS)):
+ for group in groups:
+ ids = workloads.instances(domain, group)
+ assert len(ids) == len(set(ids)), f"{domain}/{group} has duplicate task ids"
+ for domain in ("creddata", "reposvul"):
+ ids = workloads.instances(domain)
+ assert len(ids) == len(set(ids)), f"{domain} has duplicate task ids"
+
+
+def test_full_split_only_is_a_subset_and_all_xarray():
+ swebench = set(workloads.instances("swebench"))
+ assert workloads.SWEBENCH_FULL_SPLIT_ONLY <= swebench
+ assert len(workloads.SWEBENCH_FULL_SPLIT_ONLY) == 30
+ assert all(i.startswith("pydata__xarray-") for i in workloads.SWEBENCH_FULL_SPLIT_ONLY)
+
+
+def test_added_extends_sklearn_and_sympy_to_sixty_runnable():
+ """The de-saturation extension: both streams run 60 tasks, the rest are untouched."""
+ runnable = workloads.runnable_instances("swebench")
+ assert len(runnable["scikit-learn/scikit-learn"]) == 60
+ assert len(runnable["sympy/sympy"]) == 60
+ for group in ("django/django", "matplotlib/matplotlib", "sphinx-doc/sphinx",
+ "pydata/xarray"):
+ assert len(runnable[group]) == len(workloads.instances("swebench", group))
+
+
+def test_added_ids_are_new_and_grouped_under_their_own_repo():
+ canonical = set(workloads.instances("swebench"))
+ for group, ids in workloads.SWEBENCH_ADDED.items():
+ assert group in workloads.SWEBENCH_INSTANCES, f"{group} is not a known repo"
+ assert not (set(ids) & canonical), f"{group}: added ids overlap the published set"
+ assert len(ids) == len(set(ids)), f"{group}: duplicate added ids"
+
+
+def test_exclusion_still_wins_over_the_extension():
+ """14710 stays out of the 60-task sklearn stream: it does not terminate under eval."""
+ sklearn = workloads.runnable_instances("swebench", "scikit-learn/scikit-learn")
+ assert "scikit-learn__scikit-learn-14710" not in sklearn
+ assert "scikit-learn__scikit-learn-14710" in workloads.instances(
+ "swebench", "scikit-learn/scikit-learn")
+
+
+def test_added_does_not_move_the_published_record():
+ """The evaluated set is 229 tasks; additions must not touch it."""
+ assert workloads.total_tasks("swebench") == 229
+ assert workloads.task_counts("swebench")["scikit-learn/scikit-learn"] == 32
+ assert workloads.task_counts("swebench")["sympy/sympy"] == 40
+
+
+def test_added_ids_need_the_full_split():
+ """Both repos' Verified rows were already spent, so every addition is full-split-only."""
+ added = {i for ids in workloads.SWEBENCH_ADDED.values() for i in ids}
+ assert workloads.SWEBENCH_ADDED_FULL_SPLIT_ONLY == added
+ assert not (added & workloads.SWEBENCH_FULL_SPLIT_ONLY)
+
+
+def test_every_added_instance_yields_a_tag():
+ for ids in workloads.SWEBENCH_ADDED.values():
+ for iid in ids:
+ assert workloads.swebench_image_tag(iid).endswith(":latest")
+
+
+def test_copilot_extra_is_not_in_the_canonical_set():
+ assert not (workloads.SWEBENCH_COPILOT_EXTRA & set(workloads.instances("swebench")))
+
+
+def test_image_tag_uses_the_1776_separator():
+ tag = workloads.swebench_image_tag("django__django-11532")
+ assert tag == "swebench/sweb.eval.x86_64.django_1776_django-11532:latest"
+ assert "__" not in tag.split("x86_64.", 1)[1]
+
+
+def test_every_swebench_instance_yields_a_tag():
+ for iid in workloads.instances("swebench"):
+ assert workloads.swebench_image_tag(iid).endswith(":latest")
+
+
+def test_instances_rejects_unknown_domain_and_group():
+ for bad in (("nope", None), ("swebench", "pydata/xarray-typo")):
+ try:
+ workloads.instances(*bad)
+ except KeyError:
+ continue
+ raise AssertionError(f"expected KeyError for {bad}")
+
+
+def test_filter_tasks_keeps_order_and_drops_strangers():
+ ids = workloads.instances("swebench", "pydata/xarray")
+ tasks = [{"instance_id": "not-a-real-instance"}] + [{"instance_id": i} for i in ids]
+ kept = workloads.filter_tasks(tasks, "swebench", "pydata/xarray")
+ assert [t["instance_id"] for t in kept] == list(ids)
+
+
+def test_filter_tasks_returns_empty_rather_than_falling_back():
+ """The driver's --repo filter falls back to the full list on no match; this must not."""
+ assert workloads.filter_tasks([{"instance_id": "x"}], "swebench", "sympy/sympy") == []
+
+
+def test_exported_json_exists_for_every_domain():
+ for domain in workloads.DOMAINS:
+ path = os.path.join(_WORKLOAD_DIR, f"{domain}.json")
+ assert os.path.isfile(path), f"missing {path}; run python scripts/export_workloads.py"
+
+
+def test_exported_json_matches_the_module():
+ """workloads/*.json is generated from workloads.py — regenerate it when the data moves."""
+ build = _exporter().build
+ for domain in workloads.DOMAINS:
+ with open(os.path.join(_WORKLOAD_DIR, f"{domain}.json"), encoding="utf-8") as fh:
+ on_disk = json.load(fh)
+ assert on_disk == build(domain), (
+ f"workloads/{domain}.json is stale; run python scripts/export_workloads.py")
+
+
+def test_exported_json_is_small_enough_to_commit():
+ """These are ids only (no issue text), so they stay comfortably committable."""
+ for domain in workloads.DOMAINS:
+ size = os.path.getsize(os.path.join(_WORKLOAD_DIR, f"{domain}.json"))
+ assert size < 200_000, f"workloads/{domain}.json grew to {size} bytes"
+
+
+def test_exported_json_group_counts_agree_with_task_counts():
+ for domain in workloads.DOMAINS:
+ with open(os.path.join(_WORKLOAD_DIR, f"{domain}.json"), encoding="utf-8") as fh:
+ doc = json.load(fh)
+ assert doc["task_counts"] == {g: len(v) for g, v in doc["groups"].items()}
+ assert doc["total_tasks"] == sum(doc["task_counts"].values())
+
+
+def test_webbench_streams_are_contiguous():
+ """Each stream is an optional `init` followed by task-1..task-N with no gaps."""
+ for project, tasks in workloads.WEBBENCH_TASKS.items():
+ numbered = list(tasks[1:]) if tasks[0] == "init" else list(tasks)
+ assert numbered == [f"task-{n}" for n in range(1, len(numbered) + 1)], project
+
+
+# ---- the two Web-Bench registries that live OUTSIDE the repo -----------------------------
+# `workloads/*.json` is committed, so the test above can diff it. The driver task list (A)
+# and the projects tree (B) are not: A is written to $TOKENECONOMIZER_WEBBENCH_TASKS and B is
+# the Web-Bench checkout. Both are projections of WEBBENCH_TASKS and both fail LATE and
+# confusingly when stale -- a desynced A surfaces mid-run as `session is running 'X' but the
+# task belongs to 'Y'`. These skip when the env is not configured, so the suite stays
+# offline-clean, and bite on any machine that can actually launch a run.
+
+def _projects_root():
+ return os.environ.get("TOKENECONOMIZER_WEBBENCH_PROJECTS")
+
+
+def test_every_swept_webbench_project_has_a_tasks_jsonl():
+ """(B) The harness and the adapter both read `tasks.jsonl` only; a yml-only project must
+ have had one derived (scripts/webbench_tasks_from_yaml.py)."""
+ root = _projects_root()
+ if not root or not os.path.isdir(root):
+ return # checkout not present; nothing to verify
+ for project, task_ids in workloads.WEBBENCH_TASKS.items():
+ path = os.path.join(root, project, "tasks.jsonl")
+ assert os.path.isfile(path), (
+ f"{project}: no tasks.jsonl -- run "
+ f"python scripts/webbench_tasks_from_yaml.py {project}")
+ have = set()
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ if line.strip():
+ have.add(json.loads(line)["id"])
+ missing = [t for t in task_ids if t not in have]
+ assert not missing, f"{project}: tasks.jsonl lacks {missing} named by WEBBENCH_TASKS"
+
+
+def test_driver_task_list_is_not_stale():
+ """(A) $TOKENECONOMIZER_WEBBENCH_TASKS must match what WEBBENCH_TASKS projects to."""
+ root, out = _projects_root(), os.environ.get("TOKENECONOMIZER_WEBBENCH_TASKS")
+ if not (root and os.path.isdir(root) and out and os.path.isfile(out)):
+ return
+ spec = importlib.util.spec_from_file_location(
+ "_export_webbench_tasks", os.path.join(_REPO, "scripts", "export_webbench_tasks.py"))
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ with open(out, encoding="utf-8") as fh:
+ on_disk = json.load(fh)
+ assert on_disk == mod.build(root), (
+ f"{out} is stale; run python scripts/export_webbench_tasks.py")
+
+
+# ---- Web-Bench's per-segment cap must stay under the evaluator's request abort ------------
+# TE_TASK_TIMEOUT is shared with SWE-bench, where 3600 is correct; on Web-Bench anything past
+# the evaluator's ~300s undici abort loses whole attempts and truncates the stream. Measured
+# 2026-09-07: it silently killed two threejs streams at task-14 before the clamp existed.
+
+def test_webbench_task_timeout_is_clamped_under_the_evaluator_abort():
+ from te_harness.webbench import (webbench_task_timeout, WEBBENCH_TASK_TIMEOUT_MAX,
+ WEBBENCH_UNDICI_ABORT_S)
+ assert WEBBENCH_TASK_TIMEOUT_MAX < WEBBENCH_UNDICI_ABORT_S
+ for shared in ("3600", "1800", "301"):
+ assert webbench_task_timeout({"TE_TASK_TIMEOUT": shared}) == WEBBENCH_TASK_TIMEOUT_MAX
+
+
+def test_webbench_task_timeout_defaults_and_overrides():
+ from te_harness.webbench import webbench_task_timeout, WEBBENCH_TASK_TIMEOUT_DEFAULT
+ assert webbench_task_timeout({}) == WEBBENCH_TASK_TIMEOUT_DEFAULT
+ # a domain-specific value wins over the shared one, and a low value is honoured as-is
+ assert webbench_task_timeout({"TE_TASK_TIMEOUT": "3600",
+ "TE_WEBBENCH_TASK_TIMEOUT": "200"}) == 200
+ # junk must fall back, never crash a lane at task 0
+ assert webbench_task_timeout({"TE_TASK_TIMEOUT": "abc"}) == WEBBENCH_TASK_TIMEOUT_DEFAULT
+
+
+def test_swebench_task_timeout_is_untouched_by_the_webbench_clamp():
+ """SWE-bench genuinely needs 3600 (sympy pinned 1800 dead-on); the clamp is domain-local."""
+ import te_harness.swebench as sb
+ assert "WEBBENCH_TASK_TIMEOUT_MAX" not in dir(sb)
+
+
+# ---- the abort ceiling is raisable, because the evaluator's bare `fetch` uses undici's
+# GLOBAL dispatcher, which `--import te_harness/webbench_undici_preload.mjs` can reset.
+# Opt-in only: unset, every value below is what the fixed 300/260 constants produced.
+
+def test_webbench_undici_abort_defaults_to_the_stock_ceiling_and_adds_no_env():
+ from te_harness.webbench import (webbench_undici_abort, webbench_undici_env,
+ WEBBENCH_UNDICI_ABORT_DEFAULT_S)
+ assert webbench_undici_abort({}) == WEBBENCH_UNDICI_ABORT_DEFAULT_S
+ assert webbench_undici_env({}) == {} # stock spawn must be byte-identical
+ assert webbench_undici_env({"TE_WEBBENCH_UNDICI_ABORT": "300"}) == {}
+
+
+def test_raising_the_abort_ceiling_raises_the_cap_and_preloads_the_dispatcher():
+ import os
+ from te_harness.webbench import (webbench_task_timeout, webbench_undici_abort,
+ webbench_undici_env, UNDICI_PRELOAD,
+ WEBBENCH_TASK_TIMEOUT_HEADROOM_S)
+ env = {"TE_WEBBENCH_UNDICI_ABORT": "900", "TE_TASK_TIMEOUT": "3600"}
+ assert webbench_undici_abort(env) == 900
+ assert webbench_task_timeout(env) == 900 - WEBBENCH_TASK_TIMEOUT_HEADROOM_S
+ extra = webbench_undici_env(env, repo="/tmp/wb")
+ assert extra["TE_WEBBENCH_UNDICI_ABORT_MS"] == "900000"
+ assert "--import" in extra["NODE_OPTIONS"] and "webbench_undici_preload" in extra["NODE_OPTIONS"]
+ assert os.path.exists(UNDICI_PRELOAD)
+
+
+def test_raising_the_ceiling_preserves_an_existing_node_options():
+ from te_harness.webbench import webbench_undici_env
+ extra = webbench_undici_env({"TE_WEBBENCH_UNDICI_ABORT": "600",
+ "NODE_OPTIONS": "--max-old-space-size=4096"})
+ assert extra["NODE_OPTIONS"].startswith("--max-old-space-size=4096 ")
+ assert "--import" in extra["NODE_OPTIONS"]
diff --git a/experiments/workloads/README.md b/experiments/workloads/README.md
new file mode 100644
index 0000000..2ee0bfd
--- /dev/null
+++ b/experiments/workloads/README.md
@@ -0,0 +1,53 @@
+# workloads/ — the exact evaluated task sets (committed)
+
+One JSON manifest per domain naming **which** tasks the published results ran. Unlike the
+results tree (regenerated per run, git-ignored), these are **committed**: they are ids only —
+no issue text, no transcripts — so all four together are about 16 KB.
+
+| File | Tasks | Groups | Seeds |
+|---|---|---|---|
+| `swebench.json` | 229 | 6 repos | `0 1 2` |
+| `webbench.json` | 102 | 5 projects | `A B C` |
+| `creddata.json` | 100 | 1 | `0 1 2` |
+| `reposvul.json` | 100 | 1 | `0 1 2` |
+
+## Shape
+
+```json
+{
+ "domain": "swebench",
+ "group_field": "repo",
+ "seeds": [0, 1, 2],
+ "total_tasks": 229,
+ "task_counts": {"django/django": 40, "...": 0},
+ "groups": {"django/django": ["django__django-7530", "..."]},
+ "full_split_only": ["pydata__xarray-6386", "..."]
+}
+```
+
+`groups` preserves order. For Web-Bench that order is **load-bearing** — each task builds on
+the previous one's edits, so a stream must run front to back.
+
+`swebench.json` additionally carries `full_split_only`: the 30 instances that are **not** in
+SWE-bench_Verified and must come from the full `princeton-nlp/SWE-bench` test split.
+
+## Who reads them
+
+- **`scripts/launch/*.sh`** default `WORKLOAD` to `workloads/$DOMAIN.json` and pass it to the
+ driver, so a launch is pinned to the evaluated set even if your `TASKS_FILE` holds more.
+ `WORKLOAD=none` opts out.
+- **`scripts/run_experiment.py --workload`** filters `--tasks-file` to the manifest (and to
+ `--repo`'s group). Unlike the legacy `--repo` prefix match, an unknown group is an error
+ rather than a silent fall back to the whole task list.
+- **Any non-Python harness** that needs the list without importing the package.
+
+## Regenerating
+
+`tokeneconomizer/workloads.py` is the authority; these files are generated from it:
+
+```bash
+python scripts/export_workloads.py # rewrite
+python scripts/export_workloads.py --check # verify (exit 1 if stale)
+```
+
+`tests/test_workloads.py` runs the same comparison, so drift fails the test suite.
diff --git a/experiments/workloads/creddata.json b/experiments/workloads/creddata.json
new file mode 100644
index 0000000..feae27a
--- /dev/null
+++ b/experiments/workloads/creddata.json
@@ -0,0 +1,119 @@
+{
+ "domain": "creddata",
+ "description": "CredData repositories audited for secrets, graded to line-level F1.",
+ "source": "tokeneconomizer/workloads.py (regenerate: python scripts/export_workloads.py)",
+ "group_field": "instance",
+ "seeds": [
+ 0,
+ 1,
+ 2
+ ],
+ "total_tasks": 100,
+ "task_counts": {
+ "creddata": 100
+ },
+ "groups": {
+ "creddata": [
+ "00408ef6",
+ "0401c075",
+ "08295bf0",
+ "0a0d22aa",
+ "0ce78112",
+ "110f8d35",
+ "16fc0238",
+ "262e8107",
+ "28728ab4",
+ "29d5ecae",
+ "30201159",
+ "31423103",
+ "353ae7fa",
+ "35c273e2",
+ "371ea6d8",
+ "39def7b4",
+ "3d82970b",
+ "443086e9",
+ "46c52073",
+ "489fc16c",
+ "4db30bbe",
+ "4f40bf46",
+ "50b38ef0",
+ "55031e17",
+ "55aa4858",
+ "57067d8d",
+ "5cb346c3",
+ "5cecf769",
+ "5d8870b3",
+ "60f9915d",
+ "63800a10",
+ "638eb0c6",
+ "653a7746",
+ "660708f3",
+ "661abd43",
+ "69f42949",
+ "6a132a15",
+ "6d59a88a",
+ "6e2ed0e4",
+ "7529016f",
+ "75e7c64d",
+ "7c7558cf",
+ "7e01f352",
+ "7f8b28a7",
+ "81cd05d0",
+ "826a020b",
+ "848ad017",
+ "86875835",
+ "8bc560cd",
+ "8c3c80e8",
+ "8c9fcde7",
+ "8d217d6c",
+ "8f499d55",
+ "9031b2b4",
+ "91cfdb2b",
+ "93ff401b",
+ "98a9ba4e",
+ "9c7e3662",
+ "9ccb72ee",
+ "9d90ba70",
+ "9ef45116",
+ "a0cd6261",
+ "a15774b8",
+ "a5c3685e",
+ "a8bdca79",
+ "ac9be8d9",
+ "adf34139",
+ "afdd01d8",
+ "b6dbde6a",
+ "b81d3227",
+ "b8d8b8c3",
+ "bbb4193f",
+ "bcb18a48",
+ "bd7bcb9d",
+ "c2127ffb",
+ "c9b945fa",
+ "cc268724",
+ "cd82f1a2",
+ "cfdcc4f3",
+ "d2d6eedc",
+ "d518da13",
+ "d93c01fb",
+ "db218f09",
+ "db832824",
+ "df70e863",
+ "e08ff555",
+ "e097ae9a",
+ "e3c63910",
+ "e69fe1b9",
+ "eb8448ad",
+ "eb9ba732",
+ "f129be24",
+ "f43ddbf5",
+ "f710ac3c",
+ "f7ae636a",
+ "f80c571b",
+ "f85f83c9",
+ "fc8343f4",
+ "ff43ce4d",
+ "ff527e6f"
+ ]
+ }
+}
diff --git a/experiments/workloads/reposvul.json b/experiments/workloads/reposvul.json
new file mode 100644
index 0000000..1523b59
--- /dev/null
+++ b/experiments/workloads/reposvul.json
@@ -0,0 +1,119 @@
+{
+ "domain": "reposvul",
+ "description": "ReposVul instances (Python split) audited for vulnerabilities, graded to F1.",
+ "source": "tokeneconomizer/workloads.py (regenerate: python scripts/export_workloads.py)",
+ "group_field": "instance",
+ "seeds": [
+ 0,
+ 1,
+ 2
+ ],
+ "total_tasks": 100,
+ "task_counts": {
+ "reposvul": 100
+ },
+ "groups": {
+ "reposvul": [
+ "py-10081",
+ "py-10115",
+ "py-1045",
+ "py-10585",
+ "py-10746",
+ "py-10878",
+ "py-10883",
+ "py-11295",
+ "py-11519",
+ "py-11621",
+ "py-11682",
+ "py-11868",
+ "py-11990",
+ "py-12077",
+ "py-12275",
+ "py-12277",
+ "py-12279",
+ "py-12292",
+ "py-12343",
+ "py-12375",
+ "py-12440",
+ "py-12497",
+ "py-171",
+ "py-1968",
+ "py-1974",
+ "py-211",
+ "py-2210",
+ "py-2496",
+ "py-2656",
+ "py-2660",
+ "py-2754",
+ "py-2771",
+ "py-2847",
+ "py-3163",
+ "py-3380",
+ "py-3482",
+ "py-3596",
+ "py-369",
+ "py-3802",
+ "py-4061",
+ "py-4374",
+ "py-4450",
+ "py-4597",
+ "py-4615",
+ "py-4816",
+ "py-49",
+ "py-5016",
+ "py-5203",
+ "py-5259",
+ "py-5557",
+ "py-5566",
+ "py-5567",
+ "py-5569",
+ "py-569",
+ "py-5751",
+ "py-5838",
+ "py-5897",
+ "py-5934",
+ "py-5969",
+ "py-5987",
+ "py-6002",
+ "py-6200",
+ "py-623",
+ "py-6237",
+ "py-6510",
+ "py-6540",
+ "py-6565",
+ "py-6634",
+ "py-6646",
+ "py-6678",
+ "py-6764",
+ "py-6773",
+ "py-6817",
+ "py-7185",
+ "py-7458",
+ "py-7563",
+ "py-7740",
+ "py-8152",
+ "py-8320",
+ "py-8450",
+ "py-8464",
+ "py-8504",
+ "py-8523",
+ "py-8851",
+ "py-8958",
+ "py-9020",
+ "py-9030",
+ "py-9070",
+ "py-9081",
+ "py-9217",
+ "py-9275",
+ "py-940",
+ "py-941",
+ "py-9652",
+ "py-9659",
+ "py-9824",
+ "py-9859",
+ "py-9876",
+ "py-9902",
+ "py-9968"
+ ]
+ }
+}
diff --git a/experiments/workloads/swebench.json b/experiments/workloads/swebench.json
new file mode 100644
index 0000000..2086dbc
--- /dev/null
+++ b/experiments/workloads/swebench.json
@@ -0,0 +1,402 @@
+{
+ "domain": "swebench",
+ "description": "SWE-bench task streams for the Claude/Copilot trimech A/B and the Codex arms.",
+ "source": "tokeneconomizer/workloads.py (regenerate: python scripts/export_workloads.py)",
+ "group_field": "repo",
+ "seeds": [
+ 0,
+ 1,
+ 2
+ ],
+ "total_tasks": 277,
+ "task_counts": {
+ "django/django": 40,
+ "matplotlib/matplotlib": 34,
+ "scikit-learn/scikit-learn": 60,
+ "sphinx-doc/sphinx": 44,
+ "sympy/sympy": 60,
+ "pydata/xarray": 39
+ },
+ "groups": {
+ "django/django": [
+ "django__django-7530",
+ "django__django-10097",
+ "django__django-10554",
+ "django__django-10880",
+ "django__django-10914",
+ "django__django-10973",
+ "django__django-10999",
+ "django__django-11066",
+ "django__django-11087",
+ "django__django-11095",
+ "django__django-11099",
+ "django__django-11119",
+ "django__django-11133",
+ "django__django-11138",
+ "django__django-11149",
+ "django__django-11163",
+ "django__django-11179",
+ "django__django-11206",
+ "django__django-11211",
+ "django__django-11239",
+ "django__django-11265",
+ "django__django-11276",
+ "django__django-11292",
+ "django__django-11299",
+ "django__django-11333",
+ "django__django-11400",
+ "django__django-11433",
+ "django__django-11451",
+ "django__django-11477",
+ "django__django-11490",
+ "django__django-11532",
+ "django__django-11551",
+ "django__django-11555",
+ "django__django-11603",
+ "django__django-11728",
+ "django__django-11734",
+ "django__django-11740",
+ "django__django-11749",
+ "django__django-11790",
+ "django__django-11820"
+ ],
+ "matplotlib/matplotlib": [
+ "matplotlib__matplotlib-13989",
+ "matplotlib__matplotlib-14623",
+ "matplotlib__matplotlib-20488",
+ "matplotlib__matplotlib-20676",
+ "matplotlib__matplotlib-20826",
+ "matplotlib__matplotlib-20859",
+ "matplotlib__matplotlib-21568",
+ "matplotlib__matplotlib-22719",
+ "matplotlib__matplotlib-22865",
+ "matplotlib__matplotlib-22871",
+ "matplotlib__matplotlib-23299",
+ "matplotlib__matplotlib-23314",
+ "matplotlib__matplotlib-23412",
+ "matplotlib__matplotlib-23476",
+ "matplotlib__matplotlib-24026",
+ "matplotlib__matplotlib-24149",
+ "matplotlib__matplotlib-24177",
+ "matplotlib__matplotlib-24570",
+ "matplotlib__matplotlib-24627",
+ "matplotlib__matplotlib-24637",
+ "matplotlib__matplotlib-24870",
+ "matplotlib__matplotlib-24970",
+ "matplotlib__matplotlib-25122",
+ "matplotlib__matplotlib-25287",
+ "matplotlib__matplotlib-25311",
+ "matplotlib__matplotlib-25332",
+ "matplotlib__matplotlib-25479",
+ "matplotlib__matplotlib-25775",
+ "matplotlib__matplotlib-25960",
+ "matplotlib__matplotlib-26113",
+ "matplotlib__matplotlib-26208",
+ "matplotlib__matplotlib-26291",
+ "matplotlib__matplotlib-26342",
+ "matplotlib__matplotlib-26466"
+ ],
+ "scikit-learn/scikit-learn": [
+ "scikit-learn__scikit-learn-9288",
+ "scikit-learn__scikit-learn-10297",
+ "scikit-learn__scikit-learn-10844",
+ "scikit-learn__scikit-learn-10908",
+ "scikit-learn__scikit-learn-11310",
+ "scikit-learn__scikit-learn-11578",
+ "scikit-learn__scikit-learn-12585",
+ "scikit-learn__scikit-learn-12682",
+ "scikit-learn__scikit-learn-12973",
+ "scikit-learn__scikit-learn-13124",
+ "scikit-learn__scikit-learn-13135",
+ "scikit-learn__scikit-learn-13142",
+ "scikit-learn__scikit-learn-13328",
+ "scikit-learn__scikit-learn-13439",
+ "scikit-learn__scikit-learn-13496",
+ "scikit-learn__scikit-learn-13779",
+ "scikit-learn__scikit-learn-14053",
+ "scikit-learn__scikit-learn-14087",
+ "scikit-learn__scikit-learn-14141",
+ "scikit-learn__scikit-learn-14496",
+ "scikit-learn__scikit-learn-14629",
+ "scikit-learn__scikit-learn-14894",
+ "scikit-learn__scikit-learn-14983",
+ "scikit-learn__scikit-learn-15100",
+ "scikit-learn__scikit-learn-25102",
+ "scikit-learn__scikit-learn-25232",
+ "scikit-learn__scikit-learn-25747",
+ "scikit-learn__scikit-learn-25931",
+ "scikit-learn__scikit-learn-25973",
+ "scikit-learn__scikit-learn-26194",
+ "scikit-learn__scikit-learn-26323",
+ "scikit-learn__scikit-learn-3840",
+ "scikit-learn__scikit-learn-7760",
+ "scikit-learn__scikit-learn-9274",
+ "scikit-learn__scikit-learn-10459",
+ "scikit-learn__scikit-learn-10577",
+ "scikit-learn__scikit-learn-10803",
+ "scikit-learn__scikit-learn-11496",
+ "scikit-learn__scikit-learn-11585",
+ "scikit-learn__scikit-learn-11596",
+ "scikit-learn__scikit-learn-12421",
+ "scikit-learn__scikit-learn-12758",
+ "scikit-learn__scikit-learn-12827",
+ "scikit-learn__scikit-learn-12989",
+ "scikit-learn__scikit-learn-13010",
+ "scikit-learn__scikit-learn-13046",
+ "scikit-learn__scikit-learn-13363",
+ "scikit-learn__scikit-learn-13467",
+ "scikit-learn__scikit-learn-13618",
+ "scikit-learn__scikit-learn-13628",
+ "scikit-learn__scikit-learn-13864",
+ "scikit-learn__scikit-learn-13910",
+ "scikit-learn__scikit-learn-14237",
+ "scikit-learn__scikit-learn-14464",
+ "scikit-learn__scikit-learn-14544",
+ "scikit-learn__scikit-learn-14704",
+ "scikit-learn__scikit-learn-14806",
+ "scikit-learn__scikit-learn-14999",
+ "scikit-learn__scikit-learn-23099",
+ "scikit-learn__scikit-learn-25744"
+ ],
+ "sphinx-doc/sphinx": [
+ "sphinx-doc__sphinx-7440",
+ "sphinx-doc__sphinx-7454",
+ "sphinx-doc__sphinx-7462",
+ "sphinx-doc__sphinx-7590",
+ "sphinx-doc__sphinx-7748",
+ "sphinx-doc__sphinx-7757",
+ "sphinx-doc__sphinx-7889",
+ "sphinx-doc__sphinx-7910",
+ "sphinx-doc__sphinx-7985",
+ "sphinx-doc__sphinx-8035",
+ "sphinx-doc__sphinx-8056",
+ "sphinx-doc__sphinx-8120",
+ "sphinx-doc__sphinx-8265",
+ "sphinx-doc__sphinx-8269",
+ "sphinx-doc__sphinx-8459",
+ "sphinx-doc__sphinx-8475",
+ "sphinx-doc__sphinx-8548",
+ "sphinx-doc__sphinx-8551",
+ "sphinx-doc__sphinx-8593",
+ "sphinx-doc__sphinx-8595",
+ "sphinx-doc__sphinx-8621",
+ "sphinx-doc__sphinx-8638",
+ "sphinx-doc__sphinx-8721",
+ "sphinx-doc__sphinx-9229",
+ "sphinx-doc__sphinx-9230",
+ "sphinx-doc__sphinx-9258",
+ "sphinx-doc__sphinx-9281",
+ "sphinx-doc__sphinx-9320",
+ "sphinx-doc__sphinx-9367",
+ "sphinx-doc__sphinx-9461",
+ "sphinx-doc__sphinx-9591",
+ "sphinx-doc__sphinx-9602",
+ "sphinx-doc__sphinx-9658",
+ "sphinx-doc__sphinx-9673",
+ "sphinx-doc__sphinx-9698",
+ "sphinx-doc__sphinx-9711",
+ "sphinx-doc__sphinx-10323",
+ "sphinx-doc__sphinx-10435",
+ "sphinx-doc__sphinx-10449",
+ "sphinx-doc__sphinx-10466",
+ "sphinx-doc__sphinx-10614",
+ "sphinx-doc__sphinx-10673",
+ "sphinx-doc__sphinx-11445",
+ "sphinx-doc__sphinx-11510"
+ ],
+ "sympy/sympy": [
+ "sympy__sympy-11618",
+ "sympy__sympy-12096",
+ "sympy__sympy-12419",
+ "sympy__sympy-12481",
+ "sympy__sympy-12489",
+ "sympy__sympy-13031",
+ "sympy__sympy-13091",
+ "sympy__sympy-13372",
+ "sympy__sympy-13480",
+ "sympy__sympy-13551",
+ "sympy__sympy-13615",
+ "sympy__sympy-13647",
+ "sympy__sympy-13757",
+ "sympy__sympy-13798",
+ "sympy__sympy-13852",
+ "sympy__sympy-13877",
+ "sympy__sympy-13878",
+ "sympy__sympy-13974",
+ "sympy__sympy-14248",
+ "sympy__sympy-14531",
+ "sympy__sympy-14711",
+ "sympy__sympy-14976",
+ "sympy__sympy-15017",
+ "sympy__sympy-15345",
+ "sympy__sympy-15349",
+ "sympy__sympy-15599",
+ "sympy__sympy-15809",
+ "sympy__sympy-15875",
+ "sympy__sympy-15976",
+ "sympy__sympy-16450",
+ "sympy__sympy-16597",
+ "sympy__sympy-16766",
+ "sympy__sympy-16792",
+ "sympy__sympy-16886",
+ "sympy__sympy-17139",
+ "sympy__sympy-17318",
+ "sympy__sympy-17630",
+ "sympy__sympy-17655",
+ "sympy__sympy-18189",
+ "sympy__sympy-18763",
+ "sympy__sympy-11862",
+ "sympy__sympy-11989",
+ "sympy__sympy-13259",
+ "sympy__sympy-13265",
+ "sympy__sympy-13301",
+ "sympy__sympy-13988",
+ "sympy__sympy-14308",
+ "sympy__sympy-15225",
+ "sympy__sympy-15304",
+ "sympy__sympy-15555",
+ "sympy__sympy-15586",
+ "sympy__sympy-16781",
+ "sympy__sympy-17194",
+ "sympy__sympy-17313",
+ "sympy__sympy-17340",
+ "sympy__sympy-17394",
+ "sympy__sympy-18199",
+ "sympy__sympy-18256",
+ "sympy__sympy-18477",
+ "sympy__sympy-19201"
+ ],
+ "pydata/xarray": [
+ "pydata__xarray-4879",
+ "pydata__xarray-6386",
+ "pydata__xarray-6394",
+ "pydata__xarray-6400",
+ "pydata__xarray-6461",
+ "pydata__xarray-6548",
+ "pydata__xarray-6598",
+ "pydata__xarray-6599",
+ "pydata__xarray-6601",
+ "pydata__xarray-6721",
+ "pydata__xarray-6744",
+ "pydata__xarray-6798",
+ "pydata__xarray-6804",
+ "pydata__xarray-6823",
+ "pydata__xarray-6857",
+ "pydata__xarray-6882",
+ "pydata__xarray-6889",
+ "pydata__xarray-6938",
+ "pydata__xarray-6971",
+ "pydata__xarray-6992",
+ "pydata__xarray-6999",
+ "pydata__xarray-7003",
+ "pydata__xarray-7019",
+ "pydata__xarray-7052",
+ "pydata__xarray-7089",
+ "pydata__xarray-7101",
+ "pydata__xarray-7105",
+ "pydata__xarray-7112",
+ "pydata__xarray-7120",
+ "pydata__xarray-7147",
+ "pydata__xarray-7150",
+ "pydata__xarray-7203",
+ "pydata__xarray-7229",
+ "pydata__xarray-7233",
+ "pydata__xarray-7347",
+ "pydata__xarray-7391",
+ "pydata__xarray-7393",
+ "pydata__xarray-7400",
+ "pydata__xarray-7444"
+ ]
+ },
+ "excluded": {
+ "scikit-learn__scikit-learn-14710": "eval.sh does not terminate under te_harness: timed out at TE_EVAL_TIMEOUT=1800 and again at 3600, and the uncaught TimeoutExpired ended the sklearn stream at task 21 of 32 both times (that cell never produced more than 20 graded rows). The timeout is now caught (te_harness swebench.py _grade), so it would cost one full eval_timeout per run rather than the cell; excluded to stop paying that. NOT ungradeable in principle -- it graded in the published runs, so results that include it and results that exclude it are not directly comparable on scikit-learn."
+ },
+ "added": {
+ "scikit-learn/scikit-learn": [
+ "scikit-learn__scikit-learn-10459",
+ "scikit-learn__scikit-learn-10577",
+ "scikit-learn__scikit-learn-10803",
+ "scikit-learn__scikit-learn-11496",
+ "scikit-learn__scikit-learn-11585",
+ "scikit-learn__scikit-learn-11596",
+ "scikit-learn__scikit-learn-12421",
+ "scikit-learn__scikit-learn-12758",
+ "scikit-learn__scikit-learn-12827",
+ "scikit-learn__scikit-learn-12989",
+ "scikit-learn__scikit-learn-13010",
+ "scikit-learn__scikit-learn-13046",
+ "scikit-learn__scikit-learn-13363",
+ "scikit-learn__scikit-learn-13467",
+ "scikit-learn__scikit-learn-13618",
+ "scikit-learn__scikit-learn-13628",
+ "scikit-learn__scikit-learn-13864",
+ "scikit-learn__scikit-learn-13910",
+ "scikit-learn__scikit-learn-14237",
+ "scikit-learn__scikit-learn-14464",
+ "scikit-learn__scikit-learn-14544",
+ "scikit-learn__scikit-learn-14704",
+ "scikit-learn__scikit-learn-14806",
+ "scikit-learn__scikit-learn-14999",
+ "scikit-learn__scikit-learn-23099",
+ "scikit-learn__scikit-learn-25744",
+ "scikit-learn__scikit-learn-3840",
+ "scikit-learn__scikit-learn-7760",
+ "scikit-learn__scikit-learn-9274"
+ ],
+ "sympy/sympy": [
+ "sympy__sympy-11862",
+ "sympy__sympy-11989",
+ "sympy__sympy-13259",
+ "sympy__sympy-13265",
+ "sympy__sympy-13301",
+ "sympy__sympy-13988",
+ "sympy__sympy-14308",
+ "sympy__sympy-15225",
+ "sympy__sympy-15304",
+ "sympy__sympy-15555",
+ "sympy__sympy-15586",
+ "sympy__sympy-16781",
+ "sympy__sympy-17194",
+ "sympy__sympy-17313",
+ "sympy__sympy-17340",
+ "sympy__sympy-17394",
+ "sympy__sympy-18199",
+ "sympy__sympy-18256",
+ "sympy__sympy-18477",
+ "sympy__sympy-19201"
+ ]
+ },
+ "full_split_only": [
+ "pydata__xarray-4879",
+ "pydata__xarray-6386",
+ "pydata__xarray-6394",
+ "pydata__xarray-6400",
+ "pydata__xarray-6548",
+ "pydata__xarray-6598",
+ "pydata__xarray-6601",
+ "pydata__xarray-6798",
+ "pydata__xarray-6804",
+ "pydata__xarray-6823",
+ "pydata__xarray-6857",
+ "pydata__xarray-6882",
+ "pydata__xarray-6889",
+ "pydata__xarray-6971",
+ "pydata__xarray-6999",
+ "pydata__xarray-7003",
+ "pydata__xarray-7019",
+ "pydata__xarray-7052",
+ "pydata__xarray-7089",
+ "pydata__xarray-7101",
+ "pydata__xarray-7105",
+ "pydata__xarray-7112",
+ "pydata__xarray-7120",
+ "pydata__xarray-7147",
+ "pydata__xarray-7150",
+ "pydata__xarray-7203",
+ "pydata__xarray-7347",
+ "pydata__xarray-7391",
+ "pydata__xarray-7400",
+ "pydata__xarray-7444"
+ ],
+ "notes": "These ids are NOT all in SWE-bench_Verified: 'full_split_only' lists the instances that exist only in the full princeton-nlp/SWE-bench test split. Docker image tag: swebench/sweb.eval.x86_64.<id with __ -> _1776_>:latest. 'excluded' lists ids dropped from THIS run list with the reason; they are still part of the recorded workload in tokeneconomizer/workloads.py (SWEBENCH_INSTANCES) and were run by the published results, so a cell that skips them is not directly comparable to one that did not. 'added' lists ids that EXTEND a stream beyond the published workload to de-saturate it (scikit-learn and sympy, to 60 runnable tasks each); they are all outside SWE-bench_Verified, so provision them from the full test split, and results on an extended stream are not comparable to the published ones for that repo."
+}
diff --git a/experiments/workloads/webbench.json b/experiments/workloads/webbench.json
new file mode 100644
index 0000000..078d713
--- /dev/null
+++ b/experiments/workloads/webbench.json
@@ -0,0 +1,133 @@
+{
+ "domain": "webbench",
+ "description": "Web-Bench streams; task order is load-bearing (each task builds on the last).",
+ "source": "tokeneconomizer/workloads.py (regenerate: python scripts/export_workloads.py)",
+ "group_field": "project",
+ "seeds": [
+ "A",
+ "B",
+ "C"
+ ],
+ "total_tasks": 102,
+ "task_counts": {
+ "calculator": 21,
+ "expressjs": 20,
+ "react": 20,
+ "zustand": 20,
+ "table": 21
+ },
+ "groups": {
+ "calculator": [
+ "init",
+ "task-1",
+ "task-2",
+ "task-3",
+ "task-4",
+ "task-5",
+ "task-6",
+ "task-7",
+ "task-8",
+ "task-9",
+ "task-10",
+ "task-11",
+ "task-12",
+ "task-13",
+ "task-14",
+ "task-15",
+ "task-16",
+ "task-17",
+ "task-18",
+ "task-19",
+ "task-20"
+ ],
+ "expressjs": [
+ "task-1",
+ "task-2",
+ "task-3",
+ "task-4",
+ "task-5",
+ "task-6",
+ "task-7",
+ "task-8",
+ "task-9",
+ "task-10",
+ "task-11",
+ "task-12",
+ "task-13",
+ "task-14",
+ "task-15",
+ "task-16",
+ "task-17",
+ "task-18",
+ "task-19",
+ "task-20"
+ ],
+ "react": [
+ "task-1",
+ "task-2",
+ "task-3",
+ "task-4",
+ "task-5",
+ "task-6",
+ "task-7",
+ "task-8",
+ "task-9",
+ "task-10",
+ "task-11",
+ "task-12",
+ "task-13",
+ "task-14",
+ "task-15",
+ "task-16",
+ "task-17",
+ "task-18",
+ "task-19",
+ "task-20"
+ ],
+ "zustand": [
+ "task-1",
+ "task-2",
+ "task-3",
+ "task-4",
+ "task-5",
+ "task-6",
+ "task-7",
+ "task-8",
+ "task-9",
+ "task-10",
+ "task-11",
+ "task-12",
+ "task-13",
+ "task-14",
+ "task-15",
+ "task-16",
+ "task-17",
+ "task-18",
+ "task-19",
+ "task-20"
+ ],
+ "table": [
+ "init",
+ "task-1",
+ "task-2",
+ "task-3",
+ "task-4",
+ "task-5",
+ "task-6",
+ "task-7",
+ "task-8",
+ "task-9",
+ "task-10",
+ "task-11",
+ "task-12",
+ "task-13",
+ "task-14",
+ "task-15",
+ "task-16",
+ "task-17",
+ "task-18",
+ "task-19",
+ "task-20"
+ ]
+ }
+}
diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json
new file mode 100644
index 0000000..cd40162
--- /dev/null
+++ b/plugin/.claude-plugin/plugin.json
@@ -0,0 +1,10 @@
+{
+ "name": "tokeneconomizer",
+ "displayName": "TokenEconomizer",
+ "version": "0.1.0",
+ "description": "Cuts a coding agent's token bill from outside the agent: learned per-repo facts injected at session start, unused-tool deny lists you approve, and a guard that redirects a turn stuck in a failing loop.",
+ "author": {"name": "TokenEconomizer authors"},
+ "repository": "https://github.com/microsoft/TokenEconomizer",
+ "license": "MIT",
+ "keywords": ["cost", "tokens", "context", "hooks", "untooling", "spiral"]
+}
diff --git a/plugin/.codex-plugin/plugin.json b/plugin/.codex-plugin/plugin.json
new file mode 100644
index 0000000..b5961fa
--- /dev/null
+++ b/plugin/.codex-plugin/plugin.json
@@ -0,0 +1,10 @@
+{
+ "name": "tokeneconomizer",
+ "version": "0.1.0",
+ "description": "Cuts a coding agent's token bill from outside the agent: learned per-repo facts injected at session start, an unused-capability deny list you approve (written to the project's .codex/config.toml), and a guard that redirects a turn stuck in a failing loop.",
+ "author": {"name": "TokenEconomizer authors"},
+ "repository": "https://github.com/microsoft/TokenEconomizer",
+ "license": "MIT",
+ "keywords": ["cost", "tokens", "context", "hooks"],
+ "hooks": "./hooks/codex-hooks.json"
+}
diff --git a/plugin/.github/plugin/plugin.json b/plugin/.github/plugin/plugin.json
new file mode 100644
index 0000000..c61c575
--- /dev/null
+++ b/plugin/.github/plugin/plugin.json
@@ -0,0 +1,19 @@
+{
+ "name": "tokeneconomizer",
+ "version": "0.1.0",
+ "description": "Cuts a coding agent's token bill from outside the agent: learned per-repo facts injected at session start, an unused-MCP-server deny list you approve, and a guard that redirects a turn stuck in a failing loop.",
+ "author": {
+ "name": "TokenEconomizer authors"
+ },
+ "repository": "https://github.com/microsoft/TokenEconomizer",
+ "license": "MIT",
+ "keywords": [
+ "cost",
+ "tokens",
+ "context",
+ "hooks",
+ "untooling",
+ "spiral"
+ ],
+ "hooks": "hooks/copilot-hooks.json"
+}
diff --git a/plugin/README.md b/plugin/README.md
new file mode 100644
index 0000000..ca9889b
--- /dev/null
+++ b/plugin/README.md
@@ -0,0 +1,353 @@
+# TokenEconomizer plugin
+
+Runtime cost control for coding agents, from outside the agent. In each repository it works
+in, it:
+
+* **proposes removing tools you never use.** Tools that stay loaded but unused for several
+ sessions are proposed as a deny list. You review it and apply it; nothing changes until you
+ do. On Codex the unit is a capability package (sub-agents, goals, images); on the Copilot CLI
+ it is a whole MCP server, such as the built-in GitHub server.
+* **injects what it learned.** Environment facts your sessions keep rediscovering the hard way
+ ("`tox …` fails here with command not found; `python -m pytest …` works") are added to the
+ next session's context.
+* **redirects a turn stuck in a failing loop.** When a turn keeps repeating the same failing
+ edit/test cycle well past what this repository's turns usually take, the agent gets one
+ mid-turn redirect.
+
+Every action is announced in one line, every hook fails open, and by default the plugin makes no
+model calls of its own. All of its state stays on your machine, under `~/.tokeneconomizer`.
+Design, verified agent behaviour and limits: [`docs/PLUGIN.md`](../docs/PLUGIN.md).
+
+## Before you start
+
+- [ ] Claude Code installed (Codex, the Copilot CLI and Pi: see [below](#codex-copilot-and-pi))
+- [ ] Python 3.10 or newer on `PATH` (or set `TE_PYTHON=/path/to/python3.10+`)
+- [ ] Access to this repository on GitHub
+
+No extra API key is needed. See [Model calls and API keys](#model-calls-and-api-keys).
+
+## Steps (Claude Code)
+
+**1. Get the plugin.** In a terminal:
+
+```bash
+git clone https://github.com/microsoft/TokenEconomizer.git
+```
+
+**2. Install it.** In a terminal, using the path of the clone from step 1:
+
+```bash
+claude plugin marketplace add /path/to/TokenEconomizer
+claude plugin install tokeneconomizer@tokeneconomizer
+claude plugin list # should show tokeneconomizer@tokeneconomizer ... enabled
+```
+
+(Inside Claude Code the same thing is `/plugin marketplace add …` and `/plugin install …`.)
+Plugins load when a session starts, so sessions already open won't have it; start a new one.
+
+> **Where commands go.** Lines starting with `/` (`/plugin …`, `/tokeneconomizer:…`) are typed
+> into **Claude Code's prompt**, not your shell. Shell commands are in `bash` blocks.
+
+**3. Check it.** Start Claude Code in any project (`cd my-project && claude`) and type:
+
+```text
+/tokeneconomizer:status
+```
+
+It should print a short status block for the project, not an error. The first time, Claude Code
+asks permission to run the plugin's script; choose to always allow it.
+
+**4. Set up each repository, once.** In Claude Code, in the repository:
+
+```text
+/tokeneconomizer:untool refresh-catalog
+```
+
+This records which tools a session loads (no model call). **Don't skip it.** Without it the deny
+list can only see the few tools Claude Code defers, not the ones every session loads.
+
+Optional: `/tokeneconomizer:replay` estimates what the plugin would do on your past sessions in
+this repository, with no model calls.
+
+**5. Work normally.** You don't need to do anything. Across sessions the plugin learns which tools
+this repository uses, learns environment facts, and prices each session after it ends.
+
+**6. Accept the deny list (after about five sessions).** When a session starts you'll see a
+notice like *"TokenEconomizer: N tool(s) went unused in this repository's recent sessions…"*.
+Then:
+
+```text
+/tokeneconomizer:untool # see the proposed tools and the evidence
+/tokeneconomizer:untool apply # accept them
+```
+
+It takes effect **from your next session**. The rules are written to this repository's
+`.claude/settings.local.json`, which the plugin keeps out of git, and it never touches deny
+rules you wrote yourself. To undo it, run `/tokeneconomizer:untool restore`.
+
+**7. Check in whenever you like:** `/tokeneconomizer:status` and `/tokeneconomizer:facts`.
+
+For one session without installing anything: `claude --plugin-dir /path/to/TokenEconomizer/plugin`.
+
+## Commands
+
+Inside Claude Code:
+
+| Command | What it does |
+|---|---|
+| `/tokeneconomizer:status` | what it has learned in this repository, what it did, and what your sessions cost |
+| `/tokeneconomizer:facts` | the facts it injects at session start, and those still forming |
+| `/tokeneconomizer:untool` | the proposed deny list and its evidence |
+| `/tokeneconomizer:untool apply` | accept the proposal (or `apply TOOL …` for specific tools); from the next session |
+| `/tokeneconomizer:untool restore` | remove the plugin's deny rules (or `restore TOOL …`) |
+| `/tokeneconomizer:untool refresh-catalog` | re-record the tools a session loads, e.g. after a Claude Code update |
+| `/tokeneconomizer:untool measure` | prompt tokens per request with and without the proposal (two small model calls) |
+| `/tokeneconomizer:replay` | what the plugin would do, replayed over this repository's past sessions |
+| `/tokeneconomizer:config` | the settings: what is on, and every value in effect |
+| `/tokeneconomizer:config set KEY=VALUE …` | change settings for this repository (add `--global` for all); see [Settings](#settings) |
+| `/tokeneconomizer:config unset KEY …` | put a setting back to its default |
+
+From a terminal, the same commands and the settings are
+`/path/to/TokenEconomizer/plugin/scripts/te <command>`, run inside the repository:
+
+| Command | What it does |
+|---|---|
+| `te status`, `te facts`, `te untool …`, `te replay`, `te config …` | as above |
+| `te doctor` | check Python, the state directory, and the tool catalogs |
+
+## Settings
+
+Each of the three mechanisms can be turned on or off on its own, and each has numbers you can
+set by hand. Settings apply to the current repository, or to every repository with `--global`.
+
+**Turn a mechanism on or off:**
+
+```text
+/tokeneconomizer:config set spiral=off # the stuck-turn guard, in this repository
+/tokeneconomizer:config set untool=off inject=on
+/tokeneconomizer:config set spiral=off --global # in every repository
+/tokeneconomizer:config # what is on, and every value in effect
+```
+
+| Mechanism | Key | `on` means | Other values |
+|---|---|---|---|
+| Learned facts | `inject` | `envfacts`: facts found without model calls | `off`; `envfacts+distiller`, `distiller` add a paid fact proposer (need `paid=true`) |
+| Unused-tool deny list | `untool` | `propose`: proposed, applied only when you say so | `off`; `auto` applies a proposal as soon as it forms |
+| Stuck-turn guard | `spiral` | `nudge`: one mid-turn redirect | `off`; `guard` also stops a headless turn that keeps spinning after the redirect |
+
+**Tune a mechanism:**
+
+```text
+/tokeneconomizer:config set inject_max_facts=2 # inject at most two facts
+/tokeneconomizer:config set untool_evidence=10 # wait ten sessions before proposing
+/tokeneconomizer:config set spiral_nudges=2 spiral.min_turns=30
+/tokeneconomizer:config unset spiral.min_turns # back to the default
+```
+
+| Key | Default | What it sets |
+|---|---|---|
+| `inject_max_facts` | `4` | the most facts injected at session start |
+| `inject_max_tokens` | `150` | the size of the injected block, in tokens (about 4 characters each) |
+| `propose_every` | `5` | sessions between runs of the paid fact proposer |
+| `untool_evidence` | `5` | sessions a tool must go unused before it is proposed (three times as long for a few higher-risk tools; the web is never proposed) |
+| `spiral_aggr` | `default` | the guard's threshold preset: `conservative`, `default` or `aggressive` |
+| `spiral_nudges` | `1` | redirects per prompt-turn before the guard only notifies (or, with `spiral=guard`, stops) |
+| `spiral.<name>` | from the preset | one guard threshold, set by hand (below) |
+
+The guard calls a turn stuck when it keeps editing the same files and hitting the same
+failures, well past how long this repository's turns that end with passing tests usually take
+(the *envelope*). Until a repository has a few such turns, the preset's thresholds apply; after
+that, some are learned from the repository's own turns. A value you set with `spiral.<name>`
+wins over both. Defaults at each preset:
+
+| `spiral.<name>` | `conservative` | `default` | `aggressive` | What it sets |
+|---|---|---|---|---|
+| `min_turns` | 28 | 20 | 14 | tool calls in a turn before it can count as stuck |
+| `edit_churn` | 5 | 4 | 3 | edits to one file that count as churn |
+| `distinct_edit_files` | 3 | 3 | 3 | files edited that count as scattered |
+| `error_outputs` | 8 | 6 | 4 | failing tool results that count as sustained errors |
+| `stagnation` | 0.30 | 0.20 | 0.12 | share of failures that repeat an earlier one |
+| `envelope_pct` | 90 | 90 | 90 | percentile of the repository's passing turns the envelope starts from |
+| `envelope_margin` | 1.5 | 1.25 | 1.0 | multiplier on that percentile |
+| `envelope_min_samples` | 7 | 5 | 3 | passing turns needed before the envelope is used |
+| `backstop_mult` | 3.5 | 2.5 | 1.8 | past this multiple of the envelope, a stuck turn is redirected, then stopped or flagged |
+| `judge_confidence` | 0.85 | 0.7 | 0.55 | with `paid=true`: how sure the model judge must be |
+| `gate_min_turns` | 24 | 18 | 12 | with `paid=true`: tool calls before the judge is asked |
+| `gate_error_outputs` | 8 | 8 | 8 | with `paid=true`: failing results that ask the judge early |
+
+**Other settings:**
+
+| Key | Default | Values |
+|---|---|---|
+| `paid` | `false` | allow the plugin's own model calls (see below) |
+| `paid_model` | `""` | model for those calls (`haiku` when empty) |
+| `shadow` | `false` | decide and log everything, act on nothing |
+| `mode` | `auto` | `interactive` or `headless` to override what the plugin detects |
+
+**Where settings live, and when they apply.** For one session, an environment variable wins:
+`TE_INJECT`, `TE_UNTOOL`, `TE_SPIRAL`, `TE_SPIRAL_NUDGES`, `TE_INJECT_MAX_FACTS`, and so on, and
+`TE_SPIRAL_THRESHOLDS='{"min_turns": 30}'` for the guard's thresholds; `TE_DISABLE=1` turns the
+whole plugin off. Otherwise the repository's settings win over the global ones, and those over
+the defaults; `unset` goes back one level. A change applies from the agent's next hook call: the
+guard at the next tool call, facts and the deny list at the next session start. A new fact-size
+cap shows once the next session has ended.
+
+## Model calls and API keys
+
+**By default the plugin makes no model calls.** Learned facts, the deny list and the stuck-turn
+check all run locally.
+
+With `paid=true`, three helpers can call a model: the fact proposer, the stuck-turn judge and
+the redirect writer. They run through **your own agent CLI** (`claude -p`; `codex exec` on
+Codex; `copilot -p`, with every tool removed, on Copilot), so they use whatever your sessions
+already use: a Claude subscription or Anthropic API key, your Codex login or OpenAI key, or
+Copilot's subscription or BYOK provider. Their cost is billed there too, and it is shown in
+`/tokeneconomizer:status`. `untool measure` makes its two calls the same way, and only when you
+run it.
+
+## Updating and uninstalling
+
+**Update:** `git pull` in the clone, then `/plugin marketplace update tokeneconomizer` and
+`/plugin install tokeneconomizer@tokeneconomizer`. Claude Code runs an installed **copy**, so
+the pull alone changes nothing.
+
+**Uninstall:**
+
+1. In each repository where you accepted a deny list, run `/tokeneconomizer:untool restore`
+ (Codex and Copilot: `te --host codex untool restore`, `te --host copilot untool restore`).
+ The entries live in that repository's settings file and stay there otherwise.
+2. `/plugin uninstall tokeneconomizer@tokeneconomizer`
+3. Optional: `rm -rf ~/.tokeneconomizer` to delete what it learned.
+
+## Codex, Copilot and Pi
+
+The plugin runs on all three. What it can remove differs by agent.
+
+| | Learned facts | Stuck-turn redirect | Removing unused tools |
+|---|---|---|---|
+| Claude Code | yes | yes; can also stop a headless turn | tools, in `.claude/settings.local.json` |
+| Codex | yes | yes; a notice instead of a stop | capability packages, in `.codex/config.toml` |
+| Copilot CLI | yes | yes; a notice instead of a stop | whole MCP servers, in `.github/copilot/settings.local.json` |
+| Pi | yes | yes | no (Pi loads four tools) |
+
+On every agent the deny list works the same way: it is proposed after about five sessions,
+nothing changes until you run `untool apply`, it takes effect from the next session, and
+`untool restore` removes only what the plugin added. A file the plugin creates is added to
+`.git/info/exclude`.
+
+**Codex.** Codex always copies an installed plugin, so install from the self-contained bundle:
+
+```bash
+python /path/to/TokenEconomizer/scripts/build_plugin.py --out ~/te-marketplace
+codex plugin marketplace add ~/te-marketplace
+codex plugin add tokeneconomizer@tokeneconomizer
+```
+
+Then approve its hooks once in `/hooks` inside Codex. Commands are
+`~/.tokeneconomizer/bin/te --host codex <command>`, run inside the repository. The launcher is
+written at the first session; until then, use `~/te-marketplace/tokeneconomizer/scripts/te`.
+
+Codex's deny list removes the sub-agent (`collaboration`), `goals` and image (`image_media`)
+packages, each only after it went unused in every session where it was loaded. A package you
+used even once is kept. Two things to know:
+
+- Codex reads a project's `.codex/config.toml` **only in a trusted project** (Codex asks the
+ first time you start it in a repository). `te --host codex untool apply` warns you if the
+ project is not trusted, and runs `codex features list` to show that the change took effect.
+- The web and asking you questions are never proposed. The bundled skills are not proposed
+ either: Codex honours that switch only in your user-level config, and the plugin changes
+ per-repository files only.
+
+`te --host codex untool measure` shows the difference with no model call. It costs nothing
+because Codex renders its prompt locally.
+
+**Copilot CLI.** Copilot loads a plugin from a local marketplace directly from the clone,
+without copying it:
+
+```bash
+copilot plugin marketplace add /path/to/TokenEconomizer
+copilot plugin install tokeneconomizer@tokeneconomizer
+copilot plugin list # tokeneconomizer@tokeneconomizer ... (enabled)
+```
+
+Copilot uses the plugin's own manifest (`plugin/.github/plugin/plugin.json`) and hook file. In
+Copilot's interactive mode the slash commands work as on Claude Code (`/tokeneconomizer:status`,
+`…:untool`, …), and the plugin's notices appear as status lines. `copilot -p` runs neither.
+From a terminal, run `~/.tokeneconomizer/bin/te --host copilot <command>` inside the
+repository. Every session start keeps that launcher pointing at the plugin.
+
+Set up each repository once, as on Claude Code:
+
+```bash
+~/.tokeneconomizer/bin/te --host copilot untool refresh-catalog # no model call
+```
+
+Copilot has no setting that removes a single built-in tool; it only has one that turns off a
+whole MCP server. So the deny list proposes a server only when **none** of its tools was used
+in about five sessions. The built-in `github-mcp-server` (18 tools on 1.0.83) is the usual
+candidate. `te --host copilot untool measure` shows the difference for your setup. It costs
+nothing, because the plugin captures the request Copilot would send on your machine, and no
+model is called.
+
+**Copilot with your own model key (BYOK).** If Copilot answers *"Access denied by policy
+settings"*, your organization's policy refuses its models. Copilot can use your own provider
+instead, and the plugin works the same way there. Set this in your shell profile:
+
+```bash
+export COPILOT_PROVIDER_TYPE=anthropic # or openai, azure
+export COPILOT_PROVIDER_BASE_URL=https://api.anthropic.com
+export COPILOT_PROVIDER_API_KEY=... # your provider's key
+```
+
+Then start Copilot with a model id your provider knows, for example `copilot --model
+claude-sonnet-5` (an Anthropic provider wants its own ids, such as `claude-haiku-4-5`, not the
+subscription's `claude-haiku-4.5`).
+
+Through **OpenRouter**, keep Copilot on the Anthropic wire, so prompt caching works, and give it
+both ids:
+
+```bash
+export COPILOT_PROVIDER_TYPE=anthropic
+export COPILOT_PROVIDER_BASE_URL=https://openrouter.ai/api
+export COPILOT_PROVIDER_BEARER_TOKEN=... # your OpenRouter key
+export COPILOT_PROVIDER_MODEL_ID=claude-haiku-4.5 # the id Copilot configures itself for
+export COPILOT_PROVIDER_WIRE_MODEL=anthropic/claude-haiku-4.5 # the id OpenRouter bills
+```
+
+The plugin's hooks inherit the variables. Its paid helpers (`paid=true`) go to the same
+provider, on `paid_model`, or on your `COPILOT_MODEL` unless that is `auto`, or on
+`claude-haiku-4-5` for an Anthropic provider (`anthropic/claude-haiku-4.5` through OpenRouter).
+`refresh-catalog` and `measure` never call your provider. `te --host copilot doctor` shows the
+route it sees.
+
+To check the whole plugin on your own setup, run it once:
+
+```bash
+~/.tokeneconomizer/bin/te --host copilot probe # 7 short real sessions, in a temp repo
+```
+
+It builds a throwaway repository that is set up for pytest but has no pytest installed, and
+whose tests pass under `python -m unittest`, and runs the same prompt in real Copilot sessions
+with the plugin, under a state directory of its own. It reports, per session, whether the fact
+the plugin learned was in the context and which test command the model ran first. Then it
+applies the deny list it proposed, and shows the tool definition tokens of one more session
+before restoring. It makes real model calls on your key or quota, and leaves those sessions in
+Copilot's history.
+
+**Pi.** This directory is also a Pi package:
+
+```bash
+pi install /path/to/TokenEconomizer/plugin
+```
+
+Inside Pi: `/te status`, `/te facts`, `/te replay`, `/te config …`.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `/tokeneconomizer:status` prints an import error | reinstall (see Updating); the copy is stale or incomplete |
+| it prints nothing useful, or the hooks seem to do nothing | run `te doctor`; check `~/.tokeneconomizer/errors.log` |
+| `python 3.x (needs >= 3.10)` | install Python 3.10+ or set `TE_PYTHON` to one |
+| the deny list proposes nothing after many sessions | run `/tokeneconomizer:untool refresh-catalog` (step 4); on Copilot, `te --host copilot untool refresh-catalog` |
+| a Codex deny list was applied but nothing changed | trust the project in Codex; `te --host codex doctor` says whether it is trusted |
diff --git a/plugin/commands/config.md b/plugin/commands/config.md
new file mode 100644
index 0000000..b82bc54
--- /dev/null
+++ b/plugin/commands/config.md
@@ -0,0 +1,11 @@
+---
+description: Show or change TokenEconomizer's settings - turn each mechanism on or off, or tune it
+argument-hint: "[show | set KEY=VALUE ... [--global] | unset KEY ... [--global]]"
+---
+Run this exact command with the Bash tool (use `show` when no arguments were given: "$ARGUMENTS"), then show me its output verbatim in a code block:
+
+```
+te="${CLAUDE_PLUGIN_ROOT}/scripts/te"; [ -x "$te" ] || te="${TOKENECONOMIZER_HOME:-$HOME/.tokeneconomizer}/bin/te"; "$te" --cwd "$(pwd)" config $ARGUMENTS
+```
+
+Do not run `set` or `unset` unless those words appear in the arguments above. If a setting was rejected, say why in one sentence.
diff --git a/plugin/commands/facts.md b/plugin/commands/facts.md
new file mode 100644
index 0000000..8c443d9
--- /dev/null
+++ b/plugin/commands/facts.md
@@ -0,0 +1,8 @@
+---
+description: Show the learned repository facts TokenEconomizer injects at session start
+---
+Run this exact command with the Bash tool, then show me its output verbatim in a code block, with no commentary:
+
+```
+te="${CLAUDE_PLUGIN_ROOT}/scripts/te"; [ -x "$te" ] || te="${TOKENECONOMIZER_HOME:-$HOME/.tokeneconomizer}/bin/te"; "$te" --cwd "$(pwd)" facts
+```
diff --git a/plugin/commands/replay.md b/plugin/commands/replay.md
new file mode 100644
index 0000000..05bd71e
--- /dev/null
+++ b/plugin/commands/replay.md
@@ -0,0 +1,11 @@
+---
+description: Estimate what TokenEconomizer would do, by replaying it over this repository's past sessions (no LLM calls)
+argument-hint: "[--limit N] [--commit]"
+---
+Run this exact command with the Bash tool. It reads past session transcripts and can take a few minutes on a long history, so allow it a timeout of up to 10 minutes. Arguments given: "$ARGUMENTS".
+
+```
+te="${CLAUDE_PLUGIN_ROOT}/scripts/te"; [ -x "$te" ] || te="${TOKENECONOMIZER_HOME:-$HOME/.tokeneconomizer}/bin/te"; "$te" --cwd "$(pwd)" replay $ARGUMENTS
+```
+
+Show me the output verbatim in a code block, then summarize in at most three sentences what each mechanism would have done. Do not run anything else.
diff --git a/plugin/commands/status.md b/plugin/commands/status.md
new file mode 100644
index 0000000..8f01053
--- /dev/null
+++ b/plugin/commands/status.md
@@ -0,0 +1,8 @@
+---
+description: Show what TokenEconomizer has learned for this repository and what it has done
+---
+Run this exact command with the Bash tool, then show me its output verbatim in a code block. Add nothing unless the output contains a warning, in which case explain the warning in one sentence.
+
+```
+te="${CLAUDE_PLUGIN_ROOT}/scripts/te"; [ -x "$te" ] || te="${TOKENECONOMIZER_HOME:-$HOME/.tokeneconomizer}/bin/te"; "$te" --cwd "$(pwd)" status
+```
diff --git a/plugin/commands/untool.md b/plugin/commands/untool.md
new file mode 100644
index 0000000..54f448c
--- /dev/null
+++ b/plugin/commands/untool.md
@@ -0,0 +1,11 @@
+---
+description: Review or apply TokenEconomizer's proposed deny list of tools this repository never uses
+argument-hint: "[show | apply [TOOL ...] | restore [TOOL ...] | refresh-catalog | measure]"
+---
+Run this exact command with the Bash tool (use `show` when no arguments were given: "$ARGUMENTS"), then show me its output verbatim in a code block:
+
+```
+te="${CLAUDE_PLUGIN_ROOT}/scripts/te"; [ -x "$te" ] || te="${TOKENECONOMIZER_HOME:-$HOME/.tokeneconomizer}/bin/te"; "$te" --cwd "$(pwd)" untool $ARGUMENTS
+```
+
+Do not run `apply` or `restore` unless those words appear in the arguments above. If the output is a proposal, end with one line telling me how to apply it (`/tokeneconomizer:untool apply`) and that it takes effect from the next session.
diff --git a/plugin/hooks/codex-hooks.json b/plugin/hooks/codex-hooks.json
new file mode 100644
index 0000000..0010f84
--- /dev/null
+++ b/plugin/hooks/codex-hooks.json
@@ -0,0 +1,59 @@
+{
+ "hooks": {
+ "SessionStart": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT:-$CLAUDE_PLUGIN_ROOT}/scripts/te-hook\" --host codex",
+ "timeout": 15
+ }
+ ]
+ }
+ ],
+ "UserPromptSubmit": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT:-$CLAUDE_PLUGIN_ROOT}/scripts/te-hook\" --host codex",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "PostToolUse": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT:-$CLAUDE_PLUGIN_ROOT}/scripts/te-hook\" --host codex",
+ "timeout": 60
+ }
+ ]
+ }
+ ],
+ "Stop": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT:-$CLAUDE_PLUGIN_ROOT}/scripts/te-hook\" --host codex",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "SessionEnd": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT:-$CLAUDE_PLUGIN_ROOT}/scripts/te-hook\" --host codex",
+ "timeout": 3
+ }
+ ]
+ }
+ ]
+ }
+}
diff --git a/plugin/hooks/copilot-hooks.json b/plugin/hooks/copilot-hooks.json
new file mode 100644
index 0000000..6dec24d
--- /dev/null
+++ b/plugin/hooks/copilot-hooks.json
@@ -0,0 +1,71 @@
+{
+ "description": "TokenEconomizer on the GitHub Copilot CLI: inject learned repo facts at session start, guard prompt-turns against failing loops, and fold each finished session into what is learned.",
+ "hooks": {
+ "SessionStart": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 15
+ }
+ ]
+ }
+ ],
+ "UserPromptSubmit": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "PostToolUse": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 60
+ }
+ ]
+ }
+ ],
+ "PostToolUseFailure": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 60
+ }
+ ]
+ }
+ ],
+ "Stop": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "SessionEnd": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${PLUGIN_ROOT}/scripts/te-hook\" --host copilot",
+ "timeout": 10
+ }
+ ]
+ }
+ ]
+ }
+}
diff --git a/plugin/hooks/hooks.json b/plugin/hooks/hooks.json
new file mode 100644
index 0000000..2248ef8
--- /dev/null
+++ b/plugin/hooks/hooks.json
@@ -0,0 +1,60 @@
+{
+ "description": "TokenEconomizer: inject learned repo facts at session start, guard prompt-turns against failing loops, and fold each finished session into what is learned.",
+ "hooks": {
+ "SessionStart": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/te-hook\"",
+ "timeout": 15
+ }
+ ]
+ }
+ ],
+ "UserPromptSubmit": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/te-hook\"",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "PostToolUse": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/te-hook\"",
+ "timeout": 60
+ }
+ ]
+ }
+ ],
+ "Stop": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/te-hook\"",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
+ "SessionEnd": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/te-hook\"",
+ "timeout": 10
+ }
+ ]
+ }
+ ]
+ }
+}
diff --git a/plugin/lib/te_live b/plugin/lib/te_live
new file mode 100644
index 0000000..86d3486
--- /dev/null
+++ b/plugin/lib/te_live
@@ -0,0 +1 @@
+../../te_live
\ No newline at end of file
diff --git a/plugin/lib/tokeneconomizer b/plugin/lib/tokeneconomizer
new file mode 100644
index 0000000..4332233
--- /dev/null
+++ b/plugin/lib/tokeneconomizer
@@ -0,0 +1 @@
+../../tokeneconomizer
\ No newline at end of file
diff --git a/plugin/package.json b/plugin/package.json
new file mode 100644
index 0000000..7352c94
--- /dev/null
+++ b/plugin/package.json
@@ -0,0 +1,15 @@
+{
+ "name": "tokeneconomizer",
+ "version": "0.1.0",
+ "private": true,
+ "description": "TokenEconomizer for Pi: learned per-repo facts injected at session start, and a guard that redirects a turn stuck in a failing loop.",
+ "keywords": ["pi-package", "cost", "tokens", "context"],
+ "license": "MIT",
+ "repository": "https://github.com/microsoft/TokenEconomizer",
+ "pi": {
+ "extensions": ["./pi/tokeneconomizer.ts"]
+ },
+ "peerDependencies": {
+ "@earendil-works/pi-coding-agent": "*"
+ }
+}
diff --git a/plugin/pi/tokeneconomizer.ts b/plugin/pi/tokeneconomizer.ts
new file mode 100644
index 0000000..215e3e5
--- /dev/null
+++ b/plugin/pi/tokeneconomizer.ts
@@ -0,0 +1,179 @@
+/**
+ * TokenEconomizer for Pi (earendil-works/pi).
+ *
+ * Pi has no hook protocol -- extensions are functions on its event bus -- so this extension
+ * speaks the one the Claude Code and Codex integrations use, on Pi's behalf: every event
+ * below becomes one call of the shared dispatcher (`../scripts/te-hook --host pi`, JSON in,
+ * JSON out), and the answer is applied with Pi's own levers. All of the mechanism logic lives
+ * in the dispatcher; this file only translates.
+ *
+ * session_start -> SessionStart the learned manual, kept for the session
+ * before_agent_start -> UserPromptSubmit opens a prompt-turn; the manual is set as the
+ * `tokeneconomizer` system-prompt section, on every
+ * prompt, because Pi diffs sections between runs and
+ * an omitted one would be recorded as removed
+ * tool_result -> PostToolUse the guard; a nudge is delivered as a steering
+ * message, a stop as ctx.abort()
+ * agent_end -> Stop closes the turn
+ * session_shutdown -> SessionEnd hands the session to a detached fold
+ *
+ * Like the hooks on the other agents it fails open: a dispatcher that is missing, slow or
+ * broken answers "no change". It does nothing under TE_SIDECALL=1 (the plugin's own side
+ * calls) or TE_DISABLE=1. `/te <command>` runs the `te` command line for this repository.
+ */
+import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
+import { spawn } from "node:child_process";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const HERE = (() => {
+ try {
+ return dirname(fileURLToPath(import.meta.url));
+ } catch {
+ return __dirname;
+ }
+})();
+const SCRIPTS = join(HERE, "..", "scripts");
+const HOOK_TIMEOUT_MS = 60_000;
+const SECTION = "tokeneconomizer";
+
+const truthy = (v: string | undefined) => ["1", "true", "yes", "on"].includes((v ?? "").trim().toLowerCase());
+
+/** Run one of the plugin's launchers with `input` on stdin; resolve to its stdout. */
+function run(script: string, args: string[], input: string, mode: string): Promise<string> {
+ return new Promise((resolve) => {
+ let out = "";
+ let child;
+ try {
+ child = spawn(join(SCRIPTS, script), args, {
+ env: { ...process.env, TE_PI_MODE: mode },
+ stdio: ["pipe", "pipe", "ignore"],
+ });
+ } catch {
+ resolve("");
+ return;
+ }
+ const timer = setTimeout(() => {
+ child.kill("SIGKILL");
+ resolve("");
+ }, HOOK_TIMEOUT_MS);
+ child.stdout.on("data", (d: Buffer) => (out += d.toString()));
+ child.on("error", () => {
+ clearTimeout(timer);
+ resolve("");
+ });
+ child.on("close", () => {
+ clearTimeout(timer);
+ resolve(out);
+ });
+ child.stdin.on("error", () => {});
+ child.stdin.end(input);
+ });
+}
+
+/** One dispatcher call; `{}` (no change) on any failure. */
+async function hook(payload: Record<string, unknown>, mode: string): Promise<any> {
+ if (truthy(process.env.TE_SIDECALL) || truthy(process.env.TE_DISABLE)) return {};
+ const out = await run("te-hook", ["--host", "pi"], JSON.stringify(payload), mode);
+ try {
+ return JSON.parse(out.trim().split("\n").pop() || "{}") ?? {};
+ } catch {
+ return {};
+ }
+}
+
+function notify(ctx: ExtensionContext, text: unknown) {
+ if (typeof text !== "string" || !text) return;
+ if (ctx.hasUI) ctx.ui.notify(text, "info");
+ else if (truthy(process.env.TE_VERBOSE)) console.error(text);
+}
+
+const SOURCE: Record<string, string> = {
+ // A new conversation starts empty; a resumed, forked or reloaded one already carries the
+ // section it was given, in its own system-message history.
+ startup: "startup",
+ new: "clear",
+ resume: "resume",
+ fork: "resume",
+ reload: "resume",
+};
+
+export default function tokeneconomizer(pi: ExtensionAPI) {
+ let manual = "";
+ let prompts = 0;
+ let turn = "";
+
+ const base = (ctx: ExtensionContext) => ({
+ session_id: ctx.sessionManager.getSessionId(),
+ transcript_path: ctx.sessionManager.getSessionFile() ?? "",
+ cwd: ctx.cwd,
+ });
+
+ pi.on("session_start", async (event, ctx) => {
+ prompts = 0;
+ const out = await hook(
+ { ...base(ctx), hook_event_name: "SessionStart", source: SOURCE[event.reason] ?? "startup" },
+ ctx.mode,
+ );
+ manual = out?.hookSpecificOutput?.additionalContext ?? "";
+ notify(ctx, out?.systemMessage);
+ });
+
+ pi.on("before_agent_start", async (event, ctx) => {
+ prompts += 1;
+ turn = `${ctx.sessionManager.getSessionId()}:${prompts}`;
+ await hook(
+ { ...base(ctx), hook_event_name: "UserPromptSubmit", prompt_id: turn, prompt: event.prompt },
+ ctx.mode,
+ );
+ if (manual) {
+ event.systemPromptOptions.sections = { ...(event.systemPromptOptions.sections ?? {}), [SECTION]: manual };
+ }
+ });
+
+ pi.on("tool_result", async (event, ctx) => {
+ const content = (event.content ?? [])
+ .filter((c: any) => c?.type === "text")
+ .map((c: any) => c.text)
+ .join("\n");
+ const structured = (event.structuredContent ?? {}) as Record<string, unknown>;
+ const out = await hook(
+ {
+ ...base(ctx),
+ hook_event_name: "PostToolUse",
+ prompt_id: turn,
+ tool_name: event.toolName,
+ tool_use_id: event.toolCallId,
+ tool_input: event.input ?? {},
+ tool_response: { content, isError: event.isError, exit_code: structured.exit_code },
+ },
+ ctx.mode,
+ );
+ const nudge = out?.hookSpecificOutput?.additionalContext;
+ if (nudge) {
+ pi.sendMessage({ customType: SECTION, content: nudge, display: true }, { deliverAs: "steer" });
+ }
+ notify(ctx, out?.systemMessage);
+ if (out?.continue === false) {
+ notify(ctx, out?.stopReason);
+ ctx.abort();
+ }
+ });
+
+ pi.on("agent_end", async (_event, ctx) => {
+ await hook({ ...base(ctx), hook_event_name: "Stop", prompt_id: turn, stop_hook_active: false }, ctx.mode);
+ });
+
+ pi.on("session_shutdown", async (event, ctx) => {
+ await hook({ ...base(ctx), hook_event_name: "SessionEnd", reason: event.reason }, ctx.mode);
+ });
+
+ pi.registerCommand("te", {
+ description: "TokenEconomizer: status | facts | replay | config (what it learned here, what it did)",
+ handler: async (args, ctx) => {
+ const argv = (args || "status").trim().split(/\s+/).filter(Boolean);
+ const out = await run("te", ["--host", "pi", "--cwd", ctx.cwd, ...argv], "", ctx.mode);
+ notify(ctx, out.trim() || "te: no output (is Python 3.10+ on PATH?)");
+ },
+ });
+}
diff --git a/plugin/scripts/te b/plugin/scripts/te
new file mode 100644
index 0000000..833c3d5
--- /dev/null
+++ b/plugin/scripts/te
@@ -0,0 +1,10 @@
+#!/bin/sh
+# TokenEconomizer command line (`te status`, `te untool show`, `te replay`, ...).
+dir=$(CDPATH= cd -- "${0%/*}" && pwd)
+for py in "${TE_PYTHON:-}" python3 python; do
+ if [ -n "$py" ] && command -v "$py" >/dev/null 2>&1; then
+ exec "$py" "$dir/te_bootstrap.py" cli "$@"
+ fi
+done
+echo "te: no python3 on PATH (set TE_PYTHON)" >&2
+exit 1
diff --git a/plugin/scripts/te-hook b/plugin/scripts/te-hook
new file mode 100644
index 0000000..af3d45d
--- /dev/null
+++ b/plugin/scripts/te-hook
@@ -0,0 +1,12 @@
+#!/bin/sh
+# TokenEconomizer hook launcher: run the hook dispatcher under the first Python 3 on PATH.
+# Fails open -- with no usable Python the hook prints an empty object, which changes nothing.
+dir=$(CDPATH= cd -- "${0%/*}" && pwd)
+for py in "${TE_PYTHON:-}" python3 python; do
+ if [ -n "$py" ] && command -v "$py" >/dev/null 2>&1; then
+ exec "$py" "$dir/te_bootstrap.py" hook "$@"
+ fi
+done
+# Drain stdin with a builtin (no `cat` when PATH is empty), then answer "no change".
+while IFS= read -r _line; do :; done
+echo '{}'
diff --git a/plugin/scripts/te_bootstrap.py b/plugin/scripts/te_bootstrap.py
new file mode 100644
index 0000000..0114f6c
--- /dev/null
+++ b/plugin/scripts/te_bootstrap.py
@@ -0,0 +1,68 @@
+"""Locate TokenEconomizer (`tokeneconomizer` + `te_live`) and run the hook dispatcher (`hook`)
+or the command line (`cli`).
+
+The core package is found, in order:
+
+ 1. `$TE_CORE_PATH` -- a checkout you point at explicitly;
+ 2. `<plugin root>/lib` -- in the repository, symlinks to the two packages; Claude
+ Code's install copies the plugin directory into its
+ cache (even from a local marketplace) and follows them,
+ so the installed copy is self-contained. In the bundle
+ `scripts/build_plugin.py` writes, real copies;
+ 3. `<plugin root>/..` -- the repository this plugin directory lives in, for a
+ checkout whose symlinks were not created (`--plugin-dir`);
+ 4. an installed package -- `pip install` of the repository.
+
+This file must stay importable by any Python 3 so that an old interpreter can still answer a
+hook with `{}` instead of a syntax error: no f-strings, no annotations, nothing newer than 3.0.
+"""
+import os
+import sys
+
+MIN_VERSION = (3, 10)
+
+
+def core_path():
+ plugin_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ for cand in (os.environ.get("TE_CORE_PATH"), os.path.join(plugin_root, "lib"),
+ os.path.dirname(plugin_root)):
+ if cand and os.path.isfile(os.path.join(cand, "te_live", "hook.py")) and \
+ os.path.isfile(os.path.join(cand, "tokeneconomizer", "__init__.py")):
+ return cand
+ return None
+
+
+def main():
+ mode = sys.argv[1] if len(sys.argv) > 1 else "hook"
+ rest = sys.argv[2:]
+ if sys.version_info < MIN_VERSION:
+ if mode == "hook":
+ sys.stdin.read()
+ sys.stdout.write("{}\n")
+ return 0
+ sys.stderr.write("TokenEconomizer needs Python >= %d.%d (this is %s); set TE_PYTHON\n"
+ % (MIN_VERSION[0], MIN_VERSION[1], sys.version.split()[0]))
+ return 1
+ # Everything the plugin writes under its state root is derived from the agents' session
+ # logs, which the agents keep private; so are these files.
+ os.umask(0o077)
+ # Where this plugin is, for the stable launcher the hooks keep (`paths.ensure_launcher`).
+ os.environ["TE_PLUGIN_ROOT"] = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ core = core_path()
+ if core:
+ sys.path.insert(0, core)
+ try:
+ if mode == "hook":
+ from te_live import hook
+ return hook.main(rest)
+ from te_live import cli
+ return cli.main(rest)
+ except ImportError:
+ if mode == "hook":
+ sys.stdout.write("{}\n")
+ return 0
+ raise
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..9283263
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,57 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tokeneconomizer"
+version = "0.1.0"
+description = "Runtime cost control for coding agents (Claude Code, Codex, Copilot CLI, Pi), delivered from outside the agent as a plugin."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "TokenEconomizer authors" }]
+keywords = ["llm-agents", "cost-optimization", "swe-bench", "token-economy"]
+
+# The core package is intentionally STDLIB-ONLY: the three mechanisms + the vendored
+# pricing/reflect/spiral building blocks import nothing outside the standard library, so
+# `import tokeneconomizer` works in any Python >=3.10 with zero install-time dependencies.
+dependencies = []
+
+[project.scripts]
+# The live layer's command line (status, facts, untool, replay, ...). The plugin ships its own
+# launcher (plugin/scripts/te) and does not need the package installed.
+te = "te_live.cli:main"
+
+[project.optional-dependencies]
+# Test suite is offline (no Docker / no datasets).
+test = ["pytest>=7.0"]
+# The reference task-execution harness under experiments/te_harness/ (see
+# experiments/docs/HARNESS.md). It adds no
+# PyPI dependency -- it is stdlib-only at import time and the official SWE-bench grader is
+# imported lazily from a checkout named by TE_SWEBENCH_REPO -- so this extra installs
+# nothing. It exists to give "I intend to run the harness" a name, since what the harness
+# actually needs cannot come from pip: Docker, an agent CLI on PATH, an API key, the
+# benchmark images, and for Web-Bench a built Rush monorepo.
+harness = []
+# The repo-map external baseline (tokeneconomizer/baselines/repomap.py) generates aider's
+# token-budgeted map on the HOST -- the task container's Python is pinned to the issue's
+# era and cannot run a modern tree-sitter stack. This is the ONLY baseline with a host
+# dependency: rtk ships a static binary and Serena installs itself in-container.
+#
+# It stays an EXTRA, and aider is imported inside `preflight()` rather than at module
+# scope, so `import tokeneconomizer` keeps its zero-dependency guarantee even though the
+# baselines now ship inside the core package. A machine that will never run the repomap
+# arms never needs aider installed.
+baselines = ["aider-chat>=0.60"]
+
+[tool.setuptools.packages.find]
+# The core package and the plugin's live layer ONLY. The paper's reproduction code
+# (experiments/, including te_harness) is deliberately not distributed: `import
+# tokeneconomizer` must not be able to reach Docker or an agent CLI, and excluding it here is
+# what makes that structural rather than a convention.
+include = ["tokeneconomizer*", "te_live*"]
+exclude = ["experiments*", "te_harness*", "tests*", "scripts*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests", "experiments/tests"]
+addopts = "-q"
diff --git a/scripts/build_plugin.py b/scripts/build_plugin.py
new file mode 100644
index 0000000..c8fadf3
--- /dev/null
+++ b/scripts/build_plugin.py
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Assemble a self-contained TokenEconomizer plugin bundle and a marketplace around it.
+
+ python scripts/build_plugin.py [--out dist/marketplace] [--zip]
+
+Why a build step at all. Loaded in place -- `claude --plugin-dir plugin/`, or this repository
+added as a local-directory marketplace -- the plugin finds the core package one level up, in the
+checkout itself. A marketplace INSTALL is different: Claude Code copies only the plugin directory
+into its cache ("files outside the plugin directory aren't copied"), and Codex does the same, so
+an installed copy of `plugin/` alone has no `tokeneconomizer` to import. The bundle fixes that by
+vendoring the core and the live harness (`tokeneconomizer/`, `te_live/`) into `<plugin>/lib/`,
+which `scripts/te_bootstrap.py` looks in first.
+
+Both are stdlib-only, so vendoring them is the whole dependency story: the bundle runs on any
+Python >= 3.10 with nothing installed. `te_harness/`, `scripts/` and the docs are not included.
+
+Layout written to --out (a marketplace root you can `marketplace add`, publish as a branch or a
+repository of its own, or zip for `claude --plugin-url`):
+
+ <out>/.claude-plugin/marketplace.json
+ <out>/tokeneconomizer/ the plugin, with lib/{tokeneconomizer,te_live} vendored
+"""
+import argparse
+import json
+import os
+import shutil
+import sys
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+IGNORE = shutil.ignore_patterns("__pycache__", "*.pyc", ".pytest_cache")
+
+
+def build(out):
+ out = os.path.abspath(out)
+ plugin_out = os.path.join(out, "tokeneconomizer")
+ if os.path.exists(plugin_out):
+ shutil.rmtree(plugin_out)
+ # plugin/lib holds symlinks to the core for an install from a checkout (see te_bootstrap);
+ # the bundle replaces them with real copies.
+ shutil.copytree(os.path.join(REPO, "plugin"), plugin_out,
+ ignore=shutil.ignore_patterns("__pycache__", "*.pyc", ".pytest_cache", "lib"))
+ for pkg in ("tokeneconomizer", "te_live"):
+ shutil.copytree(os.path.join(REPO, pkg), os.path.join(plugin_out, "lib", pkg),
+ ignore=IGNORE)
+ with open(os.path.join(REPO, ".claude-plugin", "marketplace.json"), encoding="utf-8") as fh:
+ market = json.load(fh)
+ for entry in market.get("plugins", []):
+ if entry.get("name") == "tokeneconomizer":
+ entry["source"] = "./tokeneconomizer"
+ os.makedirs(os.path.join(out, ".claude-plugin"), exist_ok=True)
+ with open(os.path.join(out, ".claude-plugin", "marketplace.json"), "w",
+ encoding="utf-8") as fh:
+ json.dump(market, fh, indent=2)
+ fh.write("\n")
+ return plugin_out
+
+
+def main(argv=None):
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--out", default=os.path.join(REPO, "dist", "marketplace"))
+ ap.add_argument("--zip", action="store_true",
+ help="also write <out>/../tokeneconomizer-plugin.zip of the plugin directory")
+ args = ap.parse_args(argv)
+ plugin_out = build(args.out)
+ print(f"plugin bundle: {plugin_out}")
+ print(f"marketplace: {os.path.abspath(args.out)}")
+ if args.zip:
+ base = os.path.join(os.path.dirname(os.path.abspath(args.out)), "tokeneconomizer-plugin")
+ path = shutil.make_archive(base, "zip", root_dir=os.path.dirname(plugin_out),
+ base_dir=os.path.basename(plugin_out))
+ print(f"zip: {path}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/te_live/__init__.py b/te_live/__init__.py
new file mode 100644
index 0000000..591bf5a
--- /dev/null
+++ b/te_live/__init__.py
@@ -0,0 +1,39 @@
+"""The live harness: the three mechanisms inside a real coding-agent session, through the hooks
+the agent already exposes.
+
+A sibling of `te_harness`, not a submodule of the core, for the same reason: it reaches an agent
+CLI (`claude -p`, `codex exec`) for its opt-in side calls and catalog probe, and
+`import tokeneconomizer` must not be able to. `te_harness` runs the controller over benchmark
+streams in Docker; `te_live` runs it inside the user's own sessions. The core never learns which.
+
+The research driver (`experiments/scripts/run_experiment.py`) holds one `Economizer` in one process for a
+whole stream of benchmark tasks, launches the sealed agent itself, and applies each decision at
+launch: a prompt prefix, `--disallowed-tools`, a kill-and-resume on the live JSON stream. A
+plugin has none of that. The user launches the agent, and the controller runs only when the agent
+calls one of its hooks -- each call a fresh, short-lived process. This package turns the
+stream-shaped core into that event-shaped one:
+
+ * state -- per-repository, per-host state on disk (`paths`, `store`, `config`), so what
+ is learned survives between hook processes, sessions and plugin updates;
+ * hosts -- one adapter per agent (`hosts`, `transcripts`): hook payloads in, hook
+ outputs out, and the agent's own session log turned into a trace the existing
+ `trace` seam already reads;
+ * mechanisms -- `fold` runs the SAME classes the experiments ran (`PrefixStage`,
+ `ToolingStage`) over each finished session, and `guard` replays
+ `vendor.spiral.NudgeGuard`'s control flow one tool call at a time;
+ * entry points -- `hook.main` (what the plugin's hooks execute) and `cli.main` (`te`).
+
+What changes relative to the measured setting is written down where it changes, because it is
+the difference between a mechanism that ports and a number that does not:
+
+ * there is no grader, so the spiral envelope learns from a "turn ended green" proxy;
+ * a "task" is a session for injection and untooling, but a prompt-turn for the spiral guard --
+ in interactive use the human already segments work into short turns;
+ * tool removal is a deny list the user approves, applied from the next session, never a
+ mid-session change (that would also invalidate the prompt cache);
+ * a nudge is a mid-turn reminder attached to a tool result, not a resumed segment, so its
+ effect has to be re-measured rather than inherited from the published rows.
+
+See docs/PLUGIN.md. Stdlib only, like the rest of the core: a hook must start in tens of
+milliseconds and must not depend on anything the user did not install.
+"""
diff --git a/te_live/accounting.py b/te_live/accounting.py
new file mode 100644
index 0000000..46d4279
--- /dev/null
+++ b/te_live/accounting.py
@@ -0,0 +1,54 @@
+"""The session ledger: what each session cost, on the experiments' basis, and what the plugin
+itself spent.
+
+Cost is `pricing.effective_tokens` -- fresh-input-token equivalents with cached re-reads at their
+billed fraction -- plus every mechanism-overhead field `ledger.OVERHEAD_KEYS` names, exactly the
+amortized definition the results tables use. One live-only allowance: `pricing.weights_for`
+hard-fails on a billed id missing from its table (right for an experiment, where a mispriced row
+corrupts a comparison). A user's session must still be recorded, so an unpriced id is priced on
+a labelled fallback curve and the record says `priced_by: "fallback"`; Claude ids never need it,
+because every `claude-*` id already resolves to the Claude curve.
+"""
+from tokeneconomizer import ledger
+from tokeneconomizer.vendor import pricing
+
+FALLBACK_WEIGHTS = dict(input=1.0, cache_read=0.10, cache_write=1.25, output=6.0)
+
+
+def effective(usage):
+ """(effective_tokens, priced_by) for a usage breakdown; (None, "no_usage") without one."""
+ if not usage or usage.get("input_tokens") is None:
+ return None, "no_usage"
+ rec = {k: usage.get(k) for k in ("input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens", "model")}
+ try:
+ return pricing.effective_tokens(rec), "model"
+ except ValueError:
+ return pricing.effective_tokens(rec, weights=FALLBACK_WEIGHTS), "fallback"
+
+
+def overhead_total(overhead):
+ return ledger.overhead(overhead or {})
+
+
+def summarize(records):
+ """Aggregate ledger records into the numbers `te status` prints."""
+ sessions = [r for r in records if r.get("kind") == "session"]
+ priced = [r for r in sessions if isinstance(r.get("effective_tokens"), (int, float))]
+ exec_total = sum(r["effective_tokens"] for r in priced)
+ # `overhead` records are paid calls charged as they happened (the background stuck judge),
+ # so a call that outlived its session's fold is still in the account.
+ oh_total = sum(overhead_total(r.get("overhead")) for r in records
+ if r.get("kind") in ("session", "overhead"))
+ return {
+ "sessions": len(sessions),
+ "priced_sessions": len(priced),
+ "effective_tokens": exec_total,
+ "overhead_tokens": oh_total,
+ "overhead_frac": (oh_total / (exec_total + oh_total)) if (exec_total + oh_total) else 0.0,
+ "requests": sum(int(r.get("requests") or 0) for r in sessions),
+ "nudges": sum(int(r.get("nudges") or 0) for r in sessions),
+ "aborts": sum(int(r.get("aborts") or 0) for r in sessions),
+ "turns": sum(int(r.get("turns") or 0) for r in sessions),
+ "injected_sessions": sum(1 for r in sessions if r.get("injected_chars")),
+ }
diff --git a/te_live/adapter.py b/te_live/adapter.py
new file mode 100644
index 0000000..ecee7a2
--- /dev/null
+++ b/te_live/adapter.py
@@ -0,0 +1,150 @@
+"""`LiveAdapter`: the DomainAdapter for a real repository in a real session.
+
+The mechanisms read their domain only through a DomainAdapter (`adapters/base.py`), so running
+`PrefixStage` and `ToolingStage` on live sessions needs no mechanism edit -- only an adapter whose
+answers fit an interactive session instead of a sealed benchmark task:
+
+ * `protected_tools` adds what a PERSON needs to the work surface. The benchmark protected set
+ keeps Bash/Read/Edit/...; `reflect.INAPPLICABLE_TOOLS` then treats AskUserQuestion, plan
+ mode, subagents and the web as inapplicable "by construction" for a headless sealed task. A
+ person does use them interactively, so those are protected here -- the web tools always,
+ however long they go unused -- and only evidence (loaded but unused for N sessions) can
+ propose anything else.
+ * `parse_catalog_with_source` reports what the session COULD call. An interactive transcript
+ does not record the tool list the way `stream-json`'s init event does, so the catalog is the
+ union of a probed snapshot (`te untool refresh-catalog`, stored under the state root), the
+ deferred tools the transcript lists, and the tools it used. Without a snapshot only deferred
+ tools can be proposed -- they cost one listing line each, so that is honest but small.
+ * `env_vocabulary` picks the environment-fact vocabulary from the repository's own markers
+ (`package.json` -> Node, Python packaging files -> Python), overridable by config.
+ * `problem_statements` is empty: there is no issue text to give the per-task tool judge.
+"""
+import json
+import os
+
+from tokeneconomizer import trace
+from tokeneconomizer.vendor import reflect, spiral
+from . import transcripts
+
+# Tools a person relies on in an interactive session, beyond the benchmark work surface.
+INTERACTIVE_PROTECTED = frozenset({
+ "Bash", "Read", "Edit", "MultiEdit", "Write", "Grep", "Glob", "LS", "NotebookRead",
+ "BashOutput", "KillShell", "KillBash",
+ "AskUserQuestion", "EnterPlanMode", "ExitPlanMode", "TodoWrite",
+ "TaskCreate", "TaskUpdate", "TaskGet", "TaskList", "TaskOutput", "TaskStop",
+ "Agent", "Task", "Skill", "ToolSearch", "SlashCommand", "Monitor",
+ # The web is never proposed. A run of self-contained tasks can leave it unused for long
+ # stretches, and a person's next task may need it; a
+ # deny list is easy to accept without reading. Untooling is for tools a repository's work
+ # genuinely never needs, and nothing says that of the web.
+ "WebFetch", "WebSearch",
+})
+# The Copilot CLI's own work surface and a person's tools there (1.0.83 names). Only MCP servers
+# can be removed persistently on Copilot (`untool.copilot_server_proposal`), so this matters
+# for what the evidence reports, not for what can be written.
+COPILOT_PROTECTED = frozenset({
+ "bash", "read_bash", "write_bash", "stop_bash", "list_bash", "view", "edit", "create",
+ "str_replace", "apply_patch", "custom", "rg", "grep", "glob", "task", "skill",
+ "read_agent", "list_agents", "write_agent", "ask_user", "web_fetch", "web_search",
+ "report_intent", "update_todo", "sql",
+})
+# Rarely used, but expensive to lose when a task needs them: proposed only after
+# HIGH_RISK_FACTOR times the usual evidence.
+HIGH_RISK = frozenset({"NotebookEdit"})
+HIGH_RISK_FACTOR = 3
+
+_PY_MARKERS = ("pyproject.toml", "setup.py", "setup.cfg", "requirements.txt", "Pipfile",
+ "tox.ini", "environment.yml")
+_NODE_MARKERS = ("package.json",)
+
+
+def detect_vocabulary(repo_root, preference="auto"):
+ from tokeneconomizer.mechanisms.envfacts import NODE_WEB, PYTHON_GIT
+ if preference == "node-web":
+ return NODE_WEB
+ if preference == "python-git":
+ return PYTHON_GIT
+ has = lambda names: any(os.path.exists(os.path.join(repo_root or "", n)) for n in names)
+ if has(_NODE_MARKERS) and not has(_PY_MARKERS):
+ return NODE_WEB
+ return PYTHON_GIT
+
+
+def catalog_path(state_root, host):
+ return os.path.join(state_root, "hosts", host, "catalog.json")
+
+
+def load_catalog(state_root, host):
+ """{tool: description} from the last `te untool refresh-catalog` probe, {} if none."""
+ try:
+ with open(catalog_path(state_root, host), encoding="utf-8") as fh:
+ data = json.load(fh)
+ return {t: "" for t in data.get("tools", []) if isinstance(t, str)}
+ except (OSError, ValueError):
+ return {}
+
+
+class LiveAdapter:
+ name = "live"
+
+ def __init__(self, host, repo_root, vocabulary="auto", catalog=None):
+ self.host = host
+ self.repo_root = repo_root
+ self.vocabulary = vocabulary
+ self.catalog = dict(catalog or {})
+
+ # ---- trajectory reading ----
+ def load_trajectory(self, path):
+ return trace.load_trajectory(path)
+
+ def tool_usage(self, path):
+ return self.host.tool_usage(path)
+
+ def parse_catalog(self, run_dir_path, agent):
+ return self.parse_catalog_with_source(run_dir_path, agent)[0]
+
+ def parse_catalog_with_source(self, run_dir_path, agent):
+ transcript = os.path.realpath(os.path.join(run_dir_path, "transcript.jsonl"))
+ cat = dict(self.catalog)
+ observed = set(self.tool_usage(transcript))
+ if self.host.name == "claude":
+ observed |= transcripts.claude_deferred_tools(transcript)
+ elif self.host.name == "copilot":
+ # Some builds record the request's tool list in `session.usage_checkpoint`.
+ observed |= set(trace.parse_tool_names_copilot(
+ open(transcript, encoding="utf-8", errors="replace").read()
+ if os.path.exists(transcript) else ""))
+ for name in observed:
+ cat.setdefault(name, "")
+ if not cat:
+ return {}, None
+ return cat, ("probe+observed" if self.catalog else "observed")
+
+ # ---- waste signals ----
+ def detect(self, path):
+ return reflect.detect(path)
+
+ # ---- untooling tool-sets ----
+ def protected_tools(self):
+ extra = COPILOT_PROTECTED if self.host.name == "copilot" else frozenset()
+ return reflect.PROTECTED_TOOLS | INTERACTIVE_PROTECTED | extra
+
+ def inapplicable_tools(self):
+ return () # nothing is inapplicable by construction here
+
+ def hardcoded_exclusions(self, used_tools):
+ return []
+
+ # ---- spiral envelope ----
+ def spiral_snapshot(self, path):
+ return spiral.state_from_trajectory(path).snapshot()
+
+ # ---- per-task judge inputs ----
+ def problem_statements(self):
+ return {}
+
+ def env_vocabulary(self):
+ return detect_vocabulary(self.repo_root, self.vocabulary)
+
+ def classify_domain(self):
+ return None
diff --git a/te_live/cli.py b/te_live/cli.py
new file mode 100644
index 0000000..6eddbf1
--- /dev/null
+++ b/te_live/cli.py
@@ -0,0 +1,665 @@
+"""`te` -- inspect and steer what TokenEconomizer does in your sessions.
+
+ te status what it has learned for this repository, and what it did
+ te facts the manual it injects, and the facts still forming
+ te untool show the proposed deny list, and the evidence behind it
+ te untool apply [ITEM ...] write the proposal (or the named items) to the repository's
+ settings: tools (Claude Code), capabilities (Codex) or
+ MCP servers (Copilot)
+ te untool restore [ITEM ...] remove entries this plugin added
+ te untool refresh-catalog record the tool list a fresh session loads (Claude, Copilot)
+ te untool measure measure the request with and without the proposal
+ te replay [--limit N] [--commit] run the mechanisms over your existing session history
+ te config show each mechanism on or off, its settings, the guard's thresholds
+ te config set KEY=VALUE ... e.g. `spiral=off`, `inject=on`, `spiral.min_turns=12`
+ (this repository; `--global` for every repository)
+ te config unset KEY ... back to the next source down (the global file, then default)
+ te doctor check the installation
+ te --host copilot probe real Copilot sessions in a throwaway repository: does a
+ learned fact change what the model does, and what does the
+ deny list save (makes model calls; see te_live/probe.py)
+
+`--cwd` picks the repository (default: the current directory) and `--host` the agent
+(claude | codex | copilot | pi; default: `TE_HOST`, else the agent whose shell runs `te`, else
+claude). `replay` is the way to see what the mechanisms would do on
+your own sessions before turning any of them on: it folds your history into a scratch state and
+replays the spiral guard over every prompt-turn, with no LLM call. `--commit` keeps the result as
+the repository's starting state, so learning does not begin from zero.
+"""
+import argparse
+import glob
+import json
+import os
+import re
+import sys
+import tempfile
+import time
+
+from . import accounting, fold, guard, hosts, untool
+from .config import unset_config, write_config
+from .context import LiveContext
+from .paths import state_root
+from .store import Store
+
+
+def _ctx(args, env=None):
+ host = hosts.HOSTS[args.host]()
+ return LiveContext.build(host, args.cwd, env)
+
+
+def _fmt_tokens(n):
+ if n is None:
+ return "n/a"
+ n = float(n)
+ for unit, div in (("B", 1e9), ("M", 1e6), ("k", 1e3)):
+ if abs(n) >= div:
+ return f"{n / div:.1f}{unit}"
+ return f"{n:.0f}"
+
+
+# ---- status / facts ------------------------------------------------------------------------
+def cmd_status(args):
+ ctx = _ctx(args)
+ cfg = ctx.config
+ out = {"repo": ctx.repo.label, "repo_root": ctx.repo.root, "host": ctx.host.name,
+ "state": ctx.store.base, "config": cfg.as_dict(), "warnings": cfg.warnings}
+ inj = ctx.store.read("injection.json") or {}
+ unt = ctx.store.read("untool.json") or {}
+ env_tracker = guard.load_envelope(ctx.store, cfg.spiral_aggr, cfg.spiral_thresholds)
+ ledger = accounting.summarize(ctx.store.read_lines("ledger.jsonl"))
+ out.update({
+ "injection": {"profile": cfg.inject_effective,
+ "injected_lines": sum(1 for ln in (inj.get("rendered") or "").splitlines()
+ if ln.startswith("- ")),
+ "sessions_folded": inj.get("sessions", 0)},
+ "untool": {"mode": cfg.untool, "sessions": unt.get("sessions", 0),
+ "catalog_source": unt.get("catalog_source"),
+ "proposal": unt.get("proposal", []),
+ "applied": untool.applied(ctx.store)},
+ "spiral": {"mode": cfg.spiral, "envelope": env_tracker.envelope,
+ "ended_green_turns": len(env_tracker.resolved_turns)},
+ "ledger": ledger})
+ if args.json:
+ print(json.dumps(out, indent=2, default=str))
+ return 0
+ print(f"TokenEconomizer -- {out['repo']} ({ctx.host.name})")
+ print(f" state {out['state']}")
+ print(f" config inject={cfg.inject_effective} untool={cfg.untool} spiral={cfg.spiral} "
+ f"mode={cfg.mode} paid={cfg.paid} shadow={cfg.shadow}")
+ for w in cfg.warnings:
+ print(f" warning {w}")
+ print(f" sessions {ledger['sessions']} folded, {_fmt_tokens(ledger['effective_tokens'])} "
+ f"effective tokens, plugin overhead {_fmt_tokens(ledger['overhead_tokens'])} "
+ f"({ledger['overhead_frac']:.1%})")
+ i = out["injection"]
+ print(f" injection {i['injected_lines']} fact(s) injected at session start "
+ f"({i['sessions_folded']} sessions learned from)")
+ u = out["untool"]
+ if not untool.supports(ctx.host.name):
+ print(" untooling not available on this host")
+ else:
+ src = u["catalog_source"] or "none yet"
+ print(f" untooling {u['sessions']} sessions observed, catalog: {src}; "
+ f"proposed {untool.unit(ctx.host.name, plural=True)}: "
+ f"{', '.join(u['proposal']) or '-'}; applied: {', '.join(u['applied']) or '-'}")
+ sp = out["spiral"]
+ env_s = f"{sp['envelope']:.0f} tool calls" if sp["envelope"] else "not learned yet"
+ print(f" spiral envelope {env_s} ({sp['ended_green_turns']} ended-green turns); "
+ f"{ledger['nudges']} nudge(s), {ledger['aborts']} stop(s) over {ledger['turns']} turns")
+ return 0
+
+
+def cmd_facts(args):
+ ctx = _ctx(args)
+ inj = ctx.store.read("injection.json") or {}
+ text = inj.get("rendered") or ""
+ print(text.rstrip() or "(nothing injected yet: no fact has cleared its firing gate)")
+ facts = inj.get("facts") or {}
+ rows = [(k, v) for k, v in facts.items() if isinstance(v, dict) and "state" in v]
+ if rows:
+ print("\nfacts tracked:")
+ for k, v in sorted(rows):
+ print(f" {v.get('state', '?'):9s} fire={v.get('fire', '?'):>5s} "
+ f"{'INJECTED' if v.get('inject') else ' '} {k}")
+ return 0
+
+
+# ---- untool --------------------------------------------------------------------------------
+def _untool_show(ctx, unt):
+ host = ctx.host.name
+ print(f"sessions observed: {unt.get('sessions', 0)} catalog: "
+ f"{unt.get('catalog_source') or 'none yet'}")
+ print(f"applied ({untool.unit(host, plural=True)} this plugin added to "
+ f"{untool.settings_path(ctx.repo.root, host)}): "
+ f"{', '.join(untool.applied(ctx.store)) or '-'}")
+ print(f"proposed: {', '.join(unt.get('proposal', [])) or '-'}")
+ if host == "codex":
+ caps = unt.get("capabilities") or {}
+ if caps:
+ print("capabilities (removable from a project's .codex/config.toml):")
+ for c, e in sorted(caps.items()):
+ print(f" {c:14s} loaded in {e['loaded_sessions']:3d} sessions, unused in "
+ f"{e['unused_sessions']:3d}{' USED (kept)' if e['used'] else ''}")
+ trusted = untool.codex_project_trusted(ctx.repo.root)
+ if trusted is False:
+ print("note: Codex reads a project's .codex/config.toml only in a TRUSTED project; "
+ "trust this one in Codex first, or an applied proposal does nothing.")
+ return 0
+ if host == "copilot":
+ servers = unt.get("servers") or {}
+ if servers:
+ print("MCP servers (Copilot can only remove a whole server):")
+ for name, e in sorted(servers.items()):
+ state = "USED (kept)" if e["used"] else f"unused in {e['unused_sessions']} sessions"
+ print(f" {name:24s} {e['tools']:3d} tools, {state}")
+ streaks = unt.get("streaks") or {}
+ if streaks and host == "claude":
+ print("unused-session streaks (tools never used are candidates once over "
+ f"{ctx.config.untool_evidence}):")
+ for t, n in sorted(streaks.items(), key=lambda kv: -kv[1])[:30]:
+ used = "used" if t in set(unt.get("used_tools", [])) else ""
+ print(f" {n:4d} {t} {used}")
+ if not unt.get("catalog_source") or unt.get("catalog_source") == "observed":
+ print(f"note: without `te --host {host} untool refresh-catalog` only "
+ + ("deferred and observed" if host == "claude" else "observed")
+ + " tools are known, which leaves the always-loaded tools out of the evidence.")
+ return 0
+
+
+def cmd_untool(args):
+ ctx = _ctx(args)
+ host = ctx.host.name
+ if not untool.supports(host):
+ print(f"untooling is not available on {host} (it declares only a handful of tools)")
+ return 1
+ unt = ctx.store.read("untool.json") or {}
+ action = args.action
+ if action == "show":
+ return _untool_show(ctx, unt)
+ if action == "apply":
+ tools = args.tools or unt.get("proposal", [])
+ if not tools:
+ print("nothing to apply")
+ return 0
+ if host == "codex":
+ bad = [t for t in tools if t not in untool.CODEX_TOGGLES]
+ if bad:
+ print(f"not a removable Codex capability: {', '.join(bad)} (choose from "
+ f"{', '.join(untool.CODEX_LIVE_CAPABILITIES)})")
+ return 2
+ try:
+ added = untool.apply(ctx.repo.root, tools, ctx.store, host=host)
+ except RuntimeError as e:
+ print(f"not applied: {e}")
+ return 1
+ print(f"added: {', '.join(added) or '(already present)'}\n"
+ f"in {untool.settings_path(ctx.repo.root, host)} -- takes effect from the next "
+ f"session.")
+ if host == "copilot" and added:
+ if untool.copilot_folder_trusted(ctx.repo.root) is False:
+ print("warning: Copilot reads a repository's .github/copilot settings only in a "
+ "trusted folder, and this one is not. Trust it (Copilot asks when you "
+ "start it here), or the change does nothing.")
+ left = untool.verify_copilot(ctx.repo.root, untool.applied(ctx.store))
+ if left is not None:
+ print("checked with a local capture: " + ", ".join(
+ f"{k} {'removed' if not v else f'still loaded ({v} tools)'}"
+ for k, v in sorted(left.items())))
+ if host == "codex" and added:
+ if untool.codex_project_trusted(ctx.repo.root) is False:
+ print("warning: this project is not trusted in Codex, which ignores a "
+ "project's .codex/config.toml until it is. Trust it (Codex asks when "
+ "you start it here), or the change does nothing.")
+ state = untool.verify_codex(ctx.repo.root, untool.applied(ctx.store))
+ if state:
+ print("codex features list: " + ", ".join(
+ f"{k}={'on' if v else 'off'}" for k, v in sorted(state.items())))
+ return 0
+ if action == "restore":
+ try:
+ removed = untool.restore(ctx.repo.root, ctx.store, args.tools or None, host=host)
+ except RuntimeError as e:
+ print(f"not restored: {e}")
+ return 1
+ print(f"removed: {', '.join(removed) or '-'} (from the next session)")
+ return 0
+ if action == "refresh-catalog":
+ if host == "codex":
+ print("Codex needs no catalog: its removable capabilities are a fixed set "
+ f"({', '.join(untool.CODEX_LIVE_CAPABILITIES)}), read from each session.")
+ return 0
+ path, tools = untool.refresh_catalog(state_root(), host=host)
+ print(f"{len(tools)} tools recorded in {path}:\n " + ", ".join(tools))
+ return 0
+ if action == "measure":
+ tools = args.tools or unt.get("proposal", []) or untool.applied(ctx.store)
+ if not tools:
+ print("no proposal to measure")
+ return 1
+ res = untool.measure(tools, host=host)
+ print(json.dumps(res, indent=2))
+ return 0
+ return 2
+
+
+# ---- config / doctor / fold ----------------------------------------------------------------
+def _config_view(cfg):
+ """The settings as a person reads them: each mechanism on or off, its knobs, and the guard's
+ thresholds with the numbers a `spiral.<name>` setting would replace."""
+ d = cfg.as_dict()
+ pinned = d.pop("spiral_thresholds")
+ return {
+ "enabled": d.pop("enabled"),
+ "facts (inject)": {"state": "off" if cfg.inject == "off" else "on",
+ "inject": d.pop("inject"),
+ "inject_max_facts": d.pop("inject_max_facts"),
+ "inject_max_tokens": d.pop("inject_max_tokens"),
+ "propose_every": d.pop("propose_every")},
+ "unused tools (untool)": {"state": "off" if cfg.untool == "off" else "on",
+ "untool": d.pop("untool"),
+ "untool_evidence": d.pop("untool_evidence")},
+ "stuck-turn guard (spiral)": {
+ "state": "off" if cfg.spiral == "off" else "on",
+ "spiral": d.pop("spiral"), "spiral_aggr": d.pop("spiral_aggr"),
+ "spiral_nudges": d.pop("spiral_nudges"),
+ "thresholds": guard.base_thresholds(cfg.spiral_aggr, pinned),
+ "pinned": sorted(pinned)},
+ "other": d,
+ "warnings": cfg.warnings,
+ }
+
+
+def cmd_config(args):
+ ctx = _ctx(args)
+ if args.action == "show":
+ print(json.dumps(_config_view(ctx.config), indent=2))
+ return 0
+ path = (os.path.join(ctx.root, "config.json") if args.globally
+ else os.path.join(os.path.dirname(ctx.store.base), "config.json"))
+ if args.action == "unset":
+ try:
+ cur = unset_config(path, [k.strip() for k in args.pairs])
+ except ValueError as e:
+ print(f"rejected: {e}")
+ return 2
+ print(f"{path}: {json.dumps(cur, sort_keys=True)}")
+ return 0
+ updates = {}
+ for kv in args.pairs:
+ if "=" not in kv:
+ print(f"expected KEY=VALUE, got {kv!r}")
+ return 2
+ k, v = kv.split("=", 1)
+ updates[k.strip()] = v.strip()
+ if not updates:
+ print("nothing to set: give KEY=VALUE (see `te config show` for the keys)")
+ return 2
+ try:
+ cur = write_config(path, updates)
+ except ValueError as e:
+ print(f"rejected: {e}")
+ return 2
+ print(f"{path}: {json.dumps(cur, sort_keys=True)}")
+ print("applies from the next hook call; a new fact-size cap applies once the next session "
+ "has been folded")
+ return 0
+
+
+def cmd_fold(args):
+ ctx = _ctx(args)
+ sids = args.session or fold.pending_sessions(ctx)
+ for sid in sids:
+ rec = fold.fold_session(ctx, sid)
+ if rec is not None and args.verbose:
+ print(json.dumps(rec, default=str))
+ return 0
+
+
+def cmd_judge(args):
+ """The detached half of the paid stuck judge (see te_live/judge.py); not for people."""
+ from . import judge
+ judge.run_job(_ctx(args), args.session, args.job)
+ return 0
+
+
+def cmd_probe(args):
+ if args.host != "copilot":
+ print("the probe runs on Copilot only so far: `te --host copilot probe`")
+ return 2
+ from . import probe
+ return probe.run(sessions=args.sessions, model=args.model, keep=args.keep)
+
+
+def cmd_doctor(args):
+ ok = sys.version_info >= (3, 10)
+ print(f"python {sys.version.split()[0]} {'ok' if ok else '(needs >= 3.10)'}")
+ root = state_root()
+ try:
+ os.makedirs(root, exist_ok=True)
+ probe = os.path.join(root, ".write-test")
+ open(probe, "w").close()
+ os.remove(probe)
+ print(f"state root {root} (writable)")
+ except OSError as e:
+ print(f"state root {root} NOT writable: {e}")
+ ok = False
+ for host in ("claude", "copilot"):
+ cat = os.path.join(root, "hosts", host, "catalog.json")
+ print(f"{host:11s} catalog {'present' if os.path.exists(cat) else 'not probed'}")
+ from urllib.parse import urlparse
+ from .sidecall import copilot_model
+ base = os.environ.get("COPILOT_PROVIDER_BASE_URL")
+ route = (f"BYOK {os.environ.get('COPILOT_PROVIDER_TYPE') or 'openai'} at "
+ f"{urlparse(base).netloc or base}" if base else "Copilot subscription")
+ print(f"copilot {route}; side calls and `probe` use model "
+ f"{copilot_model() or '(Copilot default)'}")
+ ctrust = untool.copilot_folder_trusted(args.cwd)
+ if ctrust is not None:
+ print(f"copilot this folder is {'trusted' if ctrust else 'NOT trusted'} (an untrusted "
+ f"folder's .github/copilot settings are ignored)")
+ trusted = untool.codex_project_trusted(args.cwd)
+ if trusted is not None:
+ print(f"codex this project is {'trusted' if trusted else 'NOT trusted'} "
+ f"(an untrusted project's .codex/config.toml is ignored)")
+ log = os.path.join(root, "errors.log")
+ if os.path.exists(log):
+ tail = open(log, encoding="utf-8", errors="replace").read().splitlines()[-6:]
+ print("errors.log (tail):\n " + "\n ".join(tail))
+ else:
+ print("errors.log none")
+ return 0 if ok else 1
+
+
+# ---- replay --------------------------------------------------------------------------------
+def _claude_projects_dir():
+ base = os.environ.get("CLAUDE_CONFIG_DIR") or os.path.join(os.path.expanduser("~"), ".claude")
+ return os.path.join(base, "projects")
+
+
+def _within(path, root):
+ path, root = os.path.realpath(path or ""), os.path.realpath(root)
+ return path == root or path.startswith(root + os.sep)
+
+
+def _first_cwd(path, limit=200):
+ for i, line in enumerate(open(path, encoding="utf-8", errors="replace")):
+ if i > limit:
+ break
+ try:
+ d = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(d.get("cwd"), str):
+ return d["cwd"]
+ p = d.get("payload")
+ if isinstance(p, dict) and isinstance(p.get("cwd"), str):
+ return p["cwd"]
+ return None
+
+
+def find_history(ctx, limit=0):
+ """[(session_id, transcript)] of this repository's past sessions, oldest first."""
+ found = []
+ if ctx.host.name == "claude":
+ slug = re.sub(r"[^A-Za-z0-9]", "-", os.path.realpath(ctx.repo.root))
+ for d in glob.glob(os.path.join(_claude_projects_dir(), "*")):
+ if not os.path.basename(d).startswith(slug):
+ continue
+ for f in glob.glob(os.path.join(d, "*.jsonl")):
+ if _within(_first_cwd(f), ctx.repo.root):
+ found.append((os.path.splitext(os.path.basename(f))[0], f))
+ elif ctx.host.name == "pi":
+ from .transcripts import pi_session_header
+ agent = os.environ.get("PI_CODING_AGENT_DIR") or os.path.join(
+ os.path.expanduser("~"), ".pi", "agent")
+ for f in glob.glob(os.path.join(agent, "sessions", "*", "*.jsonl")):
+ head = pi_session_header(f)
+ if head.get("id") and _within(head.get("cwd"), ctx.repo.root):
+ found.append((head["id"], f))
+ elif ctx.host.name == "copilot":
+ from .transcripts import copilot_session_meta
+ for f in glob.glob(os.path.join(hosts.copilot_home(), "session-state", "*",
+ "events.jsonl")):
+ meta = copilot_session_meta(f)
+ if meta.get("session_id") and _within(meta.get("cwd"), ctx.repo.root):
+ found.append((meta["session_id"], f))
+ else:
+ home = os.environ.get("CODEX_HOME") or os.path.join(os.path.expanduser("~"), ".codex")
+ for f in glob.glob(os.path.join(home, "sessions", "**", "rollout-*.jsonl"),
+ recursive=True):
+ if _within(_first_cwd(f), ctx.repo.root):
+ found.append((os.path.basename(f)[len("rollout-"):-len(".jsonl")], f))
+ found.sort(key=lambda sf: os.path.getmtime(sf[1]))
+ return found[-limit:] if limit else found
+
+
+def _claude_turns(path):
+ """Prompt-turns of a Claude Code transcript: [(calls, effective_tokens)], where calls are
+ (name, args, result, exit). Usage is counted once per assistant message id, as the ledger
+ counts it (the transcript repeats a message's usage on every content block)."""
+ from tokeneconomizer.vendor import spiral
+ turns, cur, pend, usage = [], None, {}, {}
+
+ def close():
+ if cur is not None and (cur or usage):
+ eff = sum(accounting.effective(u)[0] or 0 for u in usage.values())
+ turns.append((cur, eff))
+
+ for line in open(path, encoding="utf-8", errors="replace"):
+ try:
+ d = json.loads(line)
+ except ValueError:
+ continue
+ if d.get("type") == "user" and not d.get("isMeta") and not d.get("isSidechain"):
+ c = (d.get("message") or {}).get("content")
+ kinds = ({b.get("type") for b in c if isinstance(b, dict)}
+ if isinstance(c, list) else set())
+ if (isinstance(c, str) and c.strip() and not c.startswith("<")) or (
+ "text" in kinds and "tool_result" not in kinds):
+ close()
+ cur, pend, usage = [], {}, {}
+ continue
+ if cur is None:
+ continue
+ msg = d.get("message") if isinstance(d.get("message"), dict) else {}
+ if d.get("type") == "assistant" and isinstance(msg.get("usage"), dict):
+ usage[msg.get("id") or len(usage)] = dict(msg["usage"], model=msg.get("model"))
+ for name, args, result in spiral.completed_calls(line, pend):
+ cur.append((name, args, result, None))
+ close()
+ return turns
+
+
+def _codex_turns(path):
+ from .transcripts import _rollout_item_to_exec
+ turns, order = {}, []
+ for line in open(path, encoding="utf-8", errors="replace"):
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ p = ev.get("payload") if isinstance(ev.get("payload"), dict) else {}
+ if ev.get("type") != "event_msg" or p.get("type") != "item_completed":
+ continue
+ item = _rollout_item_to_exec(p.get("item") or {}) or {}
+ tid = p.get("turn_id") or "?"
+ if tid not in turns:
+ turns[tid] = []
+ order.append(tid)
+ if item.get("type") == "command_execution":
+ turns[tid].append(("bash", {"command": item.get("command") or ""},
+ item.get("aggregated_output") or "", item.get("exit_code")))
+ elif item.get("type") == "file_change":
+ paths = [c.get("path") for c in item.get("changes") or [] if isinstance(c, dict)]
+ turns[tid].append(("apply_patch", {"path": paths[0] if paths else ""}, "", None))
+ return [(turns[t], None) for t in order if turns[t]]
+
+
+def _session_turns(host, path):
+ """Prompt-turns of one session log, whichever host wrote it."""
+ if host.name == "claude":
+ return _claude_turns(path)
+ if host.name == "copilot":
+ from .transcripts import copilot_turns
+ return [(calls, None) for calls in copilot_turns(path)]
+ if host.name == "pi":
+ from .transcripts import pi_session_to_claude
+ with tempfile.TemporaryDirectory(prefix="te-pi-") as d:
+ out = os.path.join(d, "trace.jsonl")
+ return _claude_turns(out) if pi_session_to_claude(path, out) is not None else []
+ return _codex_turns(path)
+
+
+def cmd_replay(args):
+ real = _ctx(args)
+ history = find_history(real, args.limit)
+ if not history:
+ print(f"no {real.host.name} session history found for {real.repo.root}")
+ return 1
+ scratch = None
+ ctx = real
+ if not args.commit:
+ scratch = tempfile.mkdtemp(prefix="te-replay-")
+ ctx = LiveContext(host=real.host, repo=real.repo, root=real.root,
+ store=Store(os.path.join(scratch, real.host.name)),
+ config=real.config, env=dict(real.env, TE_FOLD_SYNC="1"))
+ t0 = time.time()
+ totals = {"turns": 0, "calls": 0, "nudge_turns": 0, "stop_turns": 0, "post_fire_calls": 0}
+ turn_cost, turn_tokens = [], []
+ for i, (sid, path) in enumerate(history):
+ fold.fold_session(ctx, sid, transcript_path=path)
+ turns = _session_turns(real.host, path)
+ for calls, eff in turns:
+ if eff is not None:
+ turn_tokens.append(eff)
+ if not calls:
+ continue
+ g = guard.TurnGuard(turn_key=f"{sid}:{totals['turns']}")
+ ov = real.config.spiral_thresholds
+ tracker = guard.load_envelope(ctx.store, real.config.spiral_aggr, ov)
+ env_v, thr = tracker.envelope, guard.thresholds(tracker, ov)
+ fired_at = None
+ for j, (name, a, r, code) in enumerate(calls):
+ d = g.fold(name, a, r, code, envelope=env_v, thr=thr,
+ nudge_budget=real.config.spiral_nudges)
+ if d and d[0] == "nudge":
+ g.after_nudge()
+ totals["nudge_turns"] += 1 if fired_at is None else 0
+ fired_at = j if fired_at is None else fired_at
+ elif d and d[0] == "abort":
+ totals["stop_turns"] += 1
+ fired_at = j if fired_at is None else fired_at
+ if fired_at is not None:
+ totals["post_fire_calls"] += len(calls) - fired_at - 1
+ totals["turns"] += 1
+ totals["calls"] += len(calls)
+ turn_cost.append(len(calls))
+ if g.cum_turns:
+ guard.record_turn(ctx.store, g, level=real.config.spiral_aggr,
+ overrides=real.config.spiral_thresholds)
+ if args.verbose:
+ print(f" [{i + 1}/{len(history)}] {sid} {len(turns)} turns")
+ led = accounting.summarize(ctx.store.read_lines("ledger.jsonl"))
+ inj = ctx.store.read("injection.json") or {}
+ unt = ctx.store.read("untool.json") or {}
+ tracker = guard.load_envelope(ctx.store, real.config.spiral_aggr,
+ real.config.spiral_thresholds)
+ facts = {k: v for k, v in (inj.get("facts") or {}).items() if isinstance(v, dict)}
+ injected = [k for k, v in facts.items() if v.get("inject")]
+ forming = [k for k, v in facts.items() if v.get("state") in ("CANDIDATE", "ACTIVE")
+ and not v.get("inject")]
+ span = (time.strftime("%Y-%m-%d", time.localtime(os.path.getmtime(history[0][1]))),
+ time.strftime("%Y-%m-%d", time.localtime(os.path.getmtime(history[-1][1]))))
+ print(f"TokenEconomizer replay -- {real.repo.label} ({real.host.name}), "
+ f"{len(history)} sessions, {span[0]} -> {span[1]} ({time.time() - t0:.0f}s)")
+ tail = ""
+ if turn_tokens and sum(turn_tokens):
+ top = sorted(turn_tokens, reverse=True)[:max(1, len(turn_tokens) // 10)]
+ tail = (f"; the costliest 10% of prompt-turns carry "
+ f"{sum(top) / sum(turn_tokens):.0%} of it")
+ print(f" cost {_fmt_tokens(led['effective_tokens'])} effective tokens over "
+ f"{led['priced_sessions']} priced sessions{tail}")
+ print(f" injection {len(injected)} fact(s) would be injected now"
+ + (f": {', '.join(injected)}" if injected else "")
+ + (f"; {len(forming)} forming: {', '.join(forming[:5])}" if forming else ""))
+ if untool.supports(real.host.name):
+ print(f" untooling catalog {unt.get('catalog_source') or 'none'}; would propose "
+ f"{untool.unit(real.host.name, plural=True)}: "
+ f"{', '.join(unt.get('proposal') or []) or '-'}")
+ turns = max(totals["turns"], 1)
+ print(f" spiral {totals['turns']} prompt-turns (median "
+ f"{sorted(turn_cost)[len(turn_cost) // 2] if turn_cost else 0} tool calls); guard "
+ f"would nudge {totals['nudge_turns']} ({totals['nudge_turns'] / turns:.1%}) and stop "
+ f"{totals['stop_turns']}; {totals['post_fire_calls']} tool calls came after a fire "
+ f"({totals['post_fire_calls'] / max(totals['calls'], 1):.1%} of all calls)")
+ env_s = f"{tracker.envelope:.0f}" if tracker.envelope else "n/a"
+ print(f" learned envelope {env_s} tool calls from "
+ f"{len(tracker.resolved_turns)} ended-green turns (heuristic gate; no judge)")
+ if scratch:
+ print(f" (scratch state in {scratch}; `te replay --commit` keeps it as the starting "
+ f"state)")
+ return 0
+
+
+def build_parser():
+ ap = argparse.ArgumentParser(prog="te", description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--cwd", default=os.getcwd())
+ ap.add_argument("--host", default=hosts.default_host(), choices=sorted(hosts.HOSTS))
+ # The same two options are accepted after the subcommand too (`te fold --host codex ...`).
+ # SUPPRESS keeps a subcommand that omits them from overwriting the top-level values.
+ common = argparse.ArgumentParser(add_help=False)
+ common.add_argument("--cwd", default=argparse.SUPPRESS)
+ common.add_argument("--host", default=argparse.SUPPRESS, choices=sorted(hosts.HOSTS))
+ sub = ap.add_subparsers(dest="cmd", required=True)
+
+ def add(name):
+ return sub.add_parser(name, parents=[common])
+
+ p = add("status")
+ p.add_argument("--json", action="store_true")
+ p.set_defaults(fn=cmd_status)
+ add("facts").set_defaults(fn=cmd_facts)
+ p = add("untool")
+ p.add_argument("action", nargs="?", default="show",
+ choices=("show", "apply", "restore", "refresh-catalog", "measure"))
+ p.add_argument("tools", nargs="*")
+ p.set_defaults(fn=cmd_untool)
+ p = add("replay")
+ p.add_argument("--limit", type=int, default=0)
+ p.add_argument("--commit", action="store_true")
+ p.add_argument("--verbose", action="store_true")
+ p.set_defaults(fn=cmd_replay)
+ p = add("config")
+ p.add_argument("action", nargs="?", default="show", choices=("show", "set", "unset"))
+ p.add_argument("pairs", nargs="*")
+ p.add_argument("--global", dest="globally", action="store_true")
+ p.set_defaults(fn=cmd_config)
+ p = add("fold")
+ p.add_argument("--session", action="append")
+ p.add_argument("--verbose", action="store_true")
+ p.set_defaults(fn=cmd_fold)
+ p = add("judge")
+ p.add_argument("--session", required=True)
+ p.add_argument("--job", required=True)
+ p.set_defaults(fn=cmd_judge)
+ add("doctor").set_defaults(fn=cmd_doctor)
+ p = add("probe")
+ p.add_argument("--sessions", type=int, default=6)
+ p.add_argument("--model", default=None)
+ p.add_argument("--keep", action="store_true")
+ p.set_defaults(fn=cmd_probe)
+ return ap
+
+
+def main(argv=None):
+ args = build_parser().parse_args(argv)
+ try:
+ return args.fn(args) or 0
+ finally:
+ from .paths import secure_root
+ secure_root()
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/te_live/config.py b/te_live/config.py
new file mode 100644
index 0000000..54be66f
--- /dev/null
+++ b/te_live/config.py
@@ -0,0 +1,261 @@
+"""Live configuration: which mechanisms act in real sessions, and how far each may go.
+
+The defaults are the conservative end of every switch, chosen from what the local replay of real
+sessions showed (docs/PLUGIN.md) rather than from the benchmark arms:
+
+ * inject = "envfacts" -- the LLM-free layer only. The paid proposer (`envfacts+distiller`,
+ `distiller`) is opt-in, and refused unless `paid` is also true;
+ * untool = "propose" -- evidence is gathered and a deny list is proposed, but nothing is
+ written to the user's settings until they approve it. "auto"
+ applies proposals as they form;
+ * spiral = "nudge" -- at most one mid-turn redirect per prompt-turn. "guard" also stops a
+ turn that keeps spinning after the nudge, and only in headless mode;
+ interactive sessions get a notice instead, because there a person is
+ already the abort;
+ * paid = False -- no mechanism makes an LLM call of its own. Every paid call that is
+ allowed is charged to the session ledger, as the experiments charge
+ optimizer_tokens to an arm;
+ * shadow = False -- with True, every decision is computed and logged but none reaches
+ the agent. The way to measure a mechanism before trusting it.
+
+Each of the three mechanisms also takes a plain `on` / `off` (`on` = the value above), and each
+has hyperparameters a person can set by hand:
+
+ * inject_max_facts, inject_max_tokens -- size caps on the injected fact block;
+ * propose_every -- sessions between paid proposer runs;
+ * untool_evidence -- unused sessions before a tool is proposed;
+ * spiral_aggr -- the guard's threshold preset;
+ * spiral_nudges -- redirects per prompt-turn before it stops/notifies;
+ * spiral.<name> -- one guard threshold, pinned over the preset and over
+ what the repository's history learned (SPIRAL_KEYS).
+
+Sources, lowest precedence first: built-in defaults, `<root>/config.json`, the repository's
+`config.json`, then environment variables (`TE_INJECT`, `TE_UNTOOL`, `TE_SPIRAL`, `TE_MODE`,
+`TE_PAID`, `TE_SHADOW`, `TE_DISABLE`, and one per hyperparameter; `TE_SPIRAL_THRESHOLDS` takes
+JSON). An invalid value is ignored with a warning recorded in the result rather than raised: a
+typo in a config file must not break the agent's hooks.
+"""
+import json
+import os
+from dataclasses import asdict, dataclass, field, fields
+
+INJECT = ("off", "envfacts", "envfacts+distiller", "distiller")
+UNTOOL = ("off", "propose", "auto")
+SPIRAL = ("off", "nudge", "guard")
+MODES = ("auto", "interactive", "headless")
+AGGR = ("conservative", "default", "aggressive")
+
+# What `on` turns each mechanism to, and the words read as on / off.
+ON_VALUE = {"inject": "envfacts", "untool": "propose", "spiral": "nudge"}
+_ON = ("on", "true", "yes", "1", "enable", "enabled")
+_OFF = ("off", "false", "no", "0", "disable", "disabled")
+
+# The spiral guard's thresholds a person may pin (`spiral.<name>=<value>`), with their type and
+# range. They are the keys of `tokeneconomizer.vendor.spiral.DEFAULT_THRESHOLDS`; a turn's
+# "turns" are its tool calls here.
+SPIRAL_KEYS = {
+ "min_turns": (int, 1, None, "tool calls in a turn before it can count as stuck"),
+ "edit_churn": (int, 1, None, "edits to one file that count as churn"),
+ "distinct_edit_files": (int, 1, None, "files edited that count as scattered"),
+ "error_outputs": (int, 1, None, "failing tool results that count as sustained errors"),
+ "stagnation": (float, 0.0, 1.0, "share of failures that repeat an earlier one"),
+ "envelope_pct": (float, 1.0, 100.0, "percentile of this repository's ended-green turns"),
+ "envelope_margin": (float, 0.1, None, "multiplier on that percentile: the envelope"),
+ "envelope_min_samples": (int, 1, None, "ended-green turns needed before the envelope is used"),
+ "backstop_mult": (float, 0.1, None, "stuck past this multiple of the envelope: redirect, "
+ "then stop"),
+ "judge_confidence": (float, 0.0, 1.0, "paid judge confidence needed to act (paid=true)"),
+ "gate_min_turns": (int, 1, None, "tool calls before the paid judge is consulted"),
+ "gate_error_outputs": (int, 1, None, "failing results that wake the paid judge early"),
+}
+
+
+@dataclass
+class LiveConfig:
+ enabled: bool = True
+ mode: str = "auto"
+ inject: str = "envfacts"
+ inject_max_facts: int = 4
+ inject_max_tokens: int = 150
+ propose_every: int = 5
+ untool: str = "propose"
+ untool_evidence: int = 5
+ spiral: str = "nudge"
+ spiral_aggr: str = "default"
+ spiral_nudges: int = 1
+ spiral_thresholds: dict = field(default_factory=dict)
+ paid: bool = False
+ paid_model: str = ""
+ shadow: bool = False
+ vocabulary: str = "auto"
+ warnings: list = field(default_factory=list)
+
+ _CHOICES = {"mode": MODES, "inject": INJECT, "untool": UNTOOL, "spiral": SPIRAL,
+ "spiral_aggr": AGGR, "vocabulary": ("auto", "python-git", "node-web")}
+
+ def as_dict(self):
+ d = asdict(self)
+ d.pop("warnings", None)
+ return d
+
+ def set(self, key, value, source="?"):
+ """Apply one setting with type coercion and validation; False if rejected."""
+ if key.startswith("spiral."):
+ return self._set_threshold(key.split(".", 1)[1], value, source)
+ names = {f.name: f for f in fields(self) if f.name != "warnings"}
+ if key not in names:
+ self.warnings.append(f"{source}: unknown key {key!r}")
+ return False
+ if key == "spiral_thresholds":
+ if not isinstance(value, dict):
+ self.warnings.append(f"{source}: spiral_thresholds must be an object")
+ return False
+ return all([self._set_threshold(k, v, source) for k, v in value.items()])
+ if key in ON_VALUE and isinstance(value, (str, bool, int)):
+ word = str(value).strip().lower()
+ if word in _ON:
+ value = ON_VALUE[key]
+ elif word in _OFF:
+ value = "off"
+ cur = getattr(self, key)
+ try:
+ if isinstance(cur, bool):
+ v = value if isinstance(value, bool) else str(value).strip().lower() in (
+ "1", "true", "yes", "on")
+ elif isinstance(cur, int):
+ v = int(value)
+ if v < 1:
+ raise ValueError("must be >= 1")
+ else:
+ v = str(value).strip()
+ except (TypeError, ValueError) as e:
+ self.warnings.append(f"{source}: {key}={value!r} rejected ({e})")
+ return False
+ choices = self._CHOICES.get(key)
+ if choices and v not in choices:
+ self.warnings.append(f"{source}: {key}={v!r} not one of {choices}")
+ return False
+ setattr(self, key, v)
+ return True
+
+ def _set_threshold(self, name, value, source):
+ spec = SPIRAL_KEYS.get(name)
+ if spec is None:
+ self.warnings.append(f"{source}: unknown spiral threshold {name!r} "
+ f"(one of {', '.join(SPIRAL_KEYS)})")
+ return False
+ kind, lo, hi, _ = spec
+ try:
+ if isinstance(value, bool):
+ raise ValueError("not a number")
+ v = kind(float(value)) if kind is int and float(value).is_integer() else kind(value)
+ except (TypeError, ValueError) as e:
+ self.warnings.append(f"{source}: spiral.{name}={value!r} rejected ({e})")
+ return False
+ if (lo is not None and v < lo) or (hi is not None and v > hi):
+ rng = f">= {lo}" if hi is None else f"between {lo} and {hi}"
+ self.warnings.append(f"{source}: spiral.{name}={v!r} rejected (must be {rng})")
+ return False
+ self.spiral_thresholds[name] = v
+ return True
+
+ @property
+ def inject_effective(self):
+ """The injection profile that will actually run: a paid profile without `paid` falls
+ back to the free layer, and says so in `warnings` when the config is loaded."""
+ if self.inject in ("envfacts+distiller", "distiller") and not self.paid:
+ return "envfacts"
+ return self.inject
+
+
+_ENV = {"TE_MODE": "mode", "TE_INJECT": "inject", "TE_UNTOOL": "untool", "TE_SPIRAL": "spiral",
+ "TE_PAID": "paid", "TE_SHADOW": "shadow", "TE_PROPOSE_EVERY": "propose_every",
+ "TE_UNTOOL_EVIDENCE": "untool_evidence", "TE_SPIRAL_AGGR": "spiral_aggr",
+ "TE_PAID_MODEL": "paid_model", "TE_VOCABULARY": "vocabulary",
+ "TE_INJECT_MAX_FACTS": "inject_max_facts", "TE_INJECT_MAX_TOKENS": "inject_max_tokens",
+ "TE_SPIRAL_NUDGES": "spiral_nudges"}
+
+
+def _read(path):
+ try:
+ with open(path, encoding="utf-8") as fh:
+ obj = json.load(fh)
+ return obj if isinstance(obj, dict) else {}
+ except (OSError, ValueError):
+ return {}
+
+
+def load_config(root, repo_dir=None, env=None):
+ """Merge defaults, the global file, the repository file and the environment."""
+ env = os.environ if env is None else env
+ cfg = LiveConfig()
+ for label, path in (("global", os.path.join(root, "config.json")),
+ ("repo", os.path.join(repo_dir, "config.json") if repo_dir else None)):
+ if path:
+ for k, v in _read(path).items():
+ cfg.set(k, v, source=f"{label} {path}")
+ for var, key in _ENV.items():
+ if env.get(var) not in (None, ""):
+ cfg.set(key, env[var], source=var)
+ if env.get("TE_SPIRAL_THRESHOLDS"):
+ try:
+ cfg.set("spiral_thresholds", json.loads(env["TE_SPIRAL_THRESHOLDS"]),
+ source="TE_SPIRAL_THRESHOLDS")
+ except ValueError:
+ cfg.warnings.append("TE_SPIRAL_THRESHOLDS: not valid JSON")
+ if str(env.get("TE_DISABLE", "")).strip().lower() in ("1", "true", "yes", "on"):
+ cfg.enabled = False
+ if cfg.inject != cfg.inject_effective:
+ cfg.warnings.append(f"inject={cfg.inject!r} makes paid calls but paid=false; "
+ f"running {cfg.inject_effective!r}")
+ return cfg
+
+
+def write_config(path, updates):
+ """Merge `updates` into the JSON config at `path` (validating each key first).
+
+ A `spiral.<name>` key lands in the file's `spiral_thresholds` object."""
+ probe = LiveConfig()
+ bad = [k for k, v in updates.items() if not probe.set(k, v, source="te config")]
+ if bad:
+ raise ValueError("; ".join(probe.warnings))
+ current = _read(path)
+ for k in updates:
+ if k.startswith("spiral."):
+ name = k.split(".", 1)[1]
+ current.setdefault("spiral_thresholds", {})[name] = probe.spiral_thresholds[name]
+ elif k == "spiral_thresholds":
+ current.setdefault("spiral_thresholds", {}).update(probe.spiral_thresholds)
+ else:
+ current[k] = getattr(probe, k)
+ _write(path, current)
+ return current
+
+
+def unset_config(path, keys):
+ """Remove keys from the JSON config at `path`, so the next source down applies again.
+
+ `spiral.<name>` removes one threshold; `spiral_thresholds` removes them all."""
+ known = {f.name for f in fields(LiveConfig) if f.name != "warnings"}
+ bad = [k for k in keys if k not in known and not (
+ k.startswith("spiral.") and k.split(".", 1)[1] in SPIRAL_KEYS)]
+ if bad:
+ raise ValueError(f"unknown key(s): {', '.join(bad)}")
+ current = _read(path)
+ for k in keys:
+ if k.startswith("spiral."):
+ (current.get("spiral_thresholds") or {}).pop(k.split(".", 1)[1], None)
+ if not current.get("spiral_thresholds"):
+ current.pop("spiral_thresholds", None)
+ else:
+ current.pop(k, None)
+ _write(path, current)
+ return current
+
+
+def _write(path, obj):
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ tmp = path + ".tmp"
+ with open(tmp, "w", encoding="utf-8") as fh:
+ json.dump(obj, fh, indent=2, sort_keys=True)
+ os.replace(tmp, path)
diff --git a/te_live/context.py b/te_live/context.py
new file mode 100644
index 0000000..0bb68fb
--- /dev/null
+++ b/te_live/context.py
@@ -0,0 +1,40 @@
+"""`LiveContext`: everything one hook call or CLI command needs about where it is running."""
+import os
+from dataclasses import dataclass
+
+from .config import LiveConfig, load_config
+from .paths import RepoId, repo_identity, repo_state_dir, state_root
+from .store import Store
+
+
+@dataclass
+class LiveContext:
+ host: object # hosts.Host
+ repo: RepoId
+ root: str # the state root
+ store: Store # this host's state for this repository
+ config: LiveConfig
+ env: dict
+
+ @property
+ def mech_dir(self):
+ """Where the mechanism classes keep their own state (their `results_root`)."""
+ return self.store.ensure("mechanisms")
+
+ @property
+ def headless(self):
+ if self.config.mode == "headless":
+ return True
+ if self.config.mode == "interactive":
+ return False
+ return self.host.headless(self.env)
+
+ @classmethod
+ def build(cls, host, cwd, env=None):
+ env = dict(os.environ if env is None else env)
+ repo = repo_identity(cwd, env)
+ root = state_root(env)
+ base = repo_state_dir(repo, host.name, env)
+ store = Store(base)
+ config = load_config(root, repo_dir=os.path.dirname(base), env=env)
+ return cls(host=host, repo=repo, root=root, store=store, config=config, env=env)
diff --git a/te_live/controller.py b/te_live/controller.py
new file mode 100644
index 0000000..d04b57c
--- /dev/null
+++ b/te_live/controller.py
@@ -0,0 +1,239 @@
+"""`LiveController`: what the plugin does at each hook.
+
+ SessionStart inject the repository's learned manual (cached by the last fold), and
+ fold any earlier session that ended without its SessionEnd being seen
+ UserPromptSubmit open a prompt-turn for the spiral guard
+ PostToolUse fold the call into the turn's guard; nudge, or stop, when it says so
+ Stop close the turn; an ended-green turn feeds the repository's envelope
+ SessionEnd hand the session to a detached `te fold`
+
+Every handler returns quickly and fails open. A hook that raises would break the user's session
+for a cost optimization, so `hook.main` turns any exception into an empty output, and nothing
+here waits on the network: injection reads a cached render, and the only paid call a hook can
+cause -- the sampled stuck judge -- runs only when `paid` is on and the free gate is open, and
+then in a detached job (`te_live.judge`), never in the hook itself.
+
+Every visible action is announced in one `systemMessage` line, so a person can see why the agent
+changed course; `shadow` computes and logs every decision without acting on any of them.
+"""
+import hashlib
+import os
+import time
+
+from . import fold, guard, untool
+from .hosts import merge_outputs
+from .paths import ensure_launcher
+
+TURN_HISTORY = 200
+
+
+class LiveController:
+ def __init__(self, ctx, payload):
+ self.ctx = ctx
+ self.p = payload or {}
+ self.host = ctx.host
+ self.store = ctx.store
+ self.cfg = ctx.config
+ self.sid = self.host.session_id(self.p)
+
+ # ---- plumbing -----------------------------------------------------------------------
+ def _sess_name(self):
+ return os.path.join("sessions", f"{self.sid}.json")
+
+ def _session(self):
+ s = self.store.read(self._sess_name())
+ if not isinstance(s, dict):
+ s = {"session_id": self.sid, "host": self.host.name, "started": time.time(),
+ "turns": [], "nudges": 0, "aborts": 0, "judge_tokens": 0.0,
+ "writer_tokens": 0.0}
+ return s
+
+ def _save(self, s):
+ tp = self.host.transcript(self.p)
+ if tp:
+ s["transcript"] = tp
+ s["updated"] = time.time()
+ self.store.write(self._sess_name(), s)
+
+ def _cmd(self, name):
+ """How a person runs one of the plugin's commands on this host."""
+ if self.host.name in ("claude", "copilot"): # Copilot's TUI runs plugin commands too
+ return f"/tokeneconomizer:{name}"
+ if self.host.name == "pi":
+ return f"/te {name}" # registered by the Pi extension
+ return f"te --host {self.host.name} {name}"
+
+ def _log(self, kind, **fields):
+ self.store.append("events.jsonl", dict(
+ fields, kind=kind, at=time.time(), session_id=self.sid, shadow=self.cfg.shadow))
+
+ def handle(self):
+ # A payload without a session id is not a hook call any host makes; recording it would
+ # leave an "unknown" session behind.
+ if not self.cfg.enabled or not self.host.session_id(self.p):
+ return {}
+ event = self.p.get("hook_event_name") or ""
+ # PostToolUseFailure: Copilot reports a failed tool there, not in PostToolUse.
+ handler = {"SessionStart": self.session_start, "UserPromptSubmit": self.user_prompt,
+ "PostToolUse": self.post_tool, "PostToolUseFailure": self.post_tool,
+ "Stop": self.stop, "SessionEnd": self.session_end}.get(event)
+ return (handler() or {}) if handler else {}
+
+ # ---- SessionStart -----------------------------------------------------------------
+ def session_start(self):
+ source = self.p.get("source") or "startup"
+ try:
+ ensure_launcher(self.ctx.env)
+ except OSError:
+ pass
+ s = self._session()
+ s["source"] = source
+ s["cwd"] = self.host.cwd(self.p, self.ctx.env)
+ outputs = []
+ seen = self.store.read("notices.json") or {}
+
+ if self.cfg.inject != "off":
+ text = (self.store.read("injection.json") or {}).get("rendered") or ""
+ # A resumed or forked conversation already carries what the first start injected;
+ # a cleared or compacted one has lost it. ("new" is Copilot's word for "startup".)
+ if text and source in ("startup", "new", "clear", "compact"):
+ n_facts = sum(1 for ln in text.splitlines() if ln.startswith("- "))
+ self._log("inject", chars=len(text), facts=n_facts, source=source)
+ if not self.cfg.shadow:
+ outputs.append(self.host.context("SessionStart", text))
+ s["injected_chars"] = len(text)
+ digest = hashlib.sha1(text.encode("utf-8")).hexdigest()
+ if seen.get("injection") != digest:
+ outputs.append(self.host.notice(
+ f"TokenEconomizer: added {n_facts} learned fact(s) about "
+ f"{self.ctx.repo.label} to this session's context "
+ f"(review with `{self._cmd('facts')}`)."))
+ seen["injection"] = digest
+
+ if (self.cfg.untool == "propose" and untool.supports(self.host.name)
+ and not self.cfg.shadow):
+ prop = (self.store.read("untool.json") or {}).get("proposal") or []
+ key = ",".join(prop)
+ if prop and seen.get("untool") != key:
+ outputs.append(self.host.notice(
+ f"TokenEconomizer: {len(prop)} {untool.unit(self.host.name)}(s) went unused "
+ f"in this repository's recent sessions ({', '.join(prop[:4])}"
+ f"{'...' if len(prop) > 4 else ''}). Review with "
+ f"`{self._cmd('untool')}`; nothing changes until you apply it."))
+ seen["untool"] = key
+
+ if outputs:
+ self.store.write("notices.json", seen)
+ self._save(s)
+ fold.spawn_fold(self.ctx, fold.pending_sessions(self.ctx, exclude=(self.sid,)))
+ return merge_outputs(*outputs)
+
+ # ---- prompt-turns -------------------------------------------------------------------
+ def _close_turn(self, s, stopped_normally):
+ data = s.get("turn")
+ if not data:
+ return None
+ g = guard.TurnGuard(data=data)
+ summary = g.summary(stopped_normally)
+ if g.cum_turns and stopped_normally:
+ guard.record_turn(self.store, g, level=self.cfg.spiral_aggr,
+ overrides=self.cfg.spiral_thresholds)
+ s["turns"] = (s.get("turns") or [])[-(TURN_HISTORY - 1):] + [summary]
+ s["judge_tokens"] = float(s.get("judge_tokens", 0) or 0) + g.judge_tokens
+ s["writer_tokens"] = float(s.get("writer_tokens", 0) or 0) + g.writer_tokens
+ s["turn"] = None
+ return summary
+
+ def user_prompt(self):
+ if self.cfg.spiral == "off":
+ return {}
+ s = self._session()
+ # A turn still open here never reached Stop -- the person interrupted it. Its outcome
+ # is unknown, so it is recorded but does not feed the envelope.
+ self._close_turn(s, stopped_normally=False)
+ s["turn"] = guard.TurnGuard(turn_key=self.host.turn_key(self.p)).to_dict()
+ self._save(s)
+ return {}
+
+ def stop(self):
+ if self.cfg.spiral == "off" or self.p.get("stop_hook_active"):
+ return {}
+ s = self._session()
+ summary = self._close_turn(s, stopped_normally=True)
+ if summary:
+ self._log("turn", **summary)
+ self._save(s)
+ return {}
+
+ # ---- PostToolUse ---------------------------------------------------------------------
+ def _judge(self, thr):
+ """The paid stuck judge, run in the background (`te_live.judge`), or None.
+
+ It used to run here, inside the hook, with the nudge writer after it: ~22 s with the
+ agent waiting, per nudge, in the user-simulation pilot. Now the hook only starts a job
+ and a later call picks up its verdict, with the redirect already written."""
+ if not self.cfg.paid:
+ return None
+ from .judge import AsyncJudge
+ return AsyncJudge(self.ctx, self.sid, thr.get("judge_confidence"))
+
+ def post_tool(self):
+ if self.cfg.spiral == "off" or self.host.is_subagent(self.p):
+ return {}
+ s = self._session()
+ key = self.host.turn_key(self.p)
+ data = s.get("turn")
+ if data and key and data.get("turn_key") and data["turn_key"] != key:
+ self._close_turn(s, stopped_normally=False) # a turn we never saw end
+ data = None
+ g = guard.TurnGuard(data=data) if data else guard.TurnGuard(turn_key=key)
+ name, args, text, code = self.host.tool_call(self.p)
+ tracker = guard.load_envelope(self.store, level=self.cfg.spiral_aggr,
+ overrides=self.cfg.spiral_thresholds)
+ envelope = tracker.envelope
+ thr = guard.thresholds(tracker, self.cfg.spiral_thresholds)
+ decision = g.fold(name, args, text, code, envelope=envelope, thr=thr,
+ judge=self._judge(thr), nudge_budget=self.cfg.spiral_nudges)
+ out = {}
+ if decision:
+ action, reason = decision
+ short = reason.split(":")[0]
+ if action == "nudge":
+ body = g.nudge_body(envelope)
+ g.after_nudge()
+ s["nudges"] = int(s.get("nudges", 0)) + 1
+ self._log("nudge", reason=reason, tool_calls=g.cum_turns, envelope=envelope)
+ if not self.cfg.shadow:
+ out = merge_outputs(
+ self.host.context("PostToolUse", body),
+ self.host.notice(f"TokenEconomizer nudged this turn ({short}): the "
+ f"same failure kept recurring."))
+ elif action == "abort" and not g.abort_delivered:
+ g.abort_delivered = True
+ s["aborts"] = int(s.get("aborts", 0)) + 1
+ can_stop = (self.cfg.spiral == "guard" and self.ctx.headless
+ and self.host.can_stop_mid_turn)
+ self._log("abort", reason=reason, tool_calls=g.cum_turns, envelope=envelope,
+ delivered="stop" if can_stop else "notice")
+ if not self.cfg.shadow:
+ if can_stop:
+ out = self.host.stop(
+ f"TokenEconomizer stopped this turn: it kept repeating the same "
+ f"failing steps after a redirect ({short}).")
+ else:
+ out = self.host.notice(
+ f"TokenEconomizer: this turn still looks stuck after a redirect "
+ f"({short}, {g.cum_turns} tool calls). Consider stepping in.")
+ s["turn"] = g.to_dict()
+ self._save(s)
+ return out
+
+ # ---- SessionEnd ---------------------------------------------------------------------
+ def session_end(self):
+ s = self._session()
+ self._close_turn(s, stopped_normally=False)
+ s["ended"] = time.time()
+ s["end_reason"] = self.p.get("reason")
+ self._save(s)
+ fold.spawn_fold(self.ctx, [self.sid])
+ return {}
diff --git a/te_live/fold.py b/te_live/fold.py
new file mode 100644
index 0000000..511ff78
--- /dev/null
+++ b/te_live/fold.py
@@ -0,0 +1,329 @@
+"""Folding a finished session into what the plugin knows about its repository.
+
+This is the live counterpart of the experiments' `Economizer.after_task`, and it runs the same
+classes: `PrefixStage` learns and renders the injected manual (envfacts, plus the paid proposer
+when allowed), and `ToolingStage` accrues evidence untooling's used-tool and unused-streak state.
+A session is one "task" of the repository's stream for both, the unit they were built around.
+
+Two adaptations, both forced by the setting:
+
+ * The stage objects lived for a whole stream in one process; here each fold is a fresh
+ process. `PrefixStage` already persists its distillers; `ToolingStage`'s streaks were
+ in-memory only, so they are saved to `untool.json` and restored before each fold.
+ * Folding runs OFF the agent's critical path. SessionEnd hooks are short (Codex caps them at
+ three seconds), and parsing a long transcript -- or a paid proposer tick -- takes longer, so
+ the hook spawns `te fold` detached and returns at once. A session whose end was never seen
+ (a killed terminal) is folded by the next SessionStart once its transcript has gone quiet.
+
+Each session is folded once, at its first end, keyed by session id; a resumed session is not
+re-folded, because counting the same trajectory twice would hand the fact lifecycle confirmations
+it did not earn.
+"""
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+
+from . import accounting, sidecall, transcripts, untool
+from . import adapter as live_adapter
+from .adapter import LiveAdapter, load_catalog
+
+STALE_S = 600 # an unfolded session whose transcript is quiet this long is over
+# A fold never runs in a hook -- it is a detached process, or `te fold` -- so it can wait for
+# another fold to finish, up to one paid side call, and must never proceed without the lock:
+# two folds of one session would both find it unfolded and count it twice, in every mechanism's
+# state and in the ledger. That happened whenever a session started while the previous one's
+# paid fold was still in its side call (the next SessionStart re-folds whatever is not yet
+# marked folded). A fold that still cannot take the lock skips: the holder finishes the
+# session, or, if it died, the session stays pending and the next SessionStart folds it.
+FOLD_LOCK_WAIT_S = sidecall.TIMEOUT_S + 60
+
+
+def _run_dir(workdir, trace_path):
+ run = os.path.join(workdir, "run")
+ os.makedirs(run, exist_ok=True)
+ link = os.path.join(run, "transcript.jsonl")
+ try:
+ os.symlink(trace_path, link)
+ except OSError:
+ shutil.copyfile(trace_path, link)
+ return run
+
+
+def _fold_injection(ctx, adapter, run_dir, session_id):
+ profile = ctx.config.inject_effective
+ if profile == "off":
+ return None
+ from tokeneconomizer.config import InjectionConfig
+ from tokeneconomizer.mechanisms.injection import PrefixStage
+ icfg = InjectionConfig(enabled=True, propose_every=ctx.config.propose_every, profile=profile,
+ max_facts=ctx.config.inject_max_facts,
+ max_tokens=ctx.config.inject_max_tokens)
+ runner = sidecall.runner_for(ctx.host.name, ctx.config) if icfg.paid else None
+ stage = PrefixStage(icfg, ctx.repo.label, ctx.mech_dir, False, runner=runner,
+ optimizer_model="agent", adapter=adapter, agent=ctx.host.name)
+ srow = {"resolved": None}
+ opt_tokens, reason = stage.after_task({"instance_id": session_id}, {}, run_dir, srow)
+ rendered = stage.render() or ""
+ prior = ctx.store.read("injection.json") or {}
+ ctx.store.write("injection.json", {
+ "profile": profile, "reason": reason, "rendered": rendered,
+ "facts": srow.get("manual_facts"), "updated": time.time(),
+ "sessions": int(prior.get("sessions", 0)) + 1})
+ return {"optimizer_tokens": float(opt_tokens or 0), "rendered_chars": len(rendered)}
+
+
+def _fold_untool(ctx, adapter, run_dir, session_id, transcript=None, workdir=None):
+ if not untool.supports(ctx.host.name) or ctx.config.untool == "off":
+ return None
+ if ctx.host.name == "codex":
+ return _fold_untool_codex(ctx, adapter, session_id, transcript, workdir)
+ host = ctx.host.name
+ from tokeneconomizer.config import ToolingConfig
+ from tokeneconomizer.mechanisms.tooling import ToolingStage
+ st = ctx.store.read("untool.json") or {}
+ stage = ToolingStage(ToolingConfig(evidence=ctx.config.untool_evidence), host, "agent",
+ adapter=adapter, repo=ctx.repo.label, results_root=ctx.mech_dir,
+ fresh=True)
+ stage.used_tools = set(st.get("used_tools", []))
+ stage.tool_unused_streaks = {k: int(v) for k, v in (st.get("streaks") or {}).items()}
+ stage.catalog_tools = set(st.get("catalog_tools", []))
+ stage.catalog_desc = dict(st.get("catalog_desc") or {})
+ applied = untool.applied(ctx.store)
+ stage._exclude_tools = list(applied) # the mask this session actually ran under
+ # A tool that has since become protected (the web tools, from this version on) comes back:
+ # remove the plugin's OWN deny rule for it. Only rules it recorded adding, never one the
+ # person wrote, and only in the direction of giving a tool back.
+ protected = adapter.protected_tools()
+ restored = []
+ if any(t in protected for t in applied) and not ctx.config.shadow:
+ try:
+ restored = untool.restore(ctx.repo.root, ctx.store,
+ [t for t in applied if t in protected], host=host)
+ except RuntimeError: # an unreadable or redirected settings file: leave it
+ restored = []
+ if restored:
+ applied = untool.applied(ctx.store)
+ # The stage warns once per STREAM when no catalog parses -- a harness defect signal there.
+ # Here a stage lives for one fold, and a session that called no tool legitimately has no
+ # observed catalog; the source (or its absence) is recorded in untool.json instead.
+ stage._warned_no_catalog = True
+ srow = {}
+ stage.after_task({"instance_id": session_id}, {"untooled_excluded": applied}, run_dir, srow)
+ sessions = int(st.get("sessions", 0)) + 1
+ raw = stage.build_tools({"instance_id": f"after-{session_id}"}, sessions) or []
+ doc = {"used_tools": sorted(stage.used_tools), "streaks": stage.tool_unused_streaks,
+ "catalog_tools": sorted(stage.catalog_tools), "catalog_desc": stage.catalog_desc,
+ # A session run under an applied mask parses no catalog (the stage skips it, as the
+ # masked list is not the full one); the last source still describes the catalog.
+ "catalog_source": srow.get("untool_catalog_source") or st.get("catalog_source"),
+ "sessions": sessions, "updated": time.time()}
+ if host == "copilot":
+ # Copilot's only persistent lever is a whole MCP server (`untool` module docstring).
+ servers = transcripts.copilot_mcp_servers(ctx.env)
+ proposal = [s for s in untool.copilot_server_proposal(
+ raw, stage.catalog_tools, stage.used_tools, servers) if s not in applied]
+ doc["servers"] = _server_summary(stage, servers)
+ else:
+ proposal = [t for t in untool.filter_proposal(raw, stage.tool_unused_streaks,
+ ctx.config.untool_evidence)
+ if t not in applied]
+ doc["proposal"] = proposal
+ doc["new_in_proposal"] = sorted(set(proposal) - set(st.get("proposal", [])))
+ added = []
+ if ctx.config.untool == "auto" and proposal and not ctx.config.shadow:
+ try:
+ added = untool.apply(ctx.repo.root, proposal, ctx.store, host=host)
+ doc["proposal"], doc["auto_applied"] = [], added
+ except RuntimeError as e: # an unreadable or redirected file: keep proposing
+ doc["auto_apply_refused"] = str(e)
+ if restored:
+ doc["restored_protected"] = restored
+ ctx.store.write("untool.json", doc)
+ return {"proposal": doc["proposal"], "applied_now": added, "restored": restored}
+
+
+def _server_summary(stage, servers):
+ """{server: {tools, used, unused_sessions}} -- the Copilot evidence at the unit it acts on."""
+ out = {}
+ for t in stage.catalog_tools:
+ s = transcripts.mcp_server_of(t, servers)
+ if not s:
+ continue
+ e = out.setdefault(s, {"tools": 0, "used": False, "unused_sessions": None})
+ e["tools"] += 1
+ e["used"] = e["used"] or t in stage.used_tools
+ n = int(stage.tool_unused_streaks.get(t, 0))
+ e["unused_sessions"] = n if e["unused_sessions"] is None else min(e["unused_sessions"], n)
+ return out
+
+
+def _fold_untool_codex(ctx, adapter, session_id, transcript, workdir):
+ """Codex: the experiments' capability controller (`codex_untool.CodexUntoolProfile`, driven
+ through `ToolingStage`) decides which packages went unused, over a trace that carries the
+ model's own `exec` calls (`transcripts.codex_capability_trace`). Two live adaptations:
+
+ * the session ran under the capabilities the PERSON applied, not the profile's own mask
+ (a plugin launches nothing), so the observation is recorded against the applied set;
+ * a package is proposed only once the profile has demoted it AND it went unused in
+ `untool_evidence` sessions (x3 for image/media, the high-risk package), with no use
+ ever -- the profile alone demotes collaboration after one unused task, which was right
+ for a sealed benchmark stream and is too quick for a person's sessions."""
+ from tokeneconomizer.config import ToolingConfig
+ from tokeneconomizer.mechanisms.tooling import ToolingStage
+ if not transcript or not workdir:
+ return None
+ cap_dir = os.path.join(workdir, "capabilities")
+ os.makedirs(cap_dir, exist_ok=True)
+ cap_trace = os.path.join(cap_dir, "trace.jsonl")
+ if transcripts.codex_capability_trace(transcript, cap_trace) is None:
+ return None
+ run_dir = _run_dir(cap_dir, cap_trace)
+ stage = ToolingStage(ToolingConfig(codex_untool=True), "codex", "agent", adapter=adapter,
+ repo=ctx.repo.label, results_root=ctx.mech_dir, fresh=False)
+ stage._warned_no_catalog = True
+ stage.build_tools({"instance_id": session_id}, 0)
+ applied = untool.applied(ctx.store)
+ stage.codex_profile._pending["logical_capabilities"] = sorted(applied)
+ srow = {}
+ stage.after_task({"instance_id": session_id}, {}, run_dir, srow)
+ state = stage.codex_profile.state
+ ever_used = set(state.get("ever_completed") or ())
+ loaded = state.get("loaded_exposures") or {}
+ proposal = [c for c in untool.CODEX_LIVE_CAPABILITIES
+ if c in (state.get("current_pruned") or ()) and c not in ever_used
+ and c not in applied
+ and int(loaded.get(c, 0)) >= ctx.config.untool_evidence * (
+ live_adapter.HIGH_RISK_FACTOR if c in CODEX_HIGH_RISK else 1)]
+ prior = ctx.store.read("untool.json") or {}
+ doc = {"sessions": int(state.get("tasks_completed") or 0), "catalog_source": "capabilities",
+ "capabilities": {c: {"loaded_sessions": int(loaded.get(c, 0)),
+ "unused_sessions": int((state.get("zero_attempt_exposures")
+ or {}).get(c, 0)),
+ "used": c in ever_used}
+ for c in untool.CODEX_LIVE_CAPABILITIES},
+ "proposal": proposal, "updated": time.time(),
+ "new_in_proposal": sorted(set(proposal) - set(prior.get("proposal", [])))}
+ added = []
+ if ctx.config.untool == "auto" and proposal and not ctx.config.shadow:
+ try:
+ added = untool.apply(ctx.repo.root, proposal, ctx.store, host="codex")
+ doc["proposal"], doc["auto_applied"] = [], added
+ except RuntimeError as e: # a committed or unparseable file: keep proposing
+ doc["auto_apply_refused"] = str(e)
+ ctx.store.write("untool.json", doc)
+ return {"proposal": doc["proposal"], "applied_now": added, "restored": []}
+
+
+# Rarely used, but expensive to lose when a session needs it -- the Codex counterpart of
+# adapter.HIGH_RISK (NotebookEdit on Claude Code).
+CODEX_HIGH_RISK = frozenset({"image_media"})
+
+
+def fold_session(ctx, session_id, transcript_path=None):
+ """Fold one finished session. Returns the ledger record, or None if already folded."""
+ sess_name = os.path.join("sessions", f"{session_id}.json")
+ with ctx.store.locked("fold", timeout=FOLD_LOCK_WAIT_S) as got:
+ if not got:
+ return None
+ folded = ctx.store.read("folded.json") or {}
+ if session_id in folded:
+ return None
+ sess = ctx.store.read(sess_name) or {}
+ transcript = transcript_path or sess.get("transcript")
+ record = {"kind": "session", "session_id": session_id, "host": ctx.host.name,
+ "repo": ctx.repo.label, "folded_at": time.time(),
+ "started": sess.get("started"), "ended": sess.get("ended"),
+ "shadow": ctx.config.shadow}
+ workdir = tempfile.mkdtemp(prefix="te-fold-")
+ try:
+ trace_path = ctx.host.mechanism_trace(transcript, workdir) if transcript else None
+ if not trace_path:
+ record["error"] = "transcript unreadable"
+ else:
+ run_dir = _run_dir(workdir, trace_path)
+ adapter = LiveAdapter(ctx.host, ctx.repo.root, vocabulary=ctx.config.vocabulary,
+ catalog=load_catalog(ctx.root, ctx.host.name))
+ with ctx.store.locked("mechanisms"):
+ inj = _fold_injection(ctx, adapter, run_dir, session_id)
+ unt = _fold_untool(ctx, adapter, run_dir, session_id,
+ transcript=transcript, workdir=workdir)
+ record["injection"] = inj
+ record["untool"] = unt
+ ctx.host.settle(transcript)
+ usage = ctx.host.session_usage(transcript)
+ eff, priced_by = accounting.effective(usage)
+ record.update({k: usage.get(k) for k in (
+ "model", "requests", "input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens")})
+ record["effective_tokens"], record["priced_by"] = eff, priced_by
+ turns = sess.get("turns") or []
+ record.update({
+ "turns": len(turns),
+ "nudges": int(sess.get("nudges", 0)), "aborts": int(sess.get("aborts", 0)),
+ "injected_chars": int(sess.get("injected_chars", 0)),
+ "untool_applied": untool.applied(ctx.store),
+ "overhead": {
+ "optimizer_tokens": ((record.get("injection") or {}).get("optimizer_tokens")
+ or 0),
+ "spiral_judge_tokens": float(sess.get("judge_tokens", 0) or 0),
+ "spiral_nudge_writer_tokens": float(sess.get("writer_tokens", 0) or 0)}})
+ ctx.store.append("ledger.jsonl", record)
+ folded[session_id] = {"at": time.time(), "transcript": transcript}
+ ctx.store.write("folded.json", folded)
+ if sess:
+ sess["folded"] = True
+ ctx.store.write(sess_name, sess)
+ return record
+ finally:
+ shutil.rmtree(workdir, ignore_errors=True)
+
+
+def pending_sessions(ctx, exclude=(), stale_s=STALE_S, now=None):
+ """Recorded sessions that ended (or went quiet) without being folded."""
+ now = now or time.time()
+ folded = ctx.store.read("folded.json") or {}
+ out = []
+ sess_dir = ctx.store.path("sessions")
+ if not os.path.isdir(sess_dir):
+ return out
+ for name in sorted(os.listdir(sess_dir)):
+ if not name.endswith(".json"):
+ continue
+ sid = name[:-5]
+ if sid in folded or sid in exclude:
+ continue
+ sess = ctx.store.read(os.path.join("sessions", name)) or {}
+ tp = sess.get("transcript") or ""
+ try:
+ quiet = now - os.path.getmtime(tp) >= stale_s
+ except OSError:
+ quiet = False
+ if sess.get("ended") or quiet:
+ out.append(sid)
+ return out
+
+
+def spawn_fold(ctx, session_ids):
+ """Fold sessions in a detached process (inline when TE_FOLD_SYNC=1, for tests)."""
+ if not session_ids:
+ return None
+ if ctx.env.get("TE_FOLD_SYNC") == "1":
+ return [fold_session(ctx, sid) for sid in session_ids]
+ # The directory holding both packages: the checkout, or the bundle's lib/.
+ core = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ env = dict(ctx.env)
+ env["PYTHONPATH"] = core + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
+ cmd = [sys.executable, "-m", "te_live.cli", "fold",
+ "--host", ctx.host.name, "--cwd", ctx.repo.root] + [
+ a for sid in session_ids for a in ("--session", sid)]
+ ctx.store.ensure()
+ log = open(ctx.store.path("fold.log"), "a")
+ try:
+ subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=log, stderr=log,
+ cwd=ctx.repo.root, env=env, start_new_session=True)
+ finally:
+ log.close()
+ return True
diff --git a/te_live/guard.py b/te_live/guard.py
new file mode 100644
index 0000000..3ffb69f
--- /dev/null
+++ b/te_live/guard.py
@@ -0,0 +1,535 @@
+"""The spiral guard, one tool call at a time.
+
+`vendor.spiral.NudgeGuard` is the live controller the experiments ran: it folds each completed
+tool call of a streamed rollout into a `SpiralState`, asks `assess_abort` (cheap gate -> sampled
+judge -> envelope fallback) whether the trajectory is stuck, and answers the first stuck
+detection with a nudge and a later one with an abort. It does that inside one process that owns
+the stream. Here every tool call arrives as a separate PostToolUse hook process, so `TurnGuard`
+is the same control flow with its state on disk between calls. The decisions are the vendored
+ones -- `SpiralState.update`, `is_stuck`, `assess_abort`, `nudge_text`, the cumulative backstop,
+`nudge_on_ambiguous` -- not a re-implementation of their thresholds.
+
+Three things differ, each because the setting does:
+
+ * The unit is a PROMPT-TURN (one user prompt to the agent's stop), not a benchmark task. In
+ interactive use the person already splits work into short turns (median 5 tool calls on the
+ local replay), and a whole session folded as one trajectory looks stuck merely by being long.
+ * The nudge is delivered mid-turn, attached to the tool result, instead of by killing the agent
+ and resuming it with a continuation prompt. The per-segment reset that gives the agent a fair
+ re-assessment after a nudge is kept: `after_nudge` starts a fresh `SpiralState`, exactly as
+ `NudgeGuard.continuation` does.
+ * There is no grader, so the per-repository envelope learns from turns that ENDED GREEN: the
+ agent stopped on its own, the guard did not abort it, and no verification after its last
+ edit failed (`resolved_proxy`). The envelope is the same `EnvelopeTracker`, fed that proxy.
+"""
+import re
+import shlex
+import time
+
+from tokeneconomizer.vendor import spiral
+
+_SLOTS = spiral.SpiralState.__slots__
+
+# What counts as running the project's own checks, across ecosystems. Broader than the vendored
+# `_TEST_RUN_RE` (which is the SWE-bench/Python vocabulary the guard's thresholds were tuned
+# on); used ONLY for the ended-green proxy, never to change what `SpiralState` counts.
+VERIFY_RE = re.compile(
+ r"(?i)(?:^|[\s;&|(])(?:pytest|py\.test|tox|nox|python\d?(?:\.\d+)?\s+-m\s+(?:pytest|unittest)|"
+ r"manage\.py\s+test|runtests(?:\.py)?|jest|vitest|mocha|ava|playwright\s+test|"
+ r"(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?test|go\s+test|cargo\s+(?:test|nextest)|"
+ r"(?:mvn|mvnw|\./mvnw)\s+(?:\S+\s+)*test|(?:gradle|gradlew|\./gradlew)\s+(?:\S+\s+)*test|"
+ r"make\s+(?:test|check)|ctest|rspec|phpunit|dotnet\s+test|swift\s+test|mix\s+test|"
+ r"deno\s+test|bazel\s+test|(?:\./)?bin/test)\b")
+_EDIT = {"edit", "create"}
+
+
+# ---- edits made through the shell ----------------------------------------------------------
+# The vendored SpiralState counts an edit only when an edit TOOL makes it (Edit/Write/
+# str_replace). An agent that edits through Bash -- `sed -i`, `perl -pi`, a redirect or `tee`
+# into a source file, an inline Python script that writes the file back, `patch`/`git apply` --
+# looked like one that had edited nothing. Sonnet 5.5 edits that way; in the user-simulation
+# pilot the paid judge's digest read `edit_churn=0`, and its nudge told the agent "no files have
+# actually been edited" when four had. Such edits are folded in AFTER the vendored update, so its
+# counts and thresholds are untouched and the call still counts once.
+_HEREDOC_START = re.compile(r"<<-?\s*(['\"]?)([A-Za-z_]\w*)\1")
+_OPS = {"&&", "||", ";", ";;", "|", "|&", "&", "\n", "(", ")"}
+_OUT = {">", ">>", ">|", "&>", "&>>"} # redirects that write their operand
+_SKIP_OPERAND = {"<", "<<", "<<<", "<&", ">&", "<>"} # input, heredoc tags, fd duplication
+# Redirect and `tee` targets count only when they look like source, so `> out.log` or
+# `| tee /tmp/run.txt` -- capturing output -- is not mistaken for an edit.
+_SOURCE_FILE = re.compile(
+ r"\.(?:py|pyi|pyx|pxd|js|jsx|mjs|cjs|ts|tsx|rs|go|java|kt|kts|scala|c|cc|cpp|cxx|h|hh|hpp|"
+ r"rb|php|cs|swift|m|mm|sh|bash|zsh|lua|jl|ex|exs|erl|hs|ml|clj|sql|toml|ya?ml|json|cfg|ini|"
+ r"conf|md|rst|html?|css|scss|vue|svelte|xml|gradle|properties|in)$", re.I)
+_PYTHON = re.compile(r"(?:^|[\s/;&|(])python\d?(?:\.\d+)?\b")
+_PY_OPEN_LIT = re.compile(r"open\(\s*(['\"])([^'\"\n]+)\1\s*,\s*(['\"])([^'\"]*)\3")
+_PY_OPEN_VAR = re.compile(r"open\(\s*([A-Za-z_]\w*)\s*,\s*(['\"])([^'\"]*)\2")
+_PY_WRITE_PATH_LIT = re.compile(
+ r"(?:pathlib\.)?Path\(\s*(['\"])([^'\"\n]+)\1\s*\)\.write_(?:text|bytes)\(")
+_PY_WRITE_PATH_VAR = re.compile(r"\b([A-Za-z_]\w*)\.write_(?:text|bytes)\(")
+
+
+def _not_scratch(path):
+ return bool(path) and not path.startswith(("/dev/", "/proc/", "/tmp/"))
+
+
+def _unquote(t):
+ return t[1:-1] if len(t) >= 2 and t[0] == t[-1] and t[0] in "'\"" else t
+
+
+def _split_heredocs(cmd):
+ """(the command's shell text without heredoc bodies, the bodies)."""
+ shell, bodies, tag, body = [], [], None, []
+ for line in cmd.split("\n"):
+ if tag is not None:
+ if line.strip() == tag:
+ bodies.append("\n".join(body))
+ tag, body = None, []
+ else:
+ body.append(line)
+ continue
+ shell.append(line)
+ m = _HEREDOC_START.search(line)
+ if m:
+ tag = m.group(2)
+ if tag is not None:
+ bodies.append("\n".join(body))
+ return "\n".join(shell), bodies
+
+
+def _tokens(text):
+ """Shell words and operators. Quotes stay on quoted words, so a quoted `'>'` or `'|'` is
+ never read as an operator; ValueError on unbalanced quotes."""
+ lex = shlex.shlex(text, posix=False, punctuation_chars="();<>|&\n")
+ lex.whitespace = " \t\r"
+ lex.whitespace_split = True
+ return list(lex)
+
+
+def _commands(shell):
+ """(simple commands as unquoted word lists, the files their redirects write)."""
+ try:
+ streams = [_tokens(shell)]
+ except ValueError: # one bad line must not hide the others
+ streams = []
+ for line in shell.split("\n"):
+ try:
+ streams.append(_tokens(line) + ["\n"])
+ except ValueError:
+ continue
+ cmds, writes = [], []
+ for toks in streams:
+ cur, i = [], 0
+ while i < len(toks):
+ t = toks[i]
+ if t in _OPS:
+ if cur:
+ cmds.append(cur)
+ cur = []
+ elif t in _OUT or t in _SKIP_OPERAND:
+ if cur and cur[-1].isdigit(): # `2>`: the fd number is not an operand
+ cur.pop()
+ if t in _OUT and i + 1 < len(toks) and toks[i + 1] not in _OPS:
+ writes.append(_unquote(toks[i + 1]))
+ i += 1
+ else:
+ cur.append(_unquote(t))
+ i += 1
+ if cur:
+ cmds.append(cur)
+ return cmds, writes
+
+
+def _inplace_files(tokens, inplace_flag, script_flag):
+ """File operands of an in-place editor (`sed -i`, `perl -pi`), or [] if not in place."""
+ if not any(inplace_flag(t) for t in tokens[1:]):
+ return []
+ files, have_script, skip = [], False, False
+ for t in tokens[1:]:
+ if skip:
+ skip, have_script = False, True
+ continue
+ if not t or t.startswith("-"):
+ if t and script_flag(t):
+ skip = True # the next token is the script
+ continue
+ if not have_script:
+ have_script = True # the first operand is the script
+ continue
+ files.append(t)
+ return files
+
+
+def _py_writes(text):
+ """Paths an inline Python script writes; "(script)" for a write whose path this cannot
+ resolve."""
+ out = []
+ for m in _PY_OPEN_LIT.finditer(text):
+ if set(m.group(4)) & set("wax+"):
+ out.append(m.group(2))
+ variables = [m.group(1) for m in _PY_OPEN_VAR.finditer(text) if set(m.group(3)) & set("wax+")]
+ out += [m.group(2) for m in _PY_WRITE_PATH_LIT.finditer(text)]
+ variables += [m.group(1) for m in _PY_WRITE_PATH_VAR.finditer(text)]
+ for var in dict.fromkeys(variables):
+ found = re.findall(rf"\b{re.escape(var)}\s*=\s*(?:(?:pathlib\.)?Path\(\s*)?(['\"])"
+ rf"([^'\"\n]+)\1", text)
+ loop = re.search(rf"\bfor\s+{re.escape(var)}\s+in\s+[\[(]([^\])]*)[\])]", text)
+ paths = [p for _, p in found]
+ if loop:
+ paths += re.findall(r"['\"]([^'\"\n]+)['\"]", loop.group(1))
+ out += paths or ["(script)"]
+ return out
+
+
+def shell_edit_targets(cmd):
+ """Basenames of the files a shell command writes, in order, without repeats."""
+ if not isinstance(cmd, str) or not cmd.strip():
+ return []
+ shell, bodies = _split_heredocs(cmd)
+ cmds, writes = _commands(shell)
+ found = [w for w in writes if _SOURCE_FILE.search(w)]
+ for tokens in cmds:
+ while tokens and (tokens[0] in ("sudo", "command", "env", "time", "nohup")
+ or re.match(r"^\w+=", tokens[0])):
+ tokens = tokens[1:]
+ if not tokens:
+ continue
+ head = tokens[0].rsplit("/", 1)[-1]
+ if head == "sed":
+ found += _inplace_files(
+ tokens, lambda t: bool(re.match(r"^(?:-[a-zA-Z]*i|--in-place)", t)),
+ lambda t: t in ("-e", "-f", "--expression", "--file"))
+ elif head == "perl":
+ found += _inplace_files(tokens, lambda t: bool(re.match(r"^-[a-zA-Z]*i", t)),
+ lambda t: bool(re.match(r"^-[a-zA-Z]*e$", t)))
+ elif head == "tee":
+ found += [t for t in tokens[1:] if not t.startswith("-") and _SOURCE_FILE.search(t)]
+ elif head == "patch" or (head == "git" and len(tokens) > 1 and tokens[1] == "apply"):
+ found.append("(patch)")
+ elif head == "git" and len(tokens) > 2 and tokens[1] in ("checkout", "restore"):
+ rest = (tokens[tokens.index("--") + 1:] if "--" in tokens
+ else tokens[2:] if tokens[1] == "restore" else [])
+ found += [t for t in rest if not t.startswith("-")]
+ if _PYTHON.search(shell):
+ found += _py_writes("\n".join([shell] + bodies))
+ names = [p if p.startswith("(") else spiral._basename(p)
+ for p in found if p.startswith("(") or _not_scratch(p)]
+ return list(dict.fromkeys(n for n in names if n))
+
+
+def fold_shell_edits(st, names):
+ """Count `names` as edits in a SpiralState, and say so in its action log."""
+ for b in names:
+ st.edit_order.append(b)
+ st.edit_counts[b] = st.edit_counts.get(b, 0) + 1
+ if names and st.recent:
+ shown = ", ".join(names[:4]) + (", ..." if len(names) > 4 else "")
+ st.recent[-1] = f"{st.recent[-1]} [edits {shown}]"
+
+
+def state_to_dict(st):
+ return {k: getattr(st, k) for k in _SLOTS}
+
+
+def state_from_dict(d):
+ st = spiral.SpiralState()
+ for k in _SLOTS:
+ if k in (d or {}):
+ setattr(st, k, d[k])
+ return st
+
+
+class TurnGuard:
+ """`NudgeGuard` for one prompt-turn, persisted between hook calls via `to_dict`."""
+
+ JUDGE_BASE_GAP = 5
+ JUDGE_MAX_GAP = 20
+ # A background verdict that never came back (its job died) stops blocking the next one
+ # after this long; a side call is capped at 180 s.
+ JUDGE_STALE_S = 300
+
+ def __init__(self, turn_key="", data=None):
+ d = data or {}
+ self.turn_key = d.get("turn_key", turn_key)
+ self.st = state_from_dict(d.get("st")) # this segment; reset after a nudge
+ self.whole = state_from_dict(d.get("whole")) # the whole turn; never reset
+ self.cum_turns = int(d.get("cum_turns", 0))
+ self.segment = int(d.get("segment", 0))
+ self.nudges_done = int(d.get("nudges_done", 0))
+ self.aborted = bool(d.get("aborted", False))
+ self.abort_delivered = bool(d.get("abort_delivered", False))
+ self.events = list(d.get("events", []))
+ self.judge_last = int(d.get("judge_last", -999))
+ self.judge_gap = int(d.get("judge_gap", self.JUDGE_BASE_GAP))
+ self.judge_calls = int(d.get("judge_calls", 0))
+ self.judge_tokens = float(d.get("judge_tokens", 0) or 0)
+ self.writer_tokens = float(d.get("writer_tokens", 0) or 0)
+ self.last_edit = int(d.get("last_edit", 0))
+ self.last_verify = int(d.get("last_verify", 0))
+ self.last_verify_failed = bool(d.get("last_verify_failed", False))
+ self.edits = int(d.get("edits", 0))
+ self.verifications = int(d.get("verifications", 0))
+ self.judge_pending = d.get("judge_pending") or None # {"id", "since", "segment"}
+ self.nudge_ready = d.get("nudge_ready") or None # a redirect the judge wrote
+
+ def to_dict(self):
+ return {"turn_key": self.turn_key, "st": state_to_dict(self.st),
+ "whole": state_to_dict(self.whole),
+ "cum_turns": self.cum_turns, "segment": self.segment,
+ "nudges_done": self.nudges_done, "aborted": self.aborted,
+ "abort_delivered": self.abort_delivered, "events": self.events[-20:],
+ "judge_last": self.judge_last, "judge_gap": self.judge_gap,
+ "judge_calls": self.judge_calls, "judge_tokens": self.judge_tokens,
+ "writer_tokens": self.writer_tokens, "last_edit": self.last_edit,
+ "last_verify": self.last_verify, "last_verify_failed": self.last_verify_failed,
+ "edits": self.edits, "verifications": self.verifications,
+ "judge_pending": self.judge_pending, "nudge_ready": self.nudge_ready}
+
+ # ---- outcome tracking for the ended-green proxy ----------------------------------
+ def _track(self, name, args, result, exit_code, shell_edits=()):
+ canon = spiral._canon_tool(name) # the vendored tool-name mapping
+ if canon in _EDIT:
+ self.last_edit = self.cum_turns
+ self.edits += 1
+ return
+ cmd = (args or {}).get("command") if canon == "bash" else None
+ verifies = isinstance(cmd, str) and bool(VERIFY_RE.search(cmd))
+ if shell_edits:
+ # `sed -i ... && pytest`: the edit comes first, so the run verifies it.
+ self.last_edit = self.cum_turns - (1 if verifies else 0)
+ self.edits += 1
+ if verifies:
+ self.last_verify = self.cum_turns
+ self.verifications += 1
+ if exit_code is not None:
+ self.last_verify_failed = exit_code != 0
+ else: # Claude Code reports no exit code
+ self.last_verify_failed = bool(spiral._ERR_RE.search(result or ""))
+
+ def resolved_proxy(self, stopped_normally=True):
+ """Did this turn END GREEN? The stand-in for the grader the envelope cannot have."""
+ if not stopped_normally or self.aborted or self.cum_turns == 0:
+ return False
+ return not (self.last_verify_failed and self.last_verify > self.last_edit)
+
+ # ---- what the judge is told -------------------------------------------------------
+ # The vendored digest says "Tests are still NOT green." whenever it has an envelope: on the
+ # benchmark the guard only ever watched an unfinished task. Here the guard knows how the
+ # turn's last verification went, so the judge -- and the redirect it writes -- is told that.
+ _NOT_GREEN = "Tests are still NOT green."
+
+ def verification_state(self):
+ if not self.verifications:
+ return "No test or verification run has been seen in this turn."
+ state = "failed" if self.last_verify_failed else "passed"
+ since = " There have been edits since." if self.last_edit > self.last_verify else ""
+ return f"The last verification run {state}.{since}"
+
+ def live_digest(self, digest):
+ return (digest or "").replace(self._NOT_GREEN, self.verification_state())
+
+ # ---- the judge, rate-limited with backoff exactly as NudgeGuard._sampled_judge ----
+ def _backoff(self, v):
+ if v and v.get("stuck") is False:
+ self.judge_gap = min(self.JUDGE_MAX_GAP, self.judge_gap + self.JUDGE_BASE_GAP)
+ else:
+ self.judge_gap = self.JUDGE_BASE_GAP
+
+ def _sampled(self, judge):
+ def call(digest):
+ if self.st.turns - self.judge_last < self.judge_gap:
+ return None
+ self.judge_last = self.st.turns
+ self.judge_calls += 1
+ try:
+ v = judge(self.live_digest(digest))
+ except Exception:
+ v = None
+ if v:
+ self.judge_tokens += v.get("_tokens") or 0
+ self._backoff(v)
+ return v
+ return call
+
+ def _background(self, judge):
+ """`_sampled` for a judge that runs in the background (`te_live.judge.AsyncJudge`):
+ a verdict is taken whenever one has come back, a new job starts under the same rate
+ limit, and while one is out the judge answers nothing -- which `assess_abort` reads as
+ throttled, the same as a call the rate limit skipped."""
+ def call(digest):
+ found, v = judge.poll(self)
+ if found:
+ self.judge_pending = None
+ if v:
+ self.nudge_ready = (v.get("_nudge") or "").strip() or None
+ self._backoff(v)
+ return v or None
+ if self.judge_pending:
+ if time.time() - float(self.judge_pending.get("since") or 0) < self.JUDGE_STALE_S:
+ return None
+ self.judge_pending = None # its job never answered
+ if self.st.turns - self.judge_last < self.judge_gap:
+ return None
+ self.judge_last = self.st.turns
+ self.judge_calls += 1
+ job = judge.start(self, self.live_digest(digest))
+ if job:
+ self.judge_pending = {"id": job, "since": time.time(), "segment": self.segment}
+ return None
+ return call
+
+ # ---- one completed tool call ------------------------------------------------------
+ def fold(self, name, args, result, exit_code=None, *, envelope=None, thr=None,
+ judge=None, nudge_budget=1):
+ """Fold one call; return None, ("nudge", reason) or ("abort", reason).
+
+ The body is `NudgeGuard.monitor` for a single completed call."""
+ thr = thr or spiral.DEFAULT_THRESHOLDS
+ shell_edits = (shell_edit_targets((args or {}).get("command"))
+ if spiral._canon_tool(name) == "bash" else [])
+ self.st.update(name, args or {}, result or "")
+ self.whole.update(name, args or {}, result or "")
+ fold_shell_edits(self.st, shell_edits)
+ fold_shell_edits(self.whole, shell_edits)
+ self.cum_turns += 1
+ self._track(name, args, result, exit_code, shell_edits)
+ if self.aborted:
+ return None
+
+ backstop = thr.get("backstop_turns")
+ if backstop is None and envelope is not None:
+ backstop = envelope * thr["backstop_mult"]
+ if (backstop is not None and self.cum_turns >= backstop
+ and spiral.is_stuck(self.st, thr)):
+ # The ladder is nudge, THEN stop. In the experiments the envelope was ~30 calls, so
+ # its 2.5x backstop lay well past the stuck rule's 20-call floor and the nudge came
+ # first. An interactive repository's envelope is ~4-10 calls: its backstop falls
+ # INSIDE that floor, so the moment a turn first counts as stuck the backstop has
+ # tripped too, and it pre-empted every nudge -- found by replaying a stuck trace
+ # before a live test (2026-10-07). An unspent nudge goes first; after it, a fresh
+ # segment must be stuck again before the backstop stops anything.
+ reason = f"cum-backstop:{self.cum_turns}>={backstop:.0f}t"
+ if self.nudges_done < nudge_budget:
+ return self._stop("nudge", reason, envelope)
+ return self._stop("abort", reason, envelope)
+
+ wired = None
+ if judge is not None:
+ wired = self._background(judge) if hasattr(judge, "poll") else self._sampled(judge)
+ abort, reason = spiral.assess_abort(self.st, envelope=envelope, judge=wired, thr=thr)
+ if not abort:
+ if (thr.get("nudge_on_ambiguous") and reason == "judge-ambiguous-spared"
+ and self.nudges_done < nudge_budget):
+ return self._stop("nudge", "ambiguous-past-envelope", envelope)
+ return None
+ if self.nudges_done < nudge_budget:
+ return self._stop("nudge", reason, envelope)
+ return self._stop("abort", reason, envelope)
+
+ def _stop(self, action, reason, envelope):
+ self.events.append({"segment": self.segment, "action": action,
+ "turn_in_segment": self.st.turns, "cum_turns": self.cum_turns,
+ "envelope": envelope, "reason": reason,
+ "snapshot": self.st.snapshot()})
+ if action == "abort":
+ self.aborted = True
+ return action, reason
+
+ def nudge_body(self, envelope=None, writer=None):
+ """The redirect text: the one a background judge already wrote with its verdict, else
+ an LLM-written one when a writer is wired, else the vendored count template
+ (`spiral.nudge_text`), which mirrors only the agent's own actions."""
+ if self.nudge_ready:
+ text, self.nudge_ready = self.nudge_ready, None
+ return text
+ if writer is not None:
+ try:
+ out = writer(self.live_digest(self.st.recent_digest(envelope=envelope)))
+ except Exception:
+ out = None
+ if out:
+ self.writer_tokens += out.get("_tokens") or 0
+ text = (out.get("text") or "").strip()
+ if text:
+ return text
+ return spiral.nudge_text(self.st)
+
+ def after_nudge(self):
+ """Start the post-nudge segment with a fresh state: `NudgeGuard.continuation`'s reset,
+ which is the agent's fair re-assessment window."""
+ self.nudges_done += 1
+ self.segment += 1
+ self.st = spiral.SpiralState()
+ self.judge_last = -999
+ self.judge_gap = self.JUDGE_BASE_GAP
+ self.judge_pending = None # a verdict on the old segment is about the old one
+ self.nudge_ready = None
+
+ def summary(self, stopped_normally=True):
+ return {"turn_key": self.turn_key, "tool_calls": self.cum_turns,
+ "nudges": self.nudges_done, "aborted": self.aborted,
+ "resolved_proxy": self.resolved_proxy(stopped_normally),
+ "edits": self.edits, "verifications": self.verifications,
+ "judge_tokens": self.judge_tokens, "writer_tokens": self.writer_tokens}
+
+
+# ---- the per-repository envelope ---------------------------------------------------------
+ENVELOPE_HISTORY = 500 # recent ended-green turns kept for the percentile
+
+
+# Overrides that shape the envelope itself, passed to the tracker; the rest are pinned over the
+# thresholds it returns.
+_ENVELOPE_KEYS = {"envelope_margin": "margin", "envelope_pct": "pct",
+ "envelope_min_samples": "min_samples"}
+
+
+def load_envelope(store, level="default", overrides=None):
+ d = store.read("envelope.json") or {}
+ ov = overrides or {}
+ tracker = spiral.EnvelopeTracker(level=level, **{
+ arg: ov[key] for key, arg in _ENVELOPE_KEYS.items() if key in ov})
+ tracker.resolved_turns = [int(x) for x in d.get("resolved_turns", []) if x]
+ tracker.resolved_snaps = [s for s in d.get("resolved_snaps", []) if isinstance(s, dict)]
+ return tracker
+
+
+def thresholds(tracker, overrides=None):
+ """The thresholds a turn is checked against: the preset, what this repository's history
+ learned on top of it, then a person's pinned values (`spiral.<name>`), which win."""
+ thr = tracker.thresholds(max_turns=None)
+ for key, value in (overrides or {}).items():
+ if key not in _ENVELOPE_KEYS:
+ thr[key] = value
+ if "backstop_mult" in (overrides or {}) and tracker.envelope is not None:
+ thr["backstop_turns"] = tracker.envelope * thr["backstop_mult"]
+ return thr
+
+
+def base_thresholds(level="default", overrides=None):
+ """The thresholds before any learning: the preset with a person's pinned values on top.
+ What `te config show` reports, so the numbers a setting replaces are visible."""
+ thr = spiral.scaled_thresholds(level)
+ thr.update(overrides or {})
+ thr.pop("restart_budget", None)
+ return thr
+
+
+def save_envelope(store, tracker):
+ store.write("envelope.json", {
+ "level": tracker.level,
+ "resolved_turns": tracker.resolved_turns[-ENVELOPE_HISTORY:],
+ "resolved_snaps": tracker.resolved_snaps[-ENVELOPE_HISTORY:]})
+
+
+def record_turn(store, guard, level="default", stopped_normally=True, overrides=None):
+ """Fold a finished turn into the envelope -- under the store lock, since two sessions in
+ one repository share it. Only turns that ended green feed it, as only resolved tasks fed
+ the experiments' envelope; a turn's own outcome gates only later turns."""
+ # The snapshot is of the WHOLE turn (`guard.whole`), not the post-nudge segment: the
+ # experiments snapshot a resolved task's full trajectory (`spiral_snapshot(transcript)`).
+ with store.locked("envelope"):
+ tracker = load_envelope(store, level, overrides)
+ tracker.update(guard.cum_turns, guard.resolved_proxy(stopped_normally),
+ snapshot=guard.whole.snapshot())
+ save_envelope(store, tracker)
+ return tracker
diff --git a/te_live/hook.py b/te_live/hook.py
new file mode 100644
index 0000000..2e89917
--- /dev/null
+++ b/te_live/hook.py
@@ -0,0 +1,81 @@
+"""The entry point every plugin hook executes: hook JSON on stdin, hook JSON on stdout.
+
+ python -m te_live.hook [--host claude|codex]
+
+It never fails the agent. Whatever goes wrong -- unreadable input, a full disk, a bug -- the hook
+prints `{}` (which changes nothing) and records the traceback in `<state root>/errors.log`. It
+also does nothing at all in two cases: `TE_DISABLE=1`, and `TE_SIDECALL=1`, which marks the
+nested agent process of the plugin's own paid side calls (`live.sidecall`) so it cannot recurse
+into these hooks.
+"""
+import contextlib
+import json
+import os
+import sys
+import time
+import traceback
+
+
+def _truthy(v):
+ return str(v or "").strip().lower() in ("1", "true", "yes", "on")
+
+
+def run(payload, env, host_hint=None):
+ """Handle one hook payload; returns the output dict (no I/O, so tests call it directly)."""
+ from . import hosts
+ from .context import LiveContext
+ from .controller import LiveController
+ host = hosts.detect(payload, env, hint=host_hint)
+ ctx = LiveContext.build(host, host.cwd(payload, env), env)
+ return LiveController(ctx, payload).handle()
+
+
+def _record_error(env):
+ try:
+ from .paths import state_root
+ root = state_root(env)
+ os.makedirs(root, exist_ok=True)
+ with open(os.path.join(root, "errors.log"), "a", encoding="utf-8") as fh:
+ fh.write(f"--- {time.strftime('%Y-%m-%dT%H:%M:%S')} pid={os.getpid()}\n")
+ fh.write(traceback.format_exc())
+ except Exception:
+ pass
+
+
+def main(argv=None, stdin=None, stdout=None, env=None):
+ argv = sys.argv[1:] if argv is None else argv
+ stdin = sys.stdin if stdin is None else stdin
+ stdout = sys.stdout if stdout is None else stdout
+ env = os.environ if env is None else env
+ hint = None
+ if "--host" in argv:
+ i = argv.index("--host")
+ hint = argv[i + 1] if i + 1 < len(argv) else None
+ out = {}
+ try:
+ raw = stdin.read()
+ if not (_truthy(env.get("TE_SIDECALL")) or _truthy(env.get("TE_DISABLE"))):
+ payload = json.loads(raw) if raw.strip() else {}
+ if isinstance(payload, dict):
+ # stdout carries exactly one JSON object back to the agent. The mechanisms
+ # print progress and warnings (ToolingStage's catalog warning, for one), and a
+ # single stray line there would make the agent discard the whole hook output.
+ with contextlib.redirect_stdout(sys.stderr):
+ out = run(payload, env, host_hint=hint) or {}
+ from .paths import secure_root
+ secure_root(env)
+ except Exception:
+ _record_error(env)
+ out = {}
+ from .hosts import PROGRESS
+ for line in out.pop(PROGRESS, None) or ():
+ stdout.write(json.dumps({"type": "progress", "message": line}))
+ stdout.write("\n")
+ stdout.write(json.dumps(out))
+ stdout.write("\n")
+ stdout.flush()
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/te_live/hosts.py b/te_live/hosts.py
new file mode 100644
index 0000000..aea83e3
--- /dev/null
+++ b/te_live/hosts.py
@@ -0,0 +1,372 @@
+"""Host adapters: one per coding agent whose hooks the live controller runs under.
+
+Claude Code and Codex speak the same hook contract -- JSON on stdin with `session_id`,
+`transcript_path`, `cwd`, `hook_event_name`, and `tool_name` / `tool_input` / `tool_response` per
+tool call; JSON on stdout with `hookSpecificOutput.additionalContext`, `systemMessage`, and
+`continue` / `stopReason`. The differences that matter here are few, and each is an attribute of
+an adapter rather than a branch in a mechanism:
+
+ * Turn identity. Claude Code stamps every event of a prompt-turn with `prompt_id`; Codex
+ stamps `turn_id`.
+ * Stopping. Claude Code's `{"continue": false}` from PostToolUse halts the agent loop after the
+ current tool (verified on 2.1.286; the session stays usable). Codex's does NOT stop a turn
+ mid-flight -- it replaces the tool result and the model carries on -- so on Codex the guard's
+ last rung is a notice to the user, matching the Codex-native spiral policies, which never
+ abort by design.
+ * Tool results. Claude Code's Bash `tool_response` is `{stdout, stderr, interrupted, ...}` with
+ no exit code (verified); Codex's command items carry one, and it is used when present.
+ * The session log. Claude Code's transcript is read as-is; Codex's rollout is rewritten into
+ exec-stream lines first (`transcripts.codex_rollout_to_exec`).
+
+The GitHub Copilot CLI runs the same Claude-format hook file (PascalCase events, Claude-shaped
+snake_case payloads), with differences each verified on 1.0.83 against a local scripted model
+endpoint (docs/PLUGIN.md §3):
+
+ * Context goes in a TOP-LEVEL `additionalContext`; `hookSpecificOutput` is ignored.
+ * There is no turn identity and no `transcript_path` before Stop; the session log is always
+ `<COPILOT_HOME>/session-state/<session_id>/events.jsonl`, which the trace seam already reads.
+ * Tool names are Claude's (`Bash`, `Read`); a shell result is `tool_result.
+ text_result_for_llm` ending in `<shellId: N completed with exit code X>`, and a failed tool
+ fires `PostToolUseFailure` with an `error` string.
+ * `continue: false` does not stop a turn (three of three calls ran), so, as on Codex, the
+ guard's last rung is a notice.
+ * `systemMessage` is never shown. What the TUI does show is a stdout line
+ `{"type": "progress", "message": ...}` printed BEFORE the hook's final JSON object, which
+ still applies (verified on SessionStart, UserPromptSubmit and PostToolUse). So a Copilot
+ notice is carried as `PROGRESS` lines that `hook.main` writes ahead of the output.
+
+`detect` picks the adapter from the payload: Codex's `turn_id`, or a transcript under `.codex/`
+named `rollout-*`; Copilot's session-state transcript, or no transcript at all inside a Copilot
+hook process; Claude Code otherwise. Each plugin hook file except Claude Code's also passes
+`--host` explicitly, so detection is the fallback rather than the rule.
+"""
+import json
+import os
+import re
+import shlex
+import tempfile
+import time
+
+from . import transcripts
+
+_RESPONSE_KEYS = ("stdout", "stderr", "output", "aggregated_output", "formatted_output",
+ "content", "result", "error", "message", "text_result_for_llm")
+_EXIT_RE = re.compile(r"(?im)^\s*(?:exit code|process exited with code)[:\s]+(-?\d+)")
+# Copilot appends this line to every shell result.
+_COPILOT_EXIT_RE = re.compile(r"<shellId: [^>]*?completed with exit code (-?\d+)>")
+_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M)
+RESULT_CAP = 20000 # characters of a tool result the guard folds (errors live early)
+# Notices a host shows as progress lines ahead of the hook's JSON (Copilot); `hook.main` writes
+# them and removes the key, so it never reaches an agent as an output field.
+PROGRESS = "_progress"
+
+
+def response_text(resp, cap=RESULT_CAP):
+ """Best-effort text of a hook's `tool_response`, whatever shape the tool returned."""
+ if resp is None:
+ return ""
+ if isinstance(resp, str):
+ return resp[:cap]
+ if isinstance(resp, list):
+ return " ".join(x.get("text", "") if isinstance(x, dict) else str(x)
+ for x in resp)[:cap]
+ if isinstance(resp, dict):
+ parts = []
+ for k in _RESPONSE_KEYS:
+ v = resp.get(k)
+ if isinstance(v, str) and v:
+ parts.append(v)
+ elif isinstance(v, list):
+ parts.append(" ".join(x.get("text", "") for x in v if isinstance(x, dict)))
+ if parts:
+ return "\n".join(parts)[:cap]
+ return json.dumps(resp, sort_keys=True)[:cap]
+ return str(resp)[:cap]
+
+
+def merge_outputs(*outputs):
+ """Combine hook outputs: contexts and notices concatenate, the last stop decision wins."""
+ out = {}
+ for o in outputs:
+ if not o:
+ continue
+ for k, v in o.items():
+ if k == "systemMessage" and out.get(k):
+ out[k] = out[k] + "\n" + v
+ elif k == "additionalContext" and out.get(k):
+ out[k] = out[k] + "\n\n" + v
+ elif k == PROGRESS:
+ out[k] = list(out.get(k) or []) + list(v)
+ elif k == "hookSpecificOutput" and out.get(k):
+ a = out[k].get("additionalContext") or ""
+ b = v.get("additionalContext") or ""
+ out[k] = dict(out[k], **v)
+ if a and b:
+ out[k]["additionalContext"] = a + "\n\n" + b
+ else:
+ out[k] = v
+ return out
+
+
+class Host:
+ name = "?"
+ can_stop_mid_turn = False
+
+ def session_id(self, p):
+ return str(p.get("session_id") or "")
+
+ def transcript(self, p):
+ return p.get("transcript_path") or ""
+
+ def cwd(self, p, env):
+ return p.get("cwd") or env.get("CLAUDE_PROJECT_DIR") or os.getcwd()
+
+ def turn_key(self, p):
+ return ""
+
+ def is_subagent(self, p):
+ return bool(p.get("agent_id"))
+
+ def context(self, event, text):
+ return {"hookSpecificOutput": {"hookEventName": event, "additionalContext": text}}
+
+ def notice(self, text):
+ return {"systemMessage": text}
+
+ def stop(self, reason):
+ return None
+
+ def headless(self, env):
+ return False
+
+ def tool_call(self, p):
+ """(tool_name, args, result_text, exit_code_or_None) for a PostToolUse payload."""
+ args = p.get("tool_input") if isinstance(p.get("tool_input"), dict) else {}
+ resp = p.get("tool_response")
+ return str(p.get("tool_name") or ""), args, response_text(resp), _exit_code(resp)
+
+ # -- the finished session, in the shapes the mechanisms read --------------------------
+ def mechanism_trace(self, transcript_path, workdir):
+ """Path of a trace `trace.load_trajectory` reads, for the finished session."""
+ return transcript_path if transcript_path and os.path.exists(transcript_path) else None
+
+ def session_usage(self, transcript_path):
+ raise NotImplementedError
+
+ def settle(self, transcript_path):
+ """Wait, if this host needs to, until the finished session's log is complete."""
+
+ def tool_usage(self, trace_path):
+ from tokeneconomizer import trace
+ return trace.tool_usage(trace_path)
+
+
+def _exit_code(resp):
+ if isinstance(resp, dict):
+ for k in ("exit_code", "exitCode", "returncode", "status_code"):
+ v = resp.get(k)
+ if isinstance(v, int) and not isinstance(v, bool):
+ return v
+ meta = resp.get("metadata")
+ if isinstance(meta, dict) and isinstance(meta.get("exit_code"), int):
+ return meta["exit_code"]
+ text = resp if isinstance(resp, str) else next(
+ (resp.get(k) for k in ("output", "text_result_for_llm")
+ if isinstance(resp, dict) and isinstance(resp.get(k), str)), None)
+ if text:
+ m = _EXIT_RE.search(text[:400])
+ if m:
+ return int(m.group(1))
+ m = _COPILOT_EXIT_RE.search(text[-400:])
+ if m:
+ return int(m.group(1))
+ return None
+
+
+class ClaudeCode(Host):
+ name = "claude"
+ can_stop_mid_turn = True
+
+ def turn_key(self, p):
+ return str(p.get("prompt_id") or "")
+
+ def stop(self, reason):
+ return {"continue": False, "stopReason": reason}
+
+ def headless(self, env):
+ # Observed in a `claude -p` session's hook environment (2.1.285):
+ # CLAUDE_CODE_SESSION_ATTENDED=0 and CLAUDE_CODE_ENTRYPOINT=sdk-cli. The attended flag
+ # says what matters directly -- is a person watching -- so it wins when present.
+ attended = str(env.get("CLAUDE_CODE_SESSION_ATTENDED", "")).strip()
+ if attended in ("0", "1"):
+ return attended == "0"
+ return str(env.get("CLAUDE_CODE_ENTRYPOINT", "")).startswith("sdk")
+
+ def session_usage(self, transcript_path):
+ return transcripts.claude_session_usage(transcript_path)
+
+ def tool_usage(self, trace_path):
+ return transcripts.claude_tool_usage(os.path.realpath(trace_path))
+
+
+_CODEX_SHELL = {"bash", "shell", "exec_command", "local_shell", "unified_exec",
+ "container.exec", "exec"}
+
+
+class Codex(Host):
+ name = "codex"
+ can_stop_mid_turn = False
+
+ def turn_key(self, p):
+ return str(p.get("turn_id") or "")
+
+ def tool_call(self, p):
+ name, args, text, code = super().tool_call(p)
+ low = name.lower()
+ if low in _CODEX_SHELL:
+ cmd = args.get("command", args.get("cmd", ""))
+ if isinstance(cmd, list):
+ cmd = shlex.join(str(c) for c in cmd)
+ return "bash", {"command": str(cmd or "")}, text, code
+ if low == "apply_patch":
+ patch = next((v for v in (args.get("input"), args.get("patch"), args.get("command"))
+ if isinstance(v, str)), "")
+ paths = _PATCH_FILE_RE.findall(patch)
+ return "apply_patch", {"path": paths[0] if paths else "", "paths": paths}, text, code
+ return name, args, text, code
+
+ def mechanism_trace(self, transcript_path, workdir):
+ out = os.path.join(workdir, "codex_exec_trace.jsonl")
+ n = transcripts.codex_rollout_to_exec(transcript_path, out)
+ return out if n is not None else None
+
+ def session_usage(self, transcript_path):
+ return transcripts.codex_session_usage(transcript_path)
+
+
+class Pi(Host):
+ """Pi (earendil-works/pi), through the extension in `plugin/pi/`.
+
+ Pi has no hook protocol of its own -- extensions are TypeScript functions on its event
+ bus -- so the extension speaks THIS one on Pi's behalf: it calls the same dispatcher with
+ a Claude-shaped payload per event and applies the answer with Pi's own levers (a system
+ prompt section, a steering message, `ctx.abort()`, a UI notice). What it sends:
+
+ * `prompt_id`, which it numbers itself, for the turn identity Pi does not stamp;
+ * `tool_response = {content, isError, exit_code}` -- Pi's bash result carries a real
+ exit code in its structured content, which the ended-green proxy uses;
+ * Pi's mode in `TE_PI_MODE` on the hook's environment: print / json / rpc are headless.
+ """
+ name = "pi"
+ can_stop_mid_turn = True
+
+ def turn_key(self, p):
+ return str(p.get("prompt_id") or "")
+
+ def stop(self, reason):
+ return {"continue": False, "stopReason": reason}
+
+ def headless(self, env):
+ return str(env.get("TE_PI_MODE", "")).strip() in ("print", "json", "rpc")
+
+ def mechanism_trace(self, transcript_path, workdir):
+ out = os.path.join(workdir, "pi_trace.jsonl")
+ n = transcripts.pi_session_to_claude(transcript_path, out)
+ return out if n is not None else None
+
+ def session_usage(self, transcript_path):
+ return transcripts.pi_session_usage(transcript_path)
+
+
+def copilot_home(env=None):
+ env = os.environ if env is None else env
+ return env.get("COPILOT_HOME") or os.path.join(os.path.expanduser("~"), ".copilot")
+
+
+class Copilot(Host):
+ """The GitHub Copilot CLI, through `plugin/.github/plugin/plugin.json` and its hook file."""
+ name = "copilot"
+ can_stop_mid_turn = False
+
+ def transcript(self, p):
+ # Only Stop carries `transcript_path`; the log's location is fixed by the session id.
+ sid = self.session_id(p)
+ return p.get("transcript_path") or (
+ os.path.join(copilot_home(), "session-state", sid, "events.jsonl") if sid else "")
+
+ def cwd(self, p, env):
+ return p.get("cwd") or env.get("COPILOT_PROJECT_DIR") or os.getcwd()
+
+ def context(self, event, text):
+ return {"additionalContext": text}
+
+ def notice(self, text):
+ return {PROGRESS: [text]}
+
+ def tool_call(self, p):
+ args = p.get("tool_input") if isinstance(p.get("tool_input"), dict) else {}
+ resp = p.get("tool_result")
+ if resp is None and p.get("error") is not None: # PostToolUseFailure
+ resp = {"error": str(p.get("error")), "result_type": "failure"}
+ return str(p.get("tool_name") or ""), args, response_text(resp), _exit_code(resp)
+
+ def session_usage(self, transcript_path):
+ return transcripts.copilot_session_usage(transcript_path)
+
+ def settle(self, transcript_path, wait_s=15.0, fresh_s=60.0):
+ # Copilot writes `session.shutdown` -- the event that carries usage -- just AFTER the
+ # SessionEnd hook returns (observed order: hook.end, then session.shutdown), so the
+ # detached fold can arrive first. Wait for it while the log is still fresh; an old
+ # log without one belongs to a killed session and will never get one.
+ deadline = time.time() + wait_s
+ while time.time() < deadline:
+ try:
+ if time.time() - os.path.getmtime(transcript_path) > fresh_s:
+ return
+ with open(transcript_path, "rb") as fh:
+ fh.seek(max(0, os.path.getsize(transcript_path) - 65536))
+ if b'"session.shutdown"' in fh.read():
+ return
+ except OSError:
+ return
+ time.sleep(0.25)
+
+
+HOSTS = {"claude": ClaudeCode, "codex": Codex, "copilot": Copilot, "pi": Pi}
+
+
+def default_host(env=None):
+ """The host a `te` command run without `--host` is about: `TE_HOST`, else the agent whose
+ shell it runs in (Claude Code marks its shell `CLAUDECODE=1`, Copilot `COPILOT_CLI=1`),
+ else Claude Code."""
+ env = os.environ if env is None else env
+ if env.get("TE_HOST") in HOSTS:
+ return env["TE_HOST"]
+ if env.get("CLAUDECODE") == "1":
+ return "claude"
+ if env.get("COPILOT_CLI") == "1":
+ return "copilot"
+ return "claude"
+
+
+def detect(payload, env=None, hint=None):
+ """The adapter for this hook call: an explicit `hint`, else inferred from the payload."""
+ if hint in HOSTS:
+ return HOSTS[hint]()
+ if "turn_id" in (payload or {}):
+ return Codex()
+ tp = str((payload or {}).get("transcript_path") or "")
+ if os.path.basename(tp).startswith("rollout-") or f"{os.sep}.codex{os.sep}" in tp:
+ return Codex()
+ if f"{os.sep}session-state{os.sep}" in tp and tp.endswith("events.jsonl"):
+ return Copilot()
+ # Claude Code always sends a transcript path; Copilot sends none before Stop, and marks its
+ # hook processes. (Hooks inherit the parent's environment, so the marker alone would also
+ # tag a Claude Code session started from inside Copilot -- hence the transcript test first.)
+ if not tp and (env or {}).get("COPILOT_PLUGIN_ROOT"):
+ return Copilot()
+ return ClaudeCode()
+
+
+def scratch_dir(prefix="te-"):
+ return tempfile.mkdtemp(prefix=prefix)
diff --git a/te_live/judge.py b/te_live/judge.py
new file mode 100644
index 0000000..93a3fe9
--- /dev/null
+++ b/te_live/judge.py
@@ -0,0 +1,107 @@
+"""The paid spiral judge, off the hook's critical path.
+
+With `paid` on, the guard's gate can ask an LLM whether a turn is stuck and, when it is, have a
+second call write the redirect. Both are side calls through the agent's own CLI (`sidecall`),
+and run inside a PostToolUse hook they held the agent for ~22 s per nudge in the user-simulation
+pilot. So the hook only STARTS a job here and returns: `AsyncJudge.start` writes the request and
+spawns `te judge` detached; the job runs the judge, and the nudge writer only when the verdict is
+stuck, then leaves the verdict -- with the redirect already written -- for the next PostToolUse,
+whose `AsyncJudge.poll` hands it to the guard. The cascade the guard runs (`assess_abort`) is the
+vendored one: while a verdict is pending the judge answers nothing, which it already reads as
+"throttled", so the decision rules are unchanged; only their latency moved off the agent.
+
+What a job spends is charged to the ledger the moment it is spent, as its own `overhead` record
+for the session: a verdict that lands after its turn -- or its session -- has ended was still paid
+for, and must not drop out of the account.
+"""
+import os
+import subprocess
+import sys
+import time
+
+from . import sidecall
+
+REQUEST = "judge/{sid}.request.json"
+RESULT = "judge/{sid}.result.json"
+
+
+class AsyncJudge:
+ """The judge as `TurnGuard` sees it: `start` a job for a digest, `poll` for its verdict."""
+
+ def __init__(self, ctx, session_id, threshold):
+ self.ctx = ctx
+ self.sid = session_id
+ self.threshold = float(threshold or 0)
+
+ def start(self, guard, digest):
+ job = f"{guard.turn_key}:{guard.segment}:{guard.judge_calls}:{time.time():.3f}"
+ self.ctx.store.write(REQUEST.format(sid=self.sid), {
+ "id": job, "session_id": self.sid, "turn_key": guard.turn_key,
+ "segment": guard.segment, "digest": digest, "threshold": self.threshold,
+ "at": time.time()})
+ spawn_job(self.ctx, self.sid, job)
+ return job
+
+ def poll(self, guard):
+ """(found, verdict) for the guard's pending job; verdict is None when the call failed."""
+ pending = guard.judge_pending or {}
+ if not pending.get("id"):
+ return False, None
+ res = self.ctx.store.read(RESULT.format(sid=self.sid)) or {}
+ if res.get("id") != pending["id"]:
+ return False, None
+ return True, res.get("verdict") or None
+
+
+def run_job(ctx, session_id, job_id):
+ """The detached half: judge the digest, write the redirect if stuck, charge the ledger."""
+ from tokeneconomizer.vendor import reflect
+ req = ctx.store.read(REQUEST.format(sid=session_id)) or {}
+ if req.get("id") != job_id: # superseded by a newer request
+ return None
+ runner = sidecall.runner_for(ctx.host.name, ctx.config)
+ verdict, written = None, None
+ if runner is not None:
+ try:
+ verdict = reflect.llm_stuck_judge(req.get("digest") or "", runner, model="agent")
+ except Exception:
+ verdict = None
+ conf = float((verdict or {}).get("confidence") or 0)
+ if verdict and verdict.get("stuck") and conf >= float(req.get("threshold") or 0):
+ try:
+ written = reflect.llm_nudge_writer(req.get("digest") or "", runner,
+ model="agent")
+ except Exception:
+ written = None
+ if written and (written.get("text") or "").strip():
+ verdict["_nudge"] = written["text"].strip()
+ spend = {"spiral_judge_tokens": float((verdict or {}).get("_tokens") or 0),
+ "spiral_nudge_writer_tokens": float((written or {}).get("_tokens") or 0)}
+ if any(spend.values()):
+ # Charged before the verdict is published: a fold that runs the instant the verdict
+ # lands has already got the record.
+ ctx.store.append("ledger.jsonl", {
+ "kind": "overhead", "session_id": session_id, "host": ctx.host.name,
+ "job": job_id, "at": time.time(), "overhead": spend})
+ ctx.store.write(RESULT.format(sid=session_id),
+ {"id": job_id, "verdict": verdict, "at": time.time()})
+ return verdict
+
+
+def spawn_job(ctx, session_id, job_id):
+ """Run a judge job detached (inline when TE_JUDGE_SYNC=1, for tests)."""
+ if ctx.env.get("TE_JUDGE_SYNC") == "1":
+ return run_job(ctx, session_id, job_id)
+ core = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ env = dict(ctx.env)
+ env["PYTHONPATH"] = core + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
+ cmd = [sys.executable, "-m", "te_live.cli", "judge", "--host", ctx.host.name,
+ "--cwd", ctx.repo.root, "--session", session_id, "--job", job_id]
+ ctx.store.ensure()
+ log = open(ctx.store.path("judge.log"), "a")
+ try:
+ subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=log, stderr=log,
+ cwd=ctx.repo.root, env=env, start_new_session=True)
+ finally:
+ log.close()
+ return True
diff --git a/te_live/paths.py b/te_live/paths.py
new file mode 100644
index 0000000..e85740c
--- /dev/null
+++ b/te_live/paths.py
@@ -0,0 +1,191 @@
+"""Where live state lives, and which repository a session belongs to.
+
+State is keyed by REPOSITORY, not by directory: two worktrees of one clone, or a session started
+in a subdirectory, are the same stream of work and should share what was learned about it. The
+key is derived from the `origin` remote when there is one (so a re-clone keeps its history) and
+from the real path of the checkout otherwise. Reading the remote parses `.git/config` directly --
+a hook that shelled out to `git` would pay a subprocess on every tool call.
+
+The state root is `$TOKENECONOMIZER_HOME`, else `~/.tokeneconomizer`. It is deliberately not a
+host's plugin data directory: the CLI runs from the agent's Bash tool, where those variables are
+not exported, and one root lets Claude Code and Codex sessions on the same repository sit side
+by side (each host keeps its own subdirectory, because a turn envelope learned on one agent says
+nothing about another's turn scale).
+"""
+import hashlib
+import os
+import re
+from dataclasses import dataclass
+
+HOME_ENV = "TOKENECONOMIZER_HOME"
+
+
+def state_root(env=None):
+ """Absolute path of the live-state root."""
+ env = os.environ if env is None else env
+ root = env.get(HOME_ENV) or os.path.join(os.path.expanduser("~"), ".tokeneconomizer")
+ return os.path.abspath(root)
+
+
+def secure_root(env=None):
+ """Keep the state root private to its owner, as the agents keep their own session logs.
+
+ What the plugin stores -- repository paths, the commands sessions ran, costs, error
+ tracebacks -- is derived from those logs, and `bin/te` in it is what the slash commands run.
+ New files are created private (`te_bootstrap` sets umask 077); a root that an earlier
+ version, or another entry point, created with looser permissions is tightened here."""
+ root = state_root(env)
+ try:
+ st = os.stat(root)
+ if st.st_mode & 0o077 and st.st_uid == os.getuid():
+ os.chmod(root, 0o700)
+ except (OSError, AttributeError):
+ pass
+
+
+def launcher_path(env=None):
+ return os.path.join(state_root(env), "bin", "te")
+
+
+def ensure_launcher(env=None):
+ """Keep `<state root>/bin/te` pointing at this plugin's `te`. Returns its path, or None.
+
+ A slash command has to name the CLI's path. Claude Code substitutes `${CLAUDE_PLUGIN_ROOT}`
+ into a command's text; the Copilot CLI substitutes nothing but `$ARGUMENTS` (1.0.83:
+ `${CLAUDE_PLUGIN_ROOT}`, `${PLUGIN_ROOT}`, `${COPILOT_PLUGIN_ROOT}`, `{{plugin_root}}` all
+ arrive literally), and its shell exports none of them. So every hook session start writes
+ this one-line launcher, the commands fall back to it, and a person on any host has a stable
+ `te` to run. The plugin's location comes from the bootstrap (`TE_PLUGIN_ROOT`); the file is
+ rewritten only when it changes, as an installed copy's path does across updates."""
+ import shlex
+ env = os.environ if env is None else env
+ plugin = env.get("TE_PLUGIN_ROOT")
+ target = os.path.join(plugin, "scripts", "te") if plugin else ""
+ if not target or not os.access(target, os.X_OK):
+ return None
+ path = launcher_path(env)
+ body = ("#!/bin/sh\n# Written by the TokenEconomizer plugin at session start: its `te` "
+ "command line.\nexec %s \"$@\"\n" % shlex.quote(target))
+ try:
+ with open(path, encoding="utf-8") as fh:
+ if fh.read() == body:
+ return path
+ except OSError:
+ pass
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ tmp = "%s.%d.tmp" % (path, os.getpid())
+ with open(tmp, "w", encoding="utf-8") as fh:
+ fh.write(body)
+ os.chmod(tmp, 0o700)
+ os.replace(tmp, path)
+ return path
+
+
+def find_repo_root(start, env=None):
+ """The nearest directory at or above `start` that holds a `.git` entry, else None.
+
+ `.git` may be a directory (a clone) or a file (a worktree or submodule), so existence is the
+ test, not `isdir`. The walk honours `GIT_CEILING_DIRECTORIES` as git does: it never climbs
+ into a listed directory, so a stray `.git` in a shared parent (`/tmp`, a versioned home
+ directory) need not swallow every session beneath it."""
+ env = os.environ if env is None else env
+ ceilings = {os.path.abspath(c) for c in
+ (env.get("GIT_CEILING_DIRECTORIES") or "").split(os.pathsep) if c}
+ cur = os.path.abspath(start or os.getcwd())
+ while True:
+ if os.path.exists(os.path.join(cur, ".git")):
+ return cur
+ parent = os.path.dirname(cur)
+ if parent == cur or parent in ceilings:
+ return None
+ cur = parent
+
+
+def _git_common_dir(root):
+ """The git directory that holds `config` for a checkout, following worktree indirection."""
+ dot = os.path.join(root, ".git")
+ if os.path.isdir(dot):
+ return dot
+ try:
+ with open(dot, encoding="utf-8") as fh:
+ line = fh.read().strip()
+ except OSError:
+ return None
+ if not line.startswith("gitdir:"):
+ return None
+ gitdir = line.split(":", 1)[1].strip()
+ if not os.path.isabs(gitdir):
+ gitdir = os.path.normpath(os.path.join(root, gitdir))
+ # A worktree's gitdir is <main>/.git/worktrees/<name>; `commondir` points back at <main>/.git.
+ try:
+ with open(os.path.join(gitdir, "commondir"), encoding="utf-8") as fh:
+ common = fh.read().strip()
+ return os.path.normpath(os.path.join(gitdir, common))
+ except OSError:
+ return gitdir
+
+
+_REMOTE_RE = re.compile(r'^\s*\[remote\s+"([^"]+)"\]\s*$')
+_URL_RE = re.compile(r"^\s*url\s*=\s*(\S+)\s*$")
+
+
+def origin_url(root):
+ """The `origin` remote URL (or the first remote's), read from the git config; None if none."""
+ gitdir = _git_common_dir(root) if root else None
+ if not gitdir:
+ return None
+ try:
+ lines = open(os.path.join(gitdir, "config"), encoding="utf-8", errors="replace").read()
+ except OSError:
+ return None
+ remotes, current = {}, None
+ for line in lines.splitlines():
+ m = _REMOTE_RE.match(line)
+ if m:
+ current = m.group(1)
+ continue
+ if line.lstrip().startswith("["):
+ current = None
+ continue
+ m = _URL_RE.match(line)
+ if current and m and current not in remotes:
+ remotes[current] = m.group(1)
+ return remotes.get("origin") or (next(iter(remotes.values())) if remotes else None)
+
+
+def normalize_remote(url):
+ """`[email protected]:Owner/Repo.git` and `https://github.com/owner/repo` -> one spelling."""
+ u = (url or "").strip()
+ u = re.sub(r"^[a-z+]+://", "", u) # scheme
+ u = re.sub(r"^[^@/]+@", "", u) # user@
+ u = u.replace(":", "/", 1) if re.match(r"^[^/]+:[^/\d]", u) else u # scp-style host:path
+ u = re.sub(r"\.git$", "", u.rstrip("/"))
+ return u.lower()
+
+
+def _slug(text):
+ return re.sub(r"[^a-z0-9]+", "-", (text or "repo").lower()).strip("-")[:40] or "repo"
+
+
+@dataclass(frozen=True)
+class RepoId:
+ root: str # the checkout this session runs in
+ key: str # stable directory name for its state
+ label: str # human name, used in rendered manuals ("owner/repo" or the basename)
+ remote: str # normalized remote, "" when the checkout has none
+
+
+def repo_identity(cwd, env=None):
+ """Identify the repository a session in `cwd` belongs to."""
+ root = find_repo_root(cwd, env) or os.path.abspath(cwd or os.getcwd())
+ remote = normalize_remote(origin_url(root)) if os.path.exists(os.path.join(root, ".git")) else ""
+ canon = remote or os.path.realpath(root)
+ label = "/".join(remote.split("/")[-2:]) if remote else os.path.basename(root.rstrip("/"))
+ digest = hashlib.sha1(canon.encode("utf-8")).hexdigest()[:10]
+ return RepoId(root=root, key=f"{_slug(os.path.basename(root.rstrip('/')))}-{digest}",
+ label=label or "repo", remote=remote)
+
+
+def repo_state_dir(repo, host, env=None):
+ """`<root>/repos/<repo-key>/<host>` -- one host's state for one repository."""
+ return os.path.join(state_root(env), "repos", repo.key, host)
diff --git a/te_live/probe.py b/te_live/probe.py
new file mode 100644
index 0000000..d37395c
--- /dev/null
+++ b/te_live/probe.py
@@ -0,0 +1,240 @@
+"""`te --host copilot probe`: the plugin, end to end, in real Copilot sessions on your own model.
+
+What `docs/PLUGIN.md` §3 verified, run by anyone on their own setup -- the Copilot subscription,
+or a BYOK provider (`COPILOT_PROVIDER_*`) where an organization's policy refuses the
+subscription's models. It builds a throwaway repository with one environment trap (bare `pytest`
+fails, `python -m pytest` works), records the tool catalog, and runs the same short prompt in
+real sessions with the plugin loaded, under a state root of its own -- your learned state is
+never touched. The trap is the commonest one in Python repositories: the project is set up for
+pytest (`[tool.pytest.ini_options]`), but pytest is not installed, so bare `pytest` and
+`python -m pytest` both fail while the tests, written with `unittest`, pass under
+`python -m unittest`. (A first version made only bare `pytest` fail; Haiku never typed it, so
+nothing was learned.) The repository is created inside a folder Copilot trusts when there is
+one, because Copilot ignores an untrusted folder's settings, the deny list's file included. Then it applies the deny list it proposed, runs one more session, restores it,
+and reports:
+
+ * per session: whether the learned fact was in the context, the first test command the
+ model ran, how many test commands failed, the request's tool-definition tokens;
+ * whether the sessions WITH the fact avoided the trap that the sessions without it hit;
+ * the tool-definition tokens before and after the deny list.
+
+It makes real model calls (default 7 short sessions; ~25-60k tokens each, mostly cached) and
+leaves those sessions in Copilot's own history. Nothing else outside a temporary directory is
+written; `--keep` keeps that directory for inspection.
+"""
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+
+from . import hosts, sidecall, transcripts, untool
+from .context import LiveContext
+
+PROMPT = ("Run this repository's test suite and report how many tests pass. Do not install "
+ "anything. Keep the answer to one line.")
+
+# pytest must be missing for every interpreter the session might use, whatever is installed:
+# the trap's `python`/`python3` put this stub first on the module path, so `python -m pytest`
+# fails as it does where pytest is not installed. Nothing in the repository gives it away.
+_PYTEST_STUB = 'raise ModuleNotFoundError("No module named \'pytest\'", name="pytest")\n'
+
+
+# A test-runner invocation, and the forms that cannot work in the probe's repository. Counted
+# by attempt, not exit code: models pipe test runs into `| tail`, which hides the failure (two
+# of the first real Haiku sessions did exactly that).
+_RUNNER_RE = re.compile(r"(?:^|[\s;&|(])(?:py\.?test|python[\d.]*\s+-m\s+(?:pytest|unittest))\b")
+_TRAP_RE = re.compile(r"(?:^|[\s;&|(])(?:py\.?test|python[\d.]*\s+-m\s+pytest)\b")
+
+
+def test_commands(cmds):
+ """(runner commands in order, how many of them tried pytest) for a session's shell commands."""
+ runs = [c for c in cmds if _RUNNER_RE.search(c)]
+ return runs, sum(1 for c in runs if _TRAP_RE.search(c))
+
+
+def _write(path, text, mode=0o644):
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(text)
+ os.chmod(path, mode)
+
+
+def make_repo(root):
+ """The throwaway repository and the trap directory put first on the sessions' PATH."""
+ repo, trap = os.path.join(root, "repo"), os.path.join(root, "trap")
+ _write(os.path.join(repo, "calc.py"), "def add(a, b):\n return a + b\n")
+ _write(os.path.join(repo, "tests", "__init__.py"), "")
+ _write(os.path.join(repo, "tests", "test_calc.py"),
+ "import unittest\n\nfrom calc import add\n\n\nclass TestAdd(unittest.TestCase):\n"
+ " def test_add(self):\n self.assertEqual(add(2, 2), 4)\n")
+ _write(os.path.join(repo, "pyproject.toml"),
+ '[project]\nname = "calc"\n\n[tool.pytest.ini_options]\ntestpaths = ["tests"]\n')
+ _write(os.path.join(repo, ".gitignore"), "__pycache__/\n")
+ stub = os.path.join(trap, "pymods")
+ _write(os.path.join(stub, "pytest", "__init__.py"), _PYTEST_STUB)
+ _write(os.path.join(trap, "pytest"),
+ '#!/bin/sh\necho "bash: pytest: command not found" >&2\nexit 127\n', 0o755)
+ for name in ("python", "python3"):
+ _write(os.path.join(trap, name), f'#!/bin/sh\nPYTHONPATH="{stub}${{PYTHONPATH:+:$PYTHONPATH}}" '
+ f'exec "{sys.executable}" "$@"\n', 0o755)
+ git = ["git", "-c", "user.name=te-probe", "-c", "user.email=te-probe@localhost",
+ "-c", "commit.gpgsign=false"]
+ subprocess.run(["git", "init", "-q", repo], check=True)
+ subprocess.run(git + ["-C", repo, "add", "-A"], check=True)
+ subprocess.run(git + ["-C", repo, "commit", "-qm", "probe"], check=True)
+ return repo, trap
+
+
+def _plugin_root():
+ root = os.environ.get("TE_PLUGIN_ROOT")
+ if root and os.path.isfile(os.path.join(root, "hooks", "copilot-hooks.json")):
+ return root
+ here = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ for cand in (os.path.join(here, "plugin"), os.path.dirname(here)):
+ if os.path.isfile(os.path.join(cand, "hooks", "copilot-hooks.json")):
+ return cand
+ raise RuntimeError("cannot find the plugin directory (set TE_PLUGIN_ROOT)")
+
+
+def _session_env(state, trap):
+ env = {k: v for k, v in os.environ.items()
+ if k not in ("CLAUDECODE", "CLAUDE_CODE_SESSION_ID", "CLAUDE_PLUGIN_ROOT",
+ "TE_SIDECALL", "TE_DISABLE", "TERM_PROGRAM")}
+ # An installed copy of the plugin does not run beside the --plugin-dir one: Copilot loads
+ # one plugin per name (verified: one SessionStart hook with both present).
+ env.update(TOKENECONOMIZER_HOME=state, PATH=trap + os.pathsep + env.get("PATH", ""))
+ return env
+
+
+def run_session(repo, env, plugin, model, cli="copilot", timeout=600):
+ cmd = [cli, "-p", PROMPT, "--plugin-dir", plugin, "--allow-all-tools", "--no-auto-update",
+ "--output-format", "json"]
+ if model:
+ cmd += ["--model", model]
+ out = subprocess.run(cmd, cwd=repo, env=env, capture_output=True, text=True,
+ timeout=timeout, stdin=subprocess.DEVNULL)
+ sid, answer, error = None, "", (out.stderr or "").strip().splitlines()[-1:] or [""]
+ for line in (out.stdout or "").splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "result":
+ sid = ev.get("sessionId")
+ elif ev.get("type") == "assistant.message":
+ answer = (ev.get("data") or {}).get("content") or answer
+ return sid, answer.strip(), error[0]
+
+
+def inspect_session(sid):
+ """What one finished session did, from Copilot's own log."""
+ log = os.path.join(hosts.copilot_home(), "session-state", sid, "events.jsonl")
+ hosts.Copilot().settle(log)
+ injected = False
+ cmds = [(args or {}).get("command") or ""
+ for (name, args, _, _) in [c for t in transcripts.copilot_turns(log) for c in t]
+ if name == "bash"]
+ tests, tried_pytest = test_commands(cmds)
+ for line in open(log, encoding="utf-8", errors="replace"):
+ if '"hook.end"' in line and '"sessionStart"' in line:
+ out = (json.loads(line).get("data") or {}).get("output") or {}
+ injected = injected or "fails here" in (out.get("additionalContext") or "")
+ usage = transcripts.copilot_session_usage(log)
+ first = tests[0] if tests else None
+ if first:
+ first = re.sub(r"^cd \S+ && ", "", first)[:60]
+ return {"session": sid, "injected": injected, "tests": tests, "first_test": first,
+ "tried_pytest": tried_pytest,
+ "tool_definition_tokens": usage.get("tool_definitions_tokens"),
+ "input_per_request": (round(sum(usage.get(k) or 0 for k in (
+ "input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens"))
+ / usage["requests"]) if usage.get("requests") else None)}
+
+
+def _wait_folded(ctx, sid, timeout=90):
+ deadline = time.time() + timeout
+ while time.time() < deadline:
+ if sid in (ctx.store.read("folded.json") or {}):
+ return True
+ time.sleep(1)
+ return False
+
+
+def _line(r):
+ return (f" {r['label']:>10}: fact in context={'yes' if r['injected'] else 'no '} "
+ f"first test command: {r['first_test'] or '-'} pytest attempts: "
+ f"{r['tried_pytest']} tool definitions: {r['tool_definition_tokens']} tokens")
+
+
+def run(sessions=6, model=None, keep=False, cli="copilot", out=print):
+ if not shutil.which(cli):
+ raise RuntimeError(f"`{cli}` is not on PATH")
+ model = sidecall.copilot_model(model or "")
+ plugin = _plugin_root()
+ base = next((f for f in (untool.copilot_trusted_folders() or [])
+ if os.path.isdir(f) and os.access(f, os.W_OK)), None)
+ root = tempfile.mkdtemp(prefix=".te-probe-copilot-", dir=base)
+ state = os.path.join(root, "state")
+ try:
+ repo, trap = make_repo(root)
+ env = _session_env(state, trap)
+ route = ("BYOK " + (os.environ.get("COPILOT_PROVIDER_TYPE") or "openai")
+ if os.environ.get("COPILOT_PROVIDER_BASE_URL") else "Copilot subscription")
+ out(f"TokenEconomizer probe -- Copilot ({route}), model {model or '(Copilot default)'}, "
+ f"plugin {plugin}")
+ if not base:
+ out(" note: Copilot trusts no folder here, so the deny list cannot take effect "
+ "in the probe's repository (Copilot ignores an untrusted folder's settings)")
+ _, tools = untool.refresh_catalog(state, host="copilot")
+ out(f" catalog: {len(tools)} tools recorded (local capture, no model call)")
+ ctx = LiveContext.build(hosts.Copilot(), repo, env)
+ results = []
+ for i in range(1, sessions + 1):
+ sid, answer, err = run_session(repo, env, plugin, model, cli=cli)
+ if not sid:
+ out(f" session {i}: no session ran ({err or 'no output'})")
+ return 1
+ _wait_folded(ctx, sid)
+ r = dict(inspect_session(sid), label=f"session {i}", answer=answer)
+ results.append(r)
+ out(_line(r) + f" answer: {answer[:40]}")
+ unt = ctx.store.read("untool.json") or {}
+ proposal = unt.get("proposal") or []
+ lean = None
+ if proposal:
+ untool.apply(repo, proposal, ctx.store, host="copilot")
+ try:
+ sid, answer, err = run_session(repo, env, plugin, model, cli=cli)
+ if sid:
+ _wait_folded(ctx, sid)
+ lean = dict(inspect_session(sid), label="deny list", answer=answer)
+ out(_line(lean) + f" answer: {answer[:40]}")
+ finally:
+ untool.restore(repo, ctx.store, host="copilot")
+ # the verdict
+ everything = results + ([lean] if lean else [])
+ without = [r for r in everything if not r["injected"] and r["tests"]]
+ with_ = [r for r in everything if r["injected"] and r["tests"]]
+ out("")
+ out(f" without the fact: {sum(1 for r in without if r['tried_pytest'])}/{len(without)} "
+ f"sessions tried pytest, which is not installed; with it: "
+ f"{sum(1 for r in with_ if r['tried_pytest'])}/"
+ f"{len(with_)}" + ("" if with_ else " (the fact never cleared its firing gate; "
+ "run more --sessions)"))
+ full = [r["tool_definition_tokens"] for r in results if r["tool_definition_tokens"]]
+ if proposal and lean and full and lean["tool_definition_tokens"]:
+ out(f" deny list {', '.join(proposal)}: tool definitions {full[-1]} -> "
+ f"{lean['tool_definition_tokens']} tokens per request")
+ elif not proposal:
+ out(" deny list: nothing proposed (a server is proposed once none of its tools "
+ "was used in `untool_evidence` sessions)")
+ return 0
+ finally:
+ if keep:
+ out(f" kept: {root}")
+ else:
+ shutil.rmtree(root, ignore_errors=True)
diff --git a/te_live/sidecall.py b/te_live/sidecall.py
new file mode 100644
index 0000000..d837ff0
--- /dev/null
+++ b/te_live/sidecall.py
@@ -0,0 +1,159 @@
+"""Paid side calls -- the proposer, the stuck judge, the nudge writer -- on the user's own agent.
+
+The mechanisms call an optimizer as `runner(prompt, model) -> (text, effective_tokens)`
+(`runner.AgentRunner.optimizer_call`). In the harness that call went to the agent's model through
+the harness's own routing. Here it goes through the host CLI the user is already signed in to,
+so no extra API key is needed and the call is billed where the session is:
+
+ * Claude Code: `claude -p --output-format json --tools "" --max-turns 1`, on `paid_model`
+ (default `haiku`). This is a deliberate departure from the experiments, which ran every
+ optimizer call on the agent's own model so overhead could not look cheaper than the arm it
+ was charged to; in a user's session the cheapest capable model is the right default, and the
+ ledger records the real cost either way.
+ * Codex: `codex exec --json --disable hooks --ephemeral -s read-only`. Each such call carries
+ Codex's full base prompt (~25k tokens, mostly cached) on the user's plan quota, which is why
+ paid mechanisms default off and the Codex default injection is the free layer.
+ * Copilot CLI: `copilot -p` as a pure text completion, as `te_harness.copilot.optimizer_call`
+ runs it: a neutral cwd, no custom instructions, no built-in MCP server, every tool denied
+ and excluded. Its usage is read from the call's own session log (`result.sessionId` ->
+ `session-state/<id>/events.jsonl`), the same `session.shutdown` the ledger prices sessions
+ from. The model is `copilot_model`'s choice: under BYOK, Copilot has no router to fall back
+ on, and `COPILOT_MODEL=auto` -- a common setting, the subscription's router -- made the call
+ answer nothing (1.0.83, an Anthropic provider).
+
+Every side call runs with `TE_SIDECALL=1`, which the plugin's hook entry point treats as "do
+nothing": without it the nested agent would run this plugin's hooks and recurse. Nothing here runs
+unless `paid` is enabled.
+"""
+import json
+import os
+import subprocess
+import tempfile
+
+from . import accounting
+
+TIMEOUT_S = 180
+
+
+def _env():
+ return dict(os.environ, TE_SIDECALL="1")
+
+
+def claude_call(prompt, model="haiku", cli="claude", timeout=TIMEOUT_S):
+ cmd = [cli, "-p", "--output-format", "json", "--max-turns", "1", "--tools", "",
+ "--model", model or "haiku"]
+ with tempfile.TemporaryDirectory(prefix="te-side-") as cwd:
+ out = subprocess.run(cmd, input=prompt, capture_output=True, text=True,
+ timeout=timeout, env=_env(), cwd=cwd)
+ data = json.loads(out.stdout or "{}")
+ usage = dict(data.get("usage") or {})
+ usage["model"] = next(iter(data.get("modelUsage") or {}), None) or (model or "claude")
+ eff, _ = accounting.effective(usage)
+ return data.get("result") or "", eff
+
+
+def codex_call(prompt, model="", cli="codex", timeout=TIMEOUT_S):
+ cmd = [cli, "exec", "--json", "--disable", "hooks", "--ephemeral", "-s", "read-only",
+ "--skip-git-repo-check"]
+ if model:
+ cmd += ["-m", model]
+ cmd.append(prompt)
+ with tempfile.TemporaryDirectory(prefix="te-side-") as cwd:
+ out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout,
+ env=_env(), cwd=cwd)
+ text, usage = "", None
+ for line in (out.stdout or "").splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "item.completed" and (ev.get("item") or {}).get("type") == "agent_message":
+ text = ev["item"].get("text") or text
+ elif ev.get("type") == "turn.completed":
+ usage = ev.get("usage") or usage
+ eff = None
+ if usage:
+ inp, cached = int(usage.get("input_tokens") or 0), int(usage.get("cached_input_tokens") or 0)
+ cw = int(usage.get("cache_write_input_tokens") or 0)
+ eff, _ = accounting.effective({
+ "input_tokens": max(0, inp - cached - cw), "cache_read_input_tokens": cached,
+ "cache_creation_input_tokens": cw,
+ "output_tokens": int(usage.get("output_tokens") or 0)
+ + int(usage.get("reasoning_output_tokens") or 0),
+ "model": model or "codex"})
+ return text, eff
+
+
+# Everything the Copilot CLI offers (1.0.83 names), so the completion is offered nothing; a name
+# the installed CLI does not know is ignored.
+COPILOT_TOOLS = ("bash", "read_bash", "write_bash", "stop_bash", "list_bash", "view", "grep",
+ "rg", "glob", "edit", "create", "str_replace", "apply_patch", "web_fetch",
+ "web_search", "ask_user", "session_store_sql", "sql", "task", "skill",
+ "list_agents", "read_agent", "write_agent", "fetch_copilot_cli_documentation")
+
+
+# The cheapest capable model by BYOK provider type, in the ids that provider expects.
+BYOK_DEFAULT_MODEL = {"anthropic": "claude-haiku-4-5"}
+
+
+def copilot_model(paid_model="", env=None):
+ """The model a Copilot side call names: `paid_model`; else a concrete model the person's
+ environment names; else, under BYOK, the provider's cheap default; else "" (Copilot's own
+ default, which on the subscription is its router)."""
+ env = os.environ if env is None else env
+ if paid_model:
+ return paid_model
+ for k in ("COPILOT_PROVIDER_MODEL_ID", "COPILOT_MODEL"):
+ v = (env.get(k) or "").strip()
+ if v and v.lower() != "auto":
+ return v
+ base = env.get("COPILOT_PROVIDER_BASE_URL") or ""
+ if "openrouter.ai" in base: # OpenRouter's own ids, on either wire
+ return "anthropic/claude-haiku-4.5"
+ if base:
+ return BYOK_DEFAULT_MODEL.get((env.get("COPILOT_PROVIDER_TYPE") or "openai").lower(), "")
+ return ""
+
+
+def copilot_call(prompt, model="", cli="copilot", timeout=TIMEOUT_S):
+ from . import hosts, transcripts
+ model = copilot_model(model)
+ cmd = [cli, "-p", prompt, "--output-format", "json", "--no-color", "--no-auto-update",
+ "--no-custom-instructions", "--disable-builtin-mcps"]
+ for t in COPILOT_TOOLS:
+ cmd += ["--deny-tool", t, "--excluded-tools", t]
+ if model:
+ cmd += ["--model", model]
+ with tempfile.TemporaryDirectory(prefix="te-side-") as cwd:
+ out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout,
+ env=_env(), cwd=cwd, stdin=subprocess.DEVNULL)
+ text, sid = "", None
+ for line in (out.stdout or "").splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "assistant.message":
+ text = (ev.get("data") or {}).get("content") or text
+ elif ev.get("type") == "result":
+ sid = ev.get("sessionId") or sid
+ eff = None
+ if sid:
+ log = os.path.join(hosts.copilot_home(), "session-state", sid, "events.jsonl")
+ hosts.Copilot().settle(log, wait_s=5.0)
+ usage = transcripts.copilot_session_usage(log)
+ usage["model"] = usage.get("model") or model or "copilot"
+ eff, _ = accounting.effective(usage)
+ return text, eff
+
+
+def runner_for(host_name, config):
+ """The optimizer callable for this host, or None when paid calls are not allowed."""
+ if not config.paid:
+ return None
+ model = config.paid_model
+ if host_name == "codex":
+ return lambda prompt, _m="agent": codex_call(prompt, model=model)
+ if host_name == "copilot":
+ return lambda prompt, _m="agent": copilot_call(prompt, model=model)
+ return lambda prompt, _m="agent": claude_call(prompt, model=model or "haiku")
diff --git a/te_live/store.py b/te_live/store.py
new file mode 100644
index 0000000..9acf686
--- /dev/null
+++ b/te_live/store.py
@@ -0,0 +1,101 @@
+"""JSON state on disk, written atomically and serialized across hook processes.
+
+Two Claude Code sessions in one repository run their hooks concurrently, and a session's
+background fold can overlap the next session's start. Every read-modify-write of shared state
+therefore happens under an exclusive lock on a sibling lock file, and every write lands through a
+temporary file and `os.replace`, so a reader never sees half a document. Per-session documents are
+written by one session only and need no lock.
+
+Locking uses `fcntl.flock`. Where it is unavailable (Windows) the lock degrades to a no-op rather
+than failing the hook: a lost update is recoverable, a crashed hook is not.
+"""
+import contextlib
+import json
+import os
+import time
+
+try:
+ import fcntl
+except ImportError: # pragma: no cover - Windows
+ fcntl = None
+
+
+class Store:
+ """One directory of JSON documents and append-only JSONL logs."""
+
+ def __init__(self, base):
+ self.base = os.path.abspath(base)
+
+ def path(self, *parts):
+ return os.path.join(self.base, *parts)
+
+ def ensure(self, *parts):
+ d = self.path(*parts)
+ os.makedirs(d, exist_ok=True)
+ return d
+
+ def read(self, name, default=None):
+ try:
+ with open(self.path(name), encoding="utf-8") as fh:
+ return json.load(fh)
+ except (OSError, ValueError):
+ return default
+
+ def write(self, name, obj):
+ target = self.path(name)
+ os.makedirs(os.path.dirname(target), exist_ok=True)
+ tmp = f"{target}.{os.getpid()}.tmp"
+ with open(tmp, "w", encoding="utf-8") as fh:
+ json.dump(obj, fh, indent=2, sort_keys=True)
+ os.replace(tmp, target)
+ return target
+
+ def append(self, name, record):
+ target = self.path(name)
+ os.makedirs(os.path.dirname(target), exist_ok=True)
+ with open(target, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(record, sort_keys=True) + "\n")
+
+ def read_lines(self, name, limit=None):
+ """Records of a JSONL log, oldest first; the last `limit` when given."""
+ out = []
+ try:
+ with open(self.path(name), encoding="utf-8") as fh:
+ for line in fh:
+ try:
+ out.append(json.loads(line))
+ except ValueError:
+ continue
+ except OSError:
+ return []
+ return out[-limit:] if limit else out
+
+ @contextlib.contextmanager
+ def locked(self, name="state", timeout=10.0):
+ """Exclusive lock over `name` for the duration of the block; yields whether it was taken.
+
+ Polls a non-blocking flock rather than blocking outright, so a stuck holder costs at most
+ `timeout` seconds -- a hook must never hang the agent, and a hook's caller proceeds
+ either way. A caller that must not run unlocked (the fold) checks the yielded flag."""
+ os.makedirs(self.base, exist_ok=True)
+ if fcntl is None:
+ yield True
+ return
+ fh = open(self.path(f".{name}.lock"), "a+")
+ got = False
+ deadline = time.time() + timeout
+ try:
+ while True:
+ try:
+ fcntl.flock(fh.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ got = True
+ break
+ except OSError:
+ if time.time() >= deadline:
+ break
+ time.sleep(0.05)
+ yield got
+ finally:
+ if got:
+ fcntl.flock(fh.fileno(), fcntl.LOCK_UN)
+ fh.close()
diff --git a/te_live/transcripts.py b/te_live/transcripts.py
new file mode 100644
index 0000000..14ebd70
--- /dev/null
+++ b/te_live/transcripts.py
@@ -0,0 +1,541 @@
+"""Read a host's own session log in the shapes the mechanisms already consume.
+
+The mechanisms read rollouts through one seam (`tokeneconomizer.trace`), which knows the three
+streams the harness captured: Claude Code `stream-json`, the Copilot CLI's events, and
+`codex exec --json`. A live session leaves a different file behind, and this module bridges it
+rather than teaching every mechanism a fourth schema:
+
+ * Claude Code's interactive transcript uses the same `message.content[]` envelope as
+ `stream-json`, so `trace.load_trajectory` reads it unchanged. What differs is usage: the
+ transcript writes ONE ENTRY PER CONTENT BLOCK, each repeating its message's `usage`, so
+ summing entries over-counts by the number of blocks (measured locally: two thirds of
+ assistant message ids repeat, up to 12 times). `claude_session_usage` keys by message id.
+ Subagents write their own files under `<session>/subagents/`, and their tool calls and
+ tokens belong to the session that spawned them.
+
+ * Codex's rollout (`~/.codex/sessions/.../rollout-*.jsonl`) wraps the same item fields the
+ exec stream carries (`command`, `aggregated_output`, `exit_code`, `changes`) in
+ `event_msg/item_completed` records with PascalCase item types, an argv list for the command
+ and a path-keyed map for changes. `codex_rollout_to_exec` rewrites it into exec-stream lines,
+ after which `trace.load_trajectory`, the injection layers and the spiral parsers read it as
+ they read a harness transcript. Usage comes from its `token_count` events and is converted
+ exactly as `te_harness.codex.row_usage` converts the exec stream's, because Codex's
+ `input_tokens` is the GRAND input total including both cache fields.
+
+ A compressed rollout (`.jsonl.zst`) cannot be read with the standard library; it is reported
+ as unreadable rather than half-read.
+"""
+import glob
+import json
+import os
+import re
+import shlex
+
+_COPILOT_EXIT_RE = re.compile(r"<shellId: [^>]*?completed with exit code (-?\d+)>")
+_CLAUDE_KEYS = ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens",
+ "output_tokens")
+
+
+def _lines(path):
+ try:
+ with open(path, encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ line = line.strip()
+ if line:
+ yield line
+ except OSError:
+ return
+
+
+def claude_subagent_files(transcript_path):
+ """Subagent transcripts of a Claude Code session: `<dir>/<session-id>/subagents/*.jsonl`."""
+ if not transcript_path:
+ return []
+ stem = os.path.splitext(transcript_path)[0]
+ return sorted(glob.glob(os.path.join(stem, "subagents", "*.jsonl")))
+
+
+def claude_session_usage(transcript_path, include_subagents=True):
+ """Token breakdown of a Claude Code session, de-duplicated by assistant message id.
+
+ Returns the four breakdown fields `pricing.effective_tokens` reads, plus `model` (the most
+ used one), `requests` (distinct assistant messages carrying usage) and `files` read."""
+ files = [transcript_path] + (claude_subagent_files(transcript_path)
+ if include_subagents else [])
+ by_id, models, anon = {}, {}, 0
+ for path in files:
+ for line in _lines(path):
+ if '"usage"' not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") != "assistant":
+ continue
+ msg = ev.get("message") or {}
+ usage = msg.get("usage")
+ if not isinstance(usage, dict):
+ continue
+ mid = msg.get("id")
+ if not mid:
+ anon += 1
+ mid = f"anon-{path}-{anon}"
+ # Later entries of the same message carry the final counts; keep the last.
+ by_id[mid] = {k: int(usage.get(k) or 0) for k in _CLAUDE_KEYS}
+ model = msg.get("model")
+ if model:
+ models[model] = models.get(model, 0) + 1
+ out = {k: sum(u[k] for u in by_id.values()) for k in _CLAUDE_KEYS}
+ out["requests"] = len(by_id)
+ out["model"] = max(models, key=models.get) if models else None
+ out["files"] = len([f for f in files if os.path.exists(f)])
+ return out
+
+
+def claude_tool_usage(transcript_path, include_subagents=True):
+ """{tool: calls} over a session and, by default, its subagents -- a tool a subagent used is
+ in use, and denying it would take it from the subagent too."""
+ from tokeneconomizer import trace
+ counts = {}
+ files = [transcript_path] + (claude_subagent_files(transcript_path)
+ if include_subagents else [])
+ for path in files:
+ for name, n in trace.tool_usage(path).items():
+ counts[name] = counts.get(name, 0) + n
+ return counts
+
+
+def claude_deferred_tools(transcript_path):
+ """Tool names the session listed as deferred (`deferred_tools_delta` attachments).
+
+ Deferred tools cost one listing line each, not a schema, so they matter little to the bill;
+ they are still part of what the session could call, so the untool catalog includes them."""
+ names = set()
+ for line in _lines(transcript_path):
+ if '"deferred_tools_delta"' not in line:
+ continue
+ try:
+ att = json.loads(line).get("attachment") or {}
+ except ValueError:
+ continue
+ names.update(n for n in (att.get("addedNames") or []) if isinstance(n, str))
+ names.difference_update(n for n in (att.get("removedNames") or []) if isinstance(n, str))
+ return names
+
+
+# ---- Codex ---------------------------------------------------------------------------
+_ROLLOUT_ITEM_TYPES = {
+ "CommandExecution": "command_execution",
+ "FileChange": "file_change",
+ "AgentMessage": "agent_message",
+ "ImageView": "view_image",
+ "Extension": "web_search",
+ "McpToolCall": "mcp_tool_call",
+ "CollabToolCall": "collab_tool_call",
+}
+
+
+def _command_text(command):
+ if isinstance(command, list):
+ return shlex.join(str(c) for c in command)
+ return str(command or "")
+
+
+def _rollout_item_to_exec(item):
+ """One rollout item -> the `codex exec --json` item shape, or None to skip it."""
+ kind = _ROLLOUT_ITEM_TYPES.get(item.get("type"), None)
+ if kind is None:
+ return None
+ out = {"type": kind, "id": item.get("id")}
+ if kind == "command_execution":
+ out.update(command=_command_text(item.get("command")),
+ aggregated_output=item.get("aggregated_output")
+ or ((item.get("stdout") or "") + (item.get("stderr") or "")),
+ exit_code=item.get("exit_code"),
+ status=str(item.get("status") or "").lower() or "completed")
+ elif kind == "file_change":
+ changes = item.get("changes")
+ if isinstance(changes, dict):
+ changes = [dict({"path": p}, **({"kind": (v or {}).get("type")}
+ if isinstance(v, dict) else {}))
+ for p, v in changes.items()]
+ out.update(changes=changes or [],
+ status=str(item.get("status") or "").lower() or "completed")
+ elif kind == "agent_message":
+ content = item.get("content")
+ if isinstance(content, list):
+ content = " ".join(c.get("text", "") for c in content if isinstance(c, dict))
+ out.update(text=content or "")
+ else:
+ out.update({k: v for k, v in item.items() if k not in ("type", "id")})
+ return out
+
+
+def codex_rollout_to_exec(rollout_path, out_path):
+ """Rewrite a Codex rollout into `codex exec --json` lines at `out_path`.
+
+ Returns the number of items written, or None when the rollout cannot be read (missing, or
+ compressed). The thread id rides on a leading `thread.started` line, as in the exec stream."""
+ if not rollout_path or rollout_path.endswith(".zst") or not os.path.exists(rollout_path):
+ return None
+ n = 0
+ thread = None
+ with open(out_path, "w", encoding="utf-8") as out:
+ for line in _lines(rollout_path):
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ payload = ev.get("payload") if isinstance(ev.get("payload"), dict) else {}
+ if ev.get("type") == "session_meta" and thread is None:
+ thread = payload.get("id") or payload.get("session_id")
+ if thread:
+ out.write(json.dumps({"type": "thread.started", "thread_id": thread}) + "\n")
+ continue
+ if ev.get("type") != "event_msg" or payload.get("type") != "item_completed":
+ continue
+ item = _rollout_item_to_exec(payload.get("item") or {})
+ if item is None:
+ continue
+ out.write(json.dumps({"type": "item.completed", "item": item}) + "\n")
+ n += 1
+ return n
+
+
+def codex_capability_trace(rollout_path, out_path):
+ """The exec-stream lines `mechanisms.codex_untool.inspect_usage` reads, for a rollout.
+
+ `codex_rollout_to_exec` keeps the items the other mechanisms read; capability evidence
+ also needs the model's own tool calls. In code mode every call is one `exec` custom tool
+ whose input calls `tools.<name>(...)` -- `tools.view_image(`, `tools.spawn_agent(` -- and
+ those ride in `response_item` records, which are added here as `custom_tool_call` /
+ `function_call` items, the shape `codex_untool._item_capabilities` parses. Returns the number
+ of items written, or None when the rollout cannot be read."""
+ n = codex_rollout_to_exec(rollout_path, out_path)
+ if n is None:
+ return None
+ with open(out_path, "a", encoding="utf-8") as out:
+ for line in _lines(rollout_path):
+ if '"response_item"' not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ p = ev.get("payload") if isinstance(ev.get("payload"), dict) else {}
+ if ev.get("type") != "response_item" or p.get("type") not in (
+ "custom_tool_call", "function_call"):
+ continue
+ item = {"type": p.get("type"), "name": p.get("name"),
+ "id": p.get("call_id") or p.get("id"),
+ "status": str(p.get("status") or "completed").lower()}
+ item["input" if p.get("type") == "custom_tool_call" else "arguments"] = (
+ p.get("input") if p.get("type") == "custom_tool_call" else p.get("arguments"))
+ out.write(json.dumps({"type": "item.completed", "item": item}) + "\n")
+ n += 1
+ return n
+
+
+def codex_session_meta(rollout_path):
+ """{cwd, model, cli_version} from a rollout's session_meta / turn_context records."""
+ meta = {}
+ for line in _lines(rollout_path):
+ if '"session_meta"' not in line and '"turn_context"' not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ p = ev.get("payload") if isinstance(ev.get("payload"), dict) else {}
+ if ev.get("type") == "session_meta":
+ meta.setdefault("cwd", p.get("cwd"))
+ meta.setdefault("cli_version", p.get("cli_version"))
+ elif ev.get("type") == "turn_context" and p.get("model"):
+ meta["model"] = p.get("model")
+ return meta
+
+
+def codex_session_usage(rollout_path):
+ """Token breakdown of a Codex session from its last cumulative `token_count`.
+
+ Converted as `te_harness.codex.row_usage` converts the exec stream: fresh input is the grand
+ input total less both cache fields, and reasoning tokens are added to output."""
+ total = None
+ requests = 0
+ for line in _lines(rollout_path):
+ if '"token_count"' not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ p = ev.get("payload") if isinstance(ev.get("payload"), dict) else {}
+ if p.get("type") != "token_count":
+ continue
+ info = p.get("info") or {}
+ if info.get("total_token_usage"):
+ total = info["total_token_usage"]
+ requests += 1
+ meta = codex_session_meta(rollout_path)
+ if not total:
+ return {"input_tokens": None, "cache_read_input_tokens": None,
+ "cache_creation_input_tokens": None, "output_tokens": None,
+ "requests": 0, "model": meta.get("model")}
+ inp = int(total.get("input_tokens") or 0)
+ cached = int(total.get("cached_input_tokens") or 0)
+ cw = int(total.get("cache_write_input_tokens") or 0)
+ reasoning = int(total.get("reasoning_output_tokens") or 0)
+ return {"input_tokens": max(0, inp - cached - cw),
+ "cache_read_input_tokens": cached,
+ "cache_creation_input_tokens": cw,
+ "output_tokens": int(total.get("output_tokens") or 0) + reasoning,
+ "reasoning_output_tokens": reasoning,
+ "requests": requests, "model": meta.get("model")}
+
+
+# ---- Pi ------------------------------------------------------------------------------
+# Pi (earendil-works/pi) stores a session as a TREE of JSONL entries linked by `id` /
+# `parentId`; the conversation is the path from the current leaf back to the root, and an
+# abandoned branch stays in the file. Assistant messages carry `toolCall` blocks
+# ({id, name, arguments}) and their own usage; tool results are `toolResult` messages with
+# `isError`. `pi_session_to_claude` writes the ACTIVE branch as the Claude Code envelope the
+# trace seam already reads, so every mechanism reads a Pi session without a Pi branch.
+#
+# Usage, per pi-ai's own type: `input` excludes the cache fields, `output` already includes
+# reasoning, and `cacheWrite1h` is a subset of `cacheWrite` -- so the mapping to the ledger's
+# four fields is direct.
+
+def _pi_entries(path):
+ out = []
+ for line in _lines(path):
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(ev, dict):
+ out.append(ev)
+ return out
+
+
+def pi_session_header(path):
+ """The `session` header line ({id, cwd, version, ...}), {} if absent."""
+ for ev in _pi_entries(path)[:1]:
+ if ev.get("type") == "session":
+ return ev
+ return {}
+
+
+def pi_active_branch(entries):
+ """Entries on the path from the last entry (the leaf) back to the root, in order."""
+ by_id = {e.get("id"): e for e in entries if e.get("id")}
+ tree = [e for e in entries if e.get("id")]
+ if not tree:
+ return []
+ path, cur, seen = [], tree[-1], set()
+ while cur is not None and cur.get("id") not in seen:
+ seen.add(cur.get("id"))
+ path.append(cur)
+ cur = by_id.get(cur.get("parentId"))
+ return list(reversed(path))
+
+
+def _pi_usage(u):
+ if not isinstance(u, dict):
+ return None
+ return {"input_tokens": int(u.get("input") or 0),
+ "cache_read_input_tokens": int(u.get("cacheRead") or 0),
+ "cache_creation_input_tokens": int(u.get("cacheWrite") or 0),
+ "output_tokens": int(u.get("output") or 0),
+ "cost_usd": float((u.get("cost") or {}).get("total") or 0)}
+
+
+def _pi_text(content):
+ if isinstance(content, str):
+ return content
+ if isinstance(content, list):
+ return "\n".join(c.get("text", "") for c in content
+ if isinstance(c, dict) and c.get("type") == "text")
+ return ""
+
+
+def pi_session_to_claude(path, out_path):
+ """Write a Pi session's active branch as Claude Code transcript lines.
+
+ Returns the number of tool calls written, or None when the file is missing."""
+ if not path or not os.path.exists(path):
+ return None
+ n = 0
+ with open(out_path, "w", encoding="utf-8") as out:
+ for e in pi_active_branch(_pi_entries(path)):
+ if e.get("type") != "message":
+ continue
+ m = e.get("message") or {}
+ role = m.get("role")
+ if role == "user":
+ out.write(json.dumps({"type": "user", "message": {
+ "role": "user", "content": _pi_text(m.get("content"))}}) + "\n")
+ elif role == "assistant":
+ blocks = []
+ for b in m.get("content") or []:
+ if isinstance(b, dict) and b.get("type") == "toolCall":
+ blocks.append({"type": "tool_use", "id": b.get("id"),
+ "name": b.get("name"), "input": b.get("arguments") or {}})
+ n += 1
+ elif isinstance(b, dict) and b.get("type") == "text":
+ blocks.append({"type": "text", "text": b.get("text", "")})
+ msg = {"id": e.get("id"), "role": "assistant", "content": blocks,
+ "model": m.get("model")}
+ usage = _pi_usage(m.get("usage"))
+ if usage:
+ msg["usage"] = {k: v for k, v in usage.items() if k != "cost_usd"}
+ out.write(json.dumps({"type": "assistant", "message": msg}) + "\n")
+ elif role == "toolResult":
+ out.write(json.dumps({"type": "user", "message": {"role": "user", "content": [{
+ "type": "tool_result", "tool_use_id": m.get("toolCallId"),
+ "content": _pi_text(m.get("content")),
+ "is_error": bool(m.get("isError"))}]}}) + "\n")
+ return n
+
+
+def pi_session_usage(path):
+ """Token breakdown of a Pi session's active branch, plus Pi's own cost estimate."""
+ totals = {"input_tokens": 0, "cache_read_input_tokens": 0,
+ "cache_creation_input_tokens": 0, "output_tokens": 0}
+ cost, requests, models = 0.0, 0, {}
+ for e in pi_active_branch(_pi_entries(path)):
+ u = None
+ if e.get("type") == "message" and (e.get("message") or {}).get("role") == "assistant":
+ m = e["message"]
+ u = _pi_usage(m.get("usage"))
+ if m.get("model"):
+ models[m["model"]] = models.get(m["model"], 0) + 1
+ elif e.get("type") in ("usage", "compaction", "branch_summary"):
+ u = _pi_usage(e.get("usage"))
+ if u:
+ requests += 1
+ cost += u.pop("cost_usd")
+ for k in totals:
+ totals[k] += u[k]
+ totals.update(requests=requests, cost_usd_vendor=round(cost, 6),
+ model=max(models, key=models.get) if models else None)
+ return totals
+
+
+# ---- GitHub Copilot CLI ---------------------------------------------------------------
+# Copilot writes each session to `<COPILOT_HOME>/session-state/<id>/events.jsonl`: the same
+# `tool.execution_start` / `tool.execution_complete` event stream `copilot --output-format json`
+# prints, which `trace.load_trajectory` already reads, so no rewrite is needed. Usage is in the
+# `session.shutdown` event, per model:
+# modelMetrics[model].usage = {inputTokens, outputTokens, cacheReadTokens,
+# cacheWriteTokens, reasoningTokens}
+# `inputTokens` is the GRAND input total: against a scripted endpoint that reported
+# prompt_tokens=1000 with cached_tokens=600 on each of 4 requests, it recorded inputTokens=4000
+# and cacheReadTokens=2400 (1.0.83). Fresh input is therefore input less both cache fields, as
+# `sum_openai_usage` and the Codex reader convert theirs. `outputTokens` is the provider's
+# completion count, which already includes reasoning, so reasoning is not added again.
+# A session killed before it shut down has no usage; it is reported as unpriced.
+
+def _copilot_events(path):
+ for line in _lines(path):
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(ev, dict):
+ yield ev
+
+
+def copilot_session_meta(path):
+ """{cwd, model, cli_version, session_id} from a Copilot session's `session.start`."""
+ for ev in _copilot_events(path):
+ if ev.get("type") == "session.start":
+ d = ev.get("data") or {}
+ return {"cwd": (d.get("context") or {}).get("cwd"), "model": d.get("selectedModel"),
+ "cli_version": d.get("copilotVersion"), "session_id": d.get("sessionId")}
+ return {}
+
+
+def copilot_session_usage(path):
+ """Token breakdown of a Copilot session from its `session.shutdown` events."""
+ totals = {"input_tokens": 0, "cache_read_input_tokens": 0,
+ "cache_creation_input_tokens": 0, "output_tokens": 0}
+ requests, models, seen, tool_defs = 0, {}, False, None
+ for ev in _copilot_events(path):
+ if ev.get("type") != "session.shutdown":
+ continue
+ d = ev.get("data") or {}
+ for model, m in (d.get("modelMetrics") or {}).items():
+ u = (m or {}).get("usage") or {}
+ inp = int(u.get("inputTokens") or 0)
+ cr, cw = int(u.get("cacheReadTokens") or 0), int(u.get("cacheWriteTokens") or 0)
+ totals["input_tokens"] += max(0, inp - cr - cw)
+ totals["cache_read_input_tokens"] += cr
+ totals["cache_creation_input_tokens"] += cw
+ totals["output_tokens"] += int(u.get("outputTokens") or 0)
+ n = int(((m or {}).get("requests") or {}).get("count") or 0)
+ requests += n
+ models[model] = models.get(model, 0) + n
+ seen = True
+ if d.get("toolDefinitionsTokens") is not None:
+ tool_defs = int(d["toolDefinitionsTokens"])
+ if not seen:
+ out = {k: None for k in totals}
+ out.update(requests=0, model=copilot_session_meta(path).get("model"))
+ return out
+ totals.update(requests=requests, model=max(models, key=models.get) if models else None,
+ tool_definitions_tokens=tool_defs)
+ return totals
+
+
+def copilot_turns(path):
+ """Prompt-turns of a Copilot session: [[(name, args, result, exit_code)]], one list per
+ user message, tool calls paired start-to-complete by call id."""
+ turns, cur, pend = [], None, {}
+ for ev in _copilot_events(path):
+ t, d = ev.get("type"), ev.get("data") or {}
+ if t == "user.message":
+ if cur:
+ turns.append(cur)
+ cur, pend = [], {}
+ elif t == "tool.execution_start" and cur is not None:
+ pend[d.get("toolCallId")] = (str(d.get("toolName") or ""),
+ d.get("arguments") if isinstance(d.get("arguments"),
+ dict) else {})
+ elif t == "tool.execution_complete" and cur is not None:
+ name, args = pend.pop(d.get("toolCallId"), (None, None))
+ if name is None:
+ continue
+ res = d.get("result") or {}
+ text = res.get("content") if isinstance(res, dict) else str(res)
+ text = text if isinstance(text, str) else json.dumps(text)
+ if not d.get("success", True) and isinstance(d.get("error"), (str, dict)):
+ text = (text or "") + "\n" + json.dumps(d.get("error"))
+ m = _COPILOT_EXIT_RE.search((text or "")[-400:])
+ cur.append((name, args, text or "", int(m.group(1)) if m else None))
+ if cur:
+ turns.append(cur)
+ return turns
+
+
+def copilot_mcp_servers(env=None):
+ """MCP server names a Copilot session can load: the built-in GitHub server plus every
+ server in the user's `mcp-config.json`. Tool names are `<server>-<tool>`."""
+ env = os.environ if env is None else env
+ home = env.get("COPILOT_HOME") or os.path.join(os.path.expanduser("~"), ".copilot")
+ names = {"github-mcp-server"}
+ try:
+ with open(os.path.join(home, "mcp-config.json"), encoding="utf-8") as fh:
+ names.update((json.load(fh).get("mcpServers") or {}).keys())
+ except (OSError, ValueError, AttributeError):
+ pass
+ return names
+
+
+def mcp_server_of(tool, servers):
+ """The server a Copilot tool name belongs to (longest matching `<server>-` prefix)."""
+ best = None
+ for s in servers:
+ if tool.startswith(s + "-") and (best is None or len(s) > len(best)):
+ best = s
+ return best
diff --git a/te_live/untool.py b/te_live/untool.py
new file mode 100644
index 0000000..0137b1a
--- /dev/null
+++ b/te_live/untool.py
@@ -0,0 +1,735 @@
+"""Applying an untool proposal on each host, and measuring the request with and without it.
+
+Evidence untooling decides WHICH tools to drop (`mechanisms.tooling.ToolingStage`, driven by
+`live.fold`). On the benchmark the harness applied the decision at launch. A plugin launches
+nothing, so on each host the lever is a per-repository settings file the agent reads at session
+start, and the unit of removal is whatever that file can express:
+
+ * Claude Code -- a TOOL. A bare-name `permissions.deny` rule removes the tool from the
+ session's tool list, not merely refusing the call. Verified on 2.1.286: three deny rules
+ took a session from 30 tools to 27, the same list `--disallowed-tools` produced, and the
+ same rules in a project's `.claude/settings.local.json` took it to 23.
+ * Codex -- a CAPABILITY package (`mechanisms.codex_untool`), written as config toggles into
+ the project's `.codex/config.toml`. Verified on 0.153.1 with `codex debug prompt-input`:
+ the collaboration, goals and image toggles take effect from a project-local file -- but
+ ONLY in a trusted project; an untrusted project's `.codex/` is ignored, so `apply` checks
+ trust and says so. The `skills` package is not
+ offered: its toggle (`[[skills.config]] enabled = false` per bundled skill) takes effect in
+ the user-level config only, and a per-repository decision does not belong there.
+ * Copilot CLI -- an MCP SERVER, through `disabledMcpServers` in the repository's
+ `.github/copilot/settings.local.json`. Verified on 1.0.83: listing the built-in
+ `github-mcp-server` removes its tools from the request -- but, as on Codex, ONLY in a
+ trusted folder (Copilot's `trustedFolders`, which its first-run prompt fills): the same
+ file in an untrusted directory changes nothing.
+ `apply` checks trust and confirms the effect with a local capture. Copilot has no persistent per-tool filter (`--excluded-tools` is a launch
+ flag; `permissions.deny` refuses calls but leaves the schema in the request), so a server
+ is proposed only when every one of its tools has gone unused.
+
+Rules common to all three:
+
+ * Only the user's approval (`te untool apply`), or `untool: auto`, writes the file, and only
+ from the NEXT session: tools are fixed at session start, and changing them mid-session would
+ re-write the cached prompt prefix.
+ * The writer adds and removes exactly the entries it recorded adding (`untool_applied.json`);
+ an entry the user wrote is never touched.
+ * When the plugin creates the file, it adds the path to `.git/info/exclude`, so a file meant
+ to be per-user can never be committed by accident.
+
+Measuring: `measure` compares one request with and without the proposal. On Claude Code that is
+two small `claude -p` requests on the cheapest model. On Codex and Copilot it costs nothing: Codex renders its prompt
+input locally (`codex debug prompt-input`), and Copilot's request is captured by a scripted
+endpoint on localhost (`copilot_capture`), so no model is ever called.
+"""
+import json
+import os
+import re
+import shutil
+import subprocess
+import tempfile
+import threading
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from . import adapter as live_adapter
+from . import transcripts
+
+SETTINGS_REL = os.path.join(".claude", "settings.local.json")
+CODEX_CONFIG_REL = os.path.join(".codex", "config.toml")
+COPILOT_SETTINGS_REL = os.path.join(".github", "copilot", "settings.local.json")
+
+_SETTINGS = {"claude": SETTINGS_REL, "codex": CODEX_CONFIG_REL, "copilot": COPILOT_SETTINGS_REL}
+_UNITS = {"claude": "tool", "codex": "capability", "copilot": "MCP server"}
+
+# Codex capability -> (table or None for top level, key, TOML value). Each probed project-local
+# on 0.153.1 (see the module docstring); `te_harness.codex` holds the launch-time superset.
+CODEX_TOGGLES = {
+ "collaboration": ((None, "include_collaboration_mode_instructions", "false"),
+ ("agents", "enabled", "false"),
+ ("features", "multi_agent", "false")),
+ "goals": (("features", "goals", "false"),),
+ "image_media": (("features", "view_image", "false"),),
+}
+# What the live Codex lever can reach. `web` and `request_user_input` are a person's tools and
+# never proposed (as the web is not on Claude Code); `interactive_guidance` rewrites the base
+# prompt's guidance for talking to a user, which an interactive session is; `skills` is
+# user-level only.
+CODEX_LIVE_CAPABILITIES = tuple(sorted(CODEX_TOGGLES))
+MARK = "# tokeneconomizer"
+
+
+def supports(host):
+ return host in _SETTINGS
+
+
+def unit(host, plural=False):
+ u = _UNITS.get(host, "tool")
+ return (u[:-1] + "ies" if u.endswith("y") else u + "s") if plural else u
+
+
+def settings_path(repo_root, host="claude"):
+ return os.path.join(repo_root, _SETTINGS.get(host, SETTINGS_REL))
+
+
+def _read_settings(path):
+ """The settings object at `path`: {} when there is no file. A file that is there but is not
+ a JSON object is refused rather than written over -- it is the person's, and a parse error
+ is no reason to replace what they had in it."""
+ try:
+ with open(path, encoding="utf-8") as fh:
+ text = fh.read()
+ except FileNotFoundError:
+ return {}
+ except OSError as e:
+ raise RuntimeError(f"cannot read {path}: {e}")
+ if not text.strip():
+ return {}
+ try:
+ obj = json.loads(text)
+ except ValueError as e:
+ raise RuntimeError(f"refusing to edit {path}: it is not valid JSON ({e}); fix it first")
+ if not isinstance(obj, dict):
+ raise RuntimeError(f"refusing to edit {path}: it is not a JSON object")
+ return obj
+
+
+def _guard_target(repo_root, path):
+ """Refuse a write that a symlink would carry out of the repository.
+
+ The settings paths are the repository's, so a checkout decides what they are: a `.claude`,
+ `.codex` or `.github` that is a symlink, or a settings file that is one, would land the
+ write somewhere else -- a user-level configuration, another project."""
+ root = os.path.realpath(repo_root)
+ parent = os.path.realpath(os.path.dirname(path))
+ if os.path.islink(path) or os.path.commonpath([root, parent]) != root:
+ raise RuntimeError(f"refusing to write {path}: it is, or goes through, a symlink that "
+ "leads out of the repository")
+
+
+def _write_json(path, obj):
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ tmp = path + ".te.tmp"
+ with open(tmp, "w", encoding="utf-8") as fh:
+ json.dump(obj, fh, indent=2)
+ fh.write("\n")
+ os.replace(tmp, path)
+
+
+def _write_text(path, text):
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ tmp = path + ".te.tmp"
+ with open(tmp, "w", encoding="utf-8") as fh:
+ fh.write(text)
+ os.replace(tmp, path)
+
+
+def _exclude_from_git(repo_root, rel=SETTINGS_REL):
+ """Append `rel` to `.git/info/exclude` (local-only ignore), idempotently."""
+ from .paths import _git_common_dir
+ gitdir = _git_common_dir(repo_root)
+ if not gitdir:
+ return False
+ # A `.git` FILE names its git directory, so a checkout could point it anywhere: follow it
+ # only to something that is a git directory, and never write through a symlinked `info` or
+ # `exclude`.
+ if (os.path.isfile(os.path.join(repo_root, ".git"))
+ and not os.path.isfile(os.path.join(gitdir, "HEAD"))):
+ return False
+ path = os.path.join(gitdir, "info", "exclude")
+ if os.path.islink(os.path.dirname(path)) or os.path.islink(path):
+ return False
+ entry = "/" + rel.replace(os.sep, "/")
+ try:
+ existing = open(path, encoding="utf-8").read() if os.path.exists(path) else ""
+ if entry in existing.split():
+ return True
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "a", encoding="utf-8") as fh:
+ fh.write(("" if existing.endswith("\n") or not existing else "\n")
+ + "# added by TokenEconomizer (per-user tool deny list)\n" + entry + "\n")
+ return True
+ except OSError:
+ return False
+
+
+def applied(store):
+ return list((store.read("untool_applied.json") or {}).get("tools", []))
+
+
+def _record(store, items, path):
+ store.write("untool_applied.json", {"tools": sorted(items), "path": path,
+ "updated": time.time()})
+
+
+# ---- the JSON-list hosts: Claude Code's deny rules, Copilot's disabled MCP servers -------
+def _json_list_keys(host):
+ return ("permissions", "deny") if host == "claude" else (None, "disabledMcpServers")
+
+
+def _apply_json_list(repo_root, items, store, host):
+ path = settings_path(repo_root, host)
+ _guard_target(repo_root, path)
+ created = not os.path.exists(path)
+ data = _read_settings(path)
+ outer, key = _json_list_keys(host)
+ holder = data.setdefault(outer, {}) if outer else data
+ lst = holder.setdefault(key, [])
+ added = [t for t in items if t not in lst]
+ if not added:
+ return []
+ lst.extend(added)
+ _write_json(path, data)
+ if created:
+ _exclude_from_git(repo_root, _SETTINGS[host])
+ _record(store, set(applied(store)) | set(added), path)
+ return added
+
+
+def _restore_json_list(repo_root, store, items, host):
+ mine = set(applied(store))
+ target = mine if not items else mine & set(items)
+ if not target:
+ return []
+ path = settings_path(repo_root, host)
+ _guard_target(repo_root, path)
+ data = _read_settings(path)
+ outer, key = _json_list_keys(host)
+ holder = (data.get(outer) or {}) if outer else data
+ lst = holder.get(key) or []
+ kept = [r for r in lst if r not in target]
+ removed = [r for r in lst if r in target]
+ if key in holder:
+ holder[key] = kept
+ if not kept:
+ holder.pop(key)
+ if outer and not holder:
+ data.pop(outer, None)
+ _write_json(path, data)
+ _record(store, mine - target, path)
+ return removed
+
+
+# ---- Codex: marked lines in the project's .codex/config.toml ----------------------------
+_HEADER_RE = re.compile(r"^\s*\[\s*([A-Za-z0-9_.\-\"]+)\s*\]\s*(#.*)?$")
+_ANY_HEADER_RE = re.compile(r"^\s*\[")
+
+
+def _sections(lines):
+ """[(table or None, start, end)] -- line ranges of the top level and each [table]."""
+ out, cur, start = [], None, 0
+ for i, line in enumerate(lines):
+ if _ANY_HEADER_RE.match(line):
+ out.append((cur, start, i))
+ m = _HEADER_RE.match(line)
+ cur, start = (m.group(1) if m else "[[array]]"), i + 1
+ out.append((cur, start, len(lines)))
+ return out
+
+
+def _key_set(lines, table, key):
+ """Is `key` already set in `table` -- as `key =` under [table], or as a top-level dotted
+ `table.key =`? A key the user set is theirs, whatever its value."""
+ for t, a, b in _sections(lines):
+ for line in lines[a:b]:
+ s = line.strip()
+ if table is None and t is None and re.match(rf"{re.escape(key)}\s*=", s):
+ return True
+ if table is not None and t == table and re.match(rf"{re.escape(key)}\s*=", s):
+ return True
+ if table is not None and t is None and re.match(
+ rf"{re.escape(table)}\.{re.escape(key)}\s*=", s):
+ return True
+ return False
+
+
+def _toml_insert(text, table, key, value, tag):
+ """Insert `key = value` into `table` (None: top level), marked with `tag`."""
+ lines = text.splitlines()
+ if _key_set(lines, table, key):
+ return text, False
+ # `features = { ... }` -- an inline table cannot be extended, and adding a [features] table
+ # beside it would make the file invalid; leave that table to the person who wrote it.
+ if table is not None and any(re.match(rf"\s*{re.escape(table)}\s*=", ln) for ln in lines):
+ return text, False
+ line = f"{key} = {value} {tag}"
+ secs = _sections(lines)
+ first_header = next((a - 1 for t, a, b in secs[1:]), len(lines))
+ if table is None:
+ lines.insert(first_header, line)
+ else:
+ hdr = next((a for t, a, b in secs if t == table), None)
+ dotted = any(re.match(rf"\s*{re.escape(table)}\.\w+\s*=", ln)
+ for ln in lines[:first_header])
+ if hdr is not None:
+ lines.insert(hdr, line)
+ elif dotted:
+ lines.insert(first_header, f"{table}.{line}")
+ else:
+ if lines and lines[-1].strip():
+ lines.append("")
+ lines += [f"[{table}] {tag}", line]
+ return "\n".join(lines) + "\n", True
+
+
+def _toml_remove(text, tags):
+ """Drop every line carrying one of `tags`; a [table] header the plugin added goes too,
+ unless someone has since put a key of their own under it."""
+ lines = text.splitlines()
+ marked = lambda ln: any(ln.rstrip().endswith(t) for t in tags)
+ keep = []
+ for i, ln in enumerate(lines):
+ if not marked(ln):
+ keep.append(ln)
+ continue
+ if _HEADER_RE.match(ln):
+ j, foreign = i + 1, False
+ while j < len(lines) and not _ANY_HEADER_RE.match(lines[j]):
+ if lines[j].strip() and not marked(lines[j]):
+ foreign = True
+ break
+ j += 1
+ if foreign:
+ keep.append(ln)
+ while keep and not keep[-1].strip():
+ keep.pop()
+ return ("\n".join(keep) + "\n") if keep else ""
+
+
+def _toml_valid(text):
+ try:
+ import tomllib
+ except ImportError: # Python 3.10: Codex itself is the check (`verify_codex`)
+ return True
+ try:
+ tomllib.loads(text)
+ return True
+ except tomllib.TOMLDecodeError:
+ return False
+
+
+def _git_tracked(repo_root, rel):
+ """Is `rel` committed in this repository? False when git cannot tell."""
+ try:
+ out = subprocess.run(["git", "ls-files", "--error-unmatch", rel], cwd=repo_root,
+ capture_output=True, text=True, timeout=10)
+ return out.returncode == 0
+ except (OSError, subprocess.SubprocessError):
+ return False
+
+
+def codex_config_error(repo_root, cli="codex", timeout=30):
+ """Codex's own complaint about the configuration it would load here, or None.
+
+ `codex features list` loads the whole layered configuration and calls no model, so a file
+ that would break the person's next session fails here first. Silent (None) when the CLI is
+ not installed, or the project is untrusted and its file is not loaded at all."""
+ if not shutil.which(cli):
+ return None
+ try:
+ out = subprocess.run([cli, "features", "list"], capture_output=True, text=True,
+ cwd=repo_root, timeout=timeout, env=dict(os.environ, TE_SIDECALL="1"))
+ except (OSError, subprocess.SubprocessError):
+ return None
+ return None if out.returncode == 0 else ((out.stderr or out.stdout).strip()[:500]
+ or f"exit {out.returncode}")
+
+
+def _apply_codex(repo_root, capabilities, store):
+ path = settings_path(repo_root, "codex")
+ _guard_target(repo_root, path)
+ created = not os.path.exists(path)
+ if not created and _git_tracked(repo_root, CODEX_CONFIG_REL):
+ # Claude Code's and Copilot's files are per-user by name; a project's .codex/config.toml
+ # is often the TEAM's, committed. Marked lines in it would ride along in the next commit.
+ raise RuntimeError(f"{path} is committed in this repository; the plugin will not edit a "
+ "shared file. Add the toggles by hand, or keep them in your own "
+ "copy of it.")
+ original = open(path, encoding="utf-8").read() if not created else ""
+ text = original
+ added = []
+ for cap in capabilities:
+ if cap not in CODEX_TOGGLES or cap in applied(store):
+ continue
+ for table, key, value in CODEX_TOGGLES[cap]:
+ text, _ = _toml_insert(text, table, key, value, f"{MARK}:{cap}")
+ added.append(cap)
+ if not added:
+ return []
+ if not _toml_valid(text):
+ raise RuntimeError(f"refusing to write {path}: the result would not parse as TOML")
+ _write_text(path, text)
+ err = codex_config_error(repo_root)
+ if err:
+ if created:
+ os.remove(path)
+ else:
+ _write_text(path, original)
+ raise RuntimeError(f"Codex rejected the edited {path}, so it was put back: {err}")
+ if created:
+ _exclude_from_git(repo_root, CODEX_CONFIG_REL)
+ _record(store, set(applied(store)) | set(added), path)
+ return added
+
+
+def _restore_codex(repo_root, store, capabilities):
+ mine = set(applied(store))
+ target = mine if not capabilities else mine & set(capabilities)
+ if not target:
+ return []
+ path = settings_path(repo_root, "codex")
+ _guard_target(repo_root, path)
+ if os.path.exists(path):
+ text = _toml_remove(open(path, encoding="utf-8").read(),
+ [f"{MARK}:{c}" for c in target])
+ if text.strip():
+ _write_text(path, text)
+ else:
+ os.remove(path)
+ _record(store, mine - target, path)
+ return sorted(target)
+
+
+def copilot_trusted_folders(env=None):
+ """Copilot's `trustedFolders` (its config.json starts with `//` comment lines), or None."""
+ from .hosts import copilot_home
+ try:
+ text = open(os.path.join(copilot_home(env), "config.json"), encoding="utf-8").read()
+ doc = json.loads("\n".join(ln for ln in text.splitlines()
+ if not ln.lstrip().startswith("//")))
+ except (OSError, ValueError):
+ return None
+ return [f for f in (doc.get("trustedFolders") or []) if isinstance(f, str)]
+
+
+def copilot_folder_trusted(path, env=None):
+ """Is `path` in or under a folder Copilot trusts? None when the config is unreadable."""
+ folders = copilot_trusted_folders(env)
+ if folders is None:
+ return None
+ real = os.path.realpath(path)
+ for f in folders:
+ f = os.path.realpath(f)
+ if real == f or real.startswith(f.rstrip(os.sep) + os.sep):
+ return True
+ return False
+
+
+def verify_copilot(repo_root, servers, cli="copilot"):
+ """{server: tools still in the request} for a session started in `repo_root`, by the local
+ capture (no model call); None if no request could be captured."""
+ body = copilot_capture(cwd=repo_root, cli=cli)
+ if not body:
+ return None
+ names = _request_tools(body)
+ return {s: sum(1 for n in names if n and n.startswith(s + "-")) for s in servers}
+
+
+def codex_home(env=None):
+ env = os.environ if env is None else env
+ return env.get("CODEX_HOME") or os.path.join(os.path.expanduser("~"), ".codex")
+
+
+def codex_project_trusted(repo_root, env=None):
+ """True/False from the user's Codex config: is this checkout a trusted project? A
+ project-local `.codex/config.toml` is ignored until it is. None if the config is
+ unreadable."""
+ try:
+ text = open(os.path.join(codex_home(env), "config.toml"), encoding="utf-8").read()
+ except OSError:
+ return None
+ want = {os.path.abspath(repo_root), os.path.realpath(repo_root)}
+ current = None
+ for line in text.splitlines():
+ m = re.match(r'^\s*\[projects\.(?:"([^"]+)"|\'([^\']+)\')\]\s*$', line)
+ if m:
+ current = m.group(1) or m.group(2)
+ continue
+ if _ANY_HEADER_RE.match(line):
+ current = None
+ continue
+ if current in want and re.match(r'^\s*trust_level\s*=\s*"trusted"', line):
+ return True
+ return False
+
+
+def verify_codex(repo_root, capabilities, cli="codex", timeout=30):
+ """What `codex features list`, run in the repository, says about the applied toggles:
+ {feature: enabled} for the features those capabilities switch off, or None when the CLI
+ is unavailable. Free: it reads configuration and calls no model."""
+ feats = [k for c in capabilities for t, k, v in CODEX_TOGGLES.get(c, ())
+ if t == "features"]
+ if not feats or not shutil.which(cli):
+ return None
+ try:
+ out = subprocess.run([cli, "features", "list"], capture_output=True, text=True,
+ cwd=repo_root, timeout=timeout,
+ env=dict(os.environ, TE_SIDECALL="1"))
+ except (OSError, subprocess.SubprocessError):
+ return None
+ state = {}
+ for line in (out.stdout or "").splitlines():
+ parts = line.split()
+ if len(parts) >= 2 and parts[0] in feats:
+ state[parts[0]] = parts[-1] == "true"
+ return state or None
+
+
+# ---- dispatch ------------------------------------------------------------------------------
+def apply(repo_root, tools, store, host="claude"):
+ """Write `tools` (tools, capabilities or MCP servers, by host). Returns those newly added."""
+ if host == "codex":
+ return _apply_codex(repo_root, tools, store)
+ return _apply_json_list(repo_root, tools, store, host)
+
+
+def restore(repo_root, store, tools=None, host="claude"):
+ """Remove entries this plugin added (all of them, or just `tools`). Returns removed."""
+ if host == "codex":
+ return _restore_codex(repo_root, store, tools)
+ return _restore_json_list(repo_root, store, tools, host)
+
+
+def filter_proposal(proposal, streaks, evidence):
+ """Hold back high-risk tools until they have HIGH_RISK_FACTOR x the usual evidence."""
+ out = []
+ for t in proposal:
+ need = evidence * (live_adapter.HIGH_RISK_FACTOR if t in live_adapter.HIGH_RISK else 1)
+ if int(streaks.get(t, 0)) >= need:
+ out.append(t)
+ return out
+
+
+def copilot_server_proposal(raw, catalog, used, servers):
+ """MCP servers every one of whose catalog tools is in the evidence proposal `raw`."""
+ by_server = {}
+ for t in catalog:
+ s = transcripts.mcp_server_of(t, servers)
+ if s:
+ by_server.setdefault(s, set()).add(t)
+ raw = set(raw)
+ return sorted(s for s, tools in by_server.items()
+ if tools and tools <= raw and not (tools & set(used)))
+
+
+# ---- the catalog probe and the measurement ------------------------------------------------
+def _init_tools(extra_args, cli="claude", timeout=60):
+ """Tool list from a `claude -p` session's init event; the process is killed right after it,
+ before the model is asked anything."""
+ env = dict(os.environ, TE_SIDECALL="1")
+ cmd = [cli, "-p", "Reply with OK", "--output-format", "stream-json", "--verbose",
+ "--model", "haiku", "--max-turns", "1"] + list(extra_args)
+ with tempfile.TemporaryDirectory(prefix="te-probe-") as cwd:
+ p = subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL, text=True, env=env, cwd=cwd)
+ tools, version = None, None
+ deadline = time.time() + timeout
+ try:
+ for line in p.stdout:
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "system" and ev.get("subtype") == "init":
+ tools, version = ev.get("tools"), ev.get("claude_code_version")
+ break
+ if time.time() > deadline:
+ break
+ finally:
+ p.kill()
+ p.wait()
+ return tools, version
+
+
+class _Capture(BaseHTTPRequestHandler):
+ """A chat-completions endpoint that records the request and answers "OK"."""
+ def log_message(self, *a):
+ pass
+
+ def _send(self, body, ctype="application/json"):
+ data = body.encode() if isinstance(body, str) else body
+ self.send_response(200)
+ self.send_header("Content-Type", ctype)
+ self.send_header("Content-Length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+
+ def do_GET(self):
+ self._send(json.dumps({"object": "list", "data": [{"id": "gpt-5.4", "object": "model"}]}))
+
+ def do_POST(self):
+ n = int(self.headers.get("Content-Length") or 0)
+ try:
+ body = json.loads(self.rfile.read(n) or b"{}")
+ except ValueError:
+ body = {}
+ self.server.captured.append(body)
+ msg = {"role": "assistant", "content": "OK"}
+ usage = {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}
+ if body.get("stream"):
+ chunks = [{"choices": [{"index": 0, "delta": msg, "finish_reason": None}]},
+ {"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]},
+ {"choices": [], "usage": usage}]
+ self._send("".join(f"data: {json.dumps(dict(c, object='chat.completion.chunk'))}"
+ "\n\n" for c in chunks) + "data: [DONE]\n\n",
+ "text/event-stream")
+ else:
+ self._send(json.dumps({"object": "chat.completion", "usage": usage, "choices": [
+ {"index": 0, "message": msg, "finish_reason": "stop"}]}))
+
+
+def copilot_capture(extra_args=(), cwd=None, cli="copilot", timeout=90):
+ """The first request a fresh `copilot -p` session sends, captured on localhost.
+
+ Copilot's BYOK mode is pointed at a scripted endpoint in this process, so the request --
+ system prompt, tool schemas -- is recorded exactly as Copilot builds it, and no model,
+ credential or quota is touched. Returns the request body, or None."""
+ srv = ThreadingHTTPServer(("127.0.0.1", 0), _Capture)
+ srv.captured = []
+ th = threading.Thread(target=srv.serve_forever, daemon=True)
+ th.start()
+ # The person's own model, when the environment names one: Copilot shapes its built-in tools
+ # by model family (`edit`/`create` for Claude models, `apply_patch` for GPT ones).
+ from .sidecall import copilot_model
+ # A router's `provider/` prefix (OpenRouter: `anthropic/claude-haiku-4.5`) is not part of
+ # the id Copilot recognises.
+ model = (copilot_model() or "gpt-5.4").split("/", 1)[-1]
+ env = {k: v for k, v in os.environ.items()
+ if not k.startswith("COPILOT_PROVIDER_") and k != "COPILOT_MODEL"}
+ env.update(TE_SIDECALL="1", COPILOT_PROVIDER_TYPE="openai",
+ COPILOT_PROVIDER_BASE_URL=f"http://127.0.0.1:{srv.server_address[1]}/v1",
+ COPILOT_PROVIDER_API_KEY="te-local-probe", COPILOT_MODEL=model,
+ COPILOT_PROVIDER_WIRE_API="completions", NO_PROXY="127.0.0.1,localhost",
+ no_proxy="127.0.0.1,localhost")
+ tmp = None
+ if cwd is None:
+ tmp = tempfile.mkdtemp(prefix="te-probe-")
+ cwd = tmp
+ try:
+ subprocess.run([cli, "-p", "Reply with OK", "--no-auto-update", "--no-color"]
+ + list(extra_args), cwd=cwd, env=env, capture_output=True, text=True,
+ timeout=timeout, stdin=subprocess.DEVNULL)
+ except (OSError, subprocess.SubprocessError):
+ pass
+ finally:
+ srv.shutdown()
+ srv.server_close()
+ if tmp:
+ shutil.rmtree(tmp, ignore_errors=True)
+ return srv.captured[0] if srv.captured else None
+
+
+def _request_tools(body):
+ return [(t.get("function") or {}).get("name") or t.get("name") or t.get("type")
+ for t in (body or {}).get("tools") or []]
+
+
+def _copilot_version(cli="copilot"):
+ try:
+ out = subprocess.run([cli, "--version"], capture_output=True, text=True, timeout=30)
+ return (out.stdout or "").strip().splitlines()[0]
+ except (OSError, subprocess.SubprocessError, IndexError):
+ return None
+
+
+def refresh_catalog(state_root, cli=None, host="claude"):
+ """Record the tool list a fresh session loads on this machine."""
+ if host == "copilot":
+ body = copilot_capture(cli=cli or "copilot")
+ tools = [t for t in _request_tools(body) if t]
+ if not tools:
+ raise RuntimeError("could not capture the tool list of a `copilot -p` request")
+ path = live_adapter.catalog_path(state_root, "copilot")
+ _write_json(path, {"tools": sorted(tools), "copilot_version": _copilot_version(),
+ "probed_at": time.time()})
+ return path, sorted(tools)
+ if host != "claude":
+ raise RuntimeError(f"no catalog probe for {host}")
+ tools, version = _init_tools([], cli=cli or "claude")
+ if not tools:
+ raise RuntimeError("could not read the tool list from a `claude -p` init event")
+ path = live_adapter.catalog_path(state_root, "claude")
+ _write_json(path, {"tools": sorted(tools), "claude_code_version": version,
+ "probed_at": time.time()})
+ return path, sorted(tools)
+
+
+def _prompt_tokens(extra_args, cli="claude", timeout=180):
+ env = dict(os.environ, TE_SIDECALL="1")
+ cmd = [cli, "-p", "Reply with exactly: OK", "--output-format", "json", "--model", "haiku",
+ "--max-turns", "1"] + list(extra_args)
+ with tempfile.TemporaryDirectory(prefix="te-measure-") as cwd:
+ out = subprocess.run(cmd, capture_output=True, text=True, env=env, cwd=cwd,
+ timeout=timeout)
+ data = json.loads(out.stdout or "{}")
+ u = data.get("usage") or {}
+ return (sum(int(u.get(k) or 0) for k in ("input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens")),
+ data.get("total_cost_usd"))
+
+
+def _codex_prompt_chars(overrides, cli="codex", timeout=60):
+ cmd = [cli, "debug", "prompt-input"]
+ for table, key, value in overrides:
+ cmd += ["-c", f"{table}.{key}={value}" if table else f"{key}={value}"]
+ with tempfile.TemporaryDirectory(prefix="te-measure-") as cwd:
+ out = subprocess.run(cmd + ["OK"], capture_output=True, text=True, cwd=cwd,
+ timeout=timeout, env=dict(os.environ, TE_SIDECALL="1"))
+ return len(out.stdout or "") if out.returncode == 0 else None
+
+
+def measure(tools, cli=None, host="claude"):
+ """Request size with and without `tools` removed.
+
+ Claude Code: prompt tokens of two small requests. Codex and Copilot: characters of the
+ rendered request, and a ~4-characters-per-token estimate, at no cost."""
+ if not tools:
+ return None
+ if host == "codex":
+ toggles = [t for c in tools for t in CODEX_TOGGLES.get(c, ())]
+ full = _codex_prompt_chars([], cli=cli or "codex")
+ lean = _codex_prompt_chars(toggles, cli=cli or "codex")
+ if full is None or lean is None:
+ raise RuntimeError("`codex debug prompt-input` failed")
+ return {"capabilities": list(tools), "prompt_input_chars_full": full,
+ "prompt_input_chars_lean": lean, "saved_chars_per_request": full - lean,
+ "saved_tokens_per_request_est": (full - lean) // 4,
+ "note": "prompt input only (developer/user items); tool schemas not included",
+ "probe_cost_usd": 0.0}
+ if host == "copilot":
+ args = [a for s in tools for a in ("--disable-mcp-server", s)]
+ full = copilot_capture(cli=cli or "copilot")
+ lean = copilot_capture(args, cli=cli or "copilot")
+ if not full or not lean:
+ raise RuntimeError("could not capture a `copilot -p` request")
+ size = lambda b: len(json.dumps(b.get("tools"))) + len(json.dumps(b.get("messages")))
+ return {"mcp_servers": list(tools),
+ "tools_full": len(_request_tools(full)), "tools_lean": len(_request_tools(lean)),
+ "request_chars_full": size(full), "request_chars_lean": size(lean),
+ "saved_chars_per_request": size(full) - size(lean),
+ "saved_tokens_per_request_est": (size(full) - size(lean)) // 4,
+ "probe_cost_usd": 0.0}
+ cli = cli or "claude"
+ full, c1 = _prompt_tokens([], cli=cli)
+ lean, c2 = _prompt_tokens(["--disallowed-tools", ",".join(tools)], cli=cli)
+ return {"tools": list(tools), "prompt_tokens_full": full, "prompt_tokens_lean": lean,
+ "saved_per_request": full - lean,
+ "probe_cost_usd": round((c1 or 0) + (c2 or 0), 4)}
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 0000000..4a9df13
--- /dev/null
+++ b/tests/__init__.py
@@ -0,0 +1 @@
+"""Make the tests/ directory importable so `from fixtures import ...` works."""
diff --git a/tests/conftest.py b/tests/conftest.py
new file mode 100644
index 0000000..464475c
--- /dev/null
+++ b/tests/conftest.py
@@ -0,0 +1,9 @@
+"""pytest fixtures shared across the offline suite."""
+import os
+import sys
+
+_TESTS = os.path.dirname(os.path.abspath(__file__))
+_REPO = os.path.dirname(_TESTS)
+for p in (_REPO, _TESTS):
+ if p not in sys.path:
+ sys.path.insert(0, p)
diff --git a/tests/fixtures/__init__.py b/tests/fixtures/__init__.py
new file mode 100644
index 0000000..f2b6f5a
--- /dev/null
+++ b/tests/fixtures/__init__.py
@@ -0,0 +1,60 @@
+"""Tiny synthetic fixtures for the offline test suite (no Docker, no datasets)."""
+import json
+import os
+
+
+def write_claude_transcript(path, calls):
+ """Write a minimal Claude-schema transcript.jsonl.
+ `calls` is a list of (tool_name, input_dict, result_text)."""
+ lines = []
+ for i, (name, inp, res) in enumerate(calls):
+ tid = f"t{i}"
+ lines.append({"message": {"content": [
+ {"type": "tool_use", "id": tid, "name": name, "input": inp}]}})
+ lines.append({"message": {"content": [
+ {"type": "tool_result", "tool_use_id": tid, "content": res}]}})
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines))
+ return path
+
+
+def write_codex_transcript(path, items):
+ """Write a minimal Codex-schema transcript.jsonl.
+ `items` is a list of dicts merged into an `item.completed` event's `item`."""
+ lines = []
+ for i, item in enumerate(items):
+ it = {"id": f"i{i}", "status": "completed"}
+ it.update(item)
+ lines.append({"type": "item.completed", "item": it})
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines))
+ return path
+
+
+def run_dir_with(tmp_path, transcript_lines_writer):
+ """Make a run dir containing a transcript.jsonl written by the given writer."""
+ d = os.path.join(str(tmp_path), "run")
+ os.makedirs(d, exist_ok=True)
+ transcript_lines_writer(os.path.join(d, "transcript.jsonl"))
+ return d
+
+
+def write_claude_transcript_with_catalog(path, tools, calls):
+ """Claude transcript that also carries the `system/init` line naming the LOADED tools.
+
+ The tool catalog is only recoverable from that init line (trace.parse_catalog ->
+ reflect.parse_tool_catalog_claude), so a fixture without it cannot exercise the
+ evidence-untooling path at all: the stage would see an empty catalog forever and
+ correctly drop nothing.
+ """
+ import json as _json
+ lines = [{"type": "system", "subtype": "init", "tools": list(tools)}]
+ for i, (name, inp, res) in enumerate(calls):
+ tid = f"t{i}"
+ lines.append({"type": "assistant", "message": {"content": [
+ {"type": "tool_use", "id": tid, "name": name, "input": inp}]}})
+ lines.append({"type": "user", "message": {"content": [
+ {"type": "tool_result", "tool_use_id": tid, "content": res}]}})
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write("\n".join(_json.dumps(x) for x in lines))
+ return path
diff --git a/tests/test_adapters.py b/tests/test_adapters.py
new file mode 100644
index 0000000..a88bf13
--- /dev/null
+++ b/tests/test_adapters.py
@@ -0,0 +1,70 @@
+"""Adapters: DomainAdapter protocol conformance + env-var placeholder behavior."""
+import os
+
+import pytest
+
+from tokeneconomizer.adapters import (
+ DomainAdapter, SwebenchAdapter, WebBenchAdapter, CredDataAdapter,
+ ReposVulAdapter, CyberGymAdapter, WebArenaAdapter)
+
+
+ALL_CONCRETE = [SwebenchAdapter, CredDataAdapter, ReposVulAdapter, CyberGymAdapter]
+
+
+def test_swebench_is_domain_adapter():
+ assert isinstance(SwebenchAdapter(), DomainAdapter)
+
+
+def test_all_adapters_importable_and_named():
+ assert SwebenchAdapter().name == "swebench"
+ assert WebBenchAdapter("dom").name == "webbench"
+ assert CredDataAdapter().name == "creddata"
+ assert ReposVulAdapter().name == "reposvul"
+ assert CyberGymAdapter().name == "cybergym"
+ assert WebArenaAdapter().name == "webarena"
+
+
+def test_swebench_reading_methods_delegate():
+ a = SwebenchAdapter()
+ assert a.protected_tools() is not None
+ assert a.inapplicable_tools() is not None
+ # codex default catalog is available without a transcript
+ assert "shell" in a.parse_catalog("/nonexistent", "codex")
+
+
+def test_no_hardcoded_dataset_roots():
+ # Constructing without env vars must not bake in a host path; the error only fires
+ # when the task spec is actually requested.
+ for env in ("TOKENECONOMIZER_CREDDATA_ROOT", "TOKENECONOMIZER_REPOSVUL_ROOT"):
+ os.environ.pop(env, None)
+ with pytest.raises(RuntimeError):
+ CredDataAdapter().repo_ids()
+ with pytest.raises(RuntimeError):
+ ReposVulAdapter().instance_ids()
+
+
+def test_swebench_problem_statements_requires_config():
+ os.environ.pop("TOKENECONOMIZER_SWEBENCH_TASKS", None)
+ with pytest.raises(RuntimeError):
+ SwebenchAdapter().problem_statements()
+
+
+def test_swebench_problem_statements_from_file(tmp_path):
+ import json
+ p = tmp_path / "tasks.json"
+ p.write_text(json.dumps([
+ {"instance_id": "django-1",
+ "problem_statement": "see https://github.com/django/django/pull/42 for the fix"}]))
+ ps = SwebenchAdapter(tasks_path=str(p)).problem_statements()
+ assert "django-1" in ps
+ assert "[link redacted]" in ps["django-1"] # gold URL was redacted
+
+
+def test_creddata_classify_domain_differs():
+ assert SwebenchAdapter().classify_domain() is None
+ assert CredDataAdapter().classify_domain() is not None
+
+
+def test_webarena_stub_raises():
+ with pytest.raises(NotImplementedError):
+ WebArenaAdapter().load_trajectory("x")
diff --git a/tests/test_codex_injection.py b/tests/test_codex_injection.py
new file mode 100644
index 0000000..1af685b
--- /dev/null
+++ b/tests/test_codex_injection.py
@@ -0,0 +1,290 @@
+"""Codex-native injection: environment facts from generic detector families.
+
+Every scenario is a synthetic stream of tasks (Codex transcript schema unless stated). The
+assertions pin the four properties the mechanism exists for: facts FORM from recurring
+evidence, FIRE only while the waste recurs, ABSTAIN where the evidence says otherwise, and
+cost nothing (`opt_tokens == 0`).
+"""
+import json
+import os
+
+from tokeneconomizer.mechanisms import codex_injection as ci
+from tokeneconomizer.mechanisms.codex_injection import EnvFactsDistiller
+from tokeneconomizer.mechanisms.injection import PrefixStage
+from tokeneconomizer.config import InjectionConfig, EconomizerConfig, needs_optimizer
+from fixtures import write_codex_transcript, write_claude_transcript
+
+BASE = "/opt/env/base/bin/python"
+PROJ = "/opt/env/proj/bin/python"
+
+
+def _cmd(command, output, code):
+ return {"type": "command_execution", "command": f'/bin/bash -lc "{command}"',
+ "aggregated_output": output, "exit_code": code}
+
+
+def _edit(path):
+ return {"type": "file_change", "changes": [{"path": path, "kind": "update"}]}
+
+
+def _task(tmp_path, i, items):
+ p = str(tmp_path / f"t{i}.jsonl")
+ write_codex_transcript(p, items)
+ return p
+
+
+def _broken_env_task(tmp_path, i):
+ """Bare runner fails on the base interpreter; the explicit path works."""
+ return _task(tmp_path, i, [
+ _edit(f"pkg/mod{i}.py"),
+ _cmd("python -m pytest tests/test_a.py -q", f"{BASE}: No module named pytest", 1),
+ _cmd(f"{PROJ} -m pytest tests/test_a.py -q", "1 passed in 0.2s", 0),
+ ])
+
+
+def _stream(tmp_path, n, maker, d=None):
+ d = d or EnvFactsDistiller("acme/widget")
+ for i in range(n):
+ d.render()
+ d.update(maker(tmp_path, i), iid=f"w-{i}", resolved=True)
+ return d
+
+
+# ---- observations ---------------------------------------------------------------------
+def test_classify_forms_and_intents():
+ obs = ci.classify_command(
+ '/bin/bash -lc "source /opt/env/bin/activate proj && python -m pytest tests -q"')
+ assert [(o.intent, o.form) for o in obs] == [
+ ("run_tests", "source activate proj && python -m pytest")]
+ obs = ci.classify_command("conda run -n proj python - <<'PY'\nimport x\nPY")
+ assert [(o.intent, o.form) for o in obs] == [("run_interpreter", "conda run -n proj python")]
+ obs = ci.classify_command("PYTHONPATH=/testbed ./tests/runtests.py auth_tests -v 1")
+ assert [(o.intent, o.form) for o in obs] == [("run_tests", "python tests/runtests.py")]
+ obs = ci.classify_command(f"{PROJ} bin/test sympy/core/tests/test_a.py")
+ assert [(o.intent, o.form) for o in obs] == [("run_tests", f"{PROJ} bin/test")]
+ obs = ci.classify_command("git remote -v; git tag --contains HEAD | head; git log --oneline -3")
+ assert [(o.intent, o.probe) for o in obs] == [("probe_vcs_refs", "remote"),
+ ("probe_vcs_refs", "tag")]
+ obs = ci.classify_command("curl -sS https://api.example.org/x | head -3")
+ assert obs[0].intent == "fetch_remote" and obs[0].hosts == ("api.example.org",)
+ # a plain `git tag` lists ALL tags, past ones included: not a beyond-HEAD probe
+ assert ci.classify_command("git tag | tail -5") == []
+ # heredoc bodies never become commands
+ assert ci.classify_command("python - <<'PY'\npytest_is_a_word = 1\nPY")[0].intent == \
+ "run_interpreter"
+
+
+def test_error_classes_are_environmental_only():
+ assert ci.error_class("ModuleNotFoundError: No module named 'x'") == "module_missing"
+ assert ci.error_class("bash: pytest: command not found") == "command_missing"
+ assert ci.error_class("curl: (7) Failed to connect to h port 443: Connection refused") == \
+ "connection_refused"
+ assert ci.error_class("AssertionError: 1 != 2\nFAILED tests/test_a.py::t") is None
+
+
+# ---- FormRepair -------------------------------------------------------------------------
+def test_interpreter_fact_forms_fires_and_renders(tmp_path):
+ d = _stream(tmp_path, 3, _broken_env_task)
+ assert d.render() == "" # ACTIVE needs 3 drifted confirms AND 4 exercised
+ d.update(_broken_env_task(tmp_path, 3), iid="w-3", resolved=True)
+ text = d.render()
+ assert text.startswith(ci.HEADER)
+ assert PROJ in text and "python -m pytest …` fails here" in text
+ assert d.opt_tokens == 0
+ f = d.facts["form_repair:run_tests:python -m pytest"]
+ assert f.state == "ACTIVE" and f.life.fire_num == 4 and f.life.fire_den == 4
+ assert ci.lint_ok(text)
+
+
+def test_clean_environment_forms_nothing(tmp_path):
+ def clean(tmp_path, i):
+ return _task(tmp_path, i, [
+ _edit(f"pkg/m{i}.py"),
+ _cmd("python -m pytest tests/test_a.py -q", "1 passed", 0)])
+ d = _stream(tmp_path, 8, clean)
+ assert d.facts == {} and d.render() == ""
+
+
+def test_prevented_waste_still_counts_as_firing(tmp_path):
+ d = _stream(tmp_path, 4, _broken_env_task)
+ assert d.render() != "" # injected for the next task
+ # the agent followed the fact: used the working form straight away, nothing failed
+ d.update(_task(tmp_path, 9, [_edit("pkg/z.py"),
+ _cmd(f"{PROJ} -m pytest tests/test_z.py -q", "3 passed", 0)]),
+ iid="w-9", resolved=True)
+ f = d.facts["form_repair:run_tests:python -m pytest"]
+ assert f.life.fire_num == 5 and f.life.fire_den == 5
+ assert d.render() != ""
+
+
+def test_bare_runner_success_contradicts_and_retires(tmp_path):
+ d = _stream(tmp_path, 4, _broken_env_task)
+ assert d.render() != ""
+ # environment fixed: the bare runner now works, task after task
+ def fixed(tmp_path, i):
+ return _task(tmp_path, i, [_edit(f"pkg/f{i}.py"),
+ _cmd("python -m pytest tests/test_a.py -q", "1 passed", 0)])
+ for i in range(20, 23):
+ d.render()
+ d.update(fixed(tmp_path, i), iid=f"w-{i}", resolved=True)
+ f = d.facts["form_repair:run_tests:python -m pytest"]
+ assert f.state == "RETIRED" and d.render() == ""
+
+
+def test_bare_one_liner_success_is_not_a_contradiction(tmp_path):
+ """A pure-Python package imports from the working tree under any interpreter, so a
+ successful bare `python -c` proves nothing about the environment."""
+ d = _stream(tmp_path, 4, _broken_env_task)
+ d.update(_task(tmp_path, 9, [_edit("pkg/q.py"),
+ _cmd("python -c 'import pkg; print(pkg.__version__)'", "1.0", 0),
+ _cmd(f"{PROJ} -m pytest tests -q", "2 passed", 0)]),
+ iid="w-9", resolved=True)
+ assert d.facts["form_repair:run_tests:python -m pytest"].state == "ACTIVE"
+
+
+# ---- EmptyProbe -------------------------------------------------------------------------
+def test_history_fact_forms_only_when_probes_return_nothing(tmp_path):
+ def empty(tmp_path, i):
+ return _task(tmp_path, i, [_edit(f"pkg/e{i}.py"),
+ _cmd("git remote -v; git tag --contains HEAD | head", "", 0),
+ _cmd(f"{PROJ} -m pytest tests -q", "1 passed", 0)])
+ d = _stream(tmp_path, 4, empty)
+ assert "history ends at the checked-out commit" in d.render()
+
+ def leaky(tmp_path, i):
+ return _task(tmp_path, i, [_edit(f"pkg/l{i}.py"),
+ _cmd("git tag --contains HEAD | head", "0.22\n0.22.1\n0.23.0", 0)])
+ d2 = _stream(tmp_path, 4, leaky, EnvFactsDistiller("acme/leaky"))
+ assert d2.render() == ""
+ assert d2.facts["empty_probe:vcs_refs"].state == "RETIRED"
+
+
+# ---- Boundary ---------------------------------------------------------------------------
+def test_refused_hosts_form_a_boundary_fact_and_a_reachable_host_does_not(tmp_path):
+ def refused(tmp_path, i):
+ return _task(tmp_path, i, [
+ _edit(f"pkg/n{i}.py"),
+ _cmd("curl -sS https://raw.example.org/a/b.py",
+ "curl: (7) Failed to connect to raw.example.org port 443: Connection refused", 7),
+ _cmd("curl -sS https://api.example.org/search?q=x",
+ "curl: (7) Failed to connect to api.example.org port 443: Connection refused", 7)])
+ d = _stream(tmp_path, 4, refused)
+ text = d.render()
+ assert "raw.example.org" in text and "api.example.org" in text and "refused" in text
+ # a later successful fetch from a listed host is a contradiction (hard kill)
+ d.update(_task(tmp_path, 9, [_cmd("curl -sS https://api.example.org/ok", "{\"ok\": true}", 0)]),
+ iid="w-9", resolved=True)
+ assert d.facts["boundary:fetch_remote"].state == "RETIRED"
+
+
+# ---- rendering discipline ---------------------------------------------------------------
+def test_lint_refuses_completion_advice():
+ assert not ci.lint_ok("You are done once the tests pass; skip the rest.")
+ assert ci.lint_ok("`python …` fails here with command not found; `/x/bin/python …` works.")
+
+
+def test_block_is_capped_in_facts_and_tokens(tmp_path):
+ d = EnvFactsDistiller("acme/widget")
+ for i in range(7):
+ f = ci.EnvFact(f"form_repair:run_tests:tool{i}", "form_repair", False,
+ {"intent": "run_tests", "a": f"tool{i}",
+ "b_counts": {f"/very/long/path/number/{i}/that/keeps/going/bin/tool{i}": 3},
+ "error_counts": {"command_missing": 3}})
+ f.life.state = "ACTIVE"; f.life.fire_num = f.life.fire_den = 10
+ d.facts[f.id] = f
+ text = d.render()
+ assert text and len(text) <= ci.MAX_TOKENS * 4
+ assert text.count("\n- ") <= ci.MAX_FACTS
+
+
+def test_form_repair_facts_sharing_the_working_form_merge_into_one_line(tmp_path):
+ def two_forms(tmp_path, i):
+ return _task(tmp_path, i, [
+ _edit(f"pkg/m{i}.py"),
+ _cmd("pytest tests/test_a.py -q", "bash: pytest: command not found", 127),
+ _cmd("python -m pytest tests/test_a.py -q", f"{BASE}: No module named pytest", 1),
+ _cmd(f"{PROJ} -m pytest tests/test_a.py -q", "1 passed", 0)])
+ d = _stream(tmp_path, 4, two_forms)
+ text = d.render()
+ assert text.count("\n- ") == 1 and "pytest …`" in text and "python -m pytest …`" in text
+
+
+# ---- persistence, other transcript schemas, stage wiring ---------------------------------
+def test_save_load_roundtrip(tmp_path):
+ d = _stream(tmp_path, 4, _broken_env_task)
+ d.save(str(tmp_path))
+ d2 = EnvFactsDistiller.load("acme/widget", str(tmp_path))
+ assert d2.now == 4 and d2.render() == d.render()
+ assert json.load(open(os.path.join(str(tmp_path), ci.STATE_SUBDIR, "acme__widget.json")))["opt_tokens"] == 0.0
+
+
+def test_claude_schema_yields_the_same_fact(tmp_path):
+ d = EnvFactsDistiller("acme/widget")
+ for i in range(4):
+ p = str(tmp_path / f"c{i}.jsonl")
+ write_claude_transcript(p, [
+ ("Edit", {"file_path": f"/testbed/pkg/c{i}.py"}, "ok"),
+ ("Bash", {"command": "python -m pytest tests/test_a.py -q"},
+ f"Exit code 1\n{BASE}: No module named pytest"),
+ ("Bash", {"command": f"{PROJ} -m pytest tests/test_a.py -q"}, "1 passed")])
+ d.render(); d.update(p, iid=f"c-{i}", resolved=True)
+ assert PROJ in d.render()
+
+
+def test_prefix_stage_envfacts_profile_needs_no_runner_and_reports_zero(tmp_path):
+ cfg = InjectionConfig(enabled=True, profile="envfacts")
+ stage = PrefixStage(cfg, "acme/widget", str(tmp_path), fresh=True, runner=None,
+ agent="codex")
+ assert stage.render() == ""
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ write_codex_transcript(os.path.join(rd, "transcript.jsonl"), [
+ _cmd("python -m pytest -q", f"{BASE}: No module named pytest", 1),
+ _cmd(f"{PROJ} -m pytest -q", "1 passed", 0)])
+ srow = {"resolved": True}
+ opt, reason = stage.after_task({"instance_id": "w-1"}, {}, rd, srow)
+ assert opt == 0 and reason == "envfacts"
+ assert srow["manual_facts"]["_profile"] == "envfacts"
+ assert "form_repair:run_tests:python -m pytest" in srow["manual_facts"]
+
+
+def test_the_llm_free_arm_is_registered_and_makes_no_paid_call():
+ """`codex-inject-envfacts` is the LLM-free ablation: the layer this module implements,
+ with no proposer. (The DEFAULT Codex injection arm has carried the paid proposer on top
+ since 2026-09-12 -- see tests/test_envfacts_profiles.py.)"""
+ from tokeneconomizer.arms import ARMS
+ cfg = ARMS["codex-inject-envfacts"].cfg()
+ assert cfg.injection.enabled and cfg.injection.profile == "envfacts"
+ assert cfg.injection.paid is False
+ assert needs_optimizer(cfg) is False
+ assert ARMS["codex-inject-envfacts"].mode == "manual4"
+ assert ARMS["codex-inject-untool"].cfg().tooling.codex_untool is True
+ assert needs_optimizer(EconomizerConfig(injection=InjectionConfig(enabled=True))) is True
+
+
+# ---- rtk's Codex mode puts `rtk` in front of every command the agent types ----------------
+def test_the_classifiers_see_through_rtks_command_prefix():
+ """`codex-trimech-rtk` stacks injection and the spiral on an agent that has been told to
+ type `rtk pytest`, `rtk git status`, `rtk proxy <cmd>`. Before this, the environment-fact
+ vocabulary (first token) and the spiral's test-run detector (anchored) both stopped
+ recognising a test run under that prefix -- no fact ever formed and no green was ever
+ trusted, on exactly the arm that co-activates them. Pinned as bare == prefixed, so a new
+ wrapper spelling has to be added to trace.RTK_PREFIX_RE and nowhere else."""
+ from tokeneconomizer.mechanisms import envfacts as ef, codex_spiral as cs
+ from tokeneconomizer.trace import strip_rtk_prefix
+ cases = ("python -m pytest tests/ -q", "pytest -q", "bin/test sympy/x.py", "K=1 pytest -q",
+ "pip install foo", "curl https://x", "git status", "rg -n foo src")
+ for bare in cases:
+ for pre in ("rtk ", "rtk proxy ", "rtk K=2 "):
+ wrapped = pre + bare
+ assert ([(o.intent, o.form) for o in ef.classify_command(wrapped)]
+ == [(o.intent, o.form) for o in ef.classify_command(bare)]), wrapped
+ assert cs.is_verification_command(wrapped) == cs.is_verification_command(bare), wrapped
+ # a compound command keeps its second segment visible too
+ assert cs.is_verification_command("cd x && rtk pytest -q")
+ assert cs.is_verification_command("rtk pytest -q") and not cs.is_verification_command("rtk git status")
+ # the stripper is exact: a bare `rtk`, or a command that merely starts with the letters,
+ # is not a wrapper
+ assert strip_rtk_prefix("rtk pytest -q") == "pytest -q"
+ assert strip_rtk_prefix("rtk proxy make") == "make"
+ assert strip_rtk_prefix("rtk") == "rtk" and strip_rtk_prefix("rtkfoo bar") == "rtkfoo bar"
+ assert strip_rtk_prefix("pytest") == "pytest"
diff --git a/tests/test_config.py b/tests/test_config.py
new file mode 100644
index 0000000..e4b6868
--- /dev/null
+++ b/tests/test_config.py
@@ -0,0 +1,45 @@
+"""Config: from_args maps the kept flags onto the three mechanism switches."""
+from types import SimpleNamespace
+
+from tokeneconomizer import EconomizerConfig
+from tokeneconomizer.config import InjectionConfig, ToolingConfig, SpiralConfig
+
+
+def test_defaults_all_off():
+ cfg = EconomizerConfig()
+ assert cfg.injection.enabled is False
+ assert cfg.tooling.enabled is False
+ assert cfg.spiral.enabled is False
+
+
+def test_from_args_generic_trimech():
+ args = SimpleNamespace(
+ inject=True, propose_every=5,
+ untool_evidence=3, untool_evidence_reaudit=12, untool_judge=True,
+ spiral_guard=True, spiral_judge=True, spiral_nudge=True)
+ cfg = EconomizerConfig.from_args(args)
+ assert cfg.injection.enabled and cfg.injection.propose_every == 5
+ assert cfg.tooling.evidence == 3 and cfg.tooling.judge is True
+ assert cfg.tooling.enabled is True
+ assert cfg.spiral.guard and cfg.spiral.judge and cfg.spiral.nudge
+ assert cfg.spiral.enabled is True
+
+
+def test_from_args_codex():
+ args = SimpleNamespace(codex_untool=True, codex_spiral=True)
+ cfg = EconomizerConfig.from_args(args)
+ assert cfg.tooling.codex_untool is True and cfg.tooling.enabled is True
+ assert cfg.spiral.codex_spiral is True and cfg.spiral.enabled is True
+ assert cfg.injection.enabled is False
+
+
+def test_propose_every_defaults_when_zero():
+ args = SimpleNamespace(inject=True, propose_every=0)
+ cfg = EconomizerConfig.from_args(args)
+ assert cfg.injection.propose_every == InjectionConfig.propose_every
+
+
+def test_partial_args_default_off():
+ # A sparse args object must not crash: unknown attrs default off.
+ cfg = EconomizerConfig.from_args(SimpleNamespace())
+ assert not cfg.injection.enabled and not cfg.tooling.enabled and not cfg.spiral.enabled
diff --git a/tests/test_envfacts_profiles.py b/tests/test_envfacts_profiles.py
new file mode 100644
index 0000000..088b2e7
--- /dev/null
+++ b/tests/test_envfacts_profiles.py
@@ -0,0 +1,146 @@
+"""The shared environment-fact layer: vocabulary plug-in, composition with the paid
+proposer, and the arms that carry each profile."""
+import os
+
+from tokeneconomizer.mechanisms import envfacts as ef
+from tokeneconomizer.mechanisms import codex_injection as ci
+from tokeneconomizer.mechanisms.injection import PrefixStage, ComposedDistiller
+from tokeneconomizer.config import InjectionConfig, needs_optimizer
+from tokeneconomizer.adapters.swebench import SwebenchAdapter
+from fixtures import write_codex_transcript
+
+BASE = "/opt/env/base/bin/python"
+PROJ = "/opt/env/proj/bin/python"
+
+
+def _cmd(command, output, code):
+ return {"type": "command_execution", "command": f'/bin/bash -lc "{command}"',
+ "aggregated_output": output, "exit_code": code}
+
+
+def _edit(path):
+ return {"type": "file_change", "changes": [{"path": path, "kind": "update"}]}
+
+
+def _broken(tmp_path, i):
+ p = str(tmp_path / f"t{i}.jsonl")
+ write_codex_transcript(p, [
+ _edit(f"pkg/mod{i}.py"),
+ _cmd("python -m pytest tests/test_a.py -q", f"{BASE}: No module named pytest", 1),
+ _cmd(f"{PROJ} -m pytest tests/test_a.py -q", "1 passed", 0)])
+ return p
+
+
+# ---- the Codex module is a shim over the shared layer ------------------------------
+def test_codex_module_reexports_the_shared_layer():
+ assert ci.EnvFactsDistiller is ef.EnvFactsDistiller
+ assert ci.observe_task is ef.observe_task
+ assert ci.CODEX_DEFAULT_PROFILE == "envfacts+distiller" # since 2026-09-12
+ assert ci.CODEX_FREE_PROFILE == "envfacts"
+
+
+# ---- vocabulary plug-in ----------------------------------------------------------------
+def test_node_vocabulary_reads_a_node_repository():
+ obs = ef.classify_command("npx jest tests/ --ci", vocab=ef.NODE_WEB)
+ assert [(o.intent, o.form) for o in obs] == [("run_tests", "npx jest")]
+ obs = ef.classify_command("node scripts/build.js && npm test", vocab=ef.NODE_WEB)
+ assert [(o.intent, o.form) for o in obs] == [("run_interpreter", "node"),
+ ("run_tests", "npm test")]
+ obs = ef.classify_command("npm install left-pad", vocab=ef.NODE_WEB)
+ assert obs[0].intent == "fetch_remote"
+ # the Python vocabulary does not know jest; the Node one does not know pytest
+ assert ef.classify_command("npx jest", vocab=ef.PYTHON_GIT) == [] or \
+ ef.classify_command("npx jest", vocab=ef.PYTHON_GIT)[0].intent != "run_tests"
+ assert ef.classify_command("pytest -q", vocab=ef.NODE_WEB) == []
+ assert ef.error_class("Error: Cannot find module 'left-pad'") == "module_missing"
+ assert ef.error_class("connect ECONNREFUSED 127.0.0.1:443") == "connection_refused"
+
+
+def test_adapters_supply_a_vocabulary():
+ assert SwebenchAdapter().env_vocabulary().name == "python-git"
+ from tokeneconomizer.adapters.webbench import WebBenchAdapter
+ assert WebBenchAdapter.env_vocabulary(None).name == "node-web"
+
+
+def test_distiller_records_its_vocabulary(tmp_path):
+ d = ef.EnvFactsDistiller("acme/widget", vocab=ef.NODE_WEB)
+ d.update(_broken(tmp_path, 0), iid="w-0")
+ d.save(str(tmp_path))
+ import json
+ saved = json.load(open(os.path.join(str(tmp_path), ef.STATE_SUBDIR, "acme__widget.json")))
+ assert saved["vocab"] == "node-web"
+ assert d.debug_row()["_vocab"] == "node-web"
+
+
+# ---- composition with the paid proposer -----------------------------------------------
+class _RecordingRunner:
+ """A proposer stand-in: records every prompt, authors nothing, bills 7 eff tokens."""
+ def __init__(self):
+ self.prompts = []
+
+ def __call__(self, prompt, model="agent"):
+ self.prompts.append(prompt)
+ return '{"extractors": [], "facts": []}', 7.0
+
+
+def test_composed_profile_renders_both_layers_and_briefs_the_proposer(tmp_path):
+ runner = _RecordingRunner()
+ cfg = InjectionConfig(enabled=True, profile="envfacts+distiller", propose_every=1)
+ stage = PrefixStage(cfg, "acme/widget", str(tmp_path), fresh=True, runner=runner,
+ agent="codex", adapter=SwebenchAdapter())
+ assert isinstance(stage.distiller, ComposedDistiller)
+ total_opt = 0
+ for i in range(6):
+ block = stage.render()
+ rd = str(tmp_path / f"run{i}"); os.makedirs(rd)
+ os.replace(_broken(tmp_path, i), os.path.join(rd, "transcript.jsonl"))
+ opt, reason = stage.after_task({"instance_id": f"w-{i}"}, {}, rd, {"resolved": True})
+ total_opt += opt
+ assert reason == "envfacts+manual-distill"
+ # the free layer formed and renders first
+ assert block.startswith(ef.HEADER) and PROJ in block
+ # the proposer was called every task (cadence 1) and billed through the ledger seam
+ assert len(runner.prompts) >= 5 and total_opt == 7.0 * len(runner.prompts)
+ # once the environment fact was injectable, the proposer was told not to restate it
+ assert any("ALREADY injected" in p and "fails here" in p for p in runner.prompts[3:])
+ # telemetry names the profile and carries both layers' facts
+ srow = {"resolved": True}
+ rd = str(tmp_path / "run_last"); os.makedirs(rd)
+ os.replace(_broken(tmp_path, 9), os.path.join(rd, "transcript.jsonl"))
+ stage.after_task({"instance_id": "w-9"}, {}, rd, srow)
+ assert srow["manual_facts"]["_profile"] == "envfacts+distiller"
+ assert any(k.startswith("form_repair:") for k in srow["manual_facts"])
+
+
+def test_envfacts_alone_still_needs_no_runner(tmp_path):
+ cfg = InjectionConfig(enabled=True, profile="envfacts")
+ stage = PrefixStage(cfg, "acme/widget", str(tmp_path), fresh=True, runner=None,
+ agent="codex", adapter=SwebenchAdapter())
+ assert stage.render() == "" and stage.reason == "envfacts"
+
+
+# ---- arms ------------------------------------------------------------------------------
+def test_codex_arms_carry_the_intended_profiles():
+ """Since 2026-09-12 the Codex injection DEFAULT is the composed profile.
+
+ The free layer's value on this benchmark came from the inactive-project-environment
+ defect; with that fixed it abstains, so `codex-inject` carries the paid proposer and
+ the LLM-free variant is the named ablation `codex-inject-envfacts`.
+ """
+ from tokeneconomizer.arms import ARMS, ARM_ORDER
+ from tokeneconomizer.mechanisms import codex_injection as ci
+ default = ARMS["codex-inject"].cfg().injection
+ assert default.profile == ci.CODEX_DEFAULT_PROFILE == "envfacts+distiller"
+ assert default.paid and default.propose_every == 10
+ assert needs_optimizer(ARMS["codex-inject"].cfg()) is True
+ # the untool stack follows the default; the ablation does not
+ assert ARMS["codex-inject-untool"].cfg().injection.profile == "envfacts+distiller"
+ assert ARMS["codex-inject-untool"].cfg().tooling.codex_untool is True
+ free = ARMS["codex-inject-envfacts"].cfg().injection
+ assert free.profile == ci.CODEX_FREE_PROFILE == "envfacts" and not free.paid
+ assert needs_optimizer(ARMS["codex-inject-envfacts"].cfg()) is False
+ # the pre-2026-09-12 name of the default still resolves to the same configuration
+ assert ARMS["codex-inject-distiller"].cfg().injection == default
+ for name in ("codex-inject", "codex-inject-untool", "codex-inject-envfacts",
+ "codex-inject-distiller"):
+ assert ARMS[name].mode == "manual4" and name in ARM_ORDER
diff --git a/tests/test_facade.py b/tests/test_facade.py
new file mode 100644
index 0000000..4b6aeaf
--- /dev/null
+++ b/tests/test_facade.py
@@ -0,0 +1,126 @@
+"""Facade: the Economizer drives all three mechanisms end-to-end (offline)."""
+import os
+
+from tokeneconomizer import Economizer, EconomizerConfig
+from tokeneconomizer.config import InjectionConfig, ToolingConfig, SpiralConfig
+from tokeneconomizer.adapters import SwebenchAdapter
+from fixtures import write_claude_transcript, write_codex_transcript
+
+
+def _runner(prompt, model="agent"):
+ return ('{"extractors":[],"facts":[]}', 5.0)
+
+
+def test_generic_trimech_end_to_end(tmp_path):
+ cfg = EconomizerConfig(
+ injection=InjectionConfig(enabled=True, propose_every=5),
+ tooling=ToolingConfig(evidence=3, judge=False),
+ spiral=SpiralConfig(guard=True, nudge=True))
+ econ = Economizer(cfg, agent="claude", repo="acme/widget",
+ results_root=str(tmp_path), fresh=True, runner=_runner)
+ task = {"instance_id": "w-1", "problem_statement": "fix"}
+ assert econ.injection_prefix(task) == "" # nothing formed yet
+ assert econ.build_tools(task, 0) == [] # no catalog learned yet
+ rk = econ.run_kwargs(task)
+ assert rk["spiral_guard"] is True
+
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ write_claude_transcript(os.path.join(rd, "transcript.jsonl"),
+ [("bash", {"command": "pytest"}, "ok")])
+ row = {"resolved": True, "num_turns": 10, "tokens_total": 1000,
+ "input_tokens": 100, "cache_read_input_tokens": 800,
+ "cache_creation_input_tokens": 50, "output_tokens": 50}
+ srow = {"resolved": True, "num_turns": 10}
+ opt, reason = econ.after_task(task, row, rd, srow)
+ assert reason == "manual-distill"
+ assert "manual_facts" in srow and "aborted" in srow
+
+
+def test_codex_arm_end_to_end(tmp_path):
+ cfg = EconomizerConfig(
+ injection=InjectionConfig(enabled=False),
+ tooling=ToolingConfig(codex_untool=True),
+ spiral=SpiralConfig(codex_spiral=True))
+ econ = Economizer(cfg, agent="codex", repo="pydata/xarray",
+ results_root=str(tmp_path), fresh=True)
+ task = {"instance_id": "x-1", "problem_statement": "fix concat"}
+ assert econ.build_tools(task, 0) == [] # task 1 full contract
+ rk = econ.run_kwargs(task)
+ assert "codex_spiral_policy" in rk
+
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ write_codex_transcript(os.path.join(rd, "transcript.jsonl"),
+ [{"type": "command_execution", "command": "pytest"}])
+ row = {"resolved": True, "num_turns": 30, "tokens_total": 5000,
+ "input_tokens": 500, "cache_read_input_tokens": 4000,
+ "cache_creation_input_tokens": 200, "output_tokens": 300,
+ "codex_spiral_nudges": 0}
+ srow = {"resolved": True, "num_turns": 30}
+ opt, reason = econ.after_task(task, row, rd, srow)
+ assert reason is None # injection disabled
+ assert srow["codex_untool_active"] is True
+ assert srow["codex_spiral_active"] is True
+ # task 2: web should have been demoted (loaded-and-unused on task 1)
+ assert "sealed_web" in econ.build_tools(task, 1)
+
+
+def test_codex_trimech_end_to_end(tmp_path):
+ """All three Codex mechanisms driven through the facade at once, offline.
+
+ Built from the REGISTRY rather than a hand-written config, so this is the arm a lane
+ actually runs. What it guards is that the three do not contend: injection renders a
+ prompt prefix, untooling freezes a capability mask, and the spiral stage hands the
+ runner its policy -- on the same task, in one `run_kwargs`, with all three telemetry
+ families landing on one stream row.
+ """
+ from tokeneconomizer.arms import ARMS
+ calls = []
+
+ def runner(prompt, model="agent"):
+ calls.append(prompt)
+ return '{"extractors":[],"facts":[]}', 3.0
+
+ econ = Economizer(ARMS["codex-trimech"].cfg(), agent="codex",
+ repo="pydata/xarray", results_root=str(tmp_path), fresh=True,
+ runner=runner, adapter=SwebenchAdapter())
+ task = {"instance_id": "x-1", "problem_statement": "fix concat"}
+
+ # one run_kwargs carries BOTH codex mechanisms' runner arguments, and they do not
+ # collide -- the capability mask and the spiral policy are separate keys.
+ rk = econ.run_kwargs(task)
+ assert "codex_config_capabilities" in rk and "codex_spiral_policy" in rk
+ assert econ.build_tools(task, 0) == [] # task 1 runs the full contract
+
+ # ten tasks, because the composed profile proposes at cadence 10 -- fewer would leave
+ # the paid half of the arm untested.
+ total = 0
+ for i in range(10):
+ task = {"instance_id": f"x-{i}", "problem_statement": "fix concat"}
+ # the per-task order the driver uses: prefix -> mask -> run_kwargs -> run -> hook
+ econ.injection_prefix(task)
+ econ.build_tools(task, i)
+ econ.run_kwargs(task)
+ rd = str(tmp_path / f"run{i}"); os.makedirs(rd)
+ write_codex_transcript(os.path.join(rd, "transcript.jsonl"), [
+ {"type": "file_change", "changes": [{"path": "xarray/core/concat.py",
+ "kind": "update"}]},
+ {"type": "command_execution",
+ "command": '/bin/bash -lc "python -m pytest xarray/tests/test_concat.py -q"',
+ "aggregated_output": "1 passed", "exit_code": 0}])
+ row = {"resolved": True, "num_turns": 30, "tokens_total": 5000,
+ "input_tokens": 500, "cache_read_input_tokens": 4000,
+ "cache_creation_input_tokens": 200, "output_tokens": 300,
+ "codex_spiral_nudges": 0, "codex_spiral_tool_turns": 12}
+ srow = {"resolved": True, "num_turns": 30}
+ opt, reason = econ.after_task(task, row, rd, srow)
+ total += opt
+ # all three mechanisms reported on the same row
+ assert srow["codex_untool_active"] is True
+ assert srow["codex_spiral_active"] is True
+ assert "manual_facts" in srow
+ assert reason and reason.startswith("envfacts") # the composed profile
+
+ # the proposer's spend crossed the ledger seam rather than being invisible
+ assert calls and total == 3.0 * len(calls)
+ # untooling still learns across tasks with the other two active
+ assert "sealed_web" in econ.build_tools(task, 10)
diff --git a/tests/test_injection.py b/tests/test_injection.py
new file mode 100644
index 0000000..989b875
--- /dev/null
+++ b/tests/test_injection.py
@@ -0,0 +1,70 @@
+"""Injection: the distiller lifecycle + the PrefixStage render/after_task cycle."""
+import os
+
+from tokeneconomizer.mechanisms.injection import Distiller, PrefixStage
+from tokeneconomizer.config import InjectionConfig
+from fixtures import write_claude_transcript
+
+
+def _facts_runner(payload):
+ """An optimizer runner returning a fixed proposer payload + 5 eff tokens."""
+ return lambda prompt, model="agent": (payload, 5.0)
+
+
+def test_distiller_empty_render_before_formation(tmp_path):
+ d = Distiller("acme/widget", runner=_facts_runner('{"extractors":[],"facts":[]}'))
+ tr = str(tmp_path / "t.jsonl")
+ write_claude_transcript(tr, [("bash", {"command": "pytest"}, "ok")])
+ d.update(tr, iid="widget-1", resolved=True)
+ assert d.render() == "" # nothing has fired/formed yet
+ assert d.now == 1
+
+
+def test_distiller_persist_roundtrip(tmp_path):
+ root = str(tmp_path)
+ d = Distiller("acme/widget", runner=_facts_runner('{"extractors":[],"facts":[]}'))
+ tr = str(tmp_path / "t.jsonl")
+ write_claude_transcript(tr, [("bash", {"command": "pytest"}, "ok")])
+ d.update(tr, iid="w-1", resolved=True)
+ d.save(root)
+ d2 = Distiller.load("acme/widget", root)
+ assert d2.now == d.now
+
+
+def test_distiller_grounds_and_fires_a_fact(tmp_path):
+ # A proposer fact whose confirm regex matches a repeated pytest form; after enough
+ # drifted confirmations it should promote and (once past the firing gate) inject.
+ payload = ('{"extractors":[],"facts":[{"id":"use-pytest","claim":"pytest is canonical",'
+ '"injection":"run tests with pytest","high_stakes":true,'
+ '"detector":{"scope":"ran_tests","confirm":"passed","fired":"ran_tests"},'
+ '"cite":["w-0","w-1"]}]}')
+ d = Distiller("acme/widget", runner=_facts_runner(payload))
+ d.last_propose = -1
+ # feed several tasks that each run pytest and pass, on disjoint edited files
+ import tokeneconomizer.mechanisms.injection as inj
+ inj.PROPOSE_EVERY = 1
+ for i in range(8):
+ tr = str(tmp_path / f"t{i}.jsonl")
+ write_claude_transcript(tr, [
+ ("edit", {"file_path": f"pkg/mod{i}.py"}, "ok"),
+ ("bash", {"command": "python -m pytest tests/test_x.py"}, "1 passed")])
+ d.update(tr, iid=f"w-{i}", resolved=True)
+ assert "use-pytest" in d.facts
+ # at least reached ACTIVE or accrued firing evidence
+ f = d.facts["use-pytest"]
+ assert f.life.fire_den >= 1
+
+
+def test_prefix_stage_after_task_reports_tokens(tmp_path):
+ root = str(tmp_path)
+ cfg = InjectionConfig(enabled=True, propose_every=5)
+ stage = PrefixStage(cfg, "acme/widget", root, fresh=True,
+ runner=_facts_runner('{"extractors":[],"facts":[]}'),
+ agent="claude")
+ rd = str(tmp_path / "run"); os.makedirs(rd)
+ write_claude_transcript(os.path.join(rd, "transcript.jsonl"),
+ [("bash", {"command": "pytest"}, "ok")])
+ srow = {"resolved": True}
+ opt, reason = stage.after_task({"instance_id": "w-1"}, {}, rd, srow)
+ assert reason == "manual-distill"
+ assert "manual_facts" in srow
diff --git a/tests/test_ledger.py b/tests/test_ledger.py
new file mode 100644
index 0000000..32a28a5
--- /dev/null
+++ b/tests/test_ledger.py
@@ -0,0 +1,40 @@
+"""Ledger: overhead(), effective_exec(), and the CostLedger accumulator."""
+from tokeneconomizer import ledger
+from tokeneconomizer.ledger import CostLedger, OVERHEAD_KEYS
+
+
+def test_overhead_sums_only_known_keys():
+ row = {"optimizer_tokens": 10, "spiral_judge_tokens": 5,
+ "untool_judge_tokens": 3, "not_an_overhead_key": 999}
+ assert ledger.overhead(row) == 18
+
+
+def test_overhead_ignores_non_numeric():
+ assert ledger.overhead({"optimizer_tokens": None, "spiral_judge_tokens": "x"}) == 0
+
+
+def test_all_overhead_keys_counted():
+ row = {k: 1 for k in OVERHEAD_KEYS}
+ assert ledger.overhead(row) == len(OVERHEAD_KEYS)
+
+
+def test_effective_exec_uses_breakdown():
+ # cache_read weighted at 0.10, output at 5.0 on the Claude curve.
+ row = {"input_tokens": 100, "cache_read_input_tokens": 1000,
+ "cache_creation_input_tokens": 0, "output_tokens": 10}
+ assert ledger.effective_exec(row) == 100 * 1.0 + 1000 * 0.10 + 10 * 5.0
+
+
+def test_effective_exec_falls_back_to_total():
+ assert ledger.effective_exec({"tokens_total": 777}) == 777
+
+
+def test_cost_ledger_amortizes():
+ cl = CostLedger()
+ cl.add({"tokens_total": 100, "optimizer_tokens": 20})
+ cl.add({"tokens_total": 200})
+ assert cl.cum_exec == 300
+ assert cl.cum_overhead == 20
+ assert cl.cum_total == 320
+ assert cl.amortized_per_task == 160
+ assert 0 < cl.overhead_frac < 1
diff --git a/tests/test_live_config.py b/tests/test_live_config.py
new file mode 100644
index 0000000..c9f81ce
--- /dev/null
+++ b/tests/test_live_config.py
@@ -0,0 +1,154 @@
+"""Settings a person controls: each mechanism on or off, and its hyperparameters by hand.
+
+Every setting is read on each hook call, so these drive real hook calls (`tests.test_live_hooks`)
+with the setting in the environment or the repository's config file, and check that the
+mechanism does what the setting says.
+"""
+import json
+import os
+
+import pytest
+
+from te_live import guard
+from te_live.config import LiveConfig, load_config, unset_config, write_config
+from te_live.paths import repo_identity, repo_state_dir
+from tests.test_live_hooks import (GREEN, _first, _turn, env, repo, run_session, # noqa: F401
+ stuck_calls, trap_session)
+
+
+# ---- parsing ------------------------------------------------------------------------------
[email protected]("key,on", [("inject", "envfacts"), ("untool", "propose"),
+ ("spiral", "nudge")])
+def test_each_mechanism_takes_a_plain_on_and_off(key, on):
+ cfg = LiveConfig()
+ assert cfg.set(key, "off") and getattr(cfg, key) == "off"
+ assert cfg.set(key, "on") and getattr(cfg, key) == on
+ assert cfg.set(key, False) and getattr(cfg, key) == "off"
+ assert not cfg.set(key, "sometimes") and getattr(cfg, key) == "off"
+
+
+def test_a_guard_threshold_is_typed_and_range_checked():
+ cfg = LiveConfig()
+ assert cfg.set("spiral.min_turns", "12") and cfg.spiral_thresholds["min_turns"] == 12
+ assert cfg.set("spiral.backstop_mult", "2") and cfg.spiral_thresholds["backstop_mult"] == 2.0
+ assert not cfg.set("spiral.min_turns", "12.5") # a count is a whole number
+ assert not cfg.set("spiral.stagnation", "1.5") # a share is between 0 and 1
+ assert not cfg.set("spiral.min_turns", "0")
+ assert not cfg.set("spiral.no_such_threshold", "3")
+ assert cfg.spiral_thresholds == {"min_turns": 12, "backstop_mult": 2.0}
+ assert len(cfg.warnings) == 4
+
+
+def test_set_and_unset_keep_thresholds_in_their_own_object(tmp_path):
+ path = str(tmp_path / "config.json")
+ write_config(path, {"spiral": "off", "spiral.min_turns": "12", "spiral.edit_churn": "6"})
+ assert json.load(open(path)) == {"spiral": "off",
+ "spiral_thresholds": {"edit_churn": 6, "min_turns": 12}}
+ with pytest.raises(ValueError):
+ write_config(path, {"spiral.min_turns": "-1"}) # nothing written on a rejection
+ unset_config(path, ["spiral.min_turns", "spiral"])
+ assert json.load(open(path)) == {"spiral_thresholds": {"edit_churn": 6}}
+ unset_config(path, ["spiral.edit_churn"])
+ assert json.load(open(path)) == {}
+ with pytest.raises(ValueError):
+ unset_config(path, ["spiral.nope"])
+
+
+def test_the_repository_file_beats_the_global_one_and_the_environment_beats_both(tmp_path):
+ root, repo_dir = tmp_path / "root", tmp_path / "repo"
+ write_config(str(root / "config.json"), {"spiral_nudges": "3", "spiral.min_turns": "30"})
+ write_config(str(repo_dir / "config.json"), {"spiral.min_turns": "10"})
+ cfg = load_config(str(root), str(repo_dir), env={})
+ assert cfg.spiral_nudges == 3 and cfg.spiral_thresholds == {"min_turns": 10}
+ cfg = load_config(str(root), str(repo_dir), env={
+ "TE_SPIRAL_NUDGES": "2", "TE_SPIRAL_THRESHOLDS": '{"min_turns": 8, "stagnation": 0.5}',
+ "TE_INJECT_MAX_FACTS": "2", "TE_SPIRAL": "off"})
+ assert cfg.spiral_nudges == 2 and cfg.inject_max_facts == 2 and cfg.spiral == "off"
+ assert cfg.spiral_thresholds == {"min_turns": 8, "stagnation": 0.5}
+ assert not cfg.warnings
+
+
+def test_the_fact_caps_default_to_the_cores_own():
+ from tokeneconomizer.mechanisms import envfacts
+ cfg = LiveConfig()
+ assert (cfg.inject_max_facts, cfg.inject_max_tokens) == (envfacts.MAX_FACTS,
+ envfacts.MAX_TOKENS)
+
+
+# ---- thresholds: preset, then learned, then pinned ----------------------------------------
+class _Store:
+ def __init__(self, d):
+ self.d = d
+
+ def read(self, name):
+ return self.d.get(name)
+
+
+def test_a_pinned_threshold_wins_over_the_preset_and_over_what_was_learned():
+ store = _Store({"envelope.json": {"resolved_turns": [6, 7, 8, 8, 9]}})
+ learned = guard.thresholds(guard.load_envelope(store))
+ assert learned["min_turns"] == 8 # learned below the preset's 20
+ tracker = guard.load_envelope(store, overrides={"min_turns": 30, "backstop_mult": 4.0})
+ thr = guard.thresholds(tracker, {"min_turns": 30, "backstop_mult": 4.0})
+ assert thr["min_turns"] == 30
+ assert thr["backstop_turns"] == pytest.approx(tracker.envelope * 4.0)
+
+
+def test_the_envelope_settings_reach_the_tracker():
+ store = _Store({"envelope.json": {"resolved_turns": [6, 8]}})
+ assert guard.load_envelope(store).envelope is None # two samples: not trusted by default
+ tracker = guard.load_envelope(store, overrides={"envelope_min_samples": 2,
+ "envelope_margin": 2.0, "envelope_pct": 50})
+ assert tracker.envelope == pytest.approx(7 * 2.0)
+
+
+def test_base_thresholds_show_the_preset_with_the_pinned_values_on_top():
+ thr = guard.base_thresholds("aggressive", {"min_turns": 9})
+ assert thr["min_turns"] == 9 and thr["backstop_mult"] == 1.8
+ assert "restart_budget" not in thr # not used by the live guard
+
+
+# ---- end to end, through the hooks --------------------------------------------------------
+def _warm(env, repo):
+ for k in range(5):
+ _turn(env, repo, "s", f"g{k}", GREEN)
+
+
+def _messages(outs):
+ return [o.get("systemMessage") or "" for o in outs]
+
+
+def test_a_guard_turned_off_in_the_repository_file_never_intervenes(env, repo):
+ _warm(env, repo)
+ path = os.path.join(repo_state_dir(repo_identity(repo, env), "claude", env), "..",
+ "config.json")
+ write_config(os.path.normpath(path), {"spiral": "off"})
+ assert all(o == {} for o in _turn(env, repo, "s", "stuck", stuck_calls(40)))
+
+
+def test_more_redirects_are_given_before_the_notice_when_asked_for(env, repo):
+ _warm(env, repo)
+ outs = _turn(dict(env, TE_SPIRAL_NUDGES="2"), repo, "s", "stuck", stuck_calls(60))
+ msgs = _messages(outs)
+ nudges = [i for i, m in enumerate(msgs) if "nudged" in m]
+ notice = _first(msgs, lambda m: "still looks stuck" in m)
+ assert len(nudges) == 2 and notice is not None and notice > nudges[-1]
+
+
+def test_a_higher_min_turns_holds_the_guard_back(env, repo):
+ _warm(env, repo)
+ outs = _turn(dict(env, TE_SPIRAL_THRESHOLDS='{"min_turns": 80}'), repo, "s", "stuck",
+ stuck_calls(40))
+ assert all(o == {} for o in outs) # default thresholds nudge this turn
+
+
+def test_injection_off_and_a_tight_fact_cap_keep_the_context_clean(env, repo, tmp_path):
+ for i in range(4):
+ run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ off = run_session(dict(env, TE_INJECT="off"), repo, "s4",
+ trap_session(str(tmp_path / "s4.jsonl"), repo, 4))
+ assert off == {} # the default injects here
+ tight = dict(env, TE_INJECT_MAX_TOKENS="5") # smaller than the block's header
+ run_session(tight, repo, "s5", trap_session(str(tmp_path / "s5.jsonl"), repo, 5))
+ after = run_session(tight, repo, "s6", trap_session(str(tmp_path / "s6.jsonl"), repo, 6))
+ assert "hookSpecificOutput" not in after
diff --git a/tests/test_live_core.py b/tests/test_live_core.py
new file mode 100644
index 0000000..731623a
--- /dev/null
+++ b/tests/test_live_core.py
@@ -0,0 +1,712 @@
+"""The live layer's plumbing: repository identity, state, config, session logs, host I/O.
+
+Offline and synthetic. The transcript fixtures reproduce the two properties of real session logs
+that the harness's own transcripts never had, and that the live readers exist to handle: Claude
+Code writes one entry PER CONTENT BLOCK, each repeating its message's usage, and Codex's rollout
+wraps exec-stream item fields in PascalCase `event_msg/item_completed` records.
+"""
+import json
+import os
+
+import pytest
+
+from tokeneconomizer import trace
+from te_live import config as live_config
+from te_live import hosts, paths, transcripts
+from te_live.store import Store
+
+
+# ---- repository identity ---------------------------------------------------------------
+def _git_repo(root, remote=None):
+ os.makedirs(os.path.join(root, ".git"), exist_ok=True)
+ cfg = "[core]\n\trepositoryformatversion = 0\n"
+ if remote:
+ cfg += f'[remote "origin"]\n\turl = {remote}\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n'
+ with open(os.path.join(root, ".git", "config"), "w") as fh:
+ fh.write(cfg)
+ return root
+
+
+def test_repo_identity_follows_the_remote_not_the_spelling(tmp_path):
+ a = _git_repo(str(tmp_path / "a"), "[email protected]:Owner/Repo.git")
+ b = _git_repo(str(tmp_path / "b" / "repo"), "https://github.com/owner/repo")
+ ra, rb = paths.repo_identity(a), paths.repo_identity(b)
+ assert ra.remote == rb.remote == "github.com/owner/repo"
+ assert ra.label == "owner/repo"
+ # the key carries the checkout's name for readability, and the remote for identity
+ assert ra.key.endswith(rb.key.split("-")[-1])
+
+
+def test_a_subdirectory_and_a_worktree_share_their_repository(tmp_path):
+ main = _git_repo(str(tmp_path / "main"), "https://github.com/o/r.git")
+ os.makedirs(os.path.join(main, "src", "pkg"))
+ wt_git = os.path.join(main, ".git", "worktrees", "feature")
+ os.makedirs(wt_git)
+ with open(os.path.join(wt_git, "commondir"), "w") as fh:
+ fh.write("../..")
+ wt = str(tmp_path / "feature")
+ os.makedirs(wt)
+ with open(os.path.join(wt, ".git"), "w") as fh:
+ fh.write(f"gitdir: {wt_git}\n")
+ sub = paths.repo_identity(os.path.join(main, "src", "pkg"))
+ assert sub.root == main
+ assert paths.repo_identity(wt).remote == sub.remote == "github.com/o/r"
+
+
+def test_a_directory_without_git_is_keyed_by_its_path(tmp_path, monkeypatch):
+ d = str(tmp_path / "scratch")
+ os.makedirs(d)
+ # Fence the walk at tmp_path: a machine may well have a stray .git in /tmp itself.
+ monkeypatch.setenv("GIT_CEILING_DIRECTORIES", str(tmp_path))
+ r = paths.repo_identity(d)
+ assert r.remote == "" and r.label == "scratch" and r.root == d
+
+
+def test_state_root_honours_the_environment(tmp_path):
+ assert paths.state_root({"TOKENECONOMIZER_HOME": str(tmp_path)}) == str(tmp_path)
+
+
+# ---- store and config ----------------------------------------------------------------------
+def test_store_writes_atomically_and_logs(tmp_path):
+ s = Store(str(tmp_path / "st"))
+ s.write("a/b.json", {"x": 1})
+ assert s.read("a/b.json") == {"x": 1}
+ assert s.read("missing.json", default={}) == {}
+ s.append("log.jsonl", {"n": 1})
+ s.append("log.jsonl", {"n": 2})
+ assert [r["n"] for r in s.read_lines("log.jsonl")] == [1, 2]
+ assert [r["n"] for r in s.read_lines("log.jsonl", limit=1)] == [2]
+ with s.locked("x") as got_x:
+ with s.locked("y") as got_y: # distinct locks never deadlock
+ s.write("c.json", [1])
+ with s.locked("x", timeout=0.1) as again: # held: a hook's caller still proceeds,
+ assert again is False # and is told it is unlocked
+ assert got_x and got_y
+ assert not [f for f in os.listdir(s.base) if f.endswith(".tmp")]
+
+
+def test_config_layers_and_never_raises_on_bad_values(tmp_path):
+ root = str(tmp_path)
+ repo_dir = os.path.join(root, "repos", "r")
+ os.makedirs(repo_dir)
+ with open(os.path.join(root, "config.json"), "w") as fh:
+ json.dump({"spiral": "guard", "untool_evidence": 7, "bogus": 1}, fh)
+ with open(os.path.join(repo_dir, "config.json"), "w") as fh:
+ json.dump({"spiral": "nope", "inject": "envfacts+distiller"}, fh)
+ cfg = live_config.load_config(root, repo_dir=repo_dir, env={"TE_UNTOOL": "auto"})
+ assert cfg.spiral == "guard" # the invalid repo value is ignored
+ assert cfg.untool_evidence == 7 and cfg.untool == "auto"
+ assert cfg.inject == "envfacts+distiller" and cfg.inject_effective == "envfacts"
+ joined = " ".join(cfg.warnings)
+ assert "bogus" in joined and "nope" in joined and "paid=false" in joined
+ assert live_config.load_config(root, env={"TE_DISABLE": "1"}).enabled is False
+
+
+def test_write_config_validates_before_writing(tmp_path):
+ path = str(tmp_path / "config.json")
+ live_config.write_config(path, {"spiral": "off", "paid": "true"})
+ assert json.load(open(path)) == {"spiral": "off", "paid": True}
+ with pytest.raises(ValueError):
+ live_config.write_config(path, {"spiral": "sometimes"})
+ assert json.load(open(path))["spiral"] == "off"
+
+
+# ---- Claude Code transcripts -----------------------------------------------------------------
+def _claude_lines(path, blocks_per_message=3, n_messages=2, model="claude-opus-5-5"):
+ lines = []
+ tag = os.path.splitext(os.path.basename(path))[0] # message ids are unique per request
+ for m in range(n_messages):
+ usage = {"input_tokens": 10, "cache_read_input_tokens": 1000,
+ "cache_creation_input_tokens": 100, "output_tokens": 50}
+ for b in range(blocks_per_message): # one entry per content block, usage repeated
+ lines.append({"type": "assistant", "message": {
+ "id": f"msg_{tag}_{m}", "model": model, "role": "assistant", "usage": usage,
+ "content": [{"type": "text", "text": f"block {b}"}]}})
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
+ return path
+
+
+def test_claude_usage_counts_each_message_once_and_includes_subagents(tmp_path):
+ main = _claude_lines(str(tmp_path / "sess.jsonl"))
+ sub_dir = tmp_path / "sess" / "subagents"
+ sub_dir.mkdir(parents=True)
+ _claude_lines(str(sub_dir / "agent-1.jsonl"), blocks_per_message=2, n_messages=1)
+ u = transcripts.claude_session_usage(main)
+ assert u["requests"] == 3 # 2 main messages + 1 subagent message
+ assert u["cache_read_input_tokens"] == 3000 # not 3 x 2 x 1000 + 2 x 1000
+ assert u["model"] == "claude-opus-5-5" and u["files"] == 2
+ assert transcripts.claude_session_usage(main, include_subagents=False)["requests"] == 2
+
+
+def test_deferred_tools_are_read_from_attachments(tmp_path):
+ p = str(tmp_path / "t.jsonl")
+ with open(p, "w") as fh:
+ fh.write(json.dumps({"type": "attachment", "attachment": {
+ "type": "deferred_tools_delta", "addedNames": ["CronCreate", "WebSearch"],
+ "removedNames": []}}) + "\n")
+ fh.write(json.dumps({"type": "attachment", "attachment": {
+ "type": "deferred_tools_delta", "addedNames": [], "removedNames": ["WebSearch"]}}) + "\n")
+ assert transcripts.claude_deferred_tools(p) == {"CronCreate"}
+
+
+# ---- Codex rollouts ---------------------------------------------------------------------------
+def _rollout(path):
+ recs = [
+ {"type": "session_meta", "payload": {"id": "thread-1", "cwd": "/work/r",
+ "cli_version": "0.153.1"}},
+ {"type": "turn_context", "payload": {"turn_id": "t1", "model": "gpt-6-astra"}},
+ {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {
+ "type": "CommandExecution", "id": "c1", "command": ["/bin/bash", "-lc", "pytest -q"],
+ "aggregated_output": "1 failed", "exit_code": 1, "status": "completed"}}},
+ {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {
+ "type": "FileChange", "id": "f1", "changes": {"/work/r/pkg/a.py": {"type": "update"}},
+ "status": "completed"}}},
+ {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {
+ "type": "Reasoning", "id": "r1"}}},
+ {"type": "event_msg", "payload": {"type": "token_count", "info": {"total_token_usage": {
+ "input_tokens": 1000, "cached_input_tokens": 600, "cache_write_input_tokens": 100,
+ "output_tokens": 40, "reasoning_output_tokens": 10}}}},
+ {"type": "event_msg", "payload": {"type": "token_count", "info": {"total_token_usage": {
+ "input_tokens": 3000, "cached_input_tokens": 2000, "cache_write_input_tokens": 200,
+ "output_tokens": 90, "reasoning_output_tokens": 30}}}},
+ ]
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(r) for r in recs) + "\n")
+ return path
+
+
+def test_codex_rollout_reads_through_the_existing_trace_seam(tmp_path):
+ src = _rollout(str(tmp_path / "rollout-x.jsonl"))
+ out = str(tmp_path / "exec.jsonl")
+ assert transcripts.codex_rollout_to_exec(src, out) == 2 # reasoning is not a tool call
+ traj = trace.load_trajectory(out)
+ assert traj[0][0] == "shell" and traj[0][1]["command"] == "/bin/bash -lc 'pytest -q'"
+ assert traj[0][2] == "1 failed" and traj[0][3] is False
+ assert traj[1][0] == "apply_patch" and traj[1][1]["path"] == "/work/r/pkg/a.py"
+
+
+def test_codex_usage_is_converted_as_the_harness_converts_it(tmp_path):
+ u = transcripts.codex_session_usage(_rollout(str(tmp_path / "rollout-x.jsonl")))
+ # last cumulative total; fresh input excludes both cache fields; reasoning joins output
+ assert u["input_tokens"] == 3000 - 2000 - 200
+ assert u["cache_read_input_tokens"] == 2000 and u["cache_creation_input_tokens"] == 200
+ assert u["output_tokens"] == 90 + 30 and u["model"] == "gpt-6-astra"
+
+
+def test_codex_capability_evidence_includes_the_models_own_exec_calls(tmp_path):
+ """In code mode every tool call is one `exec` custom tool; `tools.view_image(` inside its
+ input is image use, and only the capability trace carries it."""
+ from tokeneconomizer.mechanisms import codex_untool as cu
+ src = _rollout(str(tmp_path / "rollout-x.jsonl"))
+ with open(src, "a") as fh:
+ fh.write(json.dumps({"type": "response_item", "payload": {
+ "type": "custom_tool_call", "name": "exec", "call_id": "call_9",
+ "status": "completed",
+ "input": "const r = await tools.view_image({path: 'a.png'});"}}) + "\n")
+ out = str(tmp_path / "cap.jsonl")
+ assert transcripts.codex_capability_trace(src, out) == 3
+ usage = cu.inspect_usage(out)
+ assert usage["completed"]["image_media"] == 1 and usage["completed"]["collaboration"] == 0
+ plain = str(tmp_path / "exec.jsonl")
+ transcripts.codex_rollout_to_exec(src, plain)
+ assert cu.inspect_usage(plain)["completed"]["image_media"] == 0
+
+
+def test_a_compressed_rollout_is_reported_unreadable(tmp_path):
+ p = str(tmp_path / "rollout-x.jsonl.zst")
+ open(p, "wb").write(b"\x28\xb5\x2f\xfd")
+ assert transcripts.codex_rollout_to_exec(p, str(tmp_path / "o.jsonl")) is None
+
+
+# ---- hosts ---------------------------------------------------------------------------------
+def test_host_detection():
+ assert hosts.detect({"turn_id": "t"}).name == "codex"
+ assert hosts.detect({"transcript_path": "/h/.codex/sessions/2026/rollout-a.jsonl"}).name == "codex"
+ assert hosts.detect({"transcript_path": "/h/.claude/projects/x/s.jsonl"}).name == "claude"
+ assert hosts.detect({"turn_id": "t"}, hint="claude").name == "claude"
+ assert hosts.detect({"transcript_path": "/h/.copilot/session-state/s/events.jsonl"}
+ ).name == "copilot"
+ # Copilot sends no transcript before Stop, and marks its hook processes; a Claude Code
+ # session started from inside Copilot inherits the marker but always sends a transcript.
+ copilot_env = {"COPILOT_PLUGIN_ROOT": "/p", "COPILOT_CLI": "1"}
+ assert hosts.detect({"session_id": "s"}, copilot_env).name == "copilot"
+ assert hosts.detect({"session_id": "s", "transcript_path": "/h/.claude/projects/x/s.jsonl"},
+ copilot_env).name == "claude"
+ assert hosts.detect({"session_id": "s"}, {}).name == "claude"
+
+
+def test_the_te_command_defaults_to_the_agent_whose_shell_runs_it():
+ assert hosts.default_host({}) == "claude"
+ assert hosts.default_host({"COPILOT_CLI": "1"}) == "copilot"
+ assert hosts.default_host({"COPILOT_CLI": "1", "CLAUDECODE": "1"}) == "claude"
+ assert hosts.default_host({"TE_HOST": "codex", "CLAUDECODE": "1"}) == "codex"
+ assert hosts.default_host({"TE_HOST": "nonsense"}) == "claude"
+
+
+# Payloads copilot 1.0.83 sent to a Claude-format plugin hook (PascalCase events), verbatim in
+# shape, captured against a scripted local model endpoint on 2026-10-07.
+COPILOT_POST = {"hook_event_name": "PostToolUse", "session_id": "fcb5fc6b",
+ "timestamp": "2026-10-08T04:27:38.418Z", "cwd": "/w/repo", "tool_name": "Bash",
+ "tool_input": {"command": "python3 -c 'import sys; sys.exit(3)'",
+ "description": "run it"},
+ "tool_result": {"result_type": "success",
+ "text_result_for_llm": "boom\n<shellId: 0 completed with exit code 3>"}}
+COPILOT_FAILURE = {"hook_event_name": "PostToolUseFailure", "session_id": "10bbc978",
+ "timestamp": "2026-10-08T04:27:51.286Z", "cwd": "/w/repo", "tool_name": "Read",
+ "tool_input": {"path": "/w/repo/nope.py"}, "error": "Path does not exist"}
+
+
+def test_the_copilot_payloads_observed_live_are_read():
+ c = hosts.detect(COPILOT_POST, {"COPILOT_PLUGIN_ROOT": "/p"})
+ assert c.name == "copilot" and not c.can_stop_mid_turn and c.stop("why") is None
+ name, args, text, code = c.tool_call(COPILOT_POST)
+ assert (name, args["description"], code) == ("Bash", "run it", 3) # exit code from the text
+ assert text.startswith("boom")
+ name, args, text, code = c.tool_call(COPILOT_FAILURE)
+ assert (name, args["path"], code) == ("Read", "/w/repo/nope.py", None)
+ assert "Path does not exist" in text
+ # Copilot reads a TOP-LEVEL additionalContext and ignores hookSpecificOutput, and shows a
+ # notice only as a progress line ahead of the output (both verified in its TUI)
+ assert c.context("SessionStart", "x") == {"additionalContext": "x"}
+ m = hosts.merge_outputs(c.context("SessionStart", "a"), c.notice("n"),
+ c.context("SessionStart", "b"))
+ assert m == {"additionalContext": "a\n\nb", hosts.PROGRESS: ["n"]}
+
+
+def test_a_copilot_session_log_is_found_from_the_session_id(monkeypatch, tmp_path):
+ monkeypatch.setenv("COPILOT_HOME", str(tmp_path / "cp"))
+ c = hosts.Copilot()
+ assert c.transcript({"session_id": "abc"}) == str(
+ tmp_path / "cp" / "session-state" / "abc" / "events.jsonl")
+ assert c.transcript({"session_id": "abc", "transcript_path": "/x/events.jsonl"}) == \
+ "/x/events.jsonl"
+ assert c.cwd({}, {"COPILOT_PROJECT_DIR": "/w/repo"}) == "/w/repo"
+
+
+def test_claude_tool_results_and_outputs():
+ h = hosts.ClaudeCode()
+ name, args, text, code = h.tool_call({
+ "tool_name": "Bash", "tool_input": {"command": "pytest"},
+ "tool_response": {"stdout": "1 failed", "stderr": "Traceback", "interrupted": False}})
+ assert (name, args["command"], code) == ("Bash", "pytest", None) # no exit code (verified)
+ assert "1 failed" in text and "Traceback" in text
+ assert h.context("PostToolUse", "x") == {
+ "hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": "x"}}
+ assert h.stop("why") == {"continue": False, "stopReason": "why"}
+ # the hook environment of a `claude -p` session, as observed on 2.1.285
+ assert h.headless({"CLAUDE_CODE_SESSION_ATTENDED": "0", "CLAUDE_CODE_ENTRYPOINT": "sdk-cli"})
+ assert not h.headless({"CLAUDE_CODE_SESSION_ATTENDED": "1", "CLAUDE_CODE_ENTRYPOINT": "sdk-cli"})
+ assert h.headless({"CLAUDE_CODE_ENTRYPOINT": "sdk-cli"}) and not h.headless({})
+
+
+def test_the_codex_posttooluse_payload_observed_live_is_read():
+ """The payload codex-cli 0.153.1 sent for `echo one` (gpt-5.6-sol, 2026-09-30), verbatim in
+ shape: tool_name "Bash", a string command, a plain-string result, turn_id and model."""
+ payload = {"session_id": "01a0f5e1", "turn_id": "t1", "model": "gpt-5.6-sol",
+ "permission_mode": "default", "hook_event_name": "PostToolUse",
+ "cwd": "/w/repo", "transcript_path": "/h/.codex/sessions/2026/09/30/rollout-x.jsonl",
+ "tool_name": "Bash", "tool_use_id": "call_1", "tool_input": {"command": "echo one"},
+ "tool_response": "one\n"}
+ host = hosts.detect(payload)
+ assert host.name == "codex"
+ assert host.tool_call(payload) == ("bash", {"command": "echo one"}, "one\n", None)
+
+
+def test_codex_tool_calls_normalize_and_never_stop():
+ c = hosts.Codex()
+ assert c.stop("why") is None and not c.can_stop_mid_turn
+ name, args, text, code = c.tool_call({
+ "tool_name": "exec_command", "tool_input": {"command": ["bash", "-lc", "go test ./..."]},
+ "tool_response": {"output": "FAIL", "exit_code": 1}})
+ assert (name, args["command"], code) == ("bash", "bash -lc 'go test ./...'", 1)
+ name, args, _, _ = c.tool_call({"tool_name": "apply_patch", "tool_input": {
+ "input": "*** Begin Patch\n*** Update File: src/a.py\n@@\n-x\n+y\n*** End Patch"}})
+ assert name == "apply_patch" and args["paths"] == ["src/a.py"]
+ _, _, _, code = c.tool_call({"tool_name": "shell", "tool_input": {"command": "ls"},
+ "tool_response": "Exit code: 2\nWall time: 0.1s\nOutput:\n"})
+ assert code == 2
+
+
+def test_merge_outputs_concatenates_contexts_and_notices():
+ h = hosts.ClaudeCode()
+ m = hosts.merge_outputs(h.context("SessionStart", "a"), h.notice("n1"),
+ h.context("SessionStart", "b"), h.notice("n2"), None, {})
+ assert m["hookSpecificOutput"]["additionalContext"] == "a\n\nb"
+ assert m["systemMessage"] == "n1\nn2"
+
+
+# ---- Pi sessions -------------------------------------------------------------------------
+def _pi_session(path, cwd="/w/repo", sid="pi-1", abandoned=False):
+ """A Pi v3 session tree: header, a user prompt, a bash call that fails, an edit, a bash
+ call that passes -- and, optionally, an abandoned branch hanging off the first prompt."""
+ lines = [{"type": "session", "version": 3, "id": sid, "timestamp": "2026-09-30T00:00:00Z",
+ "cwd": cwd}]
+ state = {"parent": None, "n": 0}
+
+ def add(entry, parent="__chain__"):
+ state["n"] += 1
+ eid = f"e{state['n']:03d}"
+ entry.update(id=eid, timestamp="2026-09-30T00:00:01Z",
+ parentId=state["parent"] if parent == "__chain__" else parent)
+ lines.append(entry)
+ if parent == "__chain__":
+ state["parent"] = eid
+ return eid
+
+ usage = {"input": 10, "output": 50, "cacheRead": 1000, "cacheWrite": 100, "reasoning": 5,
+ "totalTokens": 1160, "cost": {"total": 0.002}}
+
+ def call(name, args, text, err):
+ tc = f"call{state['n']}"
+ add({"type": "message", "message": {"role": "assistant", "provider": "openai",
+ "model": "gpt-5.6-sol", "usage": usage,
+ "content": [{"type": "toolCall", "id": tc, "name": name, "arguments": args}]}})
+ add({"type": "message", "message": {"role": "toolResult", "toolCallId": tc,
+ "toolName": name, "isError": err, "content": [{"type": "text", "text": text}]}})
+
+ first = add({"type": "message", "message": {"role": "user", "content": "fix the bug"}})
+ call("bash", {"command": "pytest -q"}, "bash: pytest: command not found", True)
+ call("edit", {"path": "pkg/a.py", "oldText": "x", "newText": "y"}, "ok", False)
+ call("bash", {"command": "python -m pytest -q"}, "1 passed", False)
+ if abandoned: # a branch the user walked away from, earlier in the file
+ lines.insert(3, {"type": "message", "id": "zzz", "parentId": first,
+ "timestamp": "2026-09-30T00:00:02Z", "message": {
+ "role": "assistant", "model": "gpt-5.6-sol", "usage": usage,
+ "content": [{"type": "toolCall", "id": "dead", "name": "bash",
+ "arguments": {"command": "rm -rf build"}}]}})
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
+ return path
+
+
+def test_a_pi_session_reads_through_the_trace_seam_on_its_active_branch(tmp_path):
+ src = _pi_session(str(tmp_path / "s.jsonl"), abandoned=True)
+ out = str(tmp_path / "trace.jsonl")
+ assert transcripts.pi_session_to_claude(src, out) == 3
+ traj = trace.load_trajectory(out)
+ assert [(n, a.get("command") or a.get("path"), ok) for n, a, _, ok in traj] == [
+ ("bash", "pytest -q", False), ("edit", "pkg/a.py", True), ("bash", "python -m pytest -q", True)]
+ assert "rm -rf build" not in open(out).read() # the abandoned branch is not read
+
+
+def test_pi_usage_maps_directly_and_keeps_pis_own_cost(tmp_path):
+ u = transcripts.pi_session_usage(_pi_session(str(tmp_path / "s.jsonl"), abandoned=True))
+ # three assistant messages on the active branch; output already includes reasoning
+ assert u["requests"] == 3
+ assert (u["input_tokens"], u["cache_read_input_tokens"], u["cache_creation_input_tokens"],
+ u["output_tokens"]) == (30, 3000, 300, 150)
+ assert u["cost_usd_vendor"] == pytest.approx(0.006) and u["model"] == "gpt-5.6-sol"
+ assert transcripts.pi_session_header(str(tmp_path / "s.jsonl"))["cwd"] == "/w/repo"
+
+
+def test_the_pi_host_reads_what_the_extension_sends():
+ h = hosts.HOSTS["pi"]()
+ assert h.can_stop_mid_turn and h.stop("why") == {"continue": False, "stopReason": "why"}
+ name, args, text, code = h.tool_call({"tool_name": "bash", "tool_input": {"command": "go test"},
+ "tool_response": {"content": "FAIL\nCommand exited with code 1",
+ "isError": True, "exit_code": 1}})
+ assert (name, args["command"], code) == ("bash", "go test", 1) and "FAIL" in text
+ assert h.headless({"TE_PI_MODE": "print"}) and not h.headless({"TE_PI_MODE": "tui"})
+ assert hosts.detect({"turn_id": "t"}, hint="pi").name == "pi"
+
+
+# ---- the guard sees edits made through the shell ---------------------------------------------
+from te_live import guard # noqa: E402
+
+
[email protected]("cmd,want", [
+ ("sed -i 's/a/b/' pkg/mod.py", ["mod.py"]),
+ ("cd /t && sed -i.bak -e 's/a/b/' -e 's/c/d/' x.py y.py && pytest -q", ["x.py", "y.py"]),
+ ("sed -i '' 's/a/b/' x.py", ["x.py"]), # BSD sed's empty suffix
+ ("perl -pi -e 's/foo/bar/' src/a.rs", ["a.rs"]),
+ ("cat > repro.py <<'EOF'\nprint(1 > 0)\nEOF\npython repro.py", ["repro.py"]),
+ ("echo 'x' | tee -a setup.cfg", ["setup.cfg"]),
+ # how Sonnet 5.5 edits: an inline script that reads the file and writes it back
+ ("python - <<'E'\np='sympy/printing/str.py'; s=open(p).read()\nopen(p,'w').write(s)\nE",
+ ["str.py"]),
+ ("python - <<'E'\nfor p in ['x/a.py', 'x/b.py']:\n open(p, 'w').write('')\nE",
+ ["a.py", "b.py"]),
+ ("python - <<'E'\nfrom pathlib import Path\nPath('m/n.py').write_text('x')\nE", ["n.py"]),
+ ("git apply /tmp/fix.diff", ["(patch)"]),
+ ("git checkout -- a.py b/c.py", ["a.py", "c.py"]),
+ # not edits: output captures, scratch, reads, and operators inside quotes
+ ("python -m pytest -q > /tmp/out.txt 2>&1; tail -5 /tmp/out.txt", []),
+ ("grep -n 'x' a.py > hits.txt", []),
+ ("grep -n '>' x.py", []),
+ ("grep -nE 'a|b' x.py | head", []),
+ ("python -c \"print(1>2)\"", []),
+ ("git stash -q; python bin/test x; git stash pop -q", []),
+ ("sed -n '1,20p' pkg/mod.py", []),
+ # a quoted `|` in a sed script must not split the command (seen live: sympy-13091)
+ ("sed -i -E 's/x (<|<=|>|>=) y/z/' numbers.py expr.py", ["numbers.py", "expr.py"]),
+ ("sed -i 's/a/b/' x.py 2>&1 | tail -2", ["x.py"]),
+])
+def test_edits_made_through_the_shell_are_recognised(cmd, want):
+ assert guard.shell_edit_targets(cmd) == want
+
+
+def test_shell_edits_reach_the_state_the_judge_reads_and_the_ended_green_proxy():
+ g = guard.TurnGuard(turn_key="t")
+ edit = "python - <<'E'\np='pkg/mod.py'; s=open(p).read()\nopen(p,'w').write(s[::-1])\nE"
+ for _ in range(2):
+ g.fold("Bash", {"command": edit}, "")
+ assert g.st.edit_churn == 2 and g.whole.distinct_edit_files == 1 and g.edits == 2
+ digest = g.st.recent_digest()
+ assert "edit_churn=2" in digest and "[edits mod.py]" in digest
+ # one command that edits and then fails its tests: the turn did not end green
+ g.fold("Bash", {"command": "sed -i 's/b/c/' pkg/mod.py && pytest -q"},
+ "FAILED tests/test_m.py::t - AssertionError")
+ assert g.edits == 3 and g.last_verify > g.last_edit and not g.resolved_proxy()
+
+
+def test_the_judge_is_told_how_the_last_verification_went_not_that_tests_are_red():
+ g = guard.TurnGuard(turn_key="t")
+ vendored = "turns=3\nrepo_context: ... = 1.2x that typical length. Tests are still NOT green. A"
+ assert "No test or verification run has been seen" in g.live_digest(vendored)
+ g.fold("Bash", {"command": "python bin/test sympy/core"}, "tests finished: 10 passed")
+ told = g.live_digest(vendored)
+ assert "The last verification run passed." in told and "NOT green" not in told
+ g.fold("Edit", {"file_path": "/r/a.py", "old_string": "a", "new_string": "b"}, "ok")
+ assert g.live_digest(vendored).count("There have been edits since.") == 1
+
+
+class _BackgroundJudge:
+ """`te_live.judge.AsyncJudge`'s interface, with the verdict handed in by the test."""
+
+ def __init__(self):
+ self.started, self.verdict = [], None
+
+ def start(self, g, digest):
+ self.started.append(digest)
+ return f"job{len(self.started)}"
+
+ def poll(self, g):
+ pending = g.judge_pending or {}
+ if self.verdict is None or pending.get("id") != f"job{len(self.started)}":
+ return False, None
+ v, self.verdict = self.verdict, None
+ return True, v
+
+
+def _spin(g, judge, n, envelope=6.0):
+ """An edit and an identical failing test run, alternately: a loop the gate opens on."""
+ out = []
+ for k in range(n):
+ call = (("Edit", {"file_path": "/r/a.py", "old_string": "x", "new_string": "y"}, "ok")
+ if k % 2 == 0 else
+ ("Bash", {"command": "pytest -q"}, "FAILED tests/test_a.py::t - AssertionError"))
+ out.append(g.fold(*call, envelope=envelope, judge=judge))
+ return out
+
+
+def test_a_background_judge_starts_once_and_its_verdict_nudges_a_later_call():
+ g, j = guard.TurnGuard(turn_key="t"), _BackgroundJudge()
+ assert _spin(g, j, 10) == [None] * 10 # nothing is decided while it is out
+ assert len(j.started) == 1 # and no second job is started
+ assert "NOT green" not in j.started[0]
+ j.verdict = {"stuck": True, "confidence": 0.9, "reason": "loop",
+ "_nudge": "Read the failing assertion before editing again."}
+ assert _spin(g, j, 1)[0] == ("nudge", "[email protected]:loop")
+ assert g.nudge_body(6.0) == "Read the failing assertion before editing again."
+ assert g.nudge_body(6.0) != "Read the failing assertion before editing again." # used once
+
+
+def test_a_productive_verdict_backs_the_judge_off():
+ g, j = guard.TurnGuard(turn_key="t"), _BackgroundJudge()
+ _spin(g, j, 7)
+ j.verdict = {"stuck": False, "confidence": 0.8, "reason": "progress"}
+ assert _spin(g, j, 1) == [None]
+ assert g.judge_gap == g.JUDGE_BASE_GAP * 2 and g.judge_pending is None
+
+
+def test_a_job_that_never_answers_stops_blocking_the_next_one():
+ g, j = guard.TurnGuard(turn_key="t"), _BackgroundJudge()
+ _spin(g, j, 7)
+ assert len(j.started) == 1 and g.judge_pending
+ g.judge_pending["since"] = 0 # its process died
+ _spin(g, j, g.judge_gap) # well before the backstop at 2.5x
+ assert len(j.started) == 2
+
+
+# ---- Copilot session logs ------------------------------------------------------------------
+def copilot_events(path, calls, *, shutdown=True, model="gpt-5.4", cwd="/w/repo", sid="cp1"):
+ """A Copilot `session-state/<id>/events.jsonl`: session.start, one user message, the tool
+ calls as execution_start/complete pairs, and (unless killed) a session.shutdown."""
+ ev = [{"type": "session.start", "data": {"sessionId": sid, "copilotVersion": "1.0.83",
+ "selectedModel": model, "context": {"cwd": cwd}}},
+ {"type": "user.message", "data": {"content": "fix it", "turnId": "0"}}]
+ for n, (name, args, text, ok) in enumerate(calls):
+ ev.append({"type": "tool.execution_start", "data": {
+ "toolCallId": f"c{n}", "toolName": name, "arguments": args, "turnId": "0"}})
+ ev.append({"type": "tool.execution_complete", "data": {
+ "toolCallId": f"c{n}", "success": ok, "turnId": "0",
+ "result": {"content": text, "detailedContent": text}}})
+ if shutdown:
+ ev.append({"type": "session.shutdown", "data": {
+ "modelMetrics": {model: {"requests": {"count": 4}, "usage": {
+ "inputTokens": 4000, "outputTokens": 80, "cacheReadTokens": 2400,
+ "cacheWriteTokens": 100, "reasoningTokens": 0}}},
+ "toolDefinitionsTokens": 13149}})
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(e) for e in ev) + "\n")
+ return path
+
+
+def test_copilot_logs_read_through_the_existing_trace_seam(tmp_path):
+ p = copilot_events(str(tmp_path / "s" / "events.jsonl"), [
+ ("bash", {"command": "pytest -q"}, "1 failed\n<shellId: 0 completed with exit code 1>",
+ True),
+ ("view", {"path": "/w/repo/a.py"}, "x = 1", True)])
+ traj = trace.load_trajectory(p)
+ assert [t[0] for t in traj] == ["bash", "view"] and traj[0][1]["command"] == "pytest -q"
+ assert trace.tool_usage(p) == {"bash": 1, "view": 1}
+ (turn,) = transcripts.copilot_turns(p)
+ assert turn[0][0] == "bash" and turn[0][3] == 1 and turn[1][3] is None
+ assert transcripts.copilot_session_meta(p)["cwd"] == "/w/repo"
+
+
+def test_copilot_usage_is_converted_from_its_grand_input_total(tmp_path):
+ p = copilot_events(str(tmp_path / "s" / "events.jsonl"), [])
+ u = transcripts.copilot_session_usage(p)
+ # inputTokens counts cache reads and writes (verified against a scripted endpoint)
+ assert u["input_tokens"] == 4000 - 2400 - 100
+ assert u["cache_read_input_tokens"] == 2400 and u["cache_creation_input_tokens"] == 100
+ assert u["output_tokens"] == 80 and u["requests"] == 4 and u["model"] == "gpt-5.4"
+ assert u["tool_definitions_tokens"] == 13149
+ killed = transcripts.copilot_session_usage(
+ copilot_events(str(tmp_path / "k" / "events.jsonl"), [], shutdown=False))
+ assert killed["input_tokens"] is None and killed["model"] == "gpt-5.4"
+
+
+def test_copilot_tools_are_grouped_by_mcp_server(tmp_path):
+ servers = {"github-mcp-server", "github", "acme-db"}
+ assert transcripts.mcp_server_of("github-mcp-server-list_issues", servers) == \
+ "github-mcp-server" # the longest prefix wins
+ assert transcripts.mcp_server_of("acme-db-query", servers) == "acme-db"
+ assert transcripts.mcp_server_of("bash", servers) is None
+ home = tmp_path / "cp"
+ home.mkdir()
+ (home / "mcp-config.json").write_text(json.dumps({"mcpServers": {"acme-db": {}}}))
+ assert transcripts.copilot_mcp_servers({"COPILOT_HOME": str(home)}) == {
+ "github-mcp-server", "acme-db"}
+
+
+def test_a_copilot_side_call_is_a_bare_completion_priced_from_its_own_log(tmp_path,
+ monkeypatch):
+ import subprocess
+ from te_live import sidecall
+ monkeypatch.setenv("COPILOT_HOME", str(tmp_path / "cp"))
+ copilot_events(str(tmp_path / "cp" / "session-state" / "side1" / "events.jsonl"), [])
+ seen = {}
+
+ def fake_run(cmd, **kw):
+ seen["cmd"], seen["env"] = cmd, kw["env"]
+ out = "\n".join(json.dumps(e) for e in (
+ {"type": "assistant.message", "data": {"content": "- use python -m pytest"}},
+ {"type": "result", "sessionId": "side1", "exitCode": 0}))
+ return subprocess.CompletedProcess(cmd, 0, stdout=out, stderr="")
+
+ monkeypatch.setattr(sidecall.subprocess, "run", fake_run)
+ text, eff = sidecall.copilot_call("propose facts")
+ assert text == "- use python -m pytest" and eff and eff > 0
+ cmd = seen["cmd"]
+ assert cmd[:3] == ["copilot", "-p", "propose facts"] and "--disable-builtin-mcps" in cmd
+ assert cmd.count("--excluded-tools") == len(sidecall.COPILOT_TOOLS)
+ assert seen["env"]["TE_SIDECALL"] == "1" # its own hooks must not recurse
+ cfg = live_config.LiveConfig(paid=True)
+ assert sidecall.runner_for("copilot", cfg).__code__.co_names.count("copilot_call") == 1
+
+
+def test_a_copilot_notice_is_a_progress_line_before_the_hook_output():
+ import io
+ from te_live import hook
+ out = io.StringIO()
+ hook.main([], io.StringIO("{}"), out, {"TE_DISABLE": "1"})
+ assert out.getvalue() == "{}\n"
+ # what hook.main writes for an output carrying a Copilot notice
+ from unittest import mock
+ with mock.patch.object(hook, "run", return_value=dict(
+ hosts.Copilot().notice("hello"), additionalContext="ctx")):
+ out = io.StringIO()
+ hook.main([], io.StringIO('{"session_id": "s"}'), out, {})
+ first, last = out.getvalue().splitlines()
+ assert json.loads(first) == {"type": "progress", "message": "hello"}
+ assert json.loads(last) == {"additionalContext": "ctx"}
+
+
+def test_copilot_side_calls_name_a_model_a_byok_provider_can_serve():
+ from te_live.sidecall import copilot_model
+ byok = {"COPILOT_PROVIDER_BASE_URL": "https://api.anthropic.com",
+ "COPILOT_PROVIDER_TYPE": "anthropic"}
+ # `auto` is the subscription's router; a BYOK provider has none, and the call answered nothing
+ assert copilot_model("", dict(byok, COPILOT_MODEL="auto")) == "claude-haiku-4-5"
+ assert copilot_model("", dict(byok, COPILOT_MODEL="claude-sonnet-5")) == "claude-sonnet-5"
+ assert copilot_model("claude-opus-5", byok) == "claude-opus-5" # paid_model wins
+ assert copilot_model("", {"COPILOT_MODEL": "auto"}) == "" # subscription
+ assert copilot_model("", {"COPILOT_PROVIDER_BASE_URL": "http://localhost:11434/v1"}) == ""
+ # OpenRouter wants its own ids, whichever wire Copilot speaks to it
+ for wire in ("anthropic", "openai"):
+ assert copilot_model("", {"COPILOT_PROVIDER_BASE_URL": "https://openrouter.ai/api",
+ "COPILOT_PROVIDER_TYPE": wire}) == "anthropic/claude-haiku-4.5"
+
+
+def test_the_probe_repository_carries_its_trap(tmp_path):
+ import subprocess
+ from te_live import probe
+ repo, trap = probe.make_repo(str(tmp_path))
+ bare = subprocess.run([os.path.join(trap, "pytest"), "-q"], cwd=repo, capture_output=True,
+ text=True)
+ assert bare.returncode == 127 and "command not found" in bare.stderr
+ # pytest is "not installed" for any interpreter the session runs, whatever really is
+ for py in ("python", "python3"):
+ mod = subprocess.run([os.path.join(trap, py), "-m", "pytest", "-q"], cwd=repo,
+ capture_output=True, text=True)
+ assert mod.returncode == 1 and "No module named 'pytest'" in mod.stderr
+ ok = subprocess.run([os.path.join(trap, "python"), "-m", "unittest", "-q"], cwd=repo,
+ capture_output=True, text=True)
+ assert ok.returncode == 0 and "Ran 1 test" in ok.stderr
+ assert subprocess.run(["git", "-C", repo, "status", "--porcelain"], capture_output=True,
+ text=True).stdout == "" # committed clean
+
+
+def test_the_probe_counts_pytest_attempts_not_exit_codes():
+ from te_live import probe
+ # commands two real Haiku sessions ran: the piped one exited 0 although pytest is missing
+ runs, tried = probe.test_commands([
+ 'find . -name "pytest.ini" -o -name "go.mod" | head -20',
+ "cd /r && python -m pytest tests/ -v 2>&1 | tail -20",
+ "cd /r && python -m unittest discover -s tests -v 2>&1 | tail -20"])
+ assert len(runs) == 2 and tried == 1
+ runs, tried = probe.test_commands(["pytest -q", "python3 -m unittest", "ls tests/"])
+ assert len(runs) == 2 and tried == 1
+ assert probe.test_commands(['python -m unittest discover -s tests -p "test_*.py"']) == (
+ ['python -m unittest discover -s tests -p "test_*.py"'], 0)
+
+
+def test_a_small_envelopes_backstop_never_preempts_the_nudge():
+ """Interactive repositories learn envelopes of ~4-10 calls, so the hard backstop
+ (1.8-2.5x the envelope) lies inside the stuck rule's own floor: before this fix, the first
+ stuck call tripped the backstop and the turn was stopped without ever being redirected."""
+ from te_live import guard
+ from tokeneconomizer.vendor import spiral
+ thr = spiral.scaled_thresholds("aggressive")
+ fail = ("FAIL: test_floats (tests.test_calc.TestAdd)\nTraceback (most recent call last):\n"
+ "AssertionError: 0.30000000000000004 != 0.3\n\nFAILED (failures=1)\n")
+ g = guard.TurnGuard(turn_key="t")
+ calls = [("Bash", {"command": "python -m unittest"}, fail, 1)]
+ for k in range(30):
+ calls += [("Edit", {"file_path": "src/calc/__init__.py", "old_string": "x",
+ "new_string": f"y{k}"}, "ok", None),
+ ("Bash", {"command": "python -m unittest"}, fail, 1)]
+ fired = []
+ for i, (n, a, r, c) in enumerate(calls, 1):
+ d = g.fold(n, a, r, c, envelope=4.0, thr=thr, nudge_budget=1)
+ if d:
+ fired.append((i, d[0]))
+ if d[0] == "nudge":
+ g.after_nudge()
+ else:
+ break
+ assert fired == [(14, "nudge"), (28, "abort")]
diff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.py
new file mode 100644
index 0000000..7f6badb
--- /dev/null
+++ b/tests/test_live_hooks.py
@@ -0,0 +1,743 @@
+"""The plugin's hooks, end to end, through `live.hook.main` with synthetic payloads.
+
+Each test drives a whole stretch of a repository's life -- sessions starting, prompt-turns, tool
+calls, sessions ending and being folded -- and pins one property the live layer must have:
+learned facts reach the next session's context, the spiral guard is silent until it has learned
+the repository and then nudges before it stops, an untool proposal changes nothing until it is
+applied and touches only its own deny rules, every session is folded exactly once, and nothing
+here can break the agent's session.
+
+Folds run inline (`TE_FOLD_SYNC=1`); in the plugin they are detached processes.
+"""
+import io
+import json
+import os
+
+import pytest
+
+from te_live import hook, untool
+from te_live.controller import LiveController
+from te_live.paths import repo_identity, repo_state_dir
+from te_live.store import Store
+
+
+def env(tmp_path):
+ return {"TOKENECONOMIZER_HOME": str(tmp_path / "home"), "TE_FOLD_SYNC": "1",
+ "GIT_CEILING_DIRECTORIES": str(tmp_path), "PATH": os.environ.get("PATH", "")}
+
+
+def repo(tmp_path):
+ r = tmp_path / "repo"
+ (r / ".git" / "info").mkdir(parents=True)
+ (r / ".git" / "config").write_text(
+ '[remote "origin"]\n\turl = https://github.com/acme/widget.git\n')
+ (r / "pyproject.toml").write_text("[project]\nname='w'\n")
+ return str(r)
+
+
+def call(env, payload, host=None):
+ """One hook call: its final JSON object, plus the `progress` notice lines a host shows
+ ahead of it (Copilot), when there are any."""
+ out = io.StringIO()
+ hook.main(["--host", host] if host else [], io.StringIO(json.dumps(payload)), out, env)
+ lines = out.getvalue().splitlines()
+ res = json.loads(lines[-1])
+ progress = [json.loads(ln) for ln in lines[:-1]]
+ assert all(p["type"] == "progress" for p in progress)
+ if progress:
+ res["progress"] = [p["message"] for p in progress]
+ return res
+
+
+def store_for(env, repo, host="claude"):
+ return Store(repo_state_dir(repo_identity(repo, env), host, env))
+
+
+def write_session(path, prompts, model="claude-opus-5-5"):
+ """An interactive Claude Code transcript: per prompt, a user entry, then per tool call an
+ assistant entry (tool_use, with usage) and a user entry (tool_result)."""
+ lines, n = [], 0
+ for text, calls in prompts:
+ lines.append({"type": "user", "message": {"role": "user", "content": text}})
+ for name, inp, result, is_error in calls:
+ n += 1
+ lines.append({"type": "assistant", "message": {
+ "id": f"msg_{os.path.basename(path)}_{n}", "model": model, "role": "assistant",
+ "usage": {"input_tokens": 10, "cache_read_input_tokens": 1000,
+ "cache_creation_input_tokens": 100, "output_tokens": 50},
+ "content": [{"type": "tool_use", "id": f"tu{n}", "name": name, "input": inp}]}})
+ lines.append({"type": "user", "message": {"role": "user", "content": [
+ {"type": "tool_result", "tool_use_id": f"tu{n}", "content": result,
+ "is_error": is_error}]}})
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
+ return path
+
+
+def trap_session(path, repo, i):
+ """An environment trap every session falls into: bare `pytest` is missing here."""
+ return write_session(path, [(f"fix bug {i}", [
+ ("Edit", {"file_path": f"{repo}/pkg/mod{i}.py", "old_string": "a", "new_string": "b"},
+ "ok", False),
+ ("Bash", {"command": "pytest -q tests/test_a.py"}, "bash: pytest: command not found", True),
+ ("Bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s", False),
+ ])])
+
+
+def run_session(env, repo, sid, transcript, host=None, source="startup"):
+ base = {"session_id": sid, "transcript_path": transcript, "cwd": repo}
+ start = call(env, dict(base, hook_event_name="SessionStart", source=source), host)
+ call(env, dict(base, hook_event_name="SessionEnd", reason="other"), host)
+ return start
+
+
+# ---- injection ---------------------------------------------------------------------------
+def test_a_learned_fact_reaches_the_next_session_and_is_announced_once(env, repo, tmp_path):
+ starts = [run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ for i in range(6)]
+ assert all(s == {} for s in starts[:4]) # the fact must clear its firing gate first
+ ctx = starts[4]["hookSpecificOutput"]["additionalContext"]
+ assert "`pytest …` fails here with command not found; `python -m pytest …` works." in ctx
+ assert "learned fact" in starts[4]["systemMessage"]
+ assert "systemMessage" not in starts[5] # unchanged manual: no second notice
+ assert starts[5]["hookSpecificOutput"]["additionalContext"] == ctx
+
+
+def test_resume_does_not_inject_again_and_shadow_never_injects(env, repo, tmp_path):
+ for i in range(5):
+ run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))
+ t = str(tmp_path / "later.jsonl")
+ open(t, "w").close()
+ assert call(env, {"hook_event_name": "SessionStart", "session_id": "s4", "source": "resume",
+ "transcript_path": t, "cwd": repo}) == {}
+ shadow = dict(env, TE_SHADOW="1")
+ assert call(shadow, {"hook_event_name": "SessionStart", "session_id": "z", "source": "startup",
+ "transcript_path": t, "cwd": repo}) == {}
+ events = store_for(env, repo).read_lines("events.jsonl")
+ assert any(e["kind"] == "inject" and e["shadow"] for e in events)
+
+
+# ---- the spiral guard --------------------------------------------------------------------
+def _turn(env, repo, sid, key, calls, host=None):
+ """One prompt-turn through UserPromptSubmit -> PostToolUse* -> Stop; outputs per call."""
+ idk = "turn_id" if host == "codex" else "prompt_id"
+ base = {"session_id": sid, "transcript_path": "", "cwd": repo, idk: key}
+ call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="go"), host)
+ outs = [call(env, dict(base, hook_event_name="PostToolUse", tool_name=n, tool_input=i,
+ tool_response=r), host) for n, i, r in calls]
+ call(env, dict(base, hook_event_name="Stop", stop_hook_active=False), host)
+ return outs
+
+
+GREEN = [("Read", {"file_path": "/r/a.py"}, "ok")] * 8
+
+
+def stuck_calls(n, host=None):
+ out = []
+ for k in range(n):
+ if k % 2 == 0:
+ out.append(("apply_patch", {"input": "*** Begin Patch\n*** Update File: pkg/a.py\n"},
+ "Success.") if host == "codex" else
+ ("Edit", {"file_path": "/r/pkg/a.py", "old_string": "x", "new_string": "y"},
+ "ok"))
+ else:
+ fail = "FAILED tests/test_a.py::test_x - AssertionError: assert 1 == 2"
+ out.append(("exec_command", {"command": "pytest -q"}, {"output": fail, "exit_code": 1})
+ if host == "codex" else
+ ("Bash", {"command": "pytest -q"}, {"stdout": fail, "stderr": ""}))
+ return out
+
+
+def _first(outs, pred):
+ return next((i for i, o in enumerate(outs) if pred(o)), None)
+
+
+def test_cold_start_never_intervenes(env, repo):
+ outs = _turn(env, repo, "s", "p0", stuck_calls(40))
+ assert all(o == {} for o in outs) # no envelope learned: resolve-safe
+
+
+def test_warm_guard_nudges_then_only_notifies_in_an_interactive_session(env, repo):
+ for k in range(5):
+ _turn(env, repo, "s", f"g{k}", GREEN) # five ended-green turns of 8 calls
+ outs = _turn(env, repo, "s", "stuck", stuck_calls(40))
+ nudge = _first(outs, lambda o: "hookSpecificOutput" in o)
+ assert nudge is not None
+ assert "Process check (automated)" in outs[nudge]["hookSpecificOutput"]["additionalContext"]
+ assert "nudged" in outs[nudge]["systemMessage"]
+ later = _first(outs[nudge + 1:], lambda o: o != {})
+ assert later is not None
+ after = outs[nudge + 1 + later]
+ assert "continue" not in after and "still looks stuck" in after["systemMessage"]
+ assert all(o == {} for o in outs[nudge + 2 + later:]) # the notice is delivered once
+
+
+def test_a_headless_guard_stops_the_turn_on_claude_but_not_on_codex(env, repo):
+ headless = dict(env, TE_MODE="headless", TE_SPIRAL="guard")
+ for k in range(5):
+ _turn(headless, repo, "s", f"g{k}", GREEN)
+ outs = _turn(headless, repo, "s", "stuck", stuck_calls(40))
+ assert any(o.get("continue") is False and "stopped this turn" in o["stopReason"]
+ for o in outs)
+ for k in range(5):
+ _turn(headless, repo, "c", f"g{k}", GREEN, host="codex")
+ outs = _turn(headless, repo, "c", "stuck", stuck_calls(40, "codex"), host="codex")
+ assert any("hookSpecificOutput" in o for o in outs) # nudged
+ assert not any("continue" in o for o in outs) # Codex cannot stop mid-turn
+ assert any("still looks stuck" in (o.get("systemMessage") or "") for o in outs)
+
+
+def _fake_judges(monkeypatch, seen):
+ from tokeneconomizer.vendor import reflect
+
+ def judge(digest, runner, model="agent"):
+ seen.append(digest)
+ return {"stuck": True, "confidence": 0.95, "reason": "loop", "_tokens": 1000.0}
+
+ def writer(digest, runner, model="agent"):
+ return {"text": "Read the failing assertion before editing again.", "_tokens": 300.0}
+ monkeypatch.setattr(reflect, "llm_stuck_judge", judge)
+ monkeypatch.setattr(reflect, "llm_nudge_writer", writer)
+
+
+def test_the_paid_judge_answers_from_a_job_and_its_spend_is_charged_as_it_happens(
+ env, repo, monkeypatch):
+ """The job runs inline here (TE_JUDGE_SYNC=1); in the plugin it is a detached process."""
+ from te_live import accounting
+ seen = []
+ _fake_judges(monkeypatch, seen)
+ paid = dict(env, TE_PAID="1", TE_JUDGE_SYNC="1")
+ for k in range(5):
+ _turn(paid, repo, "s", f"g{k}", GREEN)
+ outs = _turn(paid, repo, "s", "stuck", stuck_calls(40))
+ nudge = _first(outs, lambda o: "hookSpecificOutput" in o)
+ assert nudge is not None and seen
+ assert outs[nudge]["hookSpecificOutput"]["additionalContext"] == (
+ "Read the failing assertion before editing again.")
+ # The call that started the job answered nothing; its verdict arrived on a later call.
+ started_at = int(seen[0].split()[0].split("=")[1]) # the digest opens "turns=N"
+ assert nudge + 1 > started_at
+ assert all("NOT green" not in d for d in seen)
+ ledger = store_for(env, repo).read_lines("ledger.jsonl")
+ charged = [r for r in ledger if r.get("kind") == "overhead"]
+ assert charged and charged[0]["session_id"] == "s" and charged[0]["overhead"] == {
+ "spiral_judge_tokens": 1000.0, "spiral_nudge_writer_tokens": 300.0}
+ assert accounting.summarize(ledger)["overhead_tokens"] == pytest.approx(
+ 1300.0 * len(charged))
+
+
+def test_a_hook_only_starts_the_paid_judge_and_never_waits_for_it(env, repo, monkeypatch):
+ from te_live import judge as live_judge
+ seen, spawned = [], []
+ _fake_judges(monkeypatch, seen)
+ monkeypatch.setattr(live_judge, "spawn_job", lambda ctx, sid, job: spawned.append(job))
+ paid = dict(env, TE_PAID="1")
+ for k in range(5):
+ _turn(paid, repo, "s", f"g{k}", GREEN)
+ _turn(paid, repo, "s", "stuck", stuck_calls(12))
+ assert len(spawned) == 1 and seen == [] # started once, never run in the hook
+ req = store_for(env, repo).read("judge/s.request.json")
+ assert req["id"] == spawned[0] and "NOT green" not in req["digest"]
+
+
+def test_only_turns_that_end_green_feed_the_envelope(env, repo):
+ fail = {"stdout": "FAILED tests/test_a.py::t - AssertionError", "stderr": ""}
+ _turn(env, repo, "s", "red", [("Edit", {"file_path": "/r/a.py"}, "ok"),
+ ("Bash", {"command": "pytest -q"}, fail)])
+ _turn(env, repo, "s", "green", [("Edit", {"file_path": "/r/a.py"}, "ok"),
+ ("Bash", {"command": "pytest -q"}, {"stdout": "3 passed"})])
+ assert store_for(env, repo).read("envelope.json")["resolved_turns"] == [2]
+
+
+def test_subagent_tool_calls_do_not_reach_the_guard(env, repo):
+ call(env, {"hook_event_name": "PostToolUse", "session_id": "s", "cwd": repo, "prompt_id": "p",
+ "agent_id": "a1", "tool_name": "Bash", "tool_input": {"command": "ls"},
+ "tool_response": {"stdout": ""}})
+ assert store_for(env, repo).read("sessions/s.json") is None
+
+
+# ---- untooling -----------------------------------------------------------------------------
+def _catalog(env, tools):
+ d = os.path.join(env["TOKENECONOMIZER_HOME"], "hosts", "claude")
+ os.makedirs(d, exist_ok=True)
+ json.dump({"tools": tools}, open(os.path.join(d, "catalog.json"), "w"))
+
+
+def test_unused_tools_are_proposed_but_nothing_changes_until_applied(env, repo, tmp_path):
+ _catalog(env, ["Bash", "Read", "Edit", "Write", "AskUserQuestion", "Workflow",
+ "CronCreate", "ShareOnboardingGuide", "WebFetch"])
+ for i in range(5):
+ t = write_session(str(tmp_path / f"u{i}.jsonl"), [("look", [
+ ("Read", {"file_path": f"{repo}/a.py"}, "x", False),
+ ("Bash", {"command": "ls"}, "a.py", False)])])
+ run_session(env, repo, f"u{i}", t)
+ st = store_for(env, repo)
+ prop = st.read("untool.json")["proposal"]
+ # protected tools (incl. a person's AskUserQuestion, and the web) are never proposed
+ assert sorted(prop) == ["CronCreate", "ShareOnboardingGuide", "Workflow"]
+ assert not os.path.exists(untool.settings_path(repo)) # proposing writes nothing
+ t = str(tmp_path / "next.jsonl")
+ open(t, "w").close()
+ notice = call(env, {"hook_event_name": "SessionStart", "session_id": "n", "source": "startup",
+ "transcript_path": t, "cwd": repo})["systemMessage"]
+ assert "3 tool(s) went unused" in notice and "/tokeneconomizer:untool" in notice
+
+
+def test_the_web_is_never_proposed_and_an_old_deny_rule_for_it_is_given_back(
+ env, repo, tmp_path):
+ _catalog(env, ["Bash", "Read", "CronCreate", "WebFetch", "WebSearch"])
+ st = store_for(env, repo)
+ # A deny list accepted under an earlier version, which still proposed the web.
+ untool.apply(repo, ["CronCreate", "WebFetch", "WebSearch"], st)
+ for i in range(20): # far past the old 3x evidence bar
+ t = write_session(str(tmp_path / f"w{i}.jsonl"), [("look", [
+ ("Bash", {"command": "ls"}, "a.py", False)])])
+ run_session(env, repo, f"w{i}", t)
+ assert untool.applied(st) == ["CronCreate"]
+ deny = json.load(open(untool.settings_path(repo)))["permissions"]["deny"]
+ assert deny == ["CronCreate"]
+ assert not {"WebFetch", "WebSearch"} & set(st.read("untool.json")["proposal"])
+
+
+def test_apply_and_restore_touch_only_the_plugins_own_rules(repo, tmp_path):
+ st = Store(str(tmp_path / "st"))
+ path = untool.settings_path(repo)
+ os.makedirs(os.path.dirname(path))
+ json.dump({"permissions": {"deny": ["Bash(rm -rf:*)"], "allow": ["Read"]}, "model": "opus"},
+ open(path, "w"))
+ assert untool.apply(repo, ["Workflow", "CronCreate"], st) == ["Workflow", "CronCreate"]
+ data = json.load(open(path))
+ assert data["permissions"]["deny"] == ["Bash(rm -rf:*)", "Workflow", "CronCreate"]
+ assert data["model"] == "opus" and data["permissions"]["allow"] == ["Read"]
+ assert untool.restore(repo, st, ["CronCreate"]) == ["CronCreate"]
+ assert untool.restore(repo, st) == ["Workflow"]
+ assert json.load(open(path))["permissions"]["deny"] == ["Bash(rm -rf:*)"]
+ # the user wrote that file, so it was not added to the local git excludes
+ exclude = os.path.join(repo, ".git", "info", "exclude")
+ assert not os.path.exists(exclude) or "settings.local.json" not in open(exclude).read()
+
+
+def test_a_settings_file_the_plugin_creates_is_kept_out_of_git(repo, tmp_path):
+ st = Store(str(tmp_path / "st"))
+ untool.apply(repo, ["Workflow"], st)
+ assert "/.claude/settings.local.json" in open(
+ os.path.join(repo, ".git", "info", "exclude")).read().split()
+ untool.apply(repo, ["CronCreate"], st) # idempotent exclude entry
+ assert open(os.path.join(repo, ".git", "info", "exclude")).read().count(
+ "/.claude/settings.local.json") == 1
+
+
+# ---- folding and the ledger ----------------------------------------------------------------
+def test_each_session_is_folded_once_and_priced(env, repo, tmp_path):
+ t = trap_session(str(tmp_path / "s.jsonl"), repo, 0)
+ run_session(env, repo, "s", t)
+ call(env, {"hook_event_name": "SessionEnd", "session_id": "s", "transcript_path": t,
+ "cwd": repo, "reason": "other"})
+ ledger = store_for(env, repo).read_lines("ledger.jsonl")
+ assert len(ledger) == 1
+ rec = ledger[0]
+ # 3 requests x (10 fresh + 0.1 x 1000 cached + 1.25 x 100 written + 5 x 50 output)
+ assert rec["requests"] == 3 and rec["effective_tokens"] == pytest.approx(3 * 485.0)
+ assert rec["priced_by"] == "model" and set(rec["overhead"]) == {
+ "optimizer_tokens", "spiral_judge_tokens", "spiral_nudge_writer_tokens"}
+
+
+def test_a_fold_that_cannot_take_the_lock_skips_and_the_session_is_still_folded_once(
+ env, repo, tmp_path, monkeypatch):
+ """The double fold seen live: a paid fold holds the lock through its side call, a second
+ fold of the same session gave up waiting after 10 s and folded it again, unlocked."""
+ import fcntl
+ from te_live import fold, hosts
+ from te_live.context import LiveContext
+ t = trap_session(str(tmp_path / "s.jsonl"), repo, 0)
+ base = {"session_id": "s", "transcript_path": t, "cwd": repo}
+ call(env, dict(base, hook_event_name="SessionStart", source="startup"))
+ st = store_for(env, repo)
+ monkeypatch.setattr(fold, "FOLD_LOCK_WAIT_S", 0.2)
+ holder = open(st.path(".fold.lock"), "a+") # another fold, mid side call
+ fcntl.flock(holder.fileno(), fcntl.LOCK_EX)
+ try:
+ call(env, dict(base, hook_event_name="SessionEnd", reason="other")) # folds inline
+ assert st.read_lines("ledger.jsonl") == []
+ assert "s" not in (st.read("folded.json") or {})
+ ctx = LiveContext.build(hosts.HOSTS["claude"](), repo, env)
+ assert "s" in fold.pending_sessions(ctx) # left for the holder, or a later start
+ finally:
+ fcntl.flock(holder.fileno(), fcntl.LOCK_UN)
+ holder.close()
+ assert fold.fold_session(ctx, "s") is not None
+ assert fold.fold_session(ctx, "s") is None
+ assert len(st.read_lines("ledger.jsonl")) == 1
+
+
+def test_a_session_whose_end_was_missed_is_folded_by_the_next_start(env, repo, tmp_path):
+ t = trap_session(str(tmp_path / "lost.jsonl"), repo, 0)
+ call(env, {"hook_event_name": "SessionStart", "session_id": "lost", "source": "startup",
+ "transcript_path": t, "cwd": repo}) # ... then the terminal died
+ old = os.path.getmtime(t) - 3600
+ os.utime(t, (old, old))
+ t2 = str(tmp_path / "next.jsonl")
+ open(t2, "w").close()
+ call(env, {"hook_event_name": "SessionStart", "session_id": "next", "source": "startup",
+ "transcript_path": t2, "cwd": repo})
+ assert "lost" in store_for(env, repo).read("folded.json")
+
+
+# ---- never break the session ---------------------------------------------------------------
+def test_garbage_input_and_internal_errors_answer_no_change(env, repo, monkeypatch):
+ out = io.StringIO()
+ hook.main([], io.StringIO("not json"), out, env)
+ assert json.loads(out.getvalue()) == {}
+
+ def boom(self):
+ raise RuntimeError("bug")
+ monkeypatch.setattr(LiveController, "handle", boom)
+ assert call(env, {"hook_event_name": "Stop", "session_id": "s", "cwd": repo}) == {}
+ log = open(os.path.join(env["TOKENECONOMIZER_HOME"], "errors.log")).read()
+ assert "RuntimeError: bug" in log
+
+
+def test_side_calls_a_disabled_plugin_and_a_payload_without_a_session_do_nothing(env, repo):
+ for flag in ("TE_SIDECALL", "TE_DISABLE"):
+ assert call(dict(env, **{flag: "1"}), {"hook_event_name": "SessionStart",
+ "session_id": "s", "cwd": repo}) == {}
+ assert call(env, {"hook_event_name": "SessionStart", "cwd": repo}) == {}
+ assert not os.path.exists(env["TOKENECONOMIZER_HOME"])
+
+
+def test_mechanism_prints_cannot_corrupt_the_hook_output(env, repo, tmp_path, capsys):
+ """ToolingStage prints a warning when it parses no catalog; inline folds must not leak it."""
+ t = str(tmp_path / "empty.jsonl")
+ open(t, "w").close()
+ out = io.StringIO()
+ hook.main([], io.StringIO(json.dumps({"hook_event_name": "SessionEnd", "session_id": "e",
+ "transcript_path": t, "cwd": repo})), out, env)
+ assert json.loads(out.getvalue()) == {}
+
+
+# ---- Pi, through the payloads its extension sends -------------------------------------------
+def pi_trap_session(path, repo, i):
+ """The Claude trap session, as a Pi session file."""
+ lines = [{"type": "session", "version": 3, "id": f"pi{i}", "timestamp": "2026-09-30T00:00:00Z",
+ "cwd": repo}]
+ calls = [("edit", {"path": f"{repo}/pkg/mod{i}.py", "oldText": "a", "newText": "b"}, "ok", False),
+ ("bash", {"command": "pytest -q tests/test_a.py"}, "bash: pytest: command not found", True),
+ ("bash", {"command": "python -m pytest -q tests/test_a.py"}, "1 passed in 0.1s", False)]
+ parent, n = None, 0
+ entries = [{"role": "user", "content": f"fix bug {i}"}]
+ for name, args, res, err in calls:
+ entries.append({"role": "assistant", "model": "gpt-5.6-sol", "usage": {
+ "input": 10, "output": 50, "cacheRead": 1000, "cacheWrite": 100, "totalTokens": 1160,
+ "cost": {"total": 0.001}}, "content": [{"type": "toolCall", "id": f"c{name}{n}",
+ "name": name, "arguments": args}]})
+ entries.append({"role": "toolResult", "toolCallId": f"c{name}{n}", "toolName": name,
+ "isError": err, "content": [{"type": "text", "text": res}]})
+ n += 1
+ for k, m in enumerate(entries):
+ eid = f"{i}-{k}"
+ lines.append({"type": "message", "id": eid, "parentId": parent,
+ "timestamp": "2026-09-30T00:00:01Z", "message": m})
+ parent = eid
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(x) for x in lines) + "\n")
+ return path
+
+
+def test_pi_learns_from_its_own_session_files_and_injects_at_the_next_start(env, repo, tmp_path):
+ starts = [run_session(env, repo, f"pi{i}", pi_trap_session(str(tmp_path / f"p{i}.jsonl"), repo, i),
+ host="pi") for i in range(5)]
+ assert all(s == {} for s in starts[:4])
+ ctx = starts[4]["hookSpecificOutput"]["additionalContext"]
+ assert "`python -m pytest …` works." in ctx
+ assert "/te facts" in starts[4]["systemMessage"] # Pi's own command name
+ rec = store_for(env, repo, "pi").read_lines("ledger.jsonl")[0]
+ assert rec["requests"] == 3 and rec["priced_by"] == "model"
+
+
+def test_a_headless_pi_turn_that_keeps_spinning_is_stopped(env, repo):
+ headless = dict(env, TE_PI_MODE="print", TE_SPIRAL="guard")
+ for k in range(5):
+ _turn(headless, repo, "p", f"g{k}", [("read", {"path": "/r/a.py"}, {"content": "ok"})] * 8,
+ host="pi")
+ fail = {"content": "FAILED tests/test_a.py::test_x - AssertionError\nCommand exited with code 1",
+ "isError": True, "exit_code": 1}
+ calls = [("edit", {"path": "/r/pkg/a.py", "oldText": "x", "newText": "y"}, {"content": "ok"})
+ if k % 2 == 0 else ("bash", {"command": "pytest -q"}, fail) for k in range(40)]
+ outs = _turn(headless, repo, "p", "stuck", calls, host="pi")
+ assert any("hookSpecificOutput" in o for o in outs)
+ assert any(o.get("continue") is False for o in outs)
+
+
+# ---- GitHub Copilot CLI, through the payloads it sends -------------------------------------
+from tests.test_live_core import COPILOT_POST, copilot_events # noqa: E402
+
+GITHUB_TOOLS = ["github-mcp-server-list_issues", "github-mcp-server-get_commit",
+ "github-mcp-server-search_code"]
+
+
+def copilot_session(env, repo, tmp_path, sid, calls, monkeypatch, source="new"):
+ """One Copilot session, as its hook file delivers it: no transcript path before Stop (the
+ log's place is fixed by the session id), Claude tool names, a top-level context answer."""
+ home = tmp_path / "copilot-home"
+ monkeypatch.setenv("COPILOT_HOME", str(home))
+ copilot_events(str(home / "session-state" / sid / "events.jsonl"), calls, cwd=repo, sid=sid)
+ base = {"session_id": sid, "cwd": repo, "timestamp": "2026-10-08T04:22:15.157Z"}
+ start = call(env, dict(base, hook_event_name="SessionStart", source=source), "copilot")
+ call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="fix it"), "copilot")
+ for name, args, text, ok in calls:
+ if ok:
+ call(env, dict(base, hook_event_name="PostToolUse", tool_name=name.capitalize(),
+ tool_input=args, tool_result={"result_type": "success",
+ "text_result_for_llm": text}),
+ "copilot")
+ else:
+ call(env, dict(base, hook_event_name="PostToolUseFailure",
+ tool_name=name.capitalize(), tool_input=args, error=text), "copilot")
+ call(env, dict(base, hook_event_name="Stop", stop_reason="end_turn",
+ stop_hook_active=False), "copilot")
+ call(env, dict(base, hook_event_name="SessionEnd", reason="complete"), "copilot")
+ return start
+
+
+COPILOT_TRAP = [("bash", {"command": "pytest -q tests/test_a.py"},
+ "bash: pytest: command not found\n<shellId: 0 completed with exit code 127>", True),
+ ("bash", {"command": "python -m pytest -q tests/test_a.py"},
+ "1 passed in 0.1s\n<shellId: 1 completed with exit code 0>", True)]
+
+
+def test_copilot_learns_from_its_own_logs_and_injects_at_top_level(env, repo, tmp_path,
+ monkeypatch):
+ starts = [copilot_session(env, repo, tmp_path, f"cp{i}", COPILOT_TRAP, monkeypatch)
+ for i in range(5)]
+ assert all(s == {} for s in starts[:4])
+ assert "hookSpecificOutput" not in starts[4] # Copilot ignores that shape
+ assert "`python -m pytest …` works." in starts[4]["additionalContext"]
+ # Copilot never shows systemMessage; its TUI shows a progress line printed first
+ assert "systemMessage" not in starts[4]
+ (notice,) = starts[4]["progress"]
+ assert "learned fact" in notice and "/tokeneconomizer:facts" in notice
+ st = store_for(env, repo, "copilot")
+ rec = st.read_lines("ledger.jsonl")[0]
+ assert rec["host"] == "copilot" and rec["requests"] == 4 and rec["priced_by"] == "model"
+ assert rec["input_tokens"] == 4000 - 2400 - 100
+ turns = [e for e in st.read_lines("events.jsonl") if e["kind"] == "turn"]
+ assert len(turns) == 5 and all(t["tool_calls"] == 2 for t in turns)
+
+
+def test_a_failed_copilot_tool_reaches_the_guard(env, repo, tmp_path, monkeypatch):
+ copilot_session(env, repo, tmp_path, "f", [
+ ("view", {"path": f"{repo}/nope.py"}, "Path does not exist", False)], monkeypatch)
+ (turn,) = [e for e in store_for(env, repo, "copilot").read_lines("events.jsonl")
+ if e["kind"] == "turn"]
+ assert turn["tool_calls"] == 1
+
+
+def _copilot_catalog(env, tools):
+ d = os.path.join(env["TOKENECONOMIZER_HOME"], "hosts", "copilot")
+ os.makedirs(d, exist_ok=True)
+ json.dump({"tools": tools}, open(os.path.join(d, "catalog.json"), "w"))
+
+
+def test_copilot_proposes_only_whole_unused_mcp_servers(env, repo, tmp_path, monkeypatch):
+ _copilot_catalog(env, ["bash", "view", "sql", "fetch_copilot_cli_documentation"]
+ + GITHUB_TOOLS)
+ for i in range(5):
+ copilot_session(env, repo, tmp_path, f"u{i}", [
+ ("bash", {"command": "ls"}, "a.py\n<shellId: 0 completed with exit code 0>", True)],
+ monkeypatch)
+ st = store_for(env, repo, "copilot")
+ unt = st.read("untool.json")
+ # builtins went unused too, but Copilot has no persistent per-tool lever: only the server
+ assert unt["proposal"] == ["github-mcp-server"]
+ assert unt["servers"]["github-mcp-server"] == {"tools": 3, "used": False,
+ "unused_sessions": 5}
+ assert not os.path.exists(untool.settings_path(repo, "copilot"))
+ (notice,) = copilot_session(env, repo, tmp_path, "n", [], monkeypatch)["progress"]
+ assert "1 MCP server(s) went unused" in notice and "/tokeneconomizer:untool" in notice
+
+
+def test_a_server_whose_tool_was_used_once_is_never_proposed(env, repo, tmp_path, monkeypatch):
+ _copilot_catalog(env, ["bash"] + GITHUB_TOOLS)
+ copilot_session(env, repo, tmp_path, "used", [
+ ("github-mcp-server-search_code", {"query": "x"}, "[]", True)], monkeypatch)
+ for i in range(8):
+ copilot_session(env, repo, tmp_path, f"u{i}", [
+ ("bash", {"command": "ls"}, "a.py", True)], monkeypatch)
+ unt = store_for(env, repo, "copilot").read("untool.json")
+ assert unt["proposal"] == [] and unt["servers"]["github-mcp-server"]["used"] is True
+
+
+def test_copilot_apply_and_restore_touch_only_the_plugins_own_servers(repo, tmp_path):
+ st = Store(str(tmp_path / "st"))
+ path = untool.settings_path(repo, "copilot")
+ assert path.endswith(os.path.join(".github", "copilot", "settings.local.json"))
+ os.makedirs(os.path.dirname(path))
+ json.dump({"disabledMcpServers": ["mine"], "model": "gpt-5.4"}, open(path, "w"))
+ assert untool.apply(repo, ["github-mcp-server"], st, host="copilot") == ["github-mcp-server"]
+ assert json.load(open(path)) == {"disabledMcpServers": ["mine", "github-mcp-server"],
+ "model": "gpt-5.4"}
+ assert untool.restore(repo, st, host="copilot") == ["github-mcp-server"]
+ assert json.load(open(path)) == {"disabledMcpServers": ["mine"], "model": "gpt-5.4"}
+ os.remove(path)
+ untool.apply(repo, ["github-mcp-server"], st, host="copilot") # created: kept out of git
+ assert "/.github/copilot/settings.local.json" in open(
+ os.path.join(repo, ".git", "info", "exclude")).read().split()
+
+
+# ---- Codex capability untooling ----------------------------------------------------------
+def codex_rollout(path, repo, i, exec_inputs=()):
+ """A Codex rollout: one shell command and, per `exec_inputs`, one code-mode exec call."""
+ recs = [{"type": "session_meta", "payload": {"id": f"thread-{i}", "cwd": repo,
+ "cli_version": "0.153.1"}},
+ {"type": "turn_context", "payload": {"turn_id": "t1", "model": "gpt-5.6-sol"}},
+ {"type": "event_msg", "payload": {"type": "item_completed", "turn_id": "t1", "item": {
+ "type": "CommandExecution", "id": f"c{i}", "command": ["bash", "-lc", "ls"],
+ "aggregated_output": "a.py", "exit_code": 0, "status": "completed"}}}]
+ for k, src in enumerate(exec_inputs):
+ recs.append({"type": "response_item", "payload": {
+ "type": "custom_tool_call", "name": "exec", "call_id": f"x{i}-{k}",
+ "status": "completed", "input": src}})
+ recs.append({"type": "event_msg", "payload": {"type": "token_count", "info": {
+ "total_token_usage": {"input_tokens": 3000, "cached_input_tokens": 2000,
+ "cache_write_input_tokens": 0, "output_tokens": 90,
+ "reasoning_output_tokens": 10}}}})
+ with open(path, "w") as fh:
+ fh.write("\n".join(json.dumps(r) for r in recs) + "\n")
+ return path
+
+
+def test_codex_proposes_capabilities_it_never_saw_used(env, repo, tmp_path):
+ for i in range(5):
+ uses = ["await tools.view_image({path: 'a.png'})"] if i == 2 else []
+ t = codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"), repo, i, uses)
+ run_session(env, repo, f"cx{i}", t, host="codex")
+ unt = store_for(env, repo, "codex").read("untool.json")
+ # collaboration and goals went unused in all five; image was used once and is kept
+ assert unt["proposal"] == ["collaboration", "goals"]
+ assert unt["capabilities"]["image_media"]["used"] is True
+ assert unt["capabilities"]["goals"] == {"loaded_sessions": 5, "unused_sessions": 5,
+ "used": False}
+ assert not os.path.exists(untool.settings_path(repo, "codex")) # nothing written yet
+
+
+def test_codex_needs_the_full_evidence_before_proposing(env, repo, tmp_path):
+ """The experiments' controller demotes collaboration after ONE unused task; a person's
+ sessions need `untool_evidence` of them."""
+ for i in range(4):
+ run_session(env, repo, f"cx{i}", codex_rollout(str(tmp_path / f"rollout-{i}.jsonl"),
+ repo, i), host="codex")
+ assert store_for(env, repo, "codex").read("untool.json")["proposal"] == []
+
+
+def codex_accepts(monkeypatch):
+ """Codex's own check of the edited config (`codex features list`), as a stub: the tests
+ must not depend on a local Codex install or its user config."""
+ verdicts = []
+ monkeypatch.setattr(untool, "codex_config_error", lambda root, **kw: (
+ verdicts.pop(0) if verdicts else None))
+ return verdicts
+
+
+def test_codex_apply_writes_marked_lines_and_restore_leaves_the_users_config(repo, tmp_path,
+ codex_accepts):
+ st = Store(str(tmp_path / "st"))
+ path = untool.settings_path(repo, "codex")
+ os.makedirs(os.path.dirname(path))
+ user = ('model_reasoning_effort = "high"\n\n[features]\nhooks = true\n\n'
+ '[[skills.config]]\npath = "/x/SKILL.md"\nenabled = false\n')
+ open(path, "w").write(user)
+ assert untool.apply(repo, ["collaboration", "goals", "image_media"], st, host="codex") == [
+ "collaboration", "goals", "image_media"]
+ text = open(path).read()
+ lines = text.splitlines()
+ # top-level key above the first table; feature keys inside the user's [features];
+ # [agents] appended as a new table after the array of tables
+ assert lines.index("include_collaboration_mode_instructions = false "
+ "# tokeneconomizer:collaboration") < lines.index("[features]")
+ feats = lines[lines.index("[features]") + 1:lines.index("hooks = true")]
+ assert "goals = false # tokeneconomizer:goals" in feats
+ assert "view_image = false # tokeneconomizer:image_media" in feats
+ assert lines[-2:] == ["[agents] # tokeneconomizer:collaboration",
+ "enabled = false # tokeneconomizer:collaboration"]
+ assert untool.restore(repo, st, ["goals"], host="codex") == ["goals"]
+ assert "goals = false" not in open(path).read()
+ assert untool.restore(repo, st, host="codex") == ["collaboration", "image_media"]
+ assert open(path).read() == user.rstrip("\n") + "\n"
+
+
+def test_a_config_codex_rejects_is_put_back(repo, tmp_path, codex_accepts):
+ st = Store(str(tmp_path / "st"))
+ path = untool.settings_path(repo, "codex")
+ os.makedirs(os.path.dirname(path))
+ open(path, "w").write('model = "gpt-5.6-sol"\n')
+ codex_accepts.append("Error loading config.toml: duplicate key")
+ with pytest.raises(RuntimeError, match="put back"):
+ untool.apply(repo, ["goals"], st, host="codex")
+ assert open(path).read() == 'model = "gpt-5.6-sol"\n' and untool.applied(st) == []
+ os.remove(path)
+ codex_accepts.append("bad")
+ with pytest.raises(RuntimeError):
+ untool.apply(repo, ["goals"], st, host="codex")
+ assert not os.path.exists(path) # a file it created is removed again
+
+
+def test_a_committed_codex_config_is_never_edited(tmp_path, codex_accepts):
+ import subprocess
+ r = tmp_path / "shared"
+ (r / ".codex").mkdir(parents=True)
+ (r / ".codex" / "config.toml").write_text('model = "gpt-5.6-sol"\n')
+ git = ["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false"]
+ subprocess.run(["git", "init", "-q", str(r)], check=True)
+ subprocess.run(git + ["-C", str(r), "add", ".codex/config.toml"], check=True)
+ subprocess.run(git + ["-C", str(r), "commit", "-qm", "team config"], check=True)
+ with pytest.raises(RuntimeError, match="committed"):
+ untool.apply(str(r), ["goals"], Store(str(tmp_path / "st")), host="codex")
+ assert (r / ".codex" / "config.toml").read_text() == 'model = "gpt-5.6-sol"\n'
+
+
+def test_codex_never_overrides_a_key_the_user_set(tmp_path):
+ text, did = untool._toml_insert('[features]\ngoals = true\n', "features", "goals", "false",
+ "# t")
+ assert not did and text == '[features]\ngoals = true\n'
+ text, did = untool._toml_insert('features.goals = true\n', "features", "goals", "false", "# t")
+ assert not did
+ # dotted top-level keys and no [features] table: stay dotted, never add a clashing table
+ text, did = untool._toml_insert('features.hooks = true\n', "features", "view_image", "false",
+ "# t")
+ assert did and "[features]" not in text and "features.view_image = false # t" in text
+ # an inline table cannot be extended, and a [features] table beside it would be invalid
+ text, did = untool._toml_insert('features = { hooks = true }\n', "features", "goals",
+ "false", "# t")
+ assert not did and text == 'features = { hooks = true }\n'
+ # a header the plugin added, under which the user has since put a key, is kept
+ kept = untool._toml_remove('[agents] # t\nenabled = false # t\nmax_threads = 4\n', ["# t"])
+ assert kept == "[agents] # t\nmax_threads = 4\n"
+
+
+def test_codex_trust_is_read_from_the_users_config(tmp_path, repo):
+ home = tmp_path / "codex-home"
+ home.mkdir()
+ env = {"CODEX_HOME": str(home)}
+ assert untool.codex_project_trusted(repo, env) is None
+ (home / "config.toml").write_text(f'[projects."{repo}"]\ntrust_level = "trusted"\n')
+ assert untool.codex_project_trusted(repo, env) is True
+ (home / "config.toml").write_text(f'[projects."{repo}/sub"]\ntrust_level = "trusted"\n'
+ f'[projects."{repo}"]\ntrust_level = "untrusted"\n')
+ assert untool.codex_project_trusted(repo, env) is False
+
+
+def test_copilot_trust_is_read_from_its_config(tmp_path, repo):
+ home = tmp_path / "copilot-home"
+ home.mkdir()
+ env = {"COPILOT_HOME": str(home)}
+ assert untool.copilot_folder_trusted(repo, env) is None
+ # Copilot's config.json opens with `//` comment lines; trust covers subfolders
+ (home / "config.json").write_text(
+ "// User settings belong in settings.json.\n// This file is managed automatically.\n"
+ + json.dumps({"trustedFolders": [os.path.dirname(repo)]}))
+ assert untool.copilot_folder_trusted(repo, env) is True
+ (home / "config.json").write_text(json.dumps({"trustedFolders": [repo + "-other"]}))
+ assert untool.copilot_folder_trusted(repo, env) is False
diff --git a/tests/test_live_safety.py b/tests/test_live_safety.py
new file mode 100644
index 0000000..770005c
--- /dev/null
+++ b/tests/test_live_safety.py
@@ -0,0 +1,100 @@
+"""What the plugin writes stays where it belongs and stays private.
+
+ * A checkout decides what `.claude`, `.codex`, `.github` and `.git` are, so a write the
+ deny list makes must not follow a symlink, or a `.git` file, out of the repository.
+ * A settings file the plugin cannot parse is the person's: refused, never written over.
+ * The state root holds what was derived from the agents' private session logs, and the
+ launcher the slash commands run: it is private to its owner.
+"""
+import json
+import os
+import stat
+import subprocess
+import sys
+
+import pytest
+
+from te_live import untool
+from te_live.paths import ensure_launcher, secure_root
+from te_live.store import Store
+from tests.test_live_hooks import repo # noqa: F401
+
+PLUGIN = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "plugin")
+
+
+def _mode(path):
+ return stat.S_IMODE(os.stat(path).st_mode)
+
+
[email protected]("host,rel", [("claude", ".claude"), ("copilot", ".github"),
+ ("codex", ".codex")])
+def test_a_symlinked_settings_directory_is_refused(repo, tmp_path, host, rel):
+ outside = tmp_path / "outside"
+ outside.mkdir()
+ os.symlink(outside, os.path.join(repo, rel))
+ item = {"claude": "Workflow", "copilot": "github-mcp-server", "codex": "goals"}[host]
+ with pytest.raises(RuntimeError, match="symlink"):
+ untool.apply(repo, [item], Store(str(tmp_path / "st")), host=host)
+ assert list(outside.rglob("*")) == []
+
+
+def test_a_symlinked_settings_file_is_refused_and_its_target_untouched(repo, tmp_path):
+ target = tmp_path / "elsewhere.json"
+ target.write_text('{"keep": true}\n')
+ os.makedirs(os.path.join(repo, ".claude"))
+ os.symlink(target, os.path.join(repo, ".claude", "settings.local.json"))
+ st = Store(str(tmp_path / "st"))
+ with pytest.raises(RuntimeError, match="symlink"):
+ untool.apply(repo, ["Workflow"], st)
+ assert untool.restore(repo, st, ["Workflow"]) == [] # nothing of its own to remove
+ assert target.read_text() == '{"keep": true}\n'
+
+
+def test_a_settings_file_that_does_not_parse_is_refused_not_overwritten(repo, tmp_path):
+ path = os.path.join(repo, ".claude", "settings.local.json")
+ os.makedirs(os.path.dirname(path))
+ broken = '{"permissions": {"allow": ["Bash(ls)"],}}\n' # a trailing comma
+ open(path, "w").write(broken)
+ with pytest.raises(RuntimeError, match="not valid JSON"):
+ untool.apply(repo, ["Workflow"], Store(str(tmp_path / "st")))
+ assert open(path).read() == broken
+ open(path, "w").write("") # empty is the same as none
+ assert untool.apply(repo, ["Workflow"], Store(str(tmp_path / "st2"))) == ["Workflow"]
+ assert json.load(open(path)) == {"permissions": {"deny": ["Workflow"]}}
+
+
+def test_a_git_file_pointing_out_of_the_repository_gets_no_exclude_line(repo, tmp_path):
+ import shutil
+ shutil.rmtree(os.path.join(repo, ".git"))
+ elsewhere = tmp_path / "not-a-git-dir"
+ elsewhere.mkdir()
+ open(os.path.join(repo, ".git"), "w").write(f"gitdir: {elsewhere}\n")
+ untool.apply(repo, ["Workflow"], Store(str(tmp_path / "st")))
+ assert list(elsewhere.rglob("*")) == []
+
+
+def test_a_symlinked_exclude_file_is_not_written_through(repo, tmp_path):
+ victim = tmp_path / "victim"
+ victim.write_text("keep\n")
+ os.symlink(victim, os.path.join(repo, ".git", "info", "exclude"))
+ untool.apply(repo, ["Workflow"], Store(str(tmp_path / "st")))
+ assert victim.read_text() == "keep\n"
+
+
+def test_a_loose_state_root_is_made_private_and_the_launcher_is_owner_only(tmp_path):
+ env = {"TOKENECONOMIZER_HOME": str(tmp_path / "home"), "TE_PLUGIN_ROOT": PLUGIN}
+ os.makedirs(env["TOKENECONOMIZER_HOME"], mode=0o775)
+ os.chmod(env["TOKENECONOMIZER_HOME"], 0o775)
+ secure_root(env)
+ assert _mode(env["TOKENECONOMIZER_HOME"]) == 0o700
+ assert _mode(ensure_launcher(env)) == 0o700
+
+
+def test_the_command_line_creates_its_state_private(repo, tmp_path):
+ env = dict(os.environ, TOKENECONOMIZER_HOME=str(tmp_path / "home"))
+ subprocess.run(["sh", os.path.join(PLUGIN, "scripts", "te"), "--cwd", repo, "config", "set",
+ "spiral=off"], env=dict(env, TE_PYTHON=sys.executable), check=True,
+ capture_output=True)
+ made = [os.path.join(d, n) for d, ds, fs in os.walk(env["TOKENECONOMIZER_HOME"])
+ for n in ds + fs] + [env["TOKENECONOMIZER_HOME"]]
+ assert made and all(_mode(p) & 0o077 == 0 for p in made), {p: oct(_mode(p)) for p in made}
diff --git a/tests/test_plugin_package.py b/tests/test_plugin_package.py
new file mode 100644
index 0000000..33b8776
--- /dev/null
+++ b/tests/test_plugin_package.py
@@ -0,0 +1,224 @@
+"""The installable plugin: manifests, hook files, launchers, and the self-contained bundle.
+
+These pin the packaging facts a host relies on and a refactor could silently break: every hook
+file runs the one launcher that exists and is executable, the Codex file names its host and keeps
+SessionEnd under Codex's three-second cap, the marketplace points at the plugin directory, the
+launcher fails open, and the bundle `scripts/build_plugin.py` writes runs with no checkout beside
+it -- which is what a marketplace install, copying only the plugin directory, actually gets.
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+
+import pytest
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+PLUGIN = os.path.join(REPO, "plugin")
+EVENTS = {"SessionStart", "UserPromptSubmit", "PostToolUse", "Stop", "SessionEnd"}
+
+
+def _json(*parts):
+ with open(os.path.join(*parts), encoding="utf-8") as fh:
+ return json.load(fh)
+
+
+def _commands(hooks_doc):
+ for event, groups in hooks_doc["hooks"].items():
+ for group in groups:
+ for h in group["hooks"]:
+ yield event, group, h
+
+
+def test_manifests_agree_on_identity():
+ claude = _json(PLUGIN, ".claude-plugin", "plugin.json")
+ codex = _json(PLUGIN, ".codex-plugin", "plugin.json")
+ # Copilot reads `.github/plugin/plugin.json` in preference to `.claude-plugin/` (verified on
+ # 1.0.83: only the hooks it names ran), which is how it gets a hook file of its own.
+ copilot = _json(PLUGIN, ".github", "plugin", "plugin.json")
+ assert claude["name"] == codex["name"] == copilot["name"] == "tokeneconomizer"
+ assert claude["version"] == codex["version"] == copilot["version"]
+ assert codex["hooks"] == "./hooks/codex-hooks.json"
+ assert copilot["hooks"] == "hooks/copilot-hooks.json"
+ for m in (codex, copilot):
+ assert os.path.exists(os.path.join(PLUGIN, m["hooks"]))
+ assert "hooks" not in claude # Claude Code reads hooks/hooks.json by convention
+
+
[email protected]("name,host", [("hooks.json", None), ("codex-hooks.json", "codex"),
+ ("copilot-hooks.json", "copilot")])
+def test_hook_files_run_the_launcher_on_every_event(name, host):
+ doc = _json(PLUGIN, "hooks", name)
+ # no PreToolUse: the plugin never blocks a call. Copilot reports a failed tool in an event
+ # of its own, which the guard must see too.
+ want = EVENTS | ({"PostToolUseFailure"} if host == "copilot" else set())
+ assert set(doc["hooks"]) == want
+ launcher = os.path.join(PLUGIN, "scripts", "te-hook")
+ assert os.access(launcher, os.X_OK)
+ for event, group, h in _commands(doc):
+ assert h["type"] == "command" and "/scripts/te-hook" in h["command"]
+ assert "matcher" not in group # PostToolUse watches every tool
+ if host:
+ assert f"--host {host}" in h["command"]
+ if host == "copilot":
+ assert h["command"].startswith('"${PLUGIN_ROOT}/scripts/te-hook"')
+ if event == "SessionEnd" and host == "codex":
+ assert h["timeout"] <= 3 # Codex's cap for SessionEnd
+
+
+def test_marketplace_lists_the_plugin_directory():
+ m = _json(REPO, ".claude-plugin", "marketplace.json")
+ entry = {p["name"]: p for p in m["plugins"]}["tokeneconomizer"]
+ assert entry["source"] == "./plugin"
+ assert os.path.isdir(os.path.join(REPO, entry["source"], ".claude-plugin"))
+
+
+def test_no_bin_directory():
+ """Files in a plugin's bin/ go on the Bash tool's PATH, and claude.ai will not install a
+ plugin that has one; the launchers live in scripts/."""
+ assert not os.path.exists(os.path.join(PLUGIN, "bin"))
+
+
+def _run_hook(launcher, payload, env):
+ p = subprocess.run([launcher], input=json.dumps(payload), capture_output=True, text=True,
+ env=env, timeout=60)
+ return p.returncode, p.stdout, p.stderr
+
+
+def _hook_env(tmp_path, **extra):
+ env = {"PATH": os.environ.get("PATH", ""), "TOKENECONOMIZER_HOME": str(tmp_path / "home"),
+ "GIT_CEILING_DIRECTORIES": str(tmp_path), "TE_FOLD_SYNC": "1"}
+ env.update(extra)
+ return env
+
+
+def test_the_launcher_runs_the_dispatcher_from_the_checkout(tmp_path):
+ work = tmp_path / "work"
+ work.mkdir()
+ code, out, _ = _run_hook(os.path.join(PLUGIN, "scripts", "te-hook"),
+ {"hook_event_name": "SessionStart", "session_id": "s",
+ "cwd": str(work), "source": "startup"}, _hook_env(tmp_path))
+ assert code == 0 and json.loads(out) == {}
+ assert os.path.isdir(tmp_path / "home" / "repos") # it ran, and recorded the session
+
+
+def test_the_launcher_fails_open_without_python(tmp_path):
+ code, out, _ = _run_hook(os.path.join(PLUGIN, "scripts", "te-hook"),
+ {"hook_event_name": "SessionStart"},
+ {"PATH": "", "TE_PYTHON": "/nonexistent/python"})
+ assert code == 0 and json.loads(out) == {}
+
+
+def test_the_bootstrap_answers_no_change_on_an_old_python(tmp_path, monkeypatch):
+ sys.path.insert(0, os.path.join(PLUGIN, "scripts"))
+ try:
+ import te_bootstrap
+ finally:
+ sys.path.pop(0)
+ monkeypatch.setattr(te_bootstrap, "MIN_VERSION", (99, 0))
+ monkeypatch.setattr(sys, "argv", ["te_bootstrap.py", "hook"])
+ import io
+ monkeypatch.setattr(sys, "stdin", io.StringIO("{}"))
+ out = io.StringIO()
+ monkeypatch.setattr(sys, "stdout", out)
+ assert te_bootstrap.main() == 0 and out.getvalue().strip() == "{}"
+
+
+def test_the_bundle_runs_without_a_checkout(tmp_path):
+ out = tmp_path / "market"
+ subprocess.run([sys.executable, os.path.join(REPO, "scripts", "build_plugin.py"),
+ "--out", str(out)], check=True, capture_output=True)
+ bundle = out / "tokeneconomizer"
+ assert (bundle / "lib" / "te_live" / "hook.py").exists()
+ assert (bundle / ".github" / "plugin" / "plugin.json").exists()
+ assert (bundle / "hooks" / "copilot-hooks.json").exists()
+ assert (bundle / "lib" / "tokeneconomizer" / "__init__.py").exists()
+ assert not (bundle / "lib" / "te_harness").exists()
+ assert not list(bundle.rglob("__pycache__"))
+ m = json.loads((out / ".claude-plugin" / "marketplace.json").read_text())
+ assert m["plugins"][0]["source"] == "./tokeneconomizer"
+ work = tmp_path / "work"
+ work.mkdir()
+ # Run the bundled launcher with the core reachable ONLY through the bundle's lib/.
+ code, stdout, stderr = _run_hook(str(bundle / "scripts" / "te-hook"),
+ {"hook_event_name": "SessionStart", "session_id": "s",
+ "cwd": str(work), "source": "startup"},
+ _hook_env(tmp_path, PYTHONNOUSERSITE="1"))
+ assert code == 0 and json.loads(stdout) == {}, stderr
+ sys.path.insert(0, str(bundle / "scripts"))
+ try:
+ import importlib
+ import te_bootstrap
+ importlib.reload(te_bootstrap)
+ assert te_bootstrap.core_path() == str(bundle / "lib")
+ finally:
+ sys.path.pop(0)
+
+
+# ---- the layering the README promises ------------------------------------------------------
+def test_import_tokeneconomizer_cannot_reach_the_live_harness():
+ """`te_live` shells out to agent CLIs (opt-in side calls, the catalog probe); the core must
+ not be able to, which is why it is a sibling package and not `tokeneconomizer.live`."""
+ probe = ("import sys, tokeneconomizer, tokeneconomizer.arms, tokeneconomizer.mechanisms; "
+ "print(any(m.split('.')[0] == 'te_live' for m in sys.modules))")
+ out = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True, cwd=REPO)
+ assert out.stdout.strip() == "False", out.stdout + out.stderr
+ assert not os.path.exists(os.path.join(REPO, "tokeneconomizer", "live"))
+
+
+def test_the_live_harness_is_stdlib_only_and_never_touches_the_benchmark_harness():
+ import ast
+ live = os.path.join(REPO, "te_live")
+ for name in sorted(os.listdir(live)):
+ if not name.endswith(".py"):
+ continue
+ tree = ast.parse(open(os.path.join(live, name), encoding="utf-8").read())
+ for node in ast.walk(tree):
+ mods = ([a.name for a in node.names] if isinstance(node, ast.Import) else
+ [node.module or ""] if isinstance(node, ast.ImportFrom) and node.level == 0
+ else [])
+ for mod in mods:
+ root = mod.split(".")[0]
+ assert root not in ("te_harness", "docker"), f"{name} imports {mod}"
+ # tomllib: standard library from 3.11, imported behind an ImportError guard
+ assert root in sys.stdlib_module_names or root in (
+ "tokeneconomizer", "te_live", "tomllib"), \
+ f"{name} imports non-stdlib {mod}"
+
+
+def test_the_plugin_directory_is_also_a_pi_package():
+ """One directory, three agents: Pi loads the extension its `pi` manifest names, and that
+ extension drives the same launchers the Claude Code and Codex hooks run."""
+ pkg = _json(PLUGIN, "package.json")
+ assert "pi-package" in pkg["keywords"]
+ (ext,) = pkg["pi"]["extensions"]
+ src = open(os.path.join(PLUGIN, ext), encoding="utf-8").read()
+ assert 'join(HERE, "..", "scripts")' in src and '"te-hook", ["--host", "pi"]' in src
+ # Exactly the dispatcher's five events. Not `tool_call`, the one Pi event that can block a
+ # call before it runs: the plugin guards nothing, on Pi as on the hook hosts.
+ assert set(re.findall(r'pi\.on\("(\w+)"', src)) == {
+ "session_start", "before_agent_start", "tool_result", "agent_end", "session_shutdown"}
+ # no lockfile: Claude Code installs a plugin's npm dependencies only when one exists,
+ # and this package has none to install (the Pi API is a peer dependency)
+ assert not any(os.path.exists(os.path.join(PLUGIN, f)) for f in
+ ("package-lock.json", "npm-shrinkwrap.json", "bun.lock", "bun.lockb"))
+
+
+def test_the_plugin_carries_the_core_as_symlinks_an_install_copy_follows(tmp_path):
+ """Claude Code installs a COPY of plugin/ into its cache, even from a local checkout, so the
+ core must be reachable from inside plugin/. Seen live: without plugin/lib the installed copy's
+ hooks answered `{}` and every command crashed."""
+ import shutil
+ for pkg in ("tokeneconomizer", "te_live"):
+ link = os.path.join(PLUGIN, "lib", pkg)
+ assert os.path.islink(link) and os.readlink(link) == f"../../{pkg}"
+ assert os.path.isfile(os.path.join(link, "__init__.py"))
+ # What an install does: copy the directory, following links. The copy runs on its own.
+ copy = str(tmp_path / "cache" / "tokeneconomizer" / "0.1.0")
+ shutil.copytree(PLUGIN, copy, symlinks=False,
+ ignore=shutil.ignore_patterns("__pycache__", "*.pyc"))
+ out = subprocess.run([os.path.join(copy, "scripts", "te"), "doctor"], capture_output=True,
+ text=True, env=dict(os.environ, TOKENECONOMIZER_HOME=str(tmp_path / "h")),
+ timeout=60)
+ assert out.returncode == 0 and "python" in out.stdout, out.stderr[-500:]
diff --git a/tests/test_trace.py b/tests/test_trace.py
new file mode 100644
index 0000000..f2539fc
--- /dev/null
+++ b/tests/test_trace.py
@@ -0,0 +1,156 @@
+"""Trace seam: load_trajectory / tool_usage / parse_catalog over synthetic transcripts."""
+import os
+
+from tokeneconomizer import trace
+from fixtures import write_claude_transcript, write_codex_transcript
+
+
+def test_load_trajectory_claude(tmp_path):
+ p = str(tmp_path / "t.jsonl")
+ write_claude_transcript(p, [
+ ("bash", {"command": "python -m pytest tests/test_x.py"}, "2 passed"),
+ ("edit", {"file_path": "pkg/mod.py"}, "ok")])
+ traj = trace.load_trajectory(p)
+ names = [name for name, _a, _r, _ok in traj]
+ assert "bash" in names and "edit" in names
+
+
+def test_tool_usage_counts(tmp_path):
+ p = str(tmp_path / "t.jsonl")
+ write_claude_transcript(p, [
+ ("bash", {"command": "ls"}, ""),
+ ("bash", {"command": "pwd"}, ""),
+ ("edit", {"file_path": "a.py"}, "")])
+ usage = trace.tool_usage(p)
+ assert usage.get("bash", 0) >= 2
+
+
+def test_parse_catalog_codex_default():
+ cat = trace.parse_catalog("/nonexistent", "codex")
+ assert "shell" in cat and "apply_patch" in cat
+
+
+def test_parse_catalog_missing_returns_empty(tmp_path):
+ # Claude path with no transcript file -> empty catalog, no crash.
+ assert trace.parse_catalog(str(tmp_path), "claude") == {}
+
+
+# ---- Copilot tool-catalog sources: one pipeline across CLI builds -------------------
+#
+# The CLI has moved the loaded tool catalog twice, and each move made the then-current
+# reader return {} -- which untooling cannot tell apart from "no tools loaded", so it
+# degraded to a silent no-op for a whole sweep (measured: CLI 1.0.82 dropped 8 of 17 tools
+# on 30/31 tasks; 1.0.83, same harness and repo, dropped nothing on 31/31). These tests
+# pin all three source formats and the auto-selection between them.
+import json as _json
+
+from tokeneconomizer import trace as _trace
+from tokeneconomizer.vendor import reflect as _reflect
+
+_SCHEMA_LOG = (
+ '{"body":"{\\"tools\\":[{\\"name\\":\\"bash\\",\\"title\\":null,'
+ '\\"description\\":\\"Run a shell command\\"},'
+ '{\\"name\\":\\"view\\",\\"description\\":\\"Read a file\\"}]}"}'
+)
+_FLAT_LOG = '{\n "tools": [\n "bash:66921ccc",\n "view:0963ceac",\n "edit:9e640c64"\n ]\n}'
+
+
+def _checkpoint_transcript(*names):
+ return _json.dumps({
+ "type": "session.usage_checkpoint",
+ "data": {"promptCacheBreakState": [
+ {"models": {"m": {"tools": [{"name": n} for n in names]}}}]},
+ })
+
+
+def test_flat_catalog_parses_name_fingerprint_form():
+ """CLI >= 1.0.82 publishes `"tools": ["bash:66921ccc", ...]` and no schemas."""
+ cat = _reflect.parse_tool_catalog_flat(_FLAT_LOG)
+ assert set(cat) == {"bash", "view", "edit"}
+ assert all(v == "" for v in cat.values()) # names only, no descriptions
+ assert _reflect.parse_tool_catalog(_FLAT_LOG) == {} # schema reader stays schema-only
+
+
+def test_schema_catalog_still_wins_and_keeps_descriptions(tmp_path):
+ (tmp_path / "copilot_debug.log").write_text(_SCHEMA_LOG + "\n" + _FLAT_LOG)
+ cat, src = _trace.parse_catalog_with_source(str(tmp_path), "copilot")
+ assert src == "schema_debug_log" # richest: it has descriptions
+ assert cat["bash"].startswith("Run a shell command")
+
+
+def test_flat_used_when_schemas_and_checkpoint_are_gone(tmp_path):
+ """The 1.0.83 shape: no schemas, no usage_checkpoint -- must NOT parse empty."""
+ (tmp_path / "copilot_debug.log").write_text(_FLAT_LOG)
+ (tmp_path / "transcript.jsonl").write_text('{"type":"assistant.message","data":{}}')
+ cat, src = _trace.parse_catalog_with_source(str(tmp_path), "copilot")
+ assert src == "flat_debug_log"
+ assert set(cat) == {"bash", "view", "edit"}
+
+
+def test_usage_checkpoint_source_is_still_read(tmp_path):
+ """The 1.0.82 shape must keep working -- supporting the new build is not a swap."""
+ (tmp_path / "transcript.jsonl").write_text(_checkpoint_transcript("bash", "view"))
+ cat, src = _trace.parse_catalog_with_source(str(tmp_path), "copilot")
+ assert src == "usage_checkpoint"
+ assert set(cat) == {"bash", "view"}
+
+
+def test_no_source_reports_none_not_empty_catalog(tmp_path):
+ """The regression that hid the bug: {} must be distinguishable from 'no tools'."""
+ (tmp_path / "copilot_debug.log").write_text("nothing useful here")
+ cat, src = _trace.parse_catalog_with_source(str(tmp_path), "copilot")
+ assert cat == {}
+ assert src is None # defect signal, not an empty catalog
+
+
+# ---- Web-Bench pass@1 vs pass@N ------------------------------------------------------
+#
+# Web-Bench grades each task with up to `retry` attempts, so a task has TWO defensible
+# resolve metrics, and they differ materially (measured on prisma: 1-4 tasks per arm).
+# Web-Bench's own headline is pass@1. The runner breaks out of the attempt loop on the
+# first pass, which makes pass@1 recoverable from older rows that predate the explicit
+# fields -- these pin that derivation, since getting it wrong reports pass@N under the
+# pass@1 column.
+def _p1p2(row):
+ """The aggregator's per-row pass@1/pass@N resolution, mirrored for test."""
+ p1, p2 = row.get("resolved_p1", ""), row.get("resolved_p2", "")
+ att = row.get("webbench_attempts", "")
+ if p1 == "" and p2 == "":
+ if att != "":
+ att = int(att or 0)
+ if att == 0:
+ return None, None
+ p2 = row["resolved"] is True
+ p1 = p2 and att == 1
+ else:
+ p1 = p2 = row["resolved"]
+ return p1, p2
+
+
+def test_explicit_attempt_fields_win():
+ assert _p1p2({"resolved": True, "resolved_p1": False,
+ "resolved_p2": True, "webbench_attempts": 2}) == (False, True)
+
+
+def test_legacy_row_second_attempt_pass_is_not_pass_at_1():
+ """resolved=True after 2 attempts means attempt 1 FAILED."""
+ assert _p1p2({"resolved": True, "webbench_attempts": 2}) == (False, True)
+
+
+def test_legacy_row_first_attempt_pass():
+ assert _p1p2({"resolved": True, "webbench_attempts": 1}) == (True, True)
+
+
+def test_legacy_row_failed_every_attempt():
+ assert _p1p2({"resolved": False, "webbench_attempts": 2}) == (False, False)
+
+
+def test_unreached_task_counts_in_neither_metric():
+ """Never attempted is not a miss -- it must not enter numerator or denominator."""
+ assert _p1p2({"resolved": False, "webbench_attempts": 0}) == (None, None)
+
+
+def test_single_attempt_domain_collapses_to_resolved():
+ """SWE-bench rows carry no attempt fields; pass@1 and pass@N are the same number."""
+ assert _p1p2({"resolved": True}) == (True, True)
+ assert _p1p2({"resolved": False}) == (False, False)
diff --git a/tokeneconomizer/__init__.py b/tokeneconomizer/__init__.py
new file mode 100644
index 0000000..31452e9
--- /dev/null
+++ b/tokeneconomizer/__init__.py
@@ -0,0 +1,28 @@
+"""TokenEconomizer: externally-delivered, agent-agnostic middleware that aims to cut a sealed
+coding agent's amortized token cost without hurting task success.
+
+Hold the sealed agent fixed (Claude Code, Copilot CLI, Codex); make the environment learn
+a cheaper interface to itself. Three independently-switchable mechanisms read recent agent
+traces and make adaptive changes to cut effective token cost, each at a different point in
+a task's lifecycle:
+
+ * injection -- inject a firing-gated, learned per-repo operating manual before the task.
+ * tooling -- drop inapplicable tool schemas / capability packages at task construction
+ (generic evidence untooling; Codex-native capability untooling).
+ * spiral -- kill or nudge a stuck rollout in flight (generic guard; Codex-native
+ post-green finalize).
+
+One normalized trace-reading seam (`trace`) and one honest cost ledger (`ledger`) are
+shared across all three. The mechanisms are harness-agnostic: task execution and the paid
+optimizer call enter through the `runner.AgentRunner` seam, and the task domain enters
+through a `DomainAdapter` (default SWE-bench). See `facade.Economizer` for the entry point.
+
+The exact task set the experiments ran -- which instances of which repos, per domain -- is
+data in `workloads`, so it is available on a bare clone with no results tree.
+"""
+from .config import (EconomizerConfig, InjectionConfig, ToolingConfig, SpiralConfig)
+from .facade import Economizer
+from . import ledger, trace, runner, workloads
+
+__all__ = ["Economizer", "EconomizerConfig", "InjectionConfig", "ToolingConfig",
+ "SpiralConfig", "ledger", "trace", "runner", "workloads"]
diff --git a/tokeneconomizer/_armspec.py b/tokeneconomizer/_armspec.py
new file mode 100644
index 0000000..44f47df
--- /dev/null
+++ b/tokeneconomizer/_armspec.py
@@ -0,0 +1,47 @@
+"""The `Arm` type and its two constructors, shared by the arm tables.
+
+A leaf module on purpose: `mechanisms/arms.py` and `baselines/arms.py` both import from
+here, and `tokeneconomizer/arms.py` imports both of them, so the primitives cannot live in
+either of those without a cycle.
+
+An `Arm` is (mode, config-factory, env-overlay, scaffold):
+
+* ``mode`` -- the agent's prompt-prefix scheme, which lands in the run-dir name so
+ ``aggregate.py`` can recover it. It is ``manual4`` exactly when the arm
+ enables injection (that is what manual4 *is*: the micro-skill prefix) and
+ ``standalone`` otherwise. `tests/test_sweep_arms.py` pins that, because
+ the field is written per arm rather than derived and could drift.
+* ``cfg`` -- a zero-arg factory returning this arm's `EconomizerConfig`. A factory,
+ not an instance, so no two arms can share mutable config state.
+* ``env`` -- harness-environment overlay applied BEFORE the runner is constructed, for
+ arm properties the economizer does not own (reasoning effort lives in the
+ agent CLI, not in a mechanism).
+* ``scaffold`` -- names an EXTERNAL TOOL BASELINE the harness installs into every task
+ container. **A name, not an object**: the string is resolved lazily by
+ the harness through ``tokeneconomizer.baselines.get()``. That is what lets
+ an arm table mention `rtk` without importing the baselines package, which
+ in turn is what keeps `import tokeneconomizer` free of their dependencies.
+
+A scaffold deliberately does not appear in `EconomizerConfig`: the economizer must not be
+able to see or reach one. The arm tables are where a mechanism config and a scaffold name
+meet, and the only place an arm name maps to either.
+"""
+import collections
+
+from .config import EconomizerConfig, InjectionConfig, ToolingConfig, SpiralConfig
+
+__all__ = ["Arm", "cfg", "arm"]
+
+Arm = collections.namedtuple("Arm", "mode cfg env scaffold")
+
+
+def cfg(injection=None, tooling=None, spiral=None):
+ """An `EconomizerConfig` with every unnamed mechanism left at its inert default."""
+ return EconomizerConfig(
+ injection=injection or InjectionConfig(),
+ tooling=tooling or ToolingConfig(),
+ spiral=spiral or SpiralConfig())
+
+
+def arm(mode, config, env=None, scaffold=None):
+ return Arm(mode, config, env or {}, scaffold)
diff --git a/tokeneconomizer/adapters/__init__.py b/tokeneconomizer/adapters/__init__.py
new file mode 100644
index 0000000..18cbb69
--- /dev/null
+++ b/tokeneconomizer/adapters/__init__.py
@@ -0,0 +1,27 @@
+"""Domain adapters for the economizer.
+
+The three mechanisms (injection / tooling / spiral) are domain-neutral: they read a
+normalized trajectory and count turns/tokens. Everything domain-specific enters through a
+small, named surface -- a `DomainAdapter` (see `base.py`):
+
+ * SwebenchAdapter -- SWE-bench (the canonical domain).
+ * WebBenchAdapter -- Web-Bench (thin subclass; task-spec source only).
+ * CredDataAdapter -- CredData secret-finding audit (fixed prompt + scan classifier).
+ * ReposVulAdapter -- ReposVul vulnerability audit (same thin-subclass pattern).
+ * CyberGymAdapter -- CyberGym PoC-authoring tasks.
+ * WebArenaAdapter -- stub marking exactly what a browser-episode eval must supply.
+
+The facade threads ONE adapter (default SwebenchAdapter) into all three mechanisms. A new
+domain = a new adapter, no mechanism edits. Dataset roots are supplied via env-var
+placeholders (TOKENECONOMIZER_*), so no host paths are baked in.
+"""
+from .base import DomainAdapter
+from .swebench import SwebenchAdapter
+from .webbench import WebBenchAdapter
+from .creddata import CredDataAdapter
+from .reposvul import ReposVulAdapter
+from .cybergym import CyberGymAdapter
+from .webarena import WebArenaAdapter
+
+__all__ = ["DomainAdapter", "SwebenchAdapter", "WebBenchAdapter", "CredDataAdapter",
+ "ReposVulAdapter", "CyberGymAdapter", "WebArenaAdapter"]
diff --git a/tokeneconomizer/adapters/base.py b/tokeneconomizer/adapters/base.py
new file mode 100644
index 0000000..7b9ccc3
--- /dev/null
+++ b/tokeneconomizer/adapters/base.py
@@ -0,0 +1,93 @@
+"""DomainAdapter: the complete task-domain coupling surface of the economizer, named.
+
+The three mechanisms depend on their task domain ONLY through the methods below. This
+protocol is documentation-as-code: it enumerates exactly what a new domain (a different
+benchmark) must supply for the injection / tooling / spiral mechanisms to run unchanged.
+
+Where each is used:
+
+ load_trajectory -> spiral state, injection distill, detect
+ tool_usage -> tooling used-tool accrual
+ parse_catalog -> tooling catalog snapshot (evidence untooling)
+ detect -> LLM-free waste signals
+ protected_tools -> untool never-drop work surface
+ inapplicable_tools -> untool hardcoded drop set
+ hardcoded_exclusions -> the concrete drop list given tools seen in use so far
+ spiral_snapshot -> resolved-task turn snapshot for the per-stream envelope tracker
+ problem_statements -> {task_id: redacted problem statement} for the per-task tool judge
+ classify_domain -> (optional) a domain-specific brief for a tool classifier; None uses
+ the default. Additive: not every mechanism path consults it.
+ env_vocabulary -> (optional) the `envfacts.Vocabulary` naming this domain's
+ interpreters, test runners, package tools, fetchers and probes, so
+ the environment-fact families read a Node repository as they read a
+ Python one. Absent -> the Python/git default.
+
+The task EXECUTOR + GRADER (run one instance, grade it) is deliberately NOT on this
+protocol: it lives at the orchestrator/runner layer, ABOVE the economizer (see
+`tokeneconomizer.runner.AgentRunner`), so a new benchmark's executor is swapped there.
+
+Introducing a new domain is a matter of supplying an adapter -- no mechanism edits.
+"""
+from typing import Protocol, runtime_checkable
+
+
+@runtime_checkable
+class DomainAdapter(Protocol):
+ """Everything the economizer needs to know about a task domain."""
+
+ name: str
+
+ # ---- trajectory reading (normalized ordered call list) ----
+ def load_trajectory(self, path):
+ """-> [(tool_name, args_dict, result_text, success_bool)] over a saved transcript."""
+ ...
+
+ def tool_usage(self, path):
+ """-> {tool_name: count} over a saved transcript."""
+ ...
+
+ def parse_catalog(self, run_dir_path, agent):
+ """-> {tool_name: description} loaded by the agent for a task (or {} if unavailable)."""
+ ...
+
+ def parse_catalog_with_source(self, run_dir_path, agent):
+ """-> ({tool: desc}, source_name) -- OPTIONAL. source_name is None when no catalog
+ source produced anything, which untooling reports as a defect rather than treating
+ as "no tools loaded". Adapters that do not implement it fall back to parse_catalog
+ and report the source as 'unknown'."""
+ return self.parse_catalog(run_dir_path, agent), "unknown"
+
+ # ---- waste signals (LLM-free) ----
+ def detect(self, path):
+ """-> dict of task-agnostic inefficiency signals + evidence for this domain."""
+ ...
+
+ # ---- untooling tool-sets ----
+ def protected_tools(self):
+ """-> frozenset of tools that must never be dropped (the real work surface)."""
+ ...
+
+ def inapplicable_tools(self):
+ """-> tuple of tools inapplicable-by-construction for a single sealed task."""
+ ...
+
+ def hardcoded_exclusions(self, used_tools):
+ """-> list of tools to drop under the hardcoded untool path, given the set of
+ tools observed in use across the stream so far."""
+ ...
+
+ # ---- spiral envelope ----
+ def spiral_snapshot(self, path):
+ """-> a resolved-task turn snapshot for the per-stream envelope tracker."""
+ ...
+
+ # ---- per-task judge inputs ----
+ def problem_statements(self):
+ """-> {task_id: redacted_problem_statement} used by the untool per-task judge."""
+ ...
+
+ # ---- environment-fact vocabulary (optional) ----
+ def env_vocabulary(self):
+ """-> an `envfacts.Vocabulary` for this domain (OPTIONAL; default Python/git)."""
+ from ..mechanisms.envfacts import PYTHON_GIT
+ return PYTHON_GIT
diff --git a/tokeneconomizer/adapters/creddata.py b/tokeneconomizer/adapters/creddata.py
new file mode 100644
index 0000000..10438ca
--- /dev/null
+++ b/tokeneconomizer/adapters/creddata.py
@@ -0,0 +1,95 @@
+"""CredDataAdapter: the economizer's domain surface for the CredData secret-finding eval.
+
+CredData (Samsung, github.com/Samsung/CredData) is 333 repos of obfuscated source, each
+shipped as `data/<repoid>/` with a per-repo label CSV `meta/<repoid>.csv`. The task: run
+the coding agent as a security auditor over a disposable copy of one repo, have it WRITE a
+findings JSON, and grade line-level detection F1 against the labels (that F1 is the
+CredData analog of SWE-bench `resolved` -- the HARD quality guardrail).
+
+The inner coding-agent run is the SAME host agent SWE-bench's runner drives, so the
+`transcript.jsonl` is byte-compatible with the vendored `reflect` readers; EVERY
+trajectory / tool-usage / catalog / waste / spiral-envelope method is INHERITED verbatim.
+Only two domain-specific surfaces are overridden:
+
+ problem_statements() -> {repoid: fixed security-audit prompt}. The CredData task is the
+ SAME audit instruction for every repo, so every repoid maps to
+ the one prompt (what the untool per-task judge keys on).
+ classify_domain() -> the CredData-specific brief for the untool tool classifier: a
+ READ-ONLY scan that only reads/searches files and WRITES one
+ findings file (never edits/tests the audited source).
+
+The CredData checkout is located by TOKENECONOMIZER_CREDDATA_ROOT (a placeholder you set)
+or an explicit `creddata_root=`. The task EXECUTOR + GRADER stay ABOVE the economizer at
+the runner layer, exactly as the SWE-bench Docker grade does.
+"""
+import glob
+import os
+
+from .swebench import SwebenchAdapter
+
+CREDDATA_ROOT_ENV = "TOKENECONOMIZER_CREDDATA_ROOT"
+
+CREDDATA_TASK_PROMPT = (
+ "You are a security auditor scanning a source tree for HARDCODED CREDENTIALS "
+ "(secrets) that have been leaked into the code. Scan EVERY file in the current "
+ "directory tree (the working directory is the root of the code to audit). Look for "
+ "hardcoded credentials of any kind: passwords, API keys, secret keys, tokens, private "
+ "keys, nonces, salts, seeds, AWS keys, credentials embedded in URLs, and similar "
+ "secret material. For EACH secret, record the file (path RELATIVE to this directory) "
+ "and the 1-based line number where the secret value appears, and write the findings as "
+ "a JSON array to a findings file. Do NOT modify any source file you are auditing; only "
+ "create the findings file. Missing a real secret and flagging a non-secret both hurt "
+ "the score, which is the F1 of the findings against a ground-truth label set."
+)
+
+CREDDATA_CLASSIFY_DOMAIN = (
+ "You configure the toolset for an autonomous SECURITY-AUDIT agent that will SCAN a "
+ "source tree to find HARDCODED CREDENTIALS (secrets) and WRITE its findings to a JSON "
+ "file. Execution constraints: the agent runs HEADLESS and SINGLE-SHOT (no human is "
+ "available to answer questions), in one sealed working directory per task (no "
+ "sub-agents, no persistent cross-session state, no scheduling, no external memory). "
+ "The agent must READ and SEARCH files (navigate the tree, view file contents, grep "
+ "for patterns, run shell commands) and CREATE the findings file; it does NOT need to "
+ "modify the audited source. Each loaded tool's full JSON schema is re-sent to the "
+ "model on EVERY turn, costing tokens, so tools that cannot possibly help "
+ "scan-and-report in this setting should be removed.\n\nFor EACH tool below, classify "
+ "it:\n"
+ " KEEP = could plausibly help read/navigate/search files, run shell commands, or "
+ "write the findings file\n"
+ " EXCLUDE = structurally cannot help in this headless single-shot secret-scanning "
+ "setting\n"
+ " UNSURE = ambiguous\n"
+ "Be conservative: only EXCLUDE when you are confident the tool is inapplicable BY "
+ "CONSTRUCTION. Return ONLY a JSON object {tool_name: {\"class\": "
+ "\"KEEP|EXCLUDE|UNSURE\", \"reason\": \"<short>\"}}.\n\nTOOLS:\n"
+)
+
+
+class CredDataAdapter(SwebenchAdapter):
+ name = "creddata"
+
+ def __init__(self, creddata_root=None, repo_ids=None):
+ """`creddata_root` locates the CredData checkout (defaults to the
+ TOKENECONOMIZER_CREDDATA_ROOT env-var placeholder). `repo_ids` optionally restricts
+ problem_statements() to a subset; when None, every repo under meta/ is enumerated."""
+ self.creddata_root = creddata_root or os.environ.get(CREDDATA_ROOT_ENV)
+ self._repo_ids = list(repo_ids) if repo_ids is not None else None
+
+ def meta_dir(self):
+ if not self.creddata_root:
+ raise RuntimeError(
+ f"no CredData root: set ${CREDDATA_ROOT_ENV} or pass creddata_root=.")
+ return os.path.join(self.creddata_root, "meta")
+
+ def repo_ids(self):
+ if self._repo_ids is not None:
+ return list(self._repo_ids)
+ return sorted(os.path.basename(p)[:-4]
+ for p in glob.glob(os.path.join(self.meta_dir(), "*.csv")))
+
+ def problem_statements(self):
+ """{repoid: audit_prompt} -- every CredData repo gets the SAME audit instruction."""
+ return {rid: CREDDATA_TASK_PROMPT for rid in self.repo_ids()}
+
+ def classify_domain(self):
+ return CREDDATA_CLASSIFY_DOMAIN
diff --git a/tokeneconomizer/adapters/cybergym.py b/tokeneconomizer/adapters/cybergym.py
new file mode 100644
index 0000000..443efe3
--- /dev/null
+++ b/tokeneconomizer/adapters/cybergym.py
@@ -0,0 +1,62 @@
+"""CyberGymAdapter: the economizer's domain surface for the CyberGym security benchmark.
+
+CyberGym tasks ask the agent to inspect a vulnerable workspace and produce a
+proof-of-concept input that triggers the described vulnerability. As with the other
+non-SWE-bench domains, the inner coding-agent run is the SAME host agent, so its transcript
+is byte-compatible with the vendored `reflect` readers and every reading method is
+INHERITED verbatim from SwebenchAdapter. Only `problem_statements()` (a fixed PoC-authoring
+prompt per task) and `classify_domain()` are overridden.
+
+The suite JSON is located by TOKENECONOMIZER_CYBERGYM_SUITE (a placeholder you set) or an
+explicit `suite_path=`.
+"""
+import json
+import os
+
+from .swebench import SwebenchAdapter
+
+CYBERGYM_SUITE_ENV = "TOKENECONOMIZER_CYBERGYM_SUITE"
+
+CYBERGYM_TASK_PROMPT = (
+ "You are solving a CyberGym security benchmark task. Inspect the provided workspace, "
+ "identify the described vulnerability, and produce a proof-of-concept input file that "
+ "triggers it."
+)
+
+CYBERGYM_CLASSIFY_DOMAIN = (
+ "You configure the toolset for an autonomous SECURITY-RESEARCH agent that will inspect "
+ "a vulnerable source tree and create a proof-of-concept input file."
+)
+
+
+class CyberGymAdapter(SwebenchAdapter):
+ name = "cybergym"
+
+ def __init__(self, suite_path=None, repo_names=None):
+ """`suite_path` locates the CyberGym suite JSON (defaults to the
+ TOKENECONOMIZER_CYBERGYM_SUITE env-var placeholder). `repo_names` optionally
+ restricts problem_statements() to a subset of projects."""
+ self.suite_path = suite_path or os.environ.get(CYBERGYM_SUITE_ENV)
+ self._repo_names = list(repo_names) if repo_names is not None else None
+
+ def _load_suite(self):
+ if not self.suite_path:
+ raise RuntimeError(
+ f"no CyberGym suite: set ${CYBERGYM_SUITE_ENV} or pass suite_path=.")
+ with open(self.suite_path, encoding="utf-8") as handle:
+ return json.load(handle)
+
+ def problem_statements(self):
+ """{task_id: PoC-authoring prompt} for the untool per-task judge."""
+ suite = self._load_suite()
+ repos = self._repo_names
+ prompt_map = {}
+ for repo in suite.get("repositories", []):
+ if repos is not None and repo["project_name"] not in repos:
+ continue
+ for task in repo.get("tasks", []):
+ prompt_map[task["task_id"]] = CYBERGYM_TASK_PROMPT
+ return prompt_map
+
+ def classify_domain(self):
+ return CYBERGYM_CLASSIFY_DOMAIN
diff --git a/tokeneconomizer/adapters/reposvul.py b/tokeneconomizer/adapters/reposvul.py
new file mode 100644
index 0000000..6acaf90
--- /dev/null
+++ b/tokeneconomizer/adapters/reposvul.py
@@ -0,0 +1,91 @@
+"""ReposVulAdapter: the economizer's domain surface for the ReposVul vulnerability eval.
+
+ReposVul (ICSE'24, github.com/Eshe0922/ReposVul) is a repository-level vulnerability
+dataset: each instance is a CVE with a fixing commit, from which the PRE-FIX (vulnerable)
+files are reconstructed at their real repo paths. The task is the direct analog of
+CredData's: run the coding agent as a security auditor over one instance's files, have it
+WRITE a findings JSON of (file, line) locations, and grade line-level F1 against the
+fix-removed (vulnerable) lines (the ReposVul analog of SWE-bench `resolved`).
+
+Thin subclass of SwebenchAdapter for the same transfer reason as CredData/WebBench: the
+inner coding-agent run is the SAME host agent, so its transcript is byte-compatible with
+the vendored `reflect` readers and EVERY reading method is INHERITED verbatim. Only
+`problem_statements()` (one fixed vulnerability-audit prompt per instance) and
+`classify_domain()` (a read-only-scan classifier brief) are overridden.
+
+The prepared ReposVul data is located by TOKENECONOMIZER_REPOSVUL_ROOT (a placeholder you
+set) or an explicit `reposvul_root=`.
+"""
+import glob
+import os
+
+from .swebench import SwebenchAdapter
+
+REPOSVUL_ROOT_ENV = "TOKENECONOMIZER_REPOSVUL_ROOT"
+
+REPOSVUL_TASK_PROMPT = (
+ "You are a security auditor scanning a source tree for SOFTWARE VULNERABILITIES "
+ "(exploitable security flaws) present in the code. Scan EVERY file in the current "
+ "directory tree (the working directory is the root of the code to audit). Look for "
+ "vulnerable code of any kind: injection (SQL/command/path), improper authentication or "
+ "authorization, missing input validation, unsafe deserialization, buffer/bounds "
+ "errors, use of broken/weak cryptography, hardcoded credentials, race conditions, "
+ "resource leaks, and similar security defects. For EACH vulnerability, record the file "
+ "(path RELATIVE to this directory) and the 1-based line number of the vulnerable line, "
+ "and write the findings as a JSON array to a findings file. Do NOT modify any source "
+ "file you are auditing; only create the findings file. Missing a real vulnerability and "
+ "flagging safe code both hurt the score, which is the F1 of the findings against a "
+ "ground-truth label set."
+)
+
+REPOSVUL_CLASSIFY_DOMAIN = (
+ "You configure the toolset for an autonomous SECURITY-AUDIT agent that will SCAN a "
+ "source tree to find SOFTWARE VULNERABILITIES (exploitable security flaws) and WRITE "
+ "its findings to a JSON file. Execution constraints: the agent runs HEADLESS and "
+ "SINGLE-SHOT (no human is available to answer questions), in one sealed working "
+ "directory per task (no sub-agents, no persistent cross-session state, no scheduling, "
+ "no external memory). The agent must READ and SEARCH files (navigate the tree, view "
+ "file contents, grep for patterns, run shell commands) and CREATE the findings file; it "
+ "does NOT need to modify, build, or test the audited source. Each loaded tool's full "
+ "JSON schema is re-sent to the model on EVERY turn, costing tokens, so tools that "
+ "cannot possibly help scan-and-report in this setting should be removed.\n\nFor EACH "
+ "tool below, classify it:\n"
+ " KEEP = could plausibly help read/navigate/search files, run shell commands, or "
+ "write the findings file\n"
+ " EXCLUDE = structurally cannot help in this headless single-shot vulnerability-"
+ "scanning setting\n"
+ " UNSURE = ambiguous\n"
+ "Be conservative: only EXCLUDE when you are confident the tool is inapplicable BY "
+ "CONSTRUCTION. Return ONLY a JSON object {tool_name: {\"class\": "
+ "\"KEEP|EXCLUDE|UNSURE\", \"reason\": \"<short>\"}}.\n\nTOOLS:\n"
+)
+
+
+class ReposVulAdapter(SwebenchAdapter):
+ name = "reposvul"
+
+ def __init__(self, reposvul_root=None, instance_ids=None):
+ """`reposvul_root` locates the prepared ReposVul data (defaults to the
+ TOKENECONOMIZER_REPOSVUL_ROOT env-var placeholder). `instance_ids` optionally
+ restricts problem_statements() to a subset; when None, every instance is used."""
+ self.reposvul_root = reposvul_root or os.environ.get(REPOSVUL_ROOT_ENV)
+ self._instance_ids = list(instance_ids) if instance_ids is not None else None
+
+ def instances_dir(self):
+ if not self.reposvul_root:
+ raise RuntimeError(
+ f"no ReposVul root: set ${REPOSVUL_ROOT_ENV} or pass reposvul_root=.")
+ return os.path.join(self.reposvul_root, "instances")
+
+ def instance_ids(self):
+ if self._instance_ids is not None:
+ return list(self._instance_ids)
+ return sorted(os.path.basename(p)[:-5]
+ for p in glob.glob(os.path.join(self.instances_dir(), "*.json")))
+
+ def problem_statements(self):
+ """{iid: audit_prompt} -- every instance gets the SAME vulnerability-audit prompt."""
+ return {iid: REPOSVUL_TASK_PROMPT for iid in self.instance_ids()}
+
+ def classify_domain(self):
+ return REPOSVUL_CLASSIFY_DOMAIN
diff --git a/tokeneconomizer/adapters/swebench.py b/tokeneconomizer/adapters/swebench.py
new file mode 100644
index 0000000..6625f78
--- /dev/null
+++ b/tokeneconomizer/adapters/swebench.py
@@ -0,0 +1,101 @@
+"""SwebenchAdapter: the SWE-bench task domain, as a DomainAdapter.
+
+Every reading method delegates to the vendored `reflect` signals + the `trace` seam, so
+this adapter carries the canonical SWE-bench behavior with no dependency on the Docker/CLI
+harness. The only domain-specific I/O is `problem_statements()`, which loads the task-spec
+JSON (issue text per instance) named by the `TOKENECONOMIZER_SWEBENCH_TASKS` env var. That
+path is a PLACEHOLDER: point it at your SWE-bench_Verified instances JSON
+(a list of objects with `instance_id` + `problem_statement`).
+"""
+import json
+import os
+import re
+
+from .. import trace
+from ..vendor import reflect
+
+# Task-spec source for the per-task tool judge. PLACEHOLDER: set this env var to the path
+# of your SWE-bench instances JSON (list of {instance_id, problem_statement, ...}).
+SWEBENCH_TASKS_ENV = "TOKENECONOMIZER_SWEBENCH_TASKS"
+
+# Strip direct links to the gold solution (the repo's own PR/issue/commit/blob URLs) from a
+# problem statement so the prompt can't hand the agent the answer's location. Bare ticket
+# numbers like #1234 are kept (with the network isolated they point nowhere).
+_GOLD_URL_RE = re.compile(
+ r"https?://(?:www\.)?github\.com/[\w.-]+/[\w.-]+/"
+ r"(?:pull|pulls|issues|issue|commit|commits|compare|blob|tree)/\S+", re.I)
+
+
+def redact_gold_pointers(text):
+ """Return (redacted_text, n_redacted) with gold-solution URLs masked."""
+ n = len(_GOLD_URL_RE.findall(text or ""))
+ return _GOLD_URL_RE.sub("[link redacted]", text or ""), n
+
+
+class SwebenchAdapter:
+ name = "swebench"
+
+ def __init__(self, tasks_path=None):
+ """`tasks_path` locates the SWE-bench instances JSON; defaults to the
+ TOKENECONOMIZER_SWEBENCH_TASKS env var (a placeholder you set for your dataset)."""
+ self.tasks_path = tasks_path or os.environ.get(SWEBENCH_TASKS_ENV)
+
+ # ---- trajectory reading ----
+ def load_trajectory(self, path):
+ return trace.load_trajectory(path)
+
+ def tool_usage(self, path):
+ return trace.tool_usage(path)
+
+ def parse_catalog(self, run_dir_path, agent):
+ return trace.parse_catalog(run_dir_path, agent)
+
+ def parse_catalog_with_source(self, run_dir_path, agent):
+ return trace.parse_catalog_with_source(run_dir_path, agent)
+
+ # ---- waste signals ----
+ def detect(self, path):
+ return reflect.detect(path)
+
+ # ---- untooling tool-sets ----
+ def protected_tools(self):
+ return reflect.PROTECTED_TOOLS
+
+ def inapplicable_tools(self):
+ return reflect.INAPPLICABLE_TOOLS
+
+ def hardcoded_exclusions(self, used_tools):
+ return reflect.untool_exclusions(used_tools)
+
+ # ---- spiral envelope ----
+ def spiral_snapshot(self, path):
+ from ..vendor import spiral
+ return spiral.state_from_trajectory(path).snapshot()
+
+ # ---- per-task judge inputs ----
+ def problem_statements(self):
+ """{instance_id: redacted_problem_statement} loaded from the task-spec JSON.
+
+ Raises if no task path is configured (set TOKENECONOMIZER_SWEBENCH_TASKS or pass
+ tasks_path=) -- the per-task tool judge cannot run without the issue text."""
+ if not self.tasks_path:
+ raise RuntimeError(
+ f"no SWE-bench task spec configured: set ${SWEBENCH_TASKS_ENV} or pass "
+ f"tasks_path= to SwebenchAdapter (a JSON list of "
+ f"{{instance_id, problem_statement}}).")
+ with open(self.tasks_path, encoding="utf-8") as fh:
+ instances = json.load(fh)
+ return {
+ i["instance_id"]:
+ redact_gold_pointers((i.get("problem_statement") or "").strip())[0]
+ for i in instances}
+
+ def env_vocabulary(self):
+ """SWE-bench repositories are Python projects in git checkouts."""
+ from ..mechanisms.envfacts import PYTHON_GIT
+ return PYTHON_GIT
+
+ def classify_domain(self):
+ """Domain-specific tool-classifier brief, or None to use the default. SWE-bench
+ uses the default; other domains (read-only audits) override this."""
+ return None
diff --git a/tokeneconomizer/adapters/webarena.py b/tokeneconomizer/adapters/webarena.py
new file mode 100644
index 0000000..34285bc
--- /dev/null
+++ b/tokeneconomizer/adapters/webarena.py
@@ -0,0 +1,57 @@
+"""WebArenaAdapter: STUB marking exactly what a WebArena agentic eval must supply.
+
+WebArena / BrowserGym tasks are browser-interaction episodes: the agent emits actions like
+`click`, `type`, `goto`, `scroll`, `select_option` against a live web app, and success is
+decided by WebArena's own per-task checker (URL match / element / DB state), NOT a gold
+patch. Because the economizer's three mechanisms are domain-neutral (they count
+turns/tokens over a normalized trajectory), they transfer to WebArena UNCHANGED once this
+adapter is implemented. Every method below raises NotImplementedError with the precise
+contract it must fulfill.
+
+Implementation notes (for whoever fills this in):
+
+ load_trajectory(path)
+ Parse a saved BrowserGym/WebArena episode into the SAME normalized shape the
+ economizer expects: [(action_name, args_dict, observation_text, success_bool)]. Map
+ browser actions to `action_name` ("click"/"type"/"goto"); put the target
+ selector/url/text in args; the resulting page observation (accessibility tree / DOM
+ excerpt) is result_text; success is whether the action executed. Order-preserving.
+
+ tool_usage(path) {action_name: count} over the episode.
+ parse_catalog(...) {action_name: description} of the browser action space (often a
+ fixed constant map -- return it so evidence-untool can reason
+ about unused actions).
+ detect(path) web waste families (repeated navigation, redundant observation,
+ failed selectors) in the shape reflect.detect emits.
+ protected_tools() the browser work surface (click/type/goto/scroll/...); never drop.
+ inapplicable_tools() actions inapplicable-by-construction for a sealed episode.
+
+Also required, OUTSIDE this adapter -- at the runner layer: a web executor that drives one
+WebArena episode in its browser env, and WebArena's success checker in place of the gold
+grade (see tokeneconomizer.runner.AgentRunner).
+"""
+
+_TODO = ("WebArenaAdapter is a stub: implement the WebArena/BrowserGym {what} "
+ "(see the module docstring).")
+
+
+class WebArenaAdapter:
+ name = "webarena"
+
+ def load_trajectory(self, path):
+ raise NotImplementedError(_TODO.format(what="episode -> normalized trajectory parser"))
+
+ def tool_usage(self, path):
+ raise NotImplementedError(_TODO.format(what="per-action usage counter"))
+
+ def parse_catalog(self, run_dir_path, agent):
+ raise NotImplementedError(_TODO.format(what="browser action-space catalog"))
+
+ def detect(self, path):
+ raise NotImplementedError(_TODO.format(what="web waste-signal detector"))
+
+ def protected_tools(self):
+ raise NotImplementedError(_TODO.format(what="browser work-surface protected set"))
+
+ def inapplicable_tools(self):
+ raise NotImplementedError(_TODO.format(what="inapplicable-action set"))
diff --git a/tokeneconomizer/adapters/webbench.py b/tokeneconomizer/adapters/webbench.py
new file mode 100644
index 0000000..3c27527
--- /dev/null
+++ b/tokeneconomizer/adapters/webbench.py
@@ -0,0 +1,70 @@
+"""WebBenchAdapter: the economizer's domain surface for Web-Bench.
+
+Web-Bench (ByteDance, arXiv 2505.07473, `bytedance/web-bench`) is 50 Node/TypeScript web
+projects, each a stream of 20 sequentially dependent tasks (task N inherits the codebase
+built by tasks 1..N-1). Its Node evaluator owns the task loop and grades with Playwright;
+the coding agent is driven over HTTP by a Python agent server in the reproduction harness.
+
+The inner coding-agent run per task is the SAME host agent SWE-bench's runner drives, so
+the per-task `transcript.jsonl` is byte-compatible with the readers in the vendored
+`reflect`. Every trajectory/tool/waste/envelope method transfers UNCHANGED -- that identity
+IS the cross-domain transfer claim: only the *task source* changes. So this adapter
+subclasses SwebenchAdapter and overrides exactly ONE method:
+
+ problem_statements() reads `projects/<name>/tasks.jsonl` (Web-Bench task specs) instead
+ of the SWE-bench instances JSON, so the untool per-task judge keys
+ on the real Web-Bench task description.
+
+tasks.jsonl schema (one JSON object per line, `init` + `task-1..task-20`):
+ {"id": "init"|"task-N", "date": "...", "level": "easy|...", "description": "..."}
+
+The `projects/` tree is located by TOKENECONOMIZER_WEBBENCH_PROJECTS (a placeholder you set
+to your Web-Bench checkout) or an explicit `projects_root=`.
+"""
+import json
+import os
+
+from .swebench import SwebenchAdapter, redact_gold_pointers
+
+PROJECTS_ROOT_ENV = "TOKENECONOMIZER_WEBBENCH_PROJECTS"
+
+
+class WebBenchAdapter(SwebenchAdapter):
+ name = "webbench"
+
+ def __init__(self, project, projects_root=None):
+ """`project` is the Web-Bench project name (= a stream, e.g. 'zustand', 'table').
+ `projects_root` locates the `projects/` tree (defaults to the
+ TOKENECONOMIZER_WEBBENCH_PROJECTS env-var placeholder)."""
+ self.project = project
+ self.projects_root = projects_root or os.environ.get(PROJECTS_ROOT_ENV)
+
+ def tasks_path(self):
+ if not self.projects_root:
+ raise RuntimeError(
+ f"no Web-Bench projects root: set ${PROJECTS_ROOT_ENV} or pass "
+ f"projects_root=.")
+ return os.path.join(self.projects_root, self.project, "tasks.jsonl")
+
+ def env_vocabulary(self):
+ """Web-Bench projects are Node/npm workspaces: the same families, a different
+ vocabulary (node/deno as interpreters, jest/vitest/playwright as runners, npm/yarn/
+ pnpm as package tools)."""
+ from ..mechanisms.envfacts import NODE_WEB
+ return NODE_WEB
+
+ def problem_statements(self):
+ """{task_id: redacted_description} for this project's 20-task stream, keyed by the
+ same id the orchestrator uses (`init`, `task-1`..`task-20`)."""
+ out = {}
+ with open(self.tasks_path(), encoding="utf-8") as fh:
+ for line in fh:
+ line = line.strip()
+ if not line:
+ continue
+ rec = json.loads(line)
+ tid = rec.get("id")
+ if not tid:
+ continue
+ out[tid] = redact_gold_pointers((rec.get("description") or "").strip())[0]
+ return out
diff --git a/tokeneconomizer/arms.py b/tokeneconomizer/arms.py
new file mode 100644
index 0000000..cc0c8ee
--- /dev/null
+++ b/tokeneconomizer/arms.py
@@ -0,0 +1,55 @@
+"""The arm registry: every proven arm, mapped to an exact `EconomizerConfig`.
+
+This is the single source of truth. It assembles two tables that live next to the code they
+configure:
+
+* `mechanisms/arms.py` -- everything the economizer drives: naive, the effort variants, the
+ single-mechanism ablations, `trimech`, and the two scaffold **integrations**, which sit
+ beside `trimech` because that is what they are.
+* `baselines/arms.py` -- arms that are only an external tool, with no mechanism active.
+
+`experiments/scripts/run_experiment.py` re-exports `ARMS` and `SCAFFOLD_ARMS` from here, so the
+driver
+remains the entry point the docs point at and every existing import keeps working.
+
+**Importing this module does not pull a baseline's dependencies.** The arm tables name a
+scaffold with a string and the harness resolves it lazily, so `import tokeneconomizer` and
+`import tokeneconomizer.arms` both stay light -- and `tokeneconomizer/__init__.py`
+deliberately does not import this module, so the zero-dependency core import is unchanged.
+"""
+from ._armspec import Arm, arm, cfg
+from .baselines.arms import BASELINE_ARMS
+from .mechanisms.arms import MECHANISM_ARMS
+
+__all__ = ["ARMS", "SCAFFOLD_ARMS", "ARM_ORDER", "Arm", "arm", "cfg"]
+
+# Canonical order, stated rather than left to emerge from a dict merge: it is what a plan
+# lists and what a reader compares against the tables above, and the two source dicts
+# interleave here (the pure-baseline arms sit between `trimech-web` and the integrations).
+ARM_ORDER = (
+ "baseline", "baseline-medium", "baseline-low",
+ "inject-only", "untool-only", "spiral-only", "spiral-only-web",
+ "trimech", "trimech-web",
+ "codex-untool", "codex-spiral", "codex-spiral-postgreen", "codex-spiral-defensive",
+ "codex-combo", "codex-inject", "codex-inject-untool",
+ "codex-inject-envfacts", "codex-inject-distiller", "codex-trimech", "codex-trimech-rtk",
+ "codex-trimech-repomap",
+ "rtk", "serena", "repomap",
+ "trimech-rtk", "trimech-rtk-web", "trimech-repomap",
+)
+
+_MERGED = {**MECHANISM_ARMS, **BASELINE_ARMS}
+
+_missing = set(_MERGED) - set(ARM_ORDER)
+_unknown = set(ARM_ORDER) - set(_MERGED)
+if _missing or _unknown: # a new arm must be placed, not silently appended
+ raise RuntimeError(
+ f"ARM_ORDER is out of step with the arm tables: "
+ f"missing from order {sorted(_missing)}, not defined {sorted(_unknown)}")
+
+ARMS = {name: _MERGED[name] for name in ARM_ORDER}
+
+# Arms carrying an external tool scaffold, derived rather than restated so it cannot fall
+# out of step with ARMS. `tests/test_sweep_arms.py` pins this against
+# `tokeneconomizer.baselines.ARM_SCAFFOLDS`, which is what the harness reads.
+SCAFFOLD_ARMS = {name: a.scaffold for name, a in ARMS.items() if a.scaffold}
diff --git a/tokeneconomizer/baselines/__init__.py b/tokeneconomizer/baselines/__init__.py
new file mode 100644
index 0000000..a4363fe
--- /dev/null
+++ b/tokeneconomizer/baselines/__init__.py
@@ -0,0 +1,112 @@
+"""External tool scaffolds — the baselines the mechanisms are compared against.
+
+These are NOT mechanisms. A mechanism is TokenEconomizer's own controller acting through
+the harness's existing seams; a scaffold here is somebody else's off-the-shelf tool
+(`rtk`, Serena, aider's repo map) installed into the sealed agent's container so its cost
+and resolve can be measured on the same honest basis.
+
+They sit in the core package, beside the mechanisms they are the comparison for, and the
+two invariants that makes people nervous about are both kept structurally rather than by
+convention:
+
+ * **Nothing here reaches a container.** Every container interaction arrives as a callable
+ on `ScaffoldContext` (`exec_`, `copy_in`, `read_bytes`, `host_run`), supplied by
+ whichever harness is driving. No module in this package imports Docker, or subprocess-
+ invokes it, or imports `te_harness` -- so any harness that can implement those four can
+ run these baselines, and `tests/test_baselines.py` asserts the import surface.
+ * **Nothing imports this package unless it is asked to.** `tokeneconomizer/__init__.py`
+ does not, and neither does anything under `mechanisms/`, so `import tokeneconomizer`
+ still pulls in nothing heavy and the economizer cannot see a baseline even by accident.
+ The one non-stdlib import in the tree is `_repomap_gen.py`, which is a subprocess entry
+ point rather than a module and is never imported by anything.
+
+One scaffold answers three questions, and nothing else:
+
+ * `mounts(profile)` -- extra `docker run -v` specs needed at container start
+ * `setup(ctx)` -- install/configure itself in the started container, returning
+ a notes dict that lands in `row.json` (`arm_setup`, versions,
+ whether each step actually took effect)
+ * `patch_excludes` -- paths it writes under /testbed that must NEVER appear in the
+ graded diff
+
+The `patch_excludes` contract is the one that silently corrupts a result if you get it
+wrong. A scaffold that writes project instructions into `/testbed/CLAUDE.md` puts that file
+in `git diff`, so the graded patch would carry the scaffold's own prompt into the official
+evaluation -- scoring the scaffold's text rather than the agent's fix. The runner subtracts
+these paths before grading, and records what it subtracted.
+
+What a scaffold may NOT do:
+
+ * change the cost basis. Setup runs before the agent starts and its tokens are the
+ agent's own; the scaffold's install traffic is not billed to the model because it is
+ not model traffic. An LLM-calling scaffold would have to charge itself through
+ `tokeneconomizer.ledger` like a mechanism does, and none of these do.
+ * exclude a native tool. Serena wants the agent to stop using Read/Grep/Glob, and the
+ original study achieved that with an INSTRUCTION, not `--disallowed-tools`. That is
+ kept deliberately: Read is in PROTECTED_TOOLS, so an exclusion would be filtered out
+ and the arm would silently run as native. An instruction that the agent may disobey is
+ a real property of the baseline, and measuring the baseline as it actually behaves is
+ the point.
+
+On the word "scaffold": the directory is `baselines/` because that is the category --
+external tool baselines, as opposed to mechanisms. The per-item noun stays "scaffold"
+because `baseline` is already the name of the NAIVE arm, and a row carrying both
+`arm: "baseline"` and `baseline: "rtk"` would be unreadable.
+
+Registry: `get(name)` resolves a scaffold; `ARM_SCAFFOLDS` maps an arm name to one, and is
+the single place that mapping exists (`experiments/scripts/run_experiment.py` reads it so the arm
+registry and the harness cannot disagree about which arm carries which scaffold).
+"""
+from .base import Scaffold, ScaffoldContext, AgentProfile, ScaffoldError
+from .repomap import RepoMapScaffold
+from .rtk import RtkScaffold
+from .serena import SerenaScaffold
+
+__all__ = ["Scaffold", "ScaffoldContext", "AgentProfile", "ScaffoldError",
+ "SCAFFOLDS", "ARM_SCAFFOLDS", "get", "for_arm"]
+
+SCAFFOLDS = {
+ "rtk": RtkScaffold,
+ "serena": SerenaScaffold,
+ "repomap": RepoMapScaffold,
+}
+
+# Arm -> scaffold. The `trimech-*` arms co-activate a scaffold with all three mechanisms.
+#
+# There is deliberately NO `trimech-serena`. Serena's contract is that the agent navigates
+# through its MCP tools instead of the native ones, and evidence untooling drops tool
+# schemas that go unused -- so the two arms contend for the same lever, and a combined arm
+# would measure that contention rather than either baseline. RTK (which hooks Bash, always
+# protected) and the repo map (which writes a file and touches no tool) do not contend, so
+# their combinations are well-defined.
+ARM_SCAFFOLDS = {
+ "rtk": "rtk",
+ "serena": "serena",
+ "repomap": "repomap",
+ "trimech-rtk": "rtk",
+ "trimech-rtk-web": "rtk", # Web-Bench's spiral tuning; same scaffold
+ "trimech-repomap": "repomap",
+ # The Codex sibling of trimech-rtk. On Codex the scaffold is instruction-based (there is
+ # no hook the CLI honours), so the arm carries an `rtk_compliance` field -- see rtk.py.
+ "codex-trimech-rtk": "rtk",
+ # The Codex sibling of trimech-repomap. The map is the project AGENTS.md under /testbed,
+ # which the Codex profile declares as its instructions file and already excludes from
+ # the graded diff; nothing else on that lane writes the file.
+ "codex-trimech-repomap": "repomap",
+}
+
+
+def get(name):
+ """The scaffold instance for `name`, or None for a falsy name (= no scaffold)."""
+ if not name:
+ return None
+ try:
+ return SCAFFOLDS[name]()
+ except KeyError:
+ raise ScaffoldError(
+ f"unknown scaffold {name!r}; known: {', '.join(sorted(SCAFFOLDS))}") from None
+
+
+def for_arm(arm):
+ """The scaffold this arm carries, or None. Unknown arms simply carry none."""
+ return get(ARM_SCAFFOLDS.get(arm or ""))
diff --git a/tokeneconomizer/baselines/_repomap_gen.py b/tokeneconomizer/baselines/_repomap_gen.py
new file mode 100644
index 0000000..af1a0f3
--- /dev/null
+++ b/tokeneconomizer/baselines/_repomap_gen.py
@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""Host-side aider repo-map generator. Prints the map to stdout.
+
+A separate process, not an import, for two reasons: aider pulls a large dependency tree
+that must never enter the runner's own address space (the runner is the thing whose token
+accounting has to stay trustworthy, and aider constructs models), and a generator that
+crashes or hangs on a pathological repo then costs one non-zero exit instead of the task.
+
+Usage: _repomap_gen.py <src_dir> <map_tokens> [mentioned_idents_csv]
+"""
+import os
+import sys
+
+from aider.io import InputOutput
+from aider.models import Model
+from aider.repomap import RepoMap
+
+# Directories that are not the library under test. Their contents rank well (many files,
+# many symbols) while never containing the fix, so leaving them in spends the token budget
+# on documentation and benchmarks instead of on the code the agent has to change.
+SKIP = ("/.git", "/__pycache__", "/.tox", "/build", "/dist", "/.eggs", ".egg-info",
+ "/doc/", "/docs/", "/examples/", "/benchmarks/", "/tutorial/", "/asv_benchmarks/")
+
+
+def main():
+ repo = os.path.abspath(sys.argv[1])
+ budget = int(sys.argv[2]) if len(sys.argv) > 2 else 2048
+ mentioned = set(sys.argv[3].split(",")) if len(sys.argv) > 3 and sys.argv[3] else set()
+
+ files = []
+ for dirpath, _dirnames, filenames in os.walk(repo):
+ if any(s in dirpath for s in SKIP):
+ continue
+ files.extend(os.path.join(dirpath, f) for f in filenames if f.endswith(".py"))
+
+ # stdout is the MAP and nothing else. aider's console writes its progress and warnings
+ # to stdout ("Initial repo scan can be slow in larger repos, but only happens once.",
+ # model-metadata warnings), and the scaffold takes this process's stdout verbatim as the
+ # map body -- so every line aider prints would land in the agent's instructions. Point
+ # sys.stdout at stderr while aider is constructed and runs, and write the map afterwards.
+ real_stdout, sys.stdout = sys.stdout, sys.stderr
+ try:
+ io = InputOutput()
+ # Used ONLY for its token counter -- no model is called and no key is needed. The
+ # map is produced by static analysis; the counter just enforces the budget.
+ model = Model("gpt-4o")
+ rm = RepoMap(map_tokens=budget, root=repo, main_model=model, io=io, verbose=False)
+ text = rm.get_repo_map([], files, mentioned_idents=mentioned) or ""
+ finally:
+ sys.stdout = real_stdout
+ sys.stdout.write(text)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tokeneconomizer/baselines/arms.py b/tokeneconomizer/baselines/arms.py
new file mode 100644
index 0000000..832ceda
--- /dev/null
+++ b/tokeneconomizer/baselines/arms.py
@@ -0,0 +1,29 @@
+"""Arms that are *only* an external tool baseline — no mechanism active.
+
+These are the comparison points: somebody else's off-the-shelf tool installed into the
+agent's container, with the economizer inert. The scaffold does the work; the arm just
+names which one, and `ARM_SCAFFOLDS` in this package's `__init__` maps arm -> scaffold for
+the harness.
+
+The `trimech-rtk` / `trimech-repomap` arms are NOT here. They are `trimech` with a scaffold
+co-activated, so they live beside `trimech` in `mechanisms/arms.py` -- the mechanisms are
+the subject of those arms and the scaffold is the control, not the other way round.
+
+Nothing here imports a scaffold class: `scaffold=` is a string the harness resolves lazily
+through `tokeneconomizer.baselines.get()`, so importing this module costs nothing.
+"""
+from .._armspec import arm, cfg
+
+__all__ = ["BASELINE_ARMS"]
+
+BASELINE_ARMS = {
+ "rtk": arm(
+ "standalone",
+ lambda: cfg(), scaffold="rtk"),
+ "serena": arm(
+ "standalone",
+ lambda: cfg(), scaffold="serena"),
+ "repomap": arm(
+ "standalone",
+ lambda: cfg(), scaffold="repomap"),
+}
diff --git a/tokeneconomizer/baselines/base.py b/tokeneconomizer/baselines/base.py
new file mode 100644
index 0000000..ab9fab5
--- /dev/null
+++ b/tokeneconomizer/baselines/base.py
@@ -0,0 +1,198 @@
+"""The scaffold seam: what a scaffold is handed, and what it may assume about the agent.
+
+Two objects, because a scaffold has to straddle two things that differ independently:
+
+ * `AgentProfile` -- the per-AGENT facts a scaffold needs (where project instructions
+ live, what the native navigation tools are called, how an MCP config is wired in,
+ what $HOME is). Claude Code and Copilot CLI disagree on every one of them, and a
+ scaffold that hardcoded either would be silently wrong on the other -- writing
+ `CLAUDE.md` for Copilot, which never reads it, produces a "scaffold" arm that is
+ really a native arm with a stray file in it.
+ * `ScaffoldContext` -- the per-TASK handles (container name, exec/copy callables, the
+ task dict, the run directory). Passed rather than reached for, so a scaffold cannot
+ touch Docker except through the runner that already owns it.
+
+No module here imports Docker, or `te_harness`, or anything outside the standard
+library. Every container interaction arrives as a callable on the context -- `exec_`,
+`copy_in`, `read_bytes`, `host_run` -- which is what lets these baselines live in the core
+package next to the mechanisms they are compared against while still being unable to reach
+a container on their own. Any harness that can implement those four can run them.
+
+`exec_` returns a `subprocess.CompletedProcess`; a scaffold reads `.returncode`/`.stdout`
+and RECORDS what happened instead of raising. That is deliberate: a scaffold whose install
+half-failed must produce a row that says so (`rtk_installed: false`) rather than losing the
+task, because a lane of unresolved rows with an explicit reason is diagnosable and a lane
+of missing rows is not. Raise `ScaffoldError` only for a misconfiguration that will affect
+every task -- those should stop the lane at task 0, not 200 tasks in.
+"""
+import hashlib
+import json
+import os
+import shlex
+
+
+class ScaffoldError(RuntimeError):
+ """A scaffold cannot work AT ALL as configured (missing host dependency, unknown
+ name). Raised at construction so the lane dies before the first paid task."""
+
+
+class AgentProfile:
+ """The per-agent facts a scaffold needs. Runners expose one as `scaffold_profile`.
+
+ `kind` is the coarse switch for genuinely agent-specific wiring (RTK registers a
+ PreToolUse hook through Claude's settings.json and through a Copilot plugin -- there is
+ no common spelling). Everything else is data.
+ """
+
+ def __init__(self, kind, home, instructions_file, nav_tools, mcp_args,
+ patch_excludes=(), bin_dir="/usr/local/bin", env=None):
+ self.kind = kind # "claude" | "copilot" | "codex"
+ self.home = home # $HOME the agent runs under
+ self.instructions_file = instructions_file # repo-relative project-instructions path
+ self.nav_tools = tuple(nav_tools) # native code-navigation tool names
+ self._mcp_args = mcp_args # (cfg_path) -> argv fragment
+ self.patch_excludes = tuple(patch_excludes) # agent artifacts, always excluded
+ # Where a scaffold may install an executable, and where the agent's own shell will
+ # find it. `/usr/local/bin` is right in a task container, which runs as root; a
+ # harness that runs the agent on the HOST (Web-Bench) has no writable /usr/local
+ # and passes a per-run directory it also puts on PATH. A scaffold must never
+ # hardcode the path -- doing so silently no-ops the install off-container: the
+ # `chmod` fails, `&&` short-circuits, and the row reads `<tool>_installed: false`.
+ self.bin_dir = bin_dir
+ # Environment an install step must run under to land where the AGENT will look.
+ # Codex reads its global instructions from `$CODEX_HOME/AGENTS.md`, and CODEX_HOME is
+ # not $HOME (on SWE-bench it is /opt/codex-home, on Web-Bench a per-task dir), so a
+ # scaffold that ran `rtk init -g --codex` under the bare profile would write into a
+ # directory the CLI never reads -- healthy provenance over an arm that is native.
+ self.env = dict(env or {})
+
+ def mcp_args(self, cfg_path):
+ return list(self._mcp_args(cfg_path)) if cfg_path else []
+
+
+class ScaffoldContext:
+ """Per-task handles a scaffold is allowed to use."""
+
+ def __init__(self, *, profile, container, task, run_dir, exec_, copy_in, host_run,
+ read_bytes=None, log=None):
+ self.profile = profile
+ self.container = container
+ self.task = task
+ self.run_dir = run_dir
+ self.exec_ = exec_ # (script, *, timeout=, env=) -> CompletedProcess
+ self.copy_in = copy_in # (host_path, container_path) -> CompletedProcess
+ self.host_run = host_run # (argv, *, timeout=) -> CompletedProcess (HOST side)
+ # (script, *, timeout=) -> bytes: run a script in the container and return its RAW
+ # stdout. Separate from exec_ because a tar stream is not text and decoding it
+ # would corrupt it; a harness that cannot provide it simply cannot run the arms
+ # that need it, which is checked rather than discovered mid-task.
+ self._read_bytes = read_bytes
+ self._log = log or (lambda msg: None)
+
+ def read_bytes(self, script, timeout=600):
+ if self._read_bytes is None:
+ raise ScaffoldError(
+ "this harness provides no `read_bytes` on the baseline context, which "
+ "this baseline needs to stream the repository out of the container")
+ return self._read_bytes(script, timeout=timeout)
+
+ def log(self, msg):
+ self._log(msg)
+
+ # ---- helpers every scaffold needs -----------------------------------------------
+ def write_instructions(self, text):
+ """Write `text` to the agent's project-instructions file under /testbed.
+
+ Copied in rather than heredoc'd through `bash -lc`: a repo map is a couple of
+ thousand characters of arbitrary source-derived text, and any quoting scheme that
+ survives backticks, `$(`, and embedded EOF markers is a bug waiting to happen.
+ `docker cp` moves bytes.
+ """
+ rel = self.profile.instructions_file
+ path = "/testbed/" + rel
+ parent = os.path.dirname(path)
+ if parent and parent != "/testbed":
+ self.exec_(f"mkdir -p {shlex.quote(parent)}", timeout=30)
+ tmp = os.path.join(self.run_dir, "_scaffold_instructions")
+ with open(tmp, "w", encoding="utf-8") as fh:
+ fh.write(text)
+ res = self.copy_in(tmp, path)
+ os.unlink(tmp)
+ return res.returncode == 0
+
+ def write_mcp_config(self, servers, filename="scaffold.mcp.json"):
+ """Write an `{"mcpServers": {...}}` config into $HOME and return its path.
+
+ Under $HOME, never under /testbed: a config file in the repo would land in the
+ graded diff, and excluding it would be one more thing to remember.
+ """
+ cfg = f"{self.profile.home}/{filename}"
+ tmp = os.path.join(self.run_dir, "_scaffold_mcp.json")
+ with open(tmp, "w", encoding="utf-8") as fh:
+ json.dump({"mcpServers": servers}, fh, indent=2)
+ self.exec_(f"mkdir -p {shlex.quote(self.profile.home)}", timeout=30)
+ res = self.copy_in(tmp, cfg)
+ os.unlink(tmp)
+ return cfg if res.returncode == 0 else None
+
+ def merge_json_file(self, path, patch):
+ """Merge `patch` into a container-side JSON file, creating it if absent.
+
+ Used for the agent's own settings file, which the CLI may have seeded already --
+ clobbering it would drop keys the agent needs. Runs a here-doc'd Python program
+ rather than `jq`, which the era-pinned SWE-bench images do not carry.
+ """
+ prog = (
+ "import json, os, sys\n"
+ f"p = {path!r}\n"
+ f"patch = json.loads({json.dumps(json.dumps(patch))})\n"
+ "os.makedirs(os.path.dirname(p), exist_ok=True)\n"
+ "try:\n"
+ " cur = json.load(open(p))\n"
+ "except Exception:\n"
+ " cur = {}\n"
+ "for k, v in patch.items():\n"
+ " if isinstance(v, dict) and isinstance(cur.get(k), dict):\n"
+ " cur[k].update(v)\n"
+ " else:\n"
+ " cur[k] = v\n"
+ "json.dump(cur, open(p, 'w'), indent=2)\n"
+ "print('merged')\n")
+ # A UNIQUE name, not a fixed one. In a task container /tmp is private, but a
+ # harness that runs the agent on the HOST (Web-Bench) shares /tmp across lanes, and
+ # two concurrent runs on one fixed path would race: whichever wrote last decides
+ # what both merge, and the loser silently registers the wrong settings.
+ stamp = hashlib.sha1(
+ f"{self.run_dir}|{path}|{os.getpid()}".encode()).hexdigest()[:12]
+ remote = f"/tmp/_scaffold_merge_{stamp}.py"
+ tmp = os.path.join(self.run_dir, f"_scaffold_merge_{stamp}.py")
+ with open(tmp, "w", encoding="utf-8") as fh:
+ fh.write(prog)
+ self.copy_in(tmp, remote)
+ os.unlink(tmp)
+ res = self.exec_(f"python3 {remote}; rm -f {remote}", timeout=60)
+ return "merged" in (res.stdout or "")
+
+
+class Scaffold:
+ """Base class. Subclasses set `name` / `patch_excludes` and implement `setup`."""
+
+ name = "base"
+ # Paths (repo-relative, git-pathspec syntax) this scaffold writes under /testbed. The
+ # runner subtracts them from the graded diff -- see the package docstring.
+ patch_excludes = ()
+
+ def mounts(self, profile):
+ """Extra `docker run -v` spec strings. Default: none."""
+ return []
+
+ def preflight(self, profile):
+ """Raise ScaffoldError for a misconfiguration that dooms EVERY task. Called once,
+ at runner construction, so a lane fails before it spends anything."""
+ return
+
+ def setup(self, ctx):
+ """Install and configure in the started container. Return a notes dict for
+ `row.json`; it must include `arm_setup` and should record whether each step
+ actually took effect, not merely that it was attempted."""
+ raise NotImplementedError
diff --git a/tokeneconomizer/baselines/repomap.py b/tokeneconomizer/baselines/repomap.py
new file mode 100644
index 0000000..f9860fc
--- /dev/null
+++ b/tokeneconomizer/baselines/repomap.py
@@ -0,0 +1,145 @@
+"""Repo map — aider's token-budgeted ranked map of the repository, injected as context.
+
+Generated on the HOST and written into the container as project instructions. Host-side
+because aider needs a modern Python and tree-sitter, and the SWE-bench images are pinned to
+the issue's era; installing aider inside would both fail on old images and change the
+environment the agent is being measured in.
+
+Getting the source to the host is the part that differs from the original implementation.
+That one read a pre-staged worktree of every instance, which this repo does not have -- the
+repo exists only inside the container, checked out at `base_commit` by the image. So the
+source is streamed out through the context's `read_bytes` (this module never touches Docker
+itself): `git ls-files '*.py'` piped through `tar`, which is
+ * exact -- it is the container's own tree at the graded commit, not a host checkout that
+ could drift from it;
+ * cheap -- only tracked `.py` files, which is all the generator walks anyway, so a
+ matplotlib-sized repo moves a few MB rather than a `.git` directory;
+ * self-cleaning -- it lands in the task's run dir and is deleted after generation.
+
+The map is a FIXED prefix cost charged on every turn of every task, which is exactly why it
+is a baseline worth measuring rather than an obvious win: it shortens long-navigation tasks
+and taxes ones that were already localized. `map_chars` is recorded per row so that tax is
+visible in the results rather than inferred.
+"""
+import io
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+
+from .base import Scaffold, ScaffoldError
+
+DEFAULT_MAP_TOKENS = 2048
+GEN = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_repomap_gen.py")
+
+HEADER = ("# Repository map\n"
+ "Ranked key files and symbols (signatures only) for this repository. Consult "
+ "this to locate relevant code before searching.\n\n")
+
+
+def map_tokens():
+ return int(os.environ.get("TE_REPOMAP_TOKENS", DEFAULT_MAP_TOKENS))
+
+
+def extract_idents(text):
+ """Heuristic identifiers from the issue body, used to personalize the ranking.
+
+ Reads ONLY the issue text the agent is given, so there is no gold-patch leakage: the
+ map is ranked toward names the agent could have extracted itself.
+ """
+ import re
+ ids = set(re.findall(r"`([A-Za-z_][A-Za-z0-9_.]*)`", text or ""))
+ ids |= set(re.findall(r"\b([a-z_][a-z0-9_]*_[a-z0-9_]+)\b", text or ""))
+ ids |= set(re.findall(r"\b([A-Z][a-zA-Z0-9]+[A-Z][a-zA-Z0-9]*)\b", text or ""))
+ return sorted({i.split(".")[-1] for i in ids if len(i) > 2})[:60]
+
+
+class RepoMapScaffold(Scaffold):
+ name = "repomap"
+ patch_excludes = () # writes only the agent's own instructions file, excluded already
+
+ def preflight(self, profile):
+ """aider must be importable by THIS interpreter -- the generator is a subprocess of
+ it. Checked once at construction: a missing dependency that surfaces per-task would
+ produce a whole tree of `map_ok: false` rows that look like a scaffold result.
+
+ The message names the interpreter, and puts the WRONG-INTERPRETER cause first.
+ "Install aider" is the wrong advice for the likelier case: the launchers run
+ `${PYTHON:-python3}`, so a lane started from an unactivated shell -- a fresh
+ terminal, a cron job, a detached script -- picks up the system Python even though
+ `pip install -e '.[baselines]'` put aider in the venv exactly as the guide says. A
+ message that only says "install it" sends that reader to reinstall a package they
+ already have.
+ """
+ try:
+ import aider.repomap # noqa: F401
+ except ImportError as e:
+ raise ScaffoldError(
+ "the repomap scaffold needs aider importable by the interpreter running "
+ f"this lane, which is {sys.executable}. It generates the map on the HOST "
+ "because the container's Python is era-pinned. Two causes, likeliest "
+ "first:\n"
+ " 1. WRONG INTERPRETER -- aider is installed, but not here. The launchers "
+ "run `${PYTHON:-python3}`, so an unactivated shell picks up the system "
+ "Python. Fix: `source .venv/bin/activate`, or pass "
+ "`PYTHON=/path/to/.venv/bin/python` to the launcher.\n"
+ " 2. NOT INSTALLED -- `pip install -e '.[baselines]'` (or 'aider-chat') "
+ "into THAT interpreter.\n"
+ f"({e})") from None
+
+ # `git ls-files -z` + `tar --null -T -` so a path containing a space or a newline
+ # cannot split into two arguments and silently truncate the map's view of the repo.
+ TAR_SCRIPT = ("cd /testbed && git ls-files -z -- '*.py' | "
+ "tar -cf - --null -T - 2>/dev/null")
+
+ def fetch_sources(self, ctx):
+ """Stream the container's tracked `.py` files into a host directory. Returns it."""
+ dest = os.path.join(ctx.run_dir, "_repomap_src")
+ os.makedirs(dest, exist_ok=True)
+ blob = ctx.read_bytes(self.TAR_SCRIPT)
+ if not blob:
+ raise ScaffoldError("the repository stream out of the container was empty")
+ with tarfile.open(fileobj=io.BytesIO(blob)) as tf:
+ # No absolute or parent-relative members: this archive is built inside a task
+ # container, and extracting one blindly would let it write outside run_dir.
+ safe = [m for m in tf.getmembers()
+ if m.isfile() and not m.name.startswith(("/", ".."))
+ and "/../" not in m.name]
+ tf.extractall(dest, members=safe)
+ return dest
+
+ def generate(self, src_dir, idents):
+ """Run the host-side generator. Returns the map text ('' on failure)."""
+ res = subprocess.run(
+ [sys.executable, GEN, src_dir, str(map_tokens()), ",".join(idents)],
+ capture_output=True, text=True, timeout=600)
+ if res.returncode != 0:
+ return "", (res.stderr or "")[-300:]
+ return res.stdout, ""
+
+ def setup(self, ctx):
+ idents = extract_idents(ctx.task.get("problem_statement", ""))
+ notes = {"arm_setup": "repomap", "n_idents": len(idents),
+ "map_tokens_budget": map_tokens()}
+ src = None
+ try:
+ src = self.fetch_sources(ctx)
+ text, err = self.generate(src, idents)
+ except (ScaffoldError, subprocess.TimeoutExpired) as e:
+ text, err = "", str(e)[-300:]
+ finally:
+ if src and os.path.isdir(src):
+ shutil.rmtree(src, ignore_errors=True)
+
+ if not text.strip():
+ # No map means this task ran native. Flag it -- averaged into a cell as a
+ # repomap row it would dilute exactly the effect being measured.
+ ctx.log(f"repomap: generation produced no map ({err})")
+ notes.update({"map_ok": False, "map_chars": 0, "map_err": err})
+ return notes
+
+ notes["map_ok"] = ctx.write_instructions(HEADER + text)
+ notes["map_chars"] = len(text)
+ notes["instructions_file"] = ctx.profile.instructions_file
+ return notes
diff --git a/tokeneconomizer/baselines/rtk.py b/tokeneconomizer/baselines/rtk.py
new file mode 100644
index 0000000..29c1223
--- /dev/null
+++ b/tokeneconomizer/baselines/rtk.py
@@ -0,0 +1,322 @@
+"""RTK — a Bash-output rewriter, hooked in ahead of the agent's shell tool.
+
+Two integration modes, and the row says which one ran (`rtk_mode`):
+
+ * **hook** (Claude Code, Copilot): a PreToolUse hook rewrites the shell command before it
+ runs, transparently. The agent never knows. Compliance is not a question -- but WHICH
+ channel the hook is registered through is, on Copilot: that CLI has two, and they hand
+ the hook DIFFERENT payload shapes (see `_install_copilot_hook`). rtk parses one of them.
+ * **instructions** (Codex): rtk ships no hook for Codex -- `rtk init -g --codex` writes
+ `$CODEX_HOME/RTK.md` ("Always prefix shell commands with `rtk`") and an `AGENTS.md` that
+ references it, and the AGENT is expected to comply. Two consequences the harness owns:
+ - codex-cli 0.153.1 does NOT expand the `@/path/RTK.md` include rtk writes into
+ AGENTS.md (verified with `codex debug prompt-input`: the model-visible prompt carried
+ the literal `@` line and zero occurrences of the rule). The scaffold therefore
+ INLINES RTK.md into AGENTS.md itself, and records that it did.
+ - the saving depends on the agent actually typing `rtk ...`, so the runner reads the
+ transcript afterwards and reports `rtk_compliance`: the fraction of shell commands
+ that were rtk-prefixed. An arm at zero compliance is native, and the row says so.
+
+`rtk hook <agent>` is a local filter (no network) that rewrites shell output into
+token-compressed proxies. The scaffold's whole job is to make sure the hook is actually
+registered, which is the step the tool itself will not do for you: `rtk init -g` in
+non-interactive mode DECLINES to patch the agent's settings, prints a "MANUAL STEP" notice,
+and exits NONZERO (observed rc=1). Trusting it produces an arm where rtk is installed, `rtk --version` works,
+every provenance field looks healthy -- and the hook never fires, so the arm is native with
+extra steps. So the hook is written explicitly and the row records whether that took.
+
+The binary is staged on the HOST and copied in, because the sealed task container has no
+GitHub access: `install.sh` and the release tarball are both unreachable from inside. The
+in-container install is kept only as a fallback for an un-blackholed environment, and the
+row says which path was used (`rtk_staged_host`) so the two are never confused.
+
+Version is pinned. An arm whose scaffold silently tracked `latest` would not be a fixed
+baseline, and the published rows record `rtk 0.45.0`.
+"""
+import json
+import os
+import shlex
+
+from .base import Scaffold, ScaffoldError
+
+RTK_VERSION = "0.45.0"
+RTK_URL = (f"https://github.com/rtk-ai/rtk/releases/download/v{RTK_VERSION}/"
+ "rtk-x86_64-unknown-linux-musl.tar.gz")
+# In-container fallback, used only when host staging failed AND the container can reach
+# GitHub. Kept because an un-blackholed environment is a legitimate way to run this.
+RTK_INSTALL_SH = ("curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/"
+ "refs/heads/master/install.sh | sh")
+
+
+def cache_dir():
+ """Where the staged binary lives. `TE_SCAFFOLD_CACHE`, else under `$DATA`.
+
+ Not inside the repo: it is a multi-megabyte downloaded artifact, and the repo's own
+ results/artifacts are git-ignored precisely so that binaries do not accumulate in it.
+ """
+ explicit = os.environ.get("TE_SCAFFOLD_CACHE")
+ if explicit:
+ return explicit
+ data = os.environ.get("DATA")
+ if data:
+ return os.path.join(data, "scaffolds")
+ raise ScaffoldError(
+ "no scaffold cache: set TE_SCAFFOLD_CACHE (or DATA) so the rtk binary can be "
+ "staged on the host -- the sealed task container cannot reach GitHub. "
+ "See the repo's .env.example")
+
+
+class RtkScaffold(Scaffold):
+ name = "rtk"
+ # `rtk init -g` writes RTK.md and appends a reference to the agent's instructions file;
+ # both are scaffold artifacts, not the agent's fix.
+ patch_excludes = ("RTK.md", ".rtk", ".rtk/**")
+
+ # The PreToolUse hook, per agent. Claude Code reads hooks from settings.json; Copilot
+ # reads them from `$COPILOT_HOME/hooks/*.json` (NOT a plugin, and NOT settings.json --
+ # see `_install_copilot_hook` for the two channels and why only one works). There is no
+ # common spelling, so this is the one place the scaffold branches on the agent.
+ #
+ # THE COMMAND MUST BE AN ABSOLUTE PATH, not a bare `rtk`. The hook is executed by the
+ # AGENT's own subprocess, which does not inherit the PATH that `ctx.exec_` runs the
+ # install under. In a container that distinction is invisible -- bin_dir is
+ # /usr/local/bin, already on every PATH -- but on a host-run agent (Web-Bench) bin_dir
+ # is a per-task directory, and a bare `rtk` resolves to nothing.
+ #
+ # Measured 2026-09-05, Web-Bench/prisma/task-1 with the bare spelling: the debug log
+ # carried `rtk: command not found` x31 and the CLI reported the preToolUse hook
+ # erroring, while `rtk_installed` and `rtk_hook_registered` both read true. The agent
+ # flailed against the failing hook -- 612 `task` subagent calls, 553 `create`, 412
+ # `bash` -- and the task cost 75 turns / 1.79M tokens / 778 s against a 6-turn,
+ # 140k-token, 86 s baseline. So this is worse than a silent no-op: a bare hook command
+ # actively corrupts the run at ~12x the cost.
+ @staticmethod
+ def _hook_binary(profile):
+ return f"{profile.bin_dir}/rtk"
+
+ @classmethod
+ def _claude_hook(cls, profile):
+ return {"hooks": {"PreToolUse": [
+ {"matcher": "Bash", "hooks": [
+ {"type": "command",
+ "command": f"{cls._hook_binary(profile)} hook claude"}]}]}}
+
+ # The `$COPILOT_HOME/hooks/*.json` document, in rtk's OWN published shape (what
+ # `rtk init -g --copilot` writes): schema `version`, PascalCase `PreToolUse`, no
+ # matcher, a per-call timeout. The one change is the command, which rtk writes as a
+ # bare `rtk` -- see `_hook_binary` for why that must be absolute here.
+ COPILOT_HOOK_FILE = "rtk-rewrite.json"
+
+ @classmethod
+ def _copilot_hook_config(cls, profile):
+ return {"version": 1, "hooks": {"PreToolUse": [
+ {"type": "command",
+ "command": f"{cls._hook_binary(profile)} hook copilot",
+ "cwd": ".",
+ "timeout": 10}]}}
+
+ def bin_path(self):
+ return os.path.join(cache_dir(), "bin", "rtk")
+
+ def preflight(self, profile):
+ # Resolving the cache directory is what can be misconfigured; the download itself
+ # is attempted per-task and falls back, so it is not a preflight failure.
+ cache_dir()
+
+ def stage_host_binary(self, ctx):
+ """Fetch the pinned musl binary to the host cache once. Idempotent."""
+ dest = self.bin_path()
+ if os.path.exists(dest) and os.access(dest, os.X_OK):
+ return True
+ os.makedirs(os.path.dirname(dest), exist_ok=True)
+ tgz = dest + ".tar.gz"
+ res = ctx.host_run(
+ ["bash", "-lc",
+ f"curl -fsSL -o {tgz} {RTK_URL} && tar xzf {tgz} -C {os.path.dirname(dest)} "
+ f"&& chmod +x {dest} && rm -f {tgz}"],
+ timeout=300)
+ ok = res.returncode == 0 and os.access(dest, os.X_OK)
+ if not ok:
+ ctx.log(f"rtk: host staging failed ({(res.stderr or '')[-200:]})")
+ return ok
+
+ def setup(self, ctx):
+ notes = {"arm_setup": "rtk", "rtk_pinned_version": RTK_VERSION}
+
+ # 1. the binary
+ staged = self.stage_host_binary(ctx)
+ notes["rtk_staged_host"] = bool(staged)
+ if staged:
+ dest = f"{ctx.profile.bin_dir}/rtk"
+ ctx.copy_in(self.bin_path(), dest)
+ ver = ctx.exec_(f"chmod +x {dest} && rtk --version", timeout=60)
+ else:
+ ver = ctx.exec_(
+ f"({RTK_INSTALL_SH}) && "
+ f"(cp ~/.local/bin/rtk {ctx.profile.bin_dir}/ 2>/dev/null "
+ f"|| cp /root/.cargo/bin/rtk {ctx.profile.bin_dir}/ 2>/dev/null || true) && "
+ "rtk --version", timeout=300)
+ notes["rtk_installed"] = ver.returncode == 0
+ notes["rtk_version"] = (ver.stdout or "").strip()[-40:]
+
+ if ctx.profile.kind == "codex":
+ notes.update(self._codex_instructions(ctx))
+ return notes
+
+ # 2. rtk's own init (generates RTK.md and instruction references). It may decline
+ # to touch settings -- that is expected, and step 3 is why it does not matter.
+ init = ctx.exec_("rtk init -g", timeout=60,
+ env={"HOME": ctx.profile.home})
+ notes["rtk_init_rc"] = init.returncode
+
+ # 3. register the hook OURSELVES -- the step `rtk init -g` skips non-interactively.
+ hook_bin = self._hook_binary(ctx.profile)
+ notes["rtk_mode"] = "hook" # the Codex branch above returns "instructions"
+ notes["rtk_hook_cmd"] = hook_bin
+ if ctx.profile.kind == "claude":
+ notes["rtk_hook_registered"] = ctx.merge_json_file(
+ f"{ctx.profile.home}/.claude/settings.json",
+ self._claude_hook(ctx.profile))
+ else:
+ notes.update(self._install_copilot_hook(ctx))
+
+ # 3b. Registration is not the same as INVOCATION. `rtk_hook_registered` only says
+ # the settings/plugin write took; it says nothing about whether the agent can run
+ # the command. Resolve it the way the agent's hook subprocess will -- by absolute
+ # path, under a PATH that does NOT include bin_dir -- so a hook that cannot execute
+ # is a flag on the row rather than 31 `command not found` lines inside a run that
+ # still reports healthy.
+ probe = ctx.exec_(f"env PATH=/usr/bin:/bin {hook_bin} --version", timeout=60)
+ notes["rtk_hook_exec_ok"] = probe.returncode == 0
+ # 3c. Registered AND executable is still not REWRITING -- see _probe_hook_rewrites.
+ notes["rtk_hook_rewrites"] = self._probe_hook_rewrites(ctx, ctx.profile.kind)
+ if notes.get("rtk_hook_registered") and not notes["rtk_hook_rewrites"]:
+ ctx.log("rtk: WARNING the hook is registered but does NOT rewrite this "
+ "payload shape -- the arm would run as native with extra steps. "
+ "Check the channel against baselines/rtk._install_copilot_hook.")
+ if notes.get("rtk_hook_registered") and not notes["rtk_hook_exec_ok"]:
+ ctx.log(f"rtk: WARNING hook registered but {hook_bin} is not executable by the "
+ f"agent's hook subprocess -- every bash call would raise a hook error "
+ f"and the arm would cost far MORE than native, not less.")
+
+ if notes["rtk_installed"] and not notes.get("rtk_hook_registered"):
+ # The exact silent-native failure this scaffold exists to prevent. Say it
+ # loudly; the row carries the flag either way.
+ ctx.log("rtk: WARNING binary present but hook NOT registered -- this arm "
+ "would run as native. Check the agent's settings/plugin path.")
+ return notes
+
+ # What the inlined AGENTS.md must contain for the instruction to have reached the agent.
+ RTK_RULE = "Always prefix shell commands with"
+
+ def _codex_instructions(self, ctx):
+ """Codex: instruction files under $CODEX_HOME, inlined, no hook.
+
+ `rtk init -g --codex` writes `$CODEX_HOME/RTK.md` and puts a one-line
+ `@$CODEX_HOME/RTK.md` reference in `$CODEX_HOME/AGENTS.md`. That reference is a
+ Claude-Code CLAUDE.md convention; codex-cli 0.153.1 loads AGENTS.md verbatim and does
+ not expand it, so the agent would see a dangling path and no rule. The reference line
+ is replaced with RTK.md's contents, and the row records both that the files were
+ written and that the rule is now literally present in AGENTS.md.
+ """
+ env = dict(ctx.profile.env)
+ home = env.get("CODEX_HOME")
+ if not home:
+ raise ScaffoldError(
+ "the Codex profile carries no CODEX_HOME in its env: rtk would write its "
+ "instructions into a directory the CLI never reads")
+ env.setdefault("HOME", ctx.profile.home)
+ init = ctx.exec_("rtk init -g --codex", timeout=60, env=env)
+ notes = {"rtk_mode": "instructions", "rtk_init_rc": init.returncode,
+ "rtk_hook_registered": None} # not applicable on this agent
+ rule = self.RTK_RULE
+ # Replace the exact reference line with the file it points at, then count the rule.
+ inline = ctx.exec_(
+ f"H={home}; test -f $H/RTK.md && test -f $H/AGENTS.md && "
+ f"{{ grep -v -x -F \"@$H/RTK.md\" $H/AGENTS.md; printf '\\n'; cat $H/RTK.md; }} "
+ f"> $H/AGENTS.md.tmp && mv $H/AGENTS.md.tmp $H/AGENTS.md; "
+ f"grep -c '{rule}' $H/AGENTS.md", timeout=60, env=env)
+ try:
+ present = int((inline.stdout or "0").strip().splitlines()[-1])
+ except (ValueError, IndexError):
+ present = 0
+ notes["rtk_instructions_written"] = init.returncode == 0 and present > 0
+ notes["rtk_agents_inlined"] = present > 0
+ # The agent's own shell must find the binary under the PATH IT runs with -- the
+ # same probe the hook path makes, for the same reason.
+ probe = ctx.exec_(f"env PATH=/usr/bin:/bin {self._hook_binary(ctx.profile)} --version",
+ timeout=60)
+ notes["rtk_hook_exec_ok"] = probe.returncode == 0
+ if not notes["rtk_instructions_written"]:
+ ctx.log("rtk: WARNING binary present but the rtk rule did NOT reach "
+ f"{home}/AGENTS.md -- this arm would run as native. "
+ f"(init rc={init.returncode}, inline: {(inline.stderr or '')[-160:]})")
+ return notes
+
+ def _install_copilot_hook(self, ctx):
+ """Copilot: write rtk's hook into `$COPILOT_HOME/hooks/`, NOT a plugin.
+
+ Copilot CLI exposes two PreToolUse channels, and they do not hand the hook the same
+ payload -- measured on 1.0.83, 2026-09-13, by arming both at once with a dumper and
+ reading what each received:
+
+ $COPILOT_HOME/hooks/*.json {"hook_event_name","tool_name":"Bash",
+ "tool_input":{"command":...}} <- CLAUDE's shape
+ $COPILOT_HOME/settings.json {"toolName":"bash",
+ "toolArgs":{"command":...}} <- Copilot's own
+
+ **rtk parses only the first.** Handed the second it exits 0 with EMPTY stdout, which
+ the CLI reads as "no opinion" and runs the command unchanged. That is what the
+ previous implementation did: it shipped a plugin whose `hooks.json` used the
+ `preToolUse` spelling, so rtk was installed, the plugin reported "installed
+ successfully", the hook fired on every call -- and rewrote nothing. Every Copilot
+ `rtk` / `trimech-rtk` row before 2026-09-13 is that no-op.
+
+ On the hooks-dir channel the loop is closed end to end: the model asked for
+ `ls -la src`, the CLI executed `rtk ls -la src`, and the agent got rtk's compressed
+ output. rtk's reply there is Claude's own `hookSpecificOutput.updatedInput`, which
+ this CLI honours -- so Copilot is a TRANSPARENT rewrite like Claude, not the
+ instruction mode Codex needs, and no compliance field applies.
+
+ `${COPILOT_HOME:-$HOME/.copilot}` is resolved in the shell rather than here because
+ the two lanes disagree: SWE-bench leaves COPILOT_HOME unset and the CLI defaults it
+ to `$HOME/.copilot` in the container, while Web-Bench redirects it to a per-task dir
+ so a sweep cannot write into the operator's real config. One expansion is right in
+ both, and the resolved path is recorded.
+
+ The net guard keeps using `settings.json` (its own shape, its own channel), so the
+ two never contend: both were observed firing on one call.
+ """
+ cfg = json.dumps(self._copilot_hook_config(ctx.profile), indent=2)
+ name = self.COPILOT_HOOK_FILE
+ script = (
+ 'H="${COPILOT_HOME:-$HOME/.copilot}"; mkdir -p "$H/hooks" && '
+ f'cat > "$H/hooks/{name}" <<\'TE_EOF\'\n{cfg}\nTE_EOF\n'
+ f'test -s "$H/hooks/{name}" && echo "RTK_HOOK_AT=$H/hooks/{name}"')
+ res = ctx.exec_(script, timeout=60, env={"HOME": ctx.profile.home})
+ at = [ln.split("=", 1)[1] for ln in (res.stdout or "").splitlines()
+ if ln.startswith("RTK_HOOK_AT=")]
+ return {"rtk_hook_registered": bool(at),
+ "rtk_hook_channel": "copilot-hooks-dir",
+ "rtk_hook_path": at[0] if at else None,
+ "rtk_hook_log": ((res.stderr or "") + (res.stdout or ""))[-200:]}
+
+ # A payload in the shape BOTH hook channels this scaffold uses actually deliver
+ # (Claude's `tool_name`/`tool_input`; Copilot's hooks dir sends the same). Used to prove
+ # the installed binary REWRITES, rather than merely being present and executable.
+ _PROBE_PAYLOAD = '{"tool_name":"Bash","tool_input":{"command":"ls -la"}}'
+
+ def _probe_hook_rewrites(self, ctx, agent):
+ """Does `rtk hook <agent>` actually rewrite, on THIS binary and THIS payload shape?
+
+ `rtk_hook_registered` says the config was written and `rtk_hook_exec_ok` that the
+ binary runs; neither catches a hook that parses nothing and returns an empty
+ verdict, which is exactly how the Copilot arm was silently native for a week. This
+ is the effect-level check: feed the hook the payload its channel delivers and look
+ for the rewrite in its answer. Cheap, offline, and it fails loudly if a future rtk
+ changes its reply shape.
+ """
+ res = ctx.exec_(
+ f"printf '%s' {shlex.quote(self._PROBE_PAYLOAD)} | "
+ f"{self._hook_binary(ctx.profile)} hook {agent}", timeout=60)
+ return "rtk ls" in (res.stdout or "")
diff --git a/tokeneconomizer/baselines/serena.py b/tokeneconomizer/baselines/serena.py
new file mode 100644
index 0000000..4df516e
--- /dev/null
+++ b/tokeneconomizer/baselines/serena.py
@@ -0,0 +1,92 @@
+"""Serena — an off-the-shelf LSP-backed semantic-code MCP toolkit.
+
+Installed per-container through `uv` (it wants its own Python 3.13, which the era-pinned
+SWE-bench images do not have), exposed to the agent as an MCP server, and then *asked* to
+be used in place of the native navigation tools.
+
+Asked, not enforced. The original study nudged with an instruction and did not pass
+`--disallowed-tools`, and that is kept exactly: `Read` is in `PROTECTED_TOOLS`, so an
+exclusion would be filtered out by the runner and the arm would quietly run as native with
+an unused MCP server attached. An instruction the agent is free to disobey is a real
+property of this baseline -- if the agent keeps grepping, that IS the measurement.
+
+The install needs network (PyPI + astral.sh) from inside the container. Unlike RTK's
+GitHub-blackholed path there is no host-staging fallback here, because what gets installed
+is an entire Python toolchain rather than one static binary -- so on a fully sealed host
+this arm cannot run, and the row says `serena_installed: false` rather than pretending.
+"""
+from .base import Scaffold
+
+# The semantic tools the nudge names. Listed explicitly rather than discovered, because
+# the nudge has to name them for the agent and a drifting list would silently weaken it.
+SERENA_TOOLS = ("find_symbol", "get_symbols_overview", "find_referencing_symbols",
+ "find_implementations", "search_for_pattern")
+
+INSTALL = ("curl -LsSf https://astral.sh/uv/install.sh | sh >/tmp/uv.log 2>&1; "
+ "export PATH=$HOME/.local/bin:$PATH; "
+ "uv tool install -p 3.13 serena-agent >/tmp/serena_install.log 2>&1 && "
+ "serena --version")
+
+
+class SerenaScaffold(Scaffold):
+ name = "serena"
+ # Serena indexes the project into `.serena/` inside the repo.
+ patch_excludes = (".serena", ".serena/**")
+
+ def setup(self, ctx):
+ home = ctx.profile.home
+ ins = ctx.exec_(INSTALL, timeout=900, env={"HOME": home})
+ installed = ins.returncode == 0
+ notes = {"arm_setup": "serena", "serena_installed": installed,
+ "serena_version": (ins.stdout or "").strip()[-30:]}
+ if not installed:
+ # Every later step depends on the binary; recording the failure and running
+ # native is the honest outcome, and the flag makes the row unusable-as-serena
+ # rather than silently counted as one.
+ ctx.log("serena: install FAILED -- this task runs without the MCP server "
+ f"({(ins.stderr or '')[-200:]})")
+ notes["serena_nudged"] = False
+ return notes
+
+ spec = self.mcp_server(ctx.profile)
+ cfg = ctx.write_mcp_config({"serena": spec})
+ notes["mcp_config"] = cfg
+ if ctx.profile.kind == "codex":
+ # Codex takes MCP servers from `config.toml`, not from an argv flag, so the JSON
+ # above is provenance only there; the runner rewrites the config with this spec
+ # (te_harness.swebench.CodexSwebenchRunner._apply_scaffold). Returned for Codex
+ # alone so Claude and Copilot rows stay byte-identical to before.
+ notes["mcp_servers"] = {"serena": spec}
+ notes["serena_nudged"] = ctx.write_instructions(self.nudge(ctx.profile))
+ return notes
+
+ @staticmethod
+ def mcp_server(profile):
+ """The one server spec, spelled for the agent that will launch it.
+
+ Serena ships a context per client (oraios.github.io/serena, "clients"): `codex` for
+ Codex CLI, and the IDE-assistant context this scaffold has always used for the two
+ agents that take an argv MCP config. The context trims the tool set to what the
+ client can use, which is the only per-agent difference; project, transport and the
+ dashboard flag are the same everywhere so the three lanes run the SAME baseline.
+ """
+ home = profile.home
+ context = "codex" if profile.kind == "codex" else "ide-assistant"
+ return {"command": f"{home}/.local/bin/serena",
+ "args": ["start-mcp-server", "--project", "/testbed",
+ "--context", context, "--transport", "stdio",
+ "--enable-web-dashboard", "false"],
+ "env": {"PATH": f"{home}/.local/bin:/usr/local/bin:/usr/bin:/bin"}}
+
+ @staticmethod
+ def nudge(profile):
+ tools = ", ".join("`" + t + "`" for t in SERENA_TOOLS)
+ # Codex has no separate navigation tools: it reads and searches through the shell,
+ # so the tools to steer it away from are the shell ones.
+ forbid = "/".join(profile.nav_tools) or "rg/grep/sed/cat in the shell"
+ return ("# MANDATORY code-navigation policy\n"
+ "For ANY code search, symbol lookup, or 'where is X defined/used' task, "
+ "you MUST use the Serena MCP tools (use tool search to load them if "
+ f"needed): {tools}. Do NOT use {forbid} to locate or browse code -- use "
+ "Serena's semantic tools instead. They return precise, symbol-level "
+ "results and are the required way to navigate this repository.\n")
diff --git a/tokeneconomizer/config.py b/tokeneconomizer/config.py
new file mode 100644
index 0000000..40315a5
--- /dev/null
+++ b/tokeneconomizer/config.py
@@ -0,0 +1,150 @@
+"""EconomizerConfig: three independent on/off switches, one per mechanism.
+
+The three mechanisms act at three different lifecycle points and never contend for the
+same lever, so the config is just their union. `from_args` maps CLI-style flags onto the
+config; unknown attributes default off (via getattr), so a partial args object is fine.
+
+ * injection -- inject a firing-gated per-repo operating manual before the task.
+ * tooling -- drop inapplicable tool schemas (generic evidence untooling for
+ Claude/Copilot; Codex-native capability untooling for Codex).
+ * spiral -- kill/nudge a stuck rollout in flight (generic guard for Claude/Copilot;
+ Codex-native post-green finalize for Codex).
+"""
+from dataclasses import dataclass, field
+
+
+@dataclass
+class InjectionConfig:
+ """Injection (learned per-repo operating manual) stage.
+
+ `propose_every` is the paid proposer cadence (one LLM tick per this many tasks). The
+ injection Stage is the single source of truth for it and rebinds the distiller module
+ global at build time. Default 5 -- first injection a few tasks earlier and more
+ revision/grounding ticks over a stream.
+ """
+ enabled: bool = False
+ propose_every: int = 5
+ # Which distiller(s) render the prefix:
+ # distiller -- the generic paid-proposer manual (`injection.Distiller`);
+ # envfacts -- LLM-free environment facts learned from recurring evidence
+ # shapes (`envfacts.EnvFactsDistiller`). Makes NO paid call;
+ # envfacts+distiller -- both, the free layer first and its claims handed to the
+ # proposer as already-stated. Paid (the proposer).
+ profile: str = "distiller"
+ # Caps on the environment-fact block (`envfacts`): at most this many facts, and this many
+ # tokens (~4 characters each) for the whole block. 0 keeps envfacts.MAX_FACTS / MAX_TOKENS.
+ max_facts: int = 0
+ max_tokens: int = 0
+
+ PROFILES = ("distiller", "envfacts", "envfacts+distiller")
+
+ def __post_init__(self):
+ if self.profile not in self.PROFILES:
+ raise ValueError(f"injection profile {self.profile!r} is not one of {self.PROFILES}")
+
+ @property
+ def paid(self):
+ """True when this stage will make a paid between-task LLM call."""
+ return bool(self.enabled and "distiller" in self.profile)
+
+
+@dataclass
+class ToolingConfig:
+ """Untooling stage.
+
+ Generic evidence untooling (Claude / Copilot):
+ * evidence -- drop a tool loaded-but-unused for >= this many tasks (>0 on).
+ * evidence_reaudit -- periodically reload the full catalog every this many tasks.
+ * judge -- add a per-task instruction-aware LLM tool judge.
+
+ Codex-native capability untooling:
+ * codex_untool -- enable the evidence-gated, resolve-safe capability controller.
+ """
+ evidence: int = 0
+ evidence_reaudit: int = 0
+ judge: bool = False
+ codex_untool: bool = False
+
+ @property
+ def enabled(self):
+ return bool(self.evidence or self.codex_untool)
+
+
+@dataclass
+class SpiralConfig:
+ """Spiral guard stage (generic for Claude/Copilot; Codex-native for Codex)."""
+ guard: bool = False # generic spiral guard (Claude / Copilot)
+ judge: bool = False # sampled LLM trace-judge for the abort decision
+ nudge: bool = False # rung-1 same-container nudge before abort
+ nudge_llm: bool = False # LLM-written nudge redirect (vs the count template)
+ aggr: str = "default" # aggressiveness: conservative | default | aggressive
+ frozen_envelope: float = None # pre-registered resolved-turn envelope for SHORT
+ # streams where within-stream p90 never warms up;
+ # None = learn within-stream only.
+ codex_spiral: bool = False # Codex-native controller (post-green finalize)
+ # Which Codex policy: "post-green" (the shipped finalize rule alone) or "defensive"
+ # (finalize plus every deterministic spiral signature, two distinct nudges per task,
+ # never an abort).
+ codex_spiral_profile: str = "post-green"
+ CODEX_SPIRAL_PROFILES = ("post-green", "defensive")
+
+ def __post_init__(self):
+ if self.codex_spiral_profile not in self.CODEX_SPIRAL_PROFILES:
+ raise ValueError(f"codex_spiral_profile must be one of "
+ f"{self.CODEX_SPIRAL_PROFILES}, got {self.codex_spiral_profile!r}")
+
+ @property
+ def enabled(self):
+ return bool(self.guard or self.codex_spiral)
+
+
+def needs_optimizer(config):
+ """Whether ANY enabled mechanism will make a paid between-task LLM call.
+
+ The three clauses mirror `facade.Economizer.__init__` exactly -- it resolves an
+ optimizer runner for injection when `injection.paid` (enabled AND a profile that
+ carries the paid `distiller`; the LLM-free `envfacts` profile never calls), for untooling when
+ `tooling.judge`, and for spiral when the guard is on AND it has a judge or an LLM
+ nudge writer. Kept as one predicate because callers OUTSIDE the facade need the
+ aggregate answer: a harness whose mechanism transport must be preflighted should not
+ pay for (or depend on) that probe when the arm will never call it. Arm `mode` is NOT
+ the test -- `spiral-only` and `untool-only` are `standalone` and do make paid calls.
+
+ tests/test_optimizer_basis.py asserts this agrees with the facade for every arm in the
+ registry, so the two cannot drift.
+ """
+ return bool(
+ config.injection.paid
+ or config.tooling.judge
+ or (config.spiral.guard and (config.spiral.judge or config.spiral.nudge_llm))
+ )
+
+
+@dataclass
+class EconomizerConfig:
+ injection: InjectionConfig = field(default_factory=InjectionConfig)
+ tooling: ToolingConfig = field(default_factory=ToolingConfig)
+ spiral: SpiralConfig = field(default_factory=SpiralConfig)
+
+ @classmethod
+ def from_args(cls, args):
+ injection = InjectionConfig(
+ enabled=bool(getattr(args, "inject", False)),
+ propose_every=int(getattr(args, "propose_every", 0)
+ or InjectionConfig.propose_every),
+ profile=str(getattr(args, "inject_profile", "") or InjectionConfig.profile))
+ tooling = ToolingConfig(
+ evidence=int(getattr(args, "untool_evidence", 0)),
+ evidence_reaudit=int(getattr(args, "untool_evidence_reaudit", 0)),
+ judge=bool(getattr(args, "untool_judge", False)),
+ codex_untool=bool(getattr(args, "codex_untool", False)))
+ spiral = SpiralConfig(
+ guard=bool(getattr(args, "spiral_guard", False)),
+ judge=bool(getattr(args, "spiral_judge", False)),
+ nudge=bool(getattr(args, "spiral_nudge", False)),
+ nudge_llm=bool(getattr(args, "spiral_nudge_llm", False)),
+ aggr=getattr(args, "spiral_aggr", "default"),
+ frozen_envelope=getattr(args, "spiral_frozen_envelope", None),
+ codex_spiral=bool(getattr(args, "codex_spiral", False)),
+ codex_spiral_profile=getattr(args, "codex_spiral_profile", None) or "post-green")
+ return cls(injection=injection, tooling=tooling, spiral=spiral)
diff --git a/tokeneconomizer/facade.py b/tokeneconomizer/facade.py
new file mode 100644
index 0000000..c0e1526
--- /dev/null
+++ b/tokeneconomizer/facade.py
@@ -0,0 +1,98 @@
+"""Economizer facade: drives the three lifecycle mechanisms around one task.
+
+A caller builds one Economizer per repo stream and calls, per task, in this order:
+
+ injected = econ.injection_prefix(task) # learned manual for this task
+ exclude_tools = econ.build_tools(task, i) # untool exclusion set / capability mask
+ run_kwargs = econ.run_kwargs(task) # spiral params for the runner
+ ... run the task via the AgentRunner with those decisions ...
+ opt_tokens, reason = econ.after_task(task, row, run_dir_path, srow)
+
+The facade owns runner resolution (the ``runner or make_runner()`` fallback so the paid
+proposer / tool-judge / spiral-judge / nudge-writer closures work even when no standing
+runner is threaded) and calls each mechanism's post-task hook in a fixed order
+(tooling -> spiral -> injection) so the emitted row is deterministic.
+
+`adapter` is the DomainAdapter the mechanisms read the task domain through (default
+SwebenchAdapter). `runner` may be an AgentRunner or a bare ``(prompt, model) ->
+(text, eff_tokens)`` optimizer callable; it is adapted for the mechanisms automatically.
+"""
+from .config import EconomizerConfig
+from .runner import as_optimizer_callable
+from .adapters.swebench import SwebenchAdapter
+from .mechanisms.injection import PrefixStage
+from .mechanisms.tooling import ToolingStage
+from .mechanisms.spiral import SpiralStage
+
+
+class Economizer:
+ def __init__(self, config, *, agent, repo, results_root, fresh,
+ optimizer_model="agent", max_turns=80,
+ runner=None, make_runner=None, adapter=None):
+ """`runner` supplies the paid optimizer call; `make_runner` is a zero-arg factory
+ used to lazily build one only when a mechanism actually needs it. Either may be an
+ AgentRunner or a bare optimizer callable. `adapter` defaults to SwebenchAdapter."""
+ self.config = config
+ self.adapter = adapter if adapter is not None else SwebenchAdapter()
+
+ def _resolve():
+ r = runner if runner is not None else (make_runner() if make_runner else None)
+ return as_optimizer_callable(r)
+
+ # ---- injection ----
+ # The paid proposer needs a live optimizer runner; resolve one lazily. The LLM-free
+ # `envfacts` profile makes no call, so it gets none (config.needs_optimizer agrees).
+ injection_runner = _resolve() if config.injection.paid else None
+ self.injection = PrefixStage(
+ config.injection, repo, results_root, fresh,
+ runner=injection_runner, optimizer_model=optimizer_model,
+ adapter=self.adapter, agent=agent)
+
+ # ---- tooling (untooling) ----
+ judge_tool_runner = _resolve() if config.tooling.judge else None
+ self.tooling = ToolingStage(
+ config.tooling, agent, optimizer_model,
+ judge_tool_runner=judge_tool_runner,
+ adapter=self.adapter, repo=repo, results_root=results_root, fresh=fresh)
+
+ # ---- spiral ----
+ spiral_runner = None
+ if config.spiral.guard and (config.spiral.judge or config.spiral.nudge_llm):
+ spiral_runner = _resolve()
+ self.spiral = SpiralStage(
+ config.spiral, max_turns,
+ runner=spiral_runner, optimizer_model=optimizer_model,
+ adapter=self.adapter, agent=agent)
+
+ @classmethod
+ def from_args(cls, args, *, repo, results_root, fresh, runner=None,
+ make_runner=None, adapter=None):
+ return cls(EconomizerConfig.from_args(args),
+ agent=args.agent, repo=repo, results_root=results_root,
+ fresh=fresh, optimizer_model=getattr(args, "optimizer_model", "agent"),
+ max_turns=getattr(args, "max_turns", 80),
+ runner=runner, make_runner=make_runner, adapter=adapter)
+
+ # ---- per-task hooks the orchestrator calls ----
+ def injection_prefix(self, task=None):
+ """The learned per-repo manual for this task ('' if injection disabled)."""
+ return self.injection.render(task=task) if self.config.injection.enabled else ""
+
+ def build_tools(self, task, i):
+ """Tool-exclusion set / capability mask for this task (None if untooling off)."""
+ return self.tooling.build_tools(task, i)
+
+ def run_kwargs(self, task):
+ """Per-task runner kwargs from the tooling and spiral mechanisms."""
+ out = dict(self.tooling.run_kwargs(task))
+ out.update(self.spiral.run_kwargs(task))
+ return out
+
+ def after_task(self, task, row, run_dir_path, srow):
+ """Run every mechanism's post-task hook in a fixed order and return
+ (opt_tokens, reason|None). `reason` is set only by the injection stage."""
+ self.tooling.after_task(task, row, run_dir_path, srow)
+ self.spiral.after_task(task, row, run_dir_path, srow)
+ if self.config.injection.enabled:
+ return self.injection.after_task(task, row, run_dir_path, srow)
+ return 0, None
diff --git a/tokeneconomizer/ledger.py b/tokeneconomizer/ledger.py
new file mode 100644
index 0000000..6e58416
--- /dev/null
+++ b/tokeneconomizer/ledger.py
@@ -0,0 +1,79 @@
+"""One honest cost definition for the economizer.
+
+The amortized cost of a task is the agent's EXECUTION cost plus every mechanism's LLM
+OVERHEAD, on the same cache-aware effective basis (pricing.effective_tokens):
+
+ task_cost(t) = effective_exec(t) + overhead(t)
+ amortized(T) = (1/T) * sum_{t<=T} task_cost(t)
+
+`overhead()` sums the per-mechanism token fields each mechanism emits into the stream row.
+Every field here is real mechanism-internal LLM cost that must be amortized into the
+honest task cost -- so a method that mainly prunes cheap cached re-reads cannot look
+better than its real-dollar effect, and one that adds paid proposer/judge/nudge calls
+pays for them in the same ledger.
+
+This is the SINGLE definition of "cost" for the whole package: the mechanisms write these
+raw fields at emit time and the aggregation scripts (experiments/scripts/summarize_stream.py,
+experiments/scripts/aggregate.py) read the total back through this module, so the emitted ledger and
+the reported ledger can never drift.
+"""
+from .vendor import pricing
+
+# Per-task mechanism-overhead token fields, by owning mechanism. All are real
+# mechanism-internal LLM cost that must be amortized into the honest task cost.
+OVERHEAD_KEYS = (
+ "optimizer_tokens", # injection: paid between-task manual proposer (manual4)
+ "untool_classify_tokens", # untooling: one-shot tool-classifier LLM call
+ "untool_judge_tokens", # untooling: per-task instruction-aware tool judge
+ "codex_untool_adapt_tokens", # codex untooling: repository capability classifier
+ "spiral_judge_tokens", # spiral: paid live trace-judge
+ "spiral_nudge_writer_tokens", # spiral: targeted-nudge writer
+)
+
+
+def _num(x):
+ return x if isinstance(x, (int, float)) else 0
+
+
+def overhead(row):
+ """Sum every mechanism-overhead token field present on a stream row."""
+ return sum(_num(row.get(k)) for k in OVERHEAD_KEYS)
+
+
+def effective_exec(row):
+ """Cache-aware effective execution cost of a task (pricing.effective_tokens)."""
+ return pricing.effective_tokens(row)
+
+
+class CostLedger:
+ """Accumulates the amortized cost over a stream. Thin wrapper over the two canonical
+ functions above -- used by tests and any caller that wants a running total without
+ re-deriving the math. The aggregation scripts share these definitions via
+ OVERHEAD_KEYS / overhead()."""
+
+ def __init__(self):
+ self.cum_exec = 0.0
+ self.cum_overhead = 0.0
+ self.n = 0
+
+ def add(self, row, exec_cost=None):
+ """Fold one stream row. `exec_cost` overrides the effective exec (e.g. a
+ breakdown-resolved record); defaults to this row's effective exec."""
+ ec = effective_exec(row) if exec_cost is None else exec_cost
+ oh = overhead(row)
+ self.cum_exec += ec
+ self.cum_overhead += oh
+ self.n += 1
+ return ec + oh
+
+ @property
+ def cum_total(self):
+ return self.cum_exec + self.cum_overhead
+
+ @property
+ def amortized_per_task(self):
+ return (self.cum_total / self.n) if self.n else 0.0
+
+ @property
+ def overhead_frac(self):
+ return (self.cum_overhead / self.cum_total) if self.cum_total else 0.0
diff --git a/tokeneconomizer/mechanisms/__init__.py b/tokeneconomizer/mechanisms/__init__.py
new file mode 100644
index 0000000..d909ab0
--- /dev/null
+++ b/tokeneconomizer/mechanisms/__init__.py
@@ -0,0 +1,23 @@
+"""The three lifecycle mechanisms of the economizer, each acting at a different point in
+a task's lifecycle and independently switchable:
+
+ * injection (mechanisms.injection.PrefixStage) -- before the task: inject a
+ firing-gated, learned per-repo operating manual.
+ * tooling (mechanisms.tooling.ToolingStage) -- at task construction: drop
+ inapplicable tool schemas (generic evidence untooling; Codex-native
+ capability untooling).
+ * spiral (mechanisms.spiral.SpiralStage + -- in flight / at finalize: kill or
+ mechanisms.codex_spiral) nudge a stuck rollout; Codex-native
+ post-green finalize.
+
+All three read a normalized trajectory through the one trace seam (tokeneconomizer.trace)
+and charge their LLM overhead through the one ledger (tokeneconomizer.ledger).
+"""
+from .base import Stage
+from .injection import PrefixStage, Distiller
+from .tooling import ToolingStage
+from .spiral import SpiralStage
+from .codex_spiral import CodexSpiralPolicy, DEFAULT_POLICY
+
+__all__ = ["Stage", "PrefixStage", "Distiller", "ToolingStage", "SpiralStage",
+ "CodexSpiralPolicy", "DEFAULT_POLICY"]
diff --git a/tokeneconomizer/mechanisms/arms.py b/tokeneconomizer/mechanisms/arms.py
new file mode 100644
index 0000000..4d17228
--- /dev/null
+++ b/tokeneconomizer/mechanisms/arms.py
@@ -0,0 +1,252 @@
+"""Arms that involve the mechanisms — including the two scaffold integrations.
+
+The `trimech-rtk` / `trimech-repomap` arms live **here, beside `trimech`**, not in
+`baselines/arms.py`, because that is what they are: `trimech` with one external tool
+co-activated. Filing them under the baselines would make them read as scaffold arms that
+happen to have mechanisms, which is backwards — the mechanisms are the subject and the
+scaffold is the comparison. They name their scaffold with a string, so this module still
+imports nothing from `tokeneconomizer.baselines`.
+
+`baselines/arms.py` holds only the arms that are *purely* an external tool with no
+mechanism active (`rtk`, `serena`, `repomap`).
+"""
+from .._armspec import arm, cfg
+from ..config import InjectionConfig, ToolingConfig, SpiralConfig
+
+__all__ = ["MECHANISM_ARMS", "INJECT", "ENVFACTS", "ENVFACTS_PLUS", "UNTOOL", "SPIRAL",
+ "SPIRAL_WEB", "CODEX_UNTOOL", "CODEX_SPIRAL", "CODEX_SPIRAL_POSTGREEN"]
+
+# The proven single-mechanism flag sets, split out of `trimech` so each mechanism can be
+# measured on its own. They are the SAME parameters trimech co-activates -- an arm that
+# retuned its mechanism would not be an ablation of trimech, it would be a different arm.
+INJECT = dict(enabled=True, propose_every=5) # micro-skill synthesis + injection
+# The LLM-free environment-fact layer alone (mechanisms/envfacts). No paid call, so
+# `needs_optimizer` is False. An ablation of the default below, not the default itself.
+ENVFACTS = dict(enabled=True, profile="envfacts")
+# THE CODEX INJECTION DEFAULT (2026-09-12): the free layer plus the paid proposer at half
+# the generic cadence -- the environment facts are already stated, so the proposer's calls
+# are rarer and its budget goes to facts beyond them.
+#
+# Why the paid layer is the default now. The free layer's interpreter fact formed because
+# of one harness defect: the project environment was not active, so `python` resolved to
+# the base interpreter. The harness now activates the environment on every lane, and
+# replaying the layer over a post-fix baseline forms NO interpreter fact at all -- only the
+# network-boundary line. With the defect gone the free layer abstains and the
+# open-vocabulary proposer is what is left to find repository facts; its spend is charged
+# to the arm.
+ENVFACTS_PLUS = dict(enabled=True, profile="envfacts+distiller", propose_every=10)
+# The Codex-native flag sets, split out for exactly the reason the generic ones above are:
+# `codex-trimech` co-activates THESE, so each Codex single-mechanism arm is a true ablation
+# of it rather than a separately tuned arm.
+CODEX_UNTOOL = dict(codex_untool=True) # config-toggle capability mask + lean prompt
+# The spiral half of `codex-trimech` is the DEFENSIVE profile, not the shipped post-green
+# one, and that is a deliberate choice rather than an upgrade-by-default. The defensive
+# profile NEVER aborts, which is what makes it safe to stack on top of two mechanisms that already
+# change the request
+# contract and the prompt. Its ablation is
+# `codex-spiral-defensive`, not `codex-spiral`.
+CODEX_SPIRAL = dict(codex_spiral=True, codex_spiral_profile="defensive")
+# There is deliberately NO `codex-*-web` variant, and the reason is a property of the Codex
+# controller rather than an omission. `trimech-web` / `spiral-only-web` exist because the
+# GENERIC envelope is a within-stream p90 that never warms up over a 20-task Web-Bench
+# stream, so those arms pre-register one and write the nudge with an LLM.
+# `CodexEnvelopeTracker` already carries a cold start (24 tool events, used until five
+# resolved tasks have been seen) and the Codex nudges are deterministic text, so the same
+# nine Codex arms are well-defined on both domains and mean the same thing on each. Expect
+# the spiral half to be near-dormant on Web-Bench all the same: its tasks are small
+# increments and a 24-event cold start is above most of them.
+# The SHIPPED post-green policy, kept reachable and named. It is not what the spiral arms
+# carry any more, for the reason above; it is what `codex-spiral-postgreen` is for, and what
+# every pre-2026-09-12 `codex_spiral_profile` in a stored run_config resolves to.
+CODEX_SPIRAL_POSTGREEN = dict(codex_spiral=True) # SpiralConfig's own default profile
+UNTOOL = dict(evidence=3, judge=True) # evidence untooling + tool judge
+SPIRAL = dict(guard=True, judge=True, nudge=True) # spiral guard -> judge -> nudge
+# Web-Bench's streams are 20-21 tasks, too short for the within-stream p90 envelope to warm
+# up, so its spiral arms pre-register one and write the nudge with the LLM.
+SPIRAL_WEB = dict(SPIRAL, nudge_llm=True, frozen_envelope=30.0)
+
+
+def _trimech(spiral=SPIRAL, **kw):
+ """All three mechanisms at trimech's own parameters. `kw` carries the scaffold, if any.
+
+ `spiral` selects the spiral tuning: SPIRAL for SWE-bench's long streams, SPIRAL_WEB for
+ Web-Bench's 20-21 task ones (LLM nudge + frozen envelope, because a within-stream p90
+ never warms up that fast). Nothing else differs between the two domains' variants.
+ """
+ return arm("manual4",
+ lambda: cfg(injection=InjectionConfig(**INJECT),
+ tooling=ToolingConfig(**UNTOOL),
+ spiral=SpiralConfig(**spiral)), **kw)
+
+
+def _codex_trimech(**kw):
+ """All three Codex mechanisms at their single-arm parameters. `kw` carries the scaffold."""
+ return arm("manual4",
+ lambda: cfg(injection=InjectionConfig(**ENVFACTS_PLUS),
+ tooling=ToolingConfig(**CODEX_UNTOOL),
+ spiral=SpiralConfig(**CODEX_SPIRAL)), **kw)
+
+
+MECHANISM_ARMS = {
+ # ---- naive (no mechanisms) -----------------------------------------------------------
+ "baseline": arm(
+ "standalone",
+ lambda: cfg()),
+ # Reasoning-effort variants of naive. The LEVEL IS PART OF THE ARM NAME, and the arm sets
+ # TE_EFFORT_LEVEL itself rather than trusting the caller to export it: the level is
+ # invisible in an EconomizerConfig, so a mislabelled cell would be indistinguishable from
+ # a correct one after the fact, and the whole point of the comparison is that the level
+ # differs. `baseline` itself is deliberately NOT pinned here -- it inherits the harness
+ # default (high), which is what every stored row was run at.
+ "baseline-medium": arm(
+ "standalone",
+ lambda: cfg(), {"TE_EFFORT_LEVEL": "medium"}),
+ "baseline-low": arm(
+ "standalone",
+ lambda: cfg(), {"TE_EFFORT_LEVEL": "low"}),
+
+ # ---- one mechanism active ------------------------------------------------------------
+ "inject-only": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**INJECT))),
+ "untool-only": arm(
+ "standalone",
+ lambda: cfg(tooling=ToolingConfig(**UNTOOL))),
+ "spiral-only": arm(
+ "standalone",
+ lambda: cfg(spiral=SpiralConfig(**SPIRAL))),
+ "spiral-only-web": arm(
+ "standalone",
+ lambda: cfg(spiral=SpiralConfig(**SPIRAL_WEB))),
+
+ # ---- all three active ----------------------------------------------------------------
+ # trimech = manual4 injection + evidence untooling(3)+judge + spiral guard/judge/nudge
+ "trimech": _trimech(),
+ # short Web-Bench streams add the LLM-written nudge + a cold-start frozen envelope
+ "trimech-web": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**INJECT),
+ tooling=ToolingConfig(**UNTOOL),
+ spiral=SpiralConfig(**SPIRAL_WEB))),
+
+ # ---- all three, co-activated with an external tool baseline --------------------------
+ # Siblings of `trimech`, and identical to it apart from the scaffold they name. Both
+ # combinations are well-defined: rtk hooks Bash (always in PROTECTED_TOOLS, so untooling
+ # can never drop it) and the repo map writes a file and touches no tool.
+ #
+ # There is deliberately NO `trimech-serena`. Serena's contract is that the agent
+ # navigates through its MCP tools instead of the native ones, which is the very thing
+ # evidence untooling prunes when they go unused -- the combined arm would measure that
+ # contention rather than either baseline.
+ "trimech-rtk": _trimech(scaffold="rtk"),
+ "trimech-repomap": _trimech(scaffold="repomap"),
+ # Web-Bench's variant of `trimech-rtk`, for the same reason `trimech-web` exists: on a
+ # 20-task stream the within-stream p90 envelope never warms up. Pairing rtk with plain
+ # `trimech` there would spiral-tune it differently from every other spiral-bearing
+ # Web-Bench arm, so `trimech-rtk` would not be comparable to `trimech-web` -- which is
+ # the single comparison the arm exists to make.
+ "trimech-rtk-web": _trimech(spiral=SPIRAL_WEB, scaffold="rtk"),
+
+ # ---- Codex-native --------------------------------------------------------------------
+ "codex-untool": arm(
+ "standalone",
+ lambda: cfg(tooling=ToolingConfig(**CODEX_UNTOOL))),
+ # THE CODEX SPIRAL ARM, and since 2026-09-12 it carries the DEFENSIVE profile -- the
+ # same move, for the same reason, that made `codex-inject` the composed profile: the
+ # plain name should be the current configuration, and the superseded one should be
+ # reachable under a name that says which it is. The defensive profile answers every
+ # deterministic spiral signature with a same-thread nudge, never aborts, and never
+ # touches reasoning effort.
+ #
+ # This is also what makes the Codex family analogous to the generic one: `spiral-only`
+ # IS trimech's spiral, and now `codex-spiral` IS codex-trimech's.
+ "codex-spiral": arm(
+ "standalone",
+ lambda: cfg(spiral=SpiralConfig(**CODEX_SPIRAL))),
+ # The shipped post-green finalize alone -- the named ablation, the twin of
+ # `codex-inject-envfacts`. Run it to attribute a defensive result between the finalize
+ # rule and the signatures added on top of it.
+ "codex-spiral-postgreen": arm(
+ "standalone",
+ lambda: cfg(spiral=SpiralConfig(**CODEX_SPIRAL_POSTGREEN))),
+ # The name the defensive profile shipped under on 2026-09-12, kept so the stored cell of
+ # that name and any pinned script still resolve to the same configuration as
+ # `codex-spiral`. The twin of `codex-inject-distiller`.
+ "codex-spiral-defensive": arm(
+ "standalone",
+ lambda: cfg(spiral=SpiralConfig(**CODEX_SPIRAL))),
+ # Untooling + the spiral, i.e. `codex-trimech` minus injection. It reads the same two
+ # dicts trimech does, so it stays that as the parameters move; when its spiral half was
+ # pinned to post-green independently it was quietly a different arm from the one its
+ # name promised.
+ "codex-combo": arm(
+ "standalone",
+ lambda: cfg(tooling=ToolingConfig(**CODEX_UNTOOL),
+ spiral=SpiralConfig(**CODEX_SPIRAL))),
+ # Codex injection, alone and stacked with capability untooling. `manual4` is the mode
+ # label every injection arm carries (aggregate.py recovers it from the run-dir name);
+ # the PROFILE in the config is what tells the prefix schemes apart, and it is what
+ # changed on 2026-09-12: `codex-inject` is now the composed `envfacts+distiller`
+ # (see ENVFACTS_PLUS above for why), which makes it a PAID arm -- `needs_optimizer` is
+ # True, the endpoint preflight fires, and the proposer's spend is charged to the arm
+ # through `optimizer_tokens`. Cells run before that date carry `profile: envfacts` in
+ # their run_config and are the ablation, not the default.
+ "codex-inject": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**ENVFACTS_PLUS))),
+ "codex-inject-untool": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**ENVFACTS_PLUS),
+ tooling=ToolingConfig(**CODEX_UNTOOL))),
+ # The LLM-free ablation: the environment-fact layer with no proposer, zero overhead.
+ # Run it to attribute a composed result between the two layers, or on a lane that must
+ # make no paid call at all.
+ "codex-inject-envfacts": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**ENVFACTS))),
+ # The name the composed profile shipped under between 2026-09-11 and 2026-09-12, kept
+ # so the stored cell of that name and any launch script that pins it still resolve to
+ # the same configuration as `codex-inject`.
+ "codex-inject-distiller": arm(
+ "manual4",
+ lambda: cfg(injection=InjectionConfig(**ENVFACTS_PLUS))),
+
+ # ---- all three Codex mechanisms active ----------------------------------------------
+ # The Codex sibling of `trimech`, and it composes the same way: the three flag sets
+ # above, unretuned, so `codex-inject`, `codex-untool` and `codex-spiral-defensive` are
+ # each a true ablation of this arm. It is `manual4` because injection is on -- that is
+ # the mode label `aggregate.py` recovers from the run-dir name.
+ #
+ # The three act on three different surfaces and do not contend:
+ # * injection writes a prefix into the task prompt (`build_prompt`);
+ # * untooling writes the per-task `config.toml` (capability toggles) and the lean
+ # `model_instructions_file` beside it;
+ # * the spiral guard reads the live `--json` stream and, when it fires, resumes the
+ # SAME thread in the SAME container -- which still has that config.toml and that
+ # instructions file, so the resumed segment inherits the untooled contract and the
+ # prompt prefix already in its history. Nothing is re-sent and nothing is re-paid.
+ #
+ # Two compounding effects to expect in the row, neither of them a defect:
+ # * the proposer's spend lands in `optimizer_tokens` (this is a PAID arm);
+ # * a task the guard nudges reports no usage for the killed segment, so
+ # `codex.merge_usage` poisons the merge and the row is flagged `usage_incomplete`.
+ # It still counts for resolve and drops out of the paired cost figure, exactly as
+ # in `codex-spiral-defensive`. With the defensive profile that is a few percent of
+ # tasks on terra.
+ "codex-trimech": _codex_trimech(),
+ # The Codex sibling of `trimech-rtk`: the same three mechanisms with rtk co-activated.
+ # On Codex the scaffold is INSTRUCTION-based -- `rtk init -g --codex` writes AGENTS.md +
+ # RTK.md and the agent is told to prefix its commands; there is no hook this CLI honours
+ # -- so the row carries `rtk_compliance` and an arm at zero compliance is native. rtk
+ # touches no tool schema and the capability mask touches no shell command, so the two
+ # do not contend, exactly as on the generic lane.
+ "codex-trimech-rtk": _codex_trimech(scaffold="rtk"),
+ # The Codex sibling of `trimech-repomap`: the same three mechanisms with aider's repo map
+ # co-activated. The map is written as the PROJECT `AGENTS.md` (/testbed), which Codex
+ # reads at session start and carries in every request. None of the three mechanisms
+ # touches that file -- injection rides the task prompt, untooling the config and the
+ # `model_instructions_file`, the spiral the live stream -- and no capability toggle in
+ # the mask disables project instructions, so map and mask coexist however far the mask
+ # has grown. SWE-bench only, like `repomap` (the generator walks `*.py`).
+ "codex-trimech-repomap": _codex_trimech(scaffold="repomap"),
+}
diff --git a/tokeneconomizer/mechanisms/base.py b/tokeneconomizer/mechanisms/base.py
new file mode 100644
index 0000000..62da1cd
--- /dev/null
+++ b/tokeneconomizer/mechanisms/base.py
@@ -0,0 +1,29 @@
+"""Stage protocol: the lifecycle hooks the Economizer facade calls around one task.
+
+Each mechanism implements only the hooks it needs; unused hooks keep the no-op default.
+The hooks fire in this order per task (see facade.Economizer):
+
+ before_task(task, injected) -> str # pre-rollout: may augment injected prefix
+ build_tools(task) -> list|None # construction: tool-exclusion set
+ run_kwargs(task) -> dict # construction: kwargs for the agent runner
+ after_task(task, row, run_dir_path, srow) # post-rollout: update state + telemetry
+ -> opt_tokens (int/float; 0 for stages with no overhead)
+
+Hooks receive the srow (stream row dict) so a mechanism can attach its own telemetry keys.
+"""
+
+
+class Stage:
+ name = "base"
+
+ def before_task(self, task, injected):
+ return injected
+
+ def build_tools(self, task):
+ return None
+
+ def run_kwargs(self, task):
+ return {}
+
+ def after_task(self, task, row, run_dir_path, srow):
+ return 0
diff --git a/tokeneconomizer/mechanisms/codex_injection.py b/tokeneconomizer/mechanisms/codex_injection.py
new file mode 100644
index 0000000..3a457cf
--- /dev/null
+++ b/tokeneconomizer/mechanisms/codex_injection.py
@@ -0,0 +1,28 @@
+"""The Codex lane's injection defaults -- a thin layer over the shared `envfacts` module.
+
+The detector families, the environment-fact distiller and the vocabulary plug-in live in
+`mechanisms/envfacts.py` and are shared by every lane: replayed over the stored Claude Code
+and Copilot streams they form the same kinds of facts (and abstain on the one stream whose
+shell was activated), so nothing about them is Codex-specific. What IS specific to the Codex
+lane is recorded here rather than scattered:
+
+ * the default profile is `envfacts+distiller` (2026-09-12): the LLM-free layer with the
+ paid proposer on top at cadence 10. It was `envfacts` while the harness left the project
+ environment inactive, when the free layer's interpreter fact formed. The harness now
+ activates the environment, the interpreter fact no longer forms on a post-fix baseline, and the proposer
+ is what remains to find repository facts. Its spend lands in `optimizer_tokens` like anywhere else; the
+ LLM-free ablation is the arm `codex-inject-envfacts`;
+ * Codex's `command_execution` items carry exact exit codes, which the shared observation
+ layer reads directly through `trace.load_trajectory` (Claude and Copilot supply an error
+ flag and the outcome is parsed from text);
+ * Codex has a developer-role instruction channel (`developer_instructions` in the per-task
+ `config.toml`, accepted by codex-cli 0.153.1) that could carry the block instead of the
+ prompt prefix. Not wired: every lane injects through `build_prompt` today.
+
+Every public name of `envfacts` is re-exported so existing imports keep working.
+"""
+from .envfacts import * # noqa: F401,F403
+from .envfacts import _is_bare, _PASSTHROUGH, _root # noqa: F401 (offline analyses)
+
+CODEX_DEFAULT_PROFILE = "envfacts+distiller"
+CODEX_FREE_PROFILE = "envfacts" # the LLM-free ablation (arm codex-inject-envfacts)
diff --git a/tokeneconomizer/mechanisms/codex_spiral.py b/tokeneconomizer/mechanisms/codex_spiral.py
new file mode 100644
index 0000000..d021a2f
--- /dev/null
+++ b/tokeneconomizer/mechanisms/codex_spiral.py
@@ -0,0 +1,1062 @@
+#!/usr/bin/env python3
+"""Codex-native spiral detection and same-thread recovery.
+
+This module is deliberately separate from :mod:`spiral`. The historical spiral
+implementation supports several agents and restarts a stopped rollout with a fresh
+CLI invocation. Codex exposes stronger signals and a native session-resume command:
+
+* ``command_execution.exit_code`` makes successful verification unambiguous;
+* ``thread.started.thread_id`` can be passed to ``codex exec resume``;
+* file-change events contain every changed path, not just a generic edit tool name.
+
+The policy below uses only those deterministic, locally observable signals. It has
+no paid judge and never terminally aborts a task. The live v1 policy has one trigger:
+after the resolved-task tool envelope, a patch spanning at least 15 distinct files has
+failed verification after its latest edit. At most once per task it checkpoints that
+trajectory and continues the *same* Codex thread. Broader failure-loop and post-green
+rules remain available for offline experiments but are disabled by default. An
+orthogonal, opt-in completion audit can use the same thread once when deterministic
+end-state evidence shows an empty patch or a still-failing post-edit verification.
+"""
+
+from __future__ import annotations
+
+from collections import Counter, deque
+from dataclasses import asdict, dataclass, replace
+import json
+import re
+import shlex
+from typing import Iterable, Optional
+
+
+_TEST_RUN_RE = re.compile(
+ r"(?i)(?:^|[;&|]\s*|\n\s*|[\"'])"
+ # Leading `K=V` assignments, and rtk's Codex-mode prefix (`rtk pytest`, `rtk proxy pytest`,
+ # `rtk K=V pytest`): the agent is TOLD to type it on every command, and a detector that
+ # did not absorb it would see no test run on `codex-trimech-rtk` -- no trustworthy green,
+ # no finalize, every failure signature blind. Kept in step with trace.RTK_PREFIX_RE.
+ r"(?:[A-Za-z_][A-Za-z0-9_]*=\S+\s+|rtk\s+(?:proxy\s+)?)*"
+ r"(?:python(?:\d+(?:\.\d+)?)?\s+-m\s+(?:pytest|unittest|django\s+test)|"
+ r"pytest|py\.test|"
+ r"(?:python(?:\d+(?:\.\d+)?)?\s+)?(?:\./)?bin/test|"
+ r"(?:python(?:\d+(?:\.\d+)?)?\s+)?(?:\./)?(?:tests/)?runtests\.py|"
+ r"tox(?:\s|$)|nox(?:\s|$))"
+)
+_ASSERT_RE = re.compile(
+ r"(?i)(?:\bassert\b|assert_|np\.testing|testing\.assert|"
+ r"raise_if_dask_computes)"
+)
+_BLOCKED_RE = re.compile(
+ r"(?i)(?:gold-isolation|network (?:is )?unreachable|"
+ r"temporary failure in name resolution|could not resolve host|"
+ r"connection refused|request to .* blocked)"
+)
+_FAILURE_ID_RE = re.compile(
+ r"(?im)^(?:FAILED|ERROR)\s+([^\s]+)|^_{3,}\s*(.*?)\s*_{3,}$"
+)
+_FAILURE_LINE_RE = re.compile(
+ r"(?i)(?:assertionerror|traceback|\bfailed\b|\berror:|exception)"
+)
+_RESEARCH_COMMAND_RE = re.compile(
+ r"(?i)(?:\brg\b|\bgrep\b|\bsed\s+-n\b|\bfind\b|\bcurl\b|\bwget\b|"
+ r"\bgit\s+(?:log|show|blame|branch|tag|fsck)\b|\bnl\s+-ba\b|"
+ r"\bpip\s+cache\b|python(?:\d+(?:\.\d+)?)?\s+-\s*<<)"
+)
+_FINALIZATION_COMMAND_RE = re.compile(
+ r"(?i)(?:git\s+(?:diff|status)(?:\s|$)|git\s+diff\s+--check|"
+ r"compileall|py_compile|black\s+--check|ruff\s+check|pre-commit\s+run)"
+)
+# A passing test command whose green must NOT be trusted as a full success signal:
+# it narrowed the accepted set (ran a subset, deselected, or re-ran only failures).
+# Kept in sync with codex_injection_v3.PytestAcceptanceDetector._FILTERED so the
+# spiral post-green gate and the injection post-green hint share ONE definition of a
+# trustworthy green. This is the pytest plug-in's notion of "filtered"; a different
+# domain ships its own predicate and the finalize rule/nudge text stay unchanged.
+_FILTERED_GREEN_RE = re.compile(
+ r"(?i)(?:\s-k\b|--deselect\b|\bdeselect\b|\s-m\s+['\"]"
+ r"|--lf\b|--last-failed\b|--ff\b|--failed-first\b)"
+)
+
+
+def green_is_trustworthy(command: str) -> bool:
+ """True if a passing test command was a FULL (unfiltered) run, so its green can
+ gate post-green finalization. Narrowed runs (`-k`/`--deselect`/`--lf`/...) are
+ not trustworthy: the accepted set they passed is a subset."""
+ return not bool(_FILTERED_GREEN_RE.search(command or ""))
+
+
+# A pytest path/node-id token: an explicit test-file path (optionally a ``::nodeid``).
+# Used to attribute a pytest run to the concrete scope(s) it accepted or failed, so a
+# green on ONE file cannot mask an unresolved unfiltered failure on ANOTHER. A bare
+# pytest run with no path is the whole-suite scope (``_ALL_SCOPE``).
+_TEST_PATH_RE = re.compile(r"(?<![\w/.-])((?:[\w./-]*/)?[\w.-]*test[\w.-]*\.py)")
+_ALL_SCOPE = "__ALL__"
+
+
+def verification_scopes(command: str) -> Optional[set[str]]:
+ """Scopes a pytest command accepts/exercises, or ``None`` if it is not a pytest run.
+
+ Returns basenames of the test files named on the command line (e.g.
+ ``test_concat.py``), or ``{_ALL_SCOPE}`` for a bare whole-suite pytest run. Inline
+ ``python - <<`` reproducers return ``None`` -- they still drive the green/fail
+ counters via :func:`is_verification_command`, but they are not authoritative about
+ which *named* test scope passed, so they neither open nor clear outstanding failures.
+ """
+ command = command or ""
+ if not is_test_command(command):
+ return None
+ paths = {match.group(1).split("::", 1)[0].rsplit("/", 1)[-1]
+ for match in _TEST_PATH_RE.finditer(command)}
+ paths = {p for p in paths if p}
+ return paths or {_ALL_SCOPE}
+_NO_CHANGE_CLAIM_RE = re.compile(
+ r"(?i)(?:\balready (?:fixed|resolved) (?:at|in|on|by|the|this|current)\b|"
+ r"\bno (?:additional |source )?changes? (?:are|were|is|was) "
+ r"(?:necessary|required|needed)\b)"
+)
+
+COMPLETION_AUDIT_MARKER = "[AGENTEO_CODEX_COMPLETION_AUDIT]"
+
+
+@dataclass(frozen=True)
+class CodexEvent:
+ """One event relevant to the Codex spiral controller."""
+
+ kind: str
+ command: str = ""
+ output: str = ""
+ exit_code: Optional[int] = None
+ paths: tuple[str, ...] = ()
+ thread_id: str = ""
+
+
+@dataclass(frozen=True)
+class CodexSpiralPolicy:
+ """Conservative policy parameters, expressed in completed Codex tool events."""
+
+ min_tool_turns: int = 30
+ min_failed_verifications: int = 2
+ repeated_failure_count: int = 2
+ scatter_files: int = 15
+ churn_edits: int = 3
+ recent_window: int = 12
+ repeated_command_count: int = 3
+ blocked_attempts: int = 2
+ no_progress_turns: int = 8
+ post_green_research_turns: int = 4
+ scope_checkpoint_files: int = 8
+ recovery_failure_grace_turns: int = 0
+ recovery_enabled: bool = True
+ post_green_enabled: bool = False
+ scope_checkpoint_enabled: bool = False
+ repeated_failure_enabled: bool = False
+ wide_failure_enabled: bool = True
+ churn_failure_enabled: bool = False
+ repeated_command_enabled: bool = False
+ blocked_retrieval_enabled: bool = False
+ # Adaptive effort routing starts only after a recent resolved-task window. If a
+ # medium-effort task reaches that learned envelope without a current green result,
+ # checkpoint the exact thread and resume it at high effort instead of accepting a
+ # quality-risky cheap tail.
+ effort_rescue_enabled: bool = False
+ # Crossing the envelope on a fresh edit is progress, not a spiral. Let the medium
+ # segment spend a few tool events verifying that edit before paying to rebuild and
+ # resume the thread at high effort. Long localization with no edit is unaffected.
+ effort_rescue_progress_grace_turns: int = 3
+ resume_effort: str = ""
+ nudge_budget: int = 1
+ # ---- post-green finalize (codex-native-spiral-v2) ---------------------------
+ # The dominant, universal Codex cost sink is NOT thrashing-while-failing (the
+ # >=15-file wide_failure region is nearly empty on real traces): it is reaching a
+ # trustworthy green early and then wandering -- re-verifying, re-deriving the change
+ # from external history, style/lint fishing. This rule fires ONE same-thread nudge
+ # to finalize when the task's own success check has passed UNFILTERED and the
+ # trajectory has already run past the learned resolved-task envelope. It is gated on
+ # a trustworthy green, so it can never interrupt an unfinished task or trust a
+ # narrowed pass (the injection-v3 / xarray-7400 false-green trap).
+ post_green_finalize_enabled: bool = False
+ # Grace after the trustworthy green before nudging: never interrupt the accept step
+ # itself; only redirect the wandering that follows it.
+ post_green_finalize_grace_turns: int = 3
+ # Live A/B lesson (xarray, seed 0): an unfiltered green on ONE test file does not
+ # prove the patch is done when ANOTHER scope still has an outstanding unfiltered
+ # failure. Two false fires regressed resolve this way:
+ # * xarray-7400: gold `test_concat.py` failed unfiltered, was re-run only FILTERED,
+ # and the gate fired on an unfiltered green of a DIFFERENT file (`test_combine.py`).
+ # * xarray-6857: the full-suite `pytest` failed unfiltered after the last edit and
+ # was never cleared; the gate fired on a narrow green (`test_indexes.py`).
+ # When enabled, finalize additionally requires that NO named pytest scope (and not the
+ # whole-suite scope) is carrying an outstanding unfiltered failure -- i.e. every scope
+ # that ever failed unfiltered has since been re-run unfiltered and passed.
+ post_green_finalize_require_cleared_failures: bool = False
+ # ---- defensive rules (2026-09-12) ---------------------------------------------
+ # Green re-verification loop: after an edit the task's own check has passed, and
+ # then passed again `reverify_loop_count` more times with no edit in between.
+ # Re-running a passing check adds no information; it is the one post-green pattern
+ # the terra traces contain that is waste by construction.
+ reverify_loop_enabled: bool = False
+ reverify_loop_count: int = 3
+ # Runaway guard: the trajectory has run past `runaway_multiplier` times the learned
+ # resolved-task envelope (and past `runaway_floor` events), whatever its state. No
+ # baseline task on gpt-5.6-terra reached 2x the envelope, so this is a backstop for a
+ # future model or repository, not a lever.
+ runaway_enabled: bool = False
+ runaway_multiplier: float = 2.0
+ runaway_floor: int = 30
+
+ def at_threshold(self, turns: float | int) -> "CodexSpiralPolicy":
+ return replace(self, min_tool_turns=max(1, int(round(turns))))
+
+
+DEFAULT_POLICY = CodexSpiralPolicy()
+
+# The single shipped Codex spiral policy: "post-green finalize, cleared-scopes".
+#
+# The dominant Codex cost sink is NOT thrashing-while-failing (the >=15-file wide-failure
+# region is nearly empty on real traces) but reaching a trustworthy green early and then
+# WANDERING -- re-verifying, re-deriving the change from history, style/lint fishing. This
+# policy fires ONE same-thread nudge to finalize once the task's own success check has
+# passed UNFILTERED and the trajectory has run past the learned resolved-task envelope. It
+# is trustworthy-green gated (never trusts a narrowed `-k`/`--lf` pass) and, from the live
+# A/B, additionally requires that every named pytest scope that ever failed unfiltered has
+# since been re-run unfiltered and passed -- so a green on ONE file cannot mask an
+# outstanding unfiltered failure on ANOTHER (the xarray-7400 / xarray-6857 regressions its
+# predecessor hit). The wide-failure/recovery rules are disabled so this measures the
+# post-green lever alone. It never terminally aborts a task. It supersedes the single-v1,
+# hardened-v2 and post-green-v2 trials.
+PROFILE_VERSION = "codex-native-spiral-post-green-cleared-scopes"
+
+SHIPPED_POLICY = replace(
+ DEFAULT_POLICY,
+ post_green_finalize_enabled=True,
+ post_green_finalize_require_cleared_failures=True,
+ wide_failure_enabled=False,
+ recovery_enabled=False)
+
+
+def shipped_policy(threshold: float | int) -> CodexSpiralPolicy:
+ """The single live Codex spiral policy at the repository's task-frozen threshold."""
+ return SHIPPED_POLICY.at_threshold(threshold)
+
+
+# The DEFENSIVE profile (2026-09-12): the shipped post-green finalize plus every
+# deterministic spiral signature the controller knows, each a same-thread nudge and none
+# an abort, with a budget of two distinct nudges per task. It is not a cost lever -- on the
+# gpt-5.6-terra baselines the failure-loop region is empty, tails after a green are one
+# event long and nothing runs past 2x the envelope, so it fires on a few percent of tasks
+# and its expected cost effect is about zero. It is a net under trajectories that DO
+# spiral: repeated failing verification, the same command over and over, blocked
+# retrieval, a very wide or churning patch that keeps failing, a passing check re-run
+# again and again, or a run past twice the resolved envelope. Reasoning effort is never
+# touched.
+DEFENSIVE_PROFILE_VERSION = "codex-native-spiral-defensive-v1"
+DEFENSIVE_POLICY = replace(
+ SHIPPED_POLICY,
+ recovery_enabled=True,
+ repeated_failure_enabled=True,
+ wide_failure_enabled=True,
+ churn_failure_enabled=True,
+ repeated_command_enabled=True,
+ blocked_retrieval_enabled=True,
+ reverify_loop_enabled=True,
+ runaway_enabled=True,
+ nudge_budget=2)
+
+
+def defensive_policy(threshold: float | int) -> CodexSpiralPolicy:
+ """The defensive profile at the repository's task-frozen threshold."""
+ return DEFENSIVE_POLICY.at_threshold(threshold)
+
+
+def profile_policy(profile: str, threshold: float | int) -> CodexSpiralPolicy:
+ """`post-green` (the shipped default) or `defensive`, at a threshold."""
+ if profile == "defensive":
+ return defensive_policy(threshold)
+ return shipped_policy(threshold)
+
+
+def profile_version(profile: str) -> str:
+ return DEFENSIVE_PROFILE_VERSION if profile == "defensive" else PROFILE_VERSION
+
+
+# Which profile the Codex lane runs when an arm does not say otherwise -- the twin of
+# `codex_injection.CODEX_DEFAULT_PROFILE`, and settled the same way. `post-green` is the
+# SHIPPED policy and remains the dataclass default (so a config that predates this constant
+# still means what it meant). `defensive` is what the arms named for the mechanism carry,
+# and what `codex-trimech` co-activates. The shipped policy stays reachable, and named, as
+# `codex-spiral-postgreen`.
+CODEX_DEFAULT_PROFILE = "defensive"
+CODEX_SHIPPED_PROFILE = "post-green"
+
+
+def _normalize(text: str, limit: int = 180) -> str:
+ text = (text or "").lower()
+ text = re.sub(r"0x[0-9a-f]+", "<hex>", text)
+ text = re.sub(r"/tmp/\S+", "/tmp/<tmp>", text)
+ text = re.sub(r"\d+", "<n>", text)
+ text = re.sub(r"\s+", " ", text).strip()
+ return text[:limit]
+
+
+def _command_fingerprint(command: str) -> str:
+ # Codex normally reports `/bin/bash -lc '<actual command>'`. Fingerprinting the
+ # wrapper (or deleting its quoted body) makes every shell action look identical.
+ # Unwrap only the known shell form and retain the inner action; numeric/tmp-path
+ # normalization still coalesces incidental variations in genuinely repeated calls.
+ command = command or ""
+ try:
+ parts = shlex.split(command)
+ except ValueError:
+ parts = []
+ if (len(parts) >= 3 and parts[0].rsplit("/", 1)[-1] in ("bash", "sh")
+ and parts[1] in ("-c", "-lc")):
+ command = parts[2]
+ return _normalize(command, limit=320)
+
+
+def is_test_command(command: str) -> bool:
+ return bool(_TEST_RUN_RE.search(command or ""))
+
+
+def is_verification_command(command: str) -> bool:
+ command = command or ""
+ if is_test_command(command):
+ return True
+ is_python_heredoc = "python - <<" in command or "python3 - <<" in command
+ return bool(is_python_heredoc and _ASSERT_RE.search(command))
+
+
+def is_research_command(command: str) -> bool:
+ """Whether a post-green command reopens localization rather than wrapping up."""
+ command = command or ""
+ if is_verification_command(command) or _FINALIZATION_COMMAND_RE.search(command):
+ return False
+ return bool(_RESEARCH_COMMAND_RE.search(command))
+
+
+def failure_signatures(output: str) -> set[str]:
+ """Extract stable failure identities without retaining source/test contents."""
+ found = []
+ for match in _FAILURE_ID_RE.finditer(output or ""):
+ value = next((group for group in match.groups() if group), None)
+ if value:
+ found.append(_normalize(value))
+ if not found:
+ lines = [line.strip() for line in (output or "").splitlines()
+ if _FAILURE_LINE_RE.search(line)]
+ found.extend(_normalize(line) for line in lines[:2])
+ return {value for value in found if value}
+
+
+# Patches applied THROUGH THE SHELL. On some model providers the CLI stops emitting
+# `file_change` items altogether: every edit arrives as a `command_execution` whose command
+# is `apply_patch <<'PATCH' ... *** Begin Patch ... *** Update File: <p>`. A controller that
+# only reads `file_change` therefore never sees an edit there, never registers a trustworthy
+# green, and never fires. The paths are read from the patch header lines, and the
+# edit counts when the apply succeeded (the CLI's own "Success." marker, or exit 0).
+_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M)
+_GIT_APPLY_RE = re.compile(r"\bgit\s+apply\b")
+_GIT_APPLY_PATH_RE = re.compile(r"^\+\+\+ b/(.+?)\s*$", re.M)
+_PATCH_SUCCESS_RE = re.compile(r"(?i)\bSuccess\.\s*(?:Updated|Applied)")
+
+
+def patch_paths(command: str) -> tuple[str, ...]:
+ """Paths a shell-applied patch command touches, () when it is not one."""
+ command = command or ""
+ if "*** Begin Patch" in command:
+ found = _PATCH_FILE_RE.findall(command)
+ elif _GIT_APPLY_RE.search(command):
+ found = _GIT_APPLY_PATH_RE.findall(command)
+ else:
+ return ()
+ return tuple(dict.fromkeys(p.strip() for p in found if p.strip()))
+
+
+def shell_patch_applied(command: str, output: str, exit_code) -> bool:
+ """True when a shell-applied patch command actually changed files."""
+ if not patch_paths(command):
+ return False
+ if _PATCH_SUCCESS_RE.search(output or ""):
+ return True
+ try:
+ return int(exit_code) == 0
+ except (TypeError, ValueError):
+ return False
+
+
+def parse_events(line: str) -> list[CodexEvent]:
+ """Parse one ``codex exec --json`` line into controller events, in causal order.
+
+ A native `file_change` item is one edit event. A `command_execution` is one command
+ event -- preceded by an edit event when the command itself applied a patch through the
+ shell, so a verification chained after the patch in the same command is seen as
+ following the edit, exactly as it does."""
+ try:
+ event = json.loads(line)
+ except Exception:
+ return []
+ if event.get("type") == "thread.started":
+ return [CodexEvent("thread", thread_id=event.get("thread_id") or "")]
+ if event.get("type") != "item.completed":
+ return []
+ item = event.get("item") or {}
+ kind = item.get("type")
+ if kind == "command_execution":
+ command = item.get("command") or ""
+ output = item.get("aggregated_output") or ""
+ exit_code = item.get("exit_code")
+ try:
+ exit_code = int(exit_code) if exit_code is not None else None
+ except (TypeError, ValueError):
+ pass
+ out = []
+ if shell_patch_applied(command, output, exit_code):
+ out.append(CodexEvent("edit", paths=patch_paths(command)))
+ out.append(CodexEvent("command", command=command, output=output,
+ exit_code=exit_code))
+ return out
+ if kind == "file_change":
+ paths = tuple(
+ change.get("path") for change in (item.get("changes") or [])
+ if isinstance(change, dict) and change.get("path")
+ )
+ return [CodexEvent("edit", paths=paths)]
+ return []
+
+
+def parse_event(line: str) -> Optional[CodexEvent]:
+ """The PRIMARY event of a line (kept for callers that expect one); the command event
+ for a command line even when it also carries an edit -- use ``parse_events`` to see
+ both."""
+ events = parse_events(line)
+ if not events:
+ return None
+ return events[-1]
+
+
+def iter_events(lines: Iterable[str], *, first_thread_only: bool = False):
+ """Yield parsed events, optionally excluding suffixes made by an old restart."""
+ threads = 0
+ for line in lines:
+ for event in parse_events(line):
+ if event.kind == "thread":
+ threads += 1
+ if first_thread_only and threads > 1:
+ return
+ yield event
+
+
+def final_agent_message(lines: Iterable[str]) -> str:
+ """Return the last completed Codex agent message from JSONL text/lines."""
+ if isinstance(lines, str):
+ lines = lines.splitlines()
+ final = ""
+ for line in lines:
+ try:
+ event = json.loads(line)
+ except Exception:
+ continue
+ item = event.get("item") or {}
+ if (event.get("type") == "item.completed"
+ and item.get("type") == "agent_message"
+ and isinstance(item.get("text"), str)):
+ final = item["text"]
+ return final
+
+
+class CodexSpiralState:
+ """Incremental Codex-only process state used identically live and offline."""
+
+ def __init__(self, recent_window: int = DEFAULT_POLICY.recent_window):
+ self.turns = 0
+ self.command_turns = 0
+ self.edit_events = 0
+ self.edited_paths = Counter()
+ self.last_edit_turn = 0
+ self.last_green_turn = 0
+ self.last_progress_turn = 0
+ self.last_failed_verification_turn = 0
+ self.verification_runs = 0
+ self.post_green_research = 0
+ self.failed_verifications_since_green = 0
+ self.failed_verifications_after_edit = 0
+ self.failure_sigs_since_green = Counter()
+ self.recent_commands = deque(maxlen=recent_window)
+ self.recent_blocked = deque(maxlen=recent_window)
+ self.thread_id = ""
+ self.last_decision_reason = ""
+ # Turn of the first TRUSTWORTHY (unfiltered) green after an edit, and whether the
+ # current green is trustworthy. Gates post-green finalization: only a full,
+ # unfiltered pass may license "you are done, finalize".
+ self.first_trustworthy_green_turn = 0
+ self.last_green_trustworthy = False
+ # Named pytest scopes whose MOST RECENT unfiltered run failed and has not since
+ # been re-run unfiltered and passed. Whole-suite failures use ``_ALL_SCOPE``.
+ # Only pytest runs (not inline reproducers) open or clear scopes. A non-empty set
+ # means "somewhere an unfiltered test is still red" -- post-green finalize must not
+ # trust a green on a different scope while it is non-empty.
+ self.outstanding_failure_scopes: set[str] = set()
+ # Green verification runs after the FIRST green that followed the latest edit,
+ # with no edit in between -- the "re-verification loop" counter. Reset on edit.
+ self.green_reruns_since_edit = 0
+
+ @property
+ def distinct_edit_files(self) -> int:
+ return len(self.edited_paths)
+
+ @property
+ def edit_churn(self) -> int:
+ return max(self.edited_paths.values(), default=0)
+
+ @property
+ def green_after_edit(self) -> bool:
+ return bool(self.last_edit_turn and self.last_green_turn > self.last_edit_turn)
+
+ @property
+ def currently_green(self) -> bool:
+ return bool(self.last_green_turn > max(
+ self.last_edit_turn, self.last_failed_verification_turn))
+
+ @property
+ def repeated_failure_count(self) -> int:
+ return max(self.failure_sigs_since_green.values(), default=0)
+
+ def update(self, event: CodexEvent) -> None:
+ if event.kind == "thread":
+ if event.thread_id:
+ self.thread_id = event.thread_id
+ return
+ if event.kind not in ("command", "edit"):
+ return
+ self.turns += 1
+ if event.kind == "edit":
+ self.edit_events += 1
+ self.last_edit_turn = self.turns
+ self.last_progress_turn = self.turns
+ self.failed_verifications_after_edit = 0
+ self.post_green_research = 0
+ self.green_reruns_since_edit = 0
+ for path in event.paths:
+ self.edited_paths[path] += 1
+ self.recent_commands.append("<edit>")
+ self.recent_blocked.append(False)
+ return
+
+ self.command_turns += 1
+ command = event.command or ""
+ self.recent_commands.append(_command_fingerprint(command))
+ self.recent_blocked.append(bool(_BLOCKED_RE.search(event.output or "")))
+ if not is_verification_command(command):
+ if self.currently_green and is_research_command(command):
+ self.post_green_research += 1
+ return
+ self.verification_runs += 1
+ filtered = not green_is_trustworthy(command)
+ scopes = verification_scopes(command)
+ if event.exit_code == 0:
+ # A green targeted test/reproducer supersedes historical failures. This is
+ # the signal the generic spiral path lost on xarray-6599.
+ if self.last_edit_turn and self.last_green_turn > self.last_edit_turn:
+ # already green after this edit: this run re-verifies a passing state
+ self.green_reruns_since_edit += 1
+ self.last_green_turn = self.turns
+ self.last_progress_turn = self.turns
+ self.failed_verifications_since_green = 0
+ self.failed_verifications_after_edit = 0
+ self.failure_sigs_since_green.clear()
+ self.post_green_research = 0
+ # Clear outstanding unfiltered failures this UNFILTERED pass actually retired.
+ # A filtered green retires nothing (it narrowed the accepted set); an inline
+ # reproducer (scopes is None) is not authoritative about a named scope. A
+ # whole-suite unfiltered green clears everything.
+ if not filtered and scopes is not None:
+ if _ALL_SCOPE in scopes:
+ self.outstanding_failure_scopes.clear()
+ else:
+ # A NARROW green retires only its own scope(s). It must NOT clear a
+ # whole-suite (`_ALL_SCOPE`) failure -- that was the xarray-6857 trap,
+ # where a green on `test_indexes.py` masked an unresolved full-suite
+ # failure. Only a whole-suite unfiltered green clears `_ALL_SCOPE`.
+ self.outstanding_failure_scopes -= scopes
+ # Record whether THIS green was a full unfiltered pass. Only a trustworthy
+ # green (and only when it follows an edit) can license post-green finalize.
+ trustworthy = bool(self.last_edit_turn) and not filtered
+ self.last_green_trustworthy = trustworthy
+ if trustworthy and not self.first_trustworthy_green_turn:
+ self.first_trustworthy_green_turn = self.turns
+ return
+ if event.exit_code is None:
+ return
+ self.last_failed_verification_turn = self.turns
+ self.post_green_research = 0
+ self.failed_verifications_since_green += 1
+ if self.last_edit_turn:
+ self.failed_verifications_after_edit += 1
+ # An UNFILTERED pytest failure marks its scope(s) outstanding until re-greened
+ # unfiltered. Filtered failures and inline reproducers do not (a filtered run's
+ # verdict is about a subset; a reproducer names no scope).
+ if not filtered and scopes is not None:
+ self.outstanding_failure_scopes |= scopes
+ for signature in failure_signatures(event.output):
+ self.failure_sigs_since_green[signature] += 1
+
+ def decision(self, policy: CodexSpiralPolicy = DEFAULT_POLICY) -> Optional[str]:
+ """Return a deterministic nudge reason, or ``None`` to continue untouched."""
+ if self.turns < policy.min_tool_turns:
+ return None
+ # Post-green finalize (codex-native-spiral-v2): the task's own success check has
+ # passed UNFILTERED after an edit, yet the trajectory has run past the learned
+ # resolved-task envelope (enforced by min_tool_turns above) and kept going. Fire
+ # one same-thread nudge to finalize. Gated on a TRUSTWORTHY current green, so it
+ # never interrupts an unfinished task and never trusts a narrowed pass; a grace
+ # window keeps it from interrupting the accept step itself.
+ if policy.post_green_finalize_enabled:
+ grace = max(0, int(policy.post_green_finalize_grace_turns))
+ blocked = (policy.post_green_finalize_require_cleared_failures
+ and self.outstanding_failure_scopes)
+ # `blocked`: somewhere an unfiltered test is still red on a scope this green
+ # did not retire (the xarray-7400 / xarray-6857 regression class) -- do not
+ # finalize on that ground. The rules below may still apply.
+ if (not blocked and self.currently_green and self.last_green_trustworthy
+ and self.first_trustworthy_green_turn
+ and self.turns - self.first_trustworthy_green_turn >= grace):
+ return "validated-patch-should-finalize"
+ # Defensive rules. Both are checked BEFORE the
+ # edited-paths gate: a run past twice the envelope with no edit is a localization
+ # spiral, not a healthy trajectory.
+ if (policy.reverify_loop_enabled and self.currently_green
+ and self.green_reruns_since_edit >= policy.reverify_loop_count):
+ return "green-reverification-loop"
+ if policy.runaway_enabled and self.turns >= max(
+ int(policy.runaway_floor),
+ int(round(policy.runaway_multiplier * policy.min_tool_turns))):
+ return "beyond-twice-the-resolved-envelope"
+ if policy.effort_rescue_enabled and not self.currently_green:
+ grace = max(0, int(policy.effort_rescue_progress_grace_turns))
+ if (self.last_progress_turn
+ and self.turns - self.last_progress_turn < grace):
+ return None
+ # This rule intentionally also covers long localization with no edit: that
+ # is precisely where continuing indefinitely at reduced effort is riskiest.
+ return "medium-effort-envelope-rescue"
+ if not self.edited_paths:
+ return None
+ # A verified patch can still accumulate a costly Codex-specific tail when the
+ # model reopens localization instead of finalizing. Redirect only after several
+ # unmistakably research-oriented commands; ordinary diff/status/lint/test closure
+ # is deliberately excluded from this counter.
+ if policy.post_green_enabled and self.currently_green:
+ if self.post_green_research >= policy.post_green_research_turns:
+ return "validated-patch-reopened-research"
+ return None
+ # Large multi-file ports are a characteristic Codex cost sink. This is not a
+ # "you are wrong" nudge: it is a one-shot checkpoint asking the same thread to
+ # verify the accumulated patch before expanding it further. It can therefore
+ # fire before a failure, while the recovery rules below remain failure-gated.
+ if (policy.scope_checkpoint_enabled
+ and self.distinct_edit_files >= policy.scope_checkpoint_files):
+ return "wide-edit-scope-needs-verification"
+ if not policy.recovery_enabled:
+ return None
+ failures = self.failed_verifications_since_green
+ after_edit = self.failed_verifications_after_edit
+ latest_patch_failed = self.last_failed_verification_turn > self.last_edit_turn
+ if not latest_patch_failed:
+ return None
+ # Experimental rules may require a quiet grace window after the newest failure.
+ # Live v1 uses zero here because its prompt is a non-terminal checkpoint reserved
+ # for the much narrower >=15-file condition, not a generic failure intervention.
+ if (self.turns - self.last_failed_verification_turn
+ < policy.recovery_failure_grace_turns):
+ return None
+
+ if (policy.repeated_failure_enabled
+ and failures >= policy.min_failed_verifications
+ and self.repeated_failure_count >= policy.repeated_failure_count):
+ return "repeated-failing-verification"
+ if (policy.wide_failure_enabled
+ and self.distinct_edit_files >= policy.scatter_files
+ and after_edit >= policy.min_failed_verifications):
+ return "very-wide-patch-still-failing"
+ if (policy.churn_failure_enabled
+ and self.edit_churn >= policy.churn_edits
+ and after_edit >= policy.min_failed_verifications
+ and self.repeated_failure_count >= policy.repeated_failure_count):
+ return "edit-churn-still-failing"
+
+ no_progress = self.turns - self.last_progress_turn
+ commands = [value for value in self.recent_commands if value != "<edit>"]
+ same_command = max(Counter(commands).values(), default=0)
+ if (policy.repeated_command_enabled
+ and same_command >= policy.repeated_command_count
+ and no_progress >= policy.no_progress_turns
+ and failures):
+ return "repeated-command-without-progress"
+ if (policy.blocked_retrieval_enabled
+ and sum(self.recent_blocked) >= policy.blocked_attempts
+ and no_progress >= policy.no_progress_turns
+ and failures):
+ return "blocked-retrieval-without-progress"
+ return None
+
+ def snapshot(self) -> dict:
+ return {
+ "tool_turns": self.turns,
+ "command_turns": self.command_turns,
+ "edit_events": self.edit_events,
+ "distinct_edit_files": self.distinct_edit_files,
+ "edit_churn": self.edit_churn,
+ "last_edit_turn": self.last_edit_turn,
+ "last_green_turn": self.last_green_turn,
+ "last_failed_verification_turn": self.last_failed_verification_turn,
+ "failed_verifications_since_green": self.failed_verifications_since_green,
+ "failed_verifications_after_edit": self.failed_verifications_after_edit,
+ "repeated_failure_count": self.repeated_failure_count,
+ "green_after_edit": self.green_after_edit,
+ "currently_green": self.currently_green,
+ "post_green_research": self.post_green_research,
+ "verification_runs": self.verification_runs,
+ "first_trustworthy_green_turn": self.first_trustworthy_green_turn,
+ "last_green_trustworthy": self.last_green_trustworthy,
+ "outstanding_failure_scopes": sorted(self.outstanding_failure_scopes),
+ "green_reruns_since_edit": self.green_reruns_since_edit,
+ }
+
+
+def replay_events(events: Iterable[CodexEvent], policy: CodexSpiralPolicy = DEFAULT_POLICY):
+ """Return ``(first_decision, final_state)`` for an offline trajectory."""
+ state = CodexSpiralState(policy.recent_window)
+ decision = None
+ for event in events:
+ state.update(event)
+ if decision is None:
+ reason = state.decision(policy)
+ if reason:
+ decision = {
+ "turn": state.turns,
+ "reason": reason,
+ "snapshot": state.snapshot(),
+ }
+ return decision, state
+
+
+def _percentile(values: list[int], pct: float) -> Optional[float]:
+ if not values:
+ return None
+ ordered = sorted(values)
+ if len(ordered) == 1:
+ return float(ordered[0])
+ position = (pct / 100.0) * (len(ordered) - 1)
+ lower = int(position)
+ fraction = position - lower
+ if lower + 1 == len(ordered):
+ return float(ordered[-1])
+ return ordered[lower] + fraction * (ordered[lower + 1] - ordered[lower])
+
+
+class CodexEnvelopeTracker:
+ """Online resolved-task envelope measured in the same tool-event unit as live state."""
+
+ def __init__(self, *, cold_start: int = 24, floor: int = 20,
+ min_samples: int = 5, percentile: float = 90.0,
+ margin: float = 1.0):
+ self.cold_start = cold_start
+ self.floor = floor
+ self.min_samples = min_samples
+ self.percentile = percentile
+ self.margin = margin
+ self.resolved_tool_turns: list[int] = []
+
+ @property
+ def threshold(self) -> float:
+ if len(self.resolved_tool_turns) < self.min_samples:
+ return float(self.cold_start)
+ learned = _percentile(self.resolved_tool_turns, self.percentile)
+ return float(max(self.floor, (learned or 0) * self.margin))
+
+ def update(self, tool_turns: int, resolved: bool) -> None:
+ if resolved and tool_turns:
+ self.resolved_tool_turns.append(int(tool_turns))
+
+ def snapshot(self) -> dict:
+ return {
+ "threshold": self.threshold,
+ "resolved_samples": len(self.resolved_tool_turns),
+ "resolved_tool_turns": list(self.resolved_tool_turns),
+ }
+
+
+def nudge_text(reason: str) -> str:
+ if reason == "validated-patch-should-finalize":
+ # Deliberately DOMAIN-NEUTRAL: no `git diff`, no "test suite", no pytest. The
+ # firing gate (trustworthy unfiltered green + envelope) is precise and lives in
+ # the detector; the model-visible text is pure finalize-instead-of-wander
+ # guidance that reads the same for a test suite, a build, a linter, or a domain
+ # validator. It never narrows the success check.
+ return (
+ "Automated process checkpoint: the task's own success check has already "
+ "passed after your latest change, and this trajectory has run longer than "
+ "usual for a completed task. Preserve the current working state and continue "
+ "this same session -- do not restart your investigation, re-derive the change "
+ "from scratch, or reconstruct it from external history. Review what you have "
+ "already changed, run only the narrowest remaining check needed for "
+ "confidence, and finish once it still passes. Reopen the work only if that "
+ "check surfaces a concrete, current failure."
+ )
+ if reason == "green-reverification-loop":
+ return (
+ "Automated process checkpoint: the task's own success check has now passed "
+ "several times in a row with no change in between, so running it again will "
+ "not add information. Preserve the current working state and continue this "
+ "same session. If a concrete step remains, do the smallest one and check it "
+ "once; otherwise finish now."
+ )
+ if reason == "beyond-twice-the-resolved-envelope":
+ return (
+ "Automated process checkpoint: this trajectory has run more than twice as long "
+ "as completed tasks in this repository usually do. Continue this same session "
+ "and preserve the working tree. State in one line the single most likely "
+ "remaining cause, inspect the current diff and the latest concrete failure, "
+ "then commit to one targeted change and one narrow verification. Do not "
+ "restart localization or broaden the search."
+ )
+ if reason == "medium-effort-envelope-rescue":
+ return (
+ "Automated effort checkpoint: this task has crossed the recent resolved-task "
+ "tool envelope at medium reasoning effort. Continue this exact thread and "
+ "preserve the working tree; reasoning effort is now high. Inspect the current "
+ "diff and strongest local evidence, commit to the most likely remaining "
+ "hypothesis, and run the narrowest verification that can confirm it. Do not "
+ "restart localization or discard a justified partial patch."
+ )
+ if reason == "very-wide-patch-still-failing":
+ return (
+ "Automated process checkpoint: the current patch spans at least fifteen "
+ "files, and verification after the latest edit failed. Preserve the current "
+ "working tree and continue this same task. Before expanding the patch, inspect "
+ "`git diff --stat` and the latest failure, then run the narrowest diagnostic "
+ "that can distinguish a necessary cross-cutting change from an over-broad "
+ "port. Keep the breadth if local evidence requires it; otherwise reduce scope "
+ "and fix the concrete failure. Before finishing, re-read the issue and verify "
+ "every affected public entry point."
+ )
+ if reason == "wide-edit-scope-needs-verification":
+ return (
+ "Automated process checkpoint: the current patch now spans many files without "
+ "a successful verification after the latest edit. Preserve the current working "
+ "tree and do not restart localization. Pause further scope expansion, inspect "
+ "`git diff --stat`, and run the narrowest relevant verification now. Use that "
+ "result to decide whether every touched file is necessary; continue the same "
+ "implementation if the breadth is justified."
+ )
+ if reason == "validated-patch-reopened-research":
+ return (
+ "Automated process checkpoint: a verification passed after the latest edit, but "
+ "the session has returned to repository research. Preserve the validated patch "
+ "and do not restart localization. Inspect `git diff`, run only the narrowest "
+ "remaining final check needed for confidence, and finish if it stays green. "
+ "Reopen the implementation only if that check exposes a concrete failure."
+ )
+ focus = {
+ "repeated-failing-verification":
+ "The same verification failure is recurring after attempted fixes.",
+ "scattered-edits-still-failing":
+ "The patch has spread across many files and verification is still failing.",
+ "edit-churn-still-failing":
+ "Repeated edits have not changed the verification outcome.",
+ "repeated-command-without-progress":
+ "The same command pattern is recurring without a successful verification.",
+ "blocked-retrieval-without-progress":
+ "Blocked external retrieval is recurring without advancing the local fix.",
+ }.get(reason, "The recent process is repeating without a successful verification.")
+ return (
+ "Automated process check: " + focus + " Continue this same task and preserve the "
+ "current working tree, but switch to a materially different hypothesis. Inspect "
+ "`git diff` and the latest failing verification first; prefer local repository "
+ "evidence over retrying blocked external retrieval. Do not restart localization "
+ "from scratch. Once a targeted verification passes after the latest edit, finish "
+ "with the narrowest sufficient final check."
+ )
+
+
+def completion_audit_text(reason: str) -> str:
+ evidence = {
+ "claimed-already-fixed":
+ "The previous response claimed the issue was already fixed but produced no patch.",
+ "empty-patch":
+ "The previous response completed without a source patch.",
+ "latest-verification-failed":
+ "The latest relevant verification still failed after the newest edit.",
+ }[reason]
+ return (
+ COMPLETION_AUDIT_MARKER + "\n"
+ "Automated completion audit: " + evidence + " The current checkout is the "
+ "failing base for this benchmark; preserve any current work and do not assume the "
+ "issue is already fixed. Re-read the issue and `git diff`, produce the smallest "
+ "nonempty source patch, check every affected public entry point or call path, and "
+ "run the narrowest focused reproduction after the latest edit. Do not edit tests. "
+ "This continuation is capped, so prioritize the minimal fix and verification."
+ )
+
+
+class CodexResumeGuard:
+ """Live one-shot detector that continues only through native Codex resume."""
+
+ def __init__(self, policy: CodexSpiralPolicy = DEFAULT_POLICY, *, verbose: bool = False,
+ completion_audit: bool = False,
+ completion_audit_tool_turns: int = 8):
+ self.policy = policy
+ self.verbose = verbose
+ self.state = CodexSpiralState(policy.recent_window)
+ self.nudges_done = 0
+ self.pending_reason: Optional[str] = None
+ self.fired_reasons: set[str] = set() # one nudge per reason per task
+ self.events: list[dict] = []
+ self.completion_audit_enabled = bool(completion_audit)
+ self.completion_audit_tool_turns = max(1, int(completion_audit_tool_turns))
+ self.completion_audits_done = 0
+ self.completion_audit_reason: Optional[str] = None
+ self.completion_audit_active = False
+ self.completion_audit_start_turn = 0
+ self.completion_audit_capped = False
+
+ def monitor(self, line: str) -> bool:
+ events = parse_events(line)
+ if not events:
+ return False
+ for event in events:
+ self.state.update(event)
+ event = events[-1]
+ if self.completion_audit_active:
+ used = self.state.turns - self.completion_audit_start_turn
+ if (event.kind in ("command", "edit")
+ and used >= self.completion_audit_tool_turns):
+ self.completion_audit_active = False
+ self.completion_audit_capped = True
+ self.events.append({
+ "action": "completion-audit-cap",
+ "reason": self.completion_audit_reason,
+ "tool_turns": used,
+ "snapshot": self.state.snapshot(),
+ })
+ if self.verbose:
+ print(f" [codex-completion-audit] CAP at {used} tool turns")
+ return True
+ # Do not stack another spiral intervention inside the final audit.
+ return False
+ if event.kind == "thread" or self.nudges_done >= self.policy.nudge_budget:
+ return False
+ reason = self.state.decision(self.policy)
+ # A missing thread id must never degrade into the harmful fresh-session path.
+ # A reason that already produced a nudge on this task does not get a second one:
+ # the budget buys DIFFERENT interventions, never the same one twice.
+ if not reason or not self.state.thread_id or reason in self.fired_reasons:
+ return False
+ self.fired_reasons.add(reason)
+ self.pending_reason = reason
+ record = {
+ "action": "resume-nudge",
+ "reason": reason,
+ "thread_id": self.state.thread_id,
+ "snapshot": self.state.snapshot(),
+ }
+ self.events.append(record)
+ if self.verbose:
+ print(f" [codex-spiral] RESUME-NUDGE at tool turn {self.state.turns} "
+ f"({reason}) {record['snapshot']}")
+ return True
+
+ @property
+ def segment(self) -> int:
+ """Segments this guard has opened beyond the first, i.e. nudges issued.
+
+ The SHARED task loop (`te_harness.swebench` run_task) logs `guard.segment` right
+ after a continuation, and the generic `spiral.NudgeGuard` exposes it -- so a Codex
+ guard without it crashes the lane on its FIRST fire, which is exactly what happened
+ on 2026-09-12 (the sol-era nudges ran on the other harness, so nothing had ever
+ exercised this path here). `nudges_done` carries the same count and is incremented
+ in `continuation()` before the log line reads it, so the two agree by construction.
+ `tests/test_spiral.py` pins the whole attribute contract, not just this one name.
+ """
+ return self.nudges_done
+
+ def continuation(self):
+ """Return the explicit resume request consumed by ``run_agent.run_agent``."""
+ if not self.pending_reason or not self.state.thread_id:
+ return None
+ reason = self.pending_reason
+ self.pending_reason = None
+ self.nudges_done += 1
+ request = {
+ "mode": "resume",
+ "session_id": self.state.thread_id,
+ "prompt": nudge_text(reason),
+ }
+ if self.policy.resume_effort:
+ request["effort"] = self.policy.resume_effort
+ return request
+
+ def completion_continuation(self, patch: str, transcript: str):
+ """Return one deterministic same-thread end-state audit, if warranted."""
+ if (not self.completion_audit_enabled or self.completion_audits_done
+ or not self.state.thread_id):
+ return None
+ patch_empty = not (patch or "").strip()
+ final_message = final_agent_message(transcript)
+ if patch_empty and _NO_CHANGE_CLAIM_RE.search(final_message):
+ reason = "claimed-already-fixed"
+ elif patch_empty:
+ reason = "empty-patch"
+ elif (self.state.last_failed_verification_turn > self.state.last_edit_turn
+ and not self.state.currently_green):
+ reason = "latest-verification-failed"
+ else:
+ return None
+
+ self.completion_audits_done = 1
+ self.completion_audit_reason = reason
+ self.completion_audit_active = True
+ self.completion_audit_start_turn = self.state.turns
+ record = {
+ "action": "completion-audit-resume",
+ "reason": reason,
+ "thread_id": self.state.thread_id,
+ "snapshot": self.state.snapshot(),
+ }
+ self.events.append(record)
+ if self.verbose:
+ print(f" [codex-completion-audit] RESUME ({reason})")
+ return {
+ "mode": "resume",
+ "session_id": self.state.thread_id,
+ "prompt": completion_audit_text(reason),
+ }
+
+ def finish_completion_audit(self):
+ """Record a naturally completed audit (a capped one already records itself)."""
+ if not self.completion_audit_active:
+ return
+ used = self.state.turns - self.completion_audit_start_turn
+ self.completion_audit_active = False
+ self.events.append({
+ "action": "completion-audit-complete",
+ "reason": self.completion_audit_reason,
+ "tool_turns": used,
+ "snapshot": self.state.snapshot(),
+ })
+
+ def telemetry(self) -> dict:
+ return {
+ "policy": asdict(self.policy),
+ "nudges": self.nudges_done,
+ "events": list(self.events),
+ "snapshot": self.state.snapshot(),
+ "thread_id_captured": bool(self.state.thread_id),
+ "paid_judge_effective_tokens": 0,
+ "completion_audit_enabled": self.completion_audit_enabled,
+ "completion_audits": self.completion_audits_done,
+ "completion_audit_reason": self.completion_audit_reason,
+ "completion_audit_capped": self.completion_audit_capped,
+ "completion_audit_tool_turn_cap": self.completion_audit_tool_turns,
+ "resume_effort": self.policy.resume_effort or None,
+ }
diff --git a/tokeneconomizer/mechanisms/codex_untool.py b/tokeneconomizer/mechanisms/codex_untool.py
new file mode 100644
index 0000000..6ad45d8
--- /dev/null
+++ b/tokeneconomizer/mechanisms/codex_untool.py
@@ -0,0 +1,628 @@
+"""Codex-native capability untooling.
+
+Codex ships a broad default tool/instruction surface (collaboration sub-agents, web,
+skills, image/media, request-user-input, deferred wait, planning, ...). On a sealed,
+detached, network-isolated, single-agent SWE-bench task most of that surface is
+inapplicable, yet its schemas + guidance are re-sent on every model turn and billed each
+time. This controller removes the inapplicable packages from the request contract.
+
+DESIGN (causal, evidence-gated, resolve-safe):
+
+ * A fresh repository stream starts on the COMPLETE contract (task 1 observes everything).
+ * After each completed task, at most two independently-observed packages are demoted,
+ using ONLY prior completed-task evidence (never the current task, never the grade at
+ selection time). A single completed use PERMANENTLY protects a package.
+ * Higher-risk packages (image/media, skills) require more loaded-but-unused exposures
+ before demotion than collaboration/web.
+ * Narrow issue-text hints and sparse single-package probes restore one package before a
+ task without restoring the rest of the contract (fail-open recovery).
+ * Deferred `wait` is treated as CORE, not demotable: live traces established its
+ indirect utility (pruning it triggered dozens of shell-polling commands), so recent
+ experience keeps it loaded -- the key correction the shipped controller makes over its
+ predecessor. The mask is frozen within a task; grades/classifiers/reasoning-effort/
+ prefix-injection/output-compaction/spiral do not participate.
+
+This is the single, un-versioned successor to
+the codex-untool line (v1 classifier -> ... -> v5 sealed atomic packages -> v6
+recent-experience gradual -> v7 indirect-use core).
+
+WHAT THE MASK CAN REACH ON THE CURRENT HARNESS (te_harness, codex-cli 0.153.1, config
+toggles only -- no relay; recorded from the CLI's own request against a local endpoint). The
+packages a supported toggle removes are `collaboration` (the agent tools plus two
+collaboration messages), `goals` (inside the code-mode `exec` description), `image_media`
+and `skills` (the bundled system skills' listing, removed by disabling each bundled skill in
+config). `web` is already blocked on every arm; `request_user_input` has no toggle in
+0.153.1 and its demotion changes nothing there; `planning` is not in this contract at all.
+
+The ToolingStage computes the logical capability MASK here; the concrete mapping from a
+capability id to Codex config/relay prune-items is the runner's job (it owns the Codex
+request contract), so this module stays free of any Docker/CLI harness dependency.
+"""
+import copy
+import hashlib
+import json
+import os
+import re
+from collections import Counter
+
+
+# ===========================================================================
+# Transcript capability detection (which Codex packages a trace item exercises)
+# ===========================================================================
+CAPABILITY_DESCRIPTIONS = {
+ "collaboration": (
+ "Spawn, message, wait for, or close sub-agents. The benchmark runs one sealed "
+ "agent process and provides no useful parallel-agent workflow."),
+ "goals": (
+ "Create, inspect, and update long-lived product goals. These are orchestration "
+ "metadata rather than repository read/edit/test operations."),
+ "image_media": (
+ "View or transport image/audio/generated-media blocks. SWE-bench repair tasks "
+ "operate on a checked-out source repository and text test output."),
+ "interactive_guidance": (
+ "Persona, writing-style, commentary-channel and answer-formatting guidance for an "
+ "interactive chat session. A detached, single-turn run has no user reading "
+ "commentary and its final message is not graded."),
+ "planning": (
+ "Maintain a structured todo/update_plan list. This can help some long repairs but "
+ "is optional because the agent can reason and report progress without the tool."),
+ "web": (
+ "Search/open the public web and query online services. The task container is "
+ "network-isolated; empty or uncompleted attempts are not useful evidence."),
+ "request_user_input": (
+ "Pause for interactive human input. Runs are detached, autonomous, and have no "
+ "human available inside a task."),
+}
+
+CAPABILITIES = tuple(sorted(CAPABILITY_DESCRIPTIONS))
+
+_ITEM_TYPES_BY_CAPABILITY = {
+ "collaboration": frozenset((
+ "spawn_agent", "send_input", "send_message", "followup_task",
+ "resume_agent", "wait_agent", "close_agent", "interrupt_agent", "list_agents")),
+ "goals": frozenset(("create_goal", "get_goal", "update_goal")),
+ "image_media": frozenset((
+ "view_image", "image", "audio", "generated_image", "image_generation")),
+ "planning": frozenset(("update_plan", "todo_list")),
+ "web": frozenset(("web", "web_search", "web_fetch", "web__run")),
+ "request_user_input": frozenset(("request_user_input",)),
+}
+
+_NAME_TO_CAPABILITY = {
+ name: capability
+ for capability, names in _ITEM_TYPES_BY_CAPABILITY.items()
+ for name in names
+}
+_NESTED_TOOL_RE = re.compile(r"\btools\.([A-Za-z_][A-Za-z0-9_]*)\s*\(")
+
+
+def _item_capabilities(item):
+ """Return capability groups represented by one Codex transcript item."""
+ if not isinstance(item, dict):
+ return set()
+ found = set()
+ for key in ("type", "name", "tool_name"):
+ value = str(item.get(key) or "")
+ capability = _NAME_TO_CAPABILITY.get(value)
+ if capability:
+ found.add(capability)
+ # Newer Codex builds package multi-agent operations as one collaboration item; the
+ # concrete action is in ``tool`` (e.g. tool="wait"), not name/tool_name. Treat the
+ # package type itself as collaboration (mapping a bare tool="wait" globally would
+ # misclassify the unrelated deferred-wait tool).
+ if str(item.get("type") or "") == "collab_tool_call":
+ found.add("collaboration")
+ # Some builds expose a unified custom `exec` item rather than normalizing the nested
+ # helper into web_search/todo_list/etc. Inspect only custom-tool payloads; shell text
+ # may legitimately contain strings like "update_plan" and must not be false evidence.
+ kind = str(item.get("type") or "")
+ name = str(item.get("name") or "")
+ if kind in ("custom_tool_call", "function_call", "tool_call") or name == "exec":
+ for key in ("input", "arguments"):
+ raw = item.get(key)
+ if isinstance(raw, dict):
+ raw = json.dumps(raw, sort_keys=True)
+ if not isinstance(raw, str):
+ continue
+ for nested_name in _NESTED_TOOL_RE.findall(raw):
+ capability = _NAME_TO_CAPABILITY.get(nested_name)
+ if capability:
+ found.add(capability)
+ return found
+
+
+# ===========================================================================
+# Narrow issue-text hints (fail-open protection for image/skill packages)
+# ===========================================================================
+_IMAGE_HINT = re.compile(
+ r"\b(?:screenshot|png|jpe?g|gif|svg|audio|video)\b|"
+ r"\b(?:image|figure)\s+(?:file|output|render|display)\b",
+ re.I,
+)
+_SKILL_HINT = re.compile(
+ r"\b(?:OpenAI|ChatGPT|Codex|Responses API)\b|"
+ r"\b(?:generate|create|edit|transform)\s+(?:an?\s+)?"
+ r"(?:image|photo|illustration|texture|sprite|mockup)\b|"
+ r"\b(?:create|update|install|scaffold)\s+(?:an?\s+)?"
+ r"(?:Codex\s+)?(?:skill|plugin)\b|"
+ r"\bSKILL\.md\b|\.codex-plugin\b",
+ re.I,
+)
+_SKILL_USE = re.compile(
+ r"(?:\bSKILL\.md\b|[/\\]skills[/\\]|\bskills\.read\s*\()",
+ re.I,
+)
+_FAILED_STATUSES = frozenset(("failed", "error", "cancelled"))
+
+_HINT_BASE_RELAY = (
+ "deferred_wait", "image_media", "request_user_input", "skills", "sealed_web")
+
+
+def task_hints(problem_statement):
+ """Return the packages a task's issue text protects from demotion (image/skills)."""
+ text = str(problem_statement or "")
+ protected = []
+ if _IMAGE_HINT.search(text):
+ protected.append("image_media")
+ if _SKILL_HINT.search(text):
+ protected.append("skills")
+ return protected
+
+
+# ===========================================================================
+# Trace audit: attempted / completed / failed capability uses in a Codex trace
+# ===========================================================================
+_AUDIT_CAPABILITIES = (
+ "collaboration", "deferred_wait", "goals", "image_media", "interactive_guidance",
+ "request_user_input", "skills", "web")
+
+
+def _wire_text(item):
+ values = []
+ if not isinstance(item, dict):
+ return ""
+ for key in ("input", "arguments", "command", "text"):
+ value = item.get(key)
+ if isinstance(value, str):
+ values.append(value)
+ elif isinstance(value, dict):
+ values.append(json.dumps(value, sort_keys=True))
+ return "\n".join(values)
+
+
+def _audit_capabilities(item):
+ found = set(_item_capabilities(item))
+ if not isinstance(item, dict):
+ return found
+ kind = str(item.get("type") or "")
+ name = str(item.get("name") or item.get("tool_name") or "")
+ if name == "wait" or kind == "wait":
+ found.add("deferred_wait")
+ if _SKILL_USE.search(_wire_text(item)):
+ found.add("skills")
+ return found
+
+
+def inspect_usage(transcript_path):
+ """Audit capability leakage in a Codex trace, de-duplicated by transcript item id."""
+ capabilities = _AUDIT_CAPABILITIES
+ attempted = {capability: set() for capability in capabilities}
+ completed = {capability: set() for capability in capabilities}
+ failed = {capability: set() for capability in capabilities}
+ tools = Counter()
+ seen_tools = set()
+ reconnects = 0
+ try:
+ handle = open(transcript_path, encoding="utf-8", errors="replace")
+ except OSError:
+ handle = None
+ if handle is not None:
+ with handle:
+ for line_number, line in enumerate(handle, 1):
+ try:
+ event = json.loads(line)
+ except Exception:
+ continue
+ phase = str(event.get("type") or "")
+ if phase == "error" and "reconnecting" in str(
+ event.get("message") or event.get("error") or "").lower():
+ reconnects += 1
+ if phase not in ("item.started", "item.completed"):
+ continue
+ item = event.get("item") or {}
+ represented = _audit_capabilities(item) & set(capabilities)
+ if not represented:
+ continue
+ item_id = str(item.get("id") or item.get("call_id") or line_number)
+ status = str(item.get("status") or "").lower()
+ label = str(
+ item.get("name") or item.get("tool_name")
+ or item.get("tool") or item.get("type") or "unknown")
+ if item_id not in seen_tools:
+ tools[label] += 1
+ seen_tools.add(item_id)
+ for capability in represented:
+ attempted[capability].add(item_id)
+ if phase == "item.completed":
+ if status in _FAILED_STATUSES:
+ failed[capability].add(item_id)
+ else:
+ completed[capability].add(item_id)
+ return {
+ "attempted": {c: len(attempted[c]) for c in capabilities},
+ "completed": {c: len(completed[c]) for c in capabilities},
+ "failed": {c: len(failed[c]) for c in capabilities},
+ "tools": dict(sorted(tools.items())),
+ "reconnects": reconnects,
+ }
+
+
+# ===========================================================================
+# The recent-experience controller.
+#
+# `_EVIDENCE_CANDIDATES` is the full set of packages whose exposures the state machine
+# TRACKS (incl. deferred_wait, so its protection status is still recorded). The public
+# controller (CodexUntoolProfile) treats deferred_wait as CORE and only ever DEMOTES the
+# `CANDIDATE_CAPABILITIES` subset -- the shipped indirect-use-core behavior.
+# ===========================================================================
+_EVIDENCE_CANDIDATES = (
+ "collaboration", "web", "goals", "interactive_guidance", "skills", "request_user_input",
+ "image_media", "deferred_wait")
+CORE_CAPABILITIES = ("deferred_wait",)
+CANDIDATE_CAPABILITIES = tuple(
+ c for c in _EVIDENCE_CANDIDATES if c not in CORE_CAPABILITIES)
+
+MIN_LOADED_EXPOSURES = {
+ "collaboration": 1,
+ "web": 1,
+ # Product-goal metadata (create/get/update_goal): orchestration state like
+ # collaboration, structurally inapplicable to a sealed single-task repair, ~0.7k tokens
+ # of the code-mode exec description. 0 uses in 243 gpt-5.6-terra baseline tasks.
+ "goals": 1,
+ # Chat-session guidance in the base prompt (persona, commentary channel, formatting):
+ # structurally inapplicable to a detached run, like collaboration. Removed through a
+ # lean copy of the base prompt when the harness has a capture for the CLI build.
+ "interactive_guidance": 1,
+ "skills": 2,
+ "request_user_input": 2,
+ "image_media": 3,
+}
+MAX_DEMOTIONS_PER_TASK = 2
+RECENT_WINDOW = 3
+
+# Native config carries a supported toggle for these; web/image are also relay-pruned
+# because current Codex builds serialize their real declarations inside the unified exec
+# package (their native toggles are harmless defense in depth).
+# `skills` (2026-09-11): the CLI bundles system skills into $CODEX_HOME at startup and lists
+# them in a <skills_instructions> developer message (~0.64k tokens); disabling each bundled
+# skill through `[[skills.config]] enabled = false` removes the whole block. Its relay id is
+# kept for harnesses that prune by request rewrite.
+CONFIG_CAPABILITIES = frozenset(("collaboration", "web", "goals", "skills", "image_media",
+ "interactive_guidance"))
+_RELAY_CAPABILITIES = {
+ "collaboration": (),
+ "web": ("sealed_web",),
+ "goals": (),
+ "interactive_guidance": (),
+ "skills": ("skills",),
+ "request_user_input": ("request_user_input",),
+ "image_media": ("image_media",),
+ "deferred_wait": ("deferred_wait",),
+}
+
+FIRST_PROBE_AT = 24
+INITIAL_PROBE_GAP = 32
+PROBE_PRIORITY = (
+ "image_media", "skills", "request_user_input", "web", "collaboration", "goals",
+ "interactive_guidance")
+
+PROFILE_VERSION = "codex-recent-experience-untool-indirect-use-core"
+RETAINED_CAPABILITIES = ("apply_patch", "exec_command", "planning", "write_stdin")
+
+
+def relay_capabilities(logical_capabilities):
+ """Expand logical packages into relay rewrite capability ids."""
+ expanded = []
+ for capability in logical_capabilities or ():
+ expanded.extend(_RELAY_CAPABILITIES.get(capability, ()))
+ return sorted(set(expanded))
+
+
+def _contract_fingerprint():
+ return hashlib.sha256(json.dumps({
+ "profile": PROFILE_VERSION,
+ "evidence_candidates": _EVIDENCE_CANDIDATES,
+ "candidates": CANDIDATE_CAPABILITIES,
+ "core_capabilities": CORE_CAPABILITIES,
+ "min_loaded_exposures": MIN_LOADED_EXPOSURES,
+ "max_demotions_per_task": MAX_DEMOTIONS_PER_TASK,
+ "recent_window": RECENT_WINDOW,
+ "config_capabilities": sorted(CONFIG_CAPABILITIES),
+ "relay_capabilities": _RELAY_CAPABILITIES,
+ "first_probe_at": FIRST_PROBE_AT,
+ "initial_probe_gap": INITIAL_PROBE_GAP,
+ "probe_priority": PROBE_PRIORITY,
+ }, sort_keys=True).encode("utf-8")).hexdigest()
+
+
+CONTRACT_FINGERPRINT = _contract_fingerprint()
+
+
+class CodexUntoolProfile:
+ """Persistent, repository-scoped, causal full-to-pruned Codex untool controller.
+
+ Usage per task (the ToolingStage drives this):
+ p = CodexUntoolProfile(repo, results_root, fresh=...)
+ sel = p.begin_task(iid, problem_statement=...) # frozen capability mask
+ ... run the task with sel['config_capabilities'] / sel['relay_capabilities'] ...
+ rec = p.observe(inspect_usage(transcript), resolved=...) # fold into state
+ """
+
+ def __init__(self, repo, results_root, *, fresh=False):
+ self.repo = repo
+ state_dir = os.path.join(
+ os.path.abspath(results_root), "interface_state",
+ repo.replace("/", "__"))
+ self.path = os.path.join(state_dir, "codex_untool.json")
+ self._pending = None
+ self.state = self._new_state()
+ if not fresh:
+ self._load()
+
+ # ---- state ------------------------------------------------------------
+ def _new_state(self):
+ zero = {c: 0 for c in _EVIDENCE_CANDIDATES}
+ return {
+ "version": PROFILE_VERSION,
+ "contract_fingerprint": CONTRACT_FINGERPRINT,
+ "repo": self.repo,
+ "tasks_completed": 0,
+ "current_pruned": [],
+ "ever_completed": [],
+ "core_capabilities": list(CORE_CAPABILITIES),
+ "attempted_total": dict(zero),
+ "completed_total": dict(zero),
+ "failed_total": dict(zero),
+ "loaded_exposures": dict(zero),
+ "incomplete_attempt_exposures": dict(zero),
+ "zero_attempt_exposures": dict(zero),
+ "recent_loaded_outcomes": {c: [] for c in _EVIDENCE_CANDIDATES},
+ "next_probe_at": FIRST_PROBE_AT,
+ "probe_gap": INITIAL_PROBE_GAP,
+ "probe_cursor": 0,
+ "transitions": [],
+ "history": [],
+ }
+
+ def _load(self):
+ try:
+ loaded = json.load(open(self.path, encoding="utf-8"))
+ except (OSError, ValueError, TypeError):
+ return
+ if (loaded.get("version") != PROFILE_VERSION
+ or loaded.get("contract_fingerprint") != CONTRACT_FINGERPRINT
+ or loaded.get("repo") != self.repo):
+ return
+ state = self._new_state()
+ state.update(loaded)
+ for field in ("attempted_total", "completed_total", "failed_total",
+ "loaded_exposures", "incomplete_attempt_exposures",
+ "zero_attempt_exposures"):
+ merged = {c: 0 for c in _EVIDENCE_CANDIDATES}
+ merged.update({c: int(v or 0)
+ for c, v in (state.get(field) or {}).items()
+ if c in _EVIDENCE_CANDIDATES})
+ state[field] = merged
+ recent = {c: [] for c in _EVIDENCE_CANDIDATES}
+ for capability, outcomes in (
+ state.get("recent_loaded_outcomes") or {}).items():
+ if capability not in recent:
+ continue
+ recent[capability] = [
+ str(v) for v in (outcomes or ())
+ if v in ("completed", "incomplete_attempt", "unused")
+ ][-RECENT_WINDOW:]
+ state["recent_loaded_outcomes"] = recent
+ state["current_pruned"] = sorted(
+ set(state.get("current_pruned") or ()) & set(CANDIDATE_CAPABILITIES))
+ state["ever_completed"] = sorted(
+ set(state.get("ever_completed") or ()) & set(_EVIDENCE_CANDIDATES))
+ state["core_capabilities"] = list(CORE_CAPABILITIES)
+ self.state = state
+
+ def save(self):
+ os.makedirs(os.path.dirname(self.path), exist_ok=True)
+ temporary = self.path + ".tmp"
+ with open(temporary, "w", encoding="utf-8") as handle:
+ json.dump(self.state, handle, indent=2, sort_keys=True)
+ os.replace(temporary, self.path)
+
+ # ---- probe / demotion policy -----------------------------------------
+ def _choose_probe(self, applied_pruned, hints):
+ completed = int(self.state.get("tasks_completed", 0) or 0)
+ if completed < int(self.state.get("next_probe_at", FIRST_PROBE_AT)):
+ return None
+ eligible = set(applied_pruned) - set(hints)
+ if not eligible:
+ return None
+ start = int(self.state.get("probe_cursor", 0) or 0)
+ for offset in range(len(PROBE_PRIORITY)):
+ capability = PROBE_PRIORITY[(start + offset) % len(PROBE_PRIORITY)]
+ if capability in eligible:
+ return capability
+ return None
+
+ def _eligible_demotions(self, pruned, protected):
+ """Deferred_wait is CORE -> only CANDIDATE_CAPABILITIES are ever demotable."""
+ eligible = []
+ recent = self.state.get("recent_loaded_outcomes") or {}
+ for capability in CANDIDATE_CAPABILITIES:
+ if capability in pruned or capability in protected:
+ continue
+ needed = MIN_LOADED_EXPOSURES[capability]
+ outcomes = list(recent.get(capability) or ())
+ if len(outcomes) >= needed and "completed" not in outcomes[-needed:]:
+ eligible.append(capability)
+ return eligible[:MAX_DEMOTIONS_PER_TASK]
+
+ # ---- per-task selection ----------------------------------------------
+ def begin_task(self, task_id, *, problem_statement=""):
+ """Freeze a mask using ONLY state from already-completed tasks."""
+ completed = int(self.state.get("tasks_completed", 0) or 0)
+ hints = task_hints(problem_statement)
+ logical = set(self.state.get("current_pruned") or ()) - set(hints)
+ probe = self._choose_probe(logical, hints)
+ if probe:
+ logical.discard(probe)
+ logical = sorted(logical)
+ # deferred_wait is core: it must never appear in a pruned mask.
+ assert not (set(logical) & set(CORE_CAPABILITIES)), \
+ "attempted to prune an indirect-use core package"
+ if completed == 0:
+ reason = "observe-complete-contract"
+ elif probe:
+ reason = f"single-package-probe:{probe}"
+ elif hints:
+ reason = "task-hint:" + ",".join(sorted(hints))
+ else:
+ reason = "recent-experience-indirect-use-core-mask"
+ retained = sorted(set(RETAINED_CAPABILITIES) | set(CORE_CAPABILITIES))
+ self._pending = {
+ "task_id": str(task_id),
+ "profile": PROFILE_VERSION,
+ "audit": completed == 0,
+ "reason": reason,
+ "logical_capabilities": logical,
+ "config_capabilities": sorted(set(logical) & CONFIG_CAPABILITIES),
+ "relay_capabilities": relay_capabilities(logical),
+ "task_hints": sorted(hints),
+ "probe_capability": probe,
+ "probe_reason": "evidence-aging" if probe else None,
+ "tasks_completed_before": completed,
+ "core_capabilities": list(CORE_CAPABILITIES),
+ "retained_capabilities": retained,
+ }
+ return copy.deepcopy(self._pending)
+
+ def _transition(self, kind, capabilities, reason, task_id):
+ if not capabilities:
+ return
+ transitions = list(self.state.get("transitions") or ())
+ transitions.append({
+ "after_tasks": int(self.state.get("tasks_completed", 0) or 0),
+ "task_id": task_id,
+ "kind": kind,
+ "capabilities": sorted(capabilities),
+ "reason": reason,
+ })
+ self.state["transitions"] = transitions[-128:]
+
+ # ---- fold one completed trace into state (affects the NEXT task only) --
+ def observe(self, usage, *, resolved=None):
+ pending = self._pending
+ if pending is None:
+ raise RuntimeError("observe() called without begin_task()")
+ attempted = usage.get("attempted") or {}
+ completed = usage.get("completed") or {}
+ failed = usage.get("failed") or {}
+ logical_pruned = set(pending.get("logical_capabilities") or ())
+ loaded = set(_EVIDENCE_CANDIDATES) - logical_pruned
+ current_pruned = set(self.state.get("current_pruned") or ())
+ protected = set(self.state.get("ever_completed") or ())
+ completed_now = set()
+
+ for capability in _EVIDENCE_CANDIDATES:
+ attempted_n = int(attempted.get(capability, 0) or 0)
+ completed_n = int(completed.get(capability, 0) or 0)
+ failed_n = int(failed.get(capability, 0) or 0)
+ self.state["attempted_total"][capability] += attempted_n
+ self.state["completed_total"][capability] += completed_n
+ self.state["failed_total"][capability] += failed_n
+ if completed_n > 0:
+ protected.add(capability)
+ current_pruned.discard(capability)
+ completed_now.add(capability)
+ if capability not in loaded:
+ continue
+ self.state["loaded_exposures"][capability] += 1
+ if completed_n > 0:
+ outcome = "completed"
+ elif attempted_n > 0:
+ outcome = "incomplete_attempt"
+ self.state["incomplete_attempt_exposures"][capability] += 1
+ else:
+ outcome = "unused"
+ self.state["zero_attempt_exposures"][capability] += 1
+ outcomes = list(self.state["recent_loaded_outcomes"].get(capability) or ())
+ outcomes.append(outcome)
+ self.state["recent_loaded_outcomes"][capability] = outcomes[-RECENT_WINDOW:]
+
+ probe = pending.get("probe_capability")
+ if probe:
+ gap = max(1, int(self.state.get(
+ "probe_gap", INITIAL_PROBE_GAP) or INITIAL_PROBE_GAP))
+ self.state["next_probe_at"] = int(
+ self.state.get("next_probe_at", FIRST_PROBE_AT)) + gap
+ self.state["probe_gap"] = gap * 2
+ self.state["probe_cursor"] = (
+ (PROBE_PRIORITY.index(probe) + 1) % len(PROBE_PRIORITY))
+ if probe not in protected:
+ current_pruned.add(probe)
+ self._transition("probe-reprune", [probe],
+ "targeted-probe-produced-no-completed-use",
+ pending.get("task_id"))
+ else:
+ self._transition("probe-protect", [probe],
+ "targeted-probe-produced-completed-use",
+ pending.get("task_id"))
+
+ newly_pruned = self._eligible_demotions(current_pruned, protected)
+ current_pruned.update(newly_pruned)
+ current_pruned.difference_update(protected)
+ current_pruned &= set(CANDIDATE_CAPABILITIES) # core never pruned
+ before = int(self.state.get("tasks_completed", 0) or 0)
+ self.state["tasks_completed"] = before + 1
+ self.state["current_pruned"] = sorted(current_pruned)
+ self.state["ever_completed"] = sorted(protected)
+ self._transition("gradual-demotion", newly_pruned,
+ "recent-loaded-exposures-with-zero-completed-use",
+ pending.get("task_id"))
+ self._transition("completed-protect", completed_now,
+ "completed-use-permanently-protected",
+ pending.get("task_id"))
+
+ record = {
+ "task_id": pending.get("task_id"),
+ "audit": bool(pending.get("audit")),
+ "reason": pending.get("reason"),
+ "logical_capabilities": sorted(logical_pruned),
+ "config_capabilities": list(pending.get("config_capabilities") or ()),
+ "relay_capabilities": list(pending.get("relay_capabilities") or ()),
+ "task_hints": list(pending.get("task_hints") or ()),
+ "probe_capability": probe,
+ "probe_reason": pending.get("probe_reason"),
+ "resolved": resolved,
+ "loaded_capabilities": sorted(loaded),
+ "attempted": {c: int(attempted.get(c, 0) or 0)
+ for c in _EVIDENCE_CANDIDATES},
+ "completed": {c: int(completed.get(c, 0) or 0)
+ for c in _EVIDENCE_CANDIDATES},
+ "failed": {c: int(failed.get(c, 0) or 0)
+ for c in _EVIDENCE_CANDIDATES},
+ "newly_pruned": sorted(newly_pruned),
+ "mask_after": list(self.state["current_pruned"]),
+ }
+ history = list(self.state.get("history") or ())
+ history.append(record)
+ self.state["history"] = history[-256:]
+ self.save()
+ self._pending = None
+ return record
+
+ def debug_state(self):
+ state = copy.deepcopy(self.state)
+ state["next_logical_mask"] = list(state.get("current_pruned") or ())
+ state["next_relay_mask"] = relay_capabilities(state.get("current_pruned") or ())
+ state["core_capabilities"] = list(CORE_CAPABILITIES)
+ state["state_path"] = self.path
+ return state
diff --git a/tokeneconomizer/mechanisms/envfacts.py b/tokeneconomizer/mechanisms/envfacts.py
new file mode 100644
index 0000000..5b84a32
--- /dev/null
+++ b/tokeneconomizer/mechanisms/envfacts.py
@@ -0,0 +1,809 @@
+"""Environment facts: the LLM-free injection layer shared by every lane.
+
+WHAT IT DOES. As a repository's task stream runs, this controller reads each finished
+rollout through the one trace seam and looks for RECURRING SHAPES OF EVIDENCE that an
+agent re-discovers the same environment fact by failing at it, task after task. When a
+shape recurs across drifted tasks it becomes a one-line, declarative fact that is injected
+before the next task through the instruction channel (the prompt prefix on every lane). It
+runs alone (profile `envfacts`, the Codex lane's default) or underneath the paid proposer
+(profile `envfacts+distiller`), where its claims are handed to the proposer as
+already-stated so the paid budget is not spent rediscovering them.
+
+THE DESIGN RULE. The detectors describe shapes, never a harness: nothing here names a
+path, an environment manager, a repository or a benchmark. Every specific string in a
+rendered fact -- an interpreter path, a runner form, a refused host -- is a PARAMETER learned
+from the traces. The same code must form the interpreter fact on a harness whose shell is
+not activated, form nothing on one that is, and form the "history ends at HEAD" fact only
+where the probes really return nothing. The WORDS that name intents (which executables are
+interpreters, runners, package tools, fetchers, probes) are a `Vocabulary` the domain
+adapter supplies, so the same families read a Python/git repository and a Node/npm one.
+
+THE FAMILIES.
+
+ FormRepair in one task, invocation form A of an intent (run the interpreter, run the
+ tests, use the package tool) fails with an infrastructure error, and a later
+ form B of the same intent succeeds. Fact: "`A ...` fails here with <error>;
+ `B ...` works." Retires when a bare form runs the test runner successfully.
+ Boundary every call of an operation class (fetching from a host) fails with one
+ environmental error class and none ever succeeds. Fact: "Connections to
+ <hosts> are refused here." Retires when a fetch to a listed host succeeds.
+ EmptyProbe discovery probes of one kind (VCS refs beyond the checkout) return nothing in
+ every task where they run. Fact: "no remote, no refs beyond HEAD". Retires --
+ hard -- the moment a probe returns something, so it never forms on an image
+ whose history continues past the base commit.
+
+LIFECYCLE. The validated `_Fact` state machine from `mechanisms/injection.py`, unchanged:
+CANDIDATE -> ACTIVE after K_PROMOTE confirmations on drifted tasks, recency-weighted score
+with asymmetric contradiction, and a FIRING GATE so a fact is injected only while the waste
+it prevents keeps occurring (or, once injected, keeps being prevented).
+
+COST HONESTY. No LLM is involved anywhere: `opt_tokens` is exactly 0. The rendered block is
+prompt bytes, priced by the ledger like the rest of the prompt (written once at cache-write,
+re-read every step); it is capped at MAX_TOKENS and MAX_FACTS, and a lint refuses any fact
+that reads as completion or verification advice -- the failure mode of the earlier Codex
+injection variants, which lost resolves with "you are done once the tests pass" hints.
+
+Stdlib only; agent-agnostic through `trace.load_trajectory` (Codex exit codes, Claude and
+Copilot error flags all reduce to the same (form, outcome) observations).
+"""
+import json
+import os
+import re
+
+from ..trace import strip_rtk_prefix
+from collections import Counter
+from dataclasses import dataclass
+
+from .injection import _Fact, edited_pyfiles
+from ..trace import load_trajectory
+
+STATE_SUBDIR = "envfacts_state"
+MAX_FACTS = 4
+MAX_TOKENS = 150 # ~4 chars per token; the whole block, header included
+HEADER = "# environment facts learned from earlier tasks in this checkout"
+
+# --------------------------------------------------------------------------------------
+# vocabulary -- the domain-supplied words that name intents (never harness constants)
+# --------------------------------------------------------------------------------------
+@dataclass(frozen=True)
+class Vocabulary:
+ """What the families recognise as an interpreter, a test runner, a package tool, a
+ fetcher, an environment probe. A domain adapter supplies one (`env_vocabulary()`); the
+ families themselves are the same for every vocabulary."""
+ name: str
+ interpreters: frozenset # executables that run the language
+ canonical_interpreter: str # how a shebang-run script is spelled as a form
+ runner_heads: frozenset # bare test-runner executables
+ runner_scripts: tuple # scripts that ARE the runner when an interpreter runs them
+ runner_modules: frozenset # `<interpreter> -m <module>` runners
+ package_tools: frozenset
+ fetch_heads: frozenset = frozenset({"curl", "wget"})
+ env_probe_heads: frozenset = frozenset({"which", "command", "type", "whereis"})
+
+
+PYTHON_GIT = Vocabulary(
+ name="python-git",
+ interpreters=frozenset({"python", "python2", "python3", "pypy", "pypy3"}),
+ canonical_interpreter="python",
+ runner_heads=frozenset({"pytest", "py.test", "tox", "nox", "nosetests", "trial"}),
+ runner_scripts=("runtests.py", "bin/test", "bin/doctest", "manage.py"),
+ runner_modules=frozenset({"pytest", "unittest", "nose", "django", "tox", "nox"}),
+ package_tools=frozenset({"pip", "pip3", "conda", "mamba", "poetry", "pipenv", "uv",
+ "apt", "apt-get"}),
+)
+
+NODE_WEB = Vocabulary(
+ name="node-web",
+ interpreters=frozenset({"node", "nodejs", "deno", "bun", "tsx", "ts-node"}),
+ canonical_interpreter="node",
+ runner_heads=frozenset({"jest", "vitest", "mocha", "playwright", "cypress", "ava",
+ "karma"}),
+ runner_scripts=(),
+ runner_modules=frozenset(),
+ package_tools=frozenset({"npm", "yarn", "pnpm", "corepack", "apt", "apt-get"}),
+)
+
+DEFAULT_VOCAB = PYTHON_GIT
+# Module-level aliases of the default vocabulary, kept for the offline analyses.
+INTERPRETERS = PYTHON_GIT.interpreters
+RUNNER_HEADS = PYTHON_GIT.runner_heads
+RUNNER_SCRIPTS = PYTHON_GIT.runner_scripts
+RUNNER_MODULES = PYTHON_GIT.runner_modules
+PACKAGE_TOOLS = PYTHON_GIT.package_tools
+FETCH_HEADS = PYTHON_GIT.fetch_heads
+GIT_FETCH_SUBS = frozenset({"fetch", "clone", "pull", "ls-remote"})
+GIT_PROBE_SUBS = frozenset({"remote", "branch", "tag", "fsck", "stash", "show-ref",
+ "for-each-ref"})
+ENV_PROBE_HEADS = PYTHON_GIT.env_probe_heads
+PREFIX_HEADS = ("sudo", "env", "time", "exec", "nice", "nohup", "timeout")
+_PASSTHROUGH = frozenset({"cd", "echo", "head", "tail", "cat", "sort", "uniq", "tee", "true",
+ "printf", "less", "more"})
+
+_ACTIVATE_RE = re.compile(r"^(?:source|\.)\s+\S*activate\s+([\w.-]+)\s*$")
+_CONDA_ACTIVATE_RE = re.compile(r"^(?:conda|mamba)\s+activate\s+([\w.-]+)\s*$")
+_CONDA_RUN_RE = re.compile(r"^(?:conda|mamba)\s+run\s+(?:--no-capture-output\s+|--live-stream\s+)*"
+ r"(?:-n|--name)\s+([\w.-]+)\s+")
+_TOOL_RUN_RE = re.compile(r"^(poetry|pipenv|uv|npx|bundle|yarn|pnpm)\s+(?:run|exec)\s+")
+_NPX_RE = re.compile(r"^npx\s+(?:(?:--yes|-y|--no-install|--no)\s+)*")
+_INTERP_VERSIONED_RE = re.compile(r"^(?:python|node)\d(?:\.\d+)?$")
+_ENV_ASSIGN_RE = re.compile(r"^(?:[A-Za-z_][A-Za-z0-9_]*=(?:\"[^\"]*\"|'[^']*'|\S+)\s+)+")
+_WRAP_RE = re.compile(r"^/bin/(?:ba)?sh\s+-l?c\s+(.*)$", re.S)
+_HEREDOC_RE = re.compile(r"<<-?\s*['\"]?(\w+)['\"]?[^\n]*\n.*?\n\s*\1\b", re.S)
+_URL_HOST_RE = re.compile(r"https?://([^/\s'\"`]+)")
+
+# Error classes an ENVIRONMENT produces (as opposed to the code under test). Order matters:
+# the first match names the class.
+ERROR_CLASSES = (
+ ("module_missing", re.compile(r"ModuleNotFoundError|No module named|Cannot find module|"
+ r"ERR_MODULE_NOT_FOUND", re.I)),
+ ("command_missing", re.compile(r"command not found|: not found\b|is not recognized", re.I)),
+ ("permission_denied", re.compile(r"Permission denied|EACCES", re.I)),
+ ("usage_error", re.compile(r"(?m)^usage:|unrecognized arguments|no such option|"
+ r"error: argument|unknown option", re.I)),
+ ("connection_refused", re.compile(r"Connection refused|Failed to connect to|"
+ r"couldn't connect to|Could not connect|ECONNREFUSED", re.I)),
+ ("name_resolution", re.compile(r"Could not resolve host|Temporary failure in name "
+ r"resolution|Name or service not known|getaddrinfo|"
+ r"nodename nor servname|ENOTFOUND", re.I)),
+ ("unreachable", re.compile(r"Network is unreachable|No route to host|Connection timed out|"
+ r"Operation timed out|timed out after|ETIMEDOUT", re.I)),
+)
+NETWORK_ERRORS = frozenset({"connection_refused", "name_resolution", "unreachable"})
+FORM_ERRORS = frozenset({"module_missing", "command_missing", "usage_error",
+ "permission_denied"})
+
+# What "the probe returned something" looks like, per VCS probe kind. Deliberately narrow
+# patterns so the output of an unrelated segment in a compound command cannot masquerade
+# as a hit.
+_PROBE_HIT = {
+ "remote": re.compile(r"(?m)^\S+\s+\S+\s+\((?:fetch|push)\)\s*$"),
+ "branch": re.compile(r"(?m)^\s*remotes/\S+"),
+ "tag": re.compile(r"(?m)^v?\d+(?:\.\d+)+(?:[a-z]+\d*|\.post\d+|-\S+)?\s*$"),
+ "fsck": re.compile(r"(?m)^(?:unreachable|dangling)\s+(?:commit|tree|blob|tag)\s+[0-9a-f]{7,}"),
+ "stash": re.compile(r"(?m)^stash@\{\d+\}"),
+ "refs": re.compile(r"(?m)^[0-9a-f]{7,}\s+refs/"),
+}
+
+# Words that would turn a fact into completion or verification advice.
+_LINT_RE = re.compile(r"\b(skip|no need|don'?t|do not|stop|done|completed?|finish(?:ed)?|"
+ r"verify|verification|instead of (?:running|testing))\b", re.I)
+
+_ERROR_PHRASE = {
+ "module_missing": "ModuleNotFoundError",
+ "command_missing": "command not found",
+ "usage_error": "a usage error",
+ "permission_denied": "permission denied",
+}
+
+
+# --------------------------------------------------------------------------------------
+# observations
+# --------------------------------------------------------------------------------------
+def unwrap(cmd):
+ """Strip an agent's `/bin/bash -lc "<cmd>"` wrapper, if present."""
+ cmd = cmd or ""
+ m = _WRAP_RE.match(cmd.strip())
+ if not m:
+ return cmd
+ inner = m.group(1).strip()
+ if len(inner) >= 2 and inner[0] == inner[-1] and inner[0] in "\"'":
+ q = inner[0]
+ inner = inner[1:-1]
+ if q == '"':
+ inner = inner.replace('\\"', '"').replace("\\$", "$").replace("\\\\", "\\")
+ return inner
+
+
+def segments(cmd):
+ """Split a shell command into simple segments on && || ; | and newlines, outside quotes.
+ Heredoc bodies are elided first so their contents are never mistaken for commands."""
+ cmd = _HEREDOC_RE.sub(lambda m: m.group(0).split("\n", 1)[0] + " <heredoc>", cmd or "")
+ out, buf, q, i = [], [], None, 0
+ while i < len(cmd):
+ c = cmd[i]
+ if q:
+ buf.append(c)
+ if c == q:
+ q = None
+ elif c == "\\" and i + 1 < len(cmd):
+ buf.append(cmd[i + 1]); i += 1
+ elif c in "\"'":
+ q = c; buf.append(c)
+ elif c == "\n" or c == ";" or c == "|" or (c == "&" and cmd[i:i + 2] == "&&"):
+ out.append("".join(buf)); buf = []
+ if cmd[i:i + 2] in ("&&", "||"):
+ i += 1
+ else:
+ buf.append(c)
+ i += 1
+ out.append("".join(buf))
+ return [s.strip().strip("()").strip() for s in out if s.strip().strip("()").strip()]
+
+
+def _tokens(seg):
+ return seg.split()
+
+
+class Observation:
+ """One classified command segment."""
+ __slots__ = ("call", "intent", "form", "head", "prefix", "hosts", "probe", "outcome",
+ "segment")
+
+ def __init__(self, call, intent, form, head, prefix, segment, hosts=(), probe=None):
+ self.call, self.intent, self.form, self.head, self.prefix = call, intent, form, head, prefix
+ self.segment, self.hosts, self.probe, self.outcome = segment, tuple(hosts), probe, None
+
+ def __repr__(self):
+ return f"Obs({self.intent},{self.form!r},{self.outcome})"
+
+
+def classify_command(cmd, call_index=0, vocab=None):
+ """Observations for one shell command (possibly compound). Activation prefixes carry
+ across the segments of one command, which is how `source activate X && python ...`
+ becomes a single form. `vocab` names the domain's intents (default Python/git)."""
+ vocab = vocab or DEFAULT_VOCAB
+ obs = []
+ carried = ""
+ for seg in segments(unwrap(cmd)):
+ # Transparent wrappers first: `K=V rtk pytest`, `rtk K=V pytest` and `rtk proxy pytest`
+ # are all a pytest run. rtk's Codex mode puts that prefix on every command the agent
+ # types (trace.strip_rtk_prefix), and a first-token vocabulary that did not remove it
+ # would form no fact at all on the very arm that stacks injection on rtk.
+ seg = _ENV_ASSIGN_RE.sub("", seg).strip()
+ seg = _ENV_ASSIGN_RE.sub("", strip_rtk_prefix(seg)).strip()
+ if not seg:
+ continue
+ m = _ACTIVATE_RE.match(seg) or _CONDA_ACTIVATE_RE.match(seg)
+ if m:
+ carried = f"{seg.split()[0] if seg.startswith(('conda', 'mamba')) else 'source'} activate {m.group(1)} &&"
+ if carried.startswith("conda") or carried.startswith("mamba"):
+ carried = f"conda activate {m.group(1)} &&"
+ continue
+ prefix = carried
+ body = seg
+ m = _CONDA_RUN_RE.match(body)
+ if m:
+ prefix = (prefix + " " if prefix else "") + f"conda run -n {m.group(1)}"
+ body = body[m.end():]
+ m = _TOOL_RUN_RE.match(body)
+ if m:
+ prefix = (prefix + " " if prefix else "") + f"{m.group(1)} run"
+ body = body[m.end():]
+ else:
+ m = _NPX_RE.match(body)
+ if m and m.end() > 0:
+ prefix = (prefix + " " if prefix else "") + "npx"
+ body = body[m.end():]
+ toks = _tokens(body)
+ while toks and toks[0] in PREFIX_HEADS:
+ toks = toks[1:] if toks[0] != "timeout" else toks[2:]
+ if not toks:
+ continue
+ head = toks[0]
+ base = head.rsplit("/", 1)[-1]
+ rest = toks[1:]
+ o = _classify(head, base, rest, body, prefix, seg, call_index, vocab)
+ if o is not None:
+ obs.append(o)
+ return obs
+
+
+def _runner_script(rest, vocab):
+ for t in rest:
+ if t.startswith("-"):
+ continue
+ for s in vocab.runner_scripts:
+ if t == s or t.endswith("/" + s) or t.endswith(s):
+ return t.lstrip("./") if t.startswith("./") else t
+ return None
+ return None
+
+
+def _classify(head, base, rest, body, prefix, seg, call, vocab):
+ def form(expr):
+ return (prefix + " " + expr).strip() if prefix else expr
+
+ if base in vocab.interpreters or _INTERP_VERSIONED_RE.match(base):
+ if rest and all(t in ("--version", "-V", "-VV", "-v") for t in rest):
+ return Observation(call, "probe_env", form(head), head, prefix, seg)
+ if "-m" in rest:
+ mod = rest[rest.index("-m") + 1] if rest.index("-m") + 1 < len(rest) else ""
+ if mod in vocab.runner_modules:
+ return Observation(call, "run_tests", form(f"{head} -m {mod}"), head, prefix, seg)
+ script = _runner_script(rest, vocab)
+ if script and (script != "manage.py" or "test" in rest):
+ return Observation(call, "run_tests", form(f"{head} {script}"), head, prefix, seg)
+ return Observation(call, "run_interpreter", form(head), head, prefix, seg)
+ if base in vocab.runner_heads:
+ return Observation(call, "run_tests", form(head), head, prefix, seg)
+ for s in vocab.runner_scripts:
+ if head == s or head.endswith("/" + s) or head == "./" + s:
+ if s == "manage.py" and "test" not in rest:
+ break
+ # A script run through its shebang is the interpreter running it; spell the
+ # form that way so `./tests/runtests.py` and `python tests/runtests.py` pool.
+ return Observation(call, "run_tests",
+ form(f"{vocab.canonical_interpreter} " + head.lstrip("./")),
+ head, prefix, seg)
+ if base in ("make", "npm", "yarn", "pnpm", "cargo", "go") and rest and rest[0] == "test":
+ return Observation(call, "run_tests", form(f"{base} test"), head, prefix, seg)
+ if base in vocab.env_probe_heads:
+ return Observation(call, "probe_env", form(base), head, prefix, seg)
+ if base in vocab.fetch_heads:
+ return Observation(call, "fetch_remote", form(base), head, prefix, seg,
+ hosts=_URL_HOST_RE.findall(body))
+ if base == "git" and rest:
+ sub = rest[0]
+ if sub in GIT_FETCH_SUBS:
+ return Observation(call, "fetch_remote", form(f"git {sub}"), head, prefix, seg,
+ hosts=_URL_HOST_RE.findall(body))
+ if sub in GIT_PROBE_SUBS:
+ kind = _probe_kind(sub, rest[1:])
+ if kind:
+ return Observation(call, "probe_vcs_refs", form(f"git {sub}"), head, prefix,
+ seg, probe=kind)
+ return None
+ if base in vocab.package_tools:
+ if base.startswith(("pip", "npm", "yarn", "pnpm")) and rest and rest[0] in (
+ "install", "download", "add", "ci"):
+ hosts = _URL_HOST_RE.findall(body)
+ return Observation(call, "fetch_remote", form(f"{base} {rest[0]}"), head, prefix,
+ seg, hosts=hosts)
+ return Observation(call, "package_tool", form(base), head, prefix, seg)
+ return None
+
+
+def _probe_kind(sub, args):
+ a = set(args)
+ if sub == "remote":
+ return "remote"
+ if sub == "branch" and a & {"-a", "-r", "--all", "--remotes", "-ar", "-ra"}:
+ return "branch"
+ if sub == "tag" and "--contains" in a: # plain `git tag` lists PAST tags too
+ return "tag"
+ if sub == "fsck":
+ return "fsck"
+ if sub == "stash" and args[:1] == ["list"]:
+ return "stash"
+ if sub in ("show-ref", "for-each-ref"):
+ return "refs"
+ return None
+
+
+def error_class(output):
+ for name, rx in ERROR_CLASSES:
+ if rx.search(output or ""):
+ return name
+ return None
+
+
+def _exit_code_hint(output):
+ m = re.search(r"(?i)\bexit code[:=\s]+(\d+)", output or "")
+ return int(m.group(1)) if m else None
+
+
+def observe_task(transcript_path, vocab=None):
+ """Per-task digest: observations with outcomes, the edited files, the call count.
+
+ Outcome attribution inside a compound command is heuristic and conservative: an
+ infrastructure error is attributed to the segment whose executable the error names, else
+ to the last relevant segment; a call that succeeded marks every relevant segment as
+ having run; a failed call with no infrastructure error marks only the first relevant
+ segment as having run and leaves the others unknown (an `&&` chain may not have reached
+ them)."""
+ calls = load_trajectory(transcript_path)
+ obs_all = []
+ triples = []
+ for idx, (name, args, out, ok) in enumerate(calls):
+ tool = (name or "").lower()
+ triples.append((tool, args if isinstance(args, dict) else {}, out or ""))
+ if tool not in ("shell", "bash", "stop_bash"):
+ continue
+ cmd = (args.get("command") or args.get("cmd") or "") if isinstance(args, dict) else ""
+ out = out or ""
+ obs = classify_command(cmd, idx, vocab)
+ if not obs:
+ continue
+ # Pass-through segments (a leading `cd`, a trailing `| head`, an `echo ---`
+ # separator) neither add evidence nor pollute the output, so they do not break a
+ # call's homogeneity; filters that rewrite content (grep, sed, awk, wc) do.
+ n_seg = sum(1 for sg in segments(unwrap(cmd))
+ if (sg.split() or [""])[0].rsplit("/", 1)[-1] not in _PASSTHROUGH)
+ homogeneous = n_seg == len(obs) and len({o.intent for o in obs}) == 1
+ hint = _exit_code_hint(out)
+ success = bool(ok) if hint is None else (hint == 0)
+ err = error_class(out)
+ if err in FORM_ERRORS or err in NETWORK_ERRORS:
+ targets = [o for o in obs if o.intent in ("run_interpreter", "run_tests",
+ "package_tool", "fetch_remote")]
+ named = [o for o in targets
+ if (o.head and o.head in out) or (o.head and o.head.rsplit("/", 1)[-1] in
+ _error_line(out, err))]
+ if err in NETWORK_ERRORS:
+ named = [o for o in targets if o.intent == "fetch_remote"
+ and (not o.hosts or any(h in out for h in o.hosts))] or \
+ [o for o in targets if o.intent == "fetch_remote"]
+ chosen = named or (targets[-1:] if not success else [])
+ for o in chosen:
+ o.outcome = err
+ if success:
+ for o in obs:
+ if o.outcome is None:
+ o.outcome = "ran"
+ elif success:
+ for o in obs:
+ o.outcome = "ran"
+ else:
+ first = True
+ for o in obs:
+ if o.intent in ("run_interpreter", "run_tests", "package_tool", "fetch_remote"):
+ o.outcome = "ran" if first else None
+ first = False
+ else:
+ o.outcome = "ran" if o.intent != "probe_vcs_refs" else None
+ for o in obs:
+ if o.intent == "probe_vcs_refs":
+ # Judged only when every segment of the call is a probe: another segment's
+ # output (a `git describe`, a `sed`) would otherwise be read as the probe's.
+ if not homogeneous:
+ o.outcome = None
+ else:
+ o.outcome = "hit" if _PROBE_HIT[o.probe].search(out) else (
+ "empty" if success else None)
+ if o.intent == "fetch_remote" and o.outcome == "ran":
+ # Reachability needs a homogeneous call that exited 0 with real output and no
+ # network error; a compound command's exit code and output belong to its
+ # last segment, not to the fetch.
+ if not homogeneous or not success or not out.strip() or err in NETWORK_ERRORS:
+ o.outcome = None
+ obs_all.extend(obs)
+ return {"obs": obs_all, "edited": sorted(edited_pyfiles(triples)), "n_calls": len(calls)}
+
+
+def _error_line(out, err):
+ rx = dict(ERROR_CLASSES)[err]
+ m = rx.search(out or "")
+ if not m:
+ return ""
+ start = (out or "").rfind("\n", 0, m.start()) + 1
+ end = (out or "").find("\n", m.end())
+ return (out or "")[start:end if end >= 0 else None]
+
+
+def _is_bare(form):
+ """A form with no activation prefix and no explicit path is 'bare'."""
+ head = form.split()[0] if form else ""
+ return "/" not in head and " activate " not in form and " run " not in form and \
+ not form.startswith(("conda run", "poetry run", "pipenv run", "uv run", "npx "))
+
+
+# --------------------------------------------------------------------------------------
+# facts
+# --------------------------------------------------------------------------------------
+class EnvFact:
+ def __init__(self, fid, family, high, params=None):
+ self.id = fid
+ self.family = family
+ self.params = params or {}
+ self.life = _Fact(fid, high)
+
+ # ---- lifecycle passthrough ----
+ @property
+ def state(self):
+ return self.life.state
+
+ def injectable(self):
+ return self.life.injectable()
+
+ def firing_rate(self):
+ return self.life.firing_rate()
+
+ # ---- rendering ----
+ def claim(self):
+ p = self.params
+ if self.family == "form_repair":
+ b = self._best("b_counts")
+ e = self._best("error_counts") or "module_missing"
+ return f"`{p['a']} …` fails here with {_ERROR_PHRASE.get(e, e)}; `{b} …` works."
+ if self.family == "boundary":
+ hosts = [h for h, _ in Counter(p.get("hosts", {})).most_common(3)]
+ return ("Outbound connections to " + _join(hosts) +
+ " are refused here; nothing can be fetched from them.")
+ if self.family == "empty_probe":
+ return ("This checkout has no git remote and no branches, tags or unreachable "
+ "objects beyond HEAD; its history ends at the checked-out commit.")
+ return ""
+
+ def _best(self, key):
+ c = self.params.get(key) or {}
+ return max(c.items(), key=lambda kv: kv[1])[0] if c else None
+
+ def to_dict(self):
+ return {"id": self.id, "family": self.family, "params": self.params,
+ "life": self.life.to_dict()}
+
+ @classmethod
+ def from_dict(cls, d):
+ f = cls(d["id"], d["family"], d["life"].get("high", False), d.get("params") or {})
+ f.life = _Fact.from_dict(d["life"])
+ return f
+
+
+CONTRADICT_STREAK = 2 # FormRepair: consecutive contradicted tasks before one counts
+
+
+def fold(fact, now, verdict, fired, exercised, files):
+ """Fold one task's verdict into a fact's lifecycle.
+
+ Two adjustments to the shared state machine. A high-stakes CANDIDATE that is contradicted
+ retires at once (`_Fact` hard-kills only ACTIVE facts): an image whose history continues
+ past HEAD does not become clean by being probed again. And a FormRepair contradiction --
+ the bare form ran the test runner -- only counts after CONTRADICT_STREAK consecutive
+ exercised tasks said so: a single task whose tests happened to need none of the missing
+ packages is not a fixed environment, and the asymmetric contradiction weight would
+ otherwise retire a fact confirmed thirty times over."""
+ if fact.family == "form_repair" and exercised:
+ streak = fact.params.get("contra_streak", 0)
+ streak = streak + 1 if verdict == "contradict" else 0
+ fact.params["contra_streak"] = streak
+ if verdict == "contradict" and streak < CONTRADICT_STREAK:
+ verdict = None
+ fact.life.update(now, verdict, fired, exercised, files)
+ if verdict == "contradict" and fact.life.high and fact.life.state == "CANDIDATE":
+ fact.life.state, fact.life.retired_at = "RETIRED", now
+
+
+def _root(form):
+ """The part of a working form that carries the environment: its activation prefix, or
+ the directory of its explicit path."""
+ if " && " in form:
+ return form.split(" && ", 1)[0]
+ if form.startswith(("conda run", "poetry run", "pipenv run", "uv run", "npx")):
+ return " ".join(form.split()[:3])
+ head = form.split()[0] if form else ""
+ return head.rsplit("/", 1)[0] if "/" in head else head
+
+
+def _join(items):
+ items = list(items)
+ if not items:
+ return "the hosts it tried"
+ if len(items) == 1:
+ return items[0]
+ return ", ".join(items[:-1]) + " and " + items[-1]
+
+
+def lint_ok(text):
+ return not _LINT_RE.search(text or "")
+
+
+# --------------------------------------------------------------------------------------
+# the distiller
+# --------------------------------------------------------------------------------------
+class EnvFactsDistiller:
+ """Learns and injects environment facts over a repository's task stream.
+
+ Interface parity with `injection.Distiller`, so `PrefixStage` drives both the same way:
+ d = EnvFactsDistiller(repo, vocab=...) # fresh, or .load(repo, root, vocab=...)
+ prefix = d.render() # block for the NEXT task ("" if none)
+ d.update(transcript_path, iid=..., resolved=...)
+ d.save(root)
+ `runner`/`model` are accepted and ignored: nothing here is paid. `vocab` is the domain's
+ `Vocabulary` (default Python/git); `injectable_claims()` is what a paid proposer stacked
+ on top is told not to restate. `max_facts` / `max_tokens` cap the rendered block
+ (default MAX_FACTS / MAX_TOKENS).
+ """
+ def __init__(self, repo, runner=None, model="agent", vocab=None, max_facts=None,
+ max_tokens=None):
+ self.max_facts = int(max_facts or MAX_FACTS)
+ self.max_tokens = int(max_tokens or MAX_TOKENS)
+ self.repo = repo
+ self.vocab = vocab or DEFAULT_VOCAB
+ self.now = 0
+ self.facts = {} # id -> EnvFact
+ self.opt_tokens = 0.0 # invariant: LLM-free
+ self.last_injected = set() # fact ids rendered for the task now running
+ self.last_digest = None # the most recent per-task verdicts (telemetry)
+
+ # ---- render --------------------------------------------------------------------
+ def render(self, task=None):
+ chosen = self._select()
+ self.last_injected = {f.id for f in chosen}
+ if not chosen:
+ return ""
+ lines = [HEADER] + [f"- {t}" for t in self._merged_lines(chosen)]
+ return "\n".join(lines) + "\n"
+
+ def injectable_claims(self):
+ """The claims that would be injected now, one string each (no header)."""
+ return [f.claim() for f in self._select()]
+
+ def _select(self):
+ cands = [f for f in self.facts.values() if f.injectable() and lint_ok(f.claim())]
+ prio = {"form_repair": 0, "empty_probe": 1, "boundary": 2}
+ cands.sort(key=lambda f: (prio.get(f.family, 9),
+ 0 if f.params.get("intent") == "run_interpreter" else 1,
+ -f.firing_rate()))
+ chosen = []
+ for f in cands:
+ trial = chosen + [f]
+ if len(trial) > self.max_facts:
+ break
+ text = HEADER + "\n" + "\n".join(f"- {t}" for t in self._merged_lines(trial))
+ if len(text) > self.max_tokens * 4:
+ continue
+ chosen = trial
+ return chosen
+
+ def _merged_lines(self, facts):
+ """FormRepair facts whose working forms share a root (the same activation prefix
+ or the same directory) collapse into one line: they are one environment fact."""
+ lines, groups = [], {}
+ for f in facts:
+ if f.family == "form_repair":
+ groups.setdefault(_root(f._best("b_counts") or ""), []).append(f)
+ else:
+ lines.append(f.claim())
+ for _root_key, fs in groups.items():
+ if len(fs) == 1:
+ lines.insert(0, fs[0].claim())
+ continue
+ a_forms = _join([f"`{f.params['a']} …`" for f in fs])
+ errs = sorted({_ERROR_PHRASE.get(f._best("error_counts") or "", "") for f in fs}
+ - {""})
+ b_forms = _join(sorted({f"`{f._best('b_counts')} …`" for f in fs}))
+ lines.insert(0, f"{a_forms} fail here ({', '.join(errs) or 'environment error'}); "
+ f"{b_forms} work.")
+ return lines
+
+ # ---- update --------------------------------------------------------------------
+ def update(self, transcript_path, iid="", resolved=None, edited=None):
+ try:
+ digest = observe_task(transcript_path, self.vocab)
+ except Exception:
+ self.now += 1
+ self.last_injected = set()
+ return
+ obs = digest["obs"]
+ files = set(digest["edited"]) if edited is None else set(edited)
+ injected = set(self.last_injected)
+ verdicts = {}
+
+ # ---- FormRepair: discover candidates, then fold every fact ----
+ by_intent = {}
+ for o in obs:
+ by_intent.setdefault(o.intent, []).append(o)
+ for intent in ("run_interpreter", "run_tests", "package_tool"):
+ seq = by_intent.get(intent, [])
+ fails = [o for o in seq if o.outcome in FORM_ERRORS]
+ if intent != "run_tests":
+ # An explicit path or activation that fails is a missing dependency, not a
+ # resolution problem; only a bare form can be "repaired" by a fuller one.
+ fails = [o for o in fails if _is_bare(o.form)]
+ for a in fails:
+ later_ok = [o for o in seq if o.outcome == "ran" and o.call > a.call
+ and o.form != a.form
+ and (intent == "run_tests" or not _is_bare(o.form))]
+ if not later_ok:
+ continue
+ fid = f"form_repair:{intent}:{a.form}"
+ f = self.facts.get(fid)
+ if f is None:
+ f = EnvFact(fid, "form_repair", high=False,
+ params={"intent": intent, "a": a.form,
+ "b_counts": {}, "error_counts": {}})
+ self.facts[fid] = f
+ for o in later_ok:
+ f.params["b_counts"][o.form] = f.params["b_counts"].get(o.form, 0) + 1
+ f.params["error_counts"][a.outcome] = f.params["error_counts"].get(a.outcome, 0) + 1
+ for f in [x for x in self.facts.values() if x.family == "form_repair"]:
+ intent, a_form = f.params["intent"], f.params["a"]
+ seq = by_intent.get(intent, [])
+ exercised = bool(seq)
+ a_failed = any(o.form == a_form and o.outcome in FORM_ERRORS for o in seq)
+ b_forms = set(f.params["b_counts"])
+ b_ran = any(o.form in b_forms and o.outcome == "ran" for o in seq) or \
+ any(o.outcome == "ran" and not _is_bare(o.form) for o in seq)
+ # Contradiction: the bare form ran the TEST RUNNER without an environment error.
+ # A bare interpreter one-liner succeeding is not evidence (pure-Python packages
+ # import from the working tree under any interpreter).
+ a_runner_ok = any(o.form == a_form and o.outcome == "ran"
+ for o in by_intent.get("run_tests", []))
+ fired = a_failed or (f.id in injected and b_ran and not a_failed)
+ if a_failed or (b_ran and not a_runner_ok):
+ verdict = "confirm"
+ elif a_runner_ok:
+ verdict = "contradict"
+ else:
+ verdict = None
+ fold(f, self.now, verdict, fired, exercised, files)
+ verdicts[f.id] = (verdict, fired, exercised)
+
+ # ---- Boundary: hosts refused ----
+ fetches = by_intent.get("fetch_remote", [])
+ failed_hosts = Counter(h for o in fetches if o.outcome in NETWORK_ERRORS for h in o.hosts)
+ ok_hosts = {h for o in fetches if o.outcome == "ran" for h in o.hosts}
+ fid = "boundary:fetch_remote"
+ f = self.facts.get(fid)
+ if f is None and failed_hosts:
+ f = EnvFact(fid, "boundary", high=True, params={"hosts": {}})
+ self.facts[fid] = f
+ if f is not None:
+ for h, n in failed_hosts.items():
+ f.params["hosts"][h] = f.params["hosts"].get(h, 0) + n
+ known = set(f.params["hosts"])
+ exercised = any(o.hosts for o in fetches)
+ fired = bool(failed_hosts)
+ if ok_hosts & known:
+ verdict = "contradict"
+ elif failed_hosts:
+ verdict = "confirm"
+ else:
+ verdict = None
+ fold(f, self.now, verdict, fired, exercised, files)
+ verdicts[fid] = (verdict, fired, exercised)
+
+ # ---- EmptyProbe: refs beyond the checkout ----
+ probes = [o for o in by_intent.get("probe_vcs_refs", []) if o.outcome in ("hit", "empty")]
+ fid = "empty_probe:vcs_refs"
+ f = self.facts.get(fid)
+ if f is None and probes:
+ f = EnvFact(fid, "empty_probe", high=True, params={"kinds": {}})
+ self.facts[fid] = f
+ if f is not None:
+ for o in probes:
+ f.params["kinds"][o.probe] = f.params["kinds"].get(o.probe, 0) + 1
+ exercised = bool(probes)
+ fired = exercised
+ if any(o.outcome == "hit" for o in probes):
+ verdict = "contradict"
+ elif probes:
+ verdict = "confirm"
+ else:
+ verdict = None
+ fold(f, self.now, verdict, fired, exercised, files)
+ verdicts[fid] = (verdict, fired, exercised)
+
+ self.last_digest = {"iid": iid, "resolved": resolved, "injected": sorted(injected),
+ "verdicts": verdicts, "n_calls": digest["n_calls"]}
+ self.last_injected = set()
+ self.now += 1
+
+ # ---- persistence -------------------------------------------------------------
+ @staticmethod
+ def _path(root, repo):
+ d = os.path.join(root, STATE_SUBDIR)
+ os.makedirs(d, exist_ok=True)
+ return os.path.join(d, repo.replace("/", "__") + ".json")
+
+ def save(self, root):
+ with open(self._path(root, self.repo), "w", encoding="utf-8") as fh:
+ json.dump({"repo": self.repo, "now": self.now, "opt_tokens": 0.0,
+ "vocab": self.vocab.name,
+ "facts": {k: f.to_dict() for k, f in self.facts.items()}}, fh, indent=2)
+
+ @classmethod
+ def load(cls, repo, root, runner=None, model="agent", vocab=None, max_facts=None,
+ max_tokens=None):
+ self = cls(repo, vocab=vocab, max_facts=max_facts, max_tokens=max_tokens)
+ p = self._path(root, repo)
+ if not os.path.exists(p):
+ return self
+ with open(p, encoding="utf-8") as fh:
+ d = json.load(fh)
+ self.now = d.get("now", 0)
+ self.facts = {k: EnvFact.from_dict(v) for k, v in d.get("facts", {}).items()}
+ return self
+
+ # ---- telemetry ---------------------------------------------------------------
+ def debug_row(self):
+ row = {}
+ for k, f in self.facts.items():
+ row[k] = {"state": f.state, "fire": f"{f.life.fire_num}/{f.life.fire_den}",
+ "rate": round(f.firing_rate(), 2), "inject": f.injectable(),
+ "claim": f.claim()[:120], "family": f.family}
+ row["_profile"] = "envfacts"
+ row["_vocab"] = self.vocab.name
+ if self.last_digest:
+ row["_task"] = self.last_digest
+ return row
diff --git a/tokeneconomizer/mechanisms/injection.py b/tokeneconomizer/mechanisms/injection.py
new file mode 100644
index 0000000..c05f91e
--- /dev/null
+++ b/tokeneconomizer/mechanisms/injection.py
@@ -0,0 +1,832 @@
+"""Injection mechanism: a firing-gated, per-repo "operating manual" learned from the
+running task stream and injected into the next task's prefix.
+
+WHAT IT DOES. As a repo's chronological task stream runs, a paid LLM proposer (one call
+per `PROPOSE_EVERY` finished tasks, over a COMPACT digest -- never raw transcripts)
+authors two kinds of durable, repo-specific artifacts:
+
+ * facts -- one-line declarative statements about the repo an agent would want
+ stated up front (canonical test command, "a failing test already
+ reproduces the bug", a required env var, a build step, ...).
+ * extractors -- tiny host-executable predicates over the raw command/output stream
+ that surface evidence the fixed digest doesn't already expose.
+
+Every fact carries a host-run DETECTOR (regex/predicate DSL), so the hot path stays
+LLM-free: the LLM invents the facts, the host runs the checks. Each fact then flows
+through a validated lifecycle -- FORMATION (promote CANDIDATE->ACTIVE after K_PROMOTE
+confirmations across DRIFTED tasks, proving commit-stable not task-stable), EXPIRY
+(recency-weighted score with asymmetric contradiction; a single clean contradiction
+hard-kills a high-stakes fact), and a FIRING GATE (inject a fact only once its measured
+firing rate on this repo clears FIRE_GATE) -- so dead prefix bytes never accrue.
+
+One revisable knob: an existing fact's CONTRADICTION condition (`detector.contradict`)
+may be corrected after the fact was first written; everything else about a fact, and all
+extractors, are append-only. Because the contradict regex only ever RETIRES a fact (never
+confirms/fires it), revising it cannot manufacture unearned credibility -- so the earned
+lifecycle counters are kept in place and the new condition applies going forward.
+
+COST HONESTY. The proposer's effective-token cost accrues on `Distiller.opt_tokens`; the
+per-task delta is charged to the amortized ledger as `optimizer_tokens`. As extractor
+coverage matures the proposer stops needing raw excerpts and the cost decays.
+
+It is the injection arm of the generic (Claude / Copilot) tri-mechanism stack.
+
+This is the single, un-versioned successor to the manual-distiller line (v5.1 frozen ->
+v5.2 open facts -> v5.3 open extractors -> v5.4 revisable contradiction).
+"""
+import json
+import os
+import re
+
+from .base import Stage
+
+# ---- lifecycle knobs (the validated replay constants) ----------------------
+K_PROMOTE = 3 # drifted confirmations to promote CANDIDATE -> ACTIVE
+GAMMA = 0.8 # recency decay per task-ago
+LAMBDA = 5.0 # contradiction weight (asymmetry: fail toward retirement)
+SCORE_FLOOR = 0.0 # retire ACTIVE when recency score < floor
+HARD_KILL = True # high-stakes fact: single clean contradiction retires now
+FIRE_GATE = 0.40 # inject only if firing rate on this repo >= 40%
+FIRE_MIN_N = 4 # ...measured over at least this many exercised tasks
+
+# ---- proposer cadence / caps / gates ---------------------------------------
+PROPOSE_EVERY = 8 # one LLM proposer call per this many tasks (PrefixStage rebinds
+ # this module global from config; default cadence otherwise)
+WINDOW = 8 # digests handed to the proposer per tick (rolling)
+MAX_CANDS = 4 # cap facts emitted per tick
+MIN_CITE = 2 # a candidate must cite >= this many confirming task-ids
+MAX_EXTRACTORS = 12 # cap authored features (keeps the digest bounded)
+RAW_SAMPLE = 3 # raw excerpts shown to the proposer on an abstaining tick
+ABSTAIN_GATE = 0.5 # tick is "abstaining" if >= this frac of window cmds are unseen
+STATE_SUBDIR = "manual_state"
+
+# ---- regexes ---------------------------------------------------------------
+PYTEST_RE = re.compile(r"-m\s+pytest|\bpytest\b|\bpy\.test\b")
+REPRO_RE = re.compile(r"python[0-9.]*\s+-c|/tmp/\S*\.py|reproduce|repro")
+BADARG_RE = re.compile(r"unrecognized arguments|no such option|usage:|error: argument")
+
+
+# ---- transcript parsing ----------------------------------------------------
+def load_calls(fp):
+ """Ordered ``(tool_lower, args, out_text)`` calls for any supported agent.
+
+ Codex (``item.completed``) and Copilot (``tool.execution_*``) have no Claude-style
+ ``message.content[]`` envelope, so BOTH are routed through the shared trajectory
+ normalizer (trace.load_trajectory auto-detects every schema); otherwise a Copilot
+ transcript would parse to [] and the proposer would run (billed) but ground nothing.
+ """
+ try:
+ is_codex = is_copilot = False
+ with open(fp, encoding="utf-8", errors="replace") as probe:
+ for line in probe:
+ if ('"item.completed"' in line
+ or '"type": "item.completed"' in line):
+ is_codex = True
+ break
+ if ('"tool.execution_start"' in line
+ or '"tool.execution_complete"' in line):
+ is_copilot = True
+ break
+ if is_codex or is_copilot:
+ from ..trace import load_trajectory
+ return [(name.lower(), args, out)
+ for name, args, out, _ok in load_trajectory(fp)]
+ except OSError:
+ return []
+ uses, order, res = {}, [], {}
+ for line in open(fp, encoding="utf-8", errors="replace"):
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ d = json.loads(line)
+ except Exception:
+ continue
+ m = d.get("message")
+ if not isinstance(m, dict):
+ continue
+ c = m.get("content")
+ if not isinstance(c, list):
+ continue
+ for b in c:
+ if not isinstance(b, dict):
+ continue
+ if b.get("type") == "tool_use":
+ uses[b.get("id")] = (b.get("name", ""), b.get("input") or {})
+ order.append(b.get("id"))
+ elif b.get("type") == "tool_result":
+ cont = b.get("content")
+ txt = cont if isinstance(cont, str) else json.dumps(cont)
+ res[b.get("tool_use_id")] = txt
+ return [(uses[t][0].lower(), uses[t][1], res.get(t, "")) for t in order]
+
+
+def bash_cmds(calls):
+ out = []
+ for tool, args, res in calls:
+ if tool in ("bash", "shell", "stop_bash"):
+ out.append(((args.get("command") or args.get("cmd") or ""), res))
+ return out
+
+
+def edited_pyfiles(calls):
+ fs = set()
+ for tool, args, _ in calls:
+ if tool in ("edit", "multiedit", "write", "create", "str_replace_editor"):
+ p = args.get("file_path") or args.get("path") or ""
+ if p.endswith(".py"):
+ fs.add(os.path.basename(p))
+ elif tool == "apply_patch":
+ for p in args.get("paths") or ():
+ if p.endswith(".py"):
+ fs.add(os.path.basename(p))
+ return fs
+
+
+# ---- lifecycle state machine (per fact) ------------------------------------
+class _Fact:
+ def __init__(self, key, high):
+ self.key = key
+ self.high = high
+ self.state = "CANDIDATE" # CANDIDATE | ACTIVE | RETIRED
+ self.confirms = [] # [(task_idx, [edited_files])]
+ self.contradicts = [] # [task_idx]
+ self.fire_num = 0 # tasks where the waste occurred (fired)
+ self.fire_den = 0 # tasks where the fact was exercised
+ self.promoted_at = None
+ self.retired_at = None
+
+ def to_dict(self):
+ return {"key": self.key, "high": self.high, "state": self.state,
+ "confirms": self.confirms, "contradicts": self.contradicts,
+ "fire_num": self.fire_num, "fire_den": self.fire_den,
+ "promoted_at": self.promoted_at, "retired_at": self.retired_at}
+
+ @classmethod
+ def from_dict(cls, d):
+ f = cls(d["key"], d["high"])
+ f.__dict__.update(d)
+ return f
+
+ def _drifted_confirms(self):
+ kept = []
+ for idx, fs in self.confirms:
+ fs = set(fs)
+ if not fs or all(len(fs & set(k)) == 0 for _, k in kept):
+ kept.append((idx, fs))
+ return len(kept)
+
+ def score(self, now):
+ s = sum(GAMMA ** (now - i) for i, _ in self.confirms)
+ s -= LAMBDA * sum(GAMMA ** (now - i) for i in self.contradicts)
+ return s
+
+ def firing_rate(self):
+ return (self.fire_num / self.fire_den) if self.fire_den else 0.0
+
+ def injectable(self):
+ """ACTIVE and clears the firing gate -> goes into the rendered manual."""
+ return (self.state == "ACTIVE"
+ and self.fire_den >= FIRE_MIN_N
+ and self.firing_rate() >= FIRE_GATE)
+
+ def update(self, now, verdict, fired, exercised, files):
+ if exercised:
+ self.fire_den += 1
+ if fired:
+ self.fire_num += 1
+ if verdict == "confirm":
+ self.confirms.append((now, sorted(files)))
+ elif verdict == "contradict":
+ self.contradicts.append(now)
+ if self.state == "CANDIDATE":
+ if self._drifted_confirms() >= K_PROMOTE:
+ self.state, self.promoted_at = "ACTIVE", now
+ elif self.state == "ACTIVE":
+ hard = self.high and verdict == "contradict" and HARD_KILL
+ if hard or self.score(now) < SCORE_FLOOR:
+ self.state, self.retired_at = "RETIRED", now
+
+
+# ---- compact digest: one dict per finished task (the proposer's only view) --
+def build_digest(repo, transcript_path, iid, resolved=None):
+ try:
+ calls = load_calls(transcript_path)
+ except Exception:
+ return None
+ cmds = bash_cmds(calls)
+ cmd_blob = "\n".join(c for c, _ in cmds)
+ out_blob = "\n".join(o for _, o in cmds)
+ test_cmds = [c for c, _ in cmds if PYTEST_RE.search(c)
+ or re.search(r"runtests\.py|unittest|nosetests|tox", c)]
+ repro = any(REPRO_RE.search(c) and not PYTEST_RE.search(c) for c, _ in cmds)
+ return {
+ "iid": iid,
+ "resolved": bool(resolved) if resolved is not None else None,
+ "edited_files": sorted(edited_pyfiles(calls)),
+ "n_cmds": len(cmds),
+ "cmd_templates": _templates(c for c, _ in cmds),
+ "test_invocations": _templates(test_cmds)[:6],
+ "repro_written": repro,
+ "importerror": bool(re.search(
+ r"ImportError|ModuleNotFoundError|No module named", out_blob)),
+ "badarg": bool(BADARG_RE.search(out_blob)),
+ # RAW blobs kept for host-side detector eval, NOT serialized to the prompt.
+ "_cmd_blob": cmd_blob,
+ "_out_blob": out_blob,
+ }
+
+
+def _templates(cmds, cap=12):
+ """Dedup command shapes: drop long path/arg tails so the proposer sees FORMS."""
+ seen, out = set(), []
+ for c in cmds:
+ c = c.strip()
+ if not c:
+ continue
+ head = re.sub(r"\s+", " ", c)[:80]
+ head = re.sub(r"/testbed/\S+", "<path>", head)
+ if head not in seen:
+ seen.add(head)
+ out.append(head)
+ if len(out) >= cap:
+ break
+ return out
+
+
+def proposer_view(dg):
+ """The token-cheap projection actually shown to the LLM (drops raw blobs)."""
+ return {k: v for k, v in dg.items() if not k.startswith("_")}
+
+
+def _rx(pat):
+ if not pat:
+ return None
+ try:
+ return re.compile(pat)
+ except re.error:
+ return None
+
+
+# ---- detector DSL: host-executable, no LLM at eval time --------------------
+PREDICATES = {
+ "repro_written": lambda dg: dg["repro_written"],
+ "importerror": lambda dg: dg["importerror"],
+ "badarg": lambda dg: dg["badarg"],
+ "ran_tests": lambda dg: bool(dg["test_invocations"]),
+}
+
+
+class Detector:
+ """Base detector: scope/fired name a built-in predicate or a regex over commands."""
+ def __init__(self, spec):
+ self.scope = spec.get("scope", "")
+ self.confirm = _rx(spec.get("confirm"))
+ self.contra = _rx(spec.get("contradict"))
+ self.fired = spec.get("fired") or spec.get("scope")
+ self.raw = spec
+
+ def _exercised(self, dg):
+ p = PREDICATES.get(self.scope)
+ if p is not None:
+ return bool(p(dg))
+ rx = _rx(self.scope)
+ return bool(rx and rx.search(dg["_cmd_blob"]))
+
+ def _fired(self, dg):
+ p = PREDICATES.get(self.fired)
+ if p is not None:
+ return bool(p(dg))
+ rx = _rx(self.fired)
+ return bool(rx and rx.search(dg["_cmd_blob"]))
+
+ def verdict(self, dg):
+ exercised = self._exercised(dg)
+ if not exercised:
+ return None, False, False
+ blob = dg["_cmd_blob"] + "\n" + dg["_out_blob"]
+ if self.contra and self.contra.search(blob):
+ return "contradict", True, True
+ if self.confirm and self.confirm.search(blob):
+ return "confirm", self._fired(dg), True
+ return None, self._fired(dg), True
+
+
+class DetectorV3Scope(Detector):
+ """Detector whose scope/fired may reference an authored EXTRACTOR feature (resolved
+ against the digest's per-task `features`), then a built-in predicate, then a regex."""
+ def __init__(self, spec, features):
+ self.features = features
+ super().__init__(spec)
+
+ def _lookup(self, name, dg):
+ if name in self.features:
+ return bool(self.features[name])
+ if name in PREDICATES:
+ return bool(PREDICATES[name](dg))
+ rx = _rx(name)
+ return bool(rx and rx.search(dg["_cmd_blob"]))
+
+ def _exercised(self, dg):
+ return self._lookup(self.scope, dg)
+
+ def _fired(self, dg):
+ return self._lookup(self.fired, dg)
+
+
+# ---- authored extractor: raw-event predicate -> named feature --------------
+class Extractor:
+ """A proposer-authored feature over raw calls.
+ kinds: cmd_regex (bool), out_regex (bool), cmd_count (int)."""
+ def __init__(self, spec):
+ self.name = spec["name"]
+ self.kind = spec.get("kind", "cmd_regex")
+ self.rx = _rx(spec.get("pattern"))
+ self.spec = spec
+
+ def compute(self, cmds):
+ if self.rx is None:
+ return False if self.kind != "cmd_count" else 0
+ if self.kind == "out_regex":
+ return any(self.rx.search(o) for _, o in cmds)
+ if self.kind == "cmd_count":
+ return sum(1 for c, _ in cmds if self.rx.search(c))
+ return any(self.rx.search(c) for c, _ in cmds)
+
+ def to_dict(self):
+ return self.spec
+
+
+# ---- generic fact: authored payload + validated lifecycle ------------------
+class Fact:
+ def __init__(self, spec):
+ self.id = spec["id"]
+ self.claim = spec.get("claim", spec["id"])
+ self.injection = spec.get("injection", self.claim)
+ self.high = bool(spec.get("high_stakes", False))
+ self.detector = Detector(spec.get("detector", {}))
+ self.spec = spec
+ self.life = _Fact(self.id, self.high) # the validated state machine
+
+ @property
+ def state(self):
+ return self.life.state
+
+ def injectable(self):
+ return self.life.injectable()
+
+ def firing_rate(self):
+ return self.life.firing_rate()
+
+ def observe(self, now, dg, files):
+ """Fold one finished-task digest into this fact's lifecycle. The detector is
+ rebuilt per task against that task's authored features."""
+ det = DetectorV3Scope(self.spec.get("detector", {}), dg.get("features", {}))
+ verdict, fired, exercised = det.verdict(dg)
+ self.life.update(now, verdict, fired, exercised, files)
+ return verdict, fired, exercised
+
+ def to_dict(self):
+ return {"spec": self.spec, "life": self.life.to_dict()}
+
+ @classmethod
+ def from_dict(cls, d):
+ f = cls(d["spec"])
+ f.life = _Fact.from_dict(d["life"])
+ return f
+
+
+# ---- proposer prompt (facts + extractors + revisable contradiction) --------
+PROPOSER_SYSTEM = (
+ "You mine a DURABLE, REPO-SPECIFIC operating manual for a coding agent from a "
+ "window of finished-task digests in ONE repository. Each digest lists edited "
+ "files, command templates, test invocations, and boolean flags — NOT raw logs.\n\n"
+ "Emit facts that a FUTURE task in this repo would want stated up front to avoid "
+ "wasted commands: the canonical test invocation, whether a repro is redundant, a "
+ "required env var, a build/reinstall step that's needed (or NOT needed), a lint/"
+ "format hook that reformats, etc. Do NOT emit task-specific fixes or which lines "
+ "to change — only durable interface facts true across tasks.\n\n"
+ "For EACH fact return an object with:\n"
+ ' id short kebab slug\n'
+ ' claim one sentence, what is true about this repo\n'
+ ' injection the exact line to inject into the agent prefix. State it as a '
+ "DECLARATIVE fact about the repo (where things are, what the canonical command is). "
+ "NEVER phrase it as a prohibition on exploration or tool use — do NOT tell the agent "
+ "to avoid `find`/`grep`/searching/reading, or to skip discovering files. A prior "
+ "that suppresses exploration silently breaks later tasks on a growing codebase where "
+ "the agent must discover accreted files. State what is true; never forbid a tool.\n"
+ ' high_stakes true if a wrong belief would break the run (test cmd, build step)\n'
+ ' detector {scope, confirm, contradict?, fired?} where scope is a python '
+ "regex over the command string (or one of: repro_written, importerror, badarg, "
+ "ran_tests) selecting tasks where the fact is RELEVANT; confirm is a regex whose "
+ "match in commands+output means the fact HELD this task; contradict (optional) a "
+ "regex meaning the fact was FALSE this task; fired (optional) a predicate/regex "
+ "meaning the waste the fact prevents actually occurred.\n"
+ ' cite list of >=2 iids from the window where confirm matches (grounding)\n\n'
+ "Return ONLY a JSON list of <=%d such objects. Be conservative: a fact you cannot "
+ "ground with a working detector + 2 citations is worse than no fact." % MAX_CANDS
+ + "\n\nADDITIONALLY: if a durable repo fact would need evidence NOT present in "
+ "the digest flags, first author an EXTRACTOR that surfaces it, then reference the "
+ "extractor's feature name as your detector scope/fired. Extractors are cheap "
+ "host-run predicates over raw commands/outputs.\n"
+ "Return a JSON OBJECT: {\"extractors\": [{name, kind, pattern}], \"facts\": [...]}\n"
+ " extractor.kind in {cmd_regex, out_regex, cmd_count}; pattern is a python regex.\n"
+ " A fact's detector.scope/fired may name any extractor you define here or a "
+ "built-in (repro_written, importerror, badarg, ran_tests)."
+ + "\n\nREVISION: the CONTRADICTION condition of an existing note may be corrected — "
+ "and nothing else. Re-emit a note with the SAME id and the SAME "
+ "claim/injection/high_stakes and the SAME detector.scope/detector.confirm/detector.fired, "
+ "changing ONLY detector.contradict, to fix or tighten the regex that marks the fact "
+ "FALSE this task. The note KEEPS its earned firing history and the new condition applies "
+ "going forward. Re-emitting an id while changing any other field is rejected; extractors "
+ "remain append-only."
+)
+
+
+def build_proposer_prompt(repo, window, existing_ids, existing_feats, raw_excerpts,
+ external=()):
+ view = [proposer_view(d) for d in window]
+ p = (PROPOSER_SYSTEM
+ + f"\n\nRepo: {repo}"
+ + "\n\nExisting fact ids (re-emit an id with ONLY detector.contradict changed to "
+ "revise its contradiction condition, else use a NEW id):\n"
+ + (", ".join(sorted(existing_ids)) or "(none)")
+ + "\n\nExisting feature names:\n" + (", ".join(sorted(existing_feats)) or "(none)")
+ + "\n\nTask digests (chronological):\n" + json.dumps(view, indent=1))
+ if raw_excerpts:
+ p += ("\n\nRAW command excerpts from recent tasks the current extractors did "
+ "NOT cover (author extractors to reach these if they hide a durable "
+ "fact):\n" + "\n".join(f" $ {c}" for c in raw_excerpts))
+ if external:
+ # Facts a separate, LLM-free layer already injects (the environment-fact profile).
+ # Telling the proposer is what keeps its paid budget off facts it would otherwise
+ # rediscover -- on the stored streams the interpreter fact was among the first it
+ # authored, every time.
+ p += ("\n\nFacts ALREADY injected before every task by a separate layer (do NOT "
+ "restate or paraphrase them; author only facts beyond these):\n"
+ + "\n".join(f" - {c}" for c in external))
+ p += "\n\nReturn the JSON object {extractors, facts}:"
+ return p
+
+
+def parse_payload(raw):
+ m = re.search(r"\{.*\}", raw or "", re.S)
+ if not m:
+ return [], []
+ try:
+ obj = json.loads(m.group(0))
+ except Exception:
+ return [], []
+ exts = [e for e in obj.get("extractors", [])
+ if isinstance(e, dict) and e.get("name") and e.get("pattern")]
+ facts = [f for f in obj.get("facts", [])
+ if isinstance(f, dict) and f.get("id") and f.get("detector")]
+ return exts, facts
+
+
+def _reparse_cmds(dg):
+ """Recover [(cmd,out)] from a digest's stashed blobs (for retro-featurize)."""
+ cmds = dg.get("_cmd_blob", "").split("\n")
+ outs = dg.get("_out_blob", "").split("\n")
+ return list(zip(cmds, outs + [""] * (len(cmds) - len(outs))))
+
+
+# ---- distiller -------------------------------------------------------------
+class Distiller:
+ """Learns and injects a per-repo operating manual over the task stream.
+
+ Public surface (the injection Stage calls these):
+ d = Distiller(repo, runner=..., model=...) # fresh, or
+ d = Distiller.load(repo, root, runner=..., model=...) # resume state
+ prefix = d.render() # manual for the NEXT task ("" if none)
+ d.update(transcript_path, iid=..., resolved=...) # fold in the finished task
+ d.save(root)
+ """
+ def __init__(self, repo, runner=None, model="agent", external_facts=None):
+ self.repo = repo
+ # Zero-arg callable returning claims another layer already injects (see
+ # build_proposer_prompt); None when the distiller runs alone.
+ self.external_facts = external_facts
+ self.now = 0 # task index within the stream
+ self.facts = {} # id -> Fact
+ self.window = [] # rolling digests since last propose
+ self.runner = runner
+ self.model = model
+ self.opt_tokens = 0.0 # cumulative effective proposer cost
+ self.last_propose = -1
+ self.extractors = {} # name -> Extractor
+ self.raw_window = [] # bounded recent [(cmd,out)] per task
+
+ # ---- featurization -----------------------------------------------------
+ def _observe_fact(self, fact, now, dg, files):
+ return fact.observe(now, dg, files)
+
+ def _featurize(self, dg, cmds):
+ feats = {}
+ for name, ex in self.extractors.items():
+ try:
+ feats[name] = ex.compute(cmds)
+ except Exception:
+ feats[name] = False
+ dg["features"] = feats
+ return dg
+
+ def _coverage(self, cmds):
+ """Fraction of commands matched by SOME extractor or built-in test/repro sense."""
+ if not cmds:
+ return 1.0
+ seen = 0
+ for c, _ in cmds:
+ hit = (PYTEST_RE.search(c) or REPRO_RE.search(c)
+ or any(ex.rx and ex.rx.search(c) for ex in self.extractors.values()))
+ if hit:
+ seen += 1
+ return seen / len(cmds)
+
+ def _abstaining_excerpts(self):
+ """Uncovered command templates from the raw window, for extractor authoring."""
+ out = []
+ for cmds in self.raw_window[-WINDOW:]:
+ for c, _ in cmds:
+ if PYTEST_RE.search(c) or REPRO_RE.search(c):
+ continue
+ if any(ex.rx and ex.rx.search(c) for ex in self.extractors.values()):
+ continue
+ t = re.sub(r"\s+", " ", c).strip()[:100]
+ if t and t not in out:
+ out.append(t)
+ return out[:RAW_SAMPLE * 4]
+
+ # ---- proposer tick (the only paid step) --------------------------------
+ def _maybe_propose(self):
+ if self.runner is None or not self.window:
+ return
+ cov = [self._coverage(c) for c in self.raw_window[-WINDOW:]]
+ abstaining = cov and (sum(cov) / len(cov)) < ABSTAIN_GATE
+ excerpts = self._abstaining_excerpts()[:RAW_SAMPLE] if abstaining else []
+ external = list(self.external_facts()) if self.external_facts else []
+ prompt = build_proposer_prompt(self.repo, self.window, set(self.facts),
+ set(self.extractors), excerpts, external=external)
+ raw, eff = self.runner(prompt, self.model)
+ if eff is not None:
+ self.opt_tokens += eff
+ exts, facts = parse_payload(raw)
+ for e in exts:
+ self._admit_extractor(e) # append-only
+ for spec in facts[:MAX_CANDS]:
+ self._admit(spec)
+ self.last_propose = self.now
+
+ def _admit_extractor(self, spec):
+ if spec["name"] in self.extractors or len(self.extractors) >= MAX_EXTRACTORS:
+ return False
+ ex = Extractor(spec)
+ if ex.rx is None:
+ return False
+ self.extractors[ex.name] = ex
+ # retro-featurize the window so a same-tick fact can reference it
+ for dg in self.window:
+ cmds = _reparse_cmds(dg)
+ dg.setdefault("features", {})[ex.name] = ex.compute(cmds)
+ return True
+
+ # ---- fact admission: new id = append; existing id = contradict-only revise --
+ def _ground(self, spec):
+ """#window cites whose confirm regex fires under the CURRENT features."""
+ by_iid = {d["iid"]: d for d in self.window}
+ hits = 0
+ for iid in (spec.get("cite") or []):
+ dg = by_iid.get(iid)
+ if dg is None:
+ continue
+ det = DetectorV3Scope(spec.get("detector", {}), dg.get("features", {}))
+ v, _, _ = det.verdict(dg)
+ if v == "confirm":
+ hits += 1
+ return hits
+
+ def _build_fact(self, spec):
+ """Fresh Fact replayed over the window -> enters with real lifecycle state."""
+ f = Fact(spec)
+ for i, dg in enumerate(self.window):
+ self._observe_fact(
+ f, self.now - len(self.window) + i, dg,
+ set(dg["edited_files"]))
+ return f
+
+ def _frozen_view(self, spec):
+ """Everything about a note EXCEPT detector.contradict (the append-only surface)."""
+ det = dict(spec.get("detector", {}))
+ det.pop("contradict", None)
+ return {
+ "claim": spec.get("claim"),
+ "injection": spec.get("injection"),
+ "high_stakes": bool(spec.get("high_stakes", False)),
+ "detector": json.dumps(det, sort_keys=True),
+ }
+
+ def _admit(self, spec):
+ fid = spec.get("id")
+ if fid not in self.facts:
+ hits = self._ground(spec)
+ if hits < MIN_CITE:
+ return False, f"grounding-fail({hits}/{MIN_CITE})"
+ spec.setdefault("rev", 1)
+ self.facts[fid] = self._build_fact(spec)
+ return True, "admitted"
+
+ old = self.facts[fid]
+ # ONLY the contradiction condition may be revised; everything else is frozen.
+ if self._frozen_view(spec) != self._frozen_view(old.spec):
+ return False, "revise-not-contradict-only"
+ new_contra = spec.get("detector", {}).get("contradict")
+ if new_contra and _rx(new_contra) is None:
+ return False, "revise-bad-regex"
+ old_contra = old.spec.get("detector", {}).get("contradict")
+ if new_contra == old_contra:
+ return False, "revise-noop"
+ # confirm/scope unchanged => still grounded; contradict only ever RETIRES =>
+ # mutate in place, KEEP the earned lifecycle, apply going forward.
+ old.spec.setdefault("detector", {})["contradict"] = new_contra
+ old.spec["rev"] = old.spec.get("rev", 1) + 1
+ return True, "revised-contradict"
+
+ # ---- the two operations the Stage calls --------------------------------
+ def render(self, task=None):
+ """Firing-gated manual for the NEXT task. '' if nothing qualifies yet."""
+ lines = [f.injection for f in self.facts.values() if f.injectable()]
+ if not lines:
+ return ""
+ head = (f"# repo operating manual ({self.repo.split('/')[-1]}) "
+ f"— learned from prior tasks in this repo")
+ return head + "\n" + "\n".join(f"- {l}" for l in lines) + "\n"
+
+ def update(self, transcript_path, iid="", resolved=None, edited=None):
+ """Fold in the finished task, then advance the clock (may trigger a paid tick)."""
+ try:
+ calls = load_calls(transcript_path)
+ except Exception:
+ self.now += 1
+ return
+ cmds = bash_cmds(calls)
+ dg = build_digest(self.repo, transcript_path, iid, resolved)
+ if dg is None:
+ self.now += 1
+ return
+ self._featurize(dg, cmds)
+ self.raw_window.append(cmds)
+ if len(self.raw_window) > WINDOW:
+ self.raw_window = self.raw_window[-WINDOW:]
+ files = set(dg["edited_files"])
+ for f in self.facts.values():
+ self._observe_fact(f, self.now, dg, files)
+ self.window.append(dg)
+ if len(self.window) > WINDOW:
+ self.window = self.window[-WINDOW:]
+ self.now += 1
+ if self.now - self.last_propose >= PROPOSE_EVERY:
+ self._maybe_propose()
+
+ # ---- persistence -------------------------------------------------------
+ @staticmethod
+ def _path(root, repo):
+ d = os.path.join(root, STATE_SUBDIR)
+ os.makedirs(d, exist_ok=True)
+ return os.path.join(d, repo.replace("/", "__") + ".json")
+
+ def save(self, root):
+ json.dump({"repo": self.repo, "now": self.now, "opt_tokens": self.opt_tokens,
+ "last_propose": self.last_propose,
+ "extractors": {k: e.to_dict() for k, e in self.extractors.items()},
+ "facts": {k: f.to_dict() for k, f in self.facts.items()}},
+ open(self._path(root, self.repo), "w"), indent=2)
+
+ @classmethod
+ def load(cls, repo, root, runner=None, model="agent", external_facts=None):
+ p = cls._path(root, repo)
+ self = cls(repo, runner, model, external_facts=external_facts)
+ if not os.path.exists(p):
+ return self
+ d = json.load(open(p))
+ self.now = d.get("now", 0)
+ self.opt_tokens = d.get("opt_tokens", 0.0)
+ self.last_propose = d.get("last_propose", -1)
+ self.extractors = {k: Extractor(v) for k, v in d.get("extractors", {}).items()}
+ self.facts = {k: Fact.from_dict(v) for k, v in d.get("facts", {}).items()}
+ return self
+
+ def debug_row(self):
+ row = {k: {"state": f.state, "fire": f"{f.life.fire_num}/{f.life.fire_den}",
+ "rate": round(f.firing_rate(), 2), "inject": f.injectable(),
+ "claim": f.claim[:60]}
+ for k, f in self.facts.items()}
+ row["_extractors"] = sorted(self.extractors)
+ for fid, f in self.facts.items():
+ if isinstance(row.get(fid), dict):
+ row[fid]["rev"] = f.spec.get("rev", 1)
+ return row
+
+
+# ---- composition: the free environment layer under the paid manual -------------------
+class ComposedDistiller:
+ """`envfacts` first, the paid manual on top, driven as one distiller.
+
+ The environment layer renders first (its block is what the proposer is told not to
+ restate); both fold every finished task; each persists in its own state dir. Overhead is
+ the proposer's alone -- the environment layer is LLM-free by construction."""
+ def __init__(self, env, dist):
+ self.env, self.dist = env, dist
+
+ @property
+ def opt_tokens(self):
+ return self.dist.opt_tokens
+
+ @property
+ def now(self):
+ return self.dist.now
+
+ def render(self, task=None):
+ a = self.env.render(task)
+ b = self.dist.render(task)
+ return (a + ("\n" if a and b else "") + b) if (a or b) else ""
+
+ def update(self, transcript_path, iid="", resolved=None, edited=None):
+ self.env.update(transcript_path, iid=iid, resolved=resolved, edited=edited)
+ self.dist.update(transcript_path, iid=iid, resolved=resolved, edited=edited)
+
+ def save(self, root):
+ self.env.save(root)
+ self.dist.save(root)
+
+ def debug_row(self):
+ row = dict(self.dist.debug_row())
+ row.update(self.env.debug_row())
+ row["_profile"] = "envfacts+distiller"
+ return row
+
+
+# ---- lifecycle stage -------------------------------------------------------
+class PrefixStage(Stage):
+ """Injection lifecycle hook: render the manual before a task, fold the finished
+ task back into the distiller after it, and report the paid proposer's per-task
+ token delta for the amortized ledger."""
+ name = "prefix"
+
+ def __init__(self, cfg, repo, results_root, fresh,
+ runner=None, optimizer_model="agent", adapter=None, agent=None):
+ self.results_root = results_root
+ self.adapter = adapter
+ self.agent = agent
+ # The Distiller is the single source of truth for the proposer cadence: rebind
+ # the module global that update()'s gate reads (process-local, idempotent).
+ pe = int(getattr(cfg, "propose_every", 0) or 0)
+ if pe > 0:
+ global PROPOSE_EVERY
+ PROPOSE_EVERY = pe
+ profile = getattr(cfg, "profile", "distiller") or "distiller"
+ # The environment-fact layer reads intents through a domain vocabulary; the adapter
+ # supplies it when it has one (SWE-bench: Python/git; Web-Bench: Node/npm).
+ vocab = adapter.env_vocabulary() if hasattr(adapter, "env_vocabulary") else None
+ env = None
+ if "envfacts" in profile:
+ from .envfacts import EnvFactsDistiller
+ caps = {"max_facts": getattr(cfg, "max_facts", 0) or None,
+ "max_tokens": getattr(cfg, "max_tokens", 0) or None}
+ env = (EnvFactsDistiller(repo, vocab=vocab, **caps) if fresh
+ else EnvFactsDistiller.load(repo, results_root, vocab=vocab, **caps))
+ if profile == "envfacts":
+ # LLM-free environment facts alone. Same render/update/save interface, no
+ # runner, opt_tokens stays 0 -- the ledger sees no overhead.
+ self.reason = "envfacts"
+ self.distiller = env
+ else:
+ # The paid proposer; with `envfacts+distiller` it is told which claims the free
+ # layer already injects, so it authors only facts beyond them.
+ external = env.injectable_claims if env is not None else None
+ dist = (Distiller(repo, runner=runner, model=optimizer_model,
+ external_facts=external) if fresh
+ else Distiller.load(repo, results_root, runner=runner,
+ model=optimizer_model, external_facts=external))
+ if env is not None:
+ self.reason = "envfacts+manual-distill"
+ self.distiller = ComposedDistiller(env, dist)
+ else:
+ self.reason = "manual-distill"
+ self.distiller = dist
+
+ def render(self, task=None):
+ """The firing-gated repo manual injected into the NEXT task's prefix."""
+ return self.distiller.render(task=task) if self.distiller else ""
+
+ def after_task(self, task, row, run_dir_path, srow):
+ """Fold the finished task into the fact lifecycle; return (opt_tokens, reason).
+ The paid proposer accrues cumulatively on distiller.opt_tokens; capture THIS
+ task's delta for the per-task ledger (same effective-token basis)."""
+ transcript = os.path.join(run_dir_path, "transcript.jsonl")
+ before = self.distiller.opt_tokens
+ self.distiller.update(transcript, iid=task["instance_id"],
+ resolved=srow.get("resolved"))
+ self.distiller.save(self.results_root)
+ opt_tokens = self.distiller.opt_tokens - before
+ srow["manual_facts"] = self.distiller.debug_row()
+ return opt_tokens, self.reason
diff --git a/tokeneconomizer/mechanisms/spiral.py b/tokeneconomizer/mechanisms/spiral.py
new file mode 100644
index 0000000..e368254
--- /dev/null
+++ b/tokeneconomizer/mechanisms/spiral.py
@@ -0,0 +1,132 @@
+"""Spiral mechanism: kill or nudge a stuck rollout in-flight, learning a per-repo
+resolved-turn envelope so the intervention is causal and resolve-safe.
+
+Two paths, chosen by agent:
+
+ * GENERIC spiral guard (Claude / Copilot) -- `SpiralConfig.guard`:
+ Streams each rollout; a cheap LLM-free gate, an optional sampled LLM trace-judge
+ (`spiral_judge`), and a per-repo p90 resolved-turn envelope decide the cheapest
+ sufficient action. Rung 1 NUDGES (re-invokes in the same container with a
+ leakage-safe redirect built from the agent's own repeated actions; with
+ `nudge_llm`, an LLM writes the redirect); a second stuck detection ABORTS. The
+ envelope + learned thresholds advance AFTER each task (this task's outcome only
+ gates FUTURE tasks). Judge / nudge-writer tokens fold into the honest ledger.
+
+ * CODEX-NATIVE post-green finalize (Codex) -- `SpiralConfig.codex_spiral`:
+ Uses Codex's deterministic signals (exit codes, thread ids, per-file change events)
+ to fire ONE same-thread finalize nudge once a task's own success check has passed
+ UNFILTERED and every ever-failed scope has been re-greened, past the learned
+ envelope. Never terminally aborts. Policy = codex_spiral.SHIPPED_POLICY at the
+ repo's task-frozen threshold (a `CodexEnvelopeTracker`).
+
+The Stage supplies each path's params to the runner via `run_kwargs`; the runner owns the
+actual in-flight stream/kill/resume. `after_task` records telemetry and advances the
+envelope unconditionally once the task has run.
+"""
+import os
+
+from .base import Stage
+from ..vendor import reflect, spiral
+from . import codex_spiral as cs
+
+_CODEX_AGENTS = ("codex",)
+
+
+class SpiralStage(Stage):
+ name = "spiral"
+
+ def __init__(self, cfg, max_turns, runner=None, optimizer_model="agent",
+ adapter=None, agent=None):
+ self.cfg = cfg
+ self.max_turns = max_turns
+ self.adapter = adapter
+ self.agent = agent
+
+ # ---- generic guard: per-repo envelope + host-side LLM closures ----
+ self.envelope = spiral.EnvelopeTracker(
+ level=cfg.aggr, frozen_envelope=getattr(cfg, "frozen_envelope", None))
+ self.spiral_judge = None
+ self.spiral_nudge_writer = None
+ if cfg.guard and cfg.judge:
+ self.spiral_judge = (
+ lambda digest, _r=runner, _m=optimizer_model:
+ reflect.llm_stuck_judge(digest, _r, model=_m))
+ if cfg.nudge and cfg.nudge_llm:
+ self.spiral_nudge_writer = (
+ lambda digest, _r=runner, _m=optimizer_model:
+ reflect.llm_nudge_writer(digest, _r, model=_m))
+
+ # ---- Codex-native post-green finalize: per-repo threshold tracker ----
+ self.codex_tracker = None
+ if cfg.codex_spiral and agent in _CODEX_AGENTS:
+ self.codex_tracker = cs.CodexEnvelopeTracker()
+
+ def run_kwargs(self, task):
+ out = {}
+ if self.cfg.guard:
+ out.update({
+ "spiral_guard": self.cfg.guard,
+ "abort_envelope": self.envelope.envelope,
+ "spiral_thr": self.envelope.thresholds(max_turns=self.max_turns),
+ "spiral_judge": self.spiral_judge,
+ "spiral_nudge": self.cfg.nudge,
+ "spiral_nudge_writer": self.spiral_nudge_writer,
+ })
+ if self.codex_tracker is not None:
+ threshold = self.codex_tracker.threshold
+ out["codex_spiral_policy"] = (
+ cs.profile_policy(self.cfg.codex_spiral_profile, threshold)
+ if threshold is not None else None)
+ return out
+
+ def after_task(self, task, row, run_dir_path, srow):
+ cfg = self.cfg
+ transcript = os.path.join(run_dir_path, "transcript.jsonl")
+
+ # ---- generic guard telemetry + envelope advance (causal) ----
+ if cfg.guard:
+ srow["aborted"] = bool(row.get("aborted"))
+ srow["spiral_judge_tokens"] = row.get("spiral_judge_tokens") or 0
+ if row.get("spiral_abort"):
+ srow["spiral_abort"] = row["spiral_abort"]
+ if cfg.nudge:
+ srow["spiral_nudges"] = row.get("spiral_nudges", 0)
+ if row.get("spiral_events"):
+ srow["spiral_events"] = row["spiral_events"]
+ if cfg.nudge_llm:
+ srow["spiral_nudge_writer_tokens"] = (
+ row.get("spiral_nudge_writer_tokens") or 0)
+ snap = None
+ if row.get("resolved") and self.adapter is not None:
+ try:
+ snap = self.adapter.spiral_snapshot(transcript)
+ except Exception:
+ snap = None
+ self.envelope.update(row.get("num_turns") or 0,
+ bool(row.get("resolved")), snapshot=snap)
+
+ # ---- Codex-native post-green telemetry + envelope advance ----
+ if self.codex_tracker is not None:
+ srow["codex_spiral_active"] = True
+ srow["codex_spiral_profile"] = cs.profile_version(self.cfg.codex_spiral_profile)
+ srow["codex_spiral_nudges"] = row.get("codex_spiral_nudges", 0)
+ if row.get("codex_spiral_events"):
+ srow["codex_spiral_events"] = row["codex_spiral_events"]
+ srow["spiral_nudge_writer_tokens"] = (
+ row.get("codex_spiral_paid_judge_tokens") or 0)
+ # Advance the envelope in the unit the TRIGGER uses: the guard's own
+ # command+edit count, not the row's `num_turns` (agent steps = messages + tool
+ # calls, ~1.45x larger). Feeding steps raised the gate ~45% and left the
+ # controller unable to fire on most tasks; `CodexEnvelopeTracker` documents the
+ # tool-event unit and now actually receives it. Fall back to `num_turns` only
+ # when a row predates the field, and say so through the telemetry rather than
+ # silently mixing units.
+ tool_turns = row.get("codex_spiral_tool_turns")
+ srow["codex_spiral_envelope_unit"] = (
+ "tool_events" if tool_turns is not None else "steps_fallback")
+ srow["codex_spiral_tool_turns"] = tool_turns
+ srow["codex_spiral_threshold"] = self.codex_tracker.threshold
+ self.codex_tracker.update(
+ int(tool_turns if tool_turns is not None else (row.get("num_turns") or 0)),
+ bool(row.get("resolved")))
+ return 0
diff --git a/tokeneconomizer/mechanisms/tooling.py b/tokeneconomizer/mechanisms/tooling.py
new file mode 100644
index 0000000..439ee1c
--- /dev/null
+++ b/tokeneconomizer/mechanisms/tooling.py
@@ -0,0 +1,204 @@
+"""Tooling mechanism (untooling): drop inapplicable tool schemas from the per-turn prefix.
+
+A sealed agent's tool catalog is re-sent every turn and billed every turn, but on any one
+task most tools are inapplicable. Untooling removes them. Two paths, chosen by agent:
+
+ * GENERIC evidence untooling (Claude / Copilot) -- `ToolingConfig.evidence`:
+ A tool is dropped once it has been LOADED-BUT-UNUSED for `evidence` consecutive
+ tasks (never dropping a protected work-surface tool or one ever seen in use). With
+ `judge=True`, a per-task instruction-aware LLM judge additionally drops tools its
+ (redacted) issue text makes irrelevant. Periodic re-audit reloads the full catalog.
+ All judge/classify tokens fold into the honest ledger.
+
+ * CODEX-NATIVE capability untooling (Codex) -- `ToolingConfig.codex_untool`:
+ Removes whole inapplicable capability PACKAGES from Codex's request contract via the
+ evidence-gated, resolve-safe controller in `codex_untool.CodexUntoolProfile`. The
+ Stage computes the frozen capability mask; the runner maps it onto Codex config/relay
+ prune-items (that mapping is the runner's job, keeping this module harness-free).
+
+Untool state (`used_tools`, `tool_unused_streaks`, catalog snapshot) accrues every task
+even when untooling is off, so a run that toggles the switch is never wrong.
+"""
+import os
+
+from .base import Stage
+from ..vendor import reflect
+from . import codex_untool as cu
+
+_CODEX_AGENTS = ("codex",)
+
+
+class ToolingStage(Stage):
+ name = "tooling"
+
+ def __init__(self, cfg, agent, optimizer_model,
+ untool_runner=None, judge_tool_runner=None, adapter=None,
+ repo=None, results_root=None, fresh=False):
+ self.cfg = cfg
+ self.agent = agent
+ self.optimizer_model = optimizer_model
+ self.judge_tool_runner = judge_tool_runner
+ self.adapter = adapter
+
+ # ---- shared untool state (owned here, mutated across tasks) ----
+ self.used_tools = set() # tools observed in use across the stream so far
+ self.tool_unused_streaks = {} # {tool: consecutive tasks unused while loaded}
+ self.catalog_tools = set() # every tool ever seen in the full catalog
+ self._warned_no_catalog = False # one loud warning per stream, not per task
+ self.catalog_desc = {} # {tool: description} for the per-task judge
+
+ # ---- Codex-native capability controller ----
+ self.codex_profile = None
+ self._codex_selection = None
+ if cfg.codex_untool and agent in _CODEX_AGENTS:
+ if not repo or not results_root:
+ raise ValueError("codex untooling requires repo and results_root")
+ self.codex_profile = cu.CodexUntoolProfile(repo, results_root, fresh=fresh)
+
+ # ---- generic per-task instruction-aware judge: redacted-issue lookup ----
+ self.problem_by_iid = {}
+ if cfg.judge and adapter is not None:
+ try:
+ self.problem_by_iid = adapter.problem_statements()
+ except Exception as e:
+ print(f"[untool-judge] could not load problem statements ({e}); "
+ f"judge disabled")
+ self.problem_by_iid = {}
+
+ # per-task locals stashed by build_tools for after_task
+ self._ev_reaudit = False
+ self._exclude_tools = None
+ self._task_judge_tokens = 0
+ self._task_judge_dropped = []
+
+ # -- construction: this task's tool-exclusion set / codex capability mask --
+ def build_tools(self, task, i):
+ iid = task["instance_id"]
+ cfg = self.cfg
+ self._task_judge_tokens = 0
+ self._task_judge_dropped = []
+ self._ev_reaudit = False
+
+ # Codex path: freeze the capability mask; the runner maps it to prune-items.
+ if self.codex_profile is not None:
+ self._codex_selection = self.codex_profile.begin_task(
+ iid, problem_statement=task.get("problem_statement", ""))
+ self._exclude_tools = list(
+ self._codex_selection.get("relay_capabilities") or ())
+ return list(self._exclude_tools)
+
+ # Generic evidence path (Claude / Copilot).
+ if cfg.evidence:
+ ev_reaudit = (cfg.evidence_reaudit > 0 and i > 0
+ and i % cfg.evidence_reaudit == 0)
+ self._ev_reaudit = ev_reaudit
+ if ev_reaudit:
+ exclude_tools = []
+ else:
+ protected = self.adapter.protected_tools() if self.adapter else frozenset()
+ trace_excluded = [
+ t for t in self.catalog_tools
+ if t not in protected and t not in self.used_tools
+ and self.tool_unused_streaks.get(t, 0) >= cfg.evidence]
+ if cfg.judge and self.catalog_tools and self.problem_by_iid.get(iid):
+ cand = {t: self.catalog_desc.get(t, "") for t in self.catalog_tools
+ if t not in protected and t not in self.used_tools
+ and t not in trace_excluded}
+ if cand:
+ loadmap, self._task_judge_tokens = reflect.judge_tools_for_task(
+ self.problem_by_iid[iid], cand, self.judge_tool_runner,
+ model=self.optimizer_model)
+ self._task_judge_dropped = [t for t, load in loadmap.items()
+ if not load and t in cand]
+ exclude_tools = sorted(set(trace_excluded) | set(self._task_judge_dropped))
+ self._exclude_tools = exclude_tools
+ return exclude_tools
+
+ self._exclude_tools = None
+ return None
+
+ def run_kwargs(self, task):
+ """Hand the runner the frozen mask's CONFIG half (generic untooling needs none).
+
+ The relay half already crosses the seam as `build_tools`' return value, but the two
+ are not interchangeable and the config half cannot be recovered from it: the
+ `collaboration` package expands to NO relay id (see `codex_untool._RELAY_CAPABILITIES`),
+ so a runner reading only `exclude_tools` would prune it from nothing while the row
+ reported it pruned -- an overstated saving on the arm whose whole claim is the
+ saving. Passing the logical capability names keeps Layer 1 free of any Codex
+ config knowledge: which toggles remove a package is the runner's business, because
+ the runner is what owns the request contract.
+ """
+ if self.codex_profile is None:
+ return {}
+ selection = self._codex_selection or {}
+ return {"codex_config_capabilities": list(
+ selection.get("config_capabilities") or ())}
+
+ # -- post-rollout: accrue usage + mechanism bookkeeping --
+ def after_task(self, task, row, run_dir_path, srow):
+ cfg = self.cfg
+ transcript = os.path.join(run_dir_path, "transcript.jsonl")
+ task_used = (set(self.adapter.tool_usage(transcript).keys())
+ if self.adapter else set())
+ self.used_tools.update(task_used)
+
+ # ---- Codex-native capability untooling ----
+ if self.codex_profile is not None:
+ selection = self._codex_selection or {}
+ usage = cu.inspect_usage(transcript)
+ record = self.codex_profile.observe(usage, resolved=srow.get("resolved"))
+ srow["codex_untool_active"] = True
+ srow["codex_untool_profile"] = cu.PROFILE_VERSION
+ srow["codex_untool_reason"] = selection.get("reason")
+ srow["codex_untool_config_capabilities"] = list(
+ selection.get("config_capabilities") or ())
+ srow["codex_untool_relay_capabilities"] = list(
+ selection.get("relay_capabilities") or ())
+ srow["codex_untool_core_capabilities"] = list(
+ selection.get("core_capabilities") or ())
+ srow["codex_untool_newly_pruned"] = record["newly_pruned"]
+ srow["codex_untool_mask_after"] = record["mask_after"]
+ srow["codex_untool_usage_attempted"] = record["attempted"]
+ srow["codex_untool_usage_completed"] = record["completed"]
+ srow["codex_untool_adapt_tokens"] = 0
+ return 0
+
+ # ---- generic evidence untooling bookkeeping ----
+ if cfg.evidence:
+ srow["untooled_excluded"] = row.get("untooled_excluded", [])
+ srow["untool_evidence_excluded"] = list(self._exclude_tools or [])
+ srow["untool_evidence_reaudit"] = bool(self._ev_reaudit)
+ if not self._exclude_tools and self.adapter is not None:
+ # Source-aware where the adapter supports it: an empty catalog is
+ # indistinguishable from "this agent loaded no tools", and that ambiguity
+ # is exactly how a Copilot CLI upgrade silently turned untooling into a
+ # no-op for a whole sweep (trace.catalog_sources). Record which source won
+ # so the next migration shows up in the rows, and say so out loud the first
+ # time nothing parses.
+ getter = getattr(self.adapter, "parse_catalog_with_source", None)
+ if getter is not None:
+ cat, src = getter(run_dir_path, self.agent)
+ else:
+ cat, src = self.adapter.parse_catalog(run_dir_path, self.agent), "unknown"
+ srow["untool_catalog_source"] = src
+ if cat:
+ self.catalog_tools.update(cat.keys())
+ self.catalog_desc.update(cat)
+ elif not self.catalog_tools and not self._warned_no_catalog:
+ self._warned_no_catalog = True
+ print(f"[tooling] !! no tool catalog parsed for agent={self.agent} "
+ f"from {run_dir_path} -- evidence untooling can drop NOTHING and "
+ f"this arm will behave as its baseline. Likely an agent-CLI "
+ f"upgrade that moved the catalog; see trace.catalog_sources.",
+ flush=True)
+ for t in self.catalog_tools:
+ if t in task_used:
+ self.tool_unused_streaks[t] = 0
+ else:
+ self.tool_unused_streaks[t] = self.tool_unused_streaks.get(t, 0) + 1
+ srow["untool_catalog_size"] = len(self.catalog_tools)
+ if cfg.judge:
+ srow["untool_judge_tokens"] = self._task_judge_tokens or 0
+ srow["untool_judge_dropped"] = list(self._task_judge_dropped or [])
+ return 0
diff --git a/tokeneconomizer/runner.py b/tokeneconomizer/runner.py
new file mode 100644
index 0000000..320e53a
--- /dev/null
+++ b/tokeneconomizer/runner.py
@@ -0,0 +1,84 @@
+"""AgentRunner: the seam between the economizer and the task-execution harness.
+
+The economizer is harness-agnostic. It never launches Docker, calls an LLM API, or shells
+out to a coding-agent CLI itself -- that heavy, environment-specific machinery lives BELOW
+this interface. The economizer needs only two capabilities from a runner:
+
+ 1. an OPTIMIZER call -- a small paid LLM request used by the mechanisms' between-task
+ proposer / tool-judge / spiral-judge / nudge-writer. This is the ``runner(prompt,
+ model) -> (text, effective_tokens)`` callable the mechanisms already expect.
+
+ 2. (optionally) a TASK EXECUTOR -- run one benchmark instance under a chosen arm and the
+ economizer's per-task decisions (injected prefix, tool-exclusion set / capability
+ mask, spiral params), returning a normalized row + the path to the saved transcript.
+
+A concrete runner (Docker + claude/codex/copilot CLI + official gold grading) ships in the
+repo's `te_harness` package -- the one the published rows were produced with
+(experiments/docs/HARNESS.md);
+its image and dataset paths are env-var placeholders. Supply a different one with
+``--runner your_pkg.harness:make_runner``; nothing here changes, because the economizer
+never learns which harness it is talking to.
+
+`te_harness` is a SIBLING package, never imported from anywhere under `tokeneconomizer`, and
+excluded from the distributed wheel. This package depends only on the protocol below, so
+`import tokeneconomizer` cannot pull in Docker or an LLM SDK -- a property of the package
+boundary, not of anyone's discipline in respecting it.
+
+The row/transcript contract the mechanisms read:
+ * transcript.jsonl written under ``run_dir_path`` (trace-reading seam consumes it);
+ * row: a dict with at least the token-breakdown fields pricing.effective_tokens reads
+ (input_tokens/cache_read_input_tokens/cache_creation_input_tokens/output_tokens or
+ tokens_total), plus ``resolved``, ``num_turns``, and any mechanism telemetry fields
+ the runner surfaces (aborted / spiral_* / codex_* / output_compression, ...).
+"""
+from typing import Optional, Protocol, Tuple, runtime_checkable
+
+
+# The optimizer-call signature the mechanisms invoke: (prompt, model) -> (text, eff_tokens)
+# where eff_tokens is the effective (cache-aware) token cost of the call, or None.
+OptimizerRunner = "Callable[[str, str], Tuple[str, Optional[float]]]"
+
+
+@runtime_checkable
+class AgentRunner(Protocol):
+ """Everything the economizer needs from the task-execution harness."""
+
+ def optimizer_call(self, prompt: str, model: str = "agent") -> Tuple[str, Optional[float]]:
+ """Run one small paid LLM request (the between-task proposer / judge / nudge).
+ Returns (raw_text, effective_tokens). effective_tokens may be None if unknown."""
+ ...
+
+ def run_task(self, task: dict, *, arm: str, injected: Optional[str],
+ exclude_tools, run_dir_path: str, **run_kwargs) -> Tuple[dict, str]:
+ """Execute one benchmark instance under `arm` with the economizer's per-task
+ decisions applied. Returns (row, transcript_run_dir). Optional: a runner used only
+ for optimizer calls need not implement this."""
+ ...
+
+ # ---- optional: stream-level lifecycle ------------------------------------------
+ # Most harnesses own nothing wider than a single task and can ignore these. A harness
+ # that owns the whole stream implements them, and the driver calls them around the
+ # task loop (`hasattr`-guarded, so they stay optional): Web-Bench's evaluator, for
+ # example, drives a project's 20 sequentially dependent tasks itself, so it must be
+ # told the stream before the first task and shut down after the last.
+ def begin_stream(self, tasks: list, **info) -> None:
+ """Called once with the full task stream before the first run_task. `info` carries
+ the driver's context (results_root, arm, repo, seed)."""
+ ...
+
+ def end_stream(self) -> None:
+ """Called once after the last run_task, including when the loop raised."""
+ ...
+
+
+def as_optimizer_callable(runner):
+ """Adapt an AgentRunner (or a bare callable) into the ``(prompt, model) ->
+ (text, eff_tokens)`` callable the mechanisms expect. Returns None for None so the
+ LLM-free paths (disabled proposer/judge) stay no-ops."""
+ if runner is None:
+ return None
+ if hasattr(runner, "optimizer_call"):
+ return lambda prompt, model="agent": runner.optimizer_call(prompt, model)
+ if callable(runner):
+ return runner
+ raise TypeError(f"cannot use {runner!r} as an optimizer runner")
diff --git a/tokeneconomizer/trace.py b/tokeneconomizer/trace.py
new file mode 100644
index 0000000..d396845
--- /dev/null
+++ b/tokeneconomizer/trace.py
@@ -0,0 +1,161 @@
+"""Single import point for trace reading -- the one normalized seam every mechanism
+reads agent trajectories through.
+
+All three mechanisms (injection / tooling / spiral) consume a saved rollout transcript
+through the four functions below, so the transcript-schema knowledge lives in exactly one
+place. The heavy lifting is delegated to the vendored `reflect` reader, which auto-detects
+the supported transcript schemas (Claude Code / Copilot CLI).
+
+`parse_catalog` is the only agent-specific branch: it returns the tool catalog the agent
+loaded for a task. For Codex there is no per-task JSON tool dump, so we return the known
+default surface (inlined below, so this package needs no runner/`agents` dependency).
+"""
+import json
+import os
+import re
+
+from .vendor import reflect
+
+# Codex ships a fixed default tool surface (it does not dump JSON tool schemas into a
+# debug log). Inlined here -- verbatim from the original CodexDriver.DEFAULT_TOOL_CATALOG
+# -- so catalog-size telemetry reflects what is actually loaded (and prunable) without a
+# dependency on the Docker/CLI runner layer.
+CODEX_DEFAULT_TOOL_CATALOG = {
+ "shell": "Run shell commands to inspect, edit, and test the repository.",
+ "apply_patch": "Apply source-code file changes.",
+ "web_search": "Search the web (inapplicable on air-gapped SWE-bench).",
+ "view_image": "Attach a local image to context (unused on text tasks).",
+ "update_plan": "Maintain a step plan (scaffolding, unused here).",
+ "spawn_agent": "Spawn a collaborating subagent.",
+ "send_input": "Send input to a collaborating subagent.",
+ "resume_agent": "Resume a collaborating subagent.",
+ "wait_agent": "Wait for a collaborating subagent.",
+ "close_agent": "Close a collaborating subagent.",
+}
+
+
+# rtk in its Codex mode is the one scaffold that changes the COMMAND LINE rather than hooking
+# it: the agent is told to type `rtk pytest`, `rtk git status`, `rtk proxy <cmd>`. Every
+# classifier that reads a command's first token -- the environment-fact vocabulary, the
+# spiral's test-run detector -- would otherwise stop seeing a test run, a fetch or a repo
+# query on exactly the arm that co-activates them (`codex-trimech-rtk`). One definition, so
+# the classifiers cannot disagree about what counts as the wrapper. `rtk proxy` is rtk's own
+# passthrough (run unfiltered, still tracked); a bare `rtk` with no command is left alone.
+RTK_PREFIX_RE = re.compile(r"^rtk\s+(?:proxy\s+)?(?=\S)")
+
+
+def strip_rtk_prefix(command):
+ """`rtk pytest -q` -> `pytest -q`; anything else unchanged."""
+ return RTK_PREFIX_RE.sub("", (command or "").lstrip(), count=1)
+
+
+def load_trajectory(path):
+ """Ordered [(tool_name, args, result_text, success)] over a saved transcript.jsonl.
+ Auto-detects the supported agent transcript schemas (delegates to reflect)."""
+ return reflect.load_trajectory(path)
+
+
+def tool_usage(path):
+ """{tool_name: count} over a saved transcript (delegates to reflect.tool_usage)."""
+ return reflect.tool_usage(path)
+
+
+def detect(path):
+ """LLM-free signal dict over a saved transcript (delegates to reflect.detect)."""
+ return reflect.detect(path)
+
+
+# Ordered catalog sources for the Copilot CLI. The CLI has changed where it publishes the
+# loaded tool catalog at least twice, and each time the OLD reader started returning {} --
+# which untooling cannot distinguish from "this agent loaded no tools", so it degraded to a
+# silent no-op (see reflect.parse_tool_catalog). So instead of one reader with a fallback,
+# every known source is tried and the RICHEST result wins: descriptions beat names, more
+# tools beat fewer. That way one pipeline serves every CLI build we have seen, chooses per
+# run dir at runtime, and a build that publishes two sources is not at the mercy of which
+# one we happened to try first.
+#
+# schema_debug_log {name: desc} CLI <= ~1.0.81 the only source carrying descriptions
+# usage_checkpoint {name: ""} CLI ~1.0.82 transcript session.usage_checkpoint
+# flat_debug_log {name: ""} CLI >= 1.0.82 "tools": ["bash:66921ccc", ...]
+#
+# `catalog_sources` returns every source that produced anything, so a caller can record
+# which one won and notice the next migration from the rows instead of from a null result.
+def _read(path):
+ try:
+ return open(path, encoding="utf-8", errors="replace").read()
+ except OSError:
+ return ""
+
+
+def catalog_sources(run_dir_path, agent):
+ """[(source_name, {tool: desc}), ...] for every catalog source that yielded tools."""
+ if agent == "codex":
+ return [("codex_default", dict(CODEX_DEFAULT_TOOL_CATALOG))]
+ if agent == "claude":
+ cat = reflect.parse_tool_catalog_claude(
+ _read(os.path.join(run_dir_path, "transcript.jsonl")))
+ return [("claude_transcript", cat)] if cat else []
+ dbg = _read(os.path.join(run_dir_path, "copilot_debug.log"))
+ tr = _read(os.path.join(run_dir_path, "transcript.jsonl"))
+ out = []
+ for name, cat in (("schema_debug_log", reflect.parse_tool_catalog(dbg)),
+ ("usage_checkpoint", parse_tool_names_copilot(tr)),
+ ("flat_debug_log", reflect.parse_tool_catalog_flat(dbg))):
+ if cat:
+ out.append((name, cat))
+ return out
+
+
+def _catalog_rank(item):
+ """Richest-first: prefer a source with descriptions, then one with more tools."""
+ _, cat = item
+ return (sum(1 for v in cat.values() if v), len(cat))
+
+
+def parse_catalog_with_source(run_dir_path, agent):
+ """({tool: desc}, source_name) -- the richest catalog available for this run dir.
+
+ source_name is None when NO source produced anything, which is the condition that used
+ to pass silently and turn evidence untooling into a no-op. Callers should treat it as a
+ defect signal, not as an empty catalog.
+ """
+ srcs = catalog_sources(run_dir_path, agent)
+ if not srcs:
+ return {}, None
+ name, cat = max(srcs, key=_catalog_rank)
+ return cat, name
+
+
+def parse_catalog(run_dir_path, agent):
+ """Agent-agnostic parse of the loaded tool catalog ({name: desc}) from a task's run
+ dir -- Copilot debug log or Claude transcript. Returns {} if unavailable.
+
+ Codex has no per-task JSON tool dump, so its known default surface is returned instead
+ (CODEX_DEFAULT_TOOL_CATALOG) so catalog-size telemetry is honest.
+ """
+ return parse_catalog_with_source(run_dir_path, agent)[0]
+
+
+def parse_tool_names_copilot(transcript_text):
+ """{tool_name: ""} for the tools a Copilot run actually loaded.
+
+ The names ride in `session.usage_checkpoint` events under
+ data.promptCacheBreakState[].models[].tools[] -- the same list the model was sent, so
+ this reflects what the agent really had, not a static default surface.
+ """
+ names = set()
+ for line in (transcript_text or "").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ for pcb in ((ev.get("data") or {}).get("promptCacheBreakState") or []):
+ for model in (pcb.get("models") or {}).values():
+ for tool in (model.get("tools") or []):
+ name = tool.get("name")
+ if name:
+ names.add(name)
+ return {n: "" for n in sorted(names)}
diff --git a/tokeneconomizer/vendor/__init__.py b/tokeneconomizer/vendor/__init__.py
new file mode 100644
index 0000000..884e3d8
--- /dev/null
+++ b/tokeneconomizer/vendor/__init__.py
@@ -0,0 +1,17 @@
+"""Vendored, dependency-free building blocks shared by the economizer mechanisms.
+
+These three modules were lifted verbatim from the original research harness and carry
+NO third-party or cross-package dependencies (Python stdlib only). They are vendored
+here so `tokeneconomizer` is fully self-contained -- no `sys.path` injection into a
+sibling benchmark tree, no external install:
+
+ * pricing -- cache-aware effective-token cost weighting (MODEL_WEIGHTS price table)
+ * reflect -- LLM-free waste detector + gated reflection + evidence-untool tool judge
+ * spiral -- runtime spiral-state folder + cheapest-sufficient escalation ladder
+
+The only edit applied on vendoring was rewiring spiral's lazy `import reflect` to the
+relative `from . import reflect` so the two stay a matched pair inside this package.
+"""
+from . import pricing, reflect, spiral
+
+__all__ = ["pricing", "reflect", "spiral"]
diff --git a/tokeneconomizer/vendor/claude_tool_catalog.json b/tokeneconomizer/vendor/claude_tool_catalog.json
new file mode 100644
index 0000000..b4f817a
--- /dev/null
+++ b/tokeneconomizer/vendor/claude_tool_catalog.json
@@ -0,0 +1,26 @@
+{
+ "Agent": "Launch a new agent to handle complex, multi-step tasks. Each agent type has specific capabilities and tools available to it.\n\nAvailable agent types are listed in <system-reminder> messages in the conversation.\n\nWhen using the Agent tool, specify a subagent_type parameter to select which agent type t",
+ "Bash": "Executes a bash command and returns its output.\n\n- Working directory persists between calls, but prefer absolute paths \u2014 `cd` in a compound command can trigger a permission prompt. Shell state (env vars, functions) does not persist; the shell is initialized from the user's profile.\n- IMPORTANT: Avoi",
+ "CronCreate": "Schedule a prompt to be enqueued at a future time. Use for both recurring schedules and one-shot reminders.\n\nUses standard 5-field cron in the user's local timezone: minute hour day-of-month month day-of-week. \"0 9 * * *\" means 9am local \u2014 no timezone conversion needed.\n\n## One-shot tasks (recurring",
+ "CronDelete": "Cancel a cron job previously scheduled with CronCreate. Removes it from the in-memory session store.",
+ "CronList": "List all cron jobs scheduled via CronCreate in this session.",
+ "DesignSync": "Read and update the user's claude.ai/design design-system projects through their claude.ai login (or, for sessions without one, a dedicated design authorization from /design-login). Use this together with the /design-sync skill to keep a local component library in sync with a Claude Design project \u2014",
+ "Edit": "Performs exact string replacement in a file.\n\n- You must Read the file in this conversation before editing, or the call will fail.\n- `old_string` must match the file exactly, including indentation, and be unique \u2014 the edit fails otherwise. Strip the Read line prefix (line number + tab) before matchi",
+ "EnterWorktree": "Use this tool ONLY when explicitly instructed to work in a worktree \u2014 either by the user directly, or by project instructions (CLAUDE.md / memory). This tool creates an isolated git worktree and switches the current session into it.\n\n## When to Use\n\n- The user explicitly says \"worktree\" (e.g., \"star",
+ "ExitWorktree": "Exit a worktree session created by EnterWorktree and return the session to the original working directory.\n\n## Scope\n\nThis tool ONLY operates on worktrees created by EnterWorktree in this session. It will NOT touch:\n- Worktrees you created manually with `git worktree add`\n- Worktrees from a previous",
+ "ListAgents": "Lists agents you can SendMessage to \u2014 in-process subagents you spawned, other local Claude sessions on this machine, your Claude sessions running in the cloud (when this session has cloud access; a cloud session receives your message but cannot message any session back yet \u2014 do not ask it to reply, ",
+ "Monitor": "Start a background monitor that streams events from a long-running script. Each stdout line is an event \u2014 you keep working and notifications arrive in the chat. Events arrive on their own schedule and are not replies from the user, even if one lands while you're waiting for the user to answer a ques",
+ "NotebookEdit": "Replaces, inserts, or deletes a single cell in a Jupyter notebook (.ipynb file).\n\nUsage:\n- You must use the Read tool on the notebook in this conversation before editing \u2014 this tool will fail otherwise.\n- `notebook_path` must be an absolute path.\n- `cell_id` is the `id` attribute shown in the Read t",
+ "PushNotification": "This tool sends a desktop notification in the user's terminal. If Remote Control is connected, it also pushes to their phone. Either way, it pulls their attention from whatever they're doing \u2014 a meeting, another task, dinner \u2014 to this session. That's the cost. The benefit is they learn something now",
+ "Read": "Reads a file from the local filesystem.\n\n- `file_path` must be an absolute path.\n- Reads up to 2000 lines by default.\n- When you already know which part of the file you need, only read that part. This can be important for larger files.\n- Results are returned using cat -n format, with line numbers st",
+ "ReportFindings": "Report code-review findings as a typed list so the host UI can render them. Use this only when the active code-review instructions tell you to report findings with this tool; otherwise follow whatever output format those instructions specify. When reporting a review's results, call it once with the ",
+ "ScheduleWakeup": "Schedule when to resume work in /loop dynamic mode \u2014 the user invoked /loop without an interval, asking you to self-pace iterations of a specific task.\n\nDo NOT schedule a short-interval wakeup to poll for background work you started \u2014 when harness-tracked work finishes, you are re-invoked automatica",
+ "SendMessage": "# SendMessage\n\nSend a message to another agent.\n\n```json\n{\"to\": \"researcher\", \"summary\": \"assign task 1\", \"message\": \"start on task #1\"}\n```\n\n| `to` | |\n|---|---|\n| `\"researcher\"` | Teammate by name |\n| `\"main\"` | The main conversation (background subagents only) |\n| `\"worker\"` | Any agent from `Lis",
+ "Skill": "Invoke a skill.\n\nA skill is a packaged set of instructions the user or project has set up for a particular kind of task (deploy steps, a review checklist, a repo-specific workflow). Available skills appear in a system-reminder listing with one-line descriptions. When the task at hand is one a listed",
+ "TaskOutput": "DEPRECATED: Background tasks return their output file path in the tool result, and you receive a <task-notification> with the same path when the task completes.\n- For bash tasks: prefer using the Read tool on that output file path \u2014 it contains stdout/stderr.\n- For local_agent tasks: use the Agent t",
+ "TaskStop": "- Stops a running background task by its ID\n- Takes a task_id parameter identifying the task to stop\n- To stop an agent-team teammate, pass its agent ID (\"name@team\") or bare teammate name as task_id\n- To stop a background agent spawned with a name, pass that name as task_id\n- Returns a success or f",
+ "WebFetch": "Fetches a URL, converts the page to markdown, and answers `prompt` against it using a small fast model.\n\n- Fails on authenticated/private URLs \u2014 use an authenticated MCP tool or `gh` for those instead.\n- HTTP is upgraded to HTTPS. Cross-host redirects are returned to you rather than followed; call a",
+ "WebSearch": "Search the web. Returns result blocks with titles and URLs. US-only.\n\n- The current month is August 2026 \u2014 use this when searching for recent information.\n- `allowed_domains` / `blocked_domains` filter results.\n- After answering from results, end with a \"Sources:\" list of the URLs you used as markdo",
+ "Workflow": "Execute a workflow script that orchestrates multiple subagents deterministically. Workflows run in the background \u2014 this tool returns immediately with a task ID, and a <task-notification> arrives when the workflow completes. Use /workflows to watch live progress.\n\nA workflow structures work across m",
+ "Write": "Writes a file to the local filesystem, overwriting if one exists.\n\nWhen to use: creating a new file, or fully replacing one you've already Read. Overwriting an existing file you haven't Read will fail. For partial changes, use Edit instead."
+}
\ No newline at end of file
diff --git a/tokeneconomizer/vendor/pricing.py b/tokeneconomizer/vendor/pricing.py
new file mode 100644
index 0000000..4a5ebf9
--- /dev/null
+++ b/tokeneconomizer/vendor/pricing.py
@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""
+Cache-aware effective-cost weighting for token accounting.
+
+`tokens_total` (= prompt_tokens + completion_tokens, summed over every model call)
+counts EVERY token at full weight. In an agent loop the conversation is re-sent each
+turn, so ~90%+ of `tokens_total` is `cache_read_input_tokens` — re-read context that
+the provider bills at a small fraction of fresh input. Reporting raw `tokens_total`
+therefore (a) OVERSTATES the cost of cheap cached re-reads and (b) UNDERSTATES the
+cost of expensive output tokens. This biases A/B comparisons: a method that mainly
+prunes cached re-reads looks better than its real-$ effect, while one that adds output
+/ fresh-input (e.g. error-retry loops, negative transfer) looks cheaper than it is.
+
+`effective_tokens()` reweights the per-task token breakdown into fresh-input-token
+EQUIVALENTS using real price ratios, so cost A/Bs reflect real spend. Ratios are keyed on
+the exact BILLED MODEL ID (weights_for -> MODEL_WEIGHTS), because families are NOT uniform
+— e.g. gpt-5.6-sol bills output 6x + cache_write 1.25x, gpt-5.3-codex output 8x +
+cache_write 0x, Claude output 5x + cache_write 1.25x. The table is a snapshot of the
+providers' published per-model prices; a billed id with no entry HARD-FAILS rather than being
+silently mispriced.
+
+ model id (example) input cache_read cache_write output
+ claude-* (Opus/Sonnet/Haiku) 1.00 0.10 1.25 5.00
+ gpt-5.6-sol (codex default) 1.00 0.10 1.25 6.00
+ gpt-5.3-codex 1.00 0.097 0.00 8.00
+ gpt-5-codex (OpenAI direct) 1.00 0.10 1.00 8.00
+
+Callers may override with an explicit effective_tokens(rec, weights=...).
+When the per-task breakdown is absent (older runs), effective_tokens() falls back to
+`tokens_total` at full weight so nothing crashes — call has_breakdown() to detect this.
+"""
+
+# ---------------------------------------------------------------------------
+# Per-model-id price ratios (fresh-input-token equivalents; input := 1.0).
+#
+# Pricing MUST match the exact model id that was billed — model families are NOT
+# uniform (e.g. gpt-5.6-sol bills output 6x + cache_write 1.25x, while gpt-5.3-codex
+# bills output 8x + cache_write 0x). So we key on the concrete model id, not a family
+# heuristic, and HARD-FAIL if a billed id has no entry (see weights_for).
+#
+# Source: the providers' published per-model prices, snapshot 2026-07-27; ratios =
+# price / input_price. Update the dict when a provider changes its prices.
+#
+# NB (verified): codex-cli 0.145.0's built-in DEFAULT model is `gpt-5.6-sol` (the resolved id
+# is recorded in codex's session rollout file, not in its --json transcript). So an unpinned
+# `codex` run bills sol.
+MODEL_WEIGHTS = {
+ # --- per-model price snapshot (2026-07-27) ------------------------------
+ "claude-haiku-4-5": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-opus-4-6": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-opus-4-7": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-opus-4-8": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-opus-5": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-sonnet-4-5": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-sonnet-4-6": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "claude-sonnet-5": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=5.0),
+ "gemini-2.5-pro": dict(input=1.0, cache_read=0.096, cache_write=0.0, output=8.0),
+ "gemini-3-flash-preview": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=6.0),
+ "gemini-3.1-pro-preview": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=6.0),
+ "gemini-3.5-flash": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=6.0),
+ "gemini-3.6-flash": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=5.0),
+ "gpt-5-mini": dict(input=1.0, cache_read=0.08, cache_write=0.0, output=8.0),
+ "gpt-5.3-codex": dict(input=1.0, cache_read=0.0971, cache_write=0.0, output=8.0),
+ "gpt-5.4": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=6.0),
+ "gpt-5.4-mini": dict(input=1.0, cache_read=0.0933, cache_write=0.0, output=6.0),
+ "gpt-5.5": dict(input=1.0, cache_read=0.10, cache_write=0.0, output=6.0),
+ "gpt-5.6-luna": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=6.0),
+ "gpt-5.6-sol": dict(input=1.0, cache_read=0.10, cache_write=1.25, output=6.0),
+ "gpt-5.6-terra": dict(input=1.0, cache_read=0.10, cache_write=1.248, output=6.0),
+ "mai-code-1-flash-picker": dict(input=1.0, cache_read=0.0933, cache_write=0.0, output=6.0),
+ # --- OpenAI-direct-only ids (public per-1M pricing; not in copilot catalog) ----
+ # gpt-5/5.1 & *-codex: $1.25 in / $0.125 cached / $10 out => cr 0.10, out 8.0
+ "gpt-5": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ "gpt-5.1": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ "gpt-5-codex": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ "gpt-5.1-codex": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ "gpt-5.1-codex-max": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ "gpt-5.2-codex": dict(input=1.0, cache_read=0.10, cache_write=1.0, output=8.0),
+ # codex-mini-latest: $1.50 in / $0.375 cached / $6.00 out => cr 0.25, out 4.0
+ "codex-mini-latest": dict(input=1.0, cache_read=0.25, cache_write=1.0, output=4.0),
+}
+
+# Billed model id for an agent when a row does NOT pin one (rec['model'] is None).
+# Claude/Copilot agents map to None -> the Claude curve (their ratios are model-uniform
+# across Opus/Sonnet/Haiku, so an exact id is unnecessary). Codex maps to codex's built-in
+# default, gpt-5.6-sol (verified from codex's session rollout). Probed against the OpenAI
+# Platform on 2026-09-09: `gpt-5.6-sol` is both listed by /v1/models AND accepted by
+# /v1/responses (HTTP 200).
+#
+# The trap a catalog check alone will not catch: the `-codex` ALIASES are listed by
+# /v1/models and 404 on /v1/responses ("does not exist or you do not have access to it" for
+# gpt-5.1-codex, same probe). So pinning one of those fails per task while every catalog gate
+# stays green. Pin a base id.
+AGENT_DEFAULT_MODEL = {
+ "codex": "gpt-5.6-sol",
+}
+
+# Claude-family price ratios (Opus/Sonnet/Haiku share them). Used as the default when a
+# row has no resolvable model id (legacy rows, claude/copilot agents) so those stay
+# byte-identical to historical behavior. Also the value of the back-compat WEIGHTS alias.
+WEIGHTS_CLAUDE = {
+ "input": 1.0, # input_tokens = fresh/uncached prompt tokens
+ "cache_read": 0.10, # cache_read_input_tokens
+ "cache_write": 1.25, # cache_creation_input_tokens
+ "output": 5.0, # output_tokens (completion)
+}
+# Back-compat: modules that read pricing.WEIGHTS directly (figures, costmodel) were
+# written against the Claude curve; keep that meaning. Per-row/per-model selection is via
+# weights_for(); effective_tokens() applies it automatically.
+WEIGHTS = WEIGHTS_CLAUDE
+
+_BREAKDOWN_KEYS = ("input_tokens", "cache_read_input_tokens",
+ "cache_creation_input_tokens", "output_tokens")
+
+
+def resolve_model(rec):
+ """The billed model id for a row: its explicit `model`, else the agent's default
+ billed model (AGENT_DEFAULT_MODEL), else None. run_agent stamps the resolved id into
+ `model` for codex runs (read from codex's session rollout), so unpinned codex rows
+ still price by their true id."""
+ if not isinstance(rec, dict):
+ return None
+ m = rec.get("model")
+ if m:
+ # A router's `provider/model` id names a ROUTE, not a price: Copilot's BYOK through
+ # a router such as OpenRouter bills `anthropic/claude-haiku-4.5`, whose price is
+ # the tail's. Stripping it here is safe because no MODEL_WEIGHTS key contains a
+ # slash -- it can only affect an id that would otherwise hard-fail below, and
+ # hard-failing a correctly-configured BYOK row at the END of a paid rollout is the
+ # outcome this avoids.
+ return m.rsplit("/", 1)[-1] if "/" in m else m
+ return AGENT_DEFAULT_MODEL.get((rec.get("agent") or "").lower())
+
+
+def weights_for(rec):
+ """Pick the fresh-input-equivalent weight table for a row by its BILLED MODEL ID.
+
+ Resolution (pricing must match the model id):
+ * resolve the billed id (resolve_model) — explicit `model`, else agent default;
+ * id in MODEL_WEIGHTS -> that model's exact ratios;
+ * id is a claude-* variant -> Claude curve (covers dotted ids like claude-opus-4.8
+ not in the catalog snapshot; ratios are uniform);
+ * id present but UNPRICED -> raise ValueError (hard-fail: a new/unknown model must
+ be added to MODEL_WEIGHTS, never silently mispriced);
+ * no resolvable id at all -> Claude curve (legacy rows / claude / copilot: keeps
+ historical behavior, never crashes)."""
+ if not isinstance(rec, dict):
+ return WEIGHTS_CLAUDE
+ mid = resolve_model(rec)
+ if not mid:
+ return WEIGHTS_CLAUDE
+ key = mid.lower()
+ if key in MODEL_WEIGHTS:
+ return MODEL_WEIGHTS[key]
+ if "claude" in key:
+ return WEIGHTS_CLAUDE
+ raise ValueError(
+ f"pricing: no weight entry for billed model {mid!r} "
+ f"(agent={rec.get('agent')!r}). Add it to pricing.MODEL_WEIGHTS "
+ f"(from the vendor's published price list).")
+
+
+def has_breakdown(rec):
+ """True when rec carries the per-task token breakdown needed for cache-aware
+ weighting (i.e. input_tokens is populated)."""
+ return isinstance(rec, dict) and rec.get("input_tokens") is not None
+
+
+def effective_tokens(rec, weights=None):
+ """Cache-aware effective cost of one task, in fresh-input-token equivalents.
+
+ Uses the token breakdown (input / cache_read / cache_creation / output) when
+ present; otherwise falls back to `tokens_total` at full weight (legacy rows).
+ The weight table is chosen per row (weights_for) from its agent/model unless an
+ explicit `weights=` is passed. Returns 0 when neither breakdown nor total exists.
+ """
+ w = weights if weights is not None else weights_for(rec)
+ if has_breakdown(rec):
+ it = rec.get("input_tokens") or 0
+ cr = rec.get("cache_read_input_tokens") or 0
+ cw = rec.get("cache_creation_input_tokens") or 0
+ ot = rec.get("output_tokens") or 0
+ return (it * w["input"] + cr * w["cache_read"]
+ + cw * w["cache_write"] + ot * w["output"])
+ tt = rec.get("tokens_total") if isinstance(rec, dict) else None
+ return tt if isinstance(tt, (int, float)) else 0
+
+
+def cache_read_fraction(rec):
+ """Share of tokens_total that is cached re-read (diagnostic; None if unknown)."""
+ if not has_breakdown(rec):
+ return None
+ tt = rec.get("tokens_total") or 0
+ cr = rec.get("cache_read_input_tokens") or 0
+ return (cr / tt) if tt else None
diff --git a/tokeneconomizer/vendor/reflect.py b/tokeneconomizer/vendor/reflect.py
new file mode 100644
index 0000000..0116c86
--- /dev/null
+++ b/tokeneconomizer/vendor/reflect.py
@@ -0,0 +1,1377 @@
+#!/usr/bin/env python3
+"""
+Gated reflection for the amortized self-optimization loop.
+
+A single unified engine: the detector surfaces BOTH environment-discovery waste
+(re-probing the interpreter/conda env) AND navigation waste (re-searching for the
+same code, re-reading the same files), and the optimizer turns either into durable,
+task-agnostic notes (env facts + "where things live" facts) for one compounding
+manifest. (Env-discovery is largely solved by the realistic baseline; navigation is
+the remaining lever — both flow through the same machinery.)
+
+Two stages, so the paid optimizer runs only when a free check finds something:
+
+ STAGE A detect(trajectory) free, every task — arithmetic on the
+ trajectory we already capture.
+ GATE trigger(signals, window) fire the paid optimizer ONLY on an
+ obvious / recurring inefficiency; default
+ is no-op (the interface stays unchanged).
+ STAGE B optimize(signals, state, ...) paid, rare — an LLM call (model
+ configurable, default = the agent being
+ optimized) that emits durable, TASK-AGNOSTIC
+ repo notes addressing the detected waste.
+
+Because triggers fire often early in a stream and rarely once the interface
+matures, optimizer overhead is front-loaded and self-extinguishing.
+
+The detector understands the Copilot CLI transcript schema
+(tool.execution_start / tool.execution_complete). Stdlib only.
+"""
+import json
+import os
+import re
+import subprocess
+
+# ---- inefficiency signal patterns (kept in sync with trace analysis) ---------
+ENV_PROBE_RE = re.compile(
+ r"(which\s+python|ls\s+/opt/miniconda3|conda\s+env\s+list|conda\s+activate|"
+ r"source\s+activate|compgen\s+-c|/envs/|command\s+-v\s+python|"
+ r"ls\s+/usr/bin.*python|find\s+/.*python|\bpython\d?\s+--version)",
+ re.IGNORECASE)
+
+# Generic command-INTENT buckets (NOT repo-specific): agents routinely burn a
+# round-trip discovering the right *form* of a common operation. We watch the
+# test-running family because switching forms within a task ("tried pytest, then
+# fell back to the repo's runner") is a high-precision, leakage-safe, recurring
+# discovery that a durable "use <tool> to run tests" note eliminates.
+TEST_FORM_RES = {
+ "pytest": re.compile(r"\b(pytest|py\.test|python\s+-m\s+pytest)\b", re.I),
+ "runtests.py": re.compile(r"runtests(\.py)?", re.I),
+ "manage.py test": re.compile(r"manage\.py\s+test", re.I),
+ "tox": re.compile(r"\btox\b", re.I),
+ "nox": re.compile(r"\bnox\b", re.I),
+ "unittest": re.compile(r"python\s+-m\s+unittest", re.I),
+ "trial": re.compile(r"\btrial\b", re.I),
+}
+# Infrastructure-error markers (NOT test failures like "FAIL:"): these signal the
+# command itself was wrong/misconfigured. They matter because agents pipe through
+# `2>&1 | tail`, so the shell exit code is 0 and a success-based detector is blind.
+ERROR_MARKER_RE = re.compile(
+ r"(Traceback \(most recent call last\)|ModuleNotFoundError|ImportError|"
+ r"cannot import|command not found|No such file or directory|"
+ r"unrecognized arguments|no tests ran|errors during collection|"
+ r"is not a registered|ERROR: .*not found|usage: )", re.IGNORECASE)
+# External fetches: web/github tools or bash curl/wget to a non-local URL. Attempts
+# remain potential discovery waste; successful fetches are the integrity flag because
+# blocked attempts cannot import solution evidence into the sealed repository.
+EXTERNAL_URL_RE = re.compile(r"\b(curl|wget)\b[^\n]*https?://", re.IGNORECASE)
+# Backstop integrity flag: ANY mention of a GitHub gold-source host in a command,
+# regardless of the client used (curl, python urllib/requests, etc.). github.com
+# matches api.github.com too; api.githubcopilot.com (the model endpoint) is NOT
+# matched, so legitimate model traffic never trips this.
+GOLD_HOST_RE = re.compile(
+ r"(?:github\.com|githubusercontent\.com|codeload\.github)", re.IGNORECASE)
+# A network attempt is still useful diagnostic evidence, but an isolation shim or a
+# refused connection means no external bytes entered the task. Keep those attempts
+# distinct from successful fetches so integrity validation does not reject a sealed run.
+BLOCKED_FETCH_RE = re.compile(
+ r"(?:(?:gold|network)-isolation:.*blocked|failed to connect|connection refused|"
+ r"could not resolve host|network is unreachable|name or service not known|"
+ r"operation timed out)", re.IGNORECASE)
+# A command that RUNS code to verify behavior — the test/reproduction step of the
+# edit->test->fail iteration loop. Broader than TEST_FORM_RES (it also matches the
+# ad-hoc `python -c` reproductions agents use). Used to detect EXECUTION-iteration
+# waste (re-running an unchanged command), which drives the costly multi-turn spirals.
+TEST_RUN_RE = re.compile(
+ r"(runtests(\.py)?|\bpytest\b|py\.test|manage\.py\s+test|python\s+-m\s+unittest|"
+ r"\btox\b|\bnox\b|\btrial\b|python\d?\s+-c\b)", re.IGNORECASE)
+
+
+def _norm_cmd(c):
+ return re.sub(r"\s+", " ", c.strip()).lower()
+
+
+def load_trajectory(path):
+ """Return an ordered list of (toolName, command_or_args, result_text, success)
+ from a saved agent transcript.jsonl. Auto-detects the format: Copilot CLI emits
+ tool.execution_start/complete events; Claude Code emits `assistant` messages whose
+ content[] carries `tool_use` blocks (and `user` messages with `tool_result`)."""
+ out, pend = [], {}
+ if not os.path.exists(path):
+ return out
+ lines = [l.strip() for l in open(path) if l.strip()]
+ # Format sniff: Codex emits completed command/file items directly (there is no
+ # start/result pairing). Handle it before the Claude/Copilot paths so every
+ # downstream economizer stage sees the same normalized tuple stream.
+ if any('"type":"item.completed"' in line.replace(" ", "") for line in lines):
+ return _load_trajectory_codex(lines)
+ # format sniff: Claude if any line is an assistant/user message-envelope event.
+ is_claude = False
+ for line in lines:
+ if '"tool_use"' in line or ('"type": "assistant"' in line
+ or '"type":"assistant"' in line):
+ is_claude = True
+ break
+ if is_claude:
+ return _load_trajectory_claude(lines)
+ for line in lines:
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ t = ev.get("type")
+ d = ev.get("data", {}) if isinstance(ev.get("data"), dict) else {}
+ if t == "tool.execution_start":
+ pend[d.get("toolCallId")] = {"name": d.get("toolName"),
+ "args": d.get("arguments", {})}
+ elif t == "tool.execution_complete":
+ info = pend.get(d.get("toolCallId"), {"name": "?", "args": {}})
+ res = d.get("result") or {}
+ content = res.get("content") if isinstance(res, dict) else ""
+ out.append((info["name"], info["args"],
+ content if isinstance(content, str) else "",
+ d.get("success")))
+ return out
+
+
+_SHELL_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M)
+_SHELL_GIT_APPLY_PATH_RE = re.compile(r"^\+\+\+ b/(.+?)\s*$", re.M)
+
+
+def _shell_patch_paths(command):
+ """Paths touched by a patch applied through the shell, () when it is not one."""
+ command = command or ""
+ if "*** Begin Patch" in command:
+ found = _SHELL_PATCH_FILE_RE.findall(command)
+ elif re.search(r"\bgit\s+apply\b", command):
+ found = _SHELL_GIT_APPLY_PATH_RE.findall(command)
+ else:
+ return ()
+ return tuple(dict.fromkeys(p.strip() for p in found if p.strip()))
+
+
+def _load_trajectory_codex(lines):
+ """Normalize Codex ``exec --json`` item events.
+
+ Commands become ``shell`` calls with a conventional ``command`` argument;
+ file-change items become ``apply_patch`` calls carrying the changed paths.
+ Agent-message items are deliberately omitted because they are not tool calls.
+ """
+ out = []
+ for line in lines:
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ if ev.get("type") != "item.completed":
+ continue
+ item = ev.get("item") or {}
+ kind = item.get("type")
+ if kind == "command_execution":
+ command = item.get("command") or ""
+ output = item.get("aggregated_output") or ""
+ try:
+ ok = int(item.get("exit_code")) == 0
+ except (TypeError, ValueError):
+ ok = False
+ # A patch applied THROUGH THE SHELL (`apply_patch <<'PATCH' ... *** Begin
+ # Patch`) is an edit as much as a native file_change item is; on some model
+ # providers it is the only form the CLI emits (2026-09-12). Surface
+ # it as the same `apply_patch` call every consumer already reads -- injection
+ # drift, the untool audit, the waste signals -- so a provider change cannot
+ # blind them.
+ paths = _shell_patch_paths(command)
+ if paths and (ok or "Success." in output):
+ out.append((
+ "apply_patch",
+ {"path": paths[0], "paths": list(paths), "changes": [],
+ "via": "shell"},
+ "",
+ True,
+ ))
+ out.append(("shell", {"command": command}, output, ok))
+ elif kind == "file_change":
+ changes = item.get("changes") or []
+ paths = [c.get("path") for c in changes
+ if isinstance(c, dict) and c.get("path")]
+ out.append((
+ "apply_patch",
+ {"path": paths[0] if paths else "", "paths": paths,
+ "changes": changes},
+ "",
+ item.get("status") == "completed",
+ ))
+ return out
+
+
+def _load_trajectory_claude(lines):
+ """Claude Code stream-json variant of load_trajectory. Pairs each `tool_use` block
+ (in an `assistant` message) with its matching `tool_result` (in a later `user`
+ message) by tool_use id, yielding the same (name, args, result_text, success)
+ tuples the Copilot path returns so all downstream analysis is format-agnostic."""
+ out, order, seen = [], [], {}
+ results = {}
+ for line in lines:
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ msg = ev.get("message") or {}
+ content = msg.get("content")
+ if not isinstance(content, list):
+ continue
+ for b in content:
+ if not isinstance(b, dict):
+ continue
+ if b.get("type") == "tool_use":
+ tid = b.get("id")
+ seen[tid] = {"name": b.get("name"), "args": b.get("input", {})}
+ order.append(tid)
+ elif b.get("type") == "tool_result":
+ tid = b.get("tool_use_id")
+ c = b.get("content")
+ if isinstance(c, list):
+ c = " ".join(x.get("text", "") for x in c
+ if isinstance(x, dict))
+ results[tid] = (c if isinstance(c, str) else "",
+ not b.get("is_error", False))
+ for tid in order:
+ info = seen[tid]
+ res_text, ok = results.get(tid, ("", True))
+ out.append((info["name"], info["args"], res_text, ok))
+ return out
+
+
+def _basename(p):
+ return p.rstrip("/").split("/")[-1] if p else p
+
+
+# ---- v4.1 mechanism A: untooling (per-turn-prefix minimization) ---------------
+# The sealed agent re-feeds every loaded tool's JSON schema in its context EVERY
+# turn (cache_read), so unused tool schemas are a fixed per-turn tax. Because the rollout is
+# single-shot we cannot re-enable a tool mid-task, so we exclude ONLY tools that are
+# inapplicable BY CONSTRUCTION for one sealed SWE-bench task — the agentic-
+# orchestration / session / memory / scheduling / docs meta-tools that cannot help
+# edit-and-test a repo and that the agent would only invoke by mistake. File/shell/
+# search/edit tools (bash, view, grep, glob, edit, create, read_bash) are NEVER
+# excluded. The set is intentionally conservative; widen only with stream evidence.
+INAPPLICABLE_TOOLS = (
+ # sub-agent orchestration (a single sealed task has nothing to delegate to)
+ "task", "list_agents", "read_agent", "write_agent",
+ # session DB / scheduling / skills / CLI self-docs — irrelevant to solving code
+ "sql", "session_store_sql", "manage_schedule", "skill",
+ "fetch_copilot_cli_documentation",
+ # persistent memory (also off by default headless) and its vote tool
+ "store_memory", "vote_memory",
+ # interactive Q&A: headless can never get an answer
+ "ask_user",
+ # ---- Claude Code tool names (PascalCase; disjoint from the Copilot names
+ # above, so listing both is safe regardless of --agent). These are
+ # inapplicable-by-construction for one sealed headless SWE-bench task.
+ # Work surface (Bash/Edit/Read/Write/Grep/Glob) is kept.
+ # sub-agent orchestration + workflow engine (nothing to delegate to):
+ "Agent", "Workflow",
+ # task list / scheduling / cron (no session persistence in a single task):
+ "TaskCreate", "TaskUpdate", "TaskGet", "TaskList", "TaskOutput", "TaskStop",
+ "ScheduleWakeup", "CronCreate", "CronDelete", "CronList",
+ # plan-mode + git-worktree session controls (irrelevant to a sealed rollout):
+ "EnterPlanMode", "ExitPlanMode", "EnterWorktree", "ExitWorktree",
+ # skills, notebooks, and network tools (network is scrubbed in-container):
+ "Skill", "NotebookEdit", "WebSearch", "WebFetch",
+ # interactive Q&A: headless can never get an answer:
+ "AskUserQuestion",
+)
+# Tools that must always stay available regardless of usage (the real work surface).
+PROTECTED_TOOLS = frozenset({
+ "bash", "view", "grep", "glob", "edit", "create", "str_replace",
+ "read_bash", "stop_bash", "list_bash", "read_file",
+ # Codex's complete native work surface (fixed; neither schema is removable).
+ "shell", "apply_patch",
+ # Claude Code work-surface names:
+ "Bash", "Edit", "Write", "Read", "Grep", "Glob", "MultiEdit",
+})
+
+
+def tool_usage(trajectory_path):
+ """Map toolName -> call count for one trajectory (free; for untooling evidence
+ and stream-wide zero-usage learning)."""
+ counts = {}
+ for name, _args, _res, _ok in load_trajectory(trajectory_path):
+ if name:
+ counts[name] = counts.get(name, 0) + 1
+ return counts
+
+
+def untool_exclusions(used_tools=None, extra_inapplicable=()):
+ """Return the list of tool names to exclude from the agent's schema this task.
+
+ Conservative by default: the static INAPPLICABLE_TOOLS set (inapplicable by
+ construction for a sealed single task). `used_tools` is the set of tools observed
+ in use across the stream so far; any inapplicable tool that HAS been used is kept
+ (so a repo that genuinely uses one is never starved). PROTECTED_TOOLS are never
+ excluded even if listed. `extra_inapplicable` lets a caller widen the set with
+ stream evidence (e.g. tools with zero uses across a long window)."""
+ used = set(used_tools or ())
+ cand = list(INAPPLICABLE_TOOLS) + [t for t in extra_inapplicable
+ if t not in INAPPLICABLE_TOOLS]
+ return [t for t in cand if t not in PROTECTED_TOOLS and t not in used]
+
+
+# ---- v4.2: self-contained untooling (optimizer-reasoned, not hardcoded) -------
+# The v4.1 exclusion set above is a HAND-AUTHORED prior. v4.2 makes untooling
+# self-contained: introspect the agent's ACTUAL loaded tool catalog (from its debug
+# log — agent-agnostic), then let the optimizer LLM classify each tool KEEP/EXCLUDE/
+# UNSURE from its description + a fixed domain brief. EXCLUDE drops immediately; UNSURE
+# stays LOADED (safe) until observed-unused over a window, then demotes to EXCLUDE; a
+# periodic re-audit reloads the full catalog to recover from drift. Prototype validated:
+# deterministic, reproduces the high-value hand exclusions (incl. the ~9.6k-token `task`
+# schema), zero unsafe over-exclusions. The hardcoded list remains the offline fallback.
+
+# Matches `"name": "<tool>", ... "description": "<text>"` in the (JSON-in-JSON) debug
+# log. Allows optional intervening scalar fields (e.g. `"title": null,`) between name and
+# description — newer Copilot CLI schemas insert `"title"`, which previously caused every
+# tool except the few lacking a title to be missed.
+_CATALOG_RE = re.compile(
+ r'"name":\s*"([a-z_]+)"\s*,\s*(?:"[a-z_]+":\s*(?:"[^"]*"|null|true|false|[0-9.]+)\s*,\s*)*'
+ r'"description":\s*"')
+
+
+# Recovering the loaded tool catalog on the Copilot CLI, and why this fallback exists.
+#
+# Three sources have been tried, in the order the CLI killed them:
+# 1. `_CATALOG_RE` over debug-log tool SCHEMAS ("name"+"description" objects). Dead by
+# 1.0.82 -- newer builds write no schemas at all.
+# 2. `trace.parse_tool_names_copilot` over the transcript's `session.usage_checkpoint`
+# -> `data.promptCacheBreakState[].models[].tools[]`. This is what actually carried
+# 1.0.82, and 1.0.83 stopped emitting the event entirely (measured: 1 occurrence per
+# 1.0.82 transcript, 0 per 1.0.83 transcript).
+# 3. This parser: the flat catalog the debug log still writes on BOTH versions --
+#
+# "tools": [ "bash:66921ccc", "read_bash:71c40652", ... ]
+#
+# With (1) and (2) both dead on 1.0.83 the catalog parsed EMPTY, so `catalog_tools` stayed
+# empty, the drop-candidate loop had nothing to iterate, and evidence untooling silently
+# became a no-op -- every `untool-*` / `trimech*` arm ran as its own baseline while still
+# being named an untooling arm. Measured 2026-09-05 on one sklearn seed: CLI 1.0.82 gave
+# `untool_catalog_size=17` and dropped 8 tools on 30 of 31 tasks; CLI 1.0.83, same harness
+# and same repo, gave `untool_catalog_size=0` and dropped nothing on 31 of 31.
+#
+# Source (3) is preferred over (2) because it survived both upgrades. Names only -- the
+# fingerprint form carries no descriptions, the same consequence (2) had: evidence
+# untooling works (it needs names), the instruction-aware tool JUDGE has nothing to reason
+# over and will drop nothing while still billing for the call.
+_CATALOG_FLAT_BLOCK_RE = re.compile(r'"tools"\s*:\s*\[(.*?)\]', re.DOTALL)
+_CATALOG_FLAT_ITEM_RE = re.compile(r'"([a-z][a-z0-9_]*):[0-9a-f]{6,}"')
+
+
+def parse_tool_catalog_flat(debug_log_text):
+ """{tool_name: ""} from the CLI >= 1.0.83 flat `"tools": ["name:fingerprint", ...]`."""
+ out = {}
+ for block in _CATALOG_FLAT_BLOCK_RE.findall(debug_log_text or ""):
+ for name in _CATALOG_FLAT_ITEM_RE.findall(block):
+ out.setdefault(name, "")
+ return out
+
+
+def parse_tool_catalog(debug_log_text, max_desc=200):
+ """Extract {tool_name: short_description} from a saved copilot debug log. Works on
+ the JSON-in-JSON escaped payload (the tool schemas the model is sent). Agent-agnostic
+ in spirit: any agent whose log carries name/description tool defs parses the same.
+
+ Schema-only by design: the flat name:fingerprint catalog is a SEPARATE source
+ (`parse_tool_catalog_flat`), and `trace.catalog_sources` tries both and picks the
+ richest. Folding the fallback in here would make the winning source unreportable --
+ a run using the flat catalog would still claim it had parsed schemas."""
+ if not debug_log_text:
+ return {}
+ s = (debug_log_text.replace('\\"', '"').replace('\\n', '\n').replace('\\\\', '\\'))
+ out = {}
+ for m in _CATALOG_RE.finditer(s):
+ name = m.group(1)
+ if name in out:
+ continue
+ desc = s[m.end():m.end() + max_desc + 40].split('\n')[0][:max_desc]
+ out[name] = desc
+ return out
+
+
+# Claude Code emits no debug log of raw schemas the way the Copilot CLI does. Its
+# stream-json transcript carries the loaded tool NAMES (the `system` init event's
+# `tools` list) but not their descriptions. The descriptions are a static property
+# of the CLI build, so we snapshot them ONCE (host-side, from a recorded request) into
+# claude_tool_catalog.json and JOIN the transcript's live tool names against it.
+# This is the agent-agnostic analog of parse_tool_catalog: same {name: desc} output,
+# same downstream classify_tools/learned_exclusions path — only the source differs.
+_CLAUDE_CATALOG_SNAPSHOT = os.path.join(os.path.dirname(__file__),
+ "claude_tool_catalog.json")
+
+
+def _load_claude_catalog_snapshot(path=None):
+ p = path or _CLAUDE_CATALOG_SNAPSHOT
+ try:
+ with open(p, encoding="utf-8") as f:
+ return json.load(f)
+ except Exception:
+ return {}
+
+
+def parse_tool_catalog_claude(transcript_text, max_desc=200, snapshot=None):
+ """Extract {tool_name: short_description} for a Claude Code run. Reads the loaded
+ tool NAMES from the transcript's `system` init event and joins them to the
+ description snapshot (claude_tool_catalog.json). Names present in the transcript
+ but missing from the snapshot get an empty description (the classifier still sees
+ the name); snapshot entries whose tool was NOT loaded this run are skipped, so the
+ catalog reflects what the agent was actually sent. Returns {} if no system event."""
+ if not transcript_text:
+ return {}
+ snap = snapshot if snapshot is not None else _load_claude_catalog_snapshot()
+ names = []
+ for line in transcript_text.splitlines():
+ line = line.strip()
+ if not line or '"type"' not in line or '"system"' not in line:
+ continue
+ try:
+ ev = json.loads(line)
+ except Exception:
+ continue
+ if ev.get("type") == "system" and isinstance(ev.get("tools"), list):
+ names = ev["tools"]
+ break
+ return {n: (snap.get(n, "") or "")[:max_desc] for n in names}
+
+
+TOOL_CLASSIFY_DOMAIN = (
+ "You configure the toolset for an autonomous coding agent that will solve GitHub "
+ "issues by READING, EDITING, and TESTING source code in a single repository. "
+ "Execution constraints: the agent runs HEADLESS and SINGLE-SHOT (no human is "
+ "available to answer questions), in one sealed container per task (no sub-agents, no "
+ "persistent cross-session state, no scheduling, no external memory). Each loaded "
+ "tool's full JSON schema is re-sent to the model on EVERY turn, costing tokens, so "
+ "tools that cannot possibly help edit-and-test code in this setting should be "
+ "removed.\n\nFor EACH tool below, classify it:\n"
+ " KEEP = could plausibly help read/navigate/edit/run/test code in a repo\n"
+ " EXCLUDE = structurally cannot help in this headless single-shot code-fixing "
+ "setting\n"
+ " UNSURE = ambiguous\n"
+ "Be conservative: only EXCLUDE when you are confident the tool is inapplicable BY "
+ "CONSTRUCTION. Return ONLY a JSON object {tool_name: {\"class\": "
+ "\"KEEP|EXCLUDE|UNSURE\", \"reason\": \"<short>\"}}.\n\nTOOLS:\n"
+)
+
+
+def classify_tools(catalog, runner, model="agent", domain=None):
+ """Optimizer step (v4.2): classify each tool in `catalog` ({name: desc}) as
+ KEEP/EXCLUDE/UNSURE via one LLM call. Returns ({tool: {class, reason}}, tokens).
+ Leakage-safe (reads only tool metadata + a fixed domain brief, never task content).
+ On any failure returns ({}, tokens) so the caller can fall back to the hardcoded
+ list. PROTECTED_TOOLS are force-KEEP regardless of the model (never let the LLM drop
+ the core work surface).
+
+ `domain` overrides the classification brief (defaults to the SWE-bench code-fixing
+ brief TOOL_CLASSIFY_DOMAIN). A different benchmark (e.g. a read-only secret-finding
+ scan) passes its own brief so applicability is judged against ITS task, not SWE-bench's
+ — this is what makes the mechanism self-contained across benchmarks rather than
+ tied to a hardcoded SWE-bench assumption. The brief must instruct the model to return
+ the same {tool: {class, reason}} JSON."""
+ if runner is None or not catalog:
+ return {}, 0
+ prompt = ((domain or TOOL_CLASSIFY_DOMAIN)
+ + "\n".join(f"- {n}: {d}" for n, d in catalog.items())
+ + "\n\nReturn the JSON classification:")
+ raw, real = runner(prompt, model)
+ raw = raw or ""
+ tokens = real if real is not None else _approx_tokens(prompt) + _approx_tokens(raw)
+ out = {}
+ m = re.search(r"\{.*\}", raw, re.S)
+ if m:
+ try:
+ obj = json.loads(m.group(0))
+ for t, v in obj.items():
+ cls = (v.get("class") if isinstance(v, dict) else v) or "UNSURE"
+ cls = str(cls).upper()
+ if cls not in ("KEEP", "EXCLUDE", "UNSURE"):
+ cls = "UNSURE"
+ reason = (v.get("reason") if isinstance(v, dict) else "") or ""
+ out[t] = {"class": cls, "reason": str(reason)[:160]}
+ except Exception:
+ return {}, tokens
+ # safety: never EXCLUDE a protected work tool, whatever the model said.
+ for t in list(out):
+ if t in PROTECTED_TOOLS and out[t]["class"] == "EXCLUDE":
+ out[t] = {"class": "KEEP", "reason": "protected work tool (forced)"}
+ return out, tokens
+
+
+TOOL_JUDGE_DOMAIN = (
+ "You configure the toolset for an autonomous coding agent that is about to solve ONE "
+ "specific GitHub issue by READING, EDITING, and TESTING source code in a single "
+ "repository, HEADLESS and SINGLE-SHOT (no human, one sealed container, no sub-agents / "
+ "scheduling / external memory). Every LOADED tool's JSON schema is re-sent on EVERY turn "
+ "and costs tokens. Below are the ISSUE text and the currently-available tools. For THIS "
+ "specific issue, mark each tool:\n"
+ " LOAD = plausibly useful to read/navigate/edit/run/test code to solve THIS issue\n"
+ " DROP = this specific issue will not need it\n"
+ "Be conservative: DROP only when you are confident the tool is irrelevant to THIS issue. "
+ "Do NOT assume a tool is needed merely because it exists. Return ONLY a JSON object "
+ "{tool_name: {\"load\": true|false, \"reason\": \"<short>\"}}.\n"
+)
+
+
+def judge_tools_for_task(problem_statement, catalog, runner, model="agent",
+ domain=None, max_issue_chars=4000):
+ """Per-task, INSTRUCTION-AWARE tool judge (v4.4). Reads the (redacted) issue text plus
+ the candidate tool catalog ({name: desc}) and returns ({tool: load_bool}, tokens): which
+ tools to LOAD for THIS task. Complements the trace-based rule — the caller unions the
+ DROP set with the evidence exclusions (and never drops PROTECTED or ever-used tools).
+
+ Leakage-safe: the caller MUST pass the SAME redacted problem statement the agent receives
+ (run_agent._redact_gold_pointers), never the gold patch or tests. On any failure returns
+ ({}, tokens) so the caller keeps everything loaded (safe). `tokens` is the cache-aware
+ EFFECTIVE cost of this one-shot call (fresh input x1 + output x5 when approximated), on
+ the same basis as exec, so judge overhead folds into the honest amortized task cost."""
+ if runner is None or not catalog or not problem_statement:
+ return {}, 0
+ brief = domain or TOOL_JUDGE_DOMAIN
+ prompt = (brief + "\nISSUE:\n" + problem_statement[:max_issue_chars]
+ + "\n\nTOOLS:\n" + "\n".join(f"- {n}: {d}" for n, d in catalog.items())
+ + "\n\nReturn the JSON classification:")
+ try:
+ raw, real = runner(prompt, model)
+ except Exception:
+ return {}, 0
+ raw = raw or ""
+ tokens = real if real is not None else (_approx_tokens(prompt) + 5 * _approx_tokens(raw))
+ out = {}
+ m = re.search(r"\{.*\}", raw, re.S)
+ if m:
+ try:
+ obj = json.loads(m.group(0))
+ for t, v in obj.items():
+ load = v.get("load") if isinstance(v, dict) else v
+ out[t] = bool(load)
+ except Exception:
+ return {}, tokens
+ # safety: never DROP a protected work tool, whatever the judge said.
+ for t in PROTECTED_TOOLS:
+ if t in out:
+ out[t] = True
+ return out, tokens
+
+
+def learned_exclusions(classification, used_tools=None, unused_streaks=None,
+ unsure_demote_after=4):
+ """Turn a v4.2 classification into the exclusion list for the NEXT task.
+
+ EXCLUDE-classified tools are excluded (unless protected or since-observed-in-use).
+ UNSURE tools stay LOADED until they have gone `unused_streaks[tool]` >=
+ `unsure_demote_after` consecutive tasks without use, then they are excluded too
+ (evidence-based demotion — safe because they were loaded and observed). KEEP never
+ excluded. Falls back to [] when there is no classification yet (caller then loads the
+ full catalog for that task = a re-audit/warm-up turn)."""
+ if not classification:
+ return []
+ used = set(used_tools or ())
+ streaks = unused_streaks or {}
+ excl = []
+ for t, v in classification.items():
+ if t in PROTECTED_TOOLS or t in used:
+ continue
+ cls = v.get("class")
+ if cls == "EXCLUDE":
+ excl.append(t)
+ elif cls == "UNSURE" and streaks.get(t, 0) >= unsure_demote_after:
+ excl.append(t)
+ return excl
+
+
+def detect(trajectory_path):
+ """STAGE A — free. Compute inefficiency signals from a saved trajectory, plus
+ TASK-AGNOSTIC evidence the optimizer can turn into durable notes. Two families:
+
+ ENV-DISCOVERY re-probing for the interpreter / conda env (empty bash probes).
+ NAVIGATION re-finding code: repeated search queries (grep/glob) and files
+ read multiple times — i.e. the agent re-locating the same things.
+
+ Evidence is restricted to *where things are* signals (search queries, file
+ paths) — never the contents of a fix — so it cannot encode a task's solution.
+ Notes derived from it are location/architecture facts, validated leakage-safe by
+ the optimizer's task-agnostic constraint and future-only credit."""
+ calls = load_trajectory(trajectory_path)
+ bash = [c for c in calls if c[0] in ("bash", "shell")]
+ cmds = [(_norm_cmd(a.get("command", "")), res, ok) for _, a, res, ok in bash]
+
+ def is_empty(res):
+ return re.sub(r"<shellId.*", "", res or "").strip() in ("", "(empty)")
+
+ # --- env-discovery family ---
+ probe_cmds = [c for c, _, _ in cmds if ENV_PROBE_RE.search(c)]
+ empty_probe_cmds = [c for c, r, _ in cmds if is_empty(r) and ENV_PROBE_RE.search(c)]
+ failed = sum(1 for _, _, ok in cmds if ok is False)
+ seen = {}
+ for c, _, _ in cmds:
+ seen[c] = seen.get(c, 0) + 1
+ repeats = sum(v - 1 for v in seen.values() if v > 1)
+ env_evidence = []
+ for c in (empty_probe_cmds or probe_cmds):
+ if c not in env_evidence:
+ env_evidence.append(c)
+ if len(env_evidence) >= 5:
+ break
+
+ # --- navigation family (structured tools + grep-in-bash) ---
+ searches, reads = [], []
+ for name, args, _, _ in calls:
+ if name in ("grep",):
+ q = (args.get("pattern") or "").strip()
+ if q:
+ searches.append(q.lower())
+ elif name in ("glob",):
+ q = (args.get("pattern") or "").strip()
+ if q:
+ searches.append(q.lower())
+ elif name in ("view",) or "read_file" in (name or ""):
+ # built-in view OR the read_file MCP tool (navcache/shadowread arms rename
+ # reads to read_file; count them so nav signals + anchoring still work)
+ p = args.get("path") or ""
+ if p:
+ reads.append(p)
+ # grep run through bash, too
+ for c, _, _ in cmds:
+ m = re.search(r"\b(?:grep|rg)\b[^|]*?['\"]([^'\"]{2,40})['\"]", c)
+ if m:
+ searches.append(m.group(1).lower())
+
+ def _dups(seq):
+ s = {}
+ for x in seq:
+ s[x] = s.get(x, 0) + 1
+ return s
+
+ sdup, rdup = _dups(searches), _dups(reads)
+ repeated_searches = sum(v - 1 for v in sdup.values() if v > 1)
+ redundant_reads = sum(v - 1 for v in rdup.values() if v > 1)
+ # evidence: the repeated queries / files (what the agent kept re-locating)
+ nav_evidence = ([f"searched repeatedly: {q!r}" for q, n in sdup.items() if n > 1][:4]
+ + [f"re-read file: {_basename(p)}" for p, n in rdup.items() if n > 1][:4])
+
+ # --- tooling-discovery family (generic, type-agnostic friction) ---
+ # (a) superseded test-runner form: the agent used >1 distinct test-runner form
+ # in one task -> it discovered the right one the hard way. Ordered, so we can
+ # report "tried X then settled on Y".
+ test_forms_seq = []
+ for c, _, _ in cmds:
+ for form, rx in TEST_FORM_RES.items():
+ if rx.search(c):
+ if not test_forms_seq or test_forms_seq[-1] != form:
+ test_forms_seq.append(form)
+ break
+ distinct_forms = list(dict.fromkeys(test_forms_seq))
+ superseded_tools = max(0, len(distinct_forms) - 1)
+ tooling_evidence = []
+ if superseded_tools:
+ tooling_evidence.append(
+ "tried test runners in order " + " -> ".join(test_forms_seq)
+ + f"; '{distinct_forms[-1]}' is the form that stuck")
+
+ # (b) infrastructure errors in command output (masked by pipes; success=True)
+ error_cmds = [(c, r) for c, r, _ in cmds if ERROR_MARKER_RE.search(r or "")]
+ error_outputs = len(error_cmds)
+ for c, r in error_cmds[:3]:
+ m = ERROR_MARKER_RE.search(r or "")
+ snippet = (m.group(0) if m else "")[:60]
+ tooling_evidence.append(f"command errored ({snippet}): {c[:70]}")
+
+ # (c) external fetches: web/github tools or bash curl/wget to a URL. Recorded for
+ # filtering (integrity) and as possible discovery waste; NOT used to trigger
+ # the optimizer (no leakage-safe note can come from "it searched the web").
+ external_fetches = 0
+ external_fetch_successes = 0
+ for name, args, result, success in calls:
+ if name in ("web_fetch", "web_search") or (name or "").startswith("github"):
+ external_fetches += 1
+ external_fetch_successes += int(
+ success is not False and not BLOCKED_FETCH_RE.search(result or ""))
+ for c, result, success in cmds:
+ # curl/wget to any URL, OR any reference to a GitHub gold-source host via any
+ # client (python urllib/requests, etc.) — the latter is the backstop that
+ # catches non-curl exfiltration the URL regex would miss.
+ if EXTERNAL_URL_RE.search(c) or GOLD_HOST_RE.search(c):
+ external_fetches += 1
+ external_fetch_successes += int(
+ success is not False and not BLOCKED_FETCH_RE.search(result or ""))
+
+ # --- execution-iteration family (targets the edit->test->fail loop that drives
+ # the costly multi-turn spirals; cost is ~linear in turns, and these patterns
+ # are turns the agent could have avoided). Evidence is COUNTS ONLY — no
+ # filenames, commands, or test labels — so it stays strictly leakage-safe
+ # (a process-discipline note, never a hint at the fix location). ---
+ exec_evidence = []
+ # (d) re-running an UNCHANGED test/repro command (the suite is deterministic, so
+ # a repeat of the exact same command yields the exact same result = wasted turn)
+ test_cmd_counts = {}
+ for c, _, _ in cmds:
+ if TEST_RUN_RE.search(c):
+ test_cmd_counts[c] = test_cmd_counts.get(c, 0) + 1
+ repeated_test_runs = sum(v - 1 for v in test_cmd_counts.values() if v > 1)
+ if repeated_test_runs:
+ exec_evidence.append(
+ f"re-ran an identical test/repro command {repeated_test_runs} extra "
+ "time(s) (the suite is deterministic — one run gives the same result)")
+ # (e) edit churn: many edits to a SINGLE file, with re-reads interleaved — the
+ # trial-and-error edit storm. Count per file basename (count only, no names).
+ edit_seq = [] # ordered basenames of edit targets
+ file_edits = {}
+ for name, args, _, _ in calls:
+ if name in ("edit", "str_replace", "create", "apply_patch"):
+ paths = args.get("paths") or [
+ args.get("path") or args.get("filePath") or ""]
+ for path in paths:
+ p = _basename(path)
+ if p:
+ edit_seq.append(p)
+ file_edits[p] = file_edits.get(p, 0) + 1
+ edit_churn = max(file_edits.values()) if file_edits else 0
+ if edit_churn >= 3:
+ exec_evidence.append(
+ f"made {edit_churn} edits to a single file (trial-and-error iteration); "
+ "forming a hypothesis from one focused read before editing, and "
+ "verifying with the cheapest reproduction, avoids edit-test churn")
+
+ return {
+ "n_bash": len(bash),
+ "n_tool_calls": len(calls),
+ "env_probes": len(probe_cmds),
+ "empty_outputs": sum(1 for _, r, _ in cmds if is_empty(r)),
+ "failed_cmds": failed,
+ "repeated_cmds": repeats,
+ "n_searches": len(searches),
+ "repeated_searches": repeated_searches,
+ "redundant_reads": redundant_reads,
+ "superseded_tools": superseded_tools,
+ "error_outputs": error_outputs,
+ "external_fetches": external_fetches,
+ "external_fetch_successes": external_fetch_successes,
+ "repeated_test_runs": repeated_test_runs,
+ "edit_churn": edit_churn,
+ "evidence_env_cmds": env_evidence,
+ "evidence_nav": nav_evidence,
+ "evidence_tooling": tooling_evidence,
+ "evidence_exec": exec_evidence,
+ }
+
+
+# ---- the gate ----------------------------------------------------------------
+DEFAULT_THRESHOLDS = {
+ # "obvious in this task": any single signal at/above these fires
+ "env_probes": 5,
+ "empty_outputs": 6,
+ "repeated_cmds": 3,
+ "repeated_searches": 3,
+ "redundant_reads": 5,
+ "superseded_tools": 1, # one wrong-then-right tool switch is a clean discovery
+ "error_outputs": 2, # repeated infra errors -> a durable "how to" note
+ "repeated_test_runs": 1, # re-running an unchanged test/repro = a wasted turn
+ "edit_churn": 5, # >=5 edits to one file = a trial-and-error edit storm
+ # "recurring across window": a signal averaging >= this over the last W tasks
+ "window": 3,
+ "recurring_env_probes_avg": 2.0,
+ "recurring_empty_avg": 3.0,
+ "recurring_repeated_searches_avg": 2.0,
+ "recurring_superseded_avg": 0.5,
+ "recurring_edit_churn_avg": 3.0,
+}
+
+
+def trigger(signals, recent_signals, thr=None):
+ """GATE — decide whether to invoke the paid optimizer. Returns (bool, reason).
+ Fires on an obvious single-task spike (env-discovery / navigation / tooling-
+ discovery / execution-iteration) OR a pattern recurring across the window;
+ otherwise no-op (interface unchanged). external_fetches is deliberately NOT a
+ trigger (no leakage-safe note can be distilled from web/github access)."""
+ thr = thr or DEFAULT_THRESHOLDS
+ # (1) obvious single-task inefficiency.
+ # NB (v1.4): edit_churn and redundant_reads are deliberately NOT triggers. They
+ # measure trial-and-error iteration / re-reading, which is solve-loop work bound to
+ # task difficulty, not interface friction. Firing on them produced generic process-
+ # habit notes ("read the function first", "batch reads") that added upfront reading
+ # WITHOUT shrinking the edit->test loop, inflating cost on matplotlib. They are still
+ # computed in detect() and logged as diagnostics, just not acted upon.
+ for k in ("superseded_tools", "error_outputs",
+ "repeated_test_runs",
+ "env_probes", "empty_outputs", "repeated_cmds", "repeated_searches"):
+ if signals.get(k, 0) >= thr[k]:
+ return True, f"obvious:{k}={signals[k]}>={thr[k]}"
+ if METHOD == "v1.3":
+ # v1.3 also triggers on edit_churn / redundant_reads (P1, removed in v1.4).
+ for k in ("edit_churn", "redundant_reads"):
+ if signals.get(k, 0) >= thr[k]:
+ return True, f"obvious:{k}={signals[k]}>={thr[k]}"
+ # (2) recurring across the window (more general, less leakage-prone)
+ win = recent_signals[-thr["window"]:]
+ if len(win) >= thr["window"]:
+ for sig_key, thr_key in (("env_probes", "recurring_env_probes_avg"),
+ ("empty_outputs", "recurring_empty_avg"),
+ ("repeated_searches", "recurring_repeated_searches_avg"),
+ ("superseded_tools", "recurring_superseded_avg")):
+ avg = sum(s.get(sig_key, 0) for s in win) / len(win)
+ if avg >= thr[thr_key]:
+ return True, f"recurring:{sig_key}_avg={avg:.1f}"
+ return False, "below-threshold"
+
+
+# ---- coverage suppression ----------------------------------------------------
+# Signals where a SINGLE durable note addresses ALL future occurrences (e.g. "use
+# runtests.py, not pytest" covers every later pytest detour; an env-setup note covers
+# every later env re-probe). Once the manifest covers such a signal, re-firing the paid
+# optimizer on it just re-derives a duplicate and returns empty — pure waste.
+# error_outputs stays uncovered (each infra error can need a genuinely different
+# 'use X to do Y' fix). NB (v1.4): edit_churn / redundant_reads are no longer triggers
+# (see trigger()), so they are not listed here.
+GLOBALLY_COVERABLE = {"superseded_tools", "env_probes",
+ "repeated_test_runs", "repeated_searches"}
+
+
+def primary_signal(reason):
+ """Extract the signal key from a trigger() reason string, e.g.
+ 'obvious:superseded_tools=1>=1' or 'recurring:env_probes_avg=2.0' -> the key."""
+ if not reason or ":" not in reason:
+ return None
+ body = reason.split(":", 1)[1]
+ key = re.split(r"[=]", body, 1)[0]
+ key = re.sub(r"_avg$", "", key)
+ return key
+
+
+def is_covered(reason, state):
+ """True if the gate fired on a GLOBALLY_COVERABLE signal that the manifest already
+ covers — so the paid optimizer would only re-derive a duplicate. Suppress it."""
+ k = primary_signal(reason)
+ return k in GLOBALLY_COVERABLE and k in set(state.get("covered_signals", []))
+
+
+def mark_covered(reason, state):
+ """Record that a GLOBALLY_COVERABLE signal is now addressed by the manifest. Call
+ only after the optimizer actually ADDED a note in response to this firing."""
+ k = primary_signal(reason)
+ if k in GLOBALLY_COVERABLE:
+ cov = state.setdefault("covered_signals", [])
+ if k not in cov:
+ cov.append(k)
+
+
+# ---- the paid optimizer ------------------------------------------------------
+# METHOD selects the optimizer generation (set via AGENTEO_METHOD env, default v1.4):
+# v1.4 (default): bans process-habit notes (P2) and does NOT trigger on edit_churn/
+# redundant_reads (P1) — the committed milestone.
+# v1.3 : the prior hardened global manifest — ALLOWS process-discipline
+# notes (category d) and triggers on edit_churn/redundant_reads.
+# (v1.2 location notes are not exposed here; both v1.3/v1.4 forbid location facts.)
+METHOD = os.environ.get("AGENTEO_METHOD", "v1.4")
+
+OPTIMIZER_SYSTEM = (
+ "You maintain a SHORT list of durable, GENERAL operating notes for working "
+ "efficiently in a software repository. You are shown signals of wasted effort "
+ "from a recent task trajectory and the current notes. Propose at most {k} NEW "
+ "notes that would help FUTURE tasks avoid the same waste.\n\n"
+ "GOAL: capture concrete, checkable facts about THIS repo's tooling and "
+ "environment so future tasks use the right command/form the first time. NEVER "
+ "reduce the number of attempts the agent is willing to make to reach a correct, "
+ "test-passing solution, and NEVER tell it to stop early.\n\n"
+ "STRICT RULES:\n"
+ "- ALLOWED note categories ONLY: (a) the correct command/tool to "
+ "build/test/run/lint here, and which forms DON'T work; (b) environment/config "
+ "facts and gotchas (interpreter, import-path conventions, how to reproduce); "
+ "(c) infrastructure-error fixes (the durable 'use X to do Y' fact behind a "
+ "recurring traceback). EVERY note must name a concrete command, flag, path, "
+ "tool, or environment fact that an unrelated future task could act on verbatim.\n"
+ "- FORBIDDEN: (1) Do NOT write notes about WHERE specific components, symbols, "
+ "subsystems, classes, or tests LIVE (no 'X lives in file Y', no per-file or "
+ "per-symbol location facts) — these are task-specific, leak localization, and "
+ "mislead unrelated tasks. (2) Do NOT write any note that caps iteration, "
+ "discourages re-running tests, tells the agent to 'trust' a result, 'make one "
+ "consolidated change', stop early, or otherwise reduce its willingness to keep "
+ "iterating until the tests actually pass. (3) NEVER describe or hint at the "
+ "solution to any specific task, which file to change, or which lines. (4) Do NOT "
+ "write generic process/workflow-discipline advice (e.g. 'read the full function "
+ "before editing', 'batch related reads in one pass', 'trace callers/data flow "
+ "first', 'form a hypothesis before editing', 'reproduce with the cheapest check', "
+ "'reuse earlier search results instead of re-searching') — these are not repo-"
+ "specific, have no checkable command/path footprint, and slow the edit->test "
+ "solve loop without preventing real interface friction.\n"
+ "- Each note one line, concise, durable across code changes, and applicable to "
+ "ANY task in this repo (not just tasks touching one file).\n"
+ "- If nothing general, durable, and repo-wide can be added, return an empty list.\n"
+ "Return ONLY a JSON list of strings."
+)
+
+# v1.3 optimizer: the pre-v1.4 prompt — ALLOWS process-discipline notes (category d)
+# and omits FORBIDDEN(4). Location facts (FORBIDDEN 1) and effort-capping (FORBIDDEN 2)
+# stay banned (those distinguish v1.3 from the discarded v1.2).
+OPTIMIZER_SYSTEM_V13 = (
+ "You maintain a SHORT list of durable, GENERAL operating notes for working "
+ "efficiently in a software repository. You are shown signals of wasted effort "
+ "from a recent task trajectory and the current notes. Propose at most {k} NEW "
+ "notes that would help FUTURE tasks avoid the same waste.\n\n"
+ "GOAL: make each step the agent takes cheaper / more informed. NEVER reduce the "
+ "number of attempts the agent is willing to make to reach a correct, test-passing "
+ "solution.\n\n"
+ "STRICT RULES:\n"
+ "- ALLOWED note categories ONLY: (a) the correct command/tool to build/test/run/"
+ "lint here, and which forms DON'T work; (b) environment/config facts and gotchas; "
+ "(c) infrastructure-error fixes (the durable 'use X to do Y' fact behind a "
+ "recurring traceback); (d) step-cheapening workflow habits that do NOT cap effort "
+ "(e.g. read a section once before editing it, batch related reads in one pass, "
+ "confirm an import path before running a probe).\n"
+ "- FORBIDDEN: (1) Do NOT write notes about WHERE specific components, symbols, "
+ "subsystems, classes, or tests LIVE (no per-file/per-symbol location facts). "
+ "(2) Do NOT write any note that caps iteration, discourages re-running tests, "
+ "tells the agent to 'trust' a result, stop early, or otherwise reduce its "
+ "willingness to keep iterating. (3) NEVER hint at the solution to any task.\n"
+ "- Each note one line, concise, durable, applicable to ANY task in this repo.\n"
+ "- If nothing general and durable can be added, return an empty list.\n"
+ "Return ONLY a JSON list of strings."
+)
+
+
+def _optimizer_system():
+ return OPTIMIZER_SYSTEM_V13 if METHOD == "v1.3" else OPTIMIZER_SYSTEM
+
+
+def build_optimizer_prompt(signals, state, k=3):
+ cur = [n["text"] for n in state["notes"] if not n.get("pruned")]
+ env_ev = signals.get("evidence_env_cmds", [])
+ tool_ev = signals.get("evidence_tooling", [])
+ counts = {k2: v for k2, v in signals.items()
+ if not k2.startswith("evidence_")}
+ # v1.4: only the Class-A evidence blocks (tooling-discovery and wasted env-
+ # discovery commands) are surfaced. The execution-iteration (edit-churn) and
+ # repeated-navigation blocks were removed — they solicited generic process-habit
+ # notes ("read the section once", "batch reads") that inflated the solve loop
+ # without preventing real interface friction.
+ blocks = ""
+ if tool_ev:
+ blocks += ("\n\nTooling-discovery waste (the agent used the wrong command/tool "
+ "form first, then corrected, or hit an infrastructure error — infer "
+ "the durable 'use <tool/command> to do <task>' fact, NOT any fix):\n"
+ + "\n".join(f" - {c}" for c in tool_ev))
+ if env_ev:
+ blocks += ("\n\nWasted environment-discovery commands (returned nothing — "
+ "infer the durable env fact that makes them unnecessary):\n"
+ + "\n".join(f" $ {c}" for c in env_ev))
+ if METHOD == "v1.3":
+ # v1.3 also surfaces execution-iteration + repeated-navigation evidence, which
+ # solicit the process-discipline notes v1.3 allows (and v1.4 banned).
+ exec_ev = signals.get("evidence_exec", [])
+ nav_ev = signals.get("evidence_nav", [])
+ if exec_ev:
+ blocks += ("\n\nExecution-iteration waste (edit churn / re-running):\n"
+ + "\n".join(f" - {c}" for c in exec_ev))
+ if nav_ev:
+ blocks += ("\n\nRepeated-navigation waste (re-searching / re-reading):\n"
+ + "\n".join(f" - {c}" for c in nav_ev))
+ return (_optimizer_system().format(k=k)
+ + "\n\nCurrent notes:\n" + ("\n".join(f"- {c}" for c in cur) or "(none)")
+ + "\n\nWaste signal counts from the latest task:\n" + json.dumps(counts)
+ + blocks
+ + "\n\nReturn the JSON list of new notes:")
+
+
+def optimize(signals, state, model="agent", k=3, runner=None):
+ """STAGE B — paid. Ask the configured optimizer model for new durable notes.
+ `runner(prompt, model) -> (str, real_tokens)` performs the LLM call (so the model
+ is fully configurable and the call is injectable/testable). Returns
+ (notes, tokens) where tokens is the call's REAL token usage when the runner can
+ recover it, else a char-based approximation. Parses a JSON list of strings;
+ tolerant of extra prose around it."""
+ prompt = build_optimizer_prompt(signals, state, k=k)
+ if runner is None:
+ # no runner wired (offline/dry): propose nothing, cost nothing.
+ return [], 0
+ raw, real = runner(prompt, model)
+ raw = raw or ""
+ notes = _parse_notes(raw)
+ tokens = real if real is not None else _approx_tokens(prompt) + _approx_tokens(raw)
+ return notes[:k], tokens
+
+
+def _parse_notes(raw):
+ m = re.search(r"\[.*\]", raw, re.S)
+ if not m:
+ return []
+ try:
+ arr = json.loads(m.group(0))
+ return [str(x).strip() for x in arr if str(x).strip()]
+ except Exception:
+ return []
+
+
+# ---- v2.1 anchored-knowledge optimizer ---------------------------------------
+# A SEPARATE, parallel optimizer that produces FILE-ANCHORED structural facts about
+# the files the agent navigated heavily this task. Kept separate from the global
+# optimize() so the global tier is byte-identical to the v1.3 hardened optimizer
+# (clean A/B: v2.1 = v1.3 global manifest + anchored). These facts are the location/
+# structure knowledge the v1.3 global optimizer had to BAN from the global manifest —
+# safe here because they are surfaced ONLY when the agent
+# opens that file (no global pollution, no pre-leak of localization).
+ANCHORED_OPTIMIZER_SYSTEM = (
+ "You record durable, FILE-SPECIFIC structural knowledge about a code repository, "
+ "to help FUTURE tasks that edit the SAME files work with less re-exploration. You "
+ "are shown files the agent navigated/re-read heavily in a recent task. For each "
+ "file worth a note, state ONE concise, durable fact about THAT FILE that a future "
+ "editor of it would want to know up front: what it contains, key classes/base "
+ "classes and their contract, conventions, extension points, or gotchas.\n\n"
+ "STRICT RULES:\n"
+ "- The fact must be about the FILE'S STABLE STRUCTURE, true regardless of any "
+ "specific task — NOT the solution to the current task, NOT which lines to change, "
+ "NOT a fix. If the only thing you can say about a file is task-specific, skip it.\n"
+ "- One short line per file. Durable across code changes when possible.\n"
+ "- Only include files where a genuine, reusable structural fact exists.\n"
+ "Return ONLY a JSON list of objects: [{\"path\": \"<repo-relative path>\", "
+ "\"text\": \"<one-line structural fact>\"}].")
+
+
+def _heavy_files(trajectory_path, min_reads=2, top=4):
+ """Repo-relative paths the agent read >= min_reads times (navigation hotspots),
+ most-read first. These are where anchored knowledge has the most future payoff."""
+ counts = {}
+ for name, args, _, _ in load_trajectory(trajectory_path):
+ a = args if isinstance(args, dict) else {}
+ if name == "view" or "read_file" in (name or ""):
+ p = (a.get("path") or "").strip()
+ if p:
+ counts[p] = counts.get(p, 0) + 1
+ heavy = sorted((p for p, c in counts.items() if c >= min_reads),
+ key=lambda p: -counts[p])[:top]
+ return heavy
+
+
+def optimize_anchored(trajectory_path, state, model="agent", k=4, runner=None):
+ """STAGE B (anchored) — paid, parallel to optimize(). Ask the model for durable
+ file-structural facts about the navigation-hotspot files. Returns (items, tokens)
+ where items = [{path, text, tag}]. No-op (0 cost) when no heavy files or no runner."""
+ if runner is None:
+ return [], 0
+ heavy = _heavy_files(trajectory_path)
+ if not heavy:
+ return [], 0
+ existing = {p: [n["text"] for n in b] for p, b in (state.get("anchored") or {}).items()}
+ prompt = (ANCHORED_OPTIMIZER_SYSTEM
+ + "\n\nFiles the agent navigated/re-read heavily this task:\n"
+ + "\n".join(f"- {p}" for p in heavy)
+ + "\n\nExisting anchored facts (do not duplicate):\n"
+ + (json.dumps(existing) if existing else "(none)")
+ + "\n\nReturn the JSON list of {path, text}:")
+ raw, real = runner(prompt, model)
+ items = _parse_anchored(raw or "")
+ # keep only facts whose path is one we actually navigated (no hallucinated paths)
+ heavy_set = set(heavy)
+ items = [{"path": it["path"], "text": it["text"], "tag": "uncertain"}
+ for it in items if it.get("path") in heavy_set and it.get("text")][:k]
+ tokens = real if real is not None else _approx_tokens(prompt) + _approx_tokens(raw)
+ return items, tokens
+
+
+def _parse_anchored(raw):
+ m = re.search(r"\[.*\]", raw, re.S)
+ if not m:
+ return []
+ try:
+ arr = json.loads(m.group(0))
+ out = []
+ for x in arr:
+ if isinstance(x, dict) and x.get("path") and x.get("text"):
+ out.append({"path": str(x["path"]).strip(),
+ "text": str(x["text"]).strip()})
+ return out
+ except Exception:
+ return []
+
+
+def _approx_tokens(text):
+ return max(1, len(text) // 4)
+
+
+def _runner_effective_or_approx(runner, prompt, raw, real):
+ """Use recovered usage, then a runner's model-priced estimate, then legacy ×5."""
+ if real is not None:
+ return real
+ estimator = getattr(runner, "estimate_effective", None)
+ if callable(estimator):
+ try:
+ return estimator(prompt, raw)
+ except Exception:
+ pass
+ return _approx_tokens(prompt) + 5 * _approx_tokens(raw)
+
+
+# ---- v2.2 subsystem fix-map optimizer ----------------------------------------
+# v2.1's anchored tier failed on realistic streams: it anchored at FILE granularity
+# (cross-task file-revisit ~12-33%) and learned from within-task re-reads, so its
+# delivery surface was near-zero. v2.2 fixes BOTH: it captures from the RESOLVED
+# patch's EDITED files (the proven solve sites), anchors at SUBSYSTEM (dir) level —
+# where revisit is 79% on xarray — and records only SYMBOL-level structure (no line
+# numbers, no defect), so it survives the median ~69-line churn between visits
+# (measured symbol-survival = 100% median). Delivery is verify-on-inject (the reader
+# drops any mapped symbol that no longer exists), so drift can't mislead.
+FIXMAP_OPTIMIZER_SYSTEM = (
+ "You record a durable SUBSYSTEM MAP for a code repository, to help FUTURE tasks "
+ "that edit the SAME directory navigate to the fix site faster. You are shown the "
+ "directory of a just-RESOLVED task and the functions/classes that were edited. For "
+ "each, state ONE concise, durable line: the symbol name and its responsibility, "
+ "and the test file that validates it.\n\n"
+ "STRICT RULES:\n"
+ "- Anchor on the SYMBOL (function/class name) and its STABLE ROLE — NEVER line "
+ "numbers, NEVER the specific bug or fix, NEVER what to change. A future reader "
+ "must not be able to reconstruct this task's solution from your map.\n"
+ "- One short line per symbol: `symbol — role (validated by tests/<file>)`.\n"
+ "- Skip anything that is only true for this one defect.\n"
+ "Return ONLY a JSON list of {\"path\": \"<dir>\", \"text\": \"<line>\"}.")
+
+
+def _edited_symbols(patch_path):
+ """From a unified diff, map each edited .py file to the def/class symbols whose
+ bodies were touched. Returns {dir: set(symbols)}; line numbers are NOT retained."""
+ dirs = {}
+ cur, hunk_syms = None, set()
+ if not os.path.isfile(patch_path):
+ return dirs
+ sym_re = re.compile(r"(?:def|class)\s+(\w+)")
+ for ln in open(patch_path, errors="replace"):
+ if ln.startswith("+++ b/"):
+ cur = ln[6:].strip()
+ elif ln.startswith("@@") and cur and cur.endswith(".py") and "test" not in cur:
+ m = sym_re.search(ln) # hunk header often carries the enclosing def/class
+ if m:
+ dirs.setdefault(os.path.dirname(cur), set()).add(m.group(1))
+ elif (ln.startswith("+") or ln.startswith("-")) and cur and cur.endswith(".py") \
+ and "test" not in cur:
+ m = sym_re.search(ln)
+ if m:
+ dirs.setdefault(os.path.dirname(cur), set()).add(m.group(1))
+ return dirs
+
+
+def optimize_fixmap(patch_path, state, model="agent", k=4, runner=None):
+ """v2.2 — paid, fires only on RESOLVED tasks. Build symbol-level subsystem facts
+ from the edited dirs. Returns (items, tokens); items=[{path=dir, text, tag}]."""
+ if runner is None:
+ return [], 0
+ dirs = _edited_symbols(patch_path)
+ if not dirs:
+ return [], 0
+ existing = {d: [n["text"] for n in b] for d, b in (state.get("anchored") or {}).items()}
+ desc = "\n".join(f"- {d}: {', '.join(sorted(s))}" for d, s in dirs.items() if s)
+ if not desc.strip():
+ return [], 0
+ prompt = (FIXMAP_OPTIMIZER_SYSTEM
+ + "\n\nResolved task edited these subsystems (dir: symbols):\n" + desc
+ + "\n\nExisting subsystem maps (do not duplicate):\n"
+ + (json.dumps(existing) if existing else "(none)")
+ + "\n\nReturn the JSON list of {path, text}:")
+ raw, real = runner(prompt, model)
+ items = _parse_anchored(raw or "")
+ valid_dirs = set(dirs)
+ items = [{"path": it["path"], "text": it["text"], "tag": "uncertain"}
+ for it in items if it.get("path") in valid_dirs and it.get("text")][:k]
+ tokens = real if real is not None else _approx_tokens(prompt) + _approx_tokens(raw)
+ return items, tokens
+
+
+# ---- optional LLM-detector tier ----------------------------------------------
+# Answers the design question "must the detector use a fixed vocabulary?". The free
+# heuristic detect()/trigger() runs every task (cheap, and it also flags integrity /
+# external-fetch regardless). This tier is an LLM that reads a LEAKAGE-SAFE PROCESS
+# digest of one trajectory and decides, open-endedly, whether there is a recurring,
+# task-agnostic inefficiency worth a durable note — without being told the signal
+# types in advance. Because an LLM call per task would reintroduce the very overhead
+# the gate exists to remove, it is meant to be SAMPLED (every Nth task, or only when
+# the free detector is ambiguous), never run unconditionally.
+
+def trajectory_summary(trajectory_path, max_cmds=25):
+ """A compact, LEAKAGE-SAFE digest of an agent's PROCESS (not its solution): the
+ bash commands it ran, what it searched for, which files it opened (basenames),
+ and infrastructure-error snippets — deliberately EXCLUDING the patch/diff, edit
+ contents, view contents, and the issue text, so it cannot encode the fix."""
+ calls = load_trajectory(trajectory_path)
+ lines, n_view, n_edit = [], 0, 0
+ for name, args, res, ok in calls:
+ if name == "bash":
+ c = _norm_cmd(args.get("command", ""))
+ err = ERROR_MARKER_RE.search(res or "")
+ tag = f" [err: {err.group(0)[:40]}]" if err else ""
+ if c:
+ lines.append(f"$ {c[:120]}{tag}")
+ elif name in ("grep", "glob"):
+ q = (args.get("pattern") or "").strip()
+ if q:
+ lines.append(f"search {name}: {q[:60]}")
+ elif name == "view":
+ n_view += 1
+ p = _basename(args.get("path") or "")
+ if p:
+ lines.append(f"view {p}")
+ elif name in ("edit", "str_replace", "create"):
+ n_edit += 1
+ elif name in ("web_fetch", "web_search") or (name or "").startswith("github"):
+ lines.append(f"EXTERNAL {name}")
+ # cap to keep the digest cheap and process-focused
+ if len(lines) > max_cmds:
+ lines = lines[:max_cmds] + [f"... (+{len(lines)-max_cmds} more actions)"]
+ head = f"({len(calls)} tool calls; {n_view} views, {n_edit} edits)"
+ return head + "\n" + "\n".join(lines)
+
+
+LLM_DETECTOR_SYSTEM = (
+ "You audit an AI coding agent's PROCESS trace for one task in a repository it "
+ "will face MORE tasks in. Decide whether the trace shows a RECURRING, "
+ "TASK-AGNOSTIC inefficiency that a short durable repo note would prevent next "
+ "time — e.g. discovering the right build/test/run command the hard way, "
+ "re-finding where code lives, or re-probing the environment.\n\n"
+ "Do NOT infer or describe the task's solution or which lines to change. Judge "
+ "only the agent's workflow efficiency.\n"
+ "Return ONLY a JSON object: {\"worth_optimizing\": bool, \"kind\": \"<short "
+ "label e.g. tooling-discovery|navigation|env|none>\", \"evidence\": [\"<short "
+ "leakage-safe observations the optimizer can turn into a durable note>\"]}. "
+ "If the workflow was already efficient, return worth_optimizing=false, kind="
+ "\"none\", evidence=[]."
+)
+
+
+def llm_detect(trajectory_path, runner, model="agent"):
+ """SAMPLED LLM detector. Returns (verdict_dict, tokens). verdict_dict has keys
+ worth_optimizing(bool), kind(str), evidence(list[str]). On any failure returns a
+ safe negative verdict. The evidence can be fed straight into the optimizer via
+ signals['evidence_tooling']."""
+ if runner is None:
+ return {"worth_optimizing": False, "kind": "none", "evidence": []}, 0
+ summary = trajectory_summary(trajectory_path)
+ prompt = (LLM_DETECTOR_SYSTEM + "\n\nProcess trace:\n" + summary
+ + "\n\nReturn the JSON verdict:")
+ raw, real = runner(prompt, model)
+ raw = raw or ""
+ tokens = real if real is not None else _approx_tokens(prompt) + _approx_tokens(raw)
+ verdict = {"worth_optimizing": False, "kind": "none", "evidence": []}
+ m = re.search(r"\{.*\}", raw, re.S)
+ if m:
+ try:
+ obj = json.loads(m.group(0))
+ verdict["worth_optimizing"] = bool(obj.get("worth_optimizing"))
+ verdict["kind"] = str(obj.get("kind", "none"))[:40]
+ ev = obj.get("evidence", [])
+ verdict["evidence"] = [str(x)[:160] for x in ev][:5] if isinstance(ev, list) else []
+ except Exception:
+ pass
+ return verdict, tokens
+
+
+# ---- v4.1 spiral trace-judge (live abort decision; same LLM-detector machinery) ----
+# The Phase-1.5 abort decision was step-count-gated (resolved-turn envelope). This is the
+# TRACE-CONTENT tier: an LLM reads a leakage-safe digest of the agent's RECENT actions
+# (built by spiral.SpiralState.recent_digest — commands / searches / edited basenames /
+# error SIGNATURES only, never fix content) and judges whether the trajectory is an
+# unrecoverable loop or still making progress. It is the same sampled-LLM pattern as
+# llm_detect (leakage-safe process digest -> open-ended verdict), here wired to the live
+# spiral guard rather than the between-task optimizer. Meant to be called RARELY — only
+# when the free heuristic (spiral.is_stuck) already flagged a candidate stuck state.
+STUCK_JUDGE_SYSTEM = (
+ "You monitor an AI coding agent that is partway through ONE task in a software repo. "
+ "A cheap heuristic flagged its recent process as a possible UNPRODUCTIVE LOOP. Read "
+ "the leakage-safe digest of its RECENT actions and judge whether it is genuinely "
+ "STUCK — repeating the same kind of edit/test/failure with no progress toward green "
+ "tests, unlikely to recover if it keeps going the same way — versus still making "
+ "real progress (changing failures, narrowing the problem, exploring new hypotheses).\n\n"
+ "Judge ONLY the workflow from the digest; do NOT infer the task's solution or which "
+ "lines to change. Favor SPARING (stuck=false) when there is any sign of progress; "
+ "only return stuck=true with high confidence for a clear, repetitive dead-end.\n"
+ "Return ONLY a JSON object: {\"stuck\": bool, \"confidence\": <0.0-1.0>, "
+ "\"reason\": \"<short leakage-safe justification>\"}."
+)
+
+
+def llm_stuck_judge(digest, runner, model="agent"):
+ """SAMPLED LLM trace-judge for the live spiral guard. `digest` is a leakage-safe
+ recent-process string (spiral.SpiralState.recent_digest()). Returns a verdict dict
+ {stuck: bool|None, confidence: float, reason: str, _tokens: number}; stuck=None on any
+ failure (the caller treats None/ambiguous by falling back to the envelope gate).
+ `_tokens` is the REAL token cost of this paid judge call (input+output+cache, from the
+ runner's usage block; a char-based approximation if the runner couldn't recover it) so
+ the harness can fold mechanism-internal LLM cost into the amortized task cost. 0 when no
+ call was made."""
+ if runner is None or not digest:
+ return {"stuck": None, "confidence": 0.0, "reason": "no-runner", "_tokens": 0}
+ prompt = (STUCK_JUDGE_SYSTEM + "\n\nRecent process digest:\n" + digest
+ + "\n\nReturn the JSON verdict:")
+ try:
+ raw, _real = runner(prompt, model)
+ except Exception as e:
+ return {"stuck": None, "confidence": 0.0,
+ "reason": f"runner-error:{e}"[:60], "_tokens": 0}
+ raw = raw or ""
+ # EFFECTIVE (cache-aware) cost when the runner recovered usage; else approximate on the
+ # SAME basis: an uncached one-shot -> prompt is fresh input (weight 1), reply is output
+ # (weight 5). Keeps mechanism cost comparable to exec's cache-aware effective tokens.
+ _tokens = _runner_effective_or_approx(runner, prompt, raw, _real)
+ verdict = {"stuck": None, "confidence": 0.0, "reason": "", "_tokens": _tokens}
+ m = re.search(r"\{.*\}", raw, re.S)
+ if m:
+ try:
+ obj = json.loads(m.group(0))
+ verdict["stuck"] = (bool(obj["stuck"]) if "stuck" in obj
+ and obj["stuck"] is not None else None)
+ verdict["confidence"] = max(0.0, min(1.0, float(obj.get("confidence", 0) or 0)))
+ verdict["reason"] = str(obj.get("reason", ""))[:160]
+ except Exception:
+ pass
+ return verdict
+
+
+# ---- v6.1 LLM-targeted rung-1 NUDGE writer -----------------------------------
+# The v4.3 rung-1 nudge (spiral.nudge_text) is leakage-safe but GENERIC: it only
+# splices counts into a fixed template ("N edits across M files, same failure recurred
+# K times -> try something different"). This writer makes the redirect TARGETED: a
+# sampled LLM reads the SAME leakage-safe recent-process digest the stuck-judge sees
+# (spiral.SpiralState.recent_digest — commands / searches / edited basenames / error
+# SIGNATURES only) and names the specific repetitive pattern + a materially different
+# next step. It carries NO external knowledge (no issue text, no fix content, no gold
+# pointer), so it stays inside the effort-capping ban exactly like nudge_text: it may
+# only REDIRECT effort, never stop it. Falls back to the template on any failure.
+NUDGE_WRITER_SYSTEM = (
+ "An AI coding agent is stuck in a REPETITIVE, UNPRODUCTIVE loop partway through one "
+ "task in a software repo. You are given a leakage-safe digest of its RECENT ACTIONS "
+ "only (tool names, shell commands, searched patterns, edited file BASENAMES, and "
+ "error SIGNATURES — never file contents or the task description). Write a SHORT "
+ "redirect (2-4 sentences) addressed to the agent that: (1) names the specific "
+ "repetitive pattern you actually see in the digest (e.g. re-editing the same file, "
+ "re-running the same test to the same failure); (2) proposes a MATERIALLY DIFFERENT "
+ "next step (a different file/root-cause to consider, re-reading the failing test's "
+ "expectation, verifying a new hypothesis before editing again).\n"
+ "HARD CONSTRAINTS: Do NOT tell it to stop, give up, or reduce effort — only redirect. "
+ "Do NOT infer the task's solution, name specific line numbers to change, or reference "
+ "any code/identifier you cannot see in the digest. Output ONLY the redirect as plain "
+ "text — no preamble, no JSON, no headings."
+)
+
+
+def llm_nudge_writer(digest, runner, model="agent"):
+ """SAMPLED LLM writer for the v6.1 targeted rung-1 nudge. `digest` is the same
+ leakage-safe recent-process string fed to llm_stuck_judge
+ (spiral.SpiralState.recent_digest()). Returns {"text": str, "_tokens": number}: a
+ targeted, leakage-safe redirect and the REAL cache-aware effective token cost of the
+ call (folded into amortized task cost by the harness, same basis as the judge). On any
+ failure returns {"text": "", "_tokens": 0} so the caller falls back to the generic
+ spiral.nudge_text template (resolve-safe zero-LLM path)."""
+ if runner is None or not digest:
+ return {"text": "", "_tokens": 0}
+ prompt = (NUDGE_WRITER_SYSTEM + "\n\nRecent process digest:\n" + digest
+ + "\n\nWrite the redirect:")
+ try:
+ raw, _real = runner(prompt, model)
+ except Exception:
+ return {"text": "", "_tokens": 0}
+ raw = (raw or "").strip()
+ if not raw:
+ return {"text": "", "_tokens": 0}
+ # EFFECTIVE (cache-aware) cost when the runner recovered usage; else approximate on the
+ # SAME model-priced basis as llm_stuck_judge.
+ _tokens = _runner_effective_or_approx(runner, prompt, raw, _real)
+ return {"text": raw[:1200], "_tokens": _tokens}
diff --git a/tokeneconomizer/vendor/spiral.py b/tokeneconomizer/vendor/spiral.py
new file mode 100644
index 0000000..32a914e
--- /dev/null
+++ b/tokeneconomizer/vendor/spiral.py
@@ -0,0 +1,833 @@
+#!/usr/bin/env python3
+"""
+v4.1 mechanism B + C: runtime spiral detector and escalation-ladder decision.
+
+The cost of a SWE-bench stream is brutally heavy-tailed: the top ~10% of tasks carry
+~50% of cache-aware effective cost, and that tail is dominated by long, UNRESOLVED
+spirals (the agent thrashing in an edit->test->fail loop while its re-fed context
+grows ~quadratically). This module:
+
+ detect_incremental(events) STAGE B (free) — fold the trajectory so far into a
+ compact SpiralState (edit churn, distinct edited files,
+ error density, repeated near-identical edit->test->fail
+ cycles, turn count).
+ decide(state, envelope) STAGE C — given the per-repo RESOLVED-turn envelope,
+ return the cheapest sufficient ladder action:
+ none -> nudge -> compact -> restart -> abort.
+
+Design constraints (learned this milestone):
+ * High turn-count is REPO-DEPENDENT: matplotlib's long tasks mostly resolve, only
+ xarray's are doomed. So the heavy interventions (restart/abort) are gated on the
+ per-repo `envelope` = a margin beyond the longest RESOLVED task seen so far; on a
+ repo where long==hard-but-solvable the envelope is high and they rarely fire.
+ * The lightest rung (nudge) is leakage-safe BY CONSTRUCTION: it only mirrors the
+ agent's OWN repeated actions back at it ("you tried N similar things, same
+ result"); it carries no external knowledge and never says "stop", only "try
+ something different" — so it dodges the effort-capping ban.
+ * abort is the terminal cap (autonomous/CI only); interactive callers should map it
+ to a hand-off instead.
+
+Stdlib only. The event schema mirrors reflect.load_trajectory (Copilot CLI
+tool.execution_start / tool.execution_complete), so this runs offline on saved
+transcripts AND live on a streamed JSONL.
+"""
+import hashlib
+import json
+import re
+
+# An output that RAN code to verify behavior (the test/repro step of the edit->test
+# loop). Mirrors reflect.TEST_RUN_RE intent but local to keep this module standalone.
+_TEST_RUN_RE = re.compile(
+ r"(runtests(\.py)?|\bpytest\b|py\.test|manage\.py\s+test|python\s+-m\s+unittest|"
+ r"\btox\b|\bnox\b|\btrial\b|python\d?\s+-c\b)", re.IGNORECASE)
+_ERR_RE = re.compile(
+ r"(Traceback \(most recent call last\)|\b\w*Error\b|\bFAILED\b|\bassert\b|"
+ r"Exception|errors during collection)", re.IGNORECASE)
+_EDIT_TOOLS = ("edit", "str_replace", "create")
+
+# Normalize agent-specific tool names to the copilot-style canonical names the folding
+# logic below keys on, so ONE code path handles both the Copilot stream schema and Claude
+# Code's stream-json (which names tools Bash/Read/Edit/Write/Grep/Glob/MultiEdit). Unknown
+# names pass through unchanged.
+_TOOL_CANON = {
+ "bash": "bash", "shell": "bash",
+ "read": "view", "view": "view",
+ "grep": "grep",
+ "glob": "glob",
+ "edit": "edit", "multiedit": "edit", "str_replace": "edit",
+ "apply_patch": "edit",
+ "write": "create", "create": "create",
+}
+
+
+def _canon_tool(name):
+ return _TOOL_CANON.get((name or "").lower(), name or "?")
+
+
+def completed_calls(line, pend):
+ """Schema-agnostic fold of ONE streamed JSONL line into completed (name, args, result)
+ tool calls, so the live guards work for BOTH the Copilot stream
+ (tool.execution_start/complete) AND Claude Code's stream-json (assistant.tool_use /
+ user.tool_result). `pend` is a caller-owned dict {callId -> {name,args}} carried across
+ lines to pair a start/tool_use with its later complete/tool_result. Yields a list of
+ completed calls finished on THIS line (0, 1, or — for a Claude user msg batching
+ several tool_results — more). Tool names are canonicalized so downstream keying is
+ uniform; unpaired results fall back to name '?'."""
+ try:
+ ev = json.loads(line)
+ except Exception:
+ return []
+ t = ev.get("type")
+ out = []
+ # ---- Codex exec --json schema ----
+ if t == "item.completed":
+ item = ev.get("item") or {}
+ kind = item.get("type")
+ if kind == "command_execution":
+ out.append(("bash", {"command": item.get("command") or ""},
+ item.get("aggregated_output") or ""))
+ elif kind == "file_change":
+ changes = item.get("changes") or []
+ paths = [c.get("path") for c in changes
+ if isinstance(c, dict) and c.get("path")]
+ out.append(("edit", {"path": paths[0] if paths else "",
+ "paths": paths, "changes": changes}, ""))
+ return out
+ # ---- Copilot schema ----
+ if t == "tool.execution_start":
+ d = ev.get("data", {}) if isinstance(ev.get("data"), dict) else {}
+ pend[d.get("toolCallId")] = {"name": d.get("toolName"),
+ "args": d.get("arguments", {}) or {}}
+ return out
+ if t == "tool.execution_complete":
+ d = ev.get("data", {}) if isinstance(ev.get("data"), dict) else {}
+ info = pend.pop(d.get("toolCallId"), {"name": "?", "args": {}})
+ res = d.get("result") or {}
+ c = res.get("content") if isinstance(res, dict) else ""
+ out.append((_canon_tool(info["name"]), info.get("args") or {},
+ c if isinstance(c, str) else ""))
+ return out
+ # ---- Claude Code stream-json schema ----
+ if t == "assistant":
+ for b in (ev.get("message", {}) or {}).get("content", []) or []:
+ if isinstance(b, dict) and b.get("type") == "tool_use":
+ pend[b.get("id")] = {"name": b.get("name"),
+ "args": b.get("input", {}) or {}}
+ return out
+ if t == "user":
+ for b in (ev.get("message", {}) or {}).get("content", []) or []:
+ if isinstance(b, dict) and b.get("type") == "tool_result":
+ info = pend.pop(b.get("tool_use_id"), {"name": "?", "args": {}})
+ c = b.get("content")
+ if isinstance(c, list): # content can be a list of text blocks
+ c = " ".join(x.get("text", "") for x in c if isinstance(x, dict))
+ out.append((_canon_tool(info["name"]), info.get("args") or {},
+ c if isinstance(c, str) else ""))
+ return out
+ return out
+
+
+def _basename(p):
+ return p.rstrip("/").split("/")[-1] if p else (p or "")
+
+
+def _failure_sig(text):
+ """A normalized signature of a failing output: drop numbers/addresses/paths so two
+ runs that fail the SAME way hash identically (used to detect 'no progress' — the
+ same failure recurring after edits). Leakage-safe: a hash, never the content."""
+ if not text:
+ return None
+ first = None
+ for ln in text.splitlines():
+ if _ERR_RE.search(ln):
+ first = ln
+ break
+ if first is None:
+ return None
+ norm = re.sub(r"0x[0-9a-f]+|\d+|/\S+", "", first)
+ return hashlib.md5(norm.strip().encode("utf-8", "replace")).hexdigest()[:10]
+
+
+class SpiralState:
+ """Compact, incremental fold of one trajectory's spiral signals. Cheap to update
+ per event so it can run live."""
+ __slots__ = ("turns", "edit_counts", "edit_order", "error_outputs",
+ "test_runs", "fail_sigs", "repeated_fail_sigs", "recent")
+
+ def __init__(self):
+ self.turns = 0
+ self.edit_counts = {} # basename -> #edits
+ self.edit_order = [] # ordered basenames of edit targets
+ self.error_outputs = 0 # tool outputs carrying an error/failure marker
+ self.test_runs = 0 # code-execution verification steps
+ self.fail_sigs = {} # failure signature -> count
+ self.repeated_fail_sigs = 0 # # of failing outputs that REPEAT an earlier sig
+ self.recent = [] # leakage-safe rolling log of recent actions (for the
+ # LLM trace-judge); process-only, never fix content
+
+ # ---- derived signals ----
+ @property
+ def edit_churn(self):
+ return max(self.edit_counts.values()) if self.edit_counts else 0
+
+ @property
+ def distinct_edit_files(self):
+ return len(self.edit_counts)
+
+ @property
+ def stagnation(self):
+ """Fraction of failing outputs that repeat an earlier identical failure (the
+ agent edited but the SAME failure came back = no progress). 0 when no fails."""
+ total = sum(self.fail_sigs.values())
+ return (self.repeated_fail_sigs / total) if total else 0.0
+
+ def update(self, tool_name, args, result_text):
+ """Fold one completed tool call into the state."""
+ self.turns += 1
+ args = args if isinstance(args, dict) else {}
+ tool_name = _canon_tool(tool_name) # accept Copilot- or Claude-named tools
+ action = tool_name or "?"
+ if tool_name in _EDIT_TOOLS:
+ b = _basename(args.get("path") or args.get("filePath")
+ or args.get("file_path") or "")
+ if b:
+ self.edit_order.append(b)
+ self.edit_counts[b] = self.edit_counts.get(b, 0) + 1
+ action = f"edit {b}"
+ cmd = args.get("command", "") if tool_name == "bash" else ""
+ if cmd:
+ short = re.sub(r"\s+", " ", cmd.strip())[:90]
+ action = f"$ {short}"
+ if _TEST_RUN_RE.search(cmd):
+ self.test_runs += 1
+ elif tool_name in ("grep", "glob"):
+ action = f"{tool_name} {str(args.get('pattern') or '')[:40]}"
+ elif tool_name == "view":
+ action = f"view {_basename(args.get('path') or args.get('file_path') or '')}"
+ fail = None
+ if result_text and _ERR_RE.search(result_text):
+ self.error_outputs += 1
+ sig = _failure_sig(result_text)
+ if sig:
+ fail = sig
+ if sig in self.fail_sigs:
+ self.repeated_fail_sigs += 1
+ self.fail_sigs[sig] = self.fail_sigs.get(sig, 0) + 1
+ # rolling leakage-safe action log (tool name / command / search / error-SIGNATURE
+ # only — no file contents, edit bodies, or view output), capped to recent window.
+ tag = f" [err#{fail}]" if fail else ""
+ self.recent.append(f"{self.turns:>3}: {action}{tag}")
+ if len(self.recent) > 40:
+ self.recent = self.recent[-40:]
+
+ def recent_digest(self, n=24, envelope=None):
+ """A compact, LEAKAGE-SAFE digest of the agent's RECENT PROCESS (last n actions)
+ for the LLM trace-judge: commands/searches/edited-basenames and error-SIGNATURE
+ markers only — never fix content, edit bodies, view output, or the issue text.
+ Mirrors reflect.trajectory_summary's leakage stance, windowed to 'now'.
+ `envelope` (this repo's typical resolved-task length) is passed as CONTEXT so the
+ judge can recognize an abnormally-long trajectory as a likely dead-end."""
+ head = (f"turns={self.turns} edit_churn={self.edit_churn} "
+ f"distinct_edit_files={self.distinct_edit_files} "
+ f"error_outputs={self.error_outputs} test_runs={self.test_runs} "
+ f"repeated_identical_failures={self.repeated_fail_sigs} "
+ f"stagnation={self.stagnation:.2f}")
+ if envelope:
+ ratio = self.turns / envelope if envelope else 0
+ head += (f"\nrepo_context: tasks here usually RESOLVE within ~{envelope:.0f} "
+ f"turns; this trajectory is at {self.turns} turns = {ratio:.1f}x that "
+ f"typical length. Tests are still NOT green. A trajectory far past "
+ f"the typical resolved length with recurring failures is likely a "
+ f"dead-end loop, even if it touches many files.")
+ return head + "\nRecent actions (oldest->newest):\n" + "\n".join(self.recent[-n:])
+
+
+ def snapshot(self):
+ return {"turns": self.turns, "edit_churn": self.edit_churn,
+ "distinct_edit_files": self.distinct_edit_files,
+ "error_outputs": self.error_outputs, "test_runs": self.test_runs,
+ "stagnation": round(self.stagnation, 3),
+ "repeated_fail_sigs": self.repeated_fail_sigs}
+
+
+# ---- detector thresholds -----------------------------------------------------
+# The "stuck" signature (calibrated on the measured separation: spirals showed first-
+# 40-turn edit_churn ~6 vs ~1.5 normal, distinct_edit_files ~2.5 vs ~1, error_outputs
+# ~9 vs ~3). We require a CONJUNCTION (churn AND errors/scatter AND some repetition)
+# so productive-but-hard tasks (high activity, but CHANGING failures) are spared.
+DEFAULT_THRESHOLDS = {
+ "min_turns": 20, # don't judge stuck before the solve loop has run a while
+ "edit_churn": 4, # >=4 edits to one file
+ "distinct_edit_files": 3, # OR edits scattered across >=3 files
+ "error_outputs": 6, # sustained error density
+ "stagnation": 0.20, # >=20% of failures are repeats of an earlier one
+ "envelope_margin": 1.25, # restart/abort only beyond margin x the resolved-turn pctl
+ "envelope_pct": 90, # use the p90 of resolved turns (NOT max) — robust to a
+ # single very-long resolve poisoning the gate (see
+ # EnvelopeTracker). max-based envelope was disabled by one
+ # 115-turn xarray resolve; p90 ignores the lone outlier.
+ "envelope_min_samples": 5, # need >=5 resolved tasks before trusting the percentile
+ "restart_budget": 2, # max restarts before the terminal abort cap
+ "judge_confidence": 0.7, # LLM trace-judge must be >=this confident to drive an abort
+ "backstop_mult": 2.5, # HARD backstop: beyond this multiple of the envelope, a
+ # stuck trajectory aborts REGARDLESS of the judge's verdict.
+ # Empirically the judge spared xarray-7019/7179 at 5.8x the
+ # envelope (177-204 turns vs envelope 35, stagnation 0.3-0.4)
+ # by misreading multi-file thrashing as "exploration"; the
+ # judge must not be able to veto a screaming outlier.
+ # ---- cheap GATE to the judge (Tier-0): permissive "worth a paid look?" trigger.
+ # Sensitive on purpose (catch candidates); the judge supplies precision. Far cheaper
+ # to evaluate than the judge, and far more permissive than is_stuck (which is the
+ # strict heuristic DECISION used only in the no-judge fallback).
+ "gate_min_turns": 18, # don't wake the judge before the loop has had time to form
+ "gate_error_outputs": 8, # OR: a lot of errors even without an exact repeat
+}
+
+
+# ---- v6.1 aggressiveness knob ------------------------------------------------
+# A single user-facing LEVEL scales the base thresholds the guard starts from (the per-repo
+# EnvelopeTracker then learns ON TOP of this base, flooring learned signal-gates against it).
+# Calibrated on the guard-active arms (v41fix/v43-xarray, n=239): the ABORT/backstop path is
+# already near-right (only 1 task slipped past 2.5x envelope), but (a) the rung-1 NUDGE was
+# effectively DORMANT (1/239) because it shares the abort's bias-to-spare judge gate, and
+# (b) a middle band of 28 unresolved tasks ran past typical-resolved length yet under the
+# backstop with no intervention. So 'aggressive' mainly WAKES THE NUDGE (lower judge
+# confidence, earlier gate, and nudge_on_ambiguous) and tightens that middle band, while
+# only MODESTLY pulling the backstop in (2.5x -> 1.8x). 'default' == today's behavior
+# EXACTLY (empty override). 'conservative' spares harder (for churny-but-productive repos).
+# nudge_on_ambiguous: when True, a judge-AMBIGUOUS trajectory PAST the envelope fires the
+# resolve-safe NUDGE (never an abort) instead of being spared outright — safe because a
+# nudge only redirects. Only 'aggressive' sets it.
+SPIRAL_LEVELS = {
+ "conservative": {
+ "judge_confidence": 0.85, "envelope_margin": 1.5, "backstop_mult": 3.5,
+ "stagnation": 0.30, "gate_min_turns": 24, "min_turns": 28,
+ "edit_churn": 5, "error_outputs": 8, "envelope_min_samples": 7,
+ },
+ "default": {}, # verbatim DEFAULT_THRESHOLDS — reproduces pre-knob behavior exactly
+ "aggressive": {
+ "judge_confidence": 0.55, "envelope_margin": 1.0, "backstop_mult": 1.8,
+ "stagnation": 0.12, "gate_min_turns": 12, "min_turns": 14,
+ "edit_churn": 3, "error_outputs": 4, "envelope_min_samples": 3,
+ "nudge_on_ambiguous": True,
+ },
+}
+
+
+def scaled_thresholds(level="default"):
+ """Return DEFAULT_THRESHOLDS overlaid with the named LEVEL's overrides (see
+ SPIRAL_LEVELS). Unknown level falls back to 'default' (verbatim defaults). This is the
+ BASE the EnvelopeTracker starts from; per-repo learning floors on top of it."""
+ thr = dict(DEFAULT_THRESHOLDS)
+ thr.update(SPIRAL_LEVELS.get(level, SPIRAL_LEVELS["default"]))
+ return thr
+
+
+def gate_for_judge(state, thr=None):
+ """TIER-0 (free, every turn): cheap rule-based trigger deciding whether the recent
+ trace is worth ELEVATING to the paid LLM judge. Permissive — it only needs a sign of
+ POSSIBLE trouble after the loop has had time to form: a repeated identical failure
+ (the cheapest 'no progress' signal) OR a high error density. Productive trajectories
+ (changing failures, few repeats) mostly keep this gate CLOSED, so the judge is rarely
+ consulted; the judge then makes the precise stuck-vs-productive call. This separation
+ is what keeps the judge from being a per-step tax."""
+ thr = thr or DEFAULT_THRESHOLDS
+ if state.turns < thr["gate_min_turns"]:
+ return False
+ return (state.repeated_fail_sigs >= 1
+ or state.error_outputs >= thr["gate_error_outputs"])
+
+
+def is_stuck(state, thr=None):
+ """Strict heuristic 'this looks like an unproductive loop' classifier — the DECISION
+ used only in the no-judge fallback. Must have run a while AND show churn/scatter AND
+ repeated identical failures (stagnation): the last term is what separates a doomed
+ spiral from a productive hard grind. (When the judge is wired, gate_for_judge is the
+ cheap trigger and the judge is the decision; is_stuck is not used.)"""
+ thr = thr or DEFAULT_THRESHOLDS
+ if state.turns < thr["min_turns"]:
+ return False
+ churny = (state.edit_churn >= thr["edit_churn"]
+ or state.distinct_edit_files >= thr["distinct_edit_files"])
+ noisy = state.error_outputs >= thr["error_outputs"]
+ repeating = state.stagnation >= thr["stagnation"]
+ return churny and noisy and repeating
+
+
+# ladder rungs, cheapest first
+NONE, NUDGE, COMPACT, RESTART, ABORT = "none", "nudge", "compact", "restart", "abort"
+
+
+def assess_abort(state, envelope=None, judge=None, thr=None):
+ """Decide whether to abort, TRACE-FIRST, as an explicit TWO-TIER cascade. Returns
+ (abort: bool, reason: str).
+
+ TIER 0 — cheap rule-based GATE (free, every turn): gate_for_judge(). Decides only
+ whether the recent trace is WORTH a paid look. Permissive; productive trajectories
+ mostly keep it closed, so the judge is rarely consulted (this is what makes the
+ judge not a per-step tax).
+ TIER 1 — paid LLM JUDGE (only when the gate is open, and the caller additionally
+ rate-limits it): reads a leakage-safe digest of the RECENT actions and decides
+ stuck-vs-productive. The judge — not the step count — drives the outcome:
+ - STUCK, confidence>=judge_confidence -> ABORT, even within the envelope (a
+ clearly-doomed loop shouldn't burn tokens up to the envelope first);
+ - PRODUCTIVE -> SPARE, even past the envelope (progress shows in the trace);
+ - ambiguous -> defer to the envelope fallback.
+ FALLBACK — when NO judge is wired (judge=None): the strict is_stuck() heuristic plus
+ the per-repo resolved-turn envelope (the Phase-1.5 behavior), so the guard still
+ works cheaply without an LLM.
+ """
+ thr = thr or DEFAULT_THRESHOLDS
+
+ # HARD BACKSTOP (checked BEFORE the judge): an extreme outlier — far past the
+ # envelope AND stuck by the strict heuristic — aborts regardless of any judge
+ # verdict. The judge cannot veto this; it exists because the LLM judge empirically
+ # rationalized 5.8x-envelope thrashing (xarray-7019/7179) as "exploration".
+ # `backstop_turns` (learned, cap-aware) is preferred when the tracker supplied it, so
+ # the backstop stays REACHABLE under a tight turn cap; else fall back to the
+ # multiplier-on-envelope form (the Copilot-scale behavior).
+ backstop_turns = thr.get("backstop_turns")
+ if backstop_turns is None and envelope is not None:
+ backstop_turns = envelope * thr["backstop_mult"]
+ if (backstop_turns is not None
+ and state.turns >= backstop_turns
+ and is_stuck(state, thr)):
+ return True, (f"backstop:{state.turns}>={backstop_turns:.0f}"
+ f"turns stagnation={state.stagnation:.2f}")
+
+ if judge is not None:
+ # TWO-TIER: gate -> judge. The gate opens on the cheap rule-based trigger OR when
+ # the trajectory is already PAST this repo's typical resolved length (envelope) —
+ # an over-envelope run is worth a paid judge look even without an exact repeated
+ # failure yet, which is what lets the guard engage on a short-turn agent whose
+ # spirals hit the cap before accumulating a repeat signature.
+ over_envelope = envelope is not None and state.turns >= envelope
+ if not (gate_for_judge(state, thr) or over_envelope):
+ return False, "gate-closed"
+ try:
+ v = judge(state.recent_digest(envelope=envelope)) # caller rate-limits
+ except Exception:
+ v = None
+ if not v:
+ return False, "judge-throttled"
+ conf = float(v.get("confidence", 0) or 0)
+ if v.get("stuck") and conf >= thr["judge_confidence"]:
+ return True, f"judge-stuck@{conf:.2f}:{str(v.get('reason',''))[:60]}"
+ if v.get("stuck") is False:
+ return False, "judge-productive"
+ # judge AMBIGUOUS (stuck=None): do NOT abort on the (possibly under-calibrated,
+ # early-stream) envelope alone — that killed resolvable tasks like xarray-7120 at
+ # turn 28 when the p90 envelope was only 27. Ambiguity => SPARE; the hard backstop
+ # (2.5x envelope AND is_stuck, checked above) remains the only envelope-driven kill,
+ # and it sits well beyond any resolvable task's length.
+ return False, "judge-ambiguous-spared"
+
+ # FALLBACK (no judge): strict heuristic + envelope safety gate.
+ if not is_stuck(state, thr):
+ return False, "not-stuck"
+ if envelope is not None and state.turns >= envelope:
+ return True, f"envelope:{state.turns}>={envelope:.0f}"
+ return False, "stuck-but-within-envelope"
+
+
+def should_abort(state, envelope, judge=None, thr=None):
+ """Backward-compatible boolean wrapper over assess_abort()."""
+ ok, _reason = assess_abort(state, envelope=envelope, judge=judge, thr=thr)
+ return ok
+
+
+def decide(state, envelope=None, restarts_done=0, nudged=False,
+ autonomous=False, thr=None):
+ """STAGE C — choose the cheapest sufficient ladder action.
+
+ `envelope` = margin * (longest RESOLVED task's turn count seen so far) for this
+ repo, or None to disable the heavy rungs (then we only ever nudge). `restarts_done`
+ and `nudged` track what's already been tried this task so the ladder escalates.
+
+ Returns one of NONE / NUDGE / COMPACT / RESTART / ABORT.
+ """
+ thr = thr or DEFAULT_THRESHOLDS
+ if not is_stuck(state, thr):
+ return NONE
+ # rung 1: always try the free, resolve-safe nudge first.
+ if not nudged:
+ return NUDGE
+ # beyond the nudge we need the per-repo envelope to authorize heavier action — this
+ # is what keeps us off matplotlib's productive long tasks.
+ over_envelope = envelope is not None and state.turns >= envelope
+ if not over_envelope:
+ # still within the band where long==plausibly-productive: only compact (safe).
+ return COMPACT
+ if restarts_done < thr["restart_budget"]:
+ return RESTART
+ return ABORT if autonomous else COMPACT
+
+
+def nudge_text(state):
+ """The rung-1 message: mirror the agent's OWN repeated actions back at it. Contains
+ nothing but a summary of what it already did — leakage-safe by construction, and
+ framed to REDIRECT effort, never to stop it."""
+ busiest = max(state.edit_counts.items(), key=lambda kv: kv[1])[0] \
+ if state.edit_counts else "one file"
+ return (
+ "Process check (automated): so far you've made "
+ f"{sum(state.edit_counts.values())} edits across "
+ f"{state.distinct_edit_files} file(s) (most to `{busiest}`) and run "
+ f"{state.test_runs} verification(s), and the same failure has recurred "
+ f"{state.repeated_fail_sigs} time(s) without the tests going green. This "
+ "looks like a repetitive loop. Do NOT stop — instead step back and try a "
+ "MATERIALLY DIFFERENT hypothesis: re-read the failing test's expectation, "
+ "consider a different file or root cause than the ones you've been editing, "
+ "and verify the new hypothesis directly before editing again.")
+
+
+def restart_digest(state):
+ """Leakage-safe brief carried across a context RESTART: what was already tried
+ (counts + the files touched, which are the agent's own actions) and that tests are
+ still red. Never includes a fix, a gold pointer, or which lines to change."""
+ files = ", ".join(f"{b} ({n} edits)" for b, n in
+ sorted(state.edit_counts.items(), key=lambda kv: -kv[1]))
+ return (
+ "Resuming this task with a fresh, compact context after a repetitive loop. "
+ "What was already attempted (do NOT simply repeat it): edited " + (files or "—")
+ + f"; ran {state.test_runs} verifications; the tests are still failing and the "
+ f"same failure recurred {state.repeated_fail_sigs} time(s). Form a different "
+ "hypothesis about the root cause, confirm it by reading the relevant code and "
+ "the failing test's expectation, then make a focused change and verify.")
+
+
+# ---- rung-1 NUDGE live controller --------------------------------------------
+RESUME_HEADER = (
+ "\n\n---\nNOTE: you are RESUMING this same task. The repository at /testbed already "
+ "contains your OWN in-progress edits from your previous attempt (inspect them with "
+ "`git diff`); do not discard that work, build on it. The tests are still not passing.\n")
+
+
+class NudgeGuard:
+ """Live rung-1 controller: run the sealed agent as a sequence of streamed SEGMENTS in
+ ONE container, delivering the leakage-safe NUDGE between segments.
+
+ Why segments: the rollout is single-shot `copilot -p`, so a nudge cannot be injected
+ into the running context. Instead the guard STOPS the current stream the moment the
+ trajectory looks stuck (freezing the container's working tree with the agent's edits
+ intact) and the harness RE-INVOKES the agent with a continuation prompt = original task
+ + RESUME_HEADER + nudge_text(). The agent resumes on top of its own on-disk edits.
+
+ Ladder realized here (compact/restart remain P2): the FIRST stuck detection NUDGES; a
+ subsequent stuck detection (or the cumulative backstop) ABORTS. Each new segment starts
+ with a FRESH per-segment SpiralState, so the post-nudge behavior must independently
+ re-accumulate churn/errors/stagnation past the gate before it can be judged stuck again
+ — that reset IS the nudge's fair grace window (assess_abort's gate needs >=18 turns).
+ A cumulative backstop (turns summed across segments) bounds total cost regardless.
+
+ Usage (harness side):
+ guard = NudgeGuard(envelope=env, judge=spiral_judge, verbose=True)
+ rec, wall = run_agent(..., monitor=guard.monitor,
+ continuation=lambda: guard.continuation(prompt))
+ # then read guard.nudges_done / guard.aborted / guard.events for telemetry.
+ """
+
+ def __init__(self, envelope=None, judge=None, thr=None, nudge_budget=1,
+ verbose=False, nudge_writer=None, resume_header=None):
+ self.envelope = envelope
+ self.judge = judge # raw digest->verdict callable (rate-limited HERE)
+ self.thr = thr or DEFAULT_THRESHOLDS
+ self.nudge_budget = nudge_budget
+ self.verbose = verbose
+ # v6.1: optional LLM-targeted nudge writer (digest -> {"text","_tokens"}). When
+ # set, continuation() uses it to build a TARGETED redirect from the same leakage-
+ # safe digest the judge sees, falling back to the generic nudge_text template on
+ # any empty/failed result. None => the fixed count-template (v4.3 behavior).
+ self.nudge_writer = nudge_writer
+ self.resume_header = resume_header if resume_header is not None else RESUME_HEADER
+ self.nudge_writer_tokens = 0 # REAL token cost of paid nudge-writer calls
+ # per-segment stuck-detection state (reset by continuation() after each nudge)
+ self.st = SpiralState()
+ self.pend = {} # toolCallId -> {name,args} within the segment
+ # cross-segment accounting
+ self.cum_turns = 0 # turns summed over ALL segments (cumulative backstop)
+ self.segment = 0
+ self.nudges_done = 0
+ self.aborted = False # terminal abort fired
+ self.pending_action = None # 'nudge' | 'abort' set when a segment should stop
+ self.events = [] # telemetry: nudge/abort decisions
+ # sampled-judge cadence (mirror run_agent's _sampled_judge: backoff on productive)
+ self._judge_last = -999
+ self._judge_gap = 5
+ self._judge_calls = 0
+ self.judge_tokens = 0 # REAL token cost of all paid judge calls (mechanism
+ # overhead folded into amortized task cost by callers)
+ self._JUDGE_BASE_GAP = 5
+ self._JUDGE_MAX_GAP = 20
+
+ # ---- the paid judge, rate-limited with backoff (keyed on per-segment turns) -------
+ def _sampled_judge(self, digest):
+ if self.judge is None:
+ return None
+ if self.st.turns - self._judge_last < self._judge_gap:
+ return None
+ self._judge_last = self.st.turns
+ self._judge_calls += 1
+ try:
+ v = self.judge(digest)
+ except Exception:
+ v = None
+ if v:
+ self.judge_tokens += v.get("_tokens") or 0
+ if v and v.get("stuck") is False:
+ self._judge_gap = min(self._JUDGE_MAX_GAP, self._judge_gap + self._JUDGE_BASE_GAP)
+ else:
+ self._judge_gap = self._JUDGE_BASE_GAP
+ if self.verbose and v:
+ print(f" [nudge-judge] seg{self.segment} turn {self.st.turns} "
+ f"(call #{self._judge_calls}, next>={self._judge_gap}): "
+ f"stuck={v.get('stuck')} conf={v.get('confidence')} "
+ f"{str(v.get('reason', ''))[:50]}")
+ return v
+
+ # ---- STREAM monitor: fold one JSONL event, return True to STOP this segment -------
+ def monitor(self, line):
+ # completed_calls handles BOTH the Copilot schema (tool.execution_start/complete)
+ # and Claude Code's stream-json (assistant.tool_use / user.tool_result). A single
+ # Claude user message can batch several tool_results, so fold+assess EACH in order
+ # and stop on the first that trips the gate.
+ for name, args, result in completed_calls(line, self.pend):
+ self.st.update(name, args, result)
+ self.cum_turns += 1
+
+ # cumulative hard backstop across segments (per-segment st resets, so
+ # assess_abort's own backstop only sees within-segment turns; this catches a
+ # task that keeps spiraling across a nudge without any single segment exceeding
+ # the envelope). Prefer the learned cap-aware `backstop_turns` (so it stays
+ # reachable under a tight turn cap); else fall back to backstop_mult*envelope.
+ backstop_turns = self.thr.get("backstop_turns")
+ if backstop_turns is None and self.envelope is not None:
+ backstop_turns = self.envelope * self.thr["backstop_mult"]
+ if (backstop_turns is not None
+ and self.cum_turns >= backstop_turns
+ and is_stuck(self.st, self.thr)):
+ return self._stop("abort",
+ f"cum-backstop:{self.cum_turns}>={backstop_turns:.0f}t")
+
+ abort, reason = assess_abort(self.st, envelope=self.envelope,
+ judge=(self._sampled_judge if self.judge else None),
+ thr=self.thr)
+ if not abort:
+ # v6.1 aggressive knob: a judge-AMBIGUOUS trajectory PAST the envelope is
+ # spared from ABORT by assess_abort (resolve-safety), but under this level we
+ # still fire the RESOLVE-SAFE nudge on it (redirect only, never a kill) — this
+ # is what wakes the otherwise-dormant nudge on the middle band of long-running
+ # unresolved tasks. Never escalates to abort here.
+ if (self.thr.get("nudge_on_ambiguous")
+ and reason == "judge-ambiguous-spared"
+ and self.nudges_done < self.nudge_budget):
+ return self._stop("nudge", "ambiguous-past-envelope")
+ continue
+ # stuck: nudge first (while budget remains), else abort.
+ if self.nudges_done < self.nudge_budget:
+ return self._stop("nudge", reason)
+ return self._stop("abort", reason)
+ return False
+
+ def _stop(self, action, reason):
+ self.pending_action = action
+ rec = {"segment": self.segment, "action": action,
+ "turn_in_segment": self.st.turns, "cum_turns": self.cum_turns,
+ "envelope": self.envelope, "reason": reason,
+ "snapshot": self.st.snapshot()}
+ self.events.append(rec)
+ if action == "abort":
+ self.aborted = True
+ if self.verbose:
+ print(f" [nudge-guard] seg{self.segment} {action.upper()} at "
+ f"turn {self.st.turns} (cum {self.cum_turns}) ({reason}) "
+ f"{self.st.snapshot()}")
+ return True
+
+ # ---- nudge body: v6.1 LLM-targeted redirect, else the generic template ------------
+ def _nudge_body(self):
+ """Build the rung-1 nudge text. If an LLM nudge_writer is wired, feed it the SAME
+ leakage-safe recent digest the judge sees and use its targeted redirect; fold the
+ paid cost into nudge_writer_tokens. Fall back to the generic nudge_text template on
+ an absent writer or any empty/failed result (resolve-safe zero-LLM path)."""
+ if self.nudge_writer is not None:
+ try:
+ out = self.nudge_writer(self.st.recent_digest(envelope=self.envelope))
+ except Exception:
+ out = None
+ if out:
+ self.nudge_writer_tokens += out.get("_tokens") or 0
+ txt = (out.get("text") or "").strip()
+ if txt:
+ if self.verbose:
+ print(f" [nudge-writer] seg{self.segment} targeted redirect "
+ f"({len(txt)} chars, +{out.get('_tokens')} tok)")
+ return txt
+ return nudge_text(self.st)
+
+ # ---- CONTINUATION: called after a segment aborts; return next prompt or None ------
+ def continuation(self, base_prompt):
+ """Return the continuation prompt for the next segment, or None to stop for good.
+ Only a 'nudge' stop continues; an 'abort' stop (or a natural end) is terminal."""
+ if self.pending_action != "nudge":
+ return None
+ text = self._nudge_body()
+ self.nudges_done += 1
+ # fresh per-segment state -> the agent gets a fair, independent re-assessment.
+ self.st = SpiralState()
+ self.pend = {}
+ self.pending_action = None
+ self.segment += 1
+ self._judge_last = -999
+ self._judge_gap = self._JUDGE_BASE_GAP
+ return base_prompt + self.resume_header + text
+
+
+# ---- envelope tracking -------------------------------------------------------
+def _percentile(xs, p):
+ """Linear-interpolated p-th percentile of xs (0<=p<=100). xs need not be sorted."""
+ if not xs:
+ return None
+ xs = sorted(xs)
+ n = len(xs)
+ if n == 1:
+ return float(xs[0])
+ i = (p / 100.0) * (n - 1)
+ lo = int(i)
+ frac = i - lo
+ if lo + 1 >= n:
+ return float(xs[-1])
+ return xs[lo] + frac * (xs[lo + 1] - xs[lo])
+
+
+class EnvelopeTracker:
+ """Per-repo resolved-turn envelope AND runtime-learned detection thresholds.
+
+ Envelope: margin * the p<pct> of turn-counts among RESOLVED tasks seen so far. Using a
+ PERCENTILE (default p90), not the max, makes the gate robust to a single very-long
+ resolve — empirically one 115-turn xarray resolve set a max-based envelope to ~144 and
+ disabled the guard for every 40-60-turn spiral below it; p90 ignores that lone outlier
+ (envelope ~32) while staying HIGH on repos like matplotlib where long resolves are
+ COMMON (so productive-long tasks stay protected). Stays None until >=min_samples
+ resolved tasks exist (don't trust a percentile of a handful of points), so we never
+ abort on a repo we know little about yet.
+
+ Thresholds: `thresholds()` derives the FULL detector threshold dict from the SAME
+ resolved-task history, so the guard adapts to the agent/repo's actual turn scale
+ instead of the hardcoded Copilot-scale DEFAULT_THRESHOLDS (which never fired on Claude
+ Code's <=80-turn runs — its backstop landed above the turn cap). Turn-scale gates
+ scale DOWN to the learned distribution; signal gates keep DEFAULT as a FLOOR so the
+ learned dict is never MORE trigger-happy than the hand-tuned one (resolve-safety). Fed
+ only by RESOLVED tasks (same causal rule as the envelope), so a task's own outcome can
+ only gate FUTURE tasks, never itself."""
+ def __init__(self, margin=None, pct=None, min_samples=None, level="default",
+ frozen_envelope=None):
+ # v6.1: `level` selects the aggressiveness preset (SPIRAL_LEVELS). It sets the BASE
+ # thresholds this tracker starts from AND the envelope margin/percentile/min-samples;
+ # explicit margin/pct/min_samples args still win over the preset when given.
+ self.level = level
+ self.base_thr = scaled_thresholds(level)
+ self.margin = margin or self.base_thr["envelope_margin"]
+ self.pct = pct if pct is not None else self.base_thr["envelope_pct"]
+ self.min_samples = (min_samples if min_samples is not None
+ else self.base_thr["envelope_min_samples"])
+ # FROZEN envelope (pre-registered hyperparameter): when set, `envelope` returns this
+ # value from task 1, bypassing the >=min_samples warm-up. For SHORT streams (Web-Bench
+ # = 20 tasks, ~5 resolved at ~25% Pass@1) the percentile never reaches min_samples, so
+ # a within-stream-learned envelope would stay None the whole run and the guard would
+ # never engage. A frozen scalar (calibrated once from SWE-bench/held-out, IDENTICAL
+ # across all streams -- a fixed hyperparameter, NOT a prior pooled across the streams)
+ # lets the guard engage while preserving stream independence. None => today's behavior.
+ self.frozen_envelope = frozen_envelope
+ self.resolved_turns = [] # all RESOLVED tasks' turn counts (for the percentile)
+ self.resolved_snaps = [] # per-resolved-task SpiralState.snapshot() dicts (signals)
+
+ def update(self, turns, resolved, snapshot=None):
+ """Record a completed task. Only RESOLVED tasks feed the baseline. `snapshot` is an
+ optional SpiralState.snapshot() dict (edit_churn/distinct_edit_files/error_outputs/
+ stagnation/...) captured from that task's trajectory; when omitted only the
+ turn-envelope learns (back-compat with callers that don't reconstruct the state)."""
+ if resolved and turns:
+ self.resolved_turns.append(turns)
+ if snapshot:
+ self.resolved_snaps.append(snapshot)
+
+ @property
+ def envelope(self):
+ if len(self.resolved_turns) < self.min_samples:
+ # cold start: fall back to the frozen pre-registered envelope if one was set
+ # (short-stream case); else None == "don't trust a handful of points yet".
+ return self.frozen_envelope
+ return _percentile(self.resolved_turns, self.pct) * self.margin
+
+ def thresholds(self, max_turns=None):
+ """Return the detector threshold dict to hand to is_stuck/gate_for_judge/
+ assess_abort. Starts from the LEVEL-scaled base (scaled_thresholds(self.level); ==
+ DEFAULT_THRESHOLDS for level 'default') and, once >=min_samples RESOLVED tasks
+ exist, overrides the learned keys from this stream's resolved-task distribution.
+ Before warm-up (or with no data) it returns the base verbatim, so behavior is
+ unchanged until the stream has learned enough — exactly like the envelope gating
+ None today. Learned signal-gates floor against the SCALED base, so an 'aggressive'
+ level genuinely lowers those floors (and 'conservative' raises them). Pass
+ `max_turns` (the run's turn cap) to make the hard backstop REACHABLE under the cap."""
+ base = self.base_thr
+ thr = dict(base)
+ env = self.envelope
+ # cap-aware backstop: reachable even under a tight turn cap. The Copilot-scale
+ # `backstop_mult * envelope` landed ABOVE Claude's 80-turn cap (~110 turns) so it
+ # never fired; clamp it under 0.9*cap so an extreme outlier still aborts in-cap.
+ if env is not None:
+ bt = env * thr["backstop_mult"]
+ if max_turns:
+ bt = min(bt, 0.9 * max_turns)
+ thr["backstop_turns"] = bt
+ if len(self.resolved_turns) < self.min_samples:
+ return thr # cold start: level-scaled base (+ backstop_turns if env exists)
+
+ # --- turn-scale gates: relative to the learned resolved-turn distribution. On a
+ # short-turn agent (Claude p90~35) this drops the judge gate from 18 -> ~10 so the
+ # guard can engage before the cap; on a long-turn agent (Copilot p90~150+) the
+ # clamp keeps it at the base ceiling (no change). ---
+ p90t = _percentile(self.resolved_turns, 90) or 0
+
+ def _clamp(v, lo, hi):
+ return int(max(lo, min(hi, round(v))))
+ thr["gate_min_turns"] = _clamp(0.6 * p90t, 6, base["gate_min_turns"])
+ thr["min_turns"] = _clamp(0.6 * p90t, 8, base["min_turns"])
+
+ # --- signal gates: treat RESOLVED runs as the "normal" band and sit just ABOVE its
+ # p90, but never BELOW the (level-scaled) base floor (so the learned dict is never
+ # more trigger-happy than the level asks — protects resolve-safety on churny-but-
+ # productive repos, where this RAISES the bar). Needs the per-task snapshots. ---
+ if self.resolved_snaps:
+ def _sig(key):
+ return [s.get(key, 0) or 0 for s in self.resolved_snaps]
+ import math
+ thr["edit_churn"] = max(base["edit_churn"],
+ math.ceil((_percentile(_sig("edit_churn"), 90) or 0) + 1))
+ thr["distinct_edit_files"] = max(
+ base["distinct_edit_files"],
+ math.ceil((_percentile(_sig("distinct_edit_files"), 90) or 0) + 1))
+ thr["error_outputs"] = max(
+ base["error_outputs"],
+ math.ceil((_percentile(_sig("error_outputs"), 90) or 0) + 1))
+ thr["gate_error_outputs"] = max(base["gate_error_outputs"],
+ thr["error_outputs"])
+ thr["stagnation"] = max(base["stagnation"],
+ round(_percentile(_sig("stagnation"), 90) or 0, 3))
+ return thr
+
+
+# ---- offline / shadow-mode helper --------------------------------------------
+def state_from_trajectory(path, load_trajectory=None):
+ """Build a final SpiralState by replaying a saved transcript.jsonl. Reuses
+ reflect.load_trajectory by default (import lazily to avoid a hard dependency when
+ spiral.py is used standalone). Returns a SpiralState (use .snapshot() / decide()).
+ """
+ if load_trajectory is None:
+ from . import reflect
+ load_trajectory = reflect.load_trajectory
+ st = SpiralState()
+ for name, args, res, _ok in load_trajectory(path):
+ st.update(name, args, res)
+ return st
diff --git a/tokeneconomizer/workloads.py b/tokeneconomizer/workloads.py
new file mode 100644
index 0000000..ce30f52
--- /dev/null
+++ b/tokeneconomizer/workloads.py
@@ -0,0 +1,470 @@
+"""Canonical evaluated workloads — exactly which tasks the published results ran.
+
+A results tree records which tasks ran only implicitly, in its run directories, so a fresh
+user could not reproduce the workload without one. This module writes the task identities
+down: it is the authority on what to provision (dataset rows and Docker images) and what to
+run.
+
+Stdlib-only and data-only, consistent with the rest of the core package: importing it
+pulls in no dependencies and touches no filesystem, so it works on a bare clone with no
+results tree, no datasets, and no Docker.
+
+ from tokeneconomizer import workloads
+
+ workloads.SWEBENCH_INSTANCES['pydata/xarray'] # the 39 xarray instances
+ workloads.task_counts('swebench') # {'django/django': 40, ...}
+ workloads.swebench_image_tag('django__django-11532')
+
+**Provenance.** Extracted from the evaluated result trees (one row per task-run in
+``<key>/index/all_rows.csv``).
+"""
+
+__all__ = [
+ "SWEBENCH_INSTANCES", "SWEBENCH_SEEDS", "SWEBENCH_FULL_SPLIT_ONLY",
+ "SWEBENCH_EXCLUDED", "SWEBENCH_ADDED", "runnable_instances",
+ "SWEBENCH_COPILOT_EXTRA", "WEBBENCH_TASKS", "WEBBENCH_SEEDS",
+ "WEBBENCH_TASKS_PUBLISHED_2026_08",
+ "CREDDATA_INSTANCES", "CREDDATA_SEEDS", "REPOSVUL_INSTANCES", "REPOSVUL_SEEDS",
+ "CODEX_SWEBENCH_RUNS", "DOMAINS", "instances", "task_counts", "total_tasks",
+ "swebench_image_tag", "filter_tasks",
+]
+
+# ---------------------------------------------------------------------------
+# SWE-bench (Claude Code + Copilot CLI, trimech A/B; 6 repos x 3 seeds)
+# ---------------------------------------------------------------------------
+# 229 instances per seed. NOTE: this is NOT a subset of SWE-bench_Verified — 30 of the
+# 229 exist only in the FULL SWE-bench test split (all of them xarray). Provisioning only
+# Verified is therefore insufficient; see SWEBENCH_FULL_SPLIT_ONLY below.
+SWEBENCH_INSTANCES: dict[str, tuple[str, ...]] = {
+ "django/django": (
+ "django__django-7530", "django__django-10097", "django__django-10554",
+ "django__django-10880", "django__django-10914", "django__django-10973",
+ "django__django-10999", "django__django-11066", "django__django-11087",
+ "django__django-11095", "django__django-11099", "django__django-11119",
+ "django__django-11133", "django__django-11138", "django__django-11149",
+ "django__django-11163", "django__django-11179", "django__django-11206",
+ "django__django-11211", "django__django-11239", "django__django-11265",
+ "django__django-11276", "django__django-11292", "django__django-11299",
+ "django__django-11333", "django__django-11400", "django__django-11433",
+ "django__django-11451", "django__django-11477", "django__django-11490",
+ "django__django-11532", "django__django-11551", "django__django-11555",
+ "django__django-11603", "django__django-11728", "django__django-11734",
+ "django__django-11740", "django__django-11749", "django__django-11790",
+ "django__django-11820",
+ ),
+ "matplotlib/matplotlib": (
+ "matplotlib__matplotlib-13989", "matplotlib__matplotlib-14623",
+ "matplotlib__matplotlib-20488", "matplotlib__matplotlib-20676",
+ "matplotlib__matplotlib-20826", "matplotlib__matplotlib-20859",
+ "matplotlib__matplotlib-21568", "matplotlib__matplotlib-22719",
+ "matplotlib__matplotlib-22865", "matplotlib__matplotlib-22871",
+ "matplotlib__matplotlib-23299", "matplotlib__matplotlib-23314",
+ "matplotlib__matplotlib-23412", "matplotlib__matplotlib-23476",
+ "matplotlib__matplotlib-24026", "matplotlib__matplotlib-24149",
+ "matplotlib__matplotlib-24177", "matplotlib__matplotlib-24570",
+ "matplotlib__matplotlib-24627", "matplotlib__matplotlib-24637",
+ "matplotlib__matplotlib-24870", "matplotlib__matplotlib-24970",
+ "matplotlib__matplotlib-25122", "matplotlib__matplotlib-25287",
+ "matplotlib__matplotlib-25311", "matplotlib__matplotlib-25332",
+ "matplotlib__matplotlib-25479", "matplotlib__matplotlib-25775",
+ "matplotlib__matplotlib-25960", "matplotlib__matplotlib-26113",
+ "matplotlib__matplotlib-26208", "matplotlib__matplotlib-26291",
+ "matplotlib__matplotlib-26342", "matplotlib__matplotlib-26466",
+ ),
+ "scikit-learn/scikit-learn": (
+ "scikit-learn__scikit-learn-9288", "scikit-learn__scikit-learn-10297",
+ "scikit-learn__scikit-learn-10844", "scikit-learn__scikit-learn-10908",
+ "scikit-learn__scikit-learn-11310", "scikit-learn__scikit-learn-11578",
+ "scikit-learn__scikit-learn-12585", "scikit-learn__scikit-learn-12682",
+ "scikit-learn__scikit-learn-12973", "scikit-learn__scikit-learn-13124",
+ "scikit-learn__scikit-learn-13135", "scikit-learn__scikit-learn-13142",
+ "scikit-learn__scikit-learn-13328", "scikit-learn__scikit-learn-13439",
+ "scikit-learn__scikit-learn-13496", "scikit-learn__scikit-learn-13779",
+ "scikit-learn__scikit-learn-14053", "scikit-learn__scikit-learn-14087",
+ "scikit-learn__scikit-learn-14141", "scikit-learn__scikit-learn-14496",
+ "scikit-learn__scikit-learn-14629", "scikit-learn__scikit-learn-14710",
+ "scikit-learn__scikit-learn-14894", "scikit-learn__scikit-learn-14983",
+ "scikit-learn__scikit-learn-15100", "scikit-learn__scikit-learn-25102",
+ "scikit-learn__scikit-learn-25232", "scikit-learn__scikit-learn-25747",
+ "scikit-learn__scikit-learn-25931", "scikit-learn__scikit-learn-25973",
+ "scikit-learn__scikit-learn-26194", "scikit-learn__scikit-learn-26323",
+ ),
+ "sphinx-doc/sphinx": (
+ "sphinx-doc__sphinx-7440", "sphinx-doc__sphinx-7454", "sphinx-doc__sphinx-7462",
+ "sphinx-doc__sphinx-7590", "sphinx-doc__sphinx-7748", "sphinx-doc__sphinx-7757",
+ "sphinx-doc__sphinx-7889", "sphinx-doc__sphinx-7910", "sphinx-doc__sphinx-7985",
+ "sphinx-doc__sphinx-8035", "sphinx-doc__sphinx-8056", "sphinx-doc__sphinx-8120",
+ "sphinx-doc__sphinx-8265", "sphinx-doc__sphinx-8269", "sphinx-doc__sphinx-8459",
+ "sphinx-doc__sphinx-8475", "sphinx-doc__sphinx-8548", "sphinx-doc__sphinx-8551",
+ "sphinx-doc__sphinx-8593", "sphinx-doc__sphinx-8595", "sphinx-doc__sphinx-8621",
+ "sphinx-doc__sphinx-8638", "sphinx-doc__sphinx-8721", "sphinx-doc__sphinx-9229",
+ "sphinx-doc__sphinx-9230", "sphinx-doc__sphinx-9258", "sphinx-doc__sphinx-9281",
+ "sphinx-doc__sphinx-9320", "sphinx-doc__sphinx-9367", "sphinx-doc__sphinx-9461",
+ "sphinx-doc__sphinx-9591", "sphinx-doc__sphinx-9602", "sphinx-doc__sphinx-9658",
+ "sphinx-doc__sphinx-9673", "sphinx-doc__sphinx-9698", "sphinx-doc__sphinx-9711",
+ "sphinx-doc__sphinx-10323", "sphinx-doc__sphinx-10435", "sphinx-doc__sphinx-10449",
+ "sphinx-doc__sphinx-10466", "sphinx-doc__sphinx-10614", "sphinx-doc__sphinx-10673",
+ "sphinx-doc__sphinx-11445", "sphinx-doc__sphinx-11510",
+ ),
+ "sympy/sympy": (
+ "sympy__sympy-11618", "sympy__sympy-12096", "sympy__sympy-12419", "sympy__sympy-12481",
+ "sympy__sympy-12489", "sympy__sympy-13031", "sympy__sympy-13091", "sympy__sympy-13372",
+ "sympy__sympy-13480", "sympy__sympy-13551", "sympy__sympy-13615", "sympy__sympy-13647",
+ "sympy__sympy-13757", "sympy__sympy-13798", "sympy__sympy-13852", "sympy__sympy-13877",
+ "sympy__sympy-13878", "sympy__sympy-13974", "sympy__sympy-14248", "sympy__sympy-14531",
+ "sympy__sympy-14711", "sympy__sympy-14976", "sympy__sympy-15017", "sympy__sympy-15345",
+ "sympy__sympy-15349", "sympy__sympy-15599", "sympy__sympy-15809", "sympy__sympy-15875",
+ "sympy__sympy-15976", "sympy__sympy-16450", "sympy__sympy-16597", "sympy__sympy-16766",
+ "sympy__sympy-16792", "sympy__sympy-16886", "sympy__sympy-17139", "sympy__sympy-17318",
+ "sympy__sympy-17630", "sympy__sympy-17655", "sympy__sympy-18189", "sympy__sympy-18763",
+ ),
+ "pydata/xarray": (
+ "pydata__xarray-4879", "pydata__xarray-6386", "pydata__xarray-6394",
+ "pydata__xarray-6400", "pydata__xarray-6461", "pydata__xarray-6548",
+ "pydata__xarray-6598", "pydata__xarray-6599", "pydata__xarray-6601",
+ "pydata__xarray-6721", "pydata__xarray-6744", "pydata__xarray-6798",
+ "pydata__xarray-6804", "pydata__xarray-6823", "pydata__xarray-6857",
+ "pydata__xarray-6882", "pydata__xarray-6889", "pydata__xarray-6938",
+ "pydata__xarray-6971", "pydata__xarray-6992", "pydata__xarray-6999",
+ "pydata__xarray-7003", "pydata__xarray-7019", "pydata__xarray-7052",
+ "pydata__xarray-7089", "pydata__xarray-7101", "pydata__xarray-7105",
+ "pydata__xarray-7112", "pydata__xarray-7120", "pydata__xarray-7147",
+ "pydata__xarray-7150", "pydata__xarray-7203", "pydata__xarray-7229",
+ "pydata__xarray-7233", "pydata__xarray-7347", "pydata__xarray-7391",
+ "pydata__xarray-7393", "pydata__xarray-7400", "pydata__xarray-7444",
+ ),
+}
+
+# The three seeds each replay the SAME instance list; seeds vary only agent
+# nondeterminism, never task selection (see experiments/scripts/run_experiment.py --seed).
+SWEBENCH_SEEDS: tuple[int, ...] = (0, 1, 2)
+
+# Instances NOT present in princeton-nlp/SWE-bench_Verified — fetch these from the full
+# princeton-nlp/SWE-bench test split (they are the recent xarray tail).
+SWEBENCH_FULL_SPLIT_ONLY: frozenset[str] = frozenset((
+ "pydata__xarray-4879", "pydata__xarray-6386", "pydata__xarray-6394", "pydata__xarray-6400",
+ "pydata__xarray-6548", "pydata__xarray-6598", "pydata__xarray-6601", "pydata__xarray-6798",
+ "pydata__xarray-6804", "pydata__xarray-6823", "pydata__xarray-6857", "pydata__xarray-6882",
+ "pydata__xarray-6889", "pydata__xarray-6971", "pydata__xarray-6999", "pydata__xarray-7003",
+ "pydata__xarray-7019", "pydata__xarray-7052", "pydata__xarray-7089", "pydata__xarray-7101",
+ "pydata__xarray-7105", "pydata__xarray-7112", "pydata__xarray-7120", "pydata__xarray-7147",
+ "pydata__xarray-7150", "pydata__xarray-7203", "pydata__xarray-7347", "pydata__xarray-7391",
+ "pydata__xarray-7400", "pydata__xarray-7444",
+))
+
+# The Copilot sweep additionally ran this instance; the Claude sweep and the published
+# task_counts (xarray = 39) do not include it. Recorded for completeness only.
+SWEBENCH_COPILOT_EXTRA: frozenset[str] = frozenset((
+ "pydata__xarray-7179",
+))
+
+# Codex-native arms ran a subset of the same instances, one seed per repo.
+CODEX_SWEBENCH_RUNS: tuple[tuple[str, str], ...] = (
+ ("django/django", "s0"),
+ ("sympy/sympy", "s0"),
+ ("scikit-learn/scikit-learn", "s1"),
+ ("sphinx-doc/sphinx", "s0"),
+ ("pydata/xarray", "s1"),
+)
+
+# ---------------------------------------------------------------------------
+# Web-Bench (trimech-web; 5 projects, each a SEQUENTIALLY DEPENDENT stream)
+# ---------------------------------------------------------------------------
+# Order is load-bearing: task N assumes task N-1's edits, so run each tuple front to back.
+# Every project runs `task-1` .. `task-20`; `calculator` and `table` additionally open with an
+# `init` scaffold task (hence 21).
+#
+# The five are chosen on TWO MEASURED criteria, both learned the expensive way (2026-09-07,
+# a $29 lane that retired two projects):
+#
+# grounding -- the distiller admits a fact only if it can cite >= MIN_CITE=2 confirming
+# task-ids from the agent's own trajectory (mechanisms/injection.py `_admit`), and the
+# digests it reads are COMMAND-shaped. So a project where the agent barely runs anything
+# starves it: every candidate dies as `grounding-fail`. `threejs` was retired for
+# exactly this.
+#
+# wall-clock -- Web-Bench's evaluator aborts an agent request at undici's 300s default
+# (see webbench_task_timeout). A project whose tasks want more than ~150s of REQUEST time
+# cannot finish its stream. Measured: the suite norm is ~76s/task; `svg-chart` ran
+# 174-200s/task and died at task-5 of 20 in two arms of three. It was retired for that.
+#
+# `calculator` is KEPT deliberately as the short static control: dropping it would collapse
+# the 4x spread in stream length that made the starvation effect visible.
+#
+# `zustand` was ADOPTED 2026-09-08 after a baseline gate cell: 107.4s request/task (max
+# 212.5s, nothing capped) is comfortable under the 240s cap. It does NOT ground (1.8 shell
+# calls/task) and is not expected to -- grounding was dropped as a selection criterion.
+#
+# `sass` was rejected the same day on WALL-CLOCK RISK: 185.8s/task with 8 of 15 attempts
+# hitting the cap, so its pass@1 was being set by truncation rather than by the model.
+# `table` replaces it, chosen on WORST-CASE timing -- 7.0 estimated turns/task x 11.8 s/turn
+# (the slowest rate ever measured here, on svg-chart) = 82s, roughly 3x headroom under the
+# cap even if every estimate is wrong in the bad direction. Still ungated at time of writing.
+#
+# Rule of thumb the measurements support: app-logic projects run 4.7-5.9 s/turn (expressjs,
+# zustand, react); visual/CSS ones run 7.3-11.8 (calculator, threejs, svg-chart, sass). Pick
+# on turns x the WORST s/turn, not the expected one.
+#
+# Task ids are only unique WITHIN a project. Order is load-bearing: task N assumes N-1.
+WEBBENCH_TASKS: dict[str, tuple[str, ...]] = {
+ "calculator": (
+ "init", "task-1", "task-2", "task-3", "task-4", "task-5", "task-6", "task-7", "task-8",
+ "task-9", "task-10", "task-11", "task-12", "task-13", "task-14", "task-15", "task-16",
+ "task-17", "task-18", "task-19", "task-20",
+ ),
+ "expressjs": (
+ "task-1", "task-2", "task-3", "task-4", "task-5", "task-6", "task-7", "task-8",
+ "task-9", "task-10", "task-11", "task-12", "task-13", "task-14", "task-15", "task-16",
+ "task-17", "task-18", "task-19", "task-20",
+ ),
+ "react": (
+ "task-1", "task-2", "task-3", "task-4", "task-5", "task-6", "task-7", "task-8",
+ "task-9", "task-10", "task-11", "task-12", "task-13", "task-14", "task-15", "task-16",
+ "task-17", "task-18", "task-19", "task-20",
+ ),
+ "zustand": (
+ "task-1", "task-2", "task-3", "task-4", "task-5", "task-6", "task-7", "task-8",
+ "task-9", "task-10", "task-11", "task-12", "task-13", "task-14", "task-15", "task-16",
+ "task-17", "task-18", "task-19", "task-20",
+ ),
+ "table": (
+ "init", "task-1", "task-2", "task-3", "task-4", "task-5", "task-6", "task-7", "task-8",
+ "task-9", "task-10", "task-11", "task-12", "task-13", "task-14", "task-15", "task-16",
+ "task-17", "task-18", "task-19", "task-20",
+ ),
+}
+
+# The project set the 2026-08 exports were actually run on. FROZEN: it describes a run that
+# already happened and cannot be edited after the fact. Kept here so WEBBENCH_TASKS above is
+# free to move on while the record stays pinned to what was measured.
+WEBBENCH_TASKS_PUBLISHED_2026_08: dict[str, int] = {
+ "flex": 21, "expressjs": 20, "react": 20, "prisma": 20, "calculator": 21,
+}
+
+# Web-Bench seeds are LETTERED in the published trees (not 0/1/2 like the other domains).
+WEBBENCH_SEEDS: tuple[str, ...] = ("A", "B", "C")
+
+# ---------------------------------------------------------------------------
+# CredData (security / secret detection, graded to line-level F1)
+# ---------------------------------------------------------------------------
+# 100 of CredData's 333 repo ids. Each id is a directory `data/<id>/` with labels in
+# `meta/<id>.csv`; every repo gets the same read-only audit prompt.
+CREDDATA_INSTANCES: tuple[str, ...] = (
+ "00408ef6", "0401c075", "08295bf0", "0a0d22aa", "0ce78112", "110f8d35", "16fc0238",
+ "262e8107", "28728ab4", "29d5ecae", "30201159", "31423103", "353ae7fa", "35c273e2",
+ "371ea6d8", "39def7b4", "3d82970b", "443086e9", "46c52073", "489fc16c", "4db30bbe",
+ "4f40bf46", "50b38ef0", "55031e17", "55aa4858", "57067d8d", "5cb346c3", "5cecf769",
+ "5d8870b3", "60f9915d", "63800a10", "638eb0c6", "653a7746", "660708f3", "661abd43",
+ "69f42949", "6a132a15", "6d59a88a", "6e2ed0e4", "7529016f", "75e7c64d", "7c7558cf",
+ "7e01f352", "7f8b28a7", "81cd05d0", "826a020b", "848ad017", "86875835", "8bc560cd",
+ "8c3c80e8", "8c9fcde7", "8d217d6c", "8f499d55", "9031b2b4", "91cfdb2b", "93ff401b",
+ "98a9ba4e", "9c7e3662", "9ccb72ee", "9d90ba70", "9ef45116", "a0cd6261", "a15774b8",
+ "a5c3685e", "a8bdca79", "ac9be8d9", "adf34139", "afdd01d8", "b6dbde6a", "b81d3227",
+ "b8d8b8c3", "bbb4193f", "bcb18a48", "bd7bcb9d", "c2127ffb", "c9b945fa", "cc268724",
+ "cd82f1a2", "cfdcc4f3", "d2d6eedc", "d518da13", "d93c01fb", "db218f09", "db832824",
+ "df70e863", "e08ff555", "e097ae9a", "e3c63910", "e69fe1b9", "eb8448ad", "eb9ba732",
+ "f129be24", "f43ddbf5", "f710ac3c", "f7ae636a", "f80c571b", "f85f83c9", "fc8343f4",
+ "ff43ce4d", "ff527e6f",
+)
+CREDDATA_SEEDS: tuple[int, ...] = (0, 1, 2)
+
+# ---------------------------------------------------------------------------
+# ReposVul (vulnerability detection, graded to line-level F1)
+# ---------------------------------------------------------------------------
+# 100 instances, ALL from the Python split (`py-*`) — the C/Java/PHP splits were not run.
+REPOSVUL_INSTANCES: tuple[str, ...] = (
+ "py-10081", "py-10115", "py-1045", "py-10585", "py-10746", "py-10878", "py-10883",
+ "py-11295", "py-11519", "py-11621", "py-11682", "py-11868", "py-11990", "py-12077",
+ "py-12275", "py-12277", "py-12279", "py-12292", "py-12343", "py-12375", "py-12440",
+ "py-12497", "py-171", "py-1968", "py-1974", "py-211", "py-2210", "py-2496", "py-2656",
+ "py-2660", "py-2754", "py-2771", "py-2847", "py-3163", "py-3380", "py-3482", "py-3596",
+ "py-369", "py-3802", "py-4061", "py-4374", "py-4450", "py-4597", "py-4615", "py-4816",
+ "py-49", "py-5016", "py-5203", "py-5259", "py-5557", "py-5566", "py-5567", "py-5569",
+ "py-569", "py-5751", "py-5838", "py-5897", "py-5934", "py-5969", "py-5987", "py-6002",
+ "py-6200", "py-623", "py-6237", "py-6510", "py-6540", "py-6565", "py-6634", "py-6646",
+ "py-6678", "py-6764", "py-6773", "py-6817", "py-7185", "py-7458", "py-7563", "py-7740",
+ "py-8152", "py-8320", "py-8450", "py-8464", "py-8504", "py-8523", "py-8851", "py-8958",
+ "py-9020", "py-9030", "py-9070", "py-9081", "py-9217", "py-9275", "py-940", "py-941",
+ "py-9652", "py-9659", "py-9824", "py-9859", "py-9876", "py-9902", "py-9968",
+)
+REPOSVUL_SEEDS: tuple[int, ...] = (0, 1, 2)
+
+DOMAINS: tuple[str, ...] = ("swebench", "webbench", "creddata", "reposvul")
+
+_BY_DOMAIN = {
+ "swebench": SWEBENCH_INSTANCES,
+ "webbench": WEBBENCH_TASKS,
+ "creddata": {"creddata": CREDDATA_INSTANCES},
+ "reposvul": {"reposvul": REPOSVUL_INSTANCES},
+}
+
+
+def instances(domain, group=None):
+ """Task ids for `domain`, or just those of one `group` (repo / project).
+
+ `group` is the SWE-bench repo slug (``pydata/xarray``) or Web-Bench project name.
+ Raises KeyError on an unknown domain or group — never silently returns everything."""
+ try:
+ groups = _BY_DOMAIN[domain]
+ except KeyError:
+ raise KeyError(f"unknown domain {domain!r}; expected one of {DOMAINS}") from None
+ if group is None:
+ return tuple(i for g in groups.values() for i in g)
+ if group not in groups:
+ raise KeyError(f"{domain}: unknown group {group!r}; expected one of "
+ f"{tuple(groups)}")
+ return tuple(groups[group])
+
+
+# ---- forward-looking run exclusions -------------------------------------------------
+# SWEBENCH_INSTANCES above is the PROVENANCE record: exactly what the published results
+# ran, and it must not shrink (experiments/tests/test_workloads.py pins those sizes -- the
+# published runs graded every id in it).
+# This table is the separate, forward-looking question: which of those ids should a NEW run
+# skip. `runnable_instances()` is what the exporter and launchers consume; task_counts()
+# and total_tasks() deliberately do NOT subtract it, so the historical record stays intact.
+SWEBENCH_EXCLUDED: dict[str, str] = {
+ "scikit-learn__scikit-learn-14710":
+ "eval.sh does not terminate under te_harness: timed out at TE_EVAL_TIMEOUT=1800 "
+ "and again at 3600, and the uncaught TimeoutExpired ended the sklearn stream at "
+ "task 21 of 32 both times (that cell never produced more than 20 graded rows). "
+ "The timeout is now caught (te_harness swebench.py _grade), so it would cost one "
+ "full eval_timeout per run rather than the cell; excluded to stop paying that. "
+ "NOT ungradeable in principle -- it graded in the published runs, so results "
+ "that include it and results that exclude it are not directly comparable on "
+ "scikit-learn.",
+}
+
+
+# ---- forward-looking stream extensions ----------------------------------------------
+# De-saturation. On the published workload scikit-learn and sympy resolve at or near the
+# ceiling for every arm, so they no longer separate the arms; a stream that returns the same
+# number for every arm cannot support any claim made on it.
+#
+# These ids EXTEND those two streams to 60 RUNNABLE tasks each (sklearn 31 + 29,
+# sympy 40 + 20). They are additions for FUTURE runs only: `SWEBENCH_INSTANCES` above
+# stays exactly as the published results ran it, and `task_counts()` / `total_tasks()`
+# keep reporting 229, so the published record remains true.
+# `runnable_instances()` is what the exporter and launchers consume, and it folds these in.
+#
+# Selection rule (reproducible from princeton-nlp/SWE-bench, full `test` split):
+# * not already in SWEBENCH_INSTANCES;
+# * gold patch 61-200 changed lines;
+# * `version` already present in that repo's existing stream, so the per-repo operating
+# manual `injection` learns stays valid across the extension;
+# * sorted by (gold lines, id), then sampled at even spacing so the band is covered
+# rather than clustered at one end.
+#
+# Why 61-200 and not "anything big": xarray is the precedent -- it is the one stream that
+# was already extended with full-split tail ids, and its baseline rows (opus-4.8) show
+# what each band actually scores:
+# full-only <=60 lines -> 75% resolved (n=48) no harder than Verified
+# full-only 61-120 -> 44% (n=9)
+# full-only 121-200 -> 17% (n=12)
+# full-only >200 -> 5% (n=21) floor: costs the most, discriminates least
+# So being outside Verified buys nothing on its own; gold-patch SIZE is the difficulty
+# dial, and >200 overshoots into a floor. 61-200 blends to roughly 30% expected resolve,
+# which brings both streams to ~65% pooled.
+#
+# GOLD-PATCH SCREENED, 2026-09-06: 49/49. Unlike SWE-bench_Verified these ids were never
+# human-validated for solvability, so each one was screened before being committed here --
+# start the official image, apply the GOLD patch into /testbed, run the official eval
+# script, score it with the official get_eval_report. All 49 came back resolved=True with
+# the gold patch applying cleanly, every FAIL_TO_PASS passing and zero PASS_TO_PASS
+# regressions, so none is underspecified or ungradeable under this harness.
+#
+# Eval wall-clock ran 14-187 s (median 32 s), which also clears the failure mode that put
+# scikit-learn-14710 in SWEBENCH_EXCLUDED: every added instance terminates roughly an order
+# of magnitude inside the default TE_EVAL_TIMEOUT=1800, so none of them can take a stream
+# down the way that one did.
+#
+# Screen script and per-instance records: $DATA/swebench/gold_screen.py and
+# $DATA/swebench/gold_screen_added.jsonl. Re-run it if you extend this table again; a new
+# id that fails belongs nowhere near a paid sweep. Note two ids carry an empty
+# PASS_TO_PASS and so have no regression guard: scikit-learn-11585 and -11596.
+SWEBENCH_ADDED: dict[str, tuple[str, ...]] = {
+ "scikit-learn/scikit-learn": (
+ "scikit-learn__scikit-learn-3840", "scikit-learn__scikit-learn-7760",
+ "scikit-learn__scikit-learn-9274", "scikit-learn__scikit-learn-10459",
+ "scikit-learn__scikit-learn-10577", "scikit-learn__scikit-learn-10803",
+ "scikit-learn__scikit-learn-11496", "scikit-learn__scikit-learn-11585",
+ "scikit-learn__scikit-learn-11596", "scikit-learn__scikit-learn-12421",
+ "scikit-learn__scikit-learn-12758", "scikit-learn__scikit-learn-12827",
+ "scikit-learn__scikit-learn-12989", "scikit-learn__scikit-learn-13010",
+ "scikit-learn__scikit-learn-13046", "scikit-learn__scikit-learn-13363",
+ "scikit-learn__scikit-learn-13467", "scikit-learn__scikit-learn-13618",
+ "scikit-learn__scikit-learn-13628", "scikit-learn__scikit-learn-13864",
+ "scikit-learn__scikit-learn-13910", "scikit-learn__scikit-learn-14237",
+ "scikit-learn__scikit-learn-14464", "scikit-learn__scikit-learn-14544",
+ "scikit-learn__scikit-learn-14704", "scikit-learn__scikit-learn-14806",
+ "scikit-learn__scikit-learn-14999", "scikit-learn__scikit-learn-23099",
+ "scikit-learn__scikit-learn-25744",
+ ),
+ "sympy/sympy": (
+ "sympy__sympy-11862", "sympy__sympy-11989", "sympy__sympy-13259",
+ "sympy__sympy-13265", "sympy__sympy-13301", "sympy__sympy-13988",
+ "sympy__sympy-14308", "sympy__sympy-15225", "sympy__sympy-15304",
+ "sympy__sympy-15555", "sympy__sympy-15586", "sympy__sympy-16781",
+ "sympy__sympy-17194", "sympy__sympy-17313", "sympy__sympy-17340",
+ "sympy__sympy-17394", "sympy__sympy-18199", "sympy__sympy-18256",
+ "sympy__sympy-18477", "sympy__sympy-19201",
+ ),
+}
+
+# Every added id is outside SWE-bench_Verified (both repos' Verified rows were already
+# fully consumed by the published streams), so provisioning must read the FULL test split
+# for these -- exactly like SWEBENCH_FULL_SPLIT_ONLY above.
+SWEBENCH_ADDED_FULL_SPLIT_ONLY: frozenset[str] = frozenset(
+ i for ids in SWEBENCH_ADDED.values() for i in ids)
+
+
+def runnable_instances(domain, group=None):
+ """Canonical instances PLUS SWEBENCH_ADDED MINUS SWEBENCH_EXCLUDED — what a NEW
+ run should execute.
+
+ Differs from `instances()`/`task_counts()`, which report the recorded workload and
+ stay fixed at the 229 the published results ran. Today that makes scikit-learn and
+ sympy 60 runnable tasks each (the de-saturation extension) and leaves the other four
+ repos untouched. Only 'swebench' has additions/exclusions; other domains pass through.
+ """
+ groups = _BY_DOMAIN[domain]
+ skip = SWEBENCH_EXCLUDED if domain == "swebench" else {}
+ add = SWEBENCH_ADDED if domain == "swebench" else {}
+ out = {g: tuple(i for i in tuple(v) + tuple(add.get(g, ())) if i not in skip)
+ for g, v in groups.items()}
+ return out[group] if group is not None else out
+
+
+def task_counts(domain):
+ """{group: n} — the per-group workload size of the recorded workload."""
+ return {g: len(v) for g, v in _BY_DOMAIN[domain].items()}
+
+
+def total_tasks(domain):
+ """Total task ids in one seed of `domain` (multiply by the seed count for task-runs)."""
+ return len(instances(domain))
+
+
+def swebench_image_tag(instance_id):
+ """Official per-instance SWE-bench execution image for `instance_id`.
+
+ The harness convention doubles the `__` separator into `_1776_`:
+ ``django__django-11532`` -> ``swebench/sweb.eval.x86_64.django_1776_django-11532:latest``
+ """
+ return f"swebench/sweb.eval.x86_64.{instance_id.replace('__', '_1776_')}:latest"
+
+
+def filter_tasks(tasks, domain, group=None):
+ """Subset a driver task list to the canonical workload, preserving its order.
+
+ `tasks` is the list of ``{instance_id, problem_statement, repo}`` dicts the driver
+ loads from ``TOKENECONOMIZER_<DOMAIN>_TASKS``. Use this to pin a run to exactly the
+ evaluated instances instead of whatever the task file happens to contain::
+
+ tasks = json.load(open(os.environ['TOKENECONOMIZER_SWEBENCH_TASKS']))
+ tasks = workloads.filter_tasks(tasks, 'swebench', 'pydata/xarray')
+
+ Unlike the driver's ``--repo`` prefix match, an empty result is returned as empty
+ rather than falling back to the unfiltered list."""
+ wanted = set(instances(domain, group))
+ return [t for t in tasks if t.get("instance_id") in wanted]