Experience-Driven Black-Box Cost Control for Coding Agents
Python agentic-aiagentic-codingai-codingai-coding-assistantanthropicartificial-intelligenceclaude-codecodexcoding-agentcoding-agentscost-optimizationcost-reductiondeveloper-toolsgithub-copilotllmopen-sourceopenaiproductivitytoken-optimization
removed redundant comments
Aaron3cfd3f20be20
diff --git a/docs/PLUGIN.md b/docs/PLUGIN.mdindex dafea54..27fdd4c 100644--- a/docs/PLUGIN.md+++ b/docs/PLUGIN.md@@ -271,11 +271,11 @@ isolated `PI_CODING_AGENT_DIR`):the change, so one left out would be recorded as removed. Unchanged, it adds nothing afterthe first prompt.* Pi's bash result carries a real exit code (`structuredContent.exit_code`), which the- ended-green proxy uses; its session file is a tree, and only the active branch is read.-* Not exercised live: a nudge (`sendMessage` with `deliverAs: "steer"`) and a stop- (`ctx.abort()`, since removed: the plugin no longer stops a turn) — the dispatcher's side is- covered by the offline tests, Pi's by its API. Tool pruning does not apply: Pi declares four- tools by default.+ ended-green proxy uses (unless another command's status replaced it: 2026-10-26, below); its+ session file is a tree, and only the active branch is read.+* Not exercised live here: a nudge (`sendMessage` with `deliverAs: "steer"`, since exercised:+ 2026-10-21 and 2026-10-26, below) and a stop (`ctx.abort()`, since removed: the plugin no+ longer stops a turn). Tool pruning does not apply: Pi declares four tools by default.**Pi 1.2.0 — verified live, 2026-10-14** (Haiku 4.5 through Pi's built-in OpenRouter provider,set as the default provider and model in an isolated `PI_CODING_AGENT_DIR`, as a Pi user's own@@ -314,15 +314,6 @@ this plugin's guide step by step (Claude Code 2.1.300, Codex 0.153.1, Copilot CL* Pi's real git install (a local git daemon standing in for GitHub): cloned, `npm install`ed,listed, loaded; `pi update --extensions` fast-forwarded it; `pi remove` removed it.-What the run found, now fixed: Claude Code 2.1.300's `api-request-blob` entries doubled every-call read from its transcript; a working form was credited from runs that needed a-`VAR=value` it leaves out, once per failure × success, and never debited; Pi's and Copilot's-logs gave no per-call directory, so `cd <repo> && …` was learned as a separate form; the guard-counted file reads as failures; an absolute-path test runner was not a verification; facts-needed sessions that edited different files and, once injected, decayed (the fold never knew-what the session was given); a Codex install copy had no package; side calls failed silently-and left sessions in Claude Code's history; `te` without `--host` showed Claude Code's state.-**Again as a new user, interactively, 2026-10-21** -- one fresh home for all four agents (ClaudeCode 2.1.306, Codex 0.163.0, Copilot CLI 1.0.101, Pi 1.3.0; every model through OpenRouter),the main README's Quick start typed as written. `git clone https://github.com/microsoft/…` and@@ -369,17 +360,15 @@ thresholds), in its terminal UI and headless.* Then, since Claude Code hid its own line, the plugin there also asks the model to repeat thenotice word for word (`hosts.relay`). Claude Haiku 5.5 opened its next message and its finalreply with it, in the terminal UI and under `claude -p`, whose text output then began with it.- On the way, found and fixed: Claude Code rejects an answer to a failed call that names- `PostToolUse` ("Hook returned incorrect event name: expected 'PostToolUseFailure' but got- 'PostToolUse'") and drops it, so a redirect or notice that fell on a failed call, as in a- failing test run, was lost there; an answer now names the event it answers. Answering a failed- call, the hook's own line also showed under the call without Ctrl+O.+ Claude Code drops an answer to a failed call that names `PostToolUse` ("Hook returned+ incorrect event name: expected 'PostToolUseFailure' but got 'PostToolUse'"), so an answer+ names the event it answers. Answering a failed call, the hook's own line also showed under the+ call without Ctrl+O.* Telling headless from interactive: Claude Code's environment says so, and Pi's extension; Codexonly in its rollout's first record (`source: "exec"`); Copilot only in its own command line(`copilot -p`), its payloads, environment and session log being the same in both modes.-* Found and fixed: Codex 0.163.0 edits with `apply_patch <<'PATCH'` and runs the tests in the- same command, and the guard counted none of its twelve edits, so it never fired; and- `copilot -p` was told the interactive way to stop.+* Codex 0.163.0 edits with `apply_patch <<'PATCH'` and runs the tests in the same command; the+ guard counts those edits (`loopguard.shell_edits`).**Prompt caching, 2026-10-21.** A prompt cache matches a request's prefix: the tools, thesystem prompt, then the conversation in order. What the plugin changes must leave a cached@@ -398,7 +387,7 @@ OpenRouter (cache breakpoints, which move every turn, left out of the comparisonresume --last` and `copilot --continue` sent the old history with exactly the new tool list,so a conversation resumed while its cache is warm writes its whole history to the cache againinstead of reading it. The fold runs just after a session ends, when its cache is warmest and- resuming it likeliest, so `auto` now holds a new list as pending, and a detached+ resuming it likeliest, so `auto` holds a new list as pending, and a detached`te toolprune apply-pending` writes it once every session of the repository has been idle for`toolprune_quiet_minutes` (60; a hook call or a write to the session's own log counts asactivity), or once it has waited a day. End to end on Claude Code, the window set to one@@ -410,18 +399,16 @@ OpenRouter (cache breakpoints, which move every turn, left out of the comparisonrunning session alone.* A resumed conversation must carry the facts it started with, byte for byte. Claude Code andCodex replay them from their own logs (Claude Code changes the message from a block list to a- string of the same text). Two did not, and both are fixed:- * Copilot rebuilds a resumed conversation's first message without what `SessionStart` added:- the facts were gone and that message missed the cache. The plugin now records the text a- session started with and gives it again on a Copilot resume, which rebuilds the message- exactly; a session that started without facts gets none.- * Pi rebuilds its system message from the conversation's own entries, and the extension left- its section out on a resume, which Pi recorded as removed. Pi also reports `pi -c` and- `--session` as `startup`, not `resume`, and stores a section wrapped in `<name>` tags. The- extension now decides by whether the branch already holds messages, and gives such a- conversation back the section it carries, unwrapped (`carriedSection`): two resumes after a- fact was retired on disk kept the original system message, and a new session took the- current facts.+ string of the same text). Copilot and Pi rebuild it, and the plugin gives them the same text:+ * Copilot rebuilds a resumed conversation's first message without what `SessionStart` added.+ The plugin records the text a session started with and gives it again on a Copilot resume,+ which rebuilds the message exactly; a session that started without facts gets none.+ * Pi rebuilds its system message from the conversation's own entries, and records a section+ left out as removed. It reports `pi -c` and `--session` as `startup`, not `resume`, and+ stores a section wrapped in `<name>` tags. The extension decides by whether the branch+ already holds messages, and gives such a conversation back the section it carries, unwrapped+ (`carriedSection`): two resumes after a fact was retired on disk kept the original system+ message, and a new session took the current facts.**Counting prompts, 2026-10-24** (on the `experimental` branch, since merged into `main`). Can the mechanisms learn from theprompts of a session rather than from whole sessions, so a long first session counts for more?@@ -445,9 +432,7 @@ prompts of a session rather than from whole sessions, so a long first session coonce and its later prompts use the working form the conversation already shows, so the factfired in a quarter of the prompts that exercised it and never cleared the firing gate (40%over at least 4): in a stream of four-prompt sessions it was never injected, where per- session it is injected at the fifth start. A replay that split prompts into sessions of- their own had shown the opposite only because each such start recorded the fact as given.- The facts still count sessions.+ session it is injected at the fifth start. The facts count sessions.* The loop guard already counts prompt-turns: each turn that ends green feeds the envelope atits `Stop`, and the guard acts within the same session once it has five.@@ -465,12 +450,9 @@ a second clone of it, in another repository, and in a plain folder.* Copilot's unused MCP server was turned off after five sessions and announced at the next start;the clone got the list at the end of its own next session.* Pi's proposer ran after the second and fifth sessions, through `pi --print`, and was charged.--What the run found, now fixed: a session started in a subdirectory that ran-`cd <repository> && pytest` learned that as a form of its own, spelled with the machine's path (a `cd` up to the-repository now lands relative to it); and under BYOK Copilot's proposer asked to sign in every-third session, leaving an empty session behind each time (a CLI that answers it is not signed-in now pauses the plugin's model calls on that agent, and the next session start says so once).+* A session started in a subdirectory that runs `cd <repository> && pytest` runs the plain+ form, learned relative to the repository. Under BYOK, a CLI that answers it is not signed in+ pauses the plugin's model calls on that agent, and the next session start says so once.**The judge decides, 2026-10-25** (Claude Code 2.1.310, Claude Haiku 5.5 through OpenRouter, forthe agent and for the judge; thresholds lowered so a loop forms quickly, the backstop raised to@@ -490,11 +472,35 @@ admitted four facts -- where the build script, the tests and the manifest are, asessions given them still read the same files first, so in a repository this small they savednothing.-What the run's preparation found, now fixed: the proposer's window of recent sessions was not-saved with its state. The experiments keep one proposer in memory over a stream; the plugin-reloads it for every session, so each run saw only the session that had just ended, and a fact,-which must cite two, could never be admitted. Every proposer call before this was spent for-nothing.+The experiments keep one proposer in memory over a stream; the plugin reloads it for every+session, so its window of recent sessions is saved with its state.++**Pi, every mechanism, 2026-10-26** (Pi 1.3.0, Claude Haiku 5.5 through OpenRouter, for the+agent and the judge; `microskills=on`). Eight sessions in the same Node repository, then turns+told to re-run a test that cannot pass, with the judge's gate lowered to three calls. `/te+status` ran inside Pi and named tool pruning as not available there; each session was folded+within seconds of its exit; three proposer runs gave three distinct facts, in Pi's recorded+system prompt from the fifth session on; a resumed conversation (`pi -c`) carried the same+section, and its requests read the cache. The judge ran in the background as a bare `pi+--print`, found a turn of identical failures stuck, and its redirect reached the agent as a+steering message (a `custom_message` of type `tokeneconomizer`), after which it stopped+re-running the test and found the missing fixture. With every run piped (`npm test 2>&1 | tail+-n 8`), each counted as failing and the judge found the turn stuck (0.90).++**How the guard reads a test run, 2026-10-26.** A command line's exit status is its last+command's, so it may not be the test run's: `npm test 2>&1 | tail -n 8` exits with `tail`'s,+`npm test || true` and `python -m unittest; echo "exit=$?"` with 0, and `pytest -q; git diff`+with `git`'s. For such a run the runner's own summary line decides (pytest's `1 failed, 4+passed in 0.5s`, unittest's `FAILED (failures=1)` or `OK`, TAP's `# fail 1`, jest's, mocha's,+Go's, cargo's, rspec's, JUnit's; `loopguard.turn.runner_verdict`). Without one, a run whose+status a pipe, `||` or a report of it replaced is read by its output, as on an agent that+reports no status (Claude Code); a run followed by other commands keeps the status the agent+reported. Replayed through the guard, the recorded SWE-bench sessions end well in 85.6% of+turns on Claude Code (13,312 sessions), 84.5% on Copilot (6,710) and 87.8% on Codex (2,653), and+the judge's gate opens in 18.1%, 15.8% and 5.8% of them.++The plugin's proposer prompt lists each fact it holds with its claim, so a fact it has is not+authored again under a new id; the experiments' prompt names the ids alone, as the paper ran it.## 4. State and privacy@@ -570,9 +576,6 @@ open-vocabulary. On a repository you do not trust, leave micro-skills off (the dredirect but the backstop's, which still counts.* **Pi** — tool pruning is not planned there, since Pi declares only four tools by default.* **Known gaps:**- * A test run whose exit code the model masks (`python3 -m unittest; echo "exit=$?"`, seen on- Copilot) reads as passing where the agent reports exit codes (Copilot, Codex, Pi), so a turn- of such runs counts as ended green and is never flagged.* `codex exec` prints no hook messages, so a headless Codex run shows no notice; `te status`counts it afterwards. On Claude Code the notice reaches the conversation and `claude -p`'soutput only because the model repeats it, which a model may word differently or leave out;diff --git a/experiments/scripts/aggregate.py b/experiments/scripts/aggregate.pyindex 5a6471a..342afa5 100644--- a/experiments/scripts/aggregate.py+++ b/experiments/scripts/aggregate.py@@ -48,7 +48,7 @@ FIELDS = [# A row whose token fields could not be read in full (the agent was killed before its# terminal usage event, or its debug log was unparseable). Such a row is a LOWER BOUND# on cost, so it must be visible in the CSV the comparisons read -- an unflagged zero- # was previously indistinguishable from a genuinely free task.+ # is indistinguishable from a genuinely free task."usage_incomplete", "usage_incomplete_reason", "output_tokens_missing",# A row whose eval script never RAN (the grading `docker exec` failed -- usually the# container went away under us). `resolved` is False on such a row, which is@@ -122,21 +122,21 @@ def _run_order(rj):def usage_incomplete(d):"""(is_incomplete, reason) for one row: are its token fields a LOWER BOUND?- Two sources, because the flag has to cover rows written before it existed:+ Two sources, because the flag has to cover rows from a harness that does not set it:- * The harness now says so itself. `te_harness` sets `usage_incomplete` when the agent- was killed before its terminal usage event and the row had to be rebuilt from the+ * The harness says so itself. `te_harness` sets `usage_incomplete` when the agent was+ killed before its terminal usage event and the row had to be rebuilt from thetranscript (input/cache exact, `output_tokens` unrecoverable), or when Copilot'sdebug log could not be parsed.- * ROWS STORED BEFORE THAT FIX carry no flag at all -- they recorded a flat zero for- every token field and for `num_turns`. They are recognised by their signature: no- tokens, yet the task demonstrably ran (it resolved, or it left a patch). The- alternative reading, a task that really cost nothing and still produced a diff, is- not a thing that happens. Only guarded arms could produce these, so leaving them- unmarked biases every economized arm's mean DOWNWARD.-- `pricing.effective_tokens` is deliberately left alone: this reports the defect, it does- not paper over it. A recovered figure is a floor and gets labelled as one.+ * A row with no flag is recognised by its signature (`legacy_zero_usage`): a flat+ zero for every token field, yet the task demonstrably ran (it resolved, or it left+ a patch). The alternative reading, a task that really cost nothing and still+ produced a diff, is not a thing that happens. Only guarded arms kill a rollout+ mid-stream, so leaving such rows unmarked would bias every economized arm's mean+ DOWNWARD.++ `pricing.effective_tokens` is deliberately left alone: this reports the lower bound, it+ does not paper over it. A recovered figure is a floor and gets labelled as one."""if d.get("usage_incomplete"):return True, d.get("usage_incomplete_reason") or "harness_flagged"@@ -169,9 +169,9 @@ def vendor_usd(d):The runners write it as `cost_usd_vendor` (Claude Code's `total_cost_usd`, copiedfrom its final `result` event); only the older curated exports carry it under the- plain name `cost_usd`. Reading just `cost_usd` -- which this script did until the- fix -- left the column EMPTY for every tree this repo generates. The COLUMN keeps its- name so its consumers keep working; only the source key is fixed.+ plain name `cost_usd`. Reading just `cost_usd` would leave the column EMPTY for every+ tree this repo generates. The COLUMN is still named `cost_usd` so its consumers keep+ working; only the source key differs."""for k in ("cost_usd_vendor", "cost_usd"):v = d.get(k)@@ -191,7 +191,7 @@ def vendor_cost_missing(d):"""(missing, reason) for one row: did a task that RAN lose its vendor dollar figure?When the agent's final `result` event arrives without `total_cost_usd` the row stores- None, and until this flag nothing marked it. The row is not otherwise damaged, which+ None, and nothing but this flag marks it. The row is not otherwise damaged, whichis what makes it dangerous: token accounting is complete (so `usage_incomplete` staysFalse and the effective-token basis is exact) and the task grades normally, so therow looks healthy from every other column. Summing `cost_usd` across the cell then@@ -378,10 +378,9 @@ def main():# every time, so narrowing the arm list while writing to the canonical <root>/index# replaces an index describing the tree with one describing a subset -- at the path# compare_arms.py reads, under a name that says nothing about being partial. A lane- # that runs one arm per invocation (the Codex lane) is the easy way to do it, and a- # tree was once left holding one arm of eight that way. So a subset write- # to the canonical path is refused rather than warned about -- pass --out to put the- # subset somewhere of its own, or drop --arms and aggregate the tree.+ # that runs one arm per invocation (the Codex lane) is the easy way to do it. So a+ # subset write to the canonical path is refused rather than warned about -- pass --out+ # to put the subset somewhere of its own, or drop --arms and aggregate the tree.if args.arms and args.out is None:dropped = [a for a in present if a not in arms]if dropped:diff --git a/experiments/scripts/launch/_env.sh b/experiments/scripts/launch/_env.shindex 477b9b3..0d2a4a5 100644--- a/experiments/scripts/launch/_env.sh+++ b/experiments/scripts/launch/_env.sh@@ -35,9 +35,8 @@ te_load_env() {# te_dryrun_skips_billed <what> — true when this is a DRYRUN, so a BILLED probe must not run.## A dry run is how a lane is costed before spending anything, so a probe that BILLS during-# one contradicts the whole-# point: DRYRUN=1 on a Copilot lane used to spend up to three premium requests and ~240 s-# before printing "nothing was run".+# one contradicts the whole point: on a Copilot lane the auth and model probes are up to+# three premium requests and ~240 s.te_dryrun_skips_billed() {[[ "${DRYRUN:-0}" == "1" ]] || return 1echo "[env] DRYRUN: skipping the billed $1 probe" >&2@@ -253,12 +252,12 @@ te_check_routing() {# te_copilot_refused <probe-output> — did the CLI refuse the probe outright?#-# Both checks below used to key on the ABSENCE of one known error string, which meant any-# other refusal read as success. A spent subscription is exactly that case and it is not-# rare on a long sweep: the CLI prints "You have exceeded your monthly quota", the JSON-# stream carries {"errorType":"quota","statusCode":402,"errorCode":"quota_exceeded"}, and-# the auth probe still EXITS 0 -- so the launcher printed `token warm` and `model serves`-# on an account that could not run a single task, and the lane then filled its cells with+# Both checks below look for one known error string, and its ABSENCE is not success: any+# other refusal would read as one. A spent subscription is that case, and it is not rare on+# a long sweep: the CLI prints "You have exceeded your monthly quota", the JSON stream+# carries {"errorType":"quota","statusCode":402,"errorCode":"quota_exceeded"}, and the auth+# probe still EXITS 0 -- on its own the launcher would print `token warm` and `model serves`+# for an account that cannot run a single task, and the lane would fill its cells with# turns=1 / eff_tok=0 rows that nothing aborts. Keyed on the vendor's own error text and# structured code, so it covers both the JSON and the plain-text probe shapes.te_copilot_refused() {@@ -373,12 +372,12 @@ te_check_copilot_model() {out="$(timeout 120 "$bin" -p "ok" --no-color --log-level none \--model "$model" ${effort:+--effort "$effort"} 2>&1)"# Before the model verdict: a refused probe says nothing about the model, and reading it- # as "serves" is how a spent quota reached the run banner as all-green.+ # as "serves" would put a spent quota on the run banner as all-green.te_copilot_refused_die "$out" "model"# Two "not served" shapes, one per billing path. The subscription says "is not# available"; a BYOK provider says "Model 'x' not found on provider at <url> (HTTP 404)".- # Keying on the first alone let a BYOK lane print `serves; effort accepted` for an id the- # provider 404s, then write a tree of turns=1 / eff_tok=0 rows (measured 2026-09-12).+ # Keying on the first alone would let a BYOK lane print `serves; effort accepted` for an+ # id the provider 404s, then write a tree of turns=1 / eff_tok=0 rows.if te_copilot_model_unserved "$out"; then# The same model under the other separator, once: the Copilot CLI's own catalog spells# the older Opus `claude-opus-4.8` while an Anthropic-wire BYOK provider takesdiff --git a/experiments/scripts/launch/main_sweep.sh b/experiments/scripts/launch/main_sweep.shindex 0ceec82..a45eab8 100755--- a/experiments/scripts/launch/main_sweep.sh+++ b/experiments/scripts/launch/main_sweep.sh@@ -146,7 +146,7 @@ echo "[main_sweep] agent=$AGENT model=$MODEL tag=$MODEL_TAG domain=$DOMAIN"echo "[main_sweep] dest=$DEST"echo "[main_sweep] cells=${#PLAN[@]} seeds='$SEEDS' runner=$RUNNER skip_done=$SKIP_DONE"# --limit must be forwarded or the estimate describes the FULL stream while the run is-# capped: a LIMIT=1 pilot printed "task-runs= 40 ... ~$14" for one task.+# capped."$PY" "$HERE/../sweep_plan.py" --domain "$DOMAIN" "${WORKLOAD_ARGS[@]}" "${ARM_ARGS[@]}" \"${REPO_ARGS[@]}" --seeds $SEEDS --limit "$LIMIT" --format estimate | sed 's/^/[main_sweep] /'diff --git a/experiments/scripts/launch/smoke_test.sh b/experiments/scripts/launch/smoke_test.shindex 3add798..93fe5c2 100755--- a/experiments/scripts/launch/smoke_test.sh+++ b/experiments/scripts/launch/smoke_test.sh@@ -98,7 +98,7 @@ echo "[smoke] workload manifest narrows to ONE group (ids repeat across groups)"# Web-Bench numbers every project's stream 'init', 'task-1', ... so ids are unique only# WITHIN a project. Filtering on ids alone silently returns every project's tasks# interleaved -- which, for a sequentially dependent stream, is not a bigger run but a-# meaningless one. Two synthetic groups sharing all three ids catch that regression.+# meaningless one. Two synthetic groups sharing all three ids make that visible.WB_ROOT="$WORK/wb/projects"mkdir -p "$WB_ROOT/alpha" "$WB_ROOT/beta"for proj in alpha beta; dodiff --git a/experiments/scripts/run_experiment.py b/experiments/scripts/run_experiment.pyindex 54a53e9..01536e8 100644--- a/experiments/scripts/run_experiment.py+++ b/experiments/scripts/run_experiment.py@@ -298,18 +298,18 @@ def lift_overhead(row, srow):"""Copy mechanism-overhead fields a mechanism reported on `srow` into `row`.row.json is the only artifact aggregate.py reads, so an overhead field that exists- solely on the stream row is invisible to the cost basis. `untool_judge_tokens` was- exactly that: `tooling.after_task` writes the per-task instruction-aware tool judge's- spend to `srow` alone, so every arm with judge=True -- untool-only and all four trimech- variants -- had that line item silently dropped from `cost_effective_tokens`. The bias- is one-directional: a baseline runs no judge, so only the arms under test are- flattered. Same shape as the usage_incomplete trap in aggregate.py.+ solely on the stream row is invisible to the cost basis. `untool_judge_tokens` is one:+ `tooling.after_task` writes the per-task instruction-aware tool judge's spend to `srow`+ alone, so without the lift every arm with judge=True -- untool-only and all four+ trimech variants -- would drop that line item from `cost_effective_tokens`. The bias+ would be one-directional: a baseline runs no judge, so only the arms under test would+ be flattered. Same shape as usage_incomplete in aggregate.py.A value the runner already put on `row` WINS: the harness sets spiral_judge_tokensitself during run_task, and `srow` may carry a stale or zeroed copy of it.- Driven by ledger.OVERHEAD_KEYS so a newly added overhead field cannot silently- reintroduce the gap. Returns `row` for convenience.+ Driven by ledger.OVERHEAD_KEYS so a newly added overhead field cannot silently open+ the same gap. Returns `row` for convenience."""for k in ledger.OVERHEAD_KEYS:if k not in row and srow.get(k) is not None:diff --git a/experiments/scripts/summarize_stream.py b/experiments/scripts/summarize_stream.pyindex f4f568f..761b4f8 100644--- a/experiments/scripts/summarize_stream.py+++ b/experiments/scripts/summarize_stream.py@@ -112,10 +112,8 @@ def summarize(path, raw=False, exclude_init=False):# is the paid live trace-judge; spiral_nudge_writer_tokens is the v6.1 targeted-nudge# writer. All are real mechanism-internal LLM cost that must be amortized into the# honest task cost. The overhead-key set is the SINGLE definition in- # economizer.ledger.OVERHEAD_KEYS is now tokeneconomizer.ledger.OVERHEAD_KEYS,- # shared with the emit path so the two never drift.- # (This now INCLUDES spiral_nudge_writer_tokens, previously emitted but never summed- # -- a latent under-count of amortized cost, fixed per the unified-ledger decision.)+ # tokeneconomizer.ledger.OVERHEAD_KEYS, shared with the emit path so the two never+ # drift; every key above, spiral_nudge_writer_tokens included, is summed.# BASIS: rows tagged mech_tokens_basis=="effective" carry cache-aware effective# tokens (same basis as exec) — sum directly. Legacy rows (no tag) carry RAW# face-value tokens: an UNDER-count of effective (mechanism calls are uncacheddiff --git a/experiments/scripts/sweep_plan.py b/experiments/scripts/sweep_plan.pyindex db89f5c..70280a2 100755--- a/experiments/scripts/sweep_plan.py+++ b/experiments/scripts/sweep_plan.py@@ -156,10 +156,9 @@ def estimate(domain, arms, repos, seeds, sec_per_task, usd_per_task, limit=0, ma"""Task-runs, wall-clock and dollars for the requested slice.`limit` mirrors the launcher's LIMIT (cap tasks per cell, 0 = all). It has to be- honoured here or the estimate describes a run nobody asked for: a LIMIT=1 pilot used to- be costed at the full stream -- "task-runs= 40 ... ~$14" for a cell that runs ONE task- -- which is the wrong direction for a number whose whole job is to be checked before- spending money.+ honoured here or the estimate describes a run nobody asked for: a LIMIT=1 pilot would+ be costed at the full stream for a cell that runs ONE task, which is the wrong+ direction for a number whose whole job is to be checked before spending money."""sizes = group_sizes(domain, manifest)unknown = [r for r in repos if r not in sizes]diff --git a/experiments/te_bench/baselines/__init__.py b/experiments/te_bench/baselines/__init__.pyindex 2ab4cf0..80f9a88 100644--- a/experiments/te_bench/baselines/__init__.py+++ b/experiments/te_bench/baselines/__init__.py@@ -5,9 +5,8 @@ the harness's existing seams; a scaffold here is somebody else's off-the-shelf t(`rtk`, Serena, aider's repo map) installed into the sealed agent's container so its costand resolve can be measured on the same honest basis.-They sit in the core package, beside the mechanisms they are the comparison for, and the-two invariants that makes people nervous about are both kept structurally rather than by-convention:+They sit with the experiments, beside the mechanisms they are the comparison for, and two+invariants are kept structurally rather than by convention:* **Nothing here reaches a container.** Every container interaction arrives as a callableon `ScaffoldContext` (`exec_`, `copy_in`, `read_bytes`, `host_run`), supplied bydiff --git a/experiments/te_bench/baselines/base.py b/experiments/te_bench/baselines/base.pyindex ab9fab5..b54c449 100644--- a/experiments/te_bench/baselines/base.py+++ b/experiments/te_bench/baselines/base.py@@ -40,8 +40,8 @@ class AgentProfile:"""The per-agent facts a scaffold needs. Runners expose one as `scaffold_profile`.`kind` is the coarse switch for genuinely agent-specific wiring (RTK registers a- PreToolUse hook through Claude's settings.json and through a Copilot plugin -- there is- no common spelling). Everything else is data.+ PreToolUse hook through Claude's settings.json and through Copilot's+ `$COPILOT_HOME/hooks/` directory -- there is no common spelling). Everything else is data."""def __init__(self, kind, home, instructions_file, nav_tools, mcp_args,diff --git a/experiments/te_bench/baselines/rtk.py b/experiments/te_bench/baselines/rtk.pyindex 29c1223..9c19594 100644--- a/experiments/te_bench/baselines/rtk.py+++ b/experiments/te_bench/baselines/rtk.py@@ -21,9 +21,9 @@ Two integration modes, and the row says which one ran (`rtk_mode`):token-compressed proxies. The scaffold's whole job is to make sure the hook is actuallyregistered, which is the step the tool itself will not do for you: `rtk init -g` innon-interactive mode DECLINES to patch the agent's settings, prints a "MANUAL STEP" notice,-and exits NONZERO (observed rc=1). Trusting it produces an arm where rtk is installed, `rtk --version` works,-every provenance field looks healthy -- and the hook never fires, so the arm is native with-extra steps. So the hook is written explicitly and the row records whether that took.+and exits 0 (rtk 0.45.0). Trusting it produces an arm where rtk is installed, `rtk --version`+works, every provenance field looks healthy -- and the hook never fires, so the arm is native+with extra steps. So the hook is written explicitly and the row records whether that took.The binary is staged on the HOST and copied in, because the sealed task container has noGitHub access: `install.sh` and the release tarball are both unreachable from inside. The@@ -83,13 +83,10 @@ class RtkScaffold(Scaffold):# /usr/local/bin, already on every PATH -- but on a host-run agent (Web-Bench) bin_dir# is a per-task directory, and a bare `rtk` resolves to nothing.#- # Measured 2026-09-05, Web-Bench/prisma/task-1 with the bare spelling: the debug log- # carried `rtk: command not found` x31 and the CLI reported the preToolUse hook- # erroring, while `rtk_installed` and `rtk_hook_registered` both read true. The agent- # flailed against the failing hook -- 612 `task` subagent calls, 553 `create`, 412- # `bash` -- and the task cost 75 turns / 1.79M tokens / 778 s against a 6-turn,- # 140k-token, 86 s baseline. So this is worse than a silent no-op: a bare hook command- # actively corrupts the run at ~12x the cost.+ # A hook command that does not resolve is worse than a silent no-op: every shell call+ # gets `rtk: command not found` and the CLI reports the preToolUse hook erroring, while+ # `rtk_installed` and `rtk_hook_registered` both read true. The agent then works around+ # the failing hook, and the task costs many times its native run.@staticmethoddef _hook_binary(profile):return f"{profile.bin_dir}/rtk"@@ -184,8 +181,8 @@ class RtkScaffold(Scaffold):# the settings/plugin write took; it says nothing about whether the agent can run# the command. Resolve it the way the agent's hook subprocess will -- by absolute# path, under a PATH that does NOT include bin_dir -- so a hook that cannot execute- # is a flag on the row rather than 31 `command not found` lines inside a run that- # still reports healthy.+ # is a flag on the row rather than a `command not found` on every shell call inside+ # a run that still reports healthy.probe = ctx.exec_(f"env PATH=/usr/bin:/bin {hook_bin} --version", timeout=60)notes["rtk_hook_exec_ok"] = probe.returncode == 0# 3c. Registered AND executable is still not REWRITING -- see _probe_hook_rewrites.@@ -266,11 +263,10 @@ class RtkScaffold(Scaffold):"toolArgs":{"command":...}} <- Copilot's own**rtk parses only the first.** Handed the second it exits 0 with EMPTY stdout, which- the CLI reads as "no opinion" and runs the command unchanged. That is what the- previous implementation did: it shipped a plugin whose `hooks.json` used the- `preToolUse` spelling, so rtk was installed, the plugin reported "installed- successfully", the hook fired on every call -- and rewrote nothing. Every Copilot- `rtk` / `trimech-rtk` row before 2026-09-13 is that no-op.+ the CLI reads as "no opinion" and runs the command unchanged. A hook on that channel+ -- or in a plugin whose `hooks.json` uses the `preToolUse` spelling, which hands the+ hook the same Copilot shape -- installs cleanly and fires on every call, and rewrites+ nothing.On the hooks-dir channel the loop is closed end to end: the model asked for`ls -la src`, the CLI executed `rtk ls -la src`, and the agent got rtk's compressed@@ -311,8 +307,8 @@ class RtkScaffold(Scaffold):`rtk_hook_registered` says the config was written and `rtk_hook_exec_ok` that thebinary runs; neither catches a hook that parses nothing and returns an empty- verdict, which is exactly how the Copilot arm was silently native for a week. This- is the effect-level check: feed the hook the payload its channel delivers and look+ verdict, which leaves the arm native while both read true. This is the+ effect-level check: feed the hook the payload its channel delivers and lookfor the rewrite in its answer. Cheap, offline, and it fails loudly if a future rtkchanges its reply shape."""diff --git a/experiments/te_bench/codex_spiral.py b/experiments/te_bench/codex_spiral.pyindex d021a2f..501e437 100644--- a/experiments/te_bench/codex_spiral.py+++ b/experiments/te_bench/codex_spiral.py@@ -9,13 +9,22 @@ CLI invocation. Codex exposes stronger signals and a native session-resume comm* ``thread.started.thread_id`` can be passed to ``codex exec resume``;* file-change events contain every changed path, not just a generic edit tool name.-The policy below uses only those deterministic, locally observable signals. It has-no paid judge and never terminally aborts a task. The live v1 policy has one trigger:-after the resolved-task tool envelope, a patch spanning at least 15 distinct files has-failed verification after its latest edit. At most once per task it checkpoints that-trajectory and continues the *same* Codex thread. Broader failure-loop and post-green-rules remain available for offline experiments but are disabled by default. An-orthogonal, opt-in completion audit can use the same thread once when deterministic+The policies below use only those deterministic, locally observable signals. They have+no paid judge and never terminally abort a task; every rule is a nudge on the *same* Codex+thread. Two profiles ship (`profile_policy`):++* ``post-green`` (`SHIPPED_POLICY`): one nudge to finalize, once the task's own success+ check has passed unfiltered, every named pytest scope that failed unfiltered has since+ passed, and the trajectory has run past the resolved-task tool envelope;+* ``defensive`` (`DEFENSIVE_POLICY`, what the Codex lane runs unless an arm says+ otherwise, `CODEX_DEFAULT_PROFILE`): the same, plus every deterministic spiral signature+ the controller knows -- repeated failing verification, a repeated command, blocked+ retrieval, a wide or churning patch that keeps failing, a passing check re-run again and+ again, a run past twice the envelope -- at most two nudges per task.++The bare dataclass defaults (`DEFAULT_POLICY`) are a single trigger: after the envelope, a+patch spanning at least 15 distinct files has failed verification after its latest edit.+An orthogonal, opt-in completion audit can use the same thread once when deterministicend-state evidence shows an empty patch or a still-failing post-edit verification."""@@ -173,18 +182,17 @@ class CodexSpiralPolicy:# to finalize when the task's own success check has passed UNFILTERED and the# trajectory has already run past the learned resolved-task envelope. It is gated on# a trustworthy green, so it can never interrupt an unfinished task or trust a- # narrowed pass (the injection-v3 / xarray-7400 false-green trap).+ # narrowed pass (a filtered green covers only a subset of the accepted set).post_green_finalize_enabled: bool = False# Grace after the trustworthy green before nudging: never interrupt the accept step# itself; only redirect the wandering that follows it.post_green_finalize_grace_turns: int = 3- # Live A/B lesson (xarray, seed 0): an unfiltered green on ONE test file does not- # prove the patch is done when ANOTHER scope still has an outstanding unfiltered- # failure. Two false fires regressed resolve this way:- # * xarray-7400: gold `test_concat.py` failed unfiltered, was re-run only FILTERED,- # and the gate fired on an unfiltered green of a DIFFERENT file (`test_combine.py`).- # * xarray-6857: the full-suite `pytest` failed unfiltered after the last edit and- # was never cleared; the gate fired on a narrow green (`test_indexes.py`).+ # An unfiltered green on ONE test file does not prove the patch is done when ANOTHER+ # scope still has an outstanding unfiltered failure. Two shapes of it:+ # * a test file failed unfiltered and was re-run only FILTERED, while a different+ # file then passed unfiltered;+ # * the full-suite `pytest` failed unfiltered after the last edit and has not passed+ # since, while a single file then passed.# When enabled, finalize additionally requires that NO named pytest scope (and not the# whole-suite scope) is carrying an outstanding unfiltered failure -- i.e. every scope# that ever failed unfiltered has since been re-run unfiltered and passed.@@ -217,13 +225,12 @@ DEFAULT_POLICY = CodexSpiralPolicy()# WANDERING -- re-verifying, re-deriving the change from history, style/lint fishing. This# policy fires ONE same-thread nudge to finalize once the task's own success check has# passed UNFILTERED and the trajectory has run past the learned resolved-task envelope. It-# is trustworthy-green gated (never trusts a narrowed `-k`/`--lf` pass) and, from the live-# A/B, additionally requires that every named pytest scope that ever failed unfiltered has-# since been re-run unfiltered and passed -- so a green on ONE file cannot mask an-# outstanding unfiltered failure on ANOTHER (the xarray-7400 / xarray-6857 regressions its-# predecessor hit). The wide-failure/recovery rules are disabled so this measures the-# post-green lever alone. It never terminally aborts a task. It supersedes the single-v1,-# hardened-v2 and post-green-v2 trials.+# is trustworthy-green gated (never trusts a narrowed `-k`/`--lf` pass) and additionally+# requires that every named pytest scope that ever failed unfiltered has since been re-run+# unfiltered and passed -- so a green on ONE file cannot mask an outstanding unfiltered+# failure on ANOTHER (without it, a failure on another file that has not been retired+# reads as a finished patch). The wide-failure/recovery rules are disabled so this measures+# the post-green lever alone. It never terminally aborts a task.PROFILE_VERSION = "codex-native-spiral-post-green-cleared-scopes"SHIPPED_POLICY = replace(@@ -554,8 +561,8 @@ class CodexSpiralState:filtered = not green_is_trustworthy(command)scopes = verification_scopes(command)if event.exit_code == 0:- # A green targeted test/reproducer supersedes historical failures. This is- # the signal the generic spiral path lost on xarray-6599.+ # A green targeted test/reproducer supersedes earlier failures: the failure+ # counters below restart from zero.if self.last_edit_turn and self.last_green_turn > self.last_edit_turn:# already green after this edit: this run re-verifies a passing stateself.green_reruns_since_edit += 1@@ -574,8 +581,8 @@ class CodexSpiralState:self.outstanding_failure_scopes.clear()else:# A NARROW green retires only its own scope(s). It must NOT clear a- # whole-suite (`_ALL_SCOPE`) failure -- that was the xarray-6857 trap,- # where a green on `test_indexes.py` masked an unresolved full-suite+ # whole-suite (`_ALL_SCOPE`) failure: a green on one file says nothing+ # about the rest of the suite, so it would mask an unresolved full-suite# failure. Only a whole-suite unfiltered green clears `_ALL_SCOPE`.self.outstanding_failure_scopes -= scopes# Record whether THIS green was a full unfiltered pass. Only a trustworthy@@ -615,8 +622,8 @@ class CodexSpiralState:blocked = (policy.post_green_finalize_require_cleared_failuresand self.outstanding_failure_scopes)# `blocked`: somewhere an unfiltered test is still red on a scope this green- # did not retire (the xarray-7400 / xarray-6857 regression class) -- do not- # finalize on that ground. The rules below may still apply.+ # did not retire -- do not finalize on that ground. The rules below may still+ # apply.if (not blocked and self.currently_green and self.last_green_trustworthyand self.first_trustworthy_green_turnand self.turns - self.first_trustworthy_green_turn >= grace):@@ -972,10 +979,9 @@ class CodexResumeGuard:The SHARED task loop (`te_harness.swebench` run_task) logs `guard.segment` rightafter a continuation, and the generic `spiral.NudgeGuard` exposes it -- so a Codex- guard without it crashes the lane on its FIRST fire, which is exactly what happened- on 2026-09-12 (the sol-era nudges ran on the other harness, so nothing had ever- exercised this path here). `nudges_done` carries the same count and is incremented- in `continuation()` before the log line reads it, so the two agree by construction.+ guard must expose it too, or the loop raises on its FIRST nudge. `nudges_done`+ carries the same count and is incremented in `continuation()` before the log line+ reads it, so the two agree by construction.`tests/test_spiral.py` pins the whole attribute contract, not just this one name."""return self.nudges_donediff --git a/experiments/te_bench/mechanism_arms.py b/experiments/te_bench/mechanism_arms.pyindex 16fcf41..2d03583 100644--- a/experiments/te_bench/mechanism_arms.py+++ b/experiments/te_bench/mechanism_arms.py@@ -27,13 +27,11 @@ ENVFACTS = dict(enabled=True, profile="envfacts")# the generic cadence -- the environment facts are already stated, so the proposer's calls# are rarer and its budget goes to facts beyond them.#-# Why the paid layer is the default now. The free layer's interpreter fact formed because-# of one harness defect: the project environment was not active, so `python` resolved to-# the base interpreter. The harness now activates the environment on every lane, and-# replaying the layer over a post-fix baseline forms NO interpreter fact at all -- only the-# network-boundary line. With the defect gone the free layer abstains and the-# open-vocabulary proposer is what is left to find repository facts; its spend is charged-# to the arm.+# Why the paid layer is the default. The harness activates the project environment on+# every lane, so `python` resolves to the project interpreter and the free layer forms NO+# interpreter fact -- over a baseline it states only the network-boundary line. The free+# layer therefore mostly abstains, and the open-vocabulary proposer is what is left to find+# repository facts; its spend is charged to the arm.ENVFACTS_PLUS = dict(enabled=True, profile="envfacts+distiller", propose_every=10)# The Codex lane's injection profiles by name: the default, and its LLM-free ablation# (arm `codex-inject-envfacts`).@@ -180,9 +178,8 @@ MECHANISM_ARMS = {"standalone",lambda: cfg(spiral=SpiralConfig(**CODEX_SPIRAL))),# Untooling + the spiral, i.e. `codex-trimech` minus injection. It reads the same two- # dicts trimech does, so it stays that as the parameters move; when its spiral half was- # pinned to post-green independently it was quietly a different arm from the one its- # name promised.+ # dicts trimech does, so it stays that as the parameters move; a spiral half pinned+ # independently would make it a different arm from the one its name promises."codex-combo": arm("standalone",lambda: cfg(tooling=ToolingConfig(**CODEX_UNTOOL),@@ -230,7 +227,7 @@ MECHANISM_ARMS = {# instructions file, so the resumed segment inherits the untooled contract and the# prompt prefix already in its history. Nothing is re-sent and nothing is re-paid.#- # Two compounding effects to expect in the row, neither of them a defect:+ # Two compounding effects to expect in the row, both by construction:# * the proposer's spend lands in `optimizer_tokens` (this is a PAID arm);# * a task the guard nudges reports no usage for the killed segment, so# `codex.merge_usage` poisons the merge and the row is flagged `usage_incomplete`.diff --git a/experiments/te_bench/spiral_stage.py b/experiments/te_bench/spiral_stage.pyindex 6992f8b..35d5738 100644--- a/experiments/te_bench/spiral_stage.py+++ b/experiments/te_bench/spiral_stage.py@@ -116,9 +116,9 @@ class SpiralStage(Stage):row.get("codex_spiral_paid_judge_tokens") or 0)# Advance the envelope in the unit the TRIGGER uses: the guard's own# command+edit count, not the row's `num_turns` (agent steps = messages + tool- # calls, ~1.45x larger). Feeding steps raised the gate ~45% and left the- # controller unable to fire on most tasks; `CodexEnvelopeTracker` documents the- # tool-event unit and now actually receives it. Fall back to `num_turns` only+ # calls, ~1.45x larger). A gate learned in steps sits ~45% above the count it is+ # compared to, and most tasks never reach it; `CodexEnvelopeTracker` documents+ # the tool-event unit and receives it here. Fall back to `num_turns` only# when a row predates the field, and say so through the telemetry rather than# silently mixing units.tool_turns = row.get("codex_spiral_tool_turns")diff --git a/experiments/te_bench/workloads.py b/experiments/te_bench/workloads.pyindex 7171269..faa1d46 100644--- a/experiments/te_bench/workloads.py+++ b/experiments/te_bench/workloads.py@@ -174,8 +174,7 @@ CODEX_SWEBENCH_RUNS: tuple[tuple[str, str], ...] = (# Every project runs `task-1` .. `task-20`; `calculator` and `table` additionally open with an# `init` scaffold task (hence 21).#-# The five are chosen on TWO MEASURED criteria, both learned the expensive way (2026-09-07,-# a $29 lane that retired two projects):+# The five are chosen on TWO MEASURED criteria:## grounding -- the distiller admits a fact only if it can cite >= MIN_CITE=2 confirming# task-ids from the agent's own trajectory (microskills/distiller.py `_admit`), and the@@ -185,8 +184,8 @@ CODEX_SWEBENCH_RUNS: tuple[tuple[str, str], ...] = (## wall-clock -- Web-Bench's evaluator aborts an agent request at undici's 300s default# (see webbench_task_timeout). A project whose tasks want more than ~150s of REQUEST time-# cannot finish its stream. Measured: the suite norm is ~76s/task; `svg-chart` ran-# 174-200s/task and died at task-5 of 20 in two arms of three. It was retired for that.+# cannot finish its stream. Measured: the suite norm is ~76s/task; `svg-chart` runs+# 174-200s/task, too slow to finish a 20-task stream, and is out for that.## `calculator` is KEPT deliberately as the short static control: dropping it would collapse# the 4x spread in stream length that made the starvation effect visible.@@ -325,11 +324,9 @@ def instances(domain, group=None):# and total_tasks() deliberately do NOT subtract it, so the historical record stays intact.SWEBENCH_EXCLUDED: dict[str, str] = {"scikit-learn__scikit-learn-14710":- "eval.sh does not terminate under te_harness: timed out at TE_EVAL_TIMEOUT=1800 "- "and again at 3600, and the uncaught TimeoutExpired ended the sklearn stream at "- "task 21 of 32 both times (that cell never produced more than 20 graded rows). "- "The timeout is now caught (te_harness swebench.py _grade), so it would cost one "- "full eval_timeout per run rather than the cell; excluded to stop paying that. "+ "eval.sh does not terminate under te_harness within TE_EVAL_TIMEOUT (1800 s, nor "+ "3600 s), so every run of it costs one full eval_timeout (te_harness swebench.py "+ "_grade catches the timeout); excluded to stop paying that. ""NOT ungradeable in principle -- it graded in the published runs, so results ""that include it and results that exclude it are not directly comparable on ""scikit-learn.",@@ -373,10 +370,9 @@ SWEBENCH_EXCLUDED: dict[str, str] = {# the gold patch applying cleanly, every FAIL_TO_PASS passing and zero PASS_TO_PASS# regressions, so none is underspecified or ungradeable under this harness.#-# Eval wall-clock ran 14-187 s (median 32 s), which also clears the failure mode that put-# scikit-learn-14710 in SWEBENCH_EXCLUDED: every added instance terminates roughly an order-# of magnitude inside the default TE_EVAL_TIMEOUT=1800, so none of them can take a stream-# down the way that one did.+# Eval wall-clock ran 14-187 s (median 32 s): every added instance terminates roughly an+# order of magnitude inside the default TE_EVAL_TIMEOUT=1800, unlike scikit-learn-14710,+# whose eval does not terminate (see SWEBENCH_EXCLUDED).## Screen script and per-instance records: $DATA/swebench/gold_screen.py and# $DATA/swebench/gold_screen_added.jsonl. Re-run it if you extend this table again; a newdiff --git a/experiments/te_harness/codex.py b/experiments/te_harness/codex.pyindex 87de329..904176a 100644--- a/experiments/te_harness/codex.py+++ b/experiments/te_harness/codex.py@@ -66,7 +66,7 @@ EFFORT_LEVELS = frozenset({"minimal", "low", "medium", "high", "xhigh", "max"})# `apply_patch` is a harness artifact, not a mechanism result. These are CAPABILITY names# rather than tool names because that is the granularity Codex config operates at.PROTECTED_CAPABILITIES = frozenset({- "deferred_wait", # CORE in codex_untool: pruning it provoked shell-polling+ "deferred_wait", # CORE in codex_untool: pruning it provokes shell-polling"apply_patch", "exec_command", "planning", "write_stdin",})@@ -115,9 +115,9 @@ def codex_bin():def codex_vendor(binary=None):"""The host vendor triple dir holding the static-PIE ELF the container runs.- Resolved from the Node shim's install tree rather than hardcoded: the AgentEO driver- pinned an absolute path under one machine's nvm prefix, which is exactly the kind of- machine-specific path the code must not carry. TE_CODEX_VENDOR overrides.+ Resolved from the Node shim's install tree rather than hardcoded: an absolute path under+ one machine's nvm prefix is exactly the kind of machine-specific path the code must not+ carry. TE_CODEX_VENDOR overrides.Raises when the ELF is absent. Docker invents an EMPTY directory for a missing bindmount source rather than failing, so an unchecked bad path here becomes "codex: not@@ -279,7 +279,7 @@ def verify_model(model, routing):# 0.146 driver used are rejected by 0.153.1 and are NOT here:# web_search_mode = "disabled" -> now top-level web_search = "disabled"# [tools] view_image = false -> now [features] view_image = false-# Both were silently ignored by the newer CLI, which is the whole hazard: the pruning+# Both are silently ignored by the newer CLI, which is the whole hazard: the pruning# becomes a no-op and the arm measures itself as useless while still being called# economized. `validate_config` is the gate that keeps this honest across CLI upgrades.#@@ -308,13 +308,13 @@ _TOP_LEVEL_TOGGLES = {_TABLE_TOGGLES = {"collaboration": (("agents", "enabled = false"),("features", "multi_agent = false")),- # Only the TOP-LEVEL `web_search = "disabled"` (in _TOP_LEVEL_TOGGLES) remains.+ # Only the TOP-LEVEL `web_search = "disabled"` (in _TOP_LEVEL_TOGGLES) is emitted.# codex-cli 0.153.1 emits a deprecation ERROR ITEM into the transcript for# `[features].web_search` -- "deprecated because web search is enabled by default; set- # `web_search` to live/indexed/cached/disabled at the top level" -- observed on a live- # preflight 2026-09-10. An error item per task is trace noise the mechanisms then read,- # and a deprecation becomes a hard failure sooner or later, so the redundant table- # entries are dropped in favour of the spelling the CLI itself names.+ # `web_search` to live/indexed/cached/disabled at the top level". An error item per task+ # is trace noise the mechanisms then read, and a deprecation becomes a hard failure+ # sooner or later, so `web` has no table entry: the spelling the CLI itself names is the+ # only one."web": (),"goals": (("features", "goals = false"),),"skills": (),@@ -415,12 +415,11 @@ def bundled_system_skills(binary=None, home=None, timeout=45):back to DEFAULT_SYSTEM_SKILLS and say so. Nothing is billed."""binary = binary or codex_bin()# mkdtemp + an explicit RETRYING cleanup, not TemporaryDirectory. The killed CLI (or a- # child it spawned) can still be writing under CODEX_HOME when we come to remove it, and- # a "Directory not empty" here failed the offline suite about one run in four.- # `ignore_cleanup_errors=True` also makes that non-fatal, but it gives up on the first- # error and LEAVES the directory: 899 MB of throwaway CODEX_HOMEs had accumulated in- # /tmp on this host by 2026-09-13. `_rmtree_quietly` retries the transient case first- # and only then ignores, so the common case is actually cleaned up.+ # child it spawned) can still be writing under CODEX_HOME when we come to remove it, so+ # a plain removal can hit "Directory not empty". `ignore_cleanup_errors=True` would make+ # that non-fatal, but it gives up on the first error and LEAVES the directory, so every+ # probe would strand a throwaway CODEX_HOME in /tmp. `_rmtree_quietly` retries the+ # transient case first and only then ignores, so the common case is actually cleaned up.tmp = tempfile.mkdtemp(dir=home)try:with open(os.path.join(tmp, "config.toml"), "w", encoding="utf-8") as fh:@@ -458,11 +457,10 @@ def _kill_process_group(proc):"""Kill everything a probe started -- not just the process `Popen` returned.`codex` on PATH is the npm package's Node shim, which execs the native ELF as a CHILD.- `proc.kill()` reached only the shim: the native binary, pointed at a dead port, was- reparented to init and sat "waiting for network" for good. Every offline suite run- leaked one (six were alive on this host on 2026-09-30, the oldest five days old).- The probes therefore start in their own session, and the whole group goes -- also when- the shim has already exited, because its child can outlive it."""+ `proc.kill()` reaches only the shim: the native binary, pointed at a dead port, is+ reparented to init and sits "waiting for network" for good. The probes therefore start+ in their own session, and the whole group goes -- also when the shim has already+ exited, because its child can outlive it."""try:os.killpg(proc.pid, signal.SIGKILL)except (ProcessLookupError, PermissionError, AttributeError):@@ -509,10 +507,10 @@ def config_toggles(capabilities):`capabilities` is the logical mask the mechanism froze for this task. A package with nosupported config toggle contributes nothing here rather than a guess."""- # Checked against the RAW mask, before the CONFIG_CAPABILITIES filter. Filtering first- # made this guard dead code -- a protected package is not in CONFIG_CAPABILITIES, so it- # was silently dropped and the refusal never fired. The point is to reject a mask that- # ASKS for something protected, not to quietly honour the part of it that is safe.+ # Checked against the RAW mask, before the CONFIG_CAPABILITIES filter: a protected+ # package is not in CONFIG_CAPABILITIES, so filtering first would drop it silently and+ # the refusal would never fire. The point is to reject a mask that ASKS for something+ # protected, not to quietly honour the part of it that is safe.bad = PROTECTED_CAPABILITIES.intersection(capabilities or ())if bad:# The mechanism asserts this on its own side too; assert it here as well because@@ -725,8 +723,7 @@ def run_env(extra=None):# across probes in the AgentEO study, whose accounting the published Codex rows used). So# fresh input is input - cached - cache_write, and folding the cache fields into# input_tokens instead would price ~90% cached re-reads at 1.0x where the vendor bills-# 0.10x -- the same one-directional error that made Copilot rows look cheaper than Claude-# rows before the 2026-09-05 cache-write fix. Reasoning tokens are folded into output+# 0.10x, a one-directional error in the cost basis. Reasoning tokens are folded into output# because that is how they are billed._USAGE_KEYS = ("input_tokens", "cached_input_tokens", "cache_write_input_tokens","output_tokens", "reasoning_output_tokens")@@ -922,10 +919,10 @@ def optimizer_call(prompt, *, configured_model, routing, max_tokens=16384,total_in = int(usage.get("input_tokens") or 0)# A reasoning model can spend the whole cap on reasoning and emit a ZERO-character# answer -- the caller then parses nothing and acts on nothing while the call is billed- # in full. The Claude path hit exactly this (7 of 40 manual-distill calls at an 8192- # cap) and answered it by raising the cap and saying so, not by lowering effort: effort- # is part of the cost basis and cutting it here would price mechanism overhead below- # what the arm really spends. So warn, loudly, and leave the knob to the operator.+ # in full. The remedy, here as on the Claude path, is a larger cap, never a lower+ # effort: effort is part of the cost basis and cutting it here would price mechanism+ # overhead below what the arm really spends. So warn, loudly, and leave the knob to the+ # operator.if data.get("status") == "incomplete" or not text.strip():why = ((data.get("incomplete_details") or {}).get("reason")or ("empty response" if not text.strip() else "unknown"))diff --git a/experiments/te_harness/copilot.py b/experiments/te_harness/copilot.pyindex c8a4265..f99073d 100644--- a/experiments/te_harness/copilot.py+++ b/experiments/te_harness/copilot.py@@ -16,8 +16,8 @@ and none of them are shared with Claude Code:zeros, and every cost number downstream is wrong rather than missing. The realinput/cache/output breakdown is only in the CLI's own debug log, which is why everyCopilot invocation here runs with `--log-level all --log-dir ...` and mines the- result. `sum_openai_usage` is ported verbatim from the AgentEO study's- `swebench/agents.py`, which is the accounting the published Copilot rows used.+ result. `sum_openai_usage` is ported from the AgentEO study's `swebench/agents.py`,+ the accounting the published Copilot rows used, with cache writes split out.4. **A Copilot-flavoured optimizer call.** The mechanisms' between-task proposer/judge/nudge-writer is a paid LLM call. On the native Copilot path there is no Anthropic keyto make it with, so it goes through the same `copilot -p` binary -- tools denied and@@ -169,15 +169,14 @@ PROVIDER_SECRET_VARS = ("COPILOT_PROVIDER_API_KEY", "COPILOT_PROVIDER_BEARER_TOK# `--deny-tool` withholds PERMISSION: the tool stays in the model's schema and the CLI# simply refuses to run it. `--excluded-tools` removes it from what is offered to the model# at all. The optimizer call is a pure text completion that needs no tool, so it wants the-# second -- and for a long time it passed only the first.+# second.## Measured on this host (2026-09-07), one real judge-shaped prompt of 2,423 chars:# --deny-tool only 22,460 billed prompt tokens -> 29,040 effective# + --excluded-tools 5,550 billed prompt tokens -> 7,832 effective-# The 16,910-token difference is the tool schema, re-sent on EVERY mechanism call. On the-# 60-task sklearn trimech cell that was ~12,500 of the 17,577 tokens/task of mechanism-# overhead the ledger charged the arm -- overhead that bought nothing, since the call is-# forbidden to use a tool either way.+# The 16,910-token difference is the tool schema, which permission alone re-sends on EVERY+# mechanism call -- overhead the ledger charges the arm and that buys nothing, since the+# call is forbidden to use a tool either way.## Both are kept: permission AND schema. Denying a tool that is not offered is harmless, and# leaving the deny list in place means a CLI that ignores one flag still refuses the tool.@@ -216,11 +215,10 @@ def anthropic_optimizer_endpoint(env=None):one-shot text completions that need none of it. Measured on this host, the same2,4xx-char judge prompt costs ~1,730 effective tokens over the Messages API and ~4,700through the trimmed CLI -- and the CLAUDE harness has always used the API for exactly- these calls (swebench.ClaudeSwebenchRunner.optimizer_call). So the CLI transport was- also an ASYMMETRY between the two agents' mechanism overhead, one-directional and in- the same class as the cache-write split fixed on 2026-09-05: it inflated Copilot's- overhead against Claude's on identical work, biasing the comparison the sweep exists- to make.+ these calls (swebench.ClaudeSwebenchRunner.optimizer_call). So the CLI transport is+ also an ASYMMETRY between the two agents' mechanism overhead, and a one-directional+ one: it inflates Copilot's overhead against Claude's on identical work, biasing the+ comparison the sweep exists to make.WHAT IT MUST NOT DO is change the cost basis. `ledger.overhead` charges these calls tothe same basis as the task rows, so the endpoint has to bill whatever the TASKS bill.@@ -417,23 +415,19 @@ def sum_openai_usage(log_text):Returns None when no usage blocks are present, so callers can leave the token fieldsnull rather than record a confident zero.- CACHE WRITES (2026-09-05). This was ported verbatim from the AgentEO study's- swebench/agents.py, which read only `cached_tokens` and left cache_creation None on the- grounds that the API did not expose it. It does now --- `prompt_tokens_details.cache_creation_tokens` -- and because `prompt_tokens` counts- cache reads, cache writes and fresh input alike, folding the writes into- `input_tokens` priced them at 1.0x when the vendor bills 1.25x.-- Before the fix nearly all of what was booked as fresh input was cache writes, so- effective tokens were understated. That error was ONE-DIRECTIONAL and CLAUDE-vs-COPILOT ASYMMETRIC -- the Claude path records the split- correctly from the CLI's own result event -- so it made Copilot rows look cheaper than- Claude rows on identical work, biasing the exact comparison the sweep exists to make.-- `tokens_total` is untouched (it is prompt+completion either way); only the split moves.- A log with no `cache_creation_tokens` yields 0 and the old arithmetic exactly, so older- logs re-parse to their previous values. Stored rows written before this change still- carry the old split and are NOT comparable on cost to rows written after it -- they can- be corrected in place, because every run dir keeps the `copilot_debug.log` this parses.+ CACHE WRITES. The API reports them as `prompt_tokens_details.cache_creation_tokens`+ (the AgentEO study's swebench/agents.py, which this follows otherwise, reads only+ `cached_tokens`), and because `prompt_tokens` counts cache reads, cache writes and fresh+ input alike, folding the writes into `input_tokens` would price them at 1.0x when the+ vendor bills 1.25x. On a cache-heavy session nearly all of what would be booked as fresh+ input is cache writes, so effective tokens would be understated -- ONE-DIRECTIONALLY+ and CLAUDE-vs-COPILOT ASYMMETRICALLY, since the Claude path records the split from the+ CLI's own result event -- making Copilot rows look cheaper than Claude rows on+ identical work, which biases the exact comparison the sweep exists to make.++ `tokens_total` does not depend on the split (it is prompt+completion either way). A log+ with no `cache_creation_tokens` yields 0 and the AgentEO arithmetic exactly. Every run+ dir keeps the `copilot_debug.log` this parses, so a row's split can be recomputed from it."""key = '"usage": {'i, n = 0, len(log_text)diff --git a/experiments/te_harness/guards/net_guard.py b/experiments/te_harness/guards/net_guard.pyindex 734a973..db942a6 100644--- a/experiments/te_harness/guards/net_guard.py+++ b/experiments/te_harness/guards/net_guard.py@@ -3,23 +3,21 @@Web-Bench is a PUBLIC repository. `bytedance/web-bench` ships, for every project, the exactPlaywright spec each task is graded by -- so an agent that can reach GitHub is not solving-the task, it is reading the answer key. Measured 2026-09-07 in development runs: three-Web-Bench tasks curled their own grading spec-(`raw.githubusercontent.com/bytedance/web-bench/main/projects/<p>/test/task-N.spec.js`) and-two of them then passed.+the task, it is reading the answer key: each spec is one `curl` of+`raw.githubusercontent.com/bytedance/web-bench/main/projects/<p>/test/task-N.spec.js` away.-ALL THREE AGENTS SHARE THIS HOOK, and the payload spelling is no longer a guess. Codex's-`PreToolUse` contract was read out of the CLI's own embedded JSON schema+ALL THREE AGENTS SHARE THIS HOOK, and the payload spelling is read off each CLI, not+guessed. Codex's `PreToolUse` contract was read out of the CLI's own embedded JSON schema(`pre-tool-use.command.input` in the 0.153.1 binary): the required fields are `cwd`,`hook_event_name`, `model`, `permission_mode`, `session_id`, `tool_input`, `tool_name`,`tool_use_id`, `transcript_path`, `turn_id` -- i.e. `tool_name`/`tool_input`, Claude Code's-spelling exactly. So `hook_command` reads Codex payloads with the branch it already had.+spelling exactly. So `hook_command` reads Codex payloads with the Claude Code branch.ONE REFUSAL PROTOCOL, exit status 2 with the reason on stderr. Verified against all three-live CLIs -- Claude Code and Copilot on 2026-09-08, Codex on 2026-09-30.+live CLIs.CODEX CALLS IT TOO -- configured the way Codex reads hooks. Verified live on codex-cli 0.153.1-(2026-09-30, gpt-5.6-sol): with `{"hooks": {"PreToolUse": [{"matcher": "Bash", "hooks":+(gpt-5.6-sol): with `{"hooks": {"PreToolUse": [{"matcher": "Bash", "hooks":[{"type": "command", "command": "<python> net_guard.py"}]}]}}` in `$CODEX_HOME/hooks.json`and `--dangerously-bypass-hook-trust` on the command line, Codex refused `echohttps://github.com/probe` before running it ("Command blocked by PreToolUse hook: Blocked:@@ -27,39 +25,34 @@ https://github.com/probe` before running it ("Command blocked by PreToolUse hook`tool_input: {"command": "echo https://github.com/probe"}` -- a string, read by the firstbranch of `hook_command`.-The 2026-09-09 probes that concluded Codex ignores the hook got three things wrong, not the-CLI: they keyed the event kebab-case (`pre-tool-use`) where Codex wants PascalCase, gave the-handler an argv list where it wants a command string, and ran without the trust bypass ---and Codex skips every hook a person has not approved in /hooks, which an unattended run-cannot do. Rows stored before 2026-09-30 on the Codex lanes therefore carry-`net_guard_installed: false` and were guarded by the DNS layer below alone.+All three parts of that configuration are required: Codex keys the event PascalCase+(`PreToolUse`, not `pre-tool-use`), wants the handler as a command string rather than an argv+list, and skips every hook a person has not approved in /hooks -- which an unattended run+cannot do, hence the trust bypass.-WHY A HOOK AND NOT A TOOL MASK. `WebSearch`/`WebFetch` are masked out separately, but the-agent simply used `curl` inside `Bash` -- and `Bash` is in `PROTECTED_TOOLS`, so it can-never be dropped. The only place left to intervene is ahead of the shell.+WHY A HOOK AND NOT A TOOL MASK. `WebSearch`/`WebFetch` are masked out separately, but an+agent can run `curl` inside `Bash` just as well -- and `Bash` is in `PROTECTED_TOOLS`, so it+can never be dropped. The only place left to intervene is ahead of the shell.-WHY THIS PATTERN AND NOTHING WIDER. Derived from the traces, not from taste. Originally:-across 774 stored transcripts the agent made 335 network calls, 226 package-manager and 109-loopback to its own dev server, NONE mentioning GitHub, while every abusive call did.+WHY THIS PATTERN AND NOTHING WIDER. Derived from the traces, not from taste. Over 3,538+stored run dirs and their 558 network-shell calls, one case-insensitive substring separates+answer-key reads from everything else exactly:-RE-DERIVED 2026-09-09 on the full corpus -- 3,538 stored run dirs, 558 network-shell calls:-- 352 matched `github` (63%) every one of them reading a repository's source,- history, PR diffs or `.patch` files+ all matches read a repository's source, history, PR diffs or+ `.patch` files0 false positives no dependency install (pip 180, npx 14, yarn, npm)- was blocked+ and no loopback call to the agent's own dev server+ matches0 misses no call naming a `.patch`, a `/commit/`, a `/pull/`,a mirror (gitee/gitlab/codeload/jsdelivr) or a- benchmark repo escaped the rule+ benchmark repo falls outside the rule-So one case-insensitive substring still separates the two sets exactly, on a corpus 4.5x-larger than the one it was derived from. That is the whole argument for keeping it this-narrow: the alternative -- disabling web access outright -- also deletes the tool surface-that the untooling mechanism is measured on, and charges the baseline for a restriction the-cheat route never needed.+That is the whole argument for keeping it this narrow: the alternative -- disabling web+access outright -- also deletes the tool surface that the untooling mechanism is measured+on, and charges the baseline for a restriction the answer-key route does not need.This is deliberately NOT bullet-proof: an IP literal, a redirect, a mirror or a base64'd-URL all walk past both layers. It defends the pattern that actually occurred, and the row+URL all walk past both layers. It defends the plain-URL route, and the rowtelemetry is what makes a future model routing around it visible -- `net_guard_blocks`counts hook refusals, and `answer_key_attempts` counts answer-key-shaped commands found inthe transcript afterwards, which is agent-independent and therefore the number to watch on@@ -76,20 +69,20 @@ DENY = re.compile(r"github", re.I)# THE SAME POLICY, ENFORCED WHERE THE AGENT CANNOT SPELL AROUND IT.## The hook above is only as good as the CLI's cooperation, and each CLI cooperates on its own-# terms: a per-agent file, key and matcher, and on Codex a trust bypass as well. Each of those-# has failed silently at least once -- the Copilot lane's hook file sat in a directory its CLI-# never reads, and every Codex row before 2026-09-30 ran unhooked on a misread contract ---# while the provenance fields read healthy.+# terms: a per-agent file, key and matcher, and on Codex a trust bypass as well. Any one of+# those set wrongly leaves the hook unarmed while the provenance fields read healthy --+# Copilot reads its hook file only from `$COPILOT_HOME/settings.json`, and Codex skips a hook+# it has not been told to trust.## So the primary control is DNS, installed into the task container's /etc/hosts. It needs no-# cooperation from any CLI, is identical on all three agents, and catches every route the-# corpus actually used -- curl, wget, `git clone`, `python -c "import urllib.request"` ---# because all of them resolve a hostname first. Verified in a real SWE-bench container:+# cooperation from any CLI, is identical on all three agents, and catches every route that+# names a host -- curl, wget, `git clone`, `python -c "import urllib.request"` -- because+# all of them resolve a hostname first. Verified in a real SWE-bench container:# each of those fails with "Connection refused" afterwards, while pypi.org,# files.pythonhosted.org and a real `pip download` still work.#-# The host list is the `github` rule written out. Only hosts that appear in the measured-# abuse set, plus the sibling CDNs that serve the same content:+# The host list is the `github` rule written out: the GitHub hosts that serve repository+# pages, the API, archives, raw files, release objects, PR diffs and gists:ANSWER_KEY_HOSTS = ("github.com", "www.github.com", "api.github.com", "codeload.github.com","raw.githubusercontent.com", "objects.githubusercontent.com",@@ -133,7 +126,7 @@ def hook_command(payload):Claude Code {"tool_name": "Bash", "tool_input": {"command": ...}}Copilot CLI {"toolName": "bash", "toolArgs": {"command": ...}}- Captured off a live `preToolUse` hook on Copilot CLI 1.0.83, 2026-09-08.+ Captured off a live `preToolUse` hook on Copilot CLI 1.0.83.Reading only `tool_input` is the worst failure available here, which is why this is anamed function with its own test rather than an inline `.get`: the hook installs@@ -141,7 +134,7 @@ def hook_command(payload):returns not-blocked and it exits 0. Every provenance field reads healthy, the blockcount is 0 -- indistinguishable from "armed and never fired" -- and nothing is guarded.That is exactly the failure `net_guard_installed` and this block count exist to make- visible, so it must not be reintroduced by a key name.+ visible, so the key list below must carry every agent's spelling."""if not isinstance(payload, dict):return ""diff --git a/experiments/te_harness/swebench.py b/experiments/te_harness/swebench.pyindex 6cc55ca..7af8ba8 100644--- a/experiments/te_harness/swebench.py+++ b/experiments/te_harness/swebench.py@@ -41,7 +41,7 @@ resolving to whichever machine happened to write this file):(default $TOKENECONOMIZER_RESULTS/logs/optimizer_calls.jsonl;unset and with no results root, they are not recorded)TE_SWEBENCH_MAX_CPUS CPU cap for the task container, cores (default 16; 0 = no- cap, the pre-2026-09-05 behaviour). Also sets the matching+ cap). Also sets the matchingOMP/MKL/OPENBLAS/NUMEXPR/LOKY thread caps -- see _cpu_limit_argsTE_KEEP_CONTAINERS '1' leaves containers up for post-mortem (default: remove)TE_VERBOSE '1' prints per-task progress@@ -175,8 +175,7 @@ def guard_accepts(param):some versions and not in others. Passing a parameter it does not accept raises TypeErrorinside `_make_guard`/`run_task` -- i.e. at task 0, before a single row is written --so every optional guard parameter must be probed rather than assumed. Both domain- runners go through this one helper so the two cannot drift apart again; webbench.py- passed `never_abort` unconditionally and had exactly that failure mode.+ runners go through this one helper so the two cannot drift apart."""try:return param in inspect.signature(@@ -218,23 +217,21 @@ def _install_signal_cleanup():# --------------------------------------------------------------------------------------# The official images install the instance's dependencies into the `testbed` conda env and# activate it from /root/.bashrc. No lane here runs with HOME=/root, and no CLI's shell-# sources that file, so a bare `python` / `pytest` in the agent's shell resolved to the-# BASE interpreter (Python 3.11, no project deps) while TASK_TEMPLATE told the agent the-# env was active. A bare `python` then failed to import the project's own package, each-# failure costing a multi-step detour before the agent found-# /opt/miniconda3/envs/testbed/bin/python -- or gave up running tests.+# sources that file, so without activation a bare `python` / `pytest` in the agent's shell+# resolves to the BASE interpreter (Python 3.11, no project deps), which cannot import the+# project's own package, while TASK_TEMPLATE tells the agent the env is active.#-# The fix that works for EVERY shell shape (measured: `bash -lc`, `bash -c`, any HOME) is-# to prepend the env's bin dir to PATH in the agent PROCESS environment; the CLIs hand-# their environment to the shells they spawn. BASH_ENV hooks do not reach Claude Code's-# shell, so they are not used. Recorded on every row as `env_activation`, because rows-# taken before and after this change are not comparable on cost and must not be pooled.+# Prepending the env's bin dir to PATH in the agent PROCESS environment works for EVERY+# shell shape (measured: `bash -lc`, `bash -c`, any HOME); the CLIs hand their environment+# to the shells they spawn. BASH_ENV hooks do not reach Claude Code's shell, so they are+# not used. Recorded on every row as `env_activation`, because rows with and without it+# are not comparable on cost and must not be pooled._CONDA_ENVS_DIR = "/opt/miniconda3/envs"_PREFERRED_ENV = "testbed"def env_activation_enabled():- """TE_ENV_ACTIVATION=0 reproduces the pre-2026-09-11 (inactive-env) behaviour."""+ """TE_ENV_ACTIVATION=0 leaves the project env inactive (the image's own PATH)."""return os.environ.get("TE_ENV_ACTIVATION", "1") != "0"@@ -282,7 +279,7 @@ def optimizer_log_path():# made after task N is charged to task N, exactly as the ledger charges `optimizer_tokens`.# `agent` is part of it because the log is ONE file per results root while `cell`# (`<arm>/<repo>_s<seed>`) names no tree: a Claude lane and a Copilot lane over the same-# arm and repo collide on the key, and a per-cell groupby silently merged them.+# arm and repo collide on the key, and a per-cell groupby without `agent` would merge them._OPT_CTX = {}@@ -322,8 +319,8 @@ _OPT_KINDS = (# makes once per runner construction. It goes through the same seam, so it lands in# this log -- and it does so BEFORE any task, so `_OPT_CTX` is empty and its row also# carries cell=None. Labelling it is what lets a per-cell overhead groupby exclude it;- # left unlabelled it was the MAJORITY of a short run's log (measured 15 of 27) while- # reading as "unrecognised prompt", which is a different and much more alarming thing.+ # unlabelled, it can be most of a short run's log while reading as "unrecognised+ # prompt", which is a different and much more alarming thing.("preflight", "Reply with the single word: ok"),)@@ -440,13 +437,11 @@ def _import_swebench():# An agent that can browse is therefore not solving the task, it is retrieving the answer,# and the resolve verdict measures nothing.#-# This is NOT hypothetical and NOT historical. Measured 2026-09-07: across two live-# development trees, 32 tasks reached the network and 14 pulled-# something patch-shaped -- `sympy/pulls/16597/files`, `django/commit/2cbd3967.patch`,-# `sphinx-doc/commit/0eff5818.patch`, `svg-chart/test/task-4.spec.js` -- and 13 of those-# scored RESOLVED. The published claude-opus-4.8 trees are clean (0 of 800 transcripts),-# so this is a model-behaviour change, not a harness regression: the older model simply-# never reached for the network, and nothing ever stopped it.+# This is not hypothetical: given the tools, newer models do reach for the network on these+# tasks and pull something patch-shaped -- a fixing PR's files, a commit's `.patch`, a+# project's test spec -- and a patch retrieved that way grades RESOLVED like any other.+# Whether an agent browses is a matter of model behaviour, so only removing the tools+# makes the verdict independent of it.## Blocked for EVERY arm, deliberately. Leaving it to the untool mechanism would be worse# than useless: untool drops a tool only after observing DISUSE, so precisely the agents@@ -510,10 +505,10 @@ def _cpu_limit_args():An unlimited container sees every core on the HOST -- `nproc` inside one on a 384-corebox reports 384 -- and the agent's own verification step routinely runs the repo's test- suite, which for scientific repos is parallel by default. Measured here 2026-09-05:- one agent-issued `pytest -k "... or common"` on scikit-learn held **249 cores for- 183 s** (cpu-time/elapsed), peaking at 294 of 384. That is the agent checking its work,- not the grader, so it recurs on any task where the model picks a broad selector.+ suite, which for scientific repos is parallel by default: one agent-issued+ `pytest -k "... or common"` on scikit-learn holds ~250 cores for minutes+ (cpu-time/elapsed) on a 384-core host. That is the agent checking its work, not the+ grader, so it recurs on any task where the model picks a broad selector.`--cpus` ALONE IS NOT ENOUGH, and this is the part that is easy to get wrong: it capstotal CPU but does not change what `nproc` reports, so joblib would still size its@@ -521,7 +516,7 @@ def _cpu_limit_args():The thread caps are what actually bound the pool, and they must match the quota.Default 16: this harness is written for shared hosts, where an uncapped lane is- antisocial. `TE_SWEBENCH_MAX_CPUS=0` restores the previous unlimited behaviour. The+ antisocial. `TE_SWEBENCH_MAX_CPUS=0` removes the cap. Thevalue is recorded in run_config.json -> `cpu_limit` because it shapes wall-clock (and,via TE_TASK_TIMEOUT / TE_EVAL_TIMEOUT, can shape resolve)."""@@ -565,7 +560,7 @@ class ClaudeSwebenchRunner:# so a cheaper proposer would understate the overhead the arm actually incurs --# the economized arm would look better for having billed its own thinking at a# discount. One model per cell is also what the cell name claims. There is no knob:- # TE_OPTIMIZER_MODEL used to be one and is now refused rather than half-honoured.+ # TE_OPTIMIZER_MODEL is refused rather than half-honoured.if os.environ.get("TE_OPTIMIZER_MODEL"):raise RuntimeError("TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "@@ -583,7 +578,7 @@ class ClaudeSwebenchRunner:# An external tool scaffold (rtk / serena / repomap) -- a BASELINE, not a# mechanism. Resolved and preflighted here so a misconfigured one (no host cache# for rtk's binary, no aider for the repo map) kills the lane before task 0- # instead of writing 277 rows that merely look like a scaffold result.+ # instead of writing a whole stream of rows that merely look like a scaffold result.self.scaffold_name = scaffold or Noneself.scaffold = scaffolds_pkg.get(self.scaffold_name)if self.scaffold is not None:@@ -608,21 +603,20 @@ class ClaudeSwebenchRunner:# The mechanisms' judges answer with one JSON entry PER TOOL, which runs past a# small cap; a truncated answer is not a partial verdict but an unparseable one, so# the mechanism silently returns "drop nothing" while still billing for the call.- # 2048 sat right at the observed response length -- hence 8192.- # 8192 then truncated 7 of 40 manual-distill (injection) calls: the cap covers- # THINKING + response, and each of those spent all 8192 on thinking and emitted a- # ZERO-character answer -- so the manual paid full freight for nothing. Thinking is- # p50 ~4.8k but the observed distribution is CENSORED at the cap, so 16384 is a- # doubling with room for the JSON, not a measured ceiling; the truncation warning- # below is what says it was still not enough. Nudge-judge calls are unaffected- # (max 713 output tokens observed), so this only widens the injection path.+ # The cap covers THINKING + response, and a manual-distill (injection) call can+ # spend all of an 8192 cap on thinking and emit a ZERO-character answer, paying full+ # freight for nothing. Thinking is p50 ~4.8k but the observed distribution is+ # CENSORED at 8192, so 16384 is a doubling with room for the JSON, not a measured+ # ceiling; the truncation warning below is what says it is still not enough.+ # Nudge-judge calls stay far below it (max 713 output tokens observed), so the cap+ # binds only on the injection path.self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))self.task_timeout = int(os.environ.get("TE_TASK_TIMEOUT", "1800"))self.eval_timeout = int(os.environ.get("TE_EVAL_TIMEOUT", "1800"))self.keep = os.environ.get("TE_KEEP_CONTAINERS") == "1"- # Stated rather than left to run_config's getattr default: this harness has- # ALWAYS made its mechanism calls as plain Messages requests, and that is the- # transport the Copilot harness now matches. Naming it here means the field+ # Stated rather than left to run_config's getattr default: this harness makes its+ # mechanism calls as plain Messages requests, the transport the Copilot harness+ # uses too whenever its provider speaks it. Naming it here means the field# describes a decision instead of an absence.self.optimizer_transport = "api"self._preflight_optimizer_endpoint()@@ -650,17 +644,15 @@ class ClaudeSwebenchRunner:def _reap_exited(self, name):"""Remove EXITED containers from earlier attempts at this same cell.- Container names used to be one-per-(instance, arm), so `docker rm -f <name>` in- `_start` doubled as cleanup: a re-run of a cell collected whatever a previously- killed run had left behind. Names are now unique per task-run (see- `_container_name`), which is what makes two lanes safe -- but it also means that- exact-name removal no longer reaps anything, and a hard-killed driver would leak a- container per attempt.+ Names are unique per task-run (see `_container_name`), which is what makes two+ lanes on one cell safe -- but it also means the exact-name `docker rm -f <name>` in+ `_start` never reaps what a previously killed run left behind, and a hard-killed+ driver would leak a container per attempt.So reap by the readable stem instead, and ONLY containers that have exited. A- RUNNING container sharing this stem is the whole point of the rename: it belongs to- a live lane working the same cell, and force-removing it is precisely the bug this- replaced. Best-effort throughout -- failing to reap is untidy, not incorrect.+ RUNNING container sharing this stem belongs to a live lane working the same cell,+ and force-removing it would delete that lane's container mid-task. Best-effort+ throughout -- failing to reap is untidy, not incorrect."""stem = name.rsplit("-", 1)[0]try:@@ -756,9 +748,10 @@ class ClaudeSwebenchRunner:cmd += ["-v", spec]_LIVE_CONTAINERS.add(name)# Container start is the one step whose latency is set by the HOST, not by us: on- # a loaded shared box `docker run -d` can take many minutes, and a timeout here- # kills the whole lane mid-stream (the container then finishes starting and is- # orphaned). Configurable, and generous by default.+ # a loaded shared box `docker run -d` can take many minutes. A timeout here costs+ # the task (`run_task` records it unresolved, `container_start_failed`, and the+ # stream goes on) and can leave the container to finish starting on its own.+ # Configurable, and generous by default.p = _run(cmd + [image_for(instance_id), "-lc", "sleep infinity"],timeout=int(os.environ.get("TE_DOCKER_START_TIMEOUT", "900")))if p.returncode != 0:@@ -777,16 +770,13 @@ class ClaudeSwebenchRunner:# a host-side settings.json is never read by the CLI in the container.## WHY THIS AND NOT THE TOOL MASK. `network_tool_block()` drops WebSearch/WebFetch, but- # measured across 1,824 stored SWE-bench rows the shell was the BIGGER route: 120 rows- # reached the network through `Bash` against 47 through a web tool, and 16 rows fetched- # an answer-key-shaped URL (`.patch`, `/commit/`, `api.github.com/search/...`) of which- # 14 resolved -- one of them curling the instance's own fix commit as `/tmp/fix.patch`.+ # the shell is the BIGGER route to the network: an answer-key-shaped URL (`.patch`,+ # `/commit/`, `api.github.com/search/...`) is one `curl` away, no web tool needed.# `Bash` is in PROTECTED_TOOLS and can never be masked, so the hook is the only seam.# Derived from the AGENT PROFILE's home, never hardcoded: the Claude container runs- # HOME=/tmp/agent-home and the Copilot one HOME=/home/agent, and a literal path here- # meant the Copilot runner inherited a directory its `_start` never created --- # `docker cp` failed, and EVERY Copilot SWE-bench row read `net_guard_installed:- # false` (measured 0/60 armed across both stored copilot_swebench_* trees).+ # HOME=/tmp/agent-home and the Copilot one HOME=/home/agent, so a literal path here+ # would hand the Copilot runner a directory its `_start` never creates, and `docker cp`+ # would fail on every task.@propertydef NET_GUARD_SCRIPT(self):return f"{self.scaffold_profile.home}/net_guard.py"@@ -836,9 +826,9 @@ class ClaudeSwebenchRunner:# Registration succeeding says the config file was written or the plugin# installed -- neither of which means the CLI reads it, that the matcher name# is right, or that the guard can find the command in the payload it is handed- # (the two CLIs spell that differently -- guards/net_guard.hook_command). Each- # of those failed silently at least once while every provenance field read- # healthy, so the only trustworthy check is a real refusal.+ # (the two CLIs spell that differently -- guards/net_guard.hook_command). Any+ # of those can fail silently while every provenance field reads healthy, so the+ # only trustworthy check is a real refusal.self._verify_net_guard(name, pys[0])return Trueexcept Exception as e:@@ -854,7 +844,7 @@ class ClaudeSwebenchRunner:well -- and it sees only the shell tool. A layer that needs none of that is whatmakes the guard hold when one of those assumptions breaks with a CLI release.- This layer catches every route the stored corpus actually used, because all of them+ This layer catches every fetch route an agent has here, because all of themresolve a hostname first: `curl`, `wget`, `git clone`, and`python -c "import urllib.request"`. Verified in a real SWE-bench container: eachfails with "Connection refused" afterwards while pypi.org, files.pythonhosted.org@@ -1068,18 +1058,16 @@ class ClaudeSwebenchRunner:"""Remove the task container. NEVER raises -- cleanup is untidy, not incorrect.This is called from `run_task`'s `finally`, so anything it raises escapes the- per-task error handling and kills the whole STREAM. That is not hypothetical: on a- shared box at load ~96 with rootless Docker, `docker rm -f` exceeded the 120s- default and `subprocess.TimeoutExpired` propagated out of the finally, ending a- 60-task sympy lane at task 3 with 57 tasks unrun (2026-09-10).+ per-task error handling and kills the whole STREAM -- and on a loaded shared box+ with rootless Docker, `docker rm -f` can outlast a 120 s timeout.The same reasoning `_start` already carries -- "a host that cannot start containersmust not take the STREAM down with it" -- applies at least as strongly here, wherethe task is already graded and its row already built. A leaked container isreaped later by `_reap_exited`, so failing to remove one costs disk, not results.- The timeout is also generous now and configurable: removal on a loaded rootless- daemon is bounded by the HOST, not by us.+ The timeout is also generous and configurable (TE_DOCKER_STOP_TIMEOUT): removal on+ a loaded rootless daemon is bounded by the HOST, not by us."""if self.keep:_log(f"keeping container {name} (TE_KEEP_CONTAINERS=1)")@@ -1211,8 +1199,8 @@ class ClaudeSwebenchRunner:`session_output_tokens` deduplicates by message id, so double-counting is notpossible from the overlap.- Never raises and never fails a task: an unreadable log just means the row keeps the- old, honest `output_tokens=0` and stays flagged.+ Never raises and never fails a task: an unreadable log just means the row keeps an+ honest `output_tokens=0` and stays flagged."""home = self.scaffold_profile.hometry:@@ -1284,13 +1272,13 @@ class ClaudeSwebenchRunner:def _container_name(self, instance_id, arm, run_dir_path):"""Docker container name for ONE task-run. Must be unique across live lanes.- This was `te-<instance>-<arm>` and carried no agent, model or seed -- so a Claude- lane and a Copilot lane on the same (instance, arm) resolved to the same name, and- since `_start` opens with `docker rm -f <name>` the second lane DELETED the first- one's container mid-grade. Nothing reported it: the grading `docker exec` came back- empty, `eval.log` was written empty, and `get_eval_report` scored the empty log as+ `_start` opens with `docker rm -f <name>`, so a name carrying no agent, model or+ seed (`te-<instance>-<arm>`) would let a Claude lane and a Copilot lane on the same+ (instance, arm) resolve to one name, and the second lane would DELETE the first+ one's container mid-grade. Nothing would report it: the grading `docker exec` comes+ back empty, and `get_eval_report` scores an empty `eval.log` as`patch_successfully_applied: false` -- a row that reads like a bad model rather than- a lost container, on a lane that still exited 0.+ a lost container, on a lane that still exits 0.The run directory already encodes timestamp, stream index, agent, instance, arm,seed and mode, so a short digest of it is unique per task-run. The readable stem is@@ -1329,11 +1317,10 @@ class ClaudeSwebenchRunner:self._write_run_config(run_dir_path, task, arm, injected, exclude_tools,max_turns, run_kwargs, guard)- # A host that cannot start containers must not take the STREAM down with it.- # TimeoutExpired/RuntimeError from `docker run -d` used to propagate out of- # run_task and kill the launcher at whichever task it hit -- a wedged rootless- # daemon ended django s2 at task 0 of 40, and an intermittent 300s stall on- # xarray-6599 lost a whole baseline cell. Record the task as an unresolved,+ # A host that cannot start containers must not take the STREAM down with it: a+ # TimeoutExpired/RuntimeError from `docker run -d` propagating out of run_task would+ # kill the launcher at whichever task it hit, and a wedged rootless daemon or an+ # intermittent stall can hit any task. Record the task as an unresolved,# explicitly-flagged error and let the stream continue; a run where every task# carries start_error is a legible infrastructure failure, not a silent crash.net_guard_installed, net_guard_blocks = False, 0@@ -1363,7 +1350,7 @@ class ClaudeSwebenchRunner:# being replaced by it -- see `_install_net_guard`.net_guard_installed = self._install_net_guard(name)# The DNS layer, on every agent. Independent of the hook above and of each CLI's- # hook contract, which is per agent and has broken silently before.+ # hook contract, which is per agent and can break silently with a CLI release.hosts_blocked = self._install_answer_key_hosts(name)with open(transcript, "w", encoding="utf-8") as fh:result = self._segment(name, prompt, fh, exclude_tools, max_turns, guard,@@ -1382,11 +1369,10 @@ class ClaudeSwebenchRunner:# Patch extraction and grading get the SAME protection container start has,# and for the same reason: a host-side stall must not take the STREAM down.# `_graded_diff` is a `docker exec ... git diff` with a 300 s cap, and on a- # loaded shared box it does stall -- one did on 2026-09-13, at task 57 of a- # 60-task baseline, and the TimeoutExpired propagated out of run_task and ended- # the cell with four tasks unrun after 65 minutes of paid work. The agent's turn- # is already bought by this point, so the honest thing is to keep the row, say- # what happened on it, and let the stream continue: a task with+ # loaded shared box it can stall; a TimeoutExpired propagating out of run_task+ # would end the cell with every remaining task unrun. The agent's turn is+ # already bought by this point, so the honest thing is to keep the row, record+ # why it has no grade, and let the stream continue: a task with# `grading_error` is filterable, an ended stream is not recoverable.patch, resolved, report = "", False, {}try:@@ -1439,7 +1425,7 @@ class ClaudeSwebenchRunner:can be long, and the state file already holds its text."""# The SAME two-step the CLI invocation performs (see `_segment`): recomputing only- # the untool half here recorded `excluded_applied: []` on every row and made a+ # the untool half here would record `excluded_applied: []` on every row and make a# guarded run byte-identical to an unguarded one in its own provenance.drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]drop = drop + [t for t in network_tool_block() if t not in drop]@@ -1505,13 +1491,13 @@ class ClaudeSwebenchRunner:return False, {"empty_patch": True}script = self.grader.eval_script(instance_id)host_log = os.path.join(run_dir_path, "eval.log")- # A hung eval must not take the STREAM down with it. TimeoutExpired from these- # docker execs used to propagate out of run_task and kill the launcher mid-stream:- # scikit-learn-14710 ended the sklearn cell at task 21 of 32 TWICE (at eval_timeout- # 1800 and again at 3600), so that cell has never produced more than 20 graded rows.- # One pathological instance is a scored failure, not a reason to lose the other 11- # tasks behind it. Whatever the script printed before the cap is still written to- # eval.log, so the hang stays diagnosable.+ # A hung eval must not take the STREAM down with it: a TimeoutExpired from these+ # docker execs propagating out of run_task would kill the launcher mid-stream, and+ # an instance whose eval never terminates (scikit-learn-14710, see+ # workloads.SWEBENCH_EXCLUDED) would end its cell at any eval_timeout. One+ # pathological instance is a scored failure, not a reason to lose the tasks behind+ # it. Whatever the script printed before the cap is still written to eval.log, so+ # the hang stays diagnosable.try:w = self._exec(name, "cat > /tmp/eval.sh <<'TE_EOF'\n" + script + "\nTE_EOF\n""chmod +x /tmp/eval.sh")@@ -1530,9 +1516,9 @@ class ClaudeSwebenchRunner:_log(f"{instance_id}: EVAL TIMEOUT after {self.eval_timeout}s -> unresolved "f"(stream continues; see eval.log)")return False, {"eval_timeout": self.eval_timeout}- # An eval that produced NOTHING did not grade the patch -- it failed to run. The- # usual cause is that the container disappeared under us (see _container_name),- # and `docker exec` then exits nonzero with the reason on ITS stderr while the+ # An eval that produced NOTHING did not grade the patch -- it failed to run. A+ # container that disappeared under us (see _container_name) produces exactly this:+ # `docker exec` exits nonzero with the reason on ITS stderr while the# in-container `2>&1` captures nothing. Scoring that empty log would hand# get_eval_report a file with no APPLY_PATCH_PASS marker, which comes back as# `patch_successfully_applied: false, resolved: false` -- indistinguishable from a@@ -1555,8 +1541,8 @@ class ClaudeSwebenchRunner:def _eval_exec_failed(self, instance_id, host_log, out, rc, stderr, phase):"""Record a grading exec that never ran, loudly and distinguishably.- Returns the eval_report dict. `resolved` is still False -- but the row now says- the task was NOT GRADED rather than graded-and-missed, so aggregate.py and a+ Returns the eval_report dict. `resolved` is still False -- but the row says the+ task was NOT GRADED rather than graded-and-missed, so aggregate.py and ahuman reading the row can tell the two apart."""err = (stderr or "").strip()[-500:]@@ -1590,8 +1576,7 @@ def preflight_optimizer_endpoint(*, configured_model, label, remedy, api_key="",Shared by every runner whose mechanism calls use this wire -- the Claude runners, and aCopilot runner on an Anthropic-typed BYOK provider -- so the check cannot drift between- them; the two Claude runners went without it until 2026-09-07 and 83 rows were billed- on a 401 before anyone looked at the ledger.+ them.ONLY for an arm that will actually make such a call: `baseline`, `baseline-low`,`baseline-medium` and the scaffold arms make none, so probing for them would bill a call@@ -1794,14 +1779,15 @@ class _UsageRecovery:Claude Code reports its token breakdown ONLY in the stream's terminal`{"type":"result"}` event. Both break paths in `_segment` -- a spiral guard stop and a- TE_TASK_TIMEOUT -- kill the CLI mid-stream, so that event never arrives and the segment- returned `{}`. `_row_from_result({})` then produced a row of ZEROS: every token field- and `num_turns` recorded as 0 for a rollout that had really run for tens of turns and- hundreds of thousands of tokens.+ TE_TASK_TIMEOUT -- kill the CLI mid-stream, so that event never arrives. Read as `{}`,+ such a segment would make `_row_from_result` produce a row of ZEROS: every token field+ and `num_turns` 0 for a rollout that really ran for tens of turns and hundreds of+ thousands of tokens.- Because only economized arms install a guard, the understatement was ONE-DIRECTIONAL:- it made those arms look cheaper than they were and could never touch a baseline.- Grading was unaffected (that reads the patch), but turn-based claims were not.+ Because only economized arms install a guard, that understatement would be+ ONE-DIRECTIONAL: it would make those arms look cheaper than they are and could never+ touch a baseline. Grading is unaffected (that reads the patch), but turn-based claims+ are not.What is recoverable, and what is not:@@ -1821,7 +1807,7 @@ class _UsageRecovery:A recovered row is therefore honest but incomplete, and says so: `usage_incomplete` isset, with `output_tokens_missing` and `num_turns_recovered` naming which parts aresoft. Such a row is a LOWER BOUND on cost -- aggregations must not read it as exact,- and must never again read an unflagged zero as free.+ and must never read an unflagged zero as free."""def __init__(self):@@ -1849,8 +1835,8 @@ class _UsageRecovery:The flag is cleared only on PROOF of completeness: every message id the stream sawalso appears in the session log. Anything less stays flagged. Getting that backwards- would reintroduce the exact defect this class exists to prevent -- an unflagged row- that reads like a measurement and is not.+ would produce exactly what this class exists to prevent -- an unflagged row that+ reads like a measurement and is not."""if not self._by_msg:return {}@@ -1931,7 +1917,7 @@ def _row_from_result(result):"cost_usd_vendor": (result or {}).get("total_cost_usd"),# Degradation markers. Present and False on a healthy row, so "no flag" means# "not checked by this harness version" rather than "verified complete" -- and a- # zero-token row can never again be silently indistinguishable from a free one.+ # zero-token row is never silently indistinguishable from a free one."usage_incomplete": bool((result or {}).get("usage_incomplete")),"output_tokens_missing": bool((result or {}).get("output_tokens_missing")),"num_turns_recovered": bool((result or {}).get("num_turns_recovered")),@@ -1990,11 +1976,11 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):# and recording it as one makes pricing hard-fail at the END of a paid rollout.self.model = copilot.normalize_model(model or os.environ.get("COPILOT_MODEL"))copilot.validate_priceable(self.model, self.agent)- # Same rule as the Claude path: the optimizer is the agent's own model. It was- # already CALLED that way (optimizer_call substitutes self.model for the "agent"- # label), but it was RECORDED as "agent" -- so run_config.json described a cost- # basis the run did not use. A pinned id records as itself; an unpinned model is- # the CLI's own router, which is what the task ran on too, and records as "auto".+ # Same rule as the Claude path: the optimizer is the agent's own model, and it is+ # RECORDED as such -- optimizer_call substitutes self.model for the "agent" label,+ # so recording "agent" would describe a cost basis the run did not use. A pinned id+ # records as itself; an unpinned model is the CLI's own router, which is what the+ # task ran on too, and records as "auto".if os.environ.get("TE_OPTIMIZER_MODEL"):raise RuntimeError("TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "@@ -2052,9 +2038,8 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):self.cli_version = copilot.cli_version(self.copilot)self.grader = _Grader(grading_path)# An output cap applies on the API transport only -- `copilot -p` has no such flag,- # so on the CLI branch this stays a value nothing reads (it used to be omitted- # entirely for exactly that reason). Same default and same env var as the Claude- # harness, so the two agents' mechanism calls are capped alike.+ # so on the CLI branch this stays a value nothing reads. Same default and same env+ # var as the Claude harness, so the two agents' mechanism calls are capped alike.self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))self.task_timeout = int(os.environ.get("TE_TASK_TIMEOUT", "1800"))self.eval_timeout = int(os.environ.get("TE_EVAL_TIMEOUT", "1800"))@@ -2130,10 +2115,10 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):* **Which file.** `$COPILOT_HOME/settings.json`, which in the container is`/home/agent/.copilot/settings.json` (HOME is `/home/agent`, seeded per task- and thrown away with the container). The inherited implementation wrote- `.claude/settings.json`, which this CLI never reads -- so it succeeded,- reported armed, and guarded nothing. Note `hooks.json` at that level is- **not** read; only `settings.json` is.+ and thrown away with the container). This CLI never reads+ `.claude/settings.json`, so a guard written there would report armed and guard+ nothing. Note `hooks.json` at that level is **not** read; only+ `settings.json` is.* **The spelling.** `preToolUse` and matcher `bash` -- camelCase and lower-case,against Claude's `PreToolUse` and `Bash`, and a flat entry rather thanClaude's nested `hooks` list. A wrong matcher never fires.@@ -2275,9 +2260,9 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):# vendor cost stays null rather than a fabricated zero.row.setdefault("cost_usd_vendor", None)# Same degradation predicate as the Claude path, so one filter covers both agents.- # Copilot never needed _UsageRecovery -- it reads the whole log dir back AFTER the- # kill (`cat {logdir}/*.log`), which is why the audit found zero affected Copilot- # rows in 2,571. Its own failure mode is a log dir it could not parse, which+ # Copilot needs no _UsageRecovery -- it reads the whole log dir back AFTER the+ # kill (`cat {logdir}/*.log`), so a guard or timeout stop costs it no usage. Its+ # own failure mode is a log dir it could not parse, which# `row_usage` already reports as tokens_source="stream_output_only" with None (not# 0) in the unaccounted fields.src = row.get("tokens_source")@@ -2310,9 +2295,8 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):return row, out_dir-# Scaffolds whose mechanism this lane cannot deliver, and why. Refused at construction:-# each would install successfully and then do nothing, which reads as a scaffold result.-# Every scaffold is hostable on this lane, each through the channel Codex actually has:+# Every scaffold is hostable on the Codex lane, each through the channel Codex actually has+# -- none installs successfully and then does nothing, which would read as a scaffold result:# rtk -- its own Codex mode (`rtk init -g --codex`, instruction files under CODEX_HOME;# the hook mode is one this CLI ignores), measured by `rtk_compliance`;# repomap -- project instructions, which the profile declares as AGENTS.md;@@ -2320,8 +2304,6 @@ class CopilotSwebenchRunner(ClaudeSwebenchRunner):# than an argv flag: the profile's `mcp_args` is empty on purpose and# `_apply_scaffold` rewrites the config with the scaffold's spec. Measured by# `serena_tool_calls`, because a configured server is not a used one.-# A refusal table used to live here for rtk and serena; both refusals were about wiring this-# harness had not done, not about the tools.class CodexSwebenchRunner(ClaudeSwebenchRunner):@@ -2497,13 +2479,12 @@ class CodexSwebenchRunner(ClaudeSwebenchRunner):if p.returncode != 0:raise RuntimeError(f"docker run failed for {instance_id}: {p.stderr[-500:]}")# The agent home too: the net guard's script is `docker cp`'d into it, and docker cp- # into a directory that does not exist fails -- the way the Copilot lane once ran- # unguarded for a whole sweep.+ # into a directory that does not exist fails._run(["docker", "exec", name, "mkdir", "-p", codex.C_HOME, self.scaffold_profile.home])# The contract is written HERE, not as a side effect of arming the net guard.# `_install_net_guard` returns early when TE_ALLOW_NETWORK_TOOLS=1 lifts the guard,- # and with the write hanging off `_register_net_guard` that lane got NO config.toml- # at all -- no provider block, so every task failed to reach any endpoint. The+ # so a write hanging off `_register_net_guard` would leave that lane with NO+ # config.toml at all -- no provider block, so no task could reach any endpoint. The# guard's hook goes in hooks.json beside this file, never into it.self._write_config(name)self._activate_project_env(name)@@ -2558,11 +2539,10 @@ class CodexSwebenchRunner(ClaudeSwebenchRunner):PreToolUse hook: Blocked: this benchmark's grading specs are public on GitHub ..." --and logged the refusal, while a control command in the same turn ran normally.- The 2026-09-09 conclusion that Codex ignores the hook was a configuration error, not- the CLI: those probes keyed the event kebab-case (`pre-tool-use`; Codex reads- PascalCase `PreToolUse`, as Claude Code does), gave the handler an argv list rather- than a command string, and ran without the trust bypass. Codex rows stored before- 2026-09-30 therefore ran without the hook, guarded by the hosts block alone.+ Three details are load-bearing, and without any one of them the guard never runs:+ the event is keyed PascalCase `PreToolUse`, as Claude Code reads it (not kebab-case+ `pre-tool-use`); the handler is a command string, not an argv list; and the run+ passes the trust bypass.The rest is the shared lifecycle's: the script is copied in, the interpreter found,and the guard made to refuse its probe before the lane reports itself armed.@@ -2774,9 +2754,9 @@ class CodexSwebenchRunner(ClaudeSwebenchRunner):# Which runner each --agent label selects. A label with no entry is an error rather than a-# silent fallback: the old zero-arg factory always returned the Claude runner, so-# `--agent copilot` ran Claude Code and filed the rows under a copilot/ tree -- wrong-# numbers rather than a crash, which is the worst failure mode a benchmark can have.+# silent fallback: falling back to the Claude runner would run Claude Code under+# `--agent copilot` and file the rows under a copilot/ tree -- wrong numbers rather than+# a crash, which is the worst failure mode a benchmark can have.RUNNERS = {"claude": ClaudeSwebenchRunner,"copilot": CopilotSwebenchRunner,diff --git a/experiments/te_harness/webbench.py b/experiments/te_harness/webbench.pyindex 0793381..007c6ef 100644--- a/experiments/te_harness/webbench.py+++ b/experiments/te_harness/webbench.py@@ -277,47 +277,41 @@ def _ignored(rel, globs):# An agent that can browse is therefore not solving the task, it is retrieving the answer,# and the resolve verdict measures nothing.#-# This is NOT hypothetical and NOT historical. Measured 2026-09-07: across two live-# development trees, 32 tasks reached the network and 14 pulled-# something patch-shaped -- `sympy/pulls/16597/files`, `django/commit/2cbd3967.patch`,-# `sphinx-doc/commit/0eff5818.patch`, `svg-chart/test/task-4.spec.js` -- and 13 of those-# scored RESOLVED. The published claude-opus-4.8 trees are clean (0 of 800 transcripts),-# so this is a model-behaviour change, not a harness regression: the older model simply-# never reached for the network, and nothing ever stopped it.-## Blocked for EVERY arm, deliberately. Leaving it to the untool mechanism would be worse# than useless: untool drops a tool only after observing DISUSE, so precisely the agents# that browse keep the ability, and the arms would differ in whether they can cheat.# `TE_ALLOW_NETWORK_TOOLS=1` lifts it for a run that genuinely wants a browsing agent.+## The evaluator's `fetch` aborts a request after undici's default 300s headersTimeout-# (measured 2026-09-07 on node v22.23.2: UND_ERR_HEADERS_TIMEOUT at 300.9s). Past that the+# (on node v22.23.2: UND_ERR_HEADERS_TIMEOUT at 300.9s). Past that the# evaluator has already hung up, so our answer lands on a closed socket, the attempt is LOST# rather than merely slow, a retry is burned, and burning the last one ends the whole stream.# There is therefore NO Web-Bench configuration above ~300s that can help: a larger cap does# not buy the agent time, it only converts a slow task into a destroyed one.## So this CLAMPS rather than trusting the environment. `TE_TASK_TIMEOUT` is shared with-# SWE-bench, where 3600 is correct and necessary (sympy's slowest instances pinned 1800 dead-# on), and that value silently truncated two Web-Bench streams at task-14 before the clamp-# existed. `TE_WEBBENCH_TASK_TIMEOUT` overrides it for this domain only.+# SWE-bench, where 3600 is correct and necessary (sympy's slowest instances need the full+# 1800 and more), and on Web-Bench that value would lose attempts and truncate the stream.+# `TE_WEBBENCH_TASK_TIMEOUT` overrides it for this domain only.## The cap is COOPERATIVE -- checked between lines of the CLI's stream, so a turn already in# flight runs to completion. The ceiling is therefore `cap + longest single turn`, measured# at +16s and +21s on two Claude models. So the ceiling must sit well below# 300, not just under it: the default leaves ~60s for the overshoot and the hard max ~40s.# A 290 max would have allowed 290+21 = 311s, i.e. straight back over the wall.-# 2026-09-21: the "no configuration above ~300s can help" above is true of WEB-BENCH's-# config but NOT of Node's. `HttpAgent.request` (tools/http-agent) calls bare `fetch`, so it-# uses undici's GLOBAL dispatcher, and that dispatcher's headersTimeout is settable from-# outside the vendored benchmark via `--import` (webbench_undici_preload.mjs). Raising it is-# opt-in through `TE_WEBBENCH_UNDICI_ABORT`; unset, everything below is byte-identical to-# the old fixed 300/240/260. Rows taken at different ceilings are NOT comparable -- a run-# that raised it completes tasks a 300s run would have destroyed.+# The "no configuration above ~300s can help" above is true of WEB-BENCH's config but NOT+# of Node's. `HttpAgent.request` (tools/http-agent) calls bare `fetch`, so it uses undici's+# GLOBAL dispatcher, and that dispatcher's headersTimeout is settable from outside the+# vendored benchmark via `--import` (webbench_undici_preload.mjs). Raising it is opt-in+# through `TE_WEBBENCH_UNDICI_ABORT`; unset, the stock ceiling applies (abort 300s, default+# cap 240s, max cap 260s) and the evaluator is spawned unchanged. Rows taken at different+# ceilings are NOT comparable -- a run that raised it completes tasks a 300s run would+# have destroyed.WEBBENCH_UNDICI_ABORT_DEFAULT_S = 300WEBBENCH_TASK_TIMEOUT_HEADROOM_S = 40 # cooperative-cap overshoot (+21s worst) + marginWEBBENCH_TASK_TIMEOUT_DEFAULT = 240-# Back-compat names for the STOCK ceiling: the invariant they encode (cap < abort) is-# still the contract, it is now just evaluated per-run by the two helpers below.+# Names for the STOCK ceiling, kept for callers that import them: the invariant they+# encode (cap < abort) is the contract, evaluated per run by the two helpers below.WEBBENCH_UNDICI_ABORT_S = WEBBENCH_UNDICI_ABORT_DEFAULT_SWEBBENCH_TASK_TIMEOUT_MAX = (WEBBENCH_UNDICI_ABORT_DEFAULT_S- WEBBENCH_TASK_TIMEOUT_HEADROOM_S)@@ -632,7 +626,7 @@ class WebBenchRunner:# so a cheaper proposer would understate the overhead the arm actually incurs --# the economized arm would look better for having billed its own thinking at a# discount. One model per cell is also what the cell name claims. There is no knob:- # TE_OPTIMIZER_MODEL used to be one and is now refused rather than half-honoured.+ # TE_OPTIMIZER_MODEL is refused rather than half-honoured.if os.environ.get("TE_OPTIMIZER_MODEL"):raise RuntimeError("TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "@@ -669,15 +663,12 @@ class WebBenchRunner:# Output cap for the between-task proposer / judge / nudge-writer call, same# default as the SWE-bench harness so mechanism overhead is comparable across# domains. This is the CLAUDE base class and `optimizer_call` below reads it, so- # it must be set here: the Copilot subclass overrides it (that CLI has no output- # cap flag of its own), and the stale comment this replaces claimed the Copilot- # rationale for the base, leaving `self.optimizer_max_tokens` undefined. The- # effect was an AttributeError on the FIRST proposer tick -- task 5 of every- # stream -- so `inject-only` and `trimech-web` died early in each Claude Web-Bench- # cell, and no LIMIT<5 pilot could reach it.+ # it must be set here; the Copilot subclass sets its own (that CLI has no output+ # cap flag of its own). The first proposer tick is task 5 of a stream, so a pilot+ # shorter than five tasks never reaches this call.self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))- # Stated, not inferred: this harness has always used the Messages API for- # mechanism calls, which is the transport the Copilot harness now matches.+ # Stated, not inferred: this harness makes mechanism calls over the Messages API,+ # the same transport the Copilot harness uses.self.optimizer_transport = "api"# MUST stay under the evaluator's ~300s undici header timeout -- the same bound# CopilotWebBenchRunner documents, and it binds here for the identical reason.@@ -685,10 +676,7 @@ class WebBenchRunner:# so our reply lands on a closed socket (BrokenPipeError out of do_POST), the answer# is DISCARDED, and the task is scored a pass@1 miss for a plumbing reason. Worse,# it burns a retry -- and when it burns the LAST retry the evaluator ends the whole- # stream, so every later task comes back UNREACHED. Measured 2026-09-07 in a- # development run: threejs/task-14 2nd attempt clipped at- # Request 300.858s and took tasks 15-20 down with it; svg-chart hit the same wall- # from task-6 on. The old 1800s default predated a model slow enough to reach it.+ # stream, so every later task comes back UNREACHED.self.task_timeout = webbench_task_timeout()self.retry = max(1, int(os.environ.get("TE_WEBBENCH_RETRY", "2")))self.max_workers = int(os.environ.get("TE_WEBBENCH_MAX_WORKERS", "4"))@@ -928,12 +916,11 @@ class WebBenchRunner:other. Applies to EVERY arm: an agent that can read the grading spec makes theresolve verdict meaningless, which is not an arm-level concern.- **Returns a bool now, and the row records it.** It used to return a log path and- never raise, so the Copilot subclass -- which inherited it unchanged and therefore- wrote a Claude-shaped `.claude/settings.json` the Copilot CLI does not read --- failed *silently*: `os.makedirs` and the write both succeeded, nothing was armed,- and the row carried only `net_guard_blocks: 0`, which is indistinguishable from- "armed and never fired". Confirmed on a live copilot/calculator run 2026-09-08.+ **Returns a bool, and the row records it.** A successful write proves nothing: a+ Claude-shaped `.claude/settings.json` is a file the Copilot CLI never reads, so the+ write succeeds, nothing is armed, and `net_guard_blocks: 0` is indistinguishable+ from "armed and never fired". So each agent names its own settings path, hook key+ and entry shape, and the result is `_verify_net_guard`'s, not the write's."""path = self.net_guard_settings_path(home, run_dir_path)os.makedirs(os.path.dirname(path), exist_ok=True)@@ -969,7 +956,7 @@ class WebBenchRunner:payload spelling (`tool_input` vs `toolArgs`), the interpreter, or the scriptpath, and all three are exercised here. What this does NOT prove is that the CLIinvokes the hook at all; the file location, the hook key and the matcher name are- what carry that, and each is now set per agent rather than inherited.+ what carry that, and each is set per agent rather than inherited."""probe = json.dumps({"tool_name": "Bash", "toolName": "bash","tool_input": {"command": self.NET_GUARD_PROBE},@@ -1020,8 +1007,8 @@ class WebBenchRunner:Web-Bench's analogue of SWE-bench's `patch_excludes`: there the scaffold's filesare subtracted from the graded diff, here they must be kept out of the answer the- evaluator's Writer applies. With no scaffold this is empty and `_collect` behaves- byte-identically to the pre-scaffold form.+ evaluator's Writer applies. With no scaffold this is empty and `_collect` drops+ nothing on its account."""if self.scaffold is None:return frozenset()@@ -1073,10 +1060,11 @@ class WebBenchRunner:proc.stdin.close()result, stopped, deadline = {}, False, time.time() + self.task_timeout- # Same accounting hazard as the SWE-bench runner, and the same fix: the break paths- # below kill the CLI before its terminal `result` event. See- # swebench._UsageRecovery. It bites harder here -- the guard is re-made per ATTEMPT- # (`guard_per_attempt`), so there are more chances to stop mid-stream per task.+ # Same accounting hazard as the SWE-bench runner, handled the same way: the break+ # paths below kill the CLI before its terminal `result` event, so usage is recovered+ # from what streamed (swebench._UsageRecovery). It matters more here -- the guard is+ # re-made per ATTEMPT (`guard_per_attempt`), so there are more chances to stop+ # mid-stream per task.recovery, stop_reason = _UsageRecovery(), Nonefor line in proc.stdout:line = line.strip()@@ -1115,11 +1103,11 @@ class WebBenchRunner:if err.strip() and not result:_log(f"webbench agent stderr: {err.strip()[:300]}")if not result:- # Same fix as the SWE-bench runner: `output_tokens` survives the kill in the- # CLI's own session log. Here the agent runs on the HOST with $HOME relocated- # per task, so it is a plain read rather than a `docker exec` (swebench- # `_session_log_text`). It bites harder on this runner -- the guard is re-made- # per ATTEMPT -- so more rows per task stood to be flagged.+ # As in the SWE-bench runner, `output_tokens` survives the kill in the CLI's+ # own session log. Here the agent runs on the HOST with $HOME relocated per+ # task, so it is a plain read rather than a `docker exec` (swebench+ # `_session_log_text`). The guard is re-made per ATTEMPT on this runner, so more+ # rows per task are stopped mid-stream and depend on this read.result = recovery.synthesize(stop_reason or "no_result_event",_session_log_text_host(home))return result@@ -1149,7 +1137,7 @@ class WebBenchRunner:globs = _load_ignore_globs(self.repo, project_dir)supplied = set(request_files or {})# A scaffold's own artifacts are never part of the agent's answer. Empty with no- # scaffold, so the collected set is byte-identical to the pre-scaffold form. Only+ # scaffold, so nothing extra is dropped from the collected set. Only# files the REQUEST did not supply are dropped: if a project genuinely ships a file# by one of these names, it stays the agent's to edit.scaffold_files = self.scaffold_answer_excludes()@@ -1398,8 +1386,8 @@ class WebBenchRunner:# The agent-independent half of the same question, counted off the trace rather# than off the hook: `net_guard_blocks: 0` reads identically on an armed guard that# never fired and on a guard that was never registered, so a row needs both. It- # matters most on the Codex lane, where the hook is not honoured at all and- # `answer_key_hosts_blocked` is what does the guarding.+ # matters most on the Codex lane, where the hook is the second layer and+ # `answer_key_hosts_blocked` the first.row["answer_key_attempts"] = answer_key_attempts(run_dir_path)# Scaffold provenance (rtk_installed / rtk_hook_registered / scaffold_error / ...).# A failed install ran the task NATIVE, so these fields are what tells a real@@ -1455,8 +1443,8 @@ class WebBenchRunner:verbose=os.environ.get("TE_VERBOSE", "1") == "1")# `never_abort` is CONSERVATIVE-MODE-ONLY and absent from the vendored guard on# branches without trimech-safe -- see swebench.guard_accepts() for why it is- # probed. This used to pass it unconditionally, which made every guarded Web-Bench- # run on such a branch die with a TypeError at task 0.+ # probed. Passed to a guard that lacks the parameter, it would raise TypeError at+ # task 0 on every guarded Web-Bench run.if guard_accepts("never_abort"):guard_kwargs["never_abort"] = bool(run_kwargs.get("spiral_never_abort"))return NudgeGuard(**guard_kwargs)@@ -1470,9 +1458,9 @@ class WebBenchRunner:its transcript, and the evaluator's grading settings live in a config file the runwould otherwise not remember."""- # The SAME two-step the CLI invocation performs (see `_segment`): recomputing only- # the untool half here recorded `excluded_applied: []` on every row and made a- # guarded run byte-identical to an unguarded one in its own provenance.+ # The SAME two-step the CLI invocation performs (see `_segment`), so+ # `excluded_applied` includes the network block: recomputing only the untool half+ # would make a guarded run's provenance identical to an unguarded one's.drop = [t for t in (exclude_tools or []) if t not in self.protected_tools]drop = drop + [t for t in network_tool_block() if t not in drop]cfg = {@@ -1633,10 +1621,9 @@ class CopilotWebBenchRunner(WebBenchRunner):# and recording it as one makes pricing hard-fail at the END of a paid rollout.self.model = copilot.normalize_model(model or os.environ.get("COPILOT_MODEL"))copilot.validate_priceable(self.model, self.agent)- # Same rule as the Claude path: the optimizer is the agent's own model. It was- # already CALLED that way (optimizer_call substitutes self.model for the "agent"- # label), but it was RECORDED as "agent" -- so run_config.json described a cost- # basis the run did not use.+ # Same rule as the Claude path: the optimizer is the agent's own model, and it is+ # RECORDED as that model, not as the "agent" label optimizer_call substitutes it+ # for -- so run_config.json names the cost basis the run actually used.if os.environ.get("TE_OPTIMIZER_MODEL"):raise RuntimeError("TE_OPTIMIZER_MODEL is no longer honoured: the between-task optimizer "@@ -1687,10 +1674,10 @@ class CopilotWebBenchRunner(WebBenchRunner):self.scaffold.preflight(self.scaffold_profile_for("/nonexistent"))_log(f"scaffold: {self.scaffold.name} (external baseline)")# 16384, matching every other runner: one cap for the mechanisms' calls across- # both agents and both domains. It was 8192 here alone, which is a real hazard+ # both agents and both domains. A lower cap here alone would be a real hazard# rather than a tidiness point -- a cap too low truncates a judge's per-tool JSON# into an unparseable answer, which reads downstream as "no verdict" while still- # billing the call. Raising it can only remove truncations, never add them.+ # billing the call.self.optimizer_max_tokens = int(os.environ.get("TE_OPTIMIZER_MAX_TOKENS", "16384"))# Under the evaluator's ~300s undici header timeout, for the chain-integrity reason# documented on the Claude agent: a segment that outlives it breaks the stream.@@ -1739,10 +1726,9 @@ class CopilotWebBenchRunner(WebBenchRunner):return {"COPILOT_HOME": self.scaffold_copilot_home(run_dir_path)}# Copilot reads hooks from `$COPILOT_HOME/settings.json` -- NOT `$HOME/.claude/`, and- # not `hooks.json` (probed on CLI 1.0.83, 2026-09-08: a `hooks.json` at that level is- # ignored, a `settings.json` fires). Inheriting the Claude path wrote a file this CLI- # never reads, which is why every Copilot Web-Bench row was unguarded while reporting- # nothing wrong.+ # not `hooks.json` (on CLI 1.0.83 a `hooks.json` at that level is ignored, a+ # `settings.json` fires). The inherited Claude path names a file this CLI never reads,+ # so the path, the key and the entry shape are all overridden here.NET_GUARD_HOOK_KEY = "preToolUse"def net_guard_settings_path(self, home, run_dir_path):@@ -1789,13 +1775,13 @@ class CopilotWebBenchRunner(WebBenchRunner):net guard's `settings.json`. Run as-is, either would land in the operator's REALconfig and stay there after the sweep.- `COPILOT_HOME` is a separate knob from `$HOME`, so the fix is to point only that at- a per-task dir and copy in the small auth/config files (the 29MB of `session-state`- is disposable). The package cache stays where it is.+ `COPILOT_HOME` is a separate knob from `$HOME`, so only that is pointed at a+ per-task dir, with the small auth/config files copied in (the 29MB of+ `session-state` is disposable). The package cache stays where it is.Used on **every** arm, not only scaffolded ones -- the net guard is unconditional,- so there is no arm that leaves this dir untouched. (The name is historical: it was- scaffold-only when scaffolds were the only writer.)+ so there is no arm that leaves this dir untouched. (The name says scaffold, but the+ net guard writes here too.)"""dest = os.path.join(run_dir_path, ".copilot-home")if os.path.isdir(dest):@@ -1836,21 +1822,17 @@ class CopilotWebBenchRunner(WebBenchRunner):# deliberately left alone so the ~170MB package cache under $HOME/.cache/copilot is# not orphaned per task.#- # This used to be scaffold-only, on the reasoning that a no-scaffold arm writes- # nothing into the config dir and should stay byte-identical to the pre-scaffold- # form. The net guard ended that: it applies to every arm, it needs a hook in this- # file, and writing one into the operator's REAL ~/.copilot -- then leaving it- # there after the sweep -- is exactly the damage the scaffold branch existed to- # prevent. So the isolation is now unconditional, which is also one fewer- # behavioural difference between the arms.+ # Unconditional, not scaffold-only: the net guard applies to every arm and needs a+ # hook in this file, and writing one into the operator's REAL ~/.copilot -- then+ # leaving it there after the sweep -- is exactly the damage this isolation exists+ # to prevent. It is also one fewer behavioural difference between the arms.env["COPILOT_HOME"] = self.scaffold_copilot_home(os.path.dirname(workspace))# A scaffold installs its binary into the per-task bin dir, and on this lane the# AGENT's own shell has to find it -- not just the install step. rtk's hook rewrites# `ls -la` into `rtk ls -la` and hands that back to the CLI to run, so a bin dir# missing from PATH means every rewritten call dies `rtk: command not found`. That is- # not a silent no-op: reproduced 2026-09-13, the agent flails against the failing- # command (19 shell calls for one `ls`), which is the same shape as the 12x-cost- # Web-Bench incident in baselines/rtk._hook_binary. Harmless and empty with no+ # not a silent no-op: the agent retries against the failing command, many shell+ # calls for one `ls`, and the task's cost multiplies. Harmless and empty with no# scaffold. $HOME is still deliberately left alone (the package cache above).env["PATH"] = os.pathsep.join([self.scaffold_bin_dir(self.agent_home(os.path.dirname(workspace))),@@ -1967,9 +1949,11 @@ class CodexWebBenchRunner(WebBenchRunner):**The answer-key guard is the part that needed new machinery, and it is the reason thisclass exists rather than a two-line registration.** Web-Bench runs the agent on the HOSTwith no per-task container, so the other two agents are guarded by a PreToolUse hook --- which Codex provably ignores (measured on 0.153.1 with two live probes; see- `swebench.CodexSwebenchRunner._install_net_guard`). On SWE-bench the fallback is the- container's `/etc/hosts`; there is no container here. Two candidates were measured:+ which Codex runs only from its own hook file and only under+ `--dangerously-bypass-hook-trust` (see `_install_net_guard` below), so here it is the+ second layer, not the first. On SWE-bench the primary layer is the container's+ `/etc/hosts`; there is no container here, and of the two host-side ways to get the same+ block only one works:* **Codex's own sandbox** (`codex sandbox`) denies network exactly as wanted -- but on0.153.1 it also denies every write, and `sandbox_workspace_write.writable_roots` /@@ -1977,15 +1961,15 @@ class CodexWebBenchRunner(WebBenchRunner):`--strict-config` and as `-c` overrides. An agent that cannot write cannot do aWeb-Bench task, so this is unusable on this build.* **An unprivileged user+mount namespace** carrying the same `/etc/hosts` block the- SWE-bench container gets. Measured here: `github.com` resolves to 127.0.0.1 and curl+ SWE-bench container gets. Inside it `github.com` resolves to 127.0.0.1 and curlfails, `registry.npmjs.org` still resolves, writes still work, and files createdinside are owned by the real user outside. This is whatthe lane uses, so the Codex Web-Bench row is guarded by the same DNS layer as theCodex SWE-bench row -- and needs no cooperation from the CLI, which is the property- the hook lacked.+ no hook has.- `net_guard_installed` is therefore False on this lane, honestly, and- `answer_key_hosts_blocked` is the field that says the lane is guarded.+ A row records both layers: `net_guard_installed` for the hook, and+ `answer_key_hosts_blocked` for the namespace block, the primary one."""# Built once: `unshare` must exist and unprivileged user namespaces must be permitted,@@ -2045,8 +2029,7 @@ class CodexWebBenchRunner(WebBenchRunner):# Fields the INHERITED lifecycle reads. They are set here rather than left to the# base __init__ (which this class does not call, because it resolves the Anthropic# key) and a test pins the whole set, since the failure mode is an AttributeError- # raised after the evaluator has started and the first task has been paid for --- # which is exactly how the missing one was found, on a live run.+ # raised after the evaluator has started and the first task has been paid for.self.routing_base = self.routing["base_url"]self.optimizer_transport = "api" # codex.optimizer_call, always over /v1/responsesself.claude = None # no Claude binary on this lane@@ -2103,13 +2086,13 @@ class CodexWebBenchRunner(WebBenchRunner):return os.path.join(home, "hooks.json")# Set per task by `_install_net_guard`; `_segment` passes the trust bypass only when it- # is True, so an unarmed task runs exactly the argv it always did.+ # is True, so an unarmed task runs without it._net_guard_armed = Falsedef _install_net_guard(self, home, run_dir_path=None):"""The shared installer, pointed at Codex's hook file -- and Codex honours it.- Verified live on codex-cli 0.153.1 (2026-09-30): with this file and+ Verified live on codex-cli 0.153.1: with this file and`--dangerously-bypass-hook-trust` the guard refused a GitHub-bound command before itran and logged it, while a control command ran. Without the bypass Codex skips everyhook nobody has approved in /hooks, which an unattended run cannot do -- so arming@@ -2328,8 +2311,8 @@ class CodexWebBenchRunner(WebBenchRunner):row["turns_basis"] = "codex_steps"row["codex_config_capabilities"] = list(self._capabilities)row["codex_lean_instructions_tokens"] = int(self._lean_instructions_tokens or 0)- # This lane's honest guard fields: the hook is not honoured, the hosts block is.- row["net_guard_installed"] = False+ # Both layers as they were this task: the hook (`net_guard_installed`, which the+ # shared lifecycle set from whether it armed) and the namespace hosts block.row["answer_key_hosts_blocked"] = bool(self.answer_key_ns)if self.scaffold_name == "rtk":row.update(codex.rtk_compliance(os.path.join(out_dir, "transcript.jsonl")))@@ -2341,8 +2324,8 @@ class CodexWebBenchRunner(WebBenchRunner):# snapshots, session rollouts -- which is ~38 MB PER TASK-RUN and would put ~19 GB of# throwaway CLI state in the results tree over a five-project, five-seed sweep. The# SWE-bench lane has no equivalent problem because its CODEX_HOME lives inside the task- # container and dies with it; this restores that parity. TE_KEEP_WORKSPACES=1 keeps- # everything, the same knob that keeps the workspace itself.+ # container and dies with it; pruning here gives the host lane the same footprint.+ # TE_KEEP_WORKSPACES=1 keeps everything, the same knob that keeps the workspace itself._AGENT_HOME_KEEP = ("config.toml", "instructions.md", "AGENTS.md", "RTK.md")def _prune_agent_home(self, run_dir_path):diff --git a/experiments/tests/test_aggregate_index.py b/experiments/tests/test_aggregate_index.pyindex 3980a01..939a56b 100644--- a/experiments/tests/test_aggregate_index.py+++ b/experiments/tests/test_aggregate_index.py@@ -1,14 +1,11 @@"""The canonical index must describe the whole tree, or refuse to be written.`aggregate.py --arms` limits the REWRITE, not just the reading: both CSVs are written-whole on every invocation. Naming a subset while writing to `<root>/index` therefore-replaced an index describing the tree with one describing a subset, at the path-`compare_arms.py` reads, under a name that says nothing about being partial.--It is not a hypothetical. A stored Codex gpt-5.6-terra tree was found holding one arm-(`repomap`, n=3) of the eight in the tree, because the Codex lane runs one arm per-invocation and an `--arms` was passed once. Every cell was on disk; the index the-comparisons read said the tree contained three task-runs.+whole on every invocation. Naming a subset while writing to `<root>/index` would therefore+replace an index describing the tree with one describing a subset, at the path+`compare_arms.py` reads, under a name that says nothing about being partial -- with every+cell still on disk. The Codex lane runs one arm per invocation, which makes an `--arms`+an easy thing to pass, so the subset write to the canonical path is refused.These tests are offline: they build a tiny tree of synthetic `row.json` files and run`main()` against it.@@ -79,7 +76,7 @@ def test_whole_tree_is_the_default(tree):def test_subset_rewrite_of_the_canonical_index_is_refused(tree):- """The defect: `--arms repomap` used to leave the index holding only repomap."""+ """`--arms repomap` must not leave the canonical index holding only repomap."""_run([tree]) # a good index existsbefore = _cells(os.path.join(tree, "index"))diff --git a/experiments/tests/test_answer_key_guard.py b/experiments/tests/test_answer_key_guard.pyindex 7c3ccbf..d3e39fa 100644--- a/experiments/tests/test_answer_key_guard.py+++ b/experiments/tests/test_answer_key_guard.py@@ -6,20 +6,20 @@ worthless. That is not an arm-level concern, so the block applies to every arm i`baseline`.The policy is ONE case-insensitive substring, `github`, and its narrowness is the whole-argument. Re-derived 2026-09-09 over 3,538 stored run dirs / 558 network-shell calls: it-matched 352, every one of them reading a repository's source, history, PR diffs or `.patch`-files, with **0 false positives** (no `pip`/`npm`/`npx`/`yarn` install was blocked) and-**0 misses** (nothing naming a `.patch`, a `/commit/`, a `/pull/`, a mirror, or a benchmark-repo escaped it). The corpus in `_ABUSIVE` / `_LEGITIMATE` below is drawn from those calls-so a future widening or narrowing of the rule has to face them.+argument. Over 3,538 stored run dirs / 558 network-shell calls, every match reads a+repository's source, history, PR diffs or `.patch` files, with **0 false positives** (no+`pip`/`npm`/`npx`/`yarn` install matches) and **0 misses** (nothing naming a `.patch`, a+`/commit/`, a `/pull/`, a mirror, or a benchmark repo falls outside it). The corpus in+`_ABUSIVE` / `_LEGITIMATE` below is drawn from those calls so a future widening or narrowing+of the rule has to face them.It is enforced twice, because a hook is only as good as the CLI's cooperation:* the PreToolUse HOOK -- exit 2 + a reason on stderr. Honoured by all three CLIs, each onits own terms: Claude Code's settings.json, the Copilot CLI's, and Codex's`$CODEX_HOME/hooks.json` under `--dangerously-bypass-hook-trust` (verified live on- codex-cli 0.153.1, 2026-09-30; before that date the Codex lanes ran without it). It is- the layer that tells the MODEL why, which a DNS failure cannot.+ codex-cli 0.153.1). It is the layer that tells the MODEL why, which a DNS failure+ cannot.* the /etc/hosts BLOCK -- the answer-key hosts pointed at 127.0.0.1 inside the taskcontainer. Needs no cooperation from any CLI, so it is identical on all three lanes andholds when a CLI release breaks a hook contract.@@ -40,7 +40,8 @@ from te_harness.guards import net_guard_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))-# Real commands, taken verbatim (truncated) from the stored transcripts.+# Answer-key fetches the rule must block: patches, PR diffs, raw files, API searches, a+# clone and urllib, across the GitHub hosts (truncated)._ABUSIVE = ["timeout 60 curl -sSL https://github.com/django/django/commit/2cbd3967.patch -o /tmp/fix.patch","cd /tmp && curl -s -m 30 'https://patch-diff.githubusercontent.com/raw/scikit-learn/scikit-learn/pull/25102.diff'",@@ -51,7 +52,7 @@ _ABUSIVE = ["timeout 90 curl -s 'https://api.github.com/repos/sympy/sympy/pulls/17340/files?per_page=100'","curl -sL 'https://github.com/sympy/sympy/commit/9f4fb03c.diff' -o pr12419.diff",]-# Real legitimate network traffic from the same corpus. Blocking any of these would break a+# Legitimate network traffic the rule must let through. Blocking any of these would break a# lane rather than protect it._LEGITIMATE = ["pip install -e .",@@ -68,12 +69,12 @@ _LEGITIMATE = [@pytest.mark.parametrize("cmd", _ABUSIVE)-def test_every_observed_abuse_is_blocked(cmd):+def test_every_answer_key_fetch_is_blocked(cmd):assert net_guard.decide(cmd)[0] is True, cmd@pytest.mark.parametrize("cmd", _LEGITIMATE)-def test_no_observed_legitimate_call_is_blocked(cmd):+def test_no_legitimate_call_is_blocked(cmd):assert net_guard.decide(cmd)[0] is False, cmd@@ -96,7 +97,7 @@ def test_every_blocked_host_is_also_caught_by_the_shell_rule():assert net_guard.decide(f"curl https://{host}/x")[0] is True, host-def test_hosts_block_is_wellformed_and_covers_the_measured_abuse_set():+def test_hosts_block_is_wellformed_and_covers_the_answer_key_hosts():text = net_guard.hosts_block()assert text.startswith(net_guard.HOSTS_MARKER)assert text.endswith("\n")@@ -110,7 +111,7 @@ def test_hosts_block_is_wellformed_and_covers_the_measured_abuse_set():# 127.0.0.1, not 0.0.0.0: nothing listens on 443 in a task container, so the connection# is REFUSED at once instead of hanging until a timeout.assert set(entries.values()) == {"127.0.0.1"}- # The hosts that actually appeared in the 352 measured accesses.+ # The hosts that serve repository pages, the API, raw files, PR diffs and archives.for host in ("github.com", "raw.githubusercontent.com", "api.github.com","patch-diff.githubusercontent.com", "codeload.github.com"):assert host in entries, host@@ -196,10 +197,9 @@ def test_answer_key_attempts_is_none_when_there_is_no_transcript():def test_both_domains_count_answer_key_attempts_through_one_seam():"""Web-Bench rows carry it too, and off the same function.- Every Codex Web-Bench row before 2026-09-30 ran without the PreToolUse hook, guarded by- the namespace hosts block alone, so `answer_key_attempts` is the ONLY per-row leak- telemetry those rows have. It was absent from every Web-Bench row -- all three agents --- while the Codex docs told the reader to watch it.+ It is the agent-independent count, so on a row whose `net_guard_installed` is false --+ every Codex Web-Bench row, guarded by the namespace hosts block -- it is the ONLY+ per-row answer-key telemetry, and every Web-Bench row must carry it, all three agents."""from te_harness import swebench, webbenchassert webbench.answer_key_attempts is swebench.answer_key_attempts@@ -231,7 +231,7 @@ def test_the_merge_program_appends_once_and_keeps_what_was_there(tmp_path):def test_codex_arms_its_hook_in_codex_home_and_with_it_the_trust_bypass():- """Codex honours the hook in `$CODEX_HOME/hooks.json` -- verified live 2026-09-30 -- but+ """Codex honours the hook in `$CODEX_HOME/hooks.json` -- verified live on 0.153.1 -- butonly under --dangerously-bypass-hook-trust, so registering it is what turns the flag onfor every segment of the task, fresh and resumed alike."""from te_harness import codex, swebench@@ -268,11 +268,10 @@ def test_the_codex_config_declares_no_hooks_table():def test_stop_never_raises_when_docker_hangs(monkeypatch):"""`_stop` runs in run_task's `finally`, so anything it raises ends the whole STREAM.- Measured 2026-09-10: on a shared box at load ~96 with rootless Docker, `docker rm -f`- exceeded the 120 s default and the `TimeoutExpired` escaped the finally, ending a- 60-task sympy lane at task 3. The task was already graded and its row already built --- a container we failed to remove costs disk (and `_reap_exited` collects it later), not- results. Same rule `_start` already follows.+ On a loaded host with rootless Docker, `docker rm -f` can outlast its 120 s timeout, and+ a `TimeoutExpired` must not escape the finally: by then the task is graded and its row+ built, so a container that is not removed costs disk (and `_reap_exited` collects it+ later), not results. `_start` follows the same rule."""import subprocess as spfrom te_harness import swebenchdiff --git a/experiments/tests/test_baselines.py b/experiments/tests/test_baselines.pyindex 425875f..a020def 100644--- a/experiments/tests/test_baselines.py+++ b/experiments/tests/test_baselines.py@@ -324,10 +324,10 @@ def test_rtk_on_copilot_registers_through_the_hooks_dir_not_a_plugin(tmp_path, m$COPILOT_HOME/hooks/*.json -> {"tool_name": "Bash", "tool_input": {...}}$COPILOT_HOME/settings.json -> {"toolName": "bash", "toolArgs": {...}}- rtk parses only the first. The plugin this scaffold used to install delivered the- second, so rtk exited 0 with empty stdout on every call and rewrote nothing -- the arm- was native while every provenance field read healthy. Pin the channel, the document- shape rtk itself publishes, and the absolute command path."""+ rtk parses only the first. A hook handed the second (as a plugin hook is) exits 0 with+ empty stdout on every call and rewrites nothing -- the arm is native while every+ provenance field reads healthy. Pin the channel, the document shape rtk itself+ publishes, and the absolute command path."""monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))ctx = FakeCtx(COPILOT, tmp_path, exec_result=_copilot_exec)notes = scaffolds.get("rtk").setup(ctx)@@ -350,8 +350,8 @@ def test_rtk_on_copilot_registers_through_the_hooks_dir_not_a_plugin(tmp_path, mdef test_rtk_hook_effect_is_probed_not_assumed(tmp_path, monkeypatch):- """Registered and executable is still not REWRITING. The Copilot arm was silently- native for a week with both of those true, so the row carries the effect itself."""+ """Registered and executable is still not REWRITING. A hook that parses nothing leaves+ both of those true while the arm runs native, so the row carries the effect itself."""monkeypatch.setenv("TE_SCAFFOLD_CACHE", str(tmp_path / "cache"))ctx = FakeCtx(COPILOT, tmp_path, exec_result=_copilot_exec)notes = scaffolds.get("rtk").setup(ctx)@@ -360,8 +360,8 @@ def test_rtk_hook_effect_is_probed_not_assumed(tmp_path, monkeypatch):assert probe, "the hook must be fed a payload in the shape its channel delivers"assert '"tool_name"' in probe[0] and '"tool_input"' in probe[0]- # ... and a hook that parses nothing (empty stdout, rc 0 -- exactly the old failure)- # is caught and said out loud rather than reported as a healthy install.+ # ... and a hook that parses nothing (empty stdout, rc 0) is caught and said out loud+ # rather than reported as a healthy install.def mute(script):return _ok("") if "hook copilot" in script and "tool_input" in script \else _copilot_exec(script)@@ -460,7 +460,7 @@ def test_repomap_writes_the_map_and_records_its_size(tmp_path, monkeypatch):def test_repomap_preflight_demands_aider_on_the_host():"""The generator is a subprocess of this interpreter, so a missing aider must stop the- lane at construction rather than yield 277 rows of `map_ok: false`."""+ lane at construction rather than yield a lane of `map_ok: false` rows."""scaffold = scaffolds.get("repomap")try:import aider.repomap # noqa: F401@@ -475,10 +475,10 @@ def test_repomap_preflight_demands_aider_on_the_host():def test_repomap_preflight_names_the_interpreter_before_telling_you_to_install(monkeypatch):""""Install aider" is the WRONG advice for the likelier failure. The launchers run`${PYTHON:-python3}`, so a lane started from an unactivated shell picks up the system- Python even though the guide's `pip install -e '.[baselines]'` put aider in the venv.- Observed 2026-09-08: a detached lane died here on a host whose system Python carried- every other harness dependency, so only this one arm failed. The message must let that- reader diagnose it instead of reinstalling a package they already have.+ Python even though the guide's `pip install -e '.[baselines]'` put aider in the venv,+ and when that Python carries every other harness dependency this arm is the only one+ that stops. The message must let that reader diagnose it instead of reinstalling a+ package they already have."""real_import = builtins.__import__@@ -501,9 +501,9 @@ def test_repomap_generator_keeps_aiders_console_off_stdout(tmp_path):"""The scaffold takes the generator's stdout VERBATIM as the map body, and aider writesits progress line -- "Initial repo scan can be slow in larger repos, but only happensonce." -- to stdout whenever more than 100 files are uncached, which is every SWE-bench- repository on every task. Left as it was, that sentence opened every repo map an agent- read (seen live, 2026-09-13). The generator routes aider's console to stderr while the map- is built, so stdout is the map and nothing else."""+ repository on every task, so on stdout that sentence would open every repo map an agent+ reads. The generator routes aider's console to stderr while the map is built, so stdout+ is the map and nothing else."""pytest.importorskip("aider.repomap")pkg = tmp_path / "pkg"pkg.mkdir()diff --git a/experiments/tests/test_codex_routing.py b/experiments/tests/test_codex_routing.pyindex fe3fd0a..066f396 100644--- a/experiments/tests/test_codex_routing.py+++ b/experiments/tests/test_codex_routing.py@@ -4,8 +4,8 @@OPENAI_BASE_URL OpenAI's own regional endpoint, for a region-scoped key; the globalendpoint otherwise-Nothing else selects an endpoint. Pinned here, each of which failed silently at least once-while every launch banner stayed green:+Nothing else selects an endpoint. Pinned here, because a launch banner stays green when any+of them is wrong:* the RESOLUTION, including that no other variable moves the lane off the OpenAI API;* the REQUEST CONTRACT config.toml expresses, and that it carries the key by NAME only;@@ -85,8 +85,8 @@ def test_web_is_blocked_on_every_arm_including_an_empty_mask():top, tables = codex.config_toggles([])# The TOP-LEVEL spelling only. codex-cli 0.153.1 deprecated `[features].web_search`# ("web search is enabled by default; set `web_search` to disabled at the top level")- # and emits an error item into the transcript when it is present, so the redundant- # table entries were dropped -- see codex._TABLE_TOGGLES["web"].+ # and emits an error item into the transcript when it is present, so no table entry is+ # written -- see codex._TABLE_TOGGLES["web"].assert 'web_search = "disabled"' in topassert not any(t == "features" and "web_search" in line for t, line in tables)assert not any(t == "tools" and "web_search" in line for t, line in tables)@@ -138,8 +138,8 @@ def test_row_usage_decomposes_the_grand_input_total():"""`input_tokens` INCLUDES both cache fields; fresh input is what is left.Folding the cache fields into input_tokens instead would price cached re-reads at 1.0x- where the vendor bills 0.10x -- the same one-directional error that made Copilot rows- look cheaper than Claude rows before the cache-write fix.+ where the vendor bills 0.10x -- a one-directional error that would make this lane's rows+ look more expensive than the other lanes' on identical work."""row = codex.row_usage(_stream({"type": "thread.started", "thread_id": "th-1"},@@ -395,9 +395,9 @@ def _assert_reaped(pidfile):@pytest.mark.skipif(not os.path.isdir("/proc"), reason="needs /proc")def test_the_skills_probe_kills_the_cli_it_started_not_just_the_shim(tmp_path):- """`proc.kill()` reached only the Node shim; the native CLI it had started sat- "waiting for network" for good, one leaked per offline suite run (six alive on this host- on 2026-09-30). The probe now owns a process group and kills all of it."""+ """`proc.kill()` reaches only the Node shim, and the native CLI the shim started would+ sit "waiting for network" for good on an offline host. The probe owns a process group+ and kills all of it."""shim, pidfile = _fake_cli(tmp_path, ('mkdir -p "$CODEX_HOME/skills/.system/alpha"\n''touch "$CODEX_HOME/skills/.system/alpha/SKILL.md"\n'@@ -449,8 +449,7 @@ def test_lean_instructions_drops_only_the_pruned_sections_and_keeps_the_rest_verassert "# Working with the user" not in lean and "Channels." not in leanassert "When you run out of context, continue." in lean # compaction paragraph kept# the stopping rule is operational and is kept too: without it the agent has no- # statement of how a turn ends, and every untool cell ran more model calls than its- # baseline.+ # statement of how a turn ends, and runs more model calls than its baseline.assert "You end your turn by sending a final message." in leanfor kept in ("You are Codex.", "# Rules for getting work done\n\n- use rg","## Autonomy and persistence\n\nAct.", "# Using skills", "### How to use skills"):@@ -569,11 +568,10 @@ def test_mcp_tool_calls_counts_completed_items_and_keeps_a_sample_of_what_it_cancompletion, per server/tool; an item whose tool name sits somewhere unexpected is countedas unrecognised and ONE raw sample is kept.- The shape below is the one codex-cli 0.153.1 actually emits, observed live on 2026-09-13- (`item.completed`, item keys: id, type, server, tool, arguments, result, error, status) --- and the first such call on that lane was Codex's OWN `list_mcp_resources` on server- `codex`, which is exactly why the count is split per server: `mcp_tool_calls` includes it,- `serena_tool_calls` does not."""+ The shape below is the one codex-cli 0.153.1 emits (`item.completed`, item keys: id,+ type, server, tool, arguments, result, error, status). Codex also makes calls of its OWN,+ such as `list_mcp_resources` on server `codex`, which is why the count is split per+ server: `mcp_tool_calls` includes it, `serena_tool_calls` does not."""from te_harness import codeximport jsonp = str(tmp_path / "t.jsonl")@@ -600,11 +598,11 @@ def test_mcp_tool_calls_counts_completed_items_and_keeps_a_sample_of_what_it_candef test_the_codex_lane_hosts_every_scaffold_and_wires_an_mcp_one_through_the_config(tmp_path, monkeypatch):- """The refusal table is gone: rtk runs in its own Codex mode, repomap writes AGENTS.md,- and serena -- an MCP server -- is written into config.toml after setup, because `_start`- wrote the config before the scaffold existed. Both halves pinned: no scaffold name is- refused at construction, and a scaffold that returns `mcp_servers` causes exactly one- config rewrite carrying that spec."""+ """The Codex lane refuses no scaffold: rtk runs in its own Codex mode, repomap writes+ AGENTS.md, and serena -- an MCP server -- is written into config.toml after setup,+ because `_start` writes the config before the scaffold exists. Both halves pinned: no+ scaffold name is refused at construction, and a scaffold that returns `mcp_servers`+ causes exactly one config rewrite carrying that spec."""from te_harness import swebenchassert not hasattr(swebench, "_CODEX_UNAVAILABLE_SCAFFOLDS")monkeypatch.delenv("OPENAI_API_KEY", raising=False)diff --git a/experiments/tests/test_codex_webbench.py b/experiments/tests/test_codex_webbench.pyindex 8d60024..16b17a5 100644--- a/experiments/tests/test_codex_webbench.py+++ b/experiments/tests/test_codex_webbench.py@@ -6,8 +6,8 @@ have to be pinned here and nowhere else: the request contract must name HOST patcontainer path would silently name a file that does not exist), and the answer-key layersmust live in the per-task CODEX_HOME and a namespace, because there is no container to holdthem -- the PreToolUse hook in `$CODEX_HOME/hooks.json` (which Codex honours under-`--dangerously-bypass-hook-trust`, verified live 2026-09-30), and the hosts block, which-needs no cooperation from the CLI at all.+`--dangerously-bypass-hook-trust`, verified live on codex-cli 0.153.1), and the hosts block,+which needs no cooperation from the CLI at all."""import osimport subprocess@@ -36,7 +36,7 @@ def _stub(tmp_path, caps=(), ns=True):# ---- the answer-key layer ---------------------------------------------------------------def test_the_hook_is_armed_in_the_task_codex_home(tmp_path):"""`$CODEX_HOME/hooks.json`, Claude Code's shape, and only reported armed once the guard- has refused its probe -- the shared installer's rule, now on this lane too."""+ has refused its probe -- the shared installer's rule, on this lane as on the others."""import jsonr = _stub(tmp_path)home = tmp_path / "run" / ".codex-home"@@ -54,9 +54,25 @@ def test_the_hook_is_armed_in_the_task_codex_home(tmp_path):assert len(json.loads((home / "hooks.json").read_text())["hooks"]["PreToolUse"]) == 1+def test_a_row_records_the_hook_as_it_armed_and_the_hosts_block_beside_it(tmp_path,+ monkeypatch):+ """The hook is this lane's second layer and the namespace hosts block its first: a row+ records each as it was for the task, the hook's from the shared lifecycle."""+ r = _stub(tmp_path)+ r.scaffold_name = None+ monkeypatch.setattr(wb.CodexWebBenchRunner, "_prune_agent_home", lambda self, d: None)+ for armed in (True, False):+ monkeypatch.setattr(wb.WebBenchRunner, "run_task",+ lambda self, task, **kw: ({"net_guard_installed": armed},+ str(tmp_path)))+ row, _ = r.run_task({"id": "t"}, arm="baseline", injected=None, exclude_tools=None,+ run_dir_path=str(tmp_path))+ assert row["net_guard_installed"] is armed and row["answer_key_hosts_blocked"] is True++def test_only_an_armed_task_runs_with_the_trust_bypass(tmp_path, monkeypatch):"""Codex skips every hook nobody approved in /hooks, so an armed task must pass- --dangerously-bypass-hook-trust; an unarmed one keeps exactly the argv it always had.+ --dangerously-bypass-hook-trust; an unarmed one must run without it.Either way the hook's log path reaches the agent's environment."""seen = []@@ -215,8 +231,8 @@ def test_the_codex_controller_is_selected_not_the_generic_guard(tmp_path):def test_only_rtk_is_hostable_and_it_is_the_instruction_mode(tmp_path, monkeypatch):"""Web-Bench hosts one scaffold for any agent, rtk. On Codex that is rtk's INSTRUCTION- mode (files under CODEX_HOME), because the hook mode is one this CLI ignores. Anything- else is refused rather than installed and ignored."""+ mode (files under CODEX_HOME), the one `rtk init -g --codex` installs: rtk ships no hook+ for Codex. Anything else is refused rather than installed and ignored."""import pytestfor other in ("repomap", "serena"):with pytest.raises(RuntimeError, match="not wired for the Codex Web-Bench lane"):@@ -284,9 +300,8 @@ def test_the_codex_runner_sets_every_field_the_inherited_lifecycle_reads():"""The Codex runner does NOT call WebBenchRunner.__init__ -- that resolves the Anthropickey -- so every attribute the inherited methods read has to be set here by hand.- Pinned because of how the gap surfaces: not at construction, but as an AttributeError+ Pinned because a missing one surfaces not at construction, but as an AttributeErrordeep in `run_task`, after the evaluator has started and the first task has been paid for.- That is exactly how `routing_base` was found, on a live Web-Bench run."""import inspect, rebase = inspect.getsource(wb.WebBenchRunner)@@ -305,7 +320,7 @@ def test_the_per_task_codex_home_is_pruned_to_its_provenance(tmp_path):"""The CLI's own state is ~38 MB per task-run and would put ~19 GB of sqlite WALs,bundled skills and shell snapshots in the results tree over a five-project, five-seedsweep. The SWE-bench lane has no such problem -- its CODEX_HOME dies with the container --- so the host lane prunes to restore that parity, keeping the contract, which is provenance.+ so the host lane prunes to the same footprint, keeping the contract, which is provenance."""r = _stub(tmp_path)run_dir = tmp_path / "run"diff --git a/experiments/tests/test_envfacts_profiles.py b/experiments/tests/test_envfacts_profiles.pyindex 1cbfef6..efaeb0e 100644--- a/experiments/tests/test_envfacts_profiles.py+++ b/experiments/tests/test_envfacts_profiles.py@@ -121,9 +121,9 @@ def test_envfacts_alone_still_needs_no_runner(tmp_path):def test_codex_arms_carry_the_intended_profiles():"""Since 2026-09-12 the Codex injection DEFAULT is the composed profile.- The free layer's value on this benchmark came from the inactive-project-environment- defect; with that fixed it abstains, so `codex-inject` carries the paid proposer and- the LLM-free variant is the named ablation `codex-inject-envfacts`.+ With the project environment active on every lane the free layer mostly abstains, so+ `codex-inject` carries the paid proposer and the LLM-free variant is the named ablation+ `codex-inject-envfacts`."""from te_bench.arms import ARMS, ARM_ORDERfrom te_bench import mechanism_arms as cidiff --git a/experiments/tests/test_fleet_budget.py b/experiments/tests/test_fleet_budget.pyindex eb43308..664b0e2 100644--- a/experiments/tests/test_fleet_budget.py+++ b/experiments/tests/test_fleet_budget.py@@ -61,7 +61,7 @@ def test_a_report_reads_the_budget_back(tmp_path):def test_zero_leaves_the_cgroup_alone(tmp_path):- """An operator who wants the old unlimited behaviour must get it exactly, not a huge+ """An operator who wants no budget must get the unlimited cgroup exactly, not a hugenumber that still throttles."""base = _tree(tmp_path)r = _run(tmp_path, "0")diff --git a/experiments/tests/test_grading_integrity.py b/experiments/tests/test_grading_integrity.pyindex f2ba453..f1fde83 100644--- a/experiments/tests/test_grading_integrity.py+++ b/experiments/tests/test_grading_integrity.py@@ -1,20 +1,19 @@"""A row must never claim a grade the eval script did not produce.-Two defects sat behind one symptom here, and both were silent:+Two things must hold for a graded row to be trusted:- * The task container was named `te-<instance>-<arm>`, carrying no agent, model or seed,- while both SWE-bench runners open with `docker rm -f <name>`. Two lanes on the same- (instance, arm) -- exactly what you get running the Claude and Copilot lanes for- the same arm at once -- resolved to one name, and the second lane deleted the first+ * The task container's name is unique per task-run. Both SWE-bench runners open with+ `docker rm -f <name>`, so a name carrying no agent, model or seed would be shared by+ two lanes on the same (instance, arm) -- exactly what running the Claude and Copilot+ lanes for the same arm at once produces -- and the second lane would delete the firstone's container mid-grade.- * `_grade` then wrote an empty `eval.log` and handed it to `get_eval_report`, which- scored the absent APPLY_PATCH_PASS marker as `patch_successfully_applied: false,- resolved: false`. The lane exited 0. The row was indistinguishable from a model that- tried the task and missed.--So the container name must be unique per task-run, and a grading exec that did not run-must be reported as an absence rather than a miss -- and excluded from the denominator of-any resolve rate, because a task nobody evaluated is not a task the agent failed.+ * A grading exec that did not run is reported as an absence, not a miss. Handed an+ empty `eval.log`, `get_eval_report` scores the absent APPLY_PATCH_PASS marker as+ `patch_successfully_applied: false, resolved: false` on a lane that exits 0 -- a row+ indistinguishable from a model that tried the task and missed.++An ungraded row is also excluded from the denominator of any resolve rate, because a task+nobody evaluated is not a task the agent failed."""import importlib.utilimport os@@ -153,10 +152,10 @@ def test_stream_only_overhead_is_copied_into_the_row():"""The driver must lift overhead a mechanism reported on `srow` into `row`.`tooling.after_task` writes the per-task tool judge's spend to the STREAM row only,- and aggregate.py reads nothing but row.json -- so untool-only and every trimech- variant had that line item silently dropped from cost_effective_tokens. Measured on a- 6-task sympy cell: 11,244 tokens, 4.2% of the cell's reported cost, and one-directional- because a baseline runs no judge.+ and aggregate.py reads nothing but row.json -- so without the lift, untool-only and+ every trimech variant would leave that line item out of cost_effective_tokens. It is not+ small (11,244 tokens, 4.2% of a 6-task sympy cell's cost), and the omission is+ one-directional, because a baseline runs no judge."""from tokeneconomizer import ledgerrow = RX.lift_overhead({"tokens_total": 1000},@@ -172,7 +171,7 @@ def test_the_copy_never_clobbers_a_value_the_runner_already_set():def test_lift_overhead_covers_every_declared_overhead_key():- """Keyed off OVERHEAD_KEYS, so a new overhead field cannot reintroduce the gap."""+ """Keyed off OVERHEAD_KEYS, so a newly declared overhead field reaches row.json too."""from tokeneconomizer import ledgersrow = {k: 7 for k in ledger.OVERHEAD_KEYS}row = RX.lift_overhead({}, srow)@@ -189,12 +188,11 @@ def test_lift_overhead_ignores_non_overhead_stream_fields():def test_a_grading_stall_flags_the_row_instead_of_ending_the_stream():"""Patch extraction gets the guard container start already has.- `_graded_diff` is a `docker exec ... git diff` under a 300 s cap. On 2026-09-13 one- stalled at task 57 of a 60-task baseline and the TimeoutExpired propagated out of- `run_task`, ending the cell with four tasks unrun after 65 minutes of paid work -- the- exact failure the container-start guard exists to prevent, one step later in the same- function. The agent's turn is already bought by then, so the row must survive, say what- happened, and let the stream continue.+ `_graded_diff` is a `docker exec ... git diff` under a 300 s cap, and on a loaded shared+ box it can stall. A TimeoutExpired propagating out of `run_task` would end the cell with+ every remaining task unrun -- the failure the container-start guard exists to prevent,+ one step later in the same function. The agent's turn is already bought by then, so the+ row must survive, record why it has no grade, and let the stream continue."""import inspect, refrom te_harness import swebenchdiff --git a/experiments/tests/test_model_spelling.py b/experiments/tests/test_model_spelling.pyindex bd91187..bef0cb4 100644--- a/experiments/tests/test_model_spelling.py+++ b/experiments/tests/test_model_spelling.py@@ -129,10 +129,10 @@ def test_copilot_model_check_is_silent_on_a_served_id(stub_copilot):assert log.read_text().split() == ["claude-sonnet-5"]-def test_copilot_model_check_no_longer_false_passes_a_byok_404(stub_copilot):- """The check used to grep only the subscription's 'is not available'; a BYOK provider- says 'not found on provider ... (HTTP 404)', and the launch banner printed `serves`- for an id every task then 404'd on (measured 2026-09-12)."""+def test_copilot_model_check_refuses_a_byok_404(stub_copilot):+ """A BYOK provider says 'not found on provider ... (HTTP 404)', not the subscription's+ 'is not available', so the check must read both: on the first alone the launch banner+ would print `serves` for an id every task then 404s on."""env, _ = stub_copilotenv = dict(env, COPILOT_PROVIDER_BASE_URL="https://api.anthropic.com")p = _sh('te_check_copilot_model claude-sonnet-9 high', env)diff --git a/experiments/tests/test_optimizer_basis.py b/experiments/tests/test_optimizer_basis.pyindex 449e964..39d6167 100644--- a/experiments/tests/test_optimizer_basis.py+++ b/experiments/tests/test_optimizer_basis.py@@ -74,7 +74,7 @@ def test_no_runner_accepts_an_optimizer_model_argument():def test_the_old_env_knob_is_refused_not_read():- """TE_OPTIMIZER_MODEL used to select it. Silently ignoring a variable someone set is+ """TE_OPTIMIZER_MODEL is not honoured. Silently ignoring a variable someone set isworse than refusing it, so every mention must be inside a raise."""for fname in _HARNESSES:tree = _tree(fname)@@ -101,16 +101,15 @@ def test_the_driver_refuses_a_model_id_for_the_agent_label_flag():# ----------------------------------------------------------------------------------------# The mechanism call's TRANSPORT (2026-09-07).+# The mechanism call's TRANSPORT.## The MODEL is fixed by the assertions above -- the agent's own, so overhead is priced on-# the task rows' basis. The WIRE is not, and it was costing the Copilot lane an order of-# magnitude: `copilot -p` is an agent session that carries the CLI's system prompt and (until-# the --excluded-tools fix) its whole tool schema on every one-shot mechanism call. Measured-# on one 2,423-char judge prompt: 22,460 billed prompt tokens through the CLI vs 1,078 over-# the Messages API, which the Claude harness has used for these calls all along.+# the task rows' basis. The WIRE is not: `copilot -p` is an agent session that carries the+# CLI's system prompt on every one-shot mechanism call, which a bare request over the+# Messages API (what the Claude harness uses for these calls) does not. So a Copilot lane+# that can reach the Messages API sends its mechanism calls there.#-# So these guard the two things that must stay true of the new transport:+# These guard the two things that must stay true of that transport:# * it only ever points at an endpoint that bills WHAT THE TASKS BILL -- the lane's own# Anthropic-typed BYOK provider -- and# * on the GitHub subscription it never activates at all: the calls stay on the CLI.@@ -170,8 +169,8 @@ def test_a_subscription_lane_always_keeps_the_cli():def test_the_optimizer_call_excludes_tools_and_not_only_denies_them():"""--deny-tool withholds permission but leaves the schema in the prompt; only- --excluded-tools keeps it out. Passing just the first cost ~16,900 tokens of tool- schema on EVERY mechanism call, which the ledger charged to the arm."""+ --excluded-tools keeps it out. Passing just the first leaves ~16,900 tokens of tool+ schema on EVERY mechanism call, which the ledger charges to the arm."""copilot = importlib.import_module("te_harness.copilot")src = inspect.getsource(copilot.optimizer_call)assert "--excluded-tools" in src@@ -189,14 +188,14 @@ def test_both_harnesses_branch_on_the_endpoint_and_record_the_transport():# ----------------------------------------------------------------------------------------# WHO pays for the transport preflight (2026-09-07).+# WHO pays for the transport preflight.## The preflight exists because anthropic_optimizer_call never raises: a dead mechanism-# endpoint would leave an economized arm running as a baseline wearing its name. But it-# fired at every runner construction, including `baseline` -- which makes no mechanism call-# at all, so it was billing a probe it never needed and would refuse a lane over an endpoint-# it never touches. `config.needs_optimizer` is the gate, and these pin it to the facade so-# the two cannot drift.+# endpoint would leave an economized arm running as a baseline wearing its name. It runs+# only for an arm that makes mechanism calls: `baseline` makes none, so probing for it+# would bill a call it never needs and refuse a lane over an endpoint it never touches.+# `config.needs_optimizer` is the gate, and these pin it to the facade so the two cannot+# drift.# ---------------------------------------------------------------------------------------def test_needs_optimizer_agrees_with_the_facade_for_every_arm():"""The predicate must be true exactly when Economizer.__init__ would resolve a runner.@@ -263,8 +262,8 @@ def test_both_harnesses_gate_the_preflight_and_default_to_checking():def test_every_runner_caps_the_optimizer_call_at_the_same_number():"""One cap across both agents and both domains.- The Web-Bench Copilot runner sat at 8192 while the other three were 16384, which is a- hazard rather than an inconsistency: a cap too low truncates a judge's per-tool JSON+ A runner with a lower cap than the others is a hazard rather than an inconsistency: a+ cap too low truncates a judge's per-tool JSONinto an unparseable answer, and the mechanisms read that as "no verdict" while the callis still billed. So the cell pays for a judge whose verdict was thrown away, and onlythe truncation log says so. Pin them equal.@@ -286,7 +285,7 @@ def test_every_runner_caps_the_optimizer_call_at_the_same_number():# ---------------------------------------------------------------------------------------# WHERE the mechanism call goes, and WHICH CREDENTIAL it sends (2026-09-07).+# WHERE the mechanism call goes, and WHICH CREDENTIAL it sends.## Exactly one header, because anthropic_optimizer_call never raises: a request the endpoint# refuses arrives as ("", 0.0) == "no verdict", and the arm quietly runs its LLM-free path@@ -337,8 +336,8 @@ def test_a_byok_provider_gets_its_own_url_and_one_credential(monkeypatch, tmp_padef test_every_runner_preflights_the_mechanism_endpoint():- """All FOUR runners, not just the Copilot pair. The two Claude runners had no probe,- which is why a 401 could bill 83 rows before the ledger showed overhead of zero."""+ """All FOUR runners, not just the Copilot pair. A runner without the probe lets a+ refused credential (a 401) bill a lane's rows before the ledger shows overhead of zero."""for fname, classes in (("swebench.py", ("ClaudeSwebenchRunner", "CopilotSwebenchRunner")),("webbench.py", ("WebBenchRunner", "CopilotWebBenchRunner"))):tree = _tree(fname)diff --git a/experiments/tests/test_runner_registry.py b/experiments/tests/test_runner_registry.pyindex 7b5bafd..66e299e 100644--- a/experiments/tests/test_runner_registry.py+++ b/experiments/tests/test_runner_registry.py@@ -1,22 +1,21 @@"""Every runner in a RUNNERS registry must accept what its make_runner forwards.-This is the gap a two-lane blocker went through. `make_runner` fans out to a concrete-runner class:+`make_runner` fans out to a concrete runner class:RUNNERS[agent](agent=agent, scaffold=scaffold, **kwargs) # te_harness.swebench-and `CopilotSwebenchRunner.__init__` did not take `scaffold`, so every Copilot SWE-bench-cell raised TypeError at construction -- before task 0, for every arm including `baseline`-where the value is None. Two of the sweep's eight lanes could not run at all.+so a runner whose `__init__` does not take one of those keywords raises TypeError at+construction -- before task 0, for every arm on that agent, `baseline` included, where the+value is None. The whole agent's lanes cannot run.-`run_experiment.load_runner` already guards against exactly this, but it inspects the-FACTORY's signature, and a factory declared `def make_runner(agent=None, **kwargs)` accepts-anything by definition -- so the guard passes and the concrete class raises. The check has-to look at the classes the registry actually dispatches to.+`run_experiment.load_runner` guards against this too, but it inspects the FACTORY's+signature, and a factory declared `def make_runner(agent=None, **kwargs)` accepts anything+by definition -- so that guard passes while the concrete class raises. The check has to+look at the classes the registry actually dispatches to.Kept to `inspect`: no CLI, no Docker, no network, no credentials, so it runs anywhere the-rest of the suite does. That is the point -- the whole failure was a signature mismatch, and-the offline smoke test could not see it because RUNNER=stub never reaches these classes.+rest of the suite does. That is the point -- a signature mismatch is visible offline, and+the offline smoke test cannot see it because RUNNER=stub never reaches these classes."""import inspectdiff --git a/experiments/tests/test_spiral.py b/experiments/tests/test_spiral.pyindex abef81e..841f13e 100644--- a/experiments/tests/test_spiral.py+++ b/experiments/tests/test_spiral.py@@ -205,11 +205,10 @@ def test_defensive_profile_is_selectable_and_unpaid(tmp_path):def test_both_guards_expose_everything_the_shared_task_loop_reads():"""The Codex guard is driven by the SAME `run_task` loop as the generic one.- On 2026-09-12 that loop's `_log(... guard.segment)` crashed the codex-spiral lane on its- very first nudge: `CodexResumeGuard` had no `segment`. Nothing had caught it because the- only live Codex nudges on record were produced by a different harness. Rather than pin- one name, read the attributes the loop actually touches out of its source and require- both guard classes to carry them.+ The loop logs `guard.segment` on every nudge (`_log(... guard.segment)`), so the guard+ must expose it, and every other attribute the loop reads. Rather than pin one name,+ read the attributes the loop actually touches out of its source and require both guard+ classes to carry them."""import inspect, refrom te_harness import swebench as sw@@ -236,11 +235,10 @@ def test_both_guards_expose_everything_the_shared_task_loop_reads():def test_the_codex_envelope_is_advanced_in_the_unit_its_trigger_compares(tmp_path):"""`CodexEnvelopeTracker` is documented in tool events and the guard's trigger compares- `state.turns` (command + edit). The stage used to feed it the row's `num_turns`, which- for Codex is agent STEPS (messages + tool calls) and runs ~1.45x larger -- measured on- the 2026-09-12 baselines, that raised the gate ~45% and left only 3 of 60 scikit-learn- tasks able to reach it, against 10 of 60 at the documented unit. The controller ran and- could not fire.+ `state.turns` (command + edit). The row's `num_turns` is, for Codex, agent STEPS+ (messages + tool calls) and runs ~1.45x larger, so an envelope learned from it sits+ ~45% above the count it is compared to and few tasks ever reach the gate. The stage+ must feed the tracker the trigger's own unit."""from tokeneconomizer.config import SpiralConfigfrom te_bench.spiral_stage import SpiralStagediff --git a/experiments/tests/test_sweep_arms.py b/experiments/tests/test_sweep_arms.pyindex 215a0fd..d87c4b4 100644--- a/experiments/tests/test_sweep_arms.py+++ b/experiments/tests/test_sweep_arms.py@@ -253,10 +253,9 @@ def test_group_sizes_match_the_manifest_the_driver_filters_to():names. If those two numbers disagree by even one, SKIP_DONE never fires for that groupand every resume silently re-runs completed, already-paid-for cells.- This is not hypothetical: `workloads.task_counts()` is the PROVENANCE record and- deliberately does not subtract SWEBENCH_EXCLUDED, so reading it here made the planner- expect 32 scikit-learn tasks for a run that executes 60 (31 published-minus-- excluded, plus the 29-id de-saturation extension).+ `workloads.task_counts()` is not that number: it is the PROVENANCE record and+ deliberately neither subtracts SWEBENCH_EXCLUDED nor adds SWEBENCH_ADDED, so a group+ with an exclusion or a de-saturation extension runs a different count than it reports."""import jsonfor domain in SP.SWEEP_ARMS:@@ -380,9 +379,10 @@ def test_the_codex_spiral_arms_have_the_shape_the_injection_arms_have():def test_the_codex_family_relates_to_codex_trimech_as_the_generic_one_does_to_trimech():"""The whole point of the Codex arms is that they are the SAME experiment on a secondagent, so the two families must have the same shape -- not merely each be internally- consistent. Checked property by property rather than by eye, because the asymmetry that- actually occurred was invisible in the configs: every structural check passed while the- plain name `codex-spiral` carried a spiral `codex-trimech` did not compose.+ consistent. Checked property by property rather than by eye, because an asymmetry+ between the families is invisible in the configs: every other structural check passes+ when a plain single-mechanism name such as `codex-spiral` carries a mechanism setting+ its trimech does not compose."""GEN = ("trimech", "inject-only", "untool-only", "spiral-only")CDX = ("codex-trimech", "codex-inject", "codex-untool", "codex-spiral")diff --git a/experiments/tests/test_tooling.py b/experiments/tests/test_tooling.pyindex b320013..cc361fd 100644--- a/experiments/tests/test_tooling.py+++ b/experiments/tests/test_tooling.py@@ -108,18 +108,14 @@ def test_evidence_untool_drops_after_streak(tmp_path):# ---- the mask must be EARNED, never a prior ----------------------------------# The stored Web-Bench arms were produced by an older `--untool` flag that applied-# `untool_exclusions()` -- the static INAPPLICABLE_TOOLS set -- from the FIRST-# rollout of the stream. Measured consequences: the mask was live at the `init` task before-# any task had completed, it cost zero LLM tokens so the honest ledger charged nothing for-# it, and the keep-set was byte-identical across all 15 cells (5 projects x 3 seeds). The-# stored Web-Bench cost win is therefore largely an unpaid, unlearned prior rather than the-# evidence-gated mechanism, and it is not comparable to the SWE-bench arms.-#-# The current code does not do this: the static surface is still exported on the adapters but-# no longer reachable from the stage. These tests pin that shut, in BOTH domains, so it-# cannot regress silently -- the failure mode is invisible in aggregate numbers and only-# shows up as a suspiciously large cost win.+# The mask comes only from observed disuse: none before the first task has completed, and+# never one drawn from the adapters' static `untool_exclusions()` / INAPPLICABLE_TOOLS+# surface, which is still exported but not reachable from the stage. A static prior would be+# live from the first rollout, cost zero LLM tokens (so the honest ledger would charge nothing+# for it) and give a byte-identical keep-set in every cell -- an unpaid, unlearned saving+# rather than the evidence-gated mechanism. These tests pin that in BOTH domains, because the+# difference is invisible in aggregate numbers and shows up only as a suspiciously large cost+# win.from te_bench.adapters import WebBenchAdapter # noqa: E402from tokeneconomizer.toolprune import tools as reflect # noqa: E402from fixtures import write_claude_transcript_with_catalog # noqa: E402@@ -146,14 +142,13 @@ def _finish(stage, tmp_path, i, used, catalog=_CATALOG):def test_first_task_of_a_stream_is_never_masked_swebench(tmp_path):- """No evidence can exist before the first task, so the agent must see the full catalog.- This is the property the stored Web-Bench runs violated."""+ """No evidence can exist before the first task, so the agent must see the full catalog."""stage = _stage(tmp_path, SwebenchAdapter(tasks_path=None))assert stage.build_tools({"instance_id": "w-0"}, 0) == []def test_first_task_of_a_stream_is_never_masked_webbench(tmp_path):- """Same guarantee on the Web-Bench adapter -- the domain where it actually broke."""+ """Same guarantee on the Web-Bench adapter."""stage = _stage(tmp_path, WebBenchAdapter("zustand"))assert stage.build_tools({"instance_id": "task-1"}, 0) == []@@ -174,8 +169,8 @@ def test_mask_appears_only_after_the_catalog_is_learned(tmp_path):def test_static_inapplicable_prior_is_not_applied(tmp_path):"""An adapter that declares EVERYTHING inapplicable must not change the mask.- This is the specific regression: the stage must consult observed usage, never the- adapter's static `inapplicable_tools` / `hardcoded_exclusions` surface.+ The stage must consult observed usage, never the adapter's static+ `inapplicable_tools` / `hardcoded_exclusions` surface."""class _GreedyPrior(WebBenchAdapter):def inapplicable_tools(self):@@ -193,16 +188,16 @@ def test_static_inapplicable_prior_is_not_applied(tmp_path):def test_untooling_is_off_entirely_without_an_evidence_threshold(tmp_path):- """`evidence=0` (the old bare `--untool`) must mean NO mask, not a static one."""+ """`evidence=0` must mean NO mask, not a static one."""stage = _stage(tmp_path, WebBenchAdapter("zustand"), evidence=0)_finish(stage, tmp_path, 0, ["Bash"])assert stage.build_tools({"instance_id": "task-2"}, 1) is None# ---- network tools are blocked on every arm ----------------------------------------------# Both benchmarks' answers are on the public internet, and on 2026-09-07 the live opus-5 /-# sonnet-5 trees were caught fetching gold patches and grading specs. These pin the block so-# it cannot be quietly dropped, and pin the escape hatch so it stays deliberate.+# Both benchmarks' answers -- gold patches, grading specs -- are on the public internet, a+# fetch away from the agent. These pin the block so it cannot be quietly dropped, and pin the+# escape hatch so it stays deliberate.def test_network_tools_are_blocked_by_default(monkeypatch):from te_harness import webbench as wb, swebench as sb@@ -226,11 +221,11 @@ def test_network_tools_are_not_protected():# ---- the Web-Bench PreToolUse network guard -----------------------------------------------# Derived from the traces, not from taste: across 774 stored transcripts the agent made 335-# network calls (226 package-manager, 109 loopback) and NONE mentioned GitHub, while every-# observed abuse did. These pin both halves of that separation.+# Derived from the traces, not from taste: across 774 stored transcripts the network calls a+# task needs (335: 226 package-manager, 109 loopback) never mention GitHub, and every route+# to the benchmark's answer key does. These pin both halves of that separation.-def test_net_guard_blocks_the_observed_abuse():+def test_net_guard_blocks_answer_key_fetches():from te_harness.guards.net_guard import decidefor cmd in ("curl -sL https://raw.githubusercontent.com/bytedance/web-bench/main/"@@ -267,7 +262,7 @@ def test_net_guard_reads_both_agents_payload_spellings():def test_net_guard_blocks_end_to_end_on_a_copilot_shaped_payload():- """The regression that matters: rc=2 on the Copilot spelling, not just the Claude one."""+ """End to end, the hook exits rc=2 on the Copilot spelling, not just the Claude one."""import json, subprocess, sysroot = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))env = {k: v for k, v in os.environ.items() if k != "TE_ALLOW_NETWORK_TOOLS"}@@ -301,11 +296,11 @@ def test_net_guard_fails_open_on_unreadable_input(tmp_path):# ---- the guard is installed on BOTH harnesses, not just Web-Bench -------------------------# Masking WebSearch/WebFetch never covered the route that mattered on SWE-bench: measured-# across 1,824 stored rows, 120 reached the network through `Bash` against 47 through a web-# tool, and 16 fetched an answer-key-shaped URL (14 of them resolved). `Bash` is protected-# and can never be masked, so the PreToolUse hook is the only seam -- and for SWE-bench it-# has to be installed INSIDE the task container, where the agent actually runs.+# Masking WebSearch/WebFetch does not cover the agent's main route to the network on+# SWE-bench, which is `Bash` (across 1,824 stored rows, 120 reached the network through it+# against 47 through a web tool). `Bash` is protected and can never be masked, so the+# PreToolUse hook is the only seam -- and for SWE-bench it has to be installed INSIDE the+# task container, where the agent actually runs.def _method_calls(cls_name, method, path="te_harness/swebench.py"):"""The attribute names `cls_name.method` calls on self, via AST (no Docker needed)."""@@ -335,8 +330,8 @@ def test_the_swebench_guard_honours_the_same_escape_hatch(monkeypatch):def test_the_recorded_tool_mask_is_the_one_the_cli_was_given(monkeypatch):- """`run_config.tools.excluded_applied` used to omit the network block on every row,- which made a guarded run byte-identical to an unguarded one in its own provenance."""+ """`run_config.tools.excluded_applied` must include the network block: without it a+ guarded run is byte-identical to an unguarded one in its own provenance."""import astroot = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))for path in ("te_harness/swebench.py", "te_harness/webbench.py"):diff --git a/experiments/tests/test_usage_recovery.py b/experiments/tests/test_usage_recovery.pyindex c7e8bb3..f6d248b 100644--- a/experiments/tests/test_usage_recovery.py+++ b/experiments/tests/test_usage_recovery.py@@ -1,11 +1,12 @@-"""The accounting fix: a rollout killed mid-stream must not be recorded as free.+"""Usage recovery: a rollout killed mid-stream must not be recorded as free.Claude Code reports its token breakdown only in the stream's terminal `result` event, andboth of `_segment`'s break paths -- a spiral guard stop and TE_TASK_TIMEOUT -- kill the CLI-before it arrives. The segment used to return `{}`, so `_row_from_result` wrote a row of-zeros for a rollout that had really burned hundreds of thousands of tokens over tens of-turns. Only economized arms install a guard, so the understatement was one-directional and-flattered exactly the arms under test.+before it arrives. A segment that returned `{}` would make `_row_from_result` write a row of+zeros for a rollout that really burned hundreds of thousands of tokens over tens of turns,+so the segment rebuilds its usage from the stream it did see. Only economized arms install+a guard, so the understatement would be one-directional and flatter exactly the arms under+test.These tests are offline: they exercise the pure accounting functions with synthetic streamevents, no Docker and no agent. `te_harness` imports nothing heavy at module level (the@@ -32,7 +33,7 @@ def _feed(events):return rec-# ---- the bug itself ------------------------------------------------------------------+# ---- turns and tokens recovered from the stream --------------------------------------def test_killed_stream_is_not_recorded_as_free():row = _row_from_result(_feed([_assistant("m1", 2, 4000, 3000, 5),@@ -42,7 +43,7 @@ def test_killed_stream_is_not_recorded_as_free():assert row["cache_creation_input_tokens"] == 4500assert row["cache_read_input_tokens"] == 10000assert row["tokens_total"] > 0, "a killed rollout must never record zero cost"- assert row["num_turns"] == 2, "turns are recoverable; 0 was the old bug"+ assert row["num_turns"] == 2, "turns are recoverable from the stream, so never 0"assert row["usage_incomplete"] is Trueassert row["usage_incomplete_reason"] == "spiral_guard_stop"@@ -89,7 +90,7 @@ def test_healthy_row_carries_the_flags_as_false():assert row["usage_incomplete"] is Falseassert row["output_tokens_missing"] is Falseassert row["num_turns_recovered"] is False- assert row["output_tokens"] == 40, "a healthy row must be untouched by the fix"+ assert row["output_tokens"] == 40, "a healthy row must be untouched by the recovery"# ---- the spiral case: one bad segment degrades the whole task -------------------------@@ -121,11 +122,11 @@ def test_merge_reason_names_every_degraded_segment():assert _merge_usage(a, b)["usage_incomplete_reason"] == "spiral_guard_stop+task_timeout"-# ---- the guard-parameter probe (the webbench TypeError) -------------------------------+# ---- the guard-parameter probe --------------------------------------------------------def test_guard_accepts_probes_the_vendored_signature():- """`never_abort` exists in some vendored versions of the guard and not in others.- webbench.py passed it unconditionally, which killed every guarded Web-Bench run at- task 0 before a row was written. Both runners now go through this one probe."""+ """`never_abort` exists in some vendored versions of the guard and not in others, and+ passing a keyword the guard does not take raises at task 0, before a row is written. So+ neither runner passes it unconditionally: both go through this one probe."""from te_bench import nudge_guard as spiral_vendorfrom te_harness.swebench import guard_acceptsassert guard_accepts("envelope") is True@@ -137,8 +138,8 @@ def test_guard_accepts_probes_the_vendored_signature():def test_webbench_guard_constructs_on_this_checkout():- """The regression itself: `_make_guard` must not raise whatever the vendored guard's- signature is. It takes no `self` state, so it is callable unbound."""+ """`_make_guard` must not raise whatever the vendored guard's signature is. It takes no+ `self` state, so it is callable unbound."""from te_harness.webbench import WebBenchRunnerguard = WebBenchRunner._make_guard(None, {"spiral_guard": True, "spiral_nudge": True, "spiral_never_abort": True,@@ -212,7 +213,7 @@ def test_an_in_flight_turn_keeps_the_row_flagged():assert row["usage_incomplete_reason"] == "task_timeout"-def test_no_session_log_falls_back_to_the_old_honest_behaviour():+def test_no_session_log_falls_back_to_a_flagged_zero():"""An unreadable log must degrade to `output_tokens=0` + flagged, never to a crash."""rec = _feed([_assistant("m1", 2, 4000, 3000, 5)])for text in (None, "", "garbage"):diff --git a/experiments/tests/test_vendor_cost.py b/experiments/tests/test_vendor_cost.pyindex e11af71..94c6189 100644--- a/experiments/tests/test_vendor_cost.py+++ b/experiments/tests/test_vendor_cost.py@@ -1,17 +1,18 @@"""The vendor-dollar column: read the key the runners actually write, and flag its absence.-Two defects, both of which made a cost comparison quietly wrong rather than loudly broken:+Two things a cost comparison depends on; either one, missed, makes it quietly wrong rather+than loudly broken:- * `aggregate.py` emitted `cost_usd` from `d.get("cost_usd")`, but every runner writes the- figure as `cost_usd_vendor`. The column was therefore EMPTY in every tree this repo+ * `aggregate.py` emits `cost_usd` from the key every runner writes, `cost_usd_vendor`.+ Reading `d.get("cost_usd")` alone would leave the column EMPTY in every tree this repogenerates, while looking like a column that simply had no data yet.- * A row whose agent reported no `total_cost_usd` stored None and was marked by nothing:+ * A row whose agent reported no `total_cost_usd` stores None, and nothing else marks it:`usage_incomplete` stays False (the token breakdown is complete and the effective-tokenbasis is exact) and the task grades normally, so the row reads as healthy. Summing thedollar column then under-counts, because the row's tokens are in the denominator whileits dollars are not -- and because the misses cluster on the longest trajectories, the- error lands on the most expensive tasks.+ error lands on the most expensive tasks. `vendor_cost_missing` flags it.These tests are offline: they call the two pure functions with synthetic rows."""diff --git a/experiments/tests/test_workloads.py b/experiments/tests/test_workloads.pyindex fafc775..4de32a7 100644--- a/experiments/tests/test_workloads.py+++ b/experiments/tests/test_workloads.py@@ -244,8 +244,7 @@ def test_driver_task_list_is_not_stale():# ---- Web-Bench's per-segment cap must stay under the evaluator's request abort ------------# TE_TASK_TIMEOUT is shared with SWE-bench, where 3600 is correct; on Web-Bench anything past-# the evaluator's ~300s undici abort loses whole attempts and truncates the stream. Measured-# 2026-09-07: it silently killed two threejs streams at task-14 before the clamp existed.+# the evaluator's ~300s undici abort loses whole attempts and truncates the stream.def test_webbench_task_timeout_is_clamped_under_the_evaluator_abort():from te_harness.webbench import (webbench_task_timeout, WEBBENCH_TASK_TIMEOUT_MAX,@@ -266,14 +265,14 @@ def test_webbench_task_timeout_defaults_and_overrides():def test_swebench_task_timeout_is_untouched_by_the_webbench_clamp():- """SWE-bench genuinely needs 3600 (sympy pinned 1800 dead-on); the clamp is domain-local."""+ """SWE-bench genuinely needs 3600 (sympy tasks run to 1800 s); the clamp is domain-local."""import te_harness.swebench as sbassert "WEBBENCH_TASK_TIMEOUT_MAX" not in dir(sb)# ---- the abort ceiling is raisable, because the evaluator's bare `fetch` uses undici's# GLOBAL dispatcher, which `--import te_harness/webbench_undici_preload.mjs` can reset.-# Opt-in only: unset, every value below is what the fixed 300/260 constants produced.+# Opt-in only: unset, the abort ceiling stays at undici's stock 300 s and the cap at 260 s.def test_webbench_undici_abort_defaults_to_the_stock_ceiling_and_adds_no_env():from te_harness.webbench import (webbench_undici_abort, webbench_undici_env,diff --git a/experiments/workloads/swebench.json b/experiments/workloads/swebench.jsonindex e3b0a67..babfa8f 100644--- a/experiments/workloads/swebench.json+++ b/experiments/workloads/swebench.json@@ -309,7 +309,7 @@]},"excluded": {- "scikit-learn__scikit-learn-14710": "eval.sh does not terminate under te_harness: timed out at TE_EVAL_TIMEOUT=1800 and again at 3600, and the uncaught TimeoutExpired ended the sklearn stream at task 21 of 32 both times (that cell never produced more than 20 graded rows). The timeout is now caught (te_harness swebench.py _grade), so it would cost one full eval_timeout per run rather than the cell; excluded to stop paying that. NOT ungradeable in principle -- it graded in the published runs, so results that include it and results that exclude it are not directly comparable on scikit-learn."+ "scikit-learn__scikit-learn-14710": "eval.sh does not terminate under te_harness within TE_EVAL_TIMEOUT (1800 s, nor 3600 s), so every run of it costs one full eval_timeout (te_harness swebench.py _grade catches the timeout); excluded to stop paying that. NOT ungradeable in principle -- it graded in the published runs, so results that include it and results that exclude it are not directly comparable on scikit-learn."},"added": {"scikit-learn/scikit-learn": [diff --git a/tests/test_envfacts.py b/tests/test_envfacts.pyindex 3346112..3d277b0 100644--- a/tests/test_envfacts.py+++ b/tests/test_envfacts.py@@ -223,9 +223,8 @@ def d_text(d):def test_an_environment_fact_forms_in_a_repository_where_every_session_edits_one_file(tmp_path):- """Seen live on all four agents: in a small repository every session touched the same- module, so a fact needing confirmations on different files never formed. Whether `pytest`- is missing does not depend on which file a session edits."""+ """Whether `pytest` is missing does not depend on which file a session edits: in a small+ repository where every session touches the same module, the fact still forms."""def same_file(tmp_path, i):return _task(tmp_path, i, [_edit("pkg/stock.py"),diff --git a/tests/test_envfacts_args.py b/tests/test_envfacts_args.pyindex f6ee072..36bd7ba 100644--- a/tests/test_envfacts_args.py+++ b/tests/test_envfacts_args.py@@ -226,8 +226,8 @@ def test_a_cd_into_the_start_directory_by_absolute_path_is_no_cd():def test_a_cd_up_to_the_repository_from_a_subdirectory_is_no_cd():- """Seen live on Pi: a session started in `pkg/` ran `cd <repository> && pytest -q`, which- was learned as a form of its own, spelled with that machine's path."""+ """A session started in `pkg/` that runs `cd <repository> && pytest -q` runs the plain+ form: a `cd` up to the repository is no `cd`, and no form is spelled with a machine's path."""kw = dict(cwd="", root="/r/pkg", top="/r")assert observe.classify_command("cd /r && pytest -q", **kw)[0].form == "pytest"assert observe.classify_command("cd .. && pytest -q", **kw)[0].form == "pytest"@@ -311,8 +311,8 @@ def test_an_option_the_runner_rejects_is_not_a_failing_form(tmp_path):def test_a_run_whose_output_was_filtered_away_does_not_show_a_form_works(tmp_path):- """Seen live: `python runtests.py x 2>&1 | grep -E "^(OK|FAILED|Ran )"` at the root printed- nothing -- the "can't open file" was filtered out -- and counted as the form working."""+ """`python runtests.py x 2>&1 | grep -E "^(OK|FAILED|Ran )"` at the root prints nothing --+ the "can't open file" is filtered out -- and an empty output does not show a form works."""tasks = [[_at(_edit(i), ROOT), _at(_sh("python -m pytest tests/app", "Exit code 1\n""/venv/bin/python: No module named pytest"), ROOT),_at(_sh(f"python runtests.py app{i} 2>&1 | grep -E '^(OK|FAILED|Ran )'",@@ -322,9 +322,9 @@ def test_a_run_whose_output_was_filtered_away_does_not_show_a_form_works(tmp_patdef test_a_working_form_that_leaves_the_shell_where_it_is_is_the_one_named(tmp_path):- """Seen live: the fact named `cd tests && python runtests.py`, agents typed it, and their- shell stayed in `tests/` -- every later `cd tests && ...` failed. A form that does not move- the shell is named whenever one has worked, however rarely."""+ """A form that moves the shell (`cd tests && python runtests.py`), typed by an agent whose+ shell keeps its directory, leaves it in `tests/`, where every later `cd tests && ...` fails.+ A form that does not move the shell is named whenever one has worked, however rarely."""tasks = [[_at(_edit(i), ROOT), _at(_sh(f"python runtests.py app{i}", CANT_OPEN), ROOT),_at(_sh(f"cd tests && python runtests.py app{i}", OK), ROOT),_at(_sh(f"cd tests && python runtests.py app{i}.tests", OK), ROOT)]diff --git a/tests/test_generic_microskills.py b/tests/test_generic_microskills.pyindex f5f0ccf..02cf3dd 100644--- a/tests/test_generic_microskills.py+++ b/tests/test_generic_microskills.py@@ -87,9 +87,8 @@ def test_the_plugin_reads_every_repository_generically(tmp_path):def test_the_proposer_keeps_its_window_across_folds(tmp_path, monkeypatch):- """The plugin reloads the proposer for every session. Its window of recent sessions was not- saved, so each tick saw one session, and a fact -- which must cite two -- could never be- admitted, however many calls were paid for."""+ """The plugin reloads the proposer for every session, so its window of recent sessions is+ saved with its state: a fact must cite two sessions."""monkeypatch.setattr(dist, "PROPOSE_EVERY", 3)prompts = []@@ -113,6 +112,34 @@ def test_the_proposer_keeps_its_window_across_folds(tmp_path, monkeypatch):assert [dg["iid"] for dg in again.window] == ["s0", "s1"]+def test_the_proposer_is_shown_the_facts_it_already_has(tmp_path, monkeypatch):+ """The plugin's prompt shows each fact it holds with its claim, so a later run does not+ author one again under a new id."""+ monkeypatch.setattr(dist, "PROPOSE_EVERY", 3)+ prompts = []++ def runner(prompt, model):+ prompts.append(prompt)+ return json.dumps({"extractors": [], "facts": [{+ "id": "build-before-test", "claim": "make test needs make build first",+ "injection": "`make test` runs the built binary: run `make build` first.",+ "detector": {"scope": r"make test", "confirm": r"make build"},+ "cite": ["s0", "s1"]}]}), 100.0+ root = str(tmp_path / "state")+ for i in range(5):+ d = dist.Distiller.load("acme/tool", root, runner=runner, vocab=GENERIC)+ d.update(_session(tmp_path, i, [+ ("Bash", {"command": "make build"}, "built"),+ ("Bash", {"command": "make test"}, "ok ./... 0.01s")]), iid=f"s{i}")+ d.save(root)+ assert len(prompts) == 2+ assert "Existing facts, as `id: claim`" in prompts[0] and "(none)" in prompts[0]+ assert " - build-before-test: make test needs make build first" in prompts[1]+ # the experiments' prompt still names the ids alone, as the paper ran it+ p = dist.build_proposer_prompt("acme/tool", [], {"build-before-test": "x y z"}, set(), [])+ assert "Existing fact ids (re-emit" in p and "\nbuild-before-test\n" in p and "x y z" not in p++def test_the_experiments_proposer_prompt_is_the_papers():"""The generic prompt is the plugin's; the experiments' must stay what the paper ran."""assert hashlib.sha256(dist.PROPOSER_SYSTEM.encode()).hexdigest()[:16] == "fad9adfb8fda5875"diff --git a/tests/test_live_core.py b/tests/test_live_core.pyindex fbfa1ec..b5d666e 100644--- a/tests/test_live_core.py+++ b/tests/test_live_core.py@@ -462,6 +462,7 @@ def test_the_pi_host_reads_what_the_extension_sends():# ---- the guard sees edits made through the shell ---------------------------------------------from tokeneconomizer import loopguard as guard # noqa: E402+from tokeneconomizer.loopguard import turn # noqa: E402@pytest.mark.parametrize("cmd,want", [@@ -515,8 +516,8 @@ def test_shell_edits_reach_the_state_the_judge_reads_and_the_ended_green_proxy()def test_reading_a_file_full_of_asserts_is_not_a_failure():- """Seen live on Pi: a passing 13-call turn that read its test file three times counted four- error outputs and two repeated identical failures, and opened the paid judge's gate."""+ """A test file read back is full of `assert` and `Error`: reading it, three times over, is+ no failure and no repeated failure, and opens no gate to the judge."""test_file = "def test_total():\n assert total([1, 2]) == 3\n raise ValueError('no')\n"g = guard.TurnGuard(turn_key="t")for _ in range(3):@@ -646,9 +647,8 @@ def test_copilot_logs_read_through_the_existing_trace_seam(tmp_path):def test_a_copilot_call_into_the_session_directory_is_the_plain_form(tmp_path):- """Seen live: Copilot's models type `cd <repo> && ` before every command, so the repeated- `cd /w/repo && python -m unittest` and a one-off bare form were two forms, and the bare one- won as the form that leaves the shell where it is."""+ """Copilot's models type `cd <repo> && ` before every command: `cd /w/repo && python -m+ unittest` is the same form as the bare one."""from tokeneconomizer.microskills.observe import PYTHON_GIT, observe_taskp = copilot_events(str(tmp_path / "s" / "events.jsonl"), [("bash", {"command": "cd /w/repo && pytest -q"},@@ -718,8 +718,8 @@ def test_a_copilot_side_call_is_a_bare_completion_priced_from_its_own_log(tmp_pa@pytest.mark.real_sidecallsdef test_a_side_call_with_no_reply_is_noted_and_keeps_to_a_bare_completion(tmp_path,monkeypatch):- """Seen live: under Copilot's BYOK the provider's variables do not reach the hooks, so the- side call ran signed out, answered nothing, and the plugin said nothing either."""+ """Under Copilot's BYOK the provider's variables do not reach the hooks, so a side call can+ run signed out and answer nothing: that is noted, and the call stays a bare completion."""import subprocessfrom tokeneconomizer.plugin import sidecallmonkeypatch.setenv("TOKENECONOMIZER_HOME", str(tmp_path / "te"))@@ -908,7 +908,7 @@ def test_a_small_envelopes_backstop_never_preempts_the_nudge():"./runtests.py basic", ".venv/bin/pytest -q tests/test_a.py","cd web && ./node_modules/.bin/jest src", "/usr/bin/python3 -m pytest","/w/venv/bin/pytest -q", "~/.local/bin/pytest -q",- "python3 /w/repo/tests/runtests.py basic"])+ "python3 /w/repo/tests/runtests.py basic", "node --test test/"])def test_a_runner_named_by_path_counts_as_a_verification(cmd):"""A failing run after the last edit means the turn did not end green, whatever path therunner was called by -- otherwise long failing turns feed the envelope and the guard@@ -925,6 +925,97 @@ def test_a_file_named_like_a_runner_is_not_a_verification(cmd):assert not guard.VERIFY_RE.search(cmd)+TAP_FAILING = "# tests 9\n# suites 0\n# pass 8\n# fail 1\n# cancelled 0"+++def test_a_test_run_whose_status_was_replaced_is_judged_by_its_output():+ """A pipe, `||` or a report of the status gives the command line another command's status:+ `npm test 2>&1 | tail -n 8` exits with `tail`'s 0 whatever the tests did."""+ g = guard.TurnGuard(turn_key="t")+ for _ in range(3):+ g.fold("bash", {"command": "npm test 2>&1 | tail -n 8"}, TAP_FAILING, 0)+ assert (g.whole.error_outputs, g.whole.repeated_fail_sigs) == (3, 2)+ assert not g.resolved_proxy()+ # Codex records a call as `bash -lc '<script>'`+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": "/bin/bash -lc 'cd /w && timeout 60 go test ./... 2>&1 | tail -3'"},+ "--- FAIL: TestAdd (0.00s)\nFAIL\tacme/x\t0.01s", 0)+ assert g.whole.error_outputs == 1 and not g.resolved_proxy()+ for cmd in ('python3 -m unittest tests.test_x; echo "exit=$?"', "npm test || true"):+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": cmd}, "FAILED (failures=1)\nexit=1", 0)+ assert g.whole.error_outputs == 1 and not g.resolved_proxy()+ # under pipefail the status is the tests' own; `| grep` finding nothing is no failure+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": "set -o pipefail; npm test | tail -n 8"}, TAP_FAILING, 0)+ g.fold("bash", {"command": "pytest -q | grep FAILED"}, "", 1)+ assert g.whole.error_outputs == 0 and g.resolved_proxy()+++def test_a_test_run_followed_by_other_commands_is_judged_by_its_runners_summary():+ """`pytest -q; git diff` exits with `git`'s status, and its output holds the diff too: the+ runner's own summary line says how the tests went."""+ diff = "diff --git a/m.py b/m.py\n+ raise ValueError('no')\n"+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": "pytest -q; git diff"},+ "F.\n1 failed, 1 passed in 0.12s\n" + diff, 0)+ assert g.whole.error_outputs == 1 and not g.resolved_proxy()+ # passing tests, whatever the later command's status or the words in its output+ for code in (1, 0, None):+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": "pytest -q\ngit diff --exit-code"}, "2 passed in 0.10s\n" + diff,+ code)+ assert g.whole.error_outputs == 0 and g.resolved_proxy()++[email protected]("out, failed", [+ ("1 failed, 4 passed, 2 warnings in 1.23s", True), ("== 1 error in 0.12s ==", True),+ ("2 passed, 1 warning in 0.30s", False), ("no tests ran in 0.01s", None),+ ("Ran 3 tests in 0.1s\n\nFAILED (failures=1)", True), ("Ran 3 tests in 0.1s\n\nOK", False),+ ("tests finished: 10 passed, 1 failed, in 3.20 seconds", True), (TAP_FAILING, True),+ ("# pass 9\n# fail 0", False), ("Tests: 1 failed, 3 passed, 4 total", True),+ (" 3 passing (20ms)\n 1 failing", True), ("--- FAIL: TestAdd (0.00s)", True),+ ("ok \tacme/x\t0.012s", False), ("test result: FAILED. 1 passed; 1 failed", True),+ ("test result: ok. 3 passed; 0 failed", False), ("5 examples, 1 failure", True),+ ("Tests run: 4, Failures: 0, Errors: 1", True), ("[INFO] BUILD SUCCESS", False),+ ("+ raise ValueError('no')", None), ("ok 1 - add", None)])+def test_a_runners_summary_gives_its_verdict(out, failed):+ assert turn.runner_verdict(out) is failed++[email protected]("cmd", ["tar -xOf pkg.tgz pkg/x.py | sed -n 1,80p",+ "cat tests/test_x.py | head -40", "grep -rn pytest src | head",+ "python -m pytest x.py --collect-only -q | rg test_",+ "bin/test --help | head", "python repro.py 2>&1 | tail",+ "pytest -q; git diff --stat", "pytest -q\npython - <<'PY'\nx\nPY"])+def test_any_other_command_keeps_its_exit_status(cmd):+ """Piped reads of source, and a test run among other commands with no runner's summary in+ the result, keep the status the agent reported: `assert` and `Error` in such output are no+ verdict."""+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": cmd}, "def f():\n assert x\n raise ValueError('no')\n", 0)+ assert g.whole.error_outputs == 0++[email protected]("out, failing", [+ (TAP_FAILING, True), ("# pass 9\n# fail 0", False), ("not ok 3 - rates", True),+ (" 2 failing", True), ("5 examples, 0 failures", False), ("5 examples, 2 failures", True),+ ("Tests run: 4, Failures: 1, Errors: 0", True), ("Tests run: 4, Failures: 0, Errors: 0", False),+ ("--- FAIL: TestAdd (0.00s)", True), ("ok \tacme/x\t0.01s", False)])+def test_other_ecosystems_failing_test_summaries_count_as_failures(out, failing):+ g = guard.TurnGuard(turn_key="t")+ g.fold("bash", {"command": "make test"}, out, None)+ assert (g.whole.error_outputs == 1) is failing+++def test_a_unittest_failure_is_known_by_its_traceback():+ """unittest's `FAIL: test_x (...)` is not Go's `FAIL`: a Django failure is known by its+ traceback, not its test name, so the same failure in another test still repeats."""+ from tokeneconomizer.loopguard.state import _failure_sig+ out = "FAIL: test_x (validators.tests.T)\nTraceback (most recent call last):\n ..."+ assert _failure_sig(out) == _failure_sig("Traceback (most recent call last):")++def test_codex_exec_and_copilot_p_are_told_from_their_terminal_uis(tmp_path, monkeypatch):"""The still-stuck notice names the headless way to stop in `codex exec` and `copilot -p`.Codex says how a session was started only in its rollout's first record; Copilot only in itsdiff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.pyindex eb09b8b..4f27e18 100644--- a/tests/test_live_hooks.py+++ b/tests/test_live_hooks.py@@ -113,10 +113,9 @@ def prevented_session(path, repo, i):def test_an_injected_fact_that_prevents_its_failure_stays_injected(env, repo, tmp_path):- """The session-start hook injects; a fold in another process learns afterwards. Without- telling the fold which facts the session was given, a fact that prevented its failure- counted as not having fired, and decayed out of the block -- seen live on Codex, where- three injected sessions left a fact at 5/8 and later under its gate."""+ """The session-start hook injects; a fold in another process learns afterwards. The fold is+ told which facts the session was given, so a fact that prevented its failure counts as having+ fired, and stays in the block."""for i in range(5):run_session(env, repo, f"s{i}", trap_session(str(tmp_path / f"s{i}.jsonl"), repo, i))starts = [run_session(env, repo, f"p{i}", prevented_session(str(tmp_path / f"p{i}.jsonl"),@@ -814,8 +813,8 @@ def test_each_session_is_folded_once_and_priced(env, repo, tmp_path):def test_a_fold_that_cannot_take_the_lock_skips_and_the_session_is_still_folded_once(env, repo, tmp_path, monkeypatch):- """The double fold seen live: a paid fold holds the lock through its side call, a second- fold of the same session gave up waiting after 10 s and folded it again, unlocked."""+ """A paid fold holds the lock through its side call; a second fold of the same session that+ cannot take it skips, leaving the session for the holder or a later start: it is folded once."""import fcntlfrom tokeneconomizer.plugin import fold, hostsfrom tokeneconomizer.plugin.context import LiveContext@@ -1002,10 +1001,10 @@ SIGNED_OUT = ("Error: No authentication information found. Copilot can be authendef test_a_signed_out_copilot_pauses_the_model_calls_and_says_so_once(env, repo, tmp_path,monkeypatch):- """Seen live under BYOK: Copilot does not pass the provider to the plugin's hooks, so the- proposer's `copilot -p` asked to sign in, every third session, each time leaving an empty- session behind. One such answer pauses the calls; the next start says so once, the facts- that need no model call still form, and `config set microskills=on` tries again."""+ """Under BYOK Copilot does not pass the provider to the plugin's hooks, so the proposer's+ `copilot -p` can ask to sign in. One such answer pauses the calls; the next start says so+ once, the facts that need no model call still form, and `config set microskills=on` tries+ again."""import subprocessfrom tokeneconomizer.plugin import cli, sidecallfor k, v in env.items():diff --git a/tests/test_live_repos.py b/tests/test_live_repos.pyindex 673e009..c49a7b0 100644--- a/tests/test_live_repos.py+++ b/tests/test_live_repos.py@@ -2,10 +2,9 @@repository, session pins, carrying state over from the keys 0.1.0 used, per-checkout deny lists,and `te repos`.-Every case here was seen on a real state root (2026-10-19): an empty `~/.git` that filed every-project under the home directory, a clone and a worktree whose folder names split one repository-in three, a session recorded in two places because the agent `cd`-ed, and two sessions folded-twice under two keys.+The cases: an empty `~/.git` is no checkout of the projects below it; a clone and a worktree of+one repository share its key, whatever their folder names; a session that `cd`s stays under the+repository it started in; and a session is folded once, under one key."""import ioimport json@@ -301,8 +300,8 @@ def test_where_a_compaction_found_the_agent_does_not_move_its_session(tmp_path,def test_a_repositorys_own_state_is_not_merged_away_by_a_visiting_session(tmp_path, env):- """Seen on the first run against real state: the only session under a repository's key had- started in ~ and `cd`-ed in. It moves to ~; the repository's state directory stays."""+ """A repository's only session started in ~ and `cd`-ed in: it moves to ~, and the+ repository's state directory stays."""home = env["HOME"]repo_dir = git_repo(os.path.join(home, "widget"), "https://github.com/acme/widget")repo = paths.repo_identity(repo_dir, env)diff --git a/tests/test_plugin_package.py b/tests/test_plugin_package.pyindex 3073c0d..437c791 100644--- a/tests/test_plugin_package.py+++ b/tests/test_plugin_package.py@@ -313,8 +313,7 @@ def test_the_plugin_directory_is_also_a_pi_package():def test_the_plugin_carries_the_core_as_symlinks_an_install_copy_follows(tmp_path):"""Claude Code installs a COPY of plugin/ into its cache, even from a local checkout, so the- core must be reachable from inside plugin/. Seen live: without plugin/lib the installed copy's- hooks answered `{}` and every command crashed."""+ core must be reachable from inside plugin/."""import shutilassert os.listdir(os.path.join(PLUGIN, "lib")) == ["tokeneconomizer"]link = os.path.join(PLUGIN, "lib", "tokeneconomizer")@@ -331,9 +330,8 @@ def test_the_plugin_carries_the_core_as_symlinks_an_install_copy_follows(tmp_patdef test_a_codex_install_copy_finds_the_core_in_the_marketplace_it_came_from(tmp_path):- """Codex installs a COPY of plugin/ that drops the `lib` symlink (seen live on 0.153.1), so- `codex plugin marketplace add <clone>` gave a plugin whose hooks answered `{}` and whose `te`- crashed. The copy now reads the marketplace's `source` from Codex's config.toml."""+ """Codex installs a COPY of plugin/ that drops the `lib` symlink (0.153.1), so the copy+ finds the core through the marketplace's `source` in Codex's config.toml."""import shutilhome = tmp_path / "codex"copy = home / "plugins" / "cache" / "tokeneconomizer" / "tokeneconomizer" / "0.1.0"diff --git a/tests/test_trace.py b/tests/test_trace.pyindex 94a3bd1..57592d1 100644--- a/tests/test_trace.py+++ b/tests/test_trace.py@@ -119,7 +119,7 @@ def test_usage_checkpoint_source_is_still_read(tmp_path):def test_no_source_reports_none_not_empty_catalog(tmp_path):- """The regression that hid the bug: {} must be distinguishable from 'no tools'."""+ """No source must be distinguishable from an empty catalog: {} is not 'no tools'."""(tmp_path / "copilot_debug.log").write_text("nothing useful here")cat, src = _trace.parse_catalog_with_source(str(tmp_path), "copilot")assert cat == {}diff --git a/tokeneconomizer/loopguard/envelope.py b/tokeneconomizer/loopguard/envelope.pyindex 2683039..b2d8cc9 100644--- a/tokeneconomizer/loopguard/envelope.py+++ b/tokeneconomizer/loopguard/envelope.py@@ -41,8 +41,8 @@ class EnvelopeTracker:Thresholds: `thresholds()` derives the FULL detector threshold dict from the SAMEresolved-task history, so the guard adapts to the agent/repo's actual turn scale- instead of the hardcoded Copilot-scale DEFAULT_THRESHOLDS (which never fired on Claude- Code's <=80-turn runs — its backstop landed above the turn cap). Turn-scale gates+ instead of the hardcoded Copilot-scale DEFAULT_THRESHOLDS, whose backstop lies above+ Claude Code's 80-turn cap. Turn-scale gatesscale DOWN to the learned distribution; signal gates keep DEFAULT as a FLOOR so thelearned dict is never MORE trigger-happy than the hand-tuned one (resolve-safety). Fedonly by RESOLVED tasks (same causal rule as the envelope), so a task's own outcome can@@ -101,8 +101,8 @@ class EnvelopeTracker:thr = dict(base)env = self.envelope# cap-aware backstop: reachable even under a tight turn cap. The Copilot-scale- # `backstop_mult * envelope` landed ABOVE Claude's 80-turn cap (~110 turns) so it- # never fired; clamp it under 0.9*cap so an extreme outlier still aborts in-cap.+ # `backstop_mult * envelope` (~110 turns) lies above Claude's 80-turn cap, so it is+ # clamped under 0.9*cap and an extreme outlier still aborts in-cap.if env is not None:bt = env * thr["backstop_mult"]if max_turns:diff --git a/tokeneconomizer/loopguard/rules.py b/tokeneconomizer/loopguard/rules.pyindex 37b1844..a9c6801 100644--- a/tokeneconomizer/loopguard/rules.py+++ b/tokeneconomizer/loopguard/rules.py@@ -24,18 +24,16 @@ DEFAULT_THRESHOLDS = {"error_outputs": 6, # sustained error density"stagnation": 0.20, # >=20% of failures are repeats of an earlier one"envelope_margin": 1.25, # restart/abort only beyond margin x the resolved-turn pctl- "envelope_pct": 90, # use the p90 of resolved turns (NOT max) — robust to a- # single very-long resolve poisoning the gate (see- # EnvelopeTracker). max-based envelope was disabled by one- # 115-turn xarray resolve; p90 ignores the lone outlier.+ "envelope_pct": 90, # use the p90 of resolved turns (NOT max): one very long+ # resolve would set a max-based envelope far out (see+ # EnvelopeTracker); p90 ignores the lone outlier."envelope_min_samples": 5, # need >=5 resolved tasks before trusting the percentile"judge_confidence": 0.7, # LLM trace-judge must be >=this confident to drive an abort"backstop_mult": 2.5, # HARD backstop: beyond this multiple of the envelope, a# stuck trajectory aborts REGARDLESS of the judge's verdict.- # Empirically the judge spared xarray-7019/7179 at 5.8x the- # envelope (177-204 turns vs envelope 35, stagnation 0.3-0.4)- # by misreading multi-file thrashing as "exploration"; the- # judge must not be able to veto a screaming outlier.+ # A judge can read multi-file thrashing far past the envelope+ # as "exploration"; it must not be able to veto such an+ # outlier.# ---- cheap GATE to the judge (Tier-0): permissive "worth a paid look?" trigger.# Sensitive on purpose (catch candidates); the judge supplies precision. Far cheaper# to evaluate than the judge, and far more permissive than is_stuck (which is the@@ -131,8 +129,8 @@ def assess_abort(state, envelope=None, judge=None, thr=None):# HARD BACKSTOP (checked BEFORE the judge): an extreme outlier — far past the# envelope AND stuck by the strict heuristic — aborts regardless of any judge- # verdict. The judge cannot veto this; it exists because the LLM judge empirically- # rationalized 5.8x-envelope thrashing (xarray-7019/7179) as "exploration".+ # verdict. The judge cannot veto this; it exists because an LLM judge can read thrashing+ # far past the envelope as "exploration".# `backstop_turns` (learned, cap-aware) is preferred when the tracker supplied it, so# the backstop stays REACHABLE under a tight turn cap; else fall back to the# multiplier-on-envelope form (the Copilot-scale behavior).@@ -168,8 +166,8 @@ def assess_abort(state, envelope=None, judge=None, thr=None):if v.get("stuck") is False:return False, "judge-productive"# judge AMBIGUOUS (stuck=None): do NOT abort on the (possibly under-calibrated,- # early-stream) envelope alone — that killed resolvable tasks like xarray-7120 at- # turn 28 when the p90 envelope was only 27. Ambiguity => SPARE; the hard backstop+ # early-stream) envelope alone: a resolvable task can run just past an envelope+ # learned from few turns. Ambiguity => SPARE; the hard backstop# (2.5x envelope AND is_stuck, checked above) remains the only envelope-driven kill,# and it sits well beyond any resolvable task's length.return False, "judge-ambiguous-spared"diff --git a/tokeneconomizer/loopguard/state.py b/tokeneconomizer/loopguard/state.pyindex 127f10a..8b14492 100644--- a/tokeneconomizer/loopguard/state.py+++ b/tokeneconomizer/loopguard/state.py@@ -19,7 +19,13 @@ TEST_RUN_RE = re.compile(r"\btox\b|\bnox\b|\btrial\b|python\d?\s+-c\b)", re.IGNORECASE)ERR_RE = re.compile(r"(Traceback \(most recent call last\)|\b\w*Error\b|\bFAILED\b|\bassert\b|"- r"Exception|errors during collection)", re.IGNORECASE)+ r"Exception|errors during collection|"+ # Other ecosystems' failing-test summaries, which name no error: TAP (`node --test`)+ # `# fail 2` and `not ok 3`, mocha's and rspec's counts, JUnit's, and Go's `--- FAIL: T`+ # and `FAIL<tab>pkg` (not unittest's `FAIL: test_x`, which would change the line a Python+ # failure is known by, and so which failures count as repeats).+ r"^# fail [1-9]|^not ok \d|\b[1-9]\d* (?:failing|failures?)\b|Failures: [1-9]|"+ r"^(?-i:--- FAIL: |FAIL\t|FAIL$))", re.IGNORECASE | re.MULTILINE)_EDIT_TOOLS = ("edit", "str_replace", "create")# Tools whose output is the content of files, not the outcome of running anything: a test file# full of `assert` read back is not a failure, nor is reading it twice a repeated one.diff --git a/tokeneconomizer/loopguard/turn.py b/tokeneconomizer/loopguard/turn.pyindex 21185bd..ee48342 100644--- a/tokeneconomizer/loopguard/turn.py+++ b/tokeneconomizer/loopguard/turn.py@@ -16,7 +16,9 @@ window to recover.agent stopped on its own, the guard did not stop it, and no verification after its last editfailed (`resolved_proxy`)."""+import osimport re+import shleximport timefrom .rules import DEFAULT_THRESHOLDS, assess_abort, is_stuck, nudge_text@@ -37,7 +39,137 @@ VERIFY_RE = re.compile(r"(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?test|go\s+test|cargo\s+(?:test|nextest)|"r"(?:mvn|mvnw|\./mvnw)\s+(?:\S+\s+)*test|(?:gradle|gradlew|\./gradlew)\s+(?:\S+\s+)*test|"r"make\s+(?:test|check)|ctest|rspec|phpunit|dotnet\s+test|swift\s+test|mix\s+test|"- r"deno\s+test|bazel\s+test|(?:\./)?bin/test)(?![\w.\-/])")+ r"deno\s+test|node\s+--test|bazel\s+test|(?:\./)?bin/test)(?![\w.\-/])")++# A command line's exit status is its LAST command's, so it may not be the test run's: `npm test+# 2>&1 | tail -n 8` exits with `tail`'s status, `pytest | grep FAILED` with `grep`'s, `python -m+# unittest; echo "exit=$?"` and `npm test || true` with 0, and `pytest -q; git diff` with+# `git`'s. For such a test run the runner's own summary line decides (`runner_verdict`). Without+# one, a run whose status was replaced (a pipe, `||`, a report of the status) is read by its+# output's markers, as for an agent that reports no status (Claude Code); a run followed by other+# commands keeps the status the agent reported, since their output is in the result too.+_SHELLS = {"sh", "bash", "zsh", "dash", "ksh"}+_QUOTED_RE = re.compile(r"'[^']*'|\"(?:\\.|[^\"\\])*\"")+_SEP_RE = re.compile(r"(&&|\|\||\|&|(?<!>)\||;|\n)") # `|`, but not the `>|` redirect+# A runner asked for its help, version or test list, not to run the tests.+_NOT_A_RUN_RE = re.compile(r"\s(?:-h|--help|--version|--collect-only|--co|--list\S*)(?!\S)")+# What may precede the command itself: `FOO=1 timeout 60 npm test`.+_PREFIX_RE = re.compile(r"^\s*(?:(?:\w+=\S*|time|nice|nohup|command|env|"+ r"timeout(?:\s+(?:-\S+|[A-Z]+|\d\S*))+)\s+)*")+# What only reports on the run before it: `echo "exit=$?"`, `true`, `exit 0`.+_REPORT_RE = re.compile(r"^\s*(?:echo|printf|true|:|exit)(?:\s|$)")+_RANK = {"own": 0, "hidden": 1, "mixed": 2}+++def status_of_tests(cmd):+ """What `cmd`'s exit status says of the test runs in it: None when it runs none; "own" when+ it is theirs (a run is last, or followed only by `&&`, or piped under `pipefail`); "hidden"+ when a pipe, `||` or a mere report of the status replaced it; "mixed" when other commands+ ran after, their output in the result too. A command recorded as `bash -lc '<script>'`+ (Codex) is judged by its script."""+ if not isinstance(cmd, str):+ return None+ try:+ parts = shlex.split(cmd)+ except ValueError:+ parts = []+ if (len(parts) >= 3 and os.path.basename(parts[0]) in _SHELLS+ and parts[1].startswith("-") and "c" in parts[1]):+ cmd = parts[2]+ pieces = _SEP_RE.split(re.sub(r"[(){}]", " ", _QUOTED_RE.sub("Q", cmd)))+ while len(pieces) > 1 and not pieces[-1].strip(): # a trailing `;` or newline+ del pieces[-2:]+ pipefail = "pipefail" in cmd+ found = None+ for i in range(0, len(pieces), 2):+ stage = pieces[i]+ if not VERIFY_RE.match(_PREFIX_RE.sub("", stage)) or _NOT_A_RUN_RE.search(stage):+ continue+ status = "own"+ for sep, then in zip(pieces[i + 1::2], pieces[i + 2::2]):+ if sep == "&&" or (pipefail and sep in ("|", "|&")):+ continue+ if sep in ("|", "|&", "||"):+ status = "hidden"+ break+ if not _REPORT_RE.match(then):+ status = "mixed"+ break+ status = "hidden"+ if found is None or _RANK[status] > _RANK[found]:+ found = status+ return found+++# A test runner's summary of its run, which holds whatever the command line's exit status.+_RUN_FAILED_RE = re.compile("|".join((+ r"\b[1-9]\d* (?:failed|errors?)\b.*\bin \d+(?:\.\d+)?s\b", # pytest+ r"^FAILED \((?:failures|errors)=", # unittest, Django+ r"^tests finished:.*\b[1-9]\d* (?:failed|exceptions?)\b", # sympy's bin/test+ r"^# fail [1-9]", # TAP (`node --test`)+ r"^Tests:.*\b[1-9]\d* failed\b", # jest+ r"^\s*Tests\s+[1-9]\d* failed\b", # vitest+ r"^\s*[1-9]\d* failing\b", # mocha+ r"^(?:--- FAIL: |FAIL\t|FAIL$)", # go+ r"^test result: FAILED", # cargo+ r"\b\d+ examples?, [1-9]\d* failures?\b", # rspec+ r"Tests run: \d+, Failures: (?:[1-9]|0, Errors: [1-9])", # JUnit (Maven, Gradle)+ r"BUILD FAIL(?:URE|ED)")), re.MULTILINE)+_RUN_PASSED_RE = re.compile("|".join((+ r"\b\d+ passed\b.*\bin \d+(?:\.\d+)?s\b",+ r"^OK(?: \(.*\))?\s*$",+ r"^tests finished: \d+ passed",+ r"^# fail 0\b",+ r"^Tests:\s+\d+ passed",+ r"^\s*Tests\s+\d+ passed",+ r"^\s*\d+ passing\b",+ r"^ok\s+\S+\s+(?:\d+(?:\.\d+)?s|\(cached\))",+ r"^test result: ok\.",+ r"\b\d+ examples?, 0 failures\b",+ r"BUILD SUCCESS")), re.MULTILINE)+++def runner_verdict(text):+ """True when a test runner's summary in `text` says its run failed, False when one says it+ passed (and none failed), None when `text` holds no summary."""+ if not text:+ return None+ if _RUN_FAILED_RE.search(text):+ return True+ return False if _RUN_PASSED_RE.search(text) else None+ try:+ parts = shlex.split(cmd)+ except ValueError:+ parts = []+ if (len(parts) >= 3 and os.path.basename(parts[0]) in _SHELLS+ and parts[1].startswith("-") and "c" in parts[1]):+ cmd = parts[2]+ pieces = _SEP_RE.split(re.sub(r"[(){}]", " ", _QUOTED_RE.sub("Q", cmd)))+ while len(pieces) > 1 and not pieces[-1].strip(): # a trailing `;` or newline+ del pieces[-2:]+ pipefail = "pipefail" in cmd+ for i in range(0, len(pieces) - 1, 2):+ stage = pieces[i]+ if not VERIFY_RE.match(_PREFIX_RE.sub("", stage)) or _NOT_A_RUN_RE.search(stage):+ continue+ hidden = False+ for sep, then in zip(pieces[i + 1::2], pieces[i + 2::2]):+ if sep in ("|", "|&"):+ hidden = not pipefail+ break+ if sep == "||":+ hidden = True+ break+ if sep in (";", "\n"):+ if not _REPORT_RE.match(then):+ hidden = False+ break+ hidden = True+ if hidden:+ return True+ return False++_EDIT = {"edit", "create"}@@ -204,8 +336,15 @@ class TurnGuard:The body is `NudgeGuard.monitor` for a single completed call."""thr = thr or DEFAULT_THRESHOLDS- shell_edits = (shell_edit_targets((args or {}).get("command"))- if canon_tool(name) == "bash" else [])+ cmd = (args or {}).get("command") if canon_tool(name) == "bash" else None+ shell_edits = shell_edit_targets(cmd) if cmd is not None else []+ status = status_of_tests(cmd)+ if status in ("hidden", "mixed"):+ verdict = runner_verdict(result)+ if verdict is not None:+ exit_code = int(verdict)+ elif status == "hidden":+ exit_code = Nonefailed = None if exit_code is None else exit_code != 0self.st.update(name, args or {}, result or "", failed)self.whole.update(name, args or {}, result or "", failed)diff --git a/tokeneconomizer/microskills/distiller.py b/tokeneconomizer/microskills/distiller.pyindex 79ee0dc..f46e751 100644--- a/tokeneconomizer/microskills/distiller.py+++ b/tokeneconomizer/microskills/distiller.py@@ -476,14 +476,26 @@ PROPOSER_SYSTEM_GENERIC = (_GENERIC_HEAD + PROPOSER_SYSTEM[len(_PROPOSER_HEAD):]"repro_written, importerror, badarg, ran_tests", ", ".join(GENERIC_PREDICATES))-def build_proposer_prompt(repo, window, existing_ids, existing_feats, raw_excerpts,+def build_proposer_prompt(repo, window, existing, existing_feats, raw_excerpts,external=(), generic=False):+ """`existing` maps each fact id to its claim (ids alone are accepted). The plugin's+ (`generic`) prompt shows the claims, so a fact already held is not authored again under a+ new id; the experiments' prompt names the ids alone, as the paper ran it."""view = [proposer_view(d) for d in window]+ existing = existing if isinstance(existing, dict) else dict.fromkeys(existing, "")+ if generic:+ known = ("\n\nExisting facts, as `id: claim` (do NOT restate or paraphrase one under a "+ "NEW id; re-emit an id with ONLY detector.contradict changed to revise its "+ "contradiction condition, else use a NEW id for a fact not listed here):\n"+ + ("\n".join(f" - {i}: {c}" if c else f" - {i}"+ for i, c in sorted(existing.items())) or "(none)"))+ else:+ known = ("\n\nExisting fact ids (re-emit an id with ONLY detector.contradict changed to "+ "revise its contradiction condition, else use a NEW id):\n"+ + (", ".join(sorted(existing)) or "(none)"))p = ((PROPOSER_SYSTEM_GENERIC if generic else PROPOSER_SYSTEM)+ f"\n\nRepo: {repo}"- + "\n\nExisting fact ids (re-emit an id with ONLY detector.contradict changed to "- "revise its contradiction condition, else use a NEW id):\n"- + (", ".join(sorted(existing_ids)) or "(none)")+ + known+ "\n\nExisting feature names:\n" + (", ".join(sorted(existing_feats)) or "(none)")+ "\n\nTask digests (chronological):\n" + json.dumps(view, indent=1))if raw_excerpts:@@ -606,7 +618,8 @@ class Distiller:abstaining = cov and (sum(cov) / len(cov)) < ABSTAIN_GATEexcerpts = self._abstaining_excerpts()[:RAW_SAMPLE] if abstaining else []external = list(self.external_facts()) if self.external_facts else []- prompt = build_proposer_prompt(self.repo, self.window, set(self.facts),+ prompt = build_proposer_prompt(self.repo, self.window,+ {i: f.claim for i, f in self.facts.items()},set(self.extractors), excerpts, external=external,generic=self.generic)raw, eff = self.runner(prompt, self.model)diff --git a/tokeneconomizer/microskills/envfacts.py b/tokeneconomizer/microskills/envfacts.pyindex a262f06..00351f2 100644--- a/tokeneconomizer/microskills/envfacts.py+++ b/tokeneconomizer/microskills/envfacts.py@@ -188,9 +188,7 @@ def _moves_shell(form):def _usage(form, args=" …"):"""A form as a claim shows it. One that starts with `cd DIR &&` is shown as a subshell: anagent whose shell keeps its directory between calls (Claude Code) that typed the bare- `cd` would run every later command in DIR -- seen live, where "`cd tests && python- runtests.py …` works" was followed by `cd: tests: No such file or directory` and `can't- open file 'runtests.py'` over and over."""+ `cd` would run every later command in DIR, where the next `cd DIR` fails."""return f"`({form}{args})`" if _moves_shell(form) else f"`{form}{args}`"diff --git a/tokeneconomizer/plugin/repos.py b/tokeneconomizer/plugin/repos.pyindex bb9e6f4..12bed6e 100644--- a/tokeneconomizer/plugin/repos.py+++ b/tokeneconomizer/plugin/repos.py@@ -1,21 +1,20 @@"""Which repository's state a session belongs to, over time: session pins, the checkout index,carrying state over from an earlier key, and the listing behind `te repos`.-Three things used to put one repository's sessions under several keys, or several repositories'-sessions under one; each has its remedy here.+Three things can put one repository's sessions under several keys, or several repositories'+sessions under one; each is handled here.* A session that changes directory. Every hook names the directory the agent is in NOW --- Claude Code's `cwd` follows the agent's `cd` -- so a session that moved into another- repository was recorded under both. The first hook of a session pins it+ Claude Code's `cwd` follows the agent's `cd` -- so a session that moves into another+ repository would be recorded under both. The first hook of a session pins it(`<root>/sessions/<host>/<id>.json`), and every later hook, and the fold it hands off, usesthe pinned repository: the one the session started in, which is also where the agent filesits own session log.- * A key that changed. Up to 0.1.0 a checkout's folder name was part of the key, any `.git`- entry made a checkout (an empty one in a home directory took in every project below it), and- a repository's `origin` can move. The first time a checkout is seen (`index.json` remembers- the ones already seen) the state its earlier keys hold is sent where it belongs (`adopt`);- `te repos migrate` does the same for every state directory, and runs by itself, once, in- the background after an upgrade.+ * A key that changed. Up to 0.1.0 a checkout's folder name was part of the key and any `.git`+ entry made a checkout, and a repository's `origin` can move. The first time a checkout is+ seen (`index.json` remembers the ones already seen) the state its earlier keys hold is sent+ where it belongs (`adopt`); `te repos migrate` does the same for every state directory, and+ runs by itself, once, in the background after an upgrade.* Sessions filed under a repository that is not theirs. Each recorded session names thedirectory it ran in; one whose directory now resolves to another repository moves thereand is folded again, so that repository learns from it (`migrate_dir`).diff --git a/tokeneconomizer/plugin/sidecall.py b/tokeneconomizer/plugin/sidecall.pyindex 50fa543..39ef417 100644--- a/tokeneconomizer/plugin/sidecall.py+++ b/tokeneconomizer/plugin/sidecall.py@@ -56,8 +56,8 @@ _SIGNED_OUT_RE = re.compile(r"To authenticate|/login\b|\b[Nn]ot (?:logged|signeddef _no_reply(host, out):- """Note a side call that came back empty: the paid helpers otherwise fail silently -- seen- with Copilot, which does not pass a BYOK provider's variables to the hooks that start it.+ """Note a side call that came back empty, which nothing else would show -- as under+ Copilot's BYOK, which does not pass the provider's variables to the hooks that start it.A CLI that said it is not signed in pauses the host's side calls."""import timefrom .paths import state_rootdiff --git a/tokeneconomizer/toolprune/stage.py b/tokeneconomizer/toolprune/stage.pyindex 3797a5b..202041f 100644--- a/tokeneconomizer/toolprune/stage.py+++ b/tokeneconomizer/toolprune/stage.py@@ -167,11 +167,10 @@ class ToolingStage(Stage):srow["untool_evidence_reaudit"] = bool(self._ev_reaudit)if not self._exclude_tools and self.adapter is not None:# Source-aware where the adapter supports it: an empty catalog is- # indistinguishable from "this agent loaded no tools", and that ambiguity- # is exactly how a Copilot CLI upgrade silently turned pruning into a- # no-op for a whole sweep (trace.catalog_sources). Record which source won- # so the next migration shows up in the rows, and say so out loud the first- # time nothing parses.+ # indistinguishable from "this agent loaded no tools", which would leave+ # pruning a no-op unnoticed (trace.catalog_sources). Record which source won+ # so a change of log format shows up in the rows, and say so out loud the+ # first time nothing parses.getter = getattr(self.adapter, "parse_catalog_with_source", None)if getter is not None:cat, src = getter(run_dir_path, self.agent)diff --git a/tokeneconomizer/trace/catalog.py b/tokeneconomizer/trace/catalog.pyindex aa14d7e..7dfcd87 100644--- a/tokeneconomizer/trace/catalog.py+++ b/tokeneconomizer/trace/catalog.py@@ -52,11 +52,11 @@ _CATALOG_RE = re.compile(## "tools": [ "bash:66921ccc", "read_bash:71c40652", ... ]#-# With (1) and (2) both gone on 1.0.83 the catalog parsed EMPTY, and an empty catalog is-# indistinguishable from "no tools loaded": pruning silently did nothing. So every source is-# tried and the richest wins (descriptions beat names, more tools beat fewer). The flat form-# carries names only, which is enough for evidence pruning; a tool judge reasoning over-# descriptions gets nothing from it.+# (1) and (2) are both gone on 1.0.83, and an empty catalog is indistinguishable from "no+# tools loaded", which would leave pruning a no-op. So every source is tried and the richest+# wins (descriptions beat names, more tools beat fewer). The flat form carries names only,+# which is enough for evidence pruning; a tool judge reasoning over descriptions gets nothing+# from it._CATALOG_FLAT_BLOCK_RE = re.compile(r'"tools"\s*:\s*\[(.*?)\]', re.DOTALL)_CATALOG_FLAT_ITEM_RE = re.compile(r'"([a-z][a-z0-9_]*):[0-9a-f]{6,}"')