Skip to content

Experience-Driven Black-Box Cost Control for Coding Agents

9.2 KB
Raw Blame

Reproducing the experiments

This directory is the code behind the paper's results: a harness that runs Claude Code, Codex and the Copilot CLI on SWE-Bench and Web-Bench, the launchers, and the aggregation scripts. It is not part of the plugin; the method is described in the main README.

Path Contents
te_harness/ runs an agent CLI in each SWE-Bench task's Docker image (on the host for Web-Bench) and grades it with the benchmark's own grader
te_bench/ the experiment arms, the benchmark adapters, and the evaluated task sets (workloads.py)
scripts/ launch/ (the launchers), run_experiment.py (one arm on one task stream), aggregate.py and compare_arms.py
workloads/ the evaluated task ids: swebench.json (277 tasks), webbench.json (102), and the security sets
tests/ offline tests: no Docker, data or API keys

Requirements

  • Linux x86_64 with Docker usable without sudo, and about 200 GB of disk for the SWE-Bench images.
  • Python 3.10 or newer, git, and Node.js with npm (for Web-Bench).
  • An Anthropic API key and an OpenAI API key. Claude Code and the Copilot CLI bill the first, Codex the second.
  • The agent CLIs on the host, at the versions the paper used. The harness mounts them into each task container.
curl -fsSL https://claude.ai/install.sh | bash -s 2.1.170          # Claude Code
npm install -g @openai/[email protected]                                 # Codex
curl -fsSL https://gh.io/copilot-install | VERSION=1.0.84-7 bash     # Copilot CLI

Setup

1. Install and check the code. Every command after cd experiments runs from this directory.

git clone https://github.com/microsoft/TokenEconomizer.git && cd TokenEconomizer
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[test,baselines]'        # baselines: aider, for the RepoMap arms
cd experiments
python -m pytest tests -q
bash scripts/launch/smoke_test.sh         # every arm wired end to end, with a stub agent

2. SWE-Bench. Pick a data directory outside the repository, and use its canonical path: Web-Bench fails every test when reached through a symlink.

export DATA="$(readlink -f ~/te_data)"
git clone https://github.com/SWE-bench/SWE-bench.git "$DATA/SWE-bench"
pip install -e "$DATA/SWE-bench"          # the official grader, into this venv
python scripts/export_swebench_tasks.py --out-dir "$DATA/swebench"
docker login                              # anonymous pulls are rate-limited
xargs -a "$DATA/swebench/image_tags.txt" -P 4 -n 1 docker pull -q

The export writes the task list, the grading file and the list of the 277 task images.

3. Web-Bench. The five evaluated projects, built, with the Playwright browser they pin:

git clone https://github.com/bytedance/web-bench.git "$DATA/web-bench"
( cd "$DATA/web-bench" && npm install -g @microsoft/rush && rush install \
  && rush build --to @web-bench/eval --to @web-bench/calculator --to @web-bench/expressjs \
                --to @web-bench/react --to @web-bench/zustand --to @web-bench/table \
  && cd projects/calculator && npx playwright install chromium chromium-headless-shell )
export TOKENECONOMIZER_WEBBENCH_PROJECTS="$DATA/web-bench/projects"
export TOKENECONOMIZER_WEBBENCH_TASKS="$DATA/webbench_tasks.json"
python scripts/webbench_tasks_from_yaml.py --all    # zustand ships only tasks.yml
python scripts/export_webbench_tasks.py

4. Keys and paths. cp .env.example .env, then fill it in with absolute paths (the file does not expand variables):

Variable Value
ANTHROPIC_API_KEY, OPENAI_API_KEY your keys
COPILOT_PROVIDER_API_KEY your Anthropic key again: the Copilot CLI bills Anthropic through its own provider settings, already set in the file
DATA the data directory
TOKENECONOMIZER_RESULTS where results go, e.g. $DATA/results
TOKENECONOMIZER_SWEBENCH_TASKS $DATA/swebench/swebench_tasks.json
TE_SWEBENCH_REPO $DATA/SWE-bench
TOKENECONOMIZER_WEBBENCH_PROJECTS, TOKENECONOMIZER_WEBBENCH_TASKS as exported in step 3

5. One paid task. Before a full run, check that one task produces a graded row:

AGENT=claude MODEL=claude-sonnet-5 DOMAIN=swebench ARMS=baseline REPOS=sympy/sympy SEEDS=0 \
  LIMIT=1 DETACH=0 RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh

Its row.json, under $TOKENECONOMIZER_RESULTS/claude_swebench_sonnet5/baseline/sympy_s0/, should show num_turns above zero, token counts, and a resolved value.

Running the paper's experiments

main_sweep.sh runs Claude Code and the Copilot CLI, codex_ab.sh runs Codex. Each command below is one lane, one agent and model on one benchmark: every arm it names, on every repository or project, with seeds 0, 1 and 2. A lane runs in the background and prints where its log is and how to stop it, so the commands in a loop run in parallel. Run a command again to resume a lane: cells already finished are skipped. DRYRUN=1 prints a lane's plan without running it, and DETACH=0 keeps it in the foreground.

The arms, by the names the paper uses:

Arm In the paper
baseline, baseline-medium, baseline-low default execution at high, medium and low effort
rtk, repomap the fixed strategies RTK and RepoMap
trimech (codex-trimech on Codex) TokenEconomizer
trimech-rtk, trimech-repomap (codex-trimech-…) TokenEconomizer+RTK, TokenEconomizer+RepoMap
inject-only, untool-only, spiral-only micro-skills, tool pruning and the failure-loop guard alone
trimech-web TokenEconomizer on Web-Bench, whose streams are short

SWE-Bench: Table 1, Figures 1(b) and 5, and the ablation in Table 2(a).

SWE="baseline baseline-medium baseline-low rtk repomap trimech trimech-rtk trimech-repomap"
for MODEL in claude-sonnet-5 claude-opus-4-8; do
  AGENT=claude MODEL=$MODEL DOMAIN=swebench ARMS="$SWE inject-only untool-only spiral-only" \
    RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=swebench ARMS="$SWE" \
  RUNNER=te_harness.swebench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
  MODEL=$MODEL DOMAIN=swebench \
    ARMS="baseline baseline-medium baseline-low rtk repomap codex-trimech codex-trimech-rtk codex-trimech-repomap" \
    RUNNER=te_harness.swebench:make_runner bash scripts/launch/codex_ab.sh
done

Web-Bench: Table 2(b). The same agent–model settings as on SWE-Bench.

for MODEL in claude-sonnet-5 claude-opus-4-8; do
  AGENT=claude MODEL=$MODEL DOMAIN=webbench ARMS="baseline trimech-web" \
    RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
done
AGENT=copilot MODEL=claude-sonnet-5 DOMAIN=webbench ARMS="baseline trimech-web" \
  RUNNER=te_harness.webbench:make_runner bash scripts/launch/main_sweep.sh
for MODEL in gpt-5.6-terra gpt-5.6-sol; do
  MODEL=$MODEL DOMAIN=webbench ARMS="baseline codex-trimech" \
    RUNNER=te_harness.webbench:make_runner bash scripts/launch/codex_ab.sh
done

Security benchmarks. The CredData and ReposVul task sets (workloads/creddata.json, workloads/reposvul.json) and their adapters are included, but this harness does not run or grade them.

Every arm, the baseline included, runs behind an answer-key guard that blocks the benchmarks' public answers on GitHub. The rtk binary is downloaded into $DATA/scaffolds.

Results

Each lane writes one results tree, $TOKENECONOMIZER_RESULTS/<agent>_<benchmark>_<model>/ (for example claude_swebench_sonnet5), with one directory per arm and cell (<arm>/<repository>_s<seed>/) and a run directory per task holding row.json, transcript.jsonl and run_config.json. At the end of a lane, aggregate.py writes index/all_rows.csv, one row per task run, and index/summary_by_cell.csv, one row per arm, repository and seed; run python scripts/aggregate.py <tree> to refresh it. A task run again keeps both runs, and the index takes the newest.

python scripts/compare_arms.py --baseline <tree>/baseline/<cell> --arm <tree>/<arm>/<cell>

compare_arms.py pairs one cell's tasks by instance and reports cost and resolved tasks against the baseline. Cost is in cache-aware effective tokens and includes the arm's own model calls. Web-Bench rows carry resolved_p2, the Pass@2 the paper reports. Never pool two trees: the agent and the model are both part of the cost basis.

Troubleshooting

Symptom Cause
a launch prints RUNNER=stub -- SYNTHETIC run RUNNER was not set; the stub runs no agent
ModuleNotFoundError at grading the SWE-Bench grader is not installed in the venv the launcher uses (step 2)
every Web-Bench test fails and the app never renders a non-canonical data path; readlink -f it (step 2)
Executable doesn't exist at …/chromium_headless_shell-… Playwright was installed outside a project, so its browser build differs from the one the projects pin (step 3)
Codex gets 401 with a valid key a key issued for a regional endpoint; set OPENAI_BASE_URL in .env
a lane needs stopping kill -TERM -$(cat <log>.pid), as printed at launch; the leading - stops the whole process group