Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
With/without-Ix agent evaluation
This harness compares baseline and ix runs of the same agent, model, effort and budget. It records resolution, usage and time, and reports paired differences with bootstrap 90% confidence intervals. Run Claude Code and Codex as separate experiments.
Offline smoke and tests
From the repository root, with Node 22+ on Linux or macOS:
node --test bench/agent-eval/test/eval.test.mjs
node bench/agent-eval/run.mjs --dry-run --out bench/agent-eval/results/smoke
Use a fresh output directory each time. The dry run executes subprocess setup, probes, a stub agent, evaluation, metrics parsing and reporting for three tasks × two arms × two repeats (12 runs). It requires neither Docker nor a paid agent. The checked-in smoke.json contains synthetic tasks and placeholder SHAs; its reports are explicitly synthetic and cannot establish Ix effectiveness.
The CLI test bridge runs portable manifest, telemetry and statistics tests on every platform in npm test, test:unit and test:coverage. POSIX orchestration/provisioning tests and the dry run are skipped on Windows; Linux/macOS retain that integration coverage. To exercise the portable subset alone, set AGENT_EVAL_PORTABLE_ONLY=1 when invoking the Node test suite. The tests cover manifests, arm configuration/isolation, native telemetry fixtures, bootstrap bounds, committed solution patches, local provisioning and the smoke.
Pin a real task set
No real SWE-bench Verified manifest ships with this repository, and none was generated or checked during development. Obtain an official princeton-nlp/SWE-bench_Verified test-split JSON array or JSONL export locally. Save a JSON array at bench/agent-eval/local/verified.json; save JSONL as verified.jsonl and use that name below and in dataset_export. The official evaluator picks its parser by file extension, so the importer and runner reject a mismatched name before any agent runs. Keep the upstream dataset revision and license information alongside it. Do not redistribute exports or task patches without checking upstream and repository licenses. The importer validates content but cannot authenticate provenance.
mkdir -p bench/agent-eval/local
# After placing your official export at local/verified.json:
node bench/agent-eval/manifest.mjs bench/agent-eval/local/verified.json bench/agent-eval/local/manifest-50.json
# A smaller real experiment: task count, minimum repo count, seed
node bench/agent-eval/manifest.mjs bench/agent-eval/local/verified.json bench/agent-eval/local/manifest-3.json 3 3 20261013
The default selects 50 tasks across at least five repos, using stable SHA-256 ranking by seed and task id. The generated manifest pins every task id, repo and base SHA, and the complete export hash. Preserve this manifest for reproduction. The runner checks each task against the hash-verified export. Gold/test patches are omitted from the manifest and native agent prompt.
SWE-PolyBench is not enabled; adding it requires license review and its official evaluator.
Local real-run setup
Use a reachable Docker daemon, a pinned local official SWE-bench checkout, cached public task repos, a local Ix backend and the installed native agent. Docker image downloads and real agent usage may incur costs; there is no hosted benchmark dependency. Supply authentication through the native CLI's supported environment, never through benchmark JSON. Existing home-directory login caches are deliberately not copied. The harness does not provide a sandbox against a malicious solver accessing other host files; use a disposable machine with no unrelated data.
Obtain the official SWE-bench source locally at a chosen full commit SHA. Keep its tracked files clean and install its dependencies:
python3 -m venv bench/agent-eval/local/venv bench/agent-eval/local/venv/bin/python -m pip install -e bench/agent-eval/local/SWE-bench git -C bench/agent-eval/local/SWE-bench rev-parse HEADSet
evaluator.revisionto that SHA and retain dependency versions and Docker image digests with the experiment. The evaluator verifies HEAD/cleanliness and imports from that checkout. Evaluation uses the saved official export, one worker and a separate deadline.Cache a bare clone of each selected repo as
local/repos/OWNER__NAME.git. Each cache must contain the manifest's base commits. For each locally obtained public task checkout:mkdir -p bench/agent-eval/local/repos git clone --bare /path/to/public-task-checkout bench/agent-eval/local/repos/OWNER__NAME.gitThe runner only clones from these caches, with detached pinned checkouts; it never fetches task repositories.
Build Ix in this checkout and start a dedicated local backend using the repository's documented Docker setup. Keep the backend image digest fixed throughout the experiment and save it with the results. No backend reset is performed by this harness.
npm --prefix ix-cli ci npm --prefix ix-cli run build node ix-cli/dist/cli/main.js --version git rev-parse HEADCopy
bench/agent-eval/config.example.jsontobench/agent-eval/local/config.json. Fill in the native agent version, immutable model id, built Ix version, Ix source SHA and SWE-bench SHA.environment.source_pathpoints to the Ix checkout;ix_clipoints to its built CLI entrypoint;plugin_revisionis its full HEAD SHA. Tracked CLI source and skills must match that SHA. Configure the reachable loopbackendpoint(default example:http://127.0.0.1:8090). Config paths are relative to the invocation directory; run from the repository root. The example uses supplied setup/probe programs; no adapter implementation is needed.Run a small real experiment before scaling:
node bench/agent-eval/run.mjs --manifest bench/agent-eval/local/manifest-3.json --config bench/agent-eval/local/config.json --out /path/to/isolated-eval-results/verified-3 # Use manifest-50.json and a fresh output directory for the full subset.
For Codex, set both agent.family and agent.telemetry to codex. For Claude Code, use claude. Keep the supplied agent.command. agent.binary can pin an absolute executable; otherwise it is resolved before isolation. agent.args supplies shared noninteractive permission options supported by your pinned version and must not override integrations. claude --print denies anything that would prompt. The example therefore passes --permission-mode acceptEdits in agent.args and lists shared tool rules (Bash) in agent.allowed_tools. The harness emits a single --allowedTools from that list and, for the ix arm only, adds the Ix MCP server rule (mcp__plugin_ix-agent-eval_ix-memory). agent.args cannot carry --allowedTools or any plugin/MCP/settings flag, and agent.allowed_tools cannot grant MCP tools. If Claude reports an Ix tool permission denial, the run fails instead of counting it as an Ix error. For Codex, codex exec needs an explicit --sandbox mode that can write the checkout (for example workspace-write). Check that sandboxed ix commands can still reach the loopback endpoint, and use the same args in both arms. Unsupported CLI flags fail the run. Config placeholders must be replaced; they are not model/version defaults.
Supplied arm provisioning
Both arms receive fresh homes and Git/XDG/agent configuration directories. The runner constructs a private executable PATH for every solver-side stage (clone, setup, probe, agent and its hooks), excluding ambient ix launchers. The evaluator is identical for both arms and runs after the solver has exited, with the runner's own inherited environment (real HOME and PATH), so the SWE-bench Docker client sees the host's ~/.docker configuration and contexts (rootless Docker, Docker Desktop). environment.tool_paths optionally pins the tool directories to copy into that PATH. Inherited Ix routing and parent sessions are removed. Claude Code's authentication and provider variables (CLAUDE_CODE_OAUTH_TOKEN, CLAUDE_CODE_USE_BEDROCK/_VERTEX/_FOUNDRY and their SKIP_*_AUTH and client-certificate settings) are kept, identically in both arms; other CLAUDE_CODE_* variables are removed. Other environment variables are inherited but never serialized. IX_TOKEN is removed with the other IX_* variables, and each run gets a fresh IX_HOME with no stored token. The harness does not forward credentials, so the configured loopback backend must accept unauthenticated local requests: do not start it with ix docker start --local-token. A token-protected backend fails setup at ix map before any agent runs. PATH isolation prevents ordinary direct Ix invocation; it is not an OS access boundary.
The baseline has no Ix launcher, skills, MCP registration or hooks. The Ix arm gets an explicit launcher for the configured built CLI, the revision-pinned tracked skills/ix files, and the benchmark's local integration:
- Claude Code: a local plugin loaded with
--plugin-dir, containing the skill, Ix MCP registration and a PostToolUse hook for edits and Bash. The hook refreshes the map and runsix hook claude-post-editfor dependent context. The product relies on editor plugins to re-ingest edits on a debounce. The benchmark has no editor plugin, so it refreshes synchronously, and only when the checkout changed since the last refresh: the hook compares a fingerprint ofgit status,git diff HEADand the untracked files' sizes and modification times, so a read-only Bash call (ls,cat, a search) costs two Git calls rather than anix map. Each hook record says whether itmapped. The hook's Claude timeout outlasts both ix deadlines. Hook time is part ofwall_secondsand is also reported separately ashook_seconds. - Codex: the skill in the run's
$CODEX_HOME/skills/ix(the location the product installer uses,ix-cli/scripts/skill-harnesses.mjs), an isolatedmcp_servers.ix-memoryconfiguration (withIX_ENDPOINTset in itsenvtable, because Codex starts MCP servers with an allowlisted environment) and a nativenotifyhook that refreshes the map afteragent-turn-complete. Its integration receipt records these settings.codex execreceives one prompt, which is normally one turn, so this refresh runs once, after the agent finishes. Codex may exit while that hook is still running. So when the agent CLI exits, the harness stops the stage's process group, kills every remaining process except Ix hooks (a command line runninghook.mjs) and their children, and waits only for those (up to the hook timeout,2 × environment.seconds + 30) before reaping the group. Both arms go through the same step; the baseline has no hooks, so all its leftovers die at once. Nothing the agent left running (a background build, test runner or server) can change the checkout after the agent exits, in either arm, and the hooks themselves only read the checkout. This needsps(POSIX); without it the whole group is killed at exit; that wait is not part ofwall_seconds, which (like the time budget) ends when the agent CLI exits, even if a hook still holds its output open, and the hook's own time is reported inhook_seconds. A hook still running when the group is reaped (outside an agent time-budget outcome) fails the run, because its telemetry would be incomplete. Within a Codex session, Ix answers do not reflect the agent's own edits. Mid-session refresh needs a per-tool hook in the pinned Codex version, which is not yet verified. Hook timing differs across agent families; compare arms within one family.
The probe does not take the skill file's presence as proof that Codex loads it. It runs codex debug prompt-input (which renders the model-visible prompt locally, without a model call) in the arm's environment and requires the ix skill in its skill list. A pinned Codex without that subcommand, or one that does not list the skill, fails the probe.
These are supplied benchmark integrations built from this repository, rather than installations from a marketplace or separate plugin repository. Preserve the harness checkout as well as the Ix source pin. Each run's solver-facing files (task checkout, home, integrations, hook telemetry) live in a fresh work directory under work_root (optional config field; default: the system temporary directory), not in the output directory. The solver's request names only that work directory; it carries no dataset export path (the gold patches), no evaluator settings and no output path. Only the evaluator, which runs after the solver exits, receives evaluate-request.json with the export. After a run is recorded, its request.json, hooks.jsonl and map receipts are copied into the run's output directory (agent-request.json, …) and the work directory is removed, so a later solver cannot browse earlier runs' patches or reports. A failed run keeps its work directory (named in failure.json) for inspection; the failure ends the experiment. Real runs reject an output directory inside work_root. Put work_root outside project ancestry containing agent instructions/configuration (including this Ix checkout); the provisioner rejects those ancestors and names the offending path. Home directories count too: a home holding ~/.claude, ~/.codex or ~/.agents rejects every work directory beneath it, so if your temporary directory is under such a home, set work_root to a location outside it (for example a dedicated data volume). For Codex, prefer a work_root outside the system temporary directory too: codex-cli 0.162.1 warns that it will not create its helper PATH aliases when its home is under a temporary directory. The provisioner also rejects ambient task AGENTS.md, CLAUDE.md, .claude, .codex, .agents and .mcp.json configurations rather than editing task files. Choose compatible tasks or deliberately implement and document a custom isolation policy.
The probe verifies native CLI versions, pinned skill/config content, HEAD, executable availability and an exercised hook. Because isolation is only a PATH, it also asks a login shell (bash -l, plus $SHELL when it is bash or zsh, the shells Claude Code runs; Codex runs commands through bash -lc) where ix is: a profile that resets PATH (Debian's /etc/profile, macOS path_helper) and so hides the ix arm's launcher or exposes a global ix to the baseline fails the run. So does a profile that exits non-zero under the arm's PATH; the error quotes the shell's stderr. It queries live ix status and ix stats; stale, incomplete, empty or unhealthy maps invalidate the run. Ix runs that produce nonempty patches must also have hook execution receipts. Version-specific native activation is still a real-run check: unit fixtures do not prove a particular installed CLI supports its plugin/notify flags.
Every stage receives one final absolute request-file argument. Custom setup/probe/agent/evaluator adapters remain possible through argv-array configs. Probe receipts contain version, repo/base/agent identity, ix_available, mapped, plugin_installed, skills_installed, hooks_enabled, and, for Ix, mapped_commit, plugin_revision and SHA-256 file evidence. The runner validates these fields and requires evidence inside the run directory. Evaluation receipts must agree with the official task report, whose saved bytes are hashed. Empty patches require the official harness's empty-patch evidence.
Measurements and interpretation
Artifacts include harness.json (script hashes), manifest.json, config.json, schedule.json, per-run agent-request.json/evaluate-request.json, per-stage logs, binary solution patches, official report evidence, runs.jsonl, report.json and summary.md. Patches compare against the task's pinned base, including agent commits and newly created files; integration files are excluded. Runs are sequential, with seed-randomized task/arm order and two repeats by default.
Native JSONL and normalized adapter telemetry record input/output/cached tokens, turns, tool calls, search counts, Ix calls and Ix errors by code. Claude input includes cache reads and cache creation; cached tokens are the cache-read subset. Claude turns counts only the main agent's assistant messages; messages from a subagent (Task/Explore: parent_tool_use_id set, or isSidechain) are not turns, matching what --max-turns limits, though their tool calls still count. Codex usage sums completed turns. Message/tool ids are deduplicated. Missing usage or terminal events invalidate a normally completed transcript. Budget deadlines may leave partial transcripts: observed tool/turn counters are retained, missing total token usage is null, and partial usage is saved separately as observed_usage. A final interrupted JSONL write is discarded only for a budget-limited agent stage. search_calls is shell searches (rg_calls, grep_calls, find_calls) plus Claude Code's built-in Grep/Glob tool calls, which are also reported as native_search_calls (Codex has no built-in search tool, so it is 0 there). Search/Ix shell counts cover direct commands and sh/bash/zsh -c wrappers with quote-aware tokenization, including ANSI-C $'...' quotes. An unbalanced quote never stops a run; the command is counted from what was parsed. Heredoc bodies are file content, not commands, so they are never counted (command substitutions inside an unquoted heredoc are missed too). Dynamic expansions, aliases and commands hidden inside scripts require normalized tool telemetry; counts are observed invocations, not a full shell execution trace. Hook Ix calls/errors are reported as hook_ix_calls/hook_ix_errors, excluding setup/probe calls; ix_calls/ix_errors count only the agent's own Ix use. An Ix call fails when its output shows an Ix error code or a shell "ix: not found" message, so ix … 2>&1 | head and ix … || rg … failures are not hidden by a zero exit status. A nonzero exit (or Claude tool error) counts as an Ix failure only when the whole command is one ix invocation (optionally inside bash -c); a failing ix map && npm test is an Ix failure only if its output shows one. In the baseline, a failed ix attempt (the probe proved there is no launcher) is recorded as an Ix call and error, and the run continues. A baseline Ix call that certainly succeeded is an isolation leak and stops the experiment: an Ix MCP call without an error, or a command that exited 0 and is a lone ix call or an && chain containing one. Commands whose status does not prove ix ran (ix --version || echo no ix, rg foo || ix text foo, ix … | head, …; ix …) never count as a leak. Claude permission denials are counted per run as permission_denials.
The common budget sets agent wall time and post-run turn/token ceilings; budget exhaustion records a run outcome and continues the remaining schedule. Agent deadlines terminate the process group, preserve logs and the current patch, and still evaluate that patch. Native Claude error_max_turns receipts are retained even when the CLI exits nonzero. Ordinary crashes and invalid setup/probe/evaluation stages remain fatal. The official harness writes no task report when a patch fails to apply or its tests time out. It lists those instances in the run summary's error_ids, and so does the scorer. When run_instance.log names either cause, the run is scored unresolved (report_kind: error-summary, evaluation_error). Any other evaluator error, such as an image build or Docker failure, stays fatal. Claude also receives --max-turns. Codex receives no native turn limit, and its turns metric counts codex exec turns (normally one per run), not model requests; for Codex, max_turns and turn_budget are effectively inert, so rely on the time and token ceilings and do not compare turns across agent families. Turn/token ceilings are checked against observed usage after execution. The token ceiling counts uncached input plus output: cache reads are excluded, because a long session re-reads its cached context every turn; overshoots are conservatively resolved: 0 because the final patch may include work beyond the declared limit. The official test verdict is retained as official_resolved. Time-limited or native turn-limited patches produced within the ceiling retain their official verdict. Each result has a termination reason (completed, time_budget, turn_budget, or token_budget). These are resource ceilings, not guaranteed billing caps. No retries occur. There is no resume: a fatal run error stops the experiment without report.json/summary.md; completed runs remain in runs.jsonl and the failing run's failure.json, and a rerun needs a fresh output directory. Clone, checkout, setup and probe each have an environment.seconds deadline; Git checks/patch extraction add bounded 10-second stages. Evaluation has its own deadline, including Docker image builds. Only agent-stage deadlines are retained as outcomes; setup, mapping and evaluation deadlines still fail the experiment. POSIX process groups are terminated at deadlines and reaped after stage completion. Windows isolation is not supported.
Reports include per-arm means/medians, mean and median differences with paired task-cluster percentile bootstrap 90% CIs (10,000 samples in the real config), and per-task descriptive rows for every metric. Repeats stay together when resampling tasks. Missing arms, unequal repeats and duplicate runs are errors. Resolution is a fraction; differences are Ix minus baseline. All scheduled runs remain in the resolution denominator, including budget-limited runs. For metrics with missing totals, per-arm summaries and CIs use only complete paired tasks; reports show paired-task coverage and observed-run counts. If no paired usage is available, usage estimates/CIs are explicitly unavailable. This conditional usage comparison may be biased by missingness and is not evidence of a cost advantage among timed-out tasks. wall_seconds is agent execution time, including synchronous hooks; setup/evaluation/total times are also recorded. Small sets can have degenerate intervals and cannot establish effectiveness. Real-task resolve rates remain unverified until a real run completes.
Reproduce from the same saved manifest/export, config, tool versions, Ix/harness commits, backend/image digests and agent model. Remote aliases can drift. CIs quantify task sampling uncertainty, not model drift, environment bias or billing prices. Review logs before publication. Local input data and results are gitignored.