Skip to content

armgit / zed

Public Imported from GitHub
0

Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.

eval_cli: Add remote benchmark orchestration (#59802)

Anant Goel10f501d7007c
diff --git a/crates/eval_cli/.gitignore b/crates/eval_cli/.gitignore
index a6317b892a1..6b8a8da509d 100644
--- a/crates/eval_cli/.gitignore
+++ b/crates/eval_cli/.gitignore
@@ -1,4 +1,10 @@
**/jobs
**/*.egg-info
**/__pycache__
+**/.venv
+**/.pytest_cache
+**/.ruff_cache
uv.lock
+
+# Experiment artifacts, that do not need to be committed
+experiments/*
diff --git a/crates/eval_cli/Dockerfile b/crates/eval_cli/Dockerfile
index 9782e5982b9..d67453f1faa 100644
--- a/crates/eval_cli/Dockerfile
+++ b/crates/eval_cli/Dockerfile
@@ -7,11 +7,13 @@
# Or use the helper script:
# crates/eval_cli/script/build-linux
-FROM rust:1.95.0 AS builder
+FROM rust:1.95.0@sha256:f49565f188ee00bc2a18dd418183f2c5f23ef7d6e691890517ed341a598f67c3 AS builder
+
+ARG CARGO_ZIGBUILD_VERSION=0.22.3
WORKDIR /app
- # Pre-install the toolchain specified in rust-toolchain.toml so it is cached.
+# Pre-install the toolchain specified in rust-toolchain.toml so it is cached.
RUN rustup toolchain install 1.95.0 --profile minimal \
--component rustfmt --component clippy --component rust-analyzer --component rust-src \
--target wasm32-wasip2 --target wasm32-unknown-unknown --target x86_64-unknown-linux-musl --target x86_64-unknown-linux-gnu
@@ -34,7 +36,7 @@ RUN mkdir -p /opt/zig \
| tar -xJ -C /opt/zig --strip-components=1 \
&& ln -s /opt/zig/zig /usr/local/bin/zig
-RUN cargo install --locked cargo-zigbuild
+RUN cargo install --locked cargo-zigbuild --version ${CARGO_ZIGBUILD_VERSION}
COPY . .
diff --git a/crates/eval_cli/README.md b/crates/eval_cli/README.md
index a9952bbf4fe..3cb54d55376 100644
--- a/crates/eval_cli/README.md
+++ b/crates/eval_cli/README.md
@@ -1,108 +1,81 @@
# eval-cli
-Headless CLI binary for running Zed's agent in evaluation/benchmark
-environments. Designed to work inside containerized environments like
-[Harbor](https://harborframework.com/) where the repository is already
-checked out and API keys are provided via environment variables.
+Headless Rust binary for running Zed's agent in evaluation and benchmark
+environments. It is designed for containerized harnesses such as
+[Harbor](https://harborframework.com/) and Pier, where the repository is already
+checked out and model API keys are provided via environment variables.
-Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed
-editor — full agentic loop with tool calls, subagents, and retries, just
-without a GUI.
+`eval-cli` uses the same `NativeAgent` + `AcpThread` pipeline as the production
+Zed editor: a full agentic loop with tool calls, subagents, and retries, without
+a GUI.
+
+This directory also contains `zed_eval/`, the Python `zed-eval` package used to
+build this binary, launch remote benchmark runs on Modal/Harbor/Pier, and fetch
+results. For normal benchmark orchestration, start with
+[`zed_eval/README.md`](zed_eval/README.md).
## Building
-### Native (for local testing on the same OS)
+### Native, for local testing on the same OS
-```
+```sh
cargo build --release -p eval_cli
```
-### Cross-compile for Linux x86_64 (from macOS or other hosts)
+### Linux x86_64, for Harbor/Pier sandboxes
-Harbor containers run Linux x86_64. Use the Docker-based build script:
+Harbor and Pier containers run Linux x86_64. From the repository root, use the
+Docker-based build script:
-```
+```sh
crates/eval_cli/script/build-linux
```
-This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can
-also specify a custom output path:
+This produces `target/eval-cli`, an x86_64 Linux ELF binary. You can also
+specify a custom output path:
-```
+```sh
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
```
## Standalone usage
-```
+```sh
eval-cli \
--workdir /testbed \
- --model anthropic/claude-sonnet-4-6-latest \
+ --model anthropic/claude-sonnet-4-6 \
--instruction "Fix the bug described in..." \
--timeout 600 \
--output-dir /logs/agent
```
-Reads API keys from environment variables (`ANTHROPIC_API_KEY`,
-`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and
-`thread.json` to the output directory.
+`eval-cli` reads provider API keys from environment variables such as
+`ANTHROPIC_API_KEY` and `OPENAI_API_KEY`. It writes `result.json`, `thread.md`,
+and `thread.json` to the output directory.
### Exit codes
-| Code | Meaning |
-| ---- | ---------------------------------- |
-| 0 | Agent finished |
-| 1 | Error (model/auth/runtime failure) |
-| 2 | Timeout |
-| 3 | Interrupted (SIGTERM/SIGINT) |
-
-## Harbor integration
+| Code | Meaning |
+| --- | --- |
+| 0 | Agent finished |
+| 1 | Error, such as model/auth/runtime failure |
+| 2 | Timeout |
+| 3 | Interrupted by SIGTERM or SIGINT |
-The `zed_eval/` directory contains a Python package that
-implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to
-be used with `--agent-import-path` without modifying Harbor's source code.
+## Running benchmarks
-### Setup
+Most benchmark runs should use the Python `zed-eval` CLI instead of invoking
+`eval-cli` directly. From the repository root:
+```sh
+crates/eval_cli/script/install-zed-eval
+zed-eval doctor --create-volume
+zed-eval run rf --from local --n-tasks 2
```
-pip install -e crates/eval_cli/harbor/
-```
-
-### Running with a local binary
-
-Build for Linux first, then pass the binary path:
-```
-crates/eval_cli/script/build-linux
+For one-off source runs without installing the tool globally, use
+`crates/eval_cli/script/zed-eval <args>`.
-harbor run -d "swebench_verified@latest" \
- --agent-import-path zed_eval.agent:ZedAgent \
- --ae binary_path=target/eval-cli \
- -m anthropic/claude-sonnet-4-6-latest
-```
-
-The agent uploads the binary into the container during setup — no
-download URL needed during local iteration.
-
-### Running with a download URL
-
-For CI or when the binary is hosted somewhere:
-
-```
-harbor run -d "swebench_verified@latest" \
- --agent-import-path zed_eval.agent:ZedAgent \
- --ak download_url=https://example.com/eval-cli \
- -m anthropic/claude-sonnet-4-6-latest
-```
-
-### Setting a timeout
-
-Pass `EVAL_CLI_TIMEOUT` via `--ae`:
-
-```
-harbor run -d "swebench_verified@latest" \
- --agent-import-path zed_eval.agent:ZedAgent \
- --ak binary_path=target/eval-cli \
- --ae EVAL_CLI_TIMEOUT=600 \
- -m anthropic/claude-sonnet-4-6-latest
-```
+See [`zed_eval/README.md`](zed_eval/README.md) for supported benchmarks, remote
+builds, Modal setup, reporting, rejudging, baselines, and Harbor/Pier installed
+agent usage.
diff --git a/crates/eval_cli/script/install-zed-eval b/crates/eval_cli/script/install-zed-eval
new file mode 100755
index 00000000000..dd4bfaea506
--- /dev/null
+++ b/crates/eval_cli/script/install-zed-eval
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Install or update the zed-eval Python CLI as an editable uv tool.
+#
+# Usage:
+# crates/eval_cli/script/install-zed-eval
+#
+# After this, `zed-eval` is available on PATH (assuming uv's tool bin directory
+# is on PATH) and tracks changes in crates/eval_cli/zed_eval without reinstalling.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+PACKAGE_DIR="$REPO_ROOT/crates/eval_cli/zed_eval"
+
+if ! command -v uv >/dev/null 2>&1; then
+ echo "error: uv is required to install zed-eval" >&2
+ echo "Install uv from https://docs.astral.sh/uv/getting-started/installation/" >&2
+ exit 1
+fi
+
+uv tool install --editable "$PACKAGE_DIR" --force
+
+echo ""
+echo "Installed zed-eval from $PACKAGE_DIR"
+echo "If your shell cannot find 'zed-eval', run: uv tool update-shell"
diff --git a/crates/eval_cli/script/zed-eval b/crates/eval_cli/script/zed-eval
new file mode 100755
index 00000000000..53ac35de521
--- /dev/null
+++ b/crates/eval_cli/script/zed-eval
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+#
+# Run the zed-eval Python CLI from this checkout without manually setting
+# PYTHONPATH or installing the tool globally.
+#
+# Usage:
+# crates/eval_cli/script/zed-eval doctor
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+PACKAGE_DIR="$REPO_ROOT/crates/eval_cli/zed_eval"
+
+if ! command -v uv >/dev/null 2>&1; then
+ echo "error: uv is required to run zed-eval from source" >&2
+ echo "Install uv from https://docs.astral.sh/uv/getting-started/installation/" >&2
+ exit 1
+fi
+
+exec uv run --project "$PACKAGE_DIR" zed-eval "$@"
diff --git a/crates/eval_cli/src/main.rs b/crates/eval_cli/src/main.rs
index 11e36914664..ef00fc87e24 100644
--- a/crates/eval_cli/src/main.rs
+++ b/crates/eval_cli/src/main.rs
@@ -15,7 +15,8 @@
//! ## Output
//!
//! Writes to `--output-dir` (default `/logs/agent/`):
-//! - `result.json` — structured result with status, timing, and token usage
+//! - `result.json` — structured result with status, timing, token usage,
+//! step count, and tool-call counts (total and per tool)
//! - `thread.md` — full conversation as markdown
//! - `thread.json` — raw thread state as JSON
//!
@@ -72,10 +73,14 @@ struct Args {
#[arg(long, default_value = ".")]
workdir: PathBuf,
- /// Instruction/prompt text. If omitted, read from --instruction-file or stdin.
+ /// Instruction/prompt text. If omitted, read from stdin.
#[arg(long, allow_hyphen_values = true)]
instruction: Option<String>,
+ /// File containing additional instruction text appended after the task prompt.
+ #[arg(long)]
+ instruction_suffix_file: Option<PathBuf>,
+
/// Language model to use, in `provider/model` format.
#[arg(long, default_value = "anthropic/claude-sonnet-4-6-latest")]
model: String,
@@ -125,6 +130,26 @@ struct EvalResult {
cache_creation_input_tokens: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
cache_read_input_tokens: Option<u64>,
+ /// Number of agent (assistant) turns, i.e. model round-trips in the agentic
+ /// loop. Reported as "steps" by the eval harness.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ step_count: Option<u64>,
+ /// Total number of tool calls across all steps.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ tool_call_count: Option<u64>,
+ /// Tool calls broken down by tool name.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ tool_calls: Option<std::collections::BTreeMap<String, u64>>,
+}
+
+/// Per-run statistics collected from the finished thread, written into
+/// `result.json` so the post-hoc report can compute success-conditioned metrics.
+#[derive(Default)]
+struct RunStats {
+ token_usage: Option<language_model::TokenUsage>,
+ step_count: Option<u64>,
+ tool_call_count: Option<u64>,
+ tool_calls: Option<std::collections::BTreeMap<String, u64>>,
}
const EXIT_OK: i32 = 0;
@@ -166,6 +191,10 @@ fn main() {
eprintln!("Error creating output dir {}: {e}", output_dir.display());
process::exit(EXIT_ERROR);
}
+ let output_dir = output_dir.canonicalize().unwrap_or_else(|e| {
+ eprintln!("Invalid --output-dir {:?}: {e}", output_dir);
+ process::exit(EXIT_ERROR);
+ });
let http_client = Arc::new(reqwest_client::ReqwestClient::new());
let app = gpui_platform::headless().with_http_client(http_client);
@@ -174,13 +203,23 @@ fn main() {
let app_state = headless::init(cx);
cx.set_staff(!args.no_staff);
- let auth_tasks = LanguageModelRegistry::global(cx).update(cx, |registry, cx| {
- registry
- .providers()
- .iter()
- .map(|p| p.authenticate(cx))
- .collect::<Vec<_>>()
- });
+ // Eval hook: enable additional feature-flag-gated tools (e.g. the LSP
+ // navigation tools behind `lsp-tool` / `rename-tool`) so experiments can
+ // measure the agent with tools that aren't yet GA. Comma-separated flag
+ // names; unset in production.
+ if let Ok(raw_flags) = std::env::var("ZED_EVAL_ENABLE_FLAGS") {
+ let flags: Vec<String> = raw_flags
+ .split(',')
+ .map(|flag| flag.trim().to_string())
+ .filter(|flag| !flag.is_empty())
+ .collect();
+ if !flags.is_empty() {
+ cx.update_flags(!args.no_staff, flags);
+ }
+ }
+
+ let openai_compatible_providers_json = openai_compatible_providers_override();
+ let anthropic_available_models_json = anthropic_available_models_override();
let model_name = args.model.clone();
let timeout = args.timeout;
@@ -188,11 +227,44 @@ fn main() {
let reasoning_effort = args.reasoning_effort.clone();
cx.spawn(async move |cx| {
+ // Each settings change below is applied in its own `cx.update` call (rather than
+ // inline in the synchronous body of `app.run`) so that GPUI flushes the resulting
+ // `NotifyGlobalObservers` effect before we move on. Without this, the
+ // openai_compatible/anthropic provider registration (driven by an
+ // `observe_global::<SettingsStore>` callback in language_models::init) wouldn't
+ // have run yet by the time we collect `auth_tasks`, so a newly-added provider's
+ // `authenticate()` would never get called and it would be permanently stuck
+ // unauthenticated.
+ if let Some(providers_json) = &openai_compatible_providers_json {
+ let result = cx.update(|cx| apply_openai_compatible_providers(providers_json, cx));
+ if let Err(e) = result {
+ eprintln!("Error applying {OPENAI_COMPATIBLE_PROVIDERS_ENV}: {e:#}");
+ process::exit(EXIT_ERROR);
+ }
+ }
+
+ if let Some(models_json) = &anthropic_available_models_json {
+ let result = cx.update(|cx| apply_anthropic_available_models(models_json, cx));
+ if let Err(e) = result {
+ eprintln!("Error applying {ANTHROPIC_AVAILABLE_MODELS_ENV}: {e:#}");
+ process::exit(EXIT_ERROR);
+ }
+ }
+
+ let auth_tasks = cx.update(|cx| {
+ LanguageModelRegistry::global(cx).update(cx, |registry, cx| {
+ registry
+ .providers()
+ .iter()
+ .map(|p| p.authenticate(cx))
+ .collect::<Vec<_>>()
+ })
+ });
futures::future::join_all(auth_tasks).await;
let start = Instant::now();
- let (outcome, token_usage) = run_agent(
+ let (outcome, stats) = run_agent(
&app_state,
&workdir,
&instruction,
@@ -201,6 +273,8 @@ fn main() {
thinking_override,
reasoning_effort.as_deref(),
Some(&output_dir),
+ openai_compatible_providers_json.as_deref(),
+ anthropic_available_models_json.as_deref(),
cx,
)
.await;
@@ -223,6 +297,7 @@ fn main() {
}
};
+ let token_usage = stats.token_usage;
let result = EvalResult {
status,
error,
@@ -239,6 +314,9 @@ fn main() {
.as_ref()
.filter(|u| u.cache_read_input_tokens > 0)
.map(|u| u.cache_read_input_tokens),
+ step_count: stats.step_count,
+ tool_call_count: stats.tool_call_count,
+ tool_calls: stats.tool_calls,
};
match serde_json::to_string_pretty(&result) {
@@ -258,8 +336,60 @@ fn main() {
});
}
+/// Name of the env var carrying a JSON object to merge into
+/// `language_models.openai_compatible` user settings before model discovery, in
+/// the same shape as Zed's `openai_compatible` settings key (provider id ->
+/// `{ "api_url": ..., "available_models": [...] }`). Lets zed-eval route the
+/// agent itself through an OpenAI-compatible endpoint (e.g. Baseten) that isn't
+/// one of Zed's built-in providers, without hardcoding it into eval-cli.
+const OPENAI_COMPATIBLE_PROVIDERS_ENV: &str = "ZED_OPENAI_COMPATIBLE_PROVIDERS";
+
+fn openai_compatible_providers_override() -> Option<String> {
+ let raw = std::env::var(OPENAI_COMPATIBLE_PROVIDERS_ENV).ok()?;
+ if raw.trim().is_empty() {
+ return None;
+ }
+ Some(raw)
+}
+
+fn apply_openai_compatible_providers(providers_json: &str, cx: &mut gpui::App) -> Result<()> {
+ let settings = format!(r#"{{"language_models": {{"openai_compatible": {providers_json}}}}}"#);
+ SettingsStore::update_global(cx, |store, cx| {
+ store.set_user_settings(&settings, cx).result()
+ })
+ .context("applying openai_compatible provider settings")?;
+ Ok(())
+}
+
+/// Name of the env var carrying a JSON array to merge into
+/// `language_models.anthropic.available_models` user settings before model
+/// discovery, in the same shape as Zed's `anthropic.available_models` settings
+/// key (a list of `{ "name": ..., "max_tokens": ..., ... }` entries). Lets
+/// zed-eval run models that exist on the Anthropic API for the configured
+/// key (e.g. early-access-program models) but aren't returned by the live
+/// `/v1/models` listing, without hardcoding them into eval-cli.
+const ANTHROPIC_AVAILABLE_MODELS_ENV: &str = "ZED_ANTHROPIC_AVAILABLE_MODELS";
+
+fn anthropic_available_models_override() -> Option<String> {
+ let raw = std::env::var(ANTHROPIC_AVAILABLE_MODELS_ENV).ok()?;
+ if raw.trim().is_empty() {
+ return None;
+ }
+ Some(raw)
+}
+
+fn apply_anthropic_available_models(models_json: &str, cx: &mut gpui::App) -> Result<()> {
+ let settings =
+ format!(r#"{{"language_models": {{"anthropic": {{"available_models": {models_json}}}}}}}"#);
+ SettingsStore::update_global(cx, |store, cx| {
+ store.set_user_settings(&settings, cx).result()
+ })
+ .context("applying anthropic available_models settings")?;
+ Ok(())
+}
+
fn read_instruction(args: &Args) -> Result<String> {
- let text = if let Some(text) = &args.instruction {
+ let mut text = if let Some(text) = &args.instruction {
text.clone()
} else {
use std::io::Read;
@@ -270,9 +400,23 @@ fn read_instruction(args: &Args) -> Result<String> {
buf
};
anyhow::ensure!(!text.trim().is_empty(), "instruction is empty");
+
+ if let Some(path) = &args.instruction_suffix_file {
+ let suffix = read_instruction_suffix_file(path)?;
+ text.push_str("\n\n");
+ text.push_str(&suffix);
+ }
Ok(text)
}
+fn read_instruction_suffix_file(path: &PathBuf) -> Result<String> {
+ let suffix = std::fs::read_to_string(path)
+ .with_context(|| format!("reading instruction suffix file {}", path.display()))?;
+ let suffix = suffix.trim().to_string();
+ anyhow::ensure!(!suffix.is_empty(), "instruction suffix file is empty");
+ Ok(suffix)
+}
+
async fn wait_for_model(selected: &SelectedModel, cx: &mut AsyncApp) -> Result<()> {
let started_at = Instant::now();
@@ -458,15 +602,17 @@ async fn run_agent(
thinking_override: Option<bool>,
reasoning_effort: Option<&str>,
output_dir: Option<&std::path::Path>,
+ openai_compatible_providers_json: Option<&str>,
+ anthropic_available_models_json: Option<&str>,
cx: &mut AsyncApp,
-) -> (Result<AgentOutcome>, Option<language_model::TokenUsage>) {
+) -> (Result<AgentOutcome>, RunStats) {
let selected = match SelectedModel::from_str(model_name).map_err(|e| anyhow::anyhow!("{e}")) {
Ok(selected) => selected,
- Err(e) => return (Err(e), None),
+ Err(e) => return (Err(e), RunStats::default()),
};
if let Err(e) = wait_for_model(&selected, cx).await {
- return (Err(e), None);
+ return (Err(e), RunStats::default());
}
let setup_result: Result<()> = cx.update(|cx| {
@@ -498,9 +644,79 @@ async fn run_agent(
"null".to_string()
};
let provider_id = selected.provider.0.to_string();
+ // set_user_settings replaces the whole user settings buffer, so the
+ // openai_compatible/anthropic blocks applied earlier (before model
+ // discovery) have to be folded back in here, or they would be dropped
+ // by this call.
+ let mut language_models_fields = Vec::new();
+ if let Some(providers_json) = openai_compatible_providers_json {
+ language_models_fields.push(format!(r#""openai_compatible": {providers_json}"#));
+ }
+ if let Some(models_json) = anthropic_available_models_json {
+ language_models_fields.push(format!(
+ r#""anthropic": {{"available_models": {models_json}}}"#
+ ));
+ }
+ let language_models_settings = format!("{{{}}}", language_models_fields.join(","));
+ // Disable specific tools (e.g. `fetch`/`search_web` on air-gapped
+ // benchmarks) the canonical way: via the agent profile. The model only
+ // sees a built-in tool when the active profile enables it (see
+ // Thread::enabled_tools), so we define a dedicated "eval" profile that
+ // mirrors the built-in "write" profile minus the disabled tools, and make
+ // it the default profile. A fresh profile key is NOT deep-merged against
+ // the defaults, so it can't inherit "write"'s tools — the full set is
+ // listed explicitly here. Keep WRITE_TOOLS in sync with the "write"
+ // profile in assets/settings/default.json.
+ let profile_field = {
+ let raw = std::env::var("ZED_EVAL_DISABLE_TOOLS").unwrap_or_default();
+ let disabled = raw
+ .split(',')
+ .map(|name| name.trim().to_string())
+ .filter(|name| !name.is_empty())
+ .collect::<std::collections::HashSet<String>>();
+ if disabled.is_empty() {
+ String::new()
+ } else {
+ const WRITE_TOOLS: &[&str] = &[
+ "copy_path",
+ "create_directory",
+ "create_thread",
+ "delete_path",
+ "diagnostics",
+ "apply_code_action",
+ "edit_file",
+ "write_file",
+ "fetch",
+ "find_path",
+ "find_references",
+ "get_code_actions",
+ "go_to_definition",
+ "list_agents_and_models",
+ "list_directory",
+ "move_path",
+ "rename_symbol",
+ "read_file",
+ "grep",
+ "skill",
+ "spawn_agent",
+ "terminal",
+ "search_web",
+ ];
+ let tools = WRITE_TOOLS
+ .iter()
+ .filter(|tool| !disabled.contains(**tool))
+ .map(|tool| format!(r#""{tool}": true"#))
+ .collect::<Vec<_>>()
+ .join(", ");
+ format!(
+ r#","default_profile": "eval", "profiles": {{"eval": {{"name": "Eval", "enable_all_context_servers": true, "tools": {{{tools}}}}}}}"#
+ )
+ }
+ };
SettingsStore::update_global(cx, |store, cx| {
let settings = format!(
r#"{{
+ "language_models": {language_models_settings},
"agent": {{
"tool_permissions": {{"default": "allow"}},
"default_model": {{
@@ -508,7 +724,7 @@ async fn run_agent(
"model": "{model_id}",
"enable_thinking": {enable_thinking},
"effort": {effort}
- }}
+ }}{profile_field}
}},
"autosave": "off",
"format_on_save": "off"
@@ -523,7 +739,7 @@ async fn run_agent(
});
if let Err(e) = setup_result {
- return (Err(e), None);
+ return (Err(e), RunStats::default());
}
let project = cx.update(|cx| {
@@ -545,7 +761,7 @@ async fn run_agent(
let worktree = project.update(cx, |project, cx| project.create_worktree(workdir, true, cx));
let worktree = match worktree.await {
Ok(w) => w,
- Err(e) => return (Err(e).context("creating worktree"), None),
+ Err(e) => return (Err(e).context("creating worktree"), RunStats::default()),
};
let scan_result = worktree.update(cx, |tree, _cx| {
@@ -555,9 +771,39 @@ async fn run_agent(
});
match scan_result {
Ok(future) => future.await,
- Err(e) => return (Err(e), None),
+ Err(e) => return (Err(e), RunStats::default()),
+ };
+
+ let output_worktree = match output_dir {
+ Some(output_dir) if !output_dir.starts_with(workdir) => {
+ let output_worktree = project.update(cx, |project, cx| {
+ project.create_worktree(output_dir, true, cx)
+ });
+ match output_worktree.await {
+ Ok(worktree) => Some(worktree),
+ Err(e) => {
+ return (
+ Err(e).context("creating output worktree"),
+ RunStats::default(),
+ );
+ }
+ }
+ }
+ _ => None,
};
+ if let Some(output_worktree) = output_worktree {
+ let scan_result = output_worktree.update(cx, |tree, _cx| {
+ tree.as_local()
+ .context("expected local output worktree")
+ .map(|local| local.scan_complete())
+ });
+ match scan_result {
+ Ok(future) => future.await,
+ Err(e) => return (Err(e), RunStats::default()),
+ };
+ }
+
let agent = cx.update(|cx| {
let thread_store = cx.new(|cx| ThreadStore::new(cx));
NativeAgent::new(thread_store, Templates::new(), app_state.fs.clone(), cx)
@@ -573,7 +819,7 @@ async fn run_agent(
.await
{
Ok(t) => t,
- Err(e) => return (Err(e).context("creating ACP session"), None),
+ Err(e) => return (Err(e).context("creating ACP session"), RunStats::default()),
};
let _subscription = cx.subscribe(&acp_thread, |acp_thread, event, cx| {
@@ -638,9 +884,30 @@ async fn run_agent(
connection.thread(&session_id, cx)
});
+ let mut step_count = None;
+ let mut tool_call_count = None;
+ let mut tool_calls = None;
let cumulative_usage = if let Some(thread) = &thread {
let db_thread = thread.read_with(cx, |thread, cx| thread.to_db(cx));
let db_thread = db_thread.await;
+ let mut counts = std::collections::BTreeMap::<String, u64>::new();
+ let mut agent_turn_count = 0;
+ for message in &db_thread.messages {
+ let Some(agent_message) = message.as_agent_message() else {
+ continue;
+ };
+ agent_turn_count += 1;
+ for request_message in agent_message.to_request() {
+ for content in request_message.content {
+ if let language_model::MessageContent::ToolUse(tool_use) = content {
+ *counts.entry(tool_use.name.to_string()).or_default() += 1;
+ }
+ }
+ }
+ }
+ step_count = Some(agent_turn_count);
+ tool_call_count = Some(counts.values().sum());
+ tool_calls = Some(counts);
let usage = db_thread.cumulative_token_usage;
if usage.input_tokens > 0 || usage.output_tokens > 0 {
Some(usage)
@@ -682,7 +949,15 @@ async fn run_agent(
}
}
- (outcome, final_usage)
+ (
+ outcome,
+ RunStats {
+ token_usage: final_usage,
+ step_count,
+ tool_call_count,
+ tool_calls,
+ },
+ )
}
fn log_acp_thread_event(
diff --git a/crates/eval_cli/zed_eval/README.md b/crates/eval_cli/zed_eval/README.md
new file mode 100644
index 00000000000..01dfca777c2
--- /dev/null
+++ b/crates/eval_cli/zed_eval/README.md
@@ -0,0 +1,348 @@
+# zed-eval
+
+Python CLI and installed-agent package for building Zed's headless `eval-cli`
+binary, launching benchmark runs on Modal/Harbor/Pier, and fetching results.
+
+This README is for `crates/eval_cli/zed_eval/`. The Rust `eval-cli` binary is
+documented in [`../README.md`](../README.md). Commands below assume they are run
+from the repository root unless otherwise noted.
+
+Most benchmark users should start with `zed-eval`.
+
+## Quick start
+
+Install the Python CLI as an editable tool:
+
+```sh
+crates/eval_cli/script/install-zed-eval
+```
+
+This wraps `uv tool install --editable crates/eval_cli/zed_eval --force`, so the
+installed `zed-eval` command tracks checkout changes without needing
+`PYTHONPATH`. If you prefer not to install globally, run from the checkout with:
+
+```sh
+crates/eval_cli/script/zed-eval doctor
+```
+
+Check your setup and create the Modal volume if needed:
+
+```sh
+zed-eval doctor --create-volume
+```
+
+Deploy the Modal app once before launching runs, and again after changing the
+Python orchestration code:
+
+```sh
+zed-eval deploy
+```
+
+Deploying replaces the live app and can cancel in-flight runs, so avoid running
+it while evals are active.
+
+Launch a small benchmark run from your current checkout:
+
+```sh
+zed-eval run rf --from local --n-tasks 2
+```
+
+Monitor and report:
+
+```sh
+zed-eval runs # recent runs launched from this machine
+zed-eval status # status for the most recent run
+zed-eval logs <run-id> # controller log
+zed-eval report <run-id> --fetch
+```
+
+After a launch, the local run index remembers the run's namespace and benchmark,
+so most commands only need the run id. If the run was launched elsewhere, pass
+`--experiment-name` and, if needed, `--namespace`.
+
+## Development
+
+Run the package tests without manually setting `PYTHONPATH`:
+
+```sh
+uv run --project crates/eval_cli/zed_eval python -m unittest discover -s crates/eval_cli/zed_eval/tests
+```
+
+## Prerequisites
+
+You need:
+
+- access to the Modal workspace used for evals
+- a Modal token secret for the controller, default: `agent-evals-modal-token`
+- an LLM-provider secret for agent/judge API keys, default:
+ `agent-evals-llm-providers`
+
+The controller secret should contain:
+
+- `MODAL_TOKEN_ID`
+- `MODAL_TOKEN_SECRET`
+
+The LLM-provider secret should contain the keys your selected models need, such as:
+
+- `ANTHROPIC_API_KEY`
+- `OPENAI_API_KEY`
+- `BASETEN_API_KEY`
+
+Defaults can be overridden globally:
+
+```sh
+zed-eval \
+ --volume agent-evals \
+ --api-secret agent-evals-llm-providers \
+ --modal-token-secret agent-evals-modal-token \
+ doctor
+```
+
+## Launch benchmarks
+
+Use `zed-eval run` for non-interactive launches:
+
+```sh
+# One SWE-Atlas part
+zed-eval run rf --from local --n-tasks 5
+
+# Multiple benchmarks share one build when possible
+zed-eval run swe-atlas terminal-bench-2.1 --from v0.210.0 --n-tasks 5 --yes
+
+# DeepSWE runs under Pier automatically
+zed-eval run deepswe --from local --n-tasks 10
+
+# Inspect what would happen without launching
+zed-eval run swe-atlas --from v0.210.0 --plan
+zed-eval run deepswe --from local --dry-run
+```
+
+Supported benchmark selectors:
+
+| Selector | Meaning | Scoring |
+| --- | --- | --- |
+| `swe-atlas` | `qna`, `rf`, and `tw` | LLM judge |
+| `qna` / `swe-atlas-qna` | SWE-Atlas Codebase Q&A | LLM judge |
+| `rf` / `swe-atlas-rf` | SWE-Atlas Refactoring | LLM judge |
+| `tw` / `swe-atlas-tw` | SWE-Atlas Test Writing | LLM judge |
+| `terminal-bench-2.1` / `tb21` | Terminal-Bench 2.1 | tests |
+| `deepswe` | DeepSWE | tests |
+
+For an interactive prompt, use:
+
+```sh
+zed-eval swe-atlas --interactive
+```
+
+Despite the command name, the interactive flow can launch SWE-Atlas parts,
+Terminal-Bench, and DeepSWE.
+
+## Choose source and builds
+
+`--from` controls what source is built into `eval-cli`:
+
+- `--from local` builds current `HEAD` plus tracked changes.
+- `--from <ref/tag/sha>` builds a clean git ref, tag, or SHA.
+
+Builds are content-addressed and reused when possible. You can also name or reuse
+a build explicitly:
+
+```sh
+zed-eval build --from local
+zed-eval run rf --build bld-abc123 --n-tasks 2
+zed-eval builds --details
+```
+
+Untracked files are not included in builds. If they are irrelevant, opt in:
+
+```sh
+zed-eval run rf --from local --allow-untracked --n-tasks 2
+```
+
+For reproducible runs, use a clean ref/tag/SHA or require a clean tracked tree:
+
+```sh
+zed-eval run rf --from v0.210.0 --n-tasks 2
+zed-eval run rf --from local --require-clean --n-tasks 2
+```
+
+## Choose models and judges
+
+Default agent model:
+
+- `sonnet-4.6` → `anthropic/claude-sonnet-4-6`
+
+Common agent model examples:
+
+```sh
+zed-eval run rf --model sonnet-4.6 --n-tasks 2
+zed-eval run rf --model opus-4.5 --n-tasks 2
+zed-eval run rf --model baseten:kimi-k2.7-code --n-tasks 2
+zed-eval run rf --model baseten:deepseek-v4-pro --n-tasks 2
+```
+
+For SWE-Atlas judge defaults, `--judge auto` uses:
+
+- `qna`: `deepseek-v4-pro`
+- `rf` and `tw`: `kimi-k2.7-code`
+
+Override the judge when needed:
+
+```sh
+zed-eval run rf --judge leaderboard --n-tasks 2
+zed-eval run rf --judge deepseek-v4-pro --judge-model deepseek-ai/DeepSeek-V4-Pro
+```
+
+Baseten presets generate the OpenAI-compatible provider settings automatically.
+The sandbox secret must include `BASETEN_API_KEY`.
+
+## Monitor, fetch, and report
+
+Useful commands:
+
+```sh
+zed-eval runs # local, fast recent-run list
+zed-eval list --details # query runs on the Modal volume
+zed-eval list -e swe-atlas-rf --details
+zed-eval status <run-id>
+zed-eval logs <run-id>
+zed-eval fetch <run-id>
+zed-eval report <run-id> --fetch
+```
+
+`fetch` downloads the run archive and extracts it to:
+
+```text
+~/.cache/harbor/jobs/<run-id>/
+```
+
+`report` prints pass rate plus resource usage such as tokens, tool calls, and
+agent steps. Use `--json` for machine-readable output.
+
+## Multi-benchmark suites
+
+Launching more than one benchmark in one command groups the runs under a
+`suite_id` stored on each run.
+
+```sh
+zed-eval run qna rf terminal-bench-2.1 --from local --n-tasks 2 --yes
+```
+
+Suite commands operate on all runs in that group:
+
+```sh
+zed-eval suite status <suite-id>
+zed-eval suite logs <suite-id>
+zed-eval suite fetch <suite-id>
+```
+
+## Rejudge a finished run
+
+`rejudge` creates a new derived run by re-running only the judge. It does not redo
+the agent's work or modify the parent run.
+
+```sh
+zed-eval rejudge <parent-run-id> --judge deepseek-v4-pro
+zed-eval rejudge <parent-run-id> --judge kimi-k2.7-code --dry-run
+zed-eval report <derived-run-id> --fetch
+```
+
+If the parent run is not in your local run index, pass `--experiment-name` and
+possibly `--namespace`.
+
+## Baselines
+
+Baseline commands record and inspect baseline-of-record results for completed
+clean-commit runs:
+
+```sh
+zed-eval baseline record <run-id> --experiment-name swe-atlas-rf
+zed-eval baseline list
+zed-eval baseline show swe-atlas-rf --model anthropic/claude-sonnet-4-6
+```
+
+## Storage model
+
+Runs and builds live on the shared Modal volume:
+
+```text
+builds/<build-id>/
+ eval-cli
+ build-info.json
+ source-info.json
+ source.patch
+
+runs/<namespace>/<experiment-name>/<run-id>/
+ request.json
+ state.json
+ controller.log
+ harbor-command.txt
+ run-metadata.json
+ harbor-job.tar.gz
+ summary.json
+```
+
+Namespaces prevent accidental collisions but are not access control. Anyone with
+access to the Modal workspace/volume can read run manifests, logs, patches, and
+archives.
+
+## Standalone eval-cli
+
+For local or custom harness usage, you can build and invoke the Rust binary
+directly.
+
+Build for the current platform:
+
+```sh
+cargo build --release -p eval_cli
+```
+
+Build the Linux binary used by Harbor/Pier sandboxes:
+
+```sh
+crates/eval_cli/script/build-linux
+```
+
+Run directly:
+
+```sh
+eval-cli \
+ --workdir /testbed \
+ --model anthropic/claude-sonnet-4-6 \
+ --instruction "Fix the bug described in..." \
+ --timeout 600 \
+ --output-dir /logs/agent
+```
+
+`eval-cli` reads provider API keys from environment variables and writes
+`result.json`, `thread.md`, and `thread.json` to the output directory.
+
+Exit codes:
+
+| Code | Meaning |
+| --- | --- |
+| 0 | Agent finished |
+| 1 | Error, such as model/auth/runtime failure |
+| 2 | Timeout |
+| 3 | Interrupted by SIGTERM or SIGINT |
+
+## Harbor/Pier installed agent
+
+The Python package also exposes installed-agent classes used by the remote
+orchestrator:
+
+- `zed_eval.agent:ZedAgent` for Harbor
+- `zed_eval.pier_agent:ZedPierAgent` for Pier
+
+For manual Harbor experiments with a locally built Linux binary:
+
+```sh
+pip install -e crates/eval_cli/zed_eval/
+crates/eval_cli/script/build-linux
+
+harbor run -d "swebench_verified@latest" \
+ --agent-import-path zed_eval.agent:ZedAgent \
+ --ae binary_path=target/eval-cli \
+ --ae EVAL_CLI_TIMEOUT=600 \
+ -m anthropic/claude-sonnet-4-6
+```
diff --git a/crates/eval_cli/zed_eval/__init__.py b/crates/eval_cli/zed_eval/__init__.py
index 8cf07a06883..8b137891791 100644
--- a/crates/eval_cli/zed_eval/__init__.py
+++ b/crates/eval_cli/zed_eval/__init__.py
@@ -1,3 +1 @@
-from zed_eval.agent import ZedAgent
-__all__ = ["ZedAgent"]
diff --git a/crates/eval_cli/zed_eval/agent.py b/crates/eval_cli/zed_eval/agent.py
index 4720a7dbc13..50135583f78 100644
--- a/crates/eval_cli/zed_eval/agent.py
+++ b/crates/eval_cli/zed_eval/agent.py
@@ -1,40 +1,33 @@
-"""Harbor agent wrapper for Zed's eval-cli binary.
-
-Usage:
- # Build eval-cli locally first:
- cargo build --release -p eval_cli
-
- # Run via Harbor with a local binary:
- harbor run -d "dataset@version" \
- --agent-import-path zed_eval.agent:ZedAgent \
- --ae binary_path=/path/to/target/release/eval-cli \
- --agent-model anthropic/claude-sonnet-4-6-latest
-
- # Or with a download URL (for CI):
- harbor run -d "dataset@version" \
- --agent-import-path zed_eval.agent:ZedAgent \
- --ae download_url=https://example.com/eval-cli \
- --agent-model anthropic/claude-sonnet-4-6-latest
-"""
+"""Harbor installed-agent adapter for Zed's eval-cli binary."""
import json
import os
import shlex
+import tomllib
from pathlib import Path
from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
+from .agent_common import (
+ add_anthropic_available_models_env,
+ add_openai_compatible_provider_env,
+ add_zed_eval_env,
+ detect_workdir,
+ eval_cli_with_log_command,
+ patch_command,
+ populate_context_from_result,
+ provider_api_env,
+)
-class ZedAgent(BaseInstalledAgent):
- """Runs Zed's headless AI agent (eval-cli) to solve tasks.
+# Leave eval-cli time to exit cleanly and flush logs before Harbor's outer
+# asyncio.wait_for can kill the agent coroutine.
+EVAL_CLI_FINALIZE_BUFFER_SEC = 45
+EVAL_CLI_MIN_TIMEOUT_SEC = 60
- The eval-cli binary boots a headless GPUI application and uses the same
- NativeAgent + AcpThread pipeline as the production Zed editor, driving
- the full agentic loop (tool calls, subagents, retries) without a GUI.
- """
+class ZedAgent(BaseInstalledAgent):
def __init__(
self,
logs_dir: Path,
@@ -52,52 +45,12 @@ class ZedAgent(BaseInstalledAgent):
return "zed"
async def _detect_workdir(self, environment: BaseEnvironment) -> str:
- """Detect the working directory inside the container.
-
- Checks, in order:
- 1. Explicit ``EVAL_CLI_WORKDIR`` extra-env override
- 2. Well-known dirs with a ``.git`` subdirectory (SWE-bench style)
- 3. First git repo found under ``/`` (max depth 3)
- 4. Well-known dirs that exist at all (terminal-bench style)
- 5. The container's default working directory (``pwd``)
- """
- override = self._extra_env.get("EVAL_CLI_WORKDIR")
- if override:
- return override
-
- # First: try to find a git repo (SWE-bench, etc.)
- result = await self.exec_as_agent(
- environment,
- command=(
- "for d in /app /testbed /repo; do "
- ' if [ -d "$d/.git" ]; then echo "$d"; exit 0; fi; '
- "done; "
- "find / -maxdepth 3 -name .git -type d 2>/dev/null "
- '| head -1 | sed "s|/.git$||"'
- ),
- )
- workdir = (result.stdout or "").strip()
- if workdir:
- return workdir
-
- # Fallback: use the first well-known directory that exists,
- # even without .git (terminal-bench containers aren't git repos).
- result = await self.exec_as_agent(
+ return await detect_workdir(
environment,
- command=(
- "for d in /app /testbed /repo /root /home; do "
- ' if [ -d "$d" ]; then echo "$d"; exit 0; fi; '
- "done; "
- "pwd"
- ),
- )
- workdir = (result.stdout or "").strip()
- if workdir:
- return workdir
-
- raise RuntimeError(
+ self.exec_as_agent,
+ self._extra_env.get,
"Could not detect a working directory in the container. "
- "Set EVAL_CLI_WORKDIR explicitly via --ae EVAL_CLI_WORKDIR=/path/to/repo"
+ "Set EVAL_CLI_WORKDIR explicitly via --ae EVAL_CLI_WORKDIR=/path/to/repo",
)
async def install(self, environment: BaseEnvironment) -> None:
@@ -123,16 +76,25 @@ class ZedAgent(BaseInstalledAgent):
env={"DEBIAN_FRONTEND": "noninteractive"},
)
- # ── Non-essential tooling ─────────────────────────────────────
- # Everything below here (Node.js, LSPs, uv/ruff) is nice-to-have.
- # If any step fails (e.g. musl incompatibility, network issues),
- # log a warning and continue — the agent can still work without
- # pre-installed language servers.
-
+ # Tooling setup is best-effort; benchmark images vary enough that one
+ # unsupported runtime should not fail the trial before eval-cli starts.
await self._install_node(environment)
await self._install_lsps(environment)
await self._install_uv_and_ruff(environment)
+ # Modal can mount a prebuilt binary inside each sandbox, avoiding uploads.
+ container_path = self._extra_env.get("EVAL_CLI_CONTAINER_PATH")
+ if container_path:
+ await self.exec_as_root(
+ environment,
+ command=(
+ f"cp {shlex.quote(container_path)} /usr/local/bin/eval-cli && "
+ "chmod +x /usr/local/bin/eval-cli && "
+ "eval-cli --help"
+ ),
+ )
+ return
+
if self._binary_path:
binary = Path(self._binary_path)
if not binary.exists():
@@ -164,8 +126,9 @@ class ZedAgent(BaseInstalledAgent):
raise ValueError(
"No eval-cli binary provided. "
- "Either pass binary_path=/path/to/target/release/eval-cli "
- "or set download_url=/EVAL_CLI_DOWNLOAD_URL."
+ "Either pass binary_path=/path/to/target/release/eval-cli, "
+ "set download_url=/EVAL_CLI_DOWNLOAD_URL, "
+ "or set --ae EVAL_CLI_CONTAINER_PATH=/path/inside/container."
)
async def _install_node(self, environment: BaseEnvironment) -> None:
@@ -346,53 +309,100 @@ class ZedAgent(BaseInstalledAgent):
self.logger.warning("uv/ruff installation failed (non-fatal): %s", exc)
def populate_context_post_run(self, context: AgentContext) -> None:
- result_data = None
- for json_file in self.logs_dir.rglob("result.json"):
- try:
- result_data = json.loads(json_file.read_text())
- break
- except (json.JSONDecodeError, OSError):
- continue
-
- if result_data is None:
- self.logger.warning("Could not find or parse result.json from eval-cli")
- return
+ populate_context_from_result(self.logs_dir, context, self.logger)
- if result_data.get("input_tokens") is not None:
- context.n_input_tokens = result_data["input_tokens"]
- if result_data.get("output_tokens") is not None:
- context.n_output_tokens = result_data["output_tokens"]
- if result_data.get("cache_read_input_tokens") is not None:
- context.n_cache_tokens = result_data["cache_read_input_tokens"]
+ def _get_api_env(self) -> dict[str, str]:
+ env = provider_api_env(self.model_name)
+ add_openai_compatible_provider_env(
+ env, self._extra_env.get("ZED_OPENAI_COMPATIBLE_PROVIDERS")
+ )
+ add_anthropic_available_models_env(
+ env, self._extra_env.get("ZED_ANTHROPIC_AVAILABLE_MODELS")
+ )
+ return env
- context.metadata = {
- "status": result_data.get("status"),
- "duration_secs": result_data.get("duration_secs"),
- "model": result_data.get("model"),
- }
+ def _harbor_agent_budget_sec(self) -> float | None:
+ """Recover Harbor's hidden outer timeout so eval-cli can exit first.
- def _get_api_env(self) -> dict[str, str]:
- env: dict[str, str] = {}
- if not self.model_name or "/" not in self.model_name:
- return env
-
- provider = self.model_name.split("/", 1)[0]
- provider_env_map = {
- "anthropic": "ANTHROPIC_API_KEY",
- "openai": "OPENAI_API_KEY",
- "google": "GEMINI_API_KEY",
- "gemini": "GEMINI_API_KEY",
- "deepseek": "DEEPSEEK_API_KEY",
- "mistral": "MISTRAL_API_KEY",
- }
-
- env_var = provider_env_map.get(provider)
- if env_var:
- api_key = os.environ.get(env_var, "")
- if api_key:
- env[env_var] = api_key
+ Harbor 0.15.0 computes this on ``Trial`` but does not pass it to custom
+ installed agents. Replaying the calculation host-side lets us preserve
+ trajectories instead of losing them to Harbor's coroutine kill.
+ """
+ config_path = Path(self.logs_dir).parent / "config.json"
+ try:
+ trial_config = json.loads(config_path.read_text())
+ except (OSError, ValueError) as exc:
+ print(
+ f"[zed-eval] agent budget: could not read {config_path}: {exc!r}",
+ flush=True,
+ )
+ return None
- return env
+ agent_cfg = trial_config.get("agent") or {}
+ base_sec = agent_cfg.get("override_timeout_sec")
+ if base_sec is None:
+ base_sec = self._task_declared_agent_timeout_sec(
+ trial_config.get("task") or {}
+ )
+ if base_sec is None:
+ return None
+
+ max_sec = agent_cfg.get("max_timeout_sec")
+ if max_sec is not None:
+ base_sec = min(base_sec, max_sec)
+ multiplier = trial_config.get("agent_timeout_multiplier")
+ if multiplier is None:
+ multiplier = trial_config.get("timeout_multiplier", 1.0)
+ return float(base_sec) * float(multiplier)
+
+ def _task_declared_agent_timeout_sec(self, task_cfg: dict) -> float | None:
+ task_dir = self._resolve_task_dir(task_cfg)
+ if task_dir is None:
+ return None
+ task_toml = task_dir / "task.toml"
+ try:
+ data = tomllib.loads(task_toml.read_text())
+ except (OSError, ValueError) as exc:
+ print(
+ f"[zed-eval] agent budget: could not parse {task_toml}: {exc!r}",
+ flush=True,
+ )
+ return None
+ timeout_sec = (data.get("agent") or {}).get("timeout_sec")
+ return float(timeout_sec) if timeout_sec is not None else None
+
+ def _resolve_task_dir(self, task_cfg: dict) -> Path | None:
+ path = task_cfg.get("path")
+ if path:
+ return Path(path).expanduser().resolve()
+
+ # config.json omits Harbor's package content hash; use the newest cached
+ # digest that contains a task.toml.
+ name = task_cfg.get("name")
+ if name and "/" in name:
+ try:
+ from harbor.constants import PACKAGE_CACHE_DIR
+ except ImportError as exc:
+ print(
+ f"[zed-eval] agent budget: harbor PACKAGE_CACHE_DIR "
+ f"unavailable: {exc!r}",
+ flush=True,
+ )
+ return None
+ download_dir = task_cfg.get("download_dir")
+ base_dir = Path(download_dir) if download_dir else PACKAGE_CACHE_DIR
+ org, package = name.split("/", 1)
+ package_root = base_dir / org / package
+ candidates = [
+ digest_dir
+ for digest_dir in package_root.glob("*")
+ if (digest_dir / "task.toml").is_file()
+ ]
+ if not candidates:
+ return None
+ return max(candidates, key=lambda p: p.stat().st_mtime)
+
+ return None
@with_prompt_template
async def run(
@@ -401,6 +411,10 @@ class ZedAgent(BaseInstalledAgent):
escaped_instruction = shlex.quote(instruction)
env = self._get_api_env()
+ add_zed_eval_env(
+ env, self._extra_env, exclude={"ZED_EVAL_INSTRUCTION_SUFFIX_FILE"}
+ )
+
workdir = await self._detect_workdir(environment)
parts = [
@@ -412,9 +426,58 @@ class ZedAgent(BaseInstalledAgent):
if self.model_name:
parts.append(f"--model {shlex.quote(self.model_name)}")
- timeout = self._extra_env.get("EVAL_CLI_TIMEOUT")
- if timeout:
- parts.append(f"--timeout {shlex.quote(timeout)}")
+ instruction_suffix_file = self._extra_env.get(
+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE"
+ )
+ if instruction_suffix_file:
+ parts.append(
+ f"--instruction-suffix-file {shlex.quote(instruction_suffix_file)}"
+ )
+
+ # Prefer eval-cli's own timeout over Harbor's coroutine kill so logs and
+ # partial answers are still available for delivery.
+ configured_timeout_raw = self._extra_env.get("EVAL_CLI_TIMEOUT")
+ configured_timeout: int | None
+ if configured_timeout_raw:
+ try:
+ configured_timeout = int(configured_timeout_raw)
+ except ValueError:
+ print(
+ "[zed-eval] ignoring non-integer EVAL_CLI_TIMEOUT="
+ f"{configured_timeout_raw!r}",
+ flush=True,
+ )
+ configured_timeout = None
+ else:
+ configured_timeout = None
+
+ budget_sec = self._harbor_agent_budget_sec()
+ timeout_arg: int | None = None
+ if budget_sec is not None:
+ ceiling = (
+ configured_timeout
+ if configured_timeout is not None
+ else int(budget_sec)
+ )
+ timeout_arg = min(ceiling, int(budget_sec - EVAL_CLI_FINALIZE_BUFFER_SEC))
+ timeout_arg = max(timeout_arg, EVAL_CLI_MIN_TIMEOUT_SEC)
+ print(
+ f"[zed-eval] eval-cli --timeout {timeout_arg}s (harbor agent "
+ f"budget {int(budget_sec)}s - {EVAL_CLI_FINALIZE_BUFFER_SEC}s "
+ f"finalize buffer; EVAL_CLI_TIMEOUT="
+ f"{configured_timeout if configured_timeout is not None else 'unset'})",
+ flush=True,
+ )
+ elif configured_timeout is not None:
+ timeout_arg = configured_timeout
+ print(
+ f"[zed-eval] eval-cli --timeout {timeout_arg}s (harbor agent "
+ "budget unavailable; using EVAL_CLI_TIMEOUT)",
+ flush=True,
+ )
+
+ if timeout_arg is not None:
+ parts.append(f"--timeout {timeout_arg}")
staff = self._extra_env.get("EVAL_CLI_STAFF")
if staff and staff.lower() == "false":
@@ -427,39 +490,41 @@ class ZedAgent(BaseInstalledAgent):
enable_thinking = self._extra_env.get("EVAL_CLI_ENABLE_THINKING")
if enable_thinking:
if enable_thinking.lower() == "true":
- parts.append("--enable-thinking")
+ parts.append("--thinking true")
elif enable_thinking.lower() == "false":
- parts.append("--disable-thinking")
+ parts.append("--thinking false")
parts.append(f"--instruction {escaped_instruction}")
+ # Exit 2 is eval-cli's timeout, not a crash; keep the trajectory judgeable.
await self.exec_as_agent(
environment,
- command=(
- " ".join(parts) + " 2>&1 | if command -v stdbuf >/dev/null 2>&1;"
- " then stdbuf -oL tee /logs/agent/eval-cli.txt;"
- " else tee /logs/agent/eval-cli.txt; fi"
+ command=eval_cli_with_log_command(
+ parts,
+ "/logs/agent/eval-cli.txt",
+ timeout_message=(
+ "[zed-eval] eval-cli timed out (exit 2); continuing to "
+ "delivery/verification"
+ ),
+ line_buffered=True,
),
env=env,
)
- # Only generate a patch if the workdir is a git repo with a valid HEAD
- # (SWE-bench style). Terminal-bench containers aren't git repos, and
- # some harnesses mount an initialized repo before creating the first commit.
+ # Modal-style remote runs can differ in agent-log collection behavior.
await self.exec_as_agent(
environment,
command=(
- "if git rev-parse --git-dir >/dev/null 2>&1; then "
- "git add -A && "
- "if git rev-parse --verify HEAD >/dev/null 2>&1; then "
- "git diff --cached HEAD -- > /logs/agent/patch.diff && "
- 'echo "Patch size: $(wc -c < /logs/agent/patch.diff) bytes"; '
- "else "
- 'echo "Git repo has no valid HEAD, skipping patch generation"; '
- "fi; "
- "else "
- 'echo "No git repo found, skipping patch generation"; '
- "fi"
+ "mkdir -p /logs/artifacts && "
+ "for f in result.json thread.json thread.md eval-cli.txt; do "
+ ' cp "/logs/agent/$f" /logs/artifacts/ 2>/dev/null || true; '
+ "done; true"
),
+ )
+
+ # Some harnesses mount an initialized repo before creating the first commit.
+ await self.exec_as_agent(
+ environment,
+ command=patch_command("/logs/agent"),
cwd=workdir,
)
diff --git a/crates/eval_cli/zed_eval/agent_common.py b/crates/eval_cli/zed_eval/agent_common.py
new file mode 100644
index 00000000000..48079f39ca2
--- /dev/null
+++ b/crates/eval_cli/zed_eval/agent_common.py
@@ -0,0 +1,184 @@
+from __future__ import annotations
+
+import json
+import os
+import shlex
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any
+
+PROVIDER_ENV_MAP = {
+ "anthropic": "ANTHROPIC_API_KEY",
+ "openai": "OPENAI_API_KEY",
+ "google": "GEMINI_API_KEY",
+ "gemini": "GEMINI_API_KEY",
+ "deepseek": "DEEPSEEK_API_KEY",
+ "mistral": "MISTRAL_API_KEY",
+}
+
+
+def provider_api_env(
+ model_name: str | None,
+ get_env: Callable[[str], str | None] = os.environ.get,
+) -> dict[str, str]:
+ env: dict[str, str] = {}
+ if not model_name or "/" not in model_name:
+ return env
+
+ provider = model_name.split("/", 1)[0]
+ env_var = PROVIDER_ENV_MAP.get(provider) or (
+ f"{provider}_API_KEY".upper().replace("-", "_")
+ )
+ api_key = get_env(env_var)
+ if api_key:
+ env[env_var] = api_key
+ return env
+
+
+def add_openai_compatible_provider_env(
+ env: dict[str, str], providers_json: str | None
+) -> None:
+ if providers_json:
+ env["ZED_OPENAI_COMPATIBLE_PROVIDERS"] = providers_json
+
+
+def add_anthropic_available_models_env(
+ env: dict[str, str], models_json: str | None
+) -> None:
+ if models_json:
+ env["ZED_ANTHROPIC_AVAILABLE_MODELS"] = models_json
+
+
+def add_zed_eval_env(
+ env: dict[str, str], extra_env: dict[str, str], *, exclude: set[str] | None = None
+) -> None:
+ exclude = exclude or set()
+ for key, value in extra_env.items():
+ if key.startswith("ZED_EVAL_") and key not in exclude:
+ env[key] = value
+
+
+async def detect_workdir(
+ environment: Any,
+ exec_as_agent: Callable[..., Any],
+ get_env: Callable[[str], str | None],
+ error_message: str,
+) -> str:
+ override = get_env("EVAL_CLI_WORKDIR")
+ if override:
+ return override
+
+ result = await exec_as_agent(
+ environment,
+ command=(
+ "for d in /app /testbed /repo; do "
+ ' if [ -d "$d/.git" ]; then echo "$d"; exit 0; fi; '
+ "done; "
+ "find / -maxdepth 3 -name .git -type d 2>/dev/null "
+ '| head -1 | sed "s|/.git$||"'
+ ),
+ )
+ workdir = (result.stdout or "").strip()
+ if workdir:
+ return workdir
+
+ result = await exec_as_agent(
+ environment,
+ command=(
+ "for d in /app /testbed /repo /root /home; do "
+ ' if [ -d "$d" ]; then echo "$d"; exit 0; fi; '
+ "done; pwd"
+ ),
+ )
+ workdir = (result.stdout or "").strip()
+ if workdir:
+ return workdir
+ raise RuntimeError(error_message)
+
+
+def populate_context_from_result(logs_dir: Path, context: Any, logger: Any) -> None:
+ result_data = None
+ for json_file in logs_dir.rglob("result.json"):
+ try:
+ result_data = json.loads(json_file.read_text())
+ break
+ except (json.JSONDecodeError, OSError):
+ continue
+
+ if result_data is None:
+ logger.warning("Could not find or parse result.json from eval-cli")
+ return
+
+ if result_data.get("input_tokens") is not None:
+ context.n_input_tokens = result_data["input_tokens"]
+ if result_data.get("output_tokens") is not None:
+ context.n_output_tokens = result_data["output_tokens"]
+ if result_data.get("cache_read_input_tokens") is not None:
+ context.n_cache_tokens = result_data["cache_read_input_tokens"]
+ if isinstance(result_data.get("step_count"), int):
+ context.n_agent_steps = result_data["step_count"]
+
+ context.metadata = {
+ "status": result_data.get("status"),
+ "duration_secs": result_data.get("duration_secs"),
+ "model": result_data.get("model"),
+ "tool_call_count": result_data.get("tool_call_count"),
+ }
+
+
+def eval_cli_with_log_command(
+ parts: list[str],
+ log_path: str,
+ *,
+ timeout_message: str | None = None,
+ line_buffered: bool = False,
+) -> str:
+ """Run eval-cli, tee output, and preserve eval-cli's exit status.
+
+ POSIX shells return the last command's status for a pipeline, so `cmd | tee`
+ would otherwise hide eval-cli failures whenever `tee` succeeds.
+ """
+ status_file = "/tmp/zed-eval-eval-cli-status"
+ quoted_status_file = shlex.quote(status_file)
+ quoted_log_path = shlex.quote(log_path)
+ timeout_handler = 'if [ "$ec" -eq 2 ]; then ec=0; fi; '
+ if timeout_message:
+ timeout_handler = (
+ f'if [ "$ec" -eq 2 ]; then echo {shlex.quote(timeout_message)}; ec=0; fi; '
+ )
+ tee_command = f"tee {quoted_log_path}"
+ if line_buffered:
+ tee_command = (
+ "if command -v stdbuf >/dev/null 2>&1; "
+ f"then stdbuf -oL tee {quoted_log_path}; "
+ f"else tee {quoted_log_path}; fi"
+ )
+ return (
+ f'status_file={quoted_status_file}; rm -f "$status_file"; '
+ "( "
+ + " ".join(parts)
+ + "; ec=$?; "
+ + timeout_handler
+ + 'printf "%s\\n" "$ec" > "$status_file"; '
+ + 'exit "$ec" ) 2>&1 | '
+ + tee_command
+ + '; ec=1; if [ -s "$status_file" ]; then read ec < "$status_file"; fi; '
+ + 'rm -f "$status_file"; exit "$ec"'
+ )
+
+
+def patch_command(agent_dir: str) -> str:
+ patch_path = shlex.quote(f"{agent_dir}/patch.diff")
+ return (
+ "if git rev-parse --git-dir >/dev/null 2>&1; then "
+ "git add -A && "
+ "if git rev-parse --verify HEAD >/dev/null 2>&1; then "
+ f"git diff --cached HEAD -- > {patch_path} && "
+ f'echo "Patch size: $(wc -c < {patch_path}) bytes"; '
+ "else "
+ 'echo "Git repo has no valid HEAD, skipping patch generation"; '
+ "fi; "
+ "else "
+ 'echo "No git repo found, skipping patch generation"; '
+ "fi"
+ )
diff --git a/crates/eval_cli/zed_eval/baseline.py b/crates/eval_cli/zed_eval/baseline.py
new file mode 100644
index 00000000000..a1dcf547443
--- /dev/null
+++ b/crates/eval_cli/zed_eval/baseline.py
@@ -0,0 +1,205 @@
+"""Baseline-of-record management.
+
+A "baseline" is a completed run promoted to the canonical reference for its
+`(experiment, model)` pair. It only qualifies if its build came from a *clean*
+commit reachable from `origin/main` (no local patch) — so baselines are both
+obviously discoverable (the `baselines/` tree + `zed-eval baseline list`) and
+provably pinned to a commit on main. The judge preset is recorded but not gated,
+so non-leaderboard baselines are allowed.
+
+Records live on the shared volume at:
+
+ baselines/index.json # roll-up for discovery
+ baselines/<experiment>/<model-slug>/current.json # canonical pointer
+ baselines/<experiment>/<model-slug>/history/<sha>.json # superseded baselines
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from typing import Any
+
+from . import config, source
+from .common import default_namespace, deployed_function, utc_now
+
+
+def _model_slug(model: str) -> str:
+ return source.sanitize_namespace(model)
+
+
+def _record_from_provenance(
+ *,
+ namespace: str,
+ fallback_experiment: str,
+ run_id: str,
+ provenance: dict[str, Any],
+) -> tuple[dict[str, Any], bool]:
+ request = provenance["request"]
+ summary = provenance["summary"]
+ build_info = provenance["build_info"]
+ source_info = build_info.get("source") or {}
+
+ experiment = request.get("experiment_name") or fallback_experiment
+ model = request.get("agent_model") or config.DEFAULT_MODEL
+ base_sha = build_info.get("base_sha")
+
+ record = {
+ "experiment": experiment,
+ "model": model,
+ "model_slug": _model_slug(model),
+ "judge": request.get("judge_preset") or request.get("judge"),
+ "base_sha": base_sha,
+ "base_ref": source_info.get("base_ref"),
+ "on_main": None,
+ "clean": not bool(build_info.get("patch_sha256")),
+ "build_id": build_info.get("build_id") or request.get("build_id"),
+ "run": {
+ "namespace": namespace,
+ "experiment_name": experiment,
+ "run_id": run_id,
+ },
+ "resources": {
+ "override_cpus": request.get("override_cpus"),
+ "override_memory_mb": request.get("override_memory_mb"),
+ },
+ "summary": {
+ "status": summary.get("status"),
+ "trial_count": summary.get("trial_count"),
+ },
+ "recorded_at": utc_now(),
+ "recorded_by": namespace,
+ }
+ return record, bool(build_info)
+
+
+def _baseline_problems(
+ *,
+ record: dict[str, Any],
+ has_build_info: bool,
+ repo_url: str,
+ allow_dirty: bool,
+ allow_off_main: bool,
+) -> tuple[list[str], bool | None]:
+ problems: list[str] = []
+ run_id = (record.get("run") or {}).get("run_id")
+ status = (record.get("summary") or {}).get("status")
+ base_sha = record.get("base_sha")
+
+ if not has_build_info:
+ problems.append(
+ f"no build-info found for run {run_id}; cannot verify provenance"
+ )
+ if status != "completed":
+ problems.append(f"run status is {status!r}, expected 'completed'")
+ if not record.get("clean") and not allow_dirty:
+ problems.append(
+ "build carries a local patch (not a clean commit); pass --allow-dirty to override"
+ )
+
+ on_main: bool | None = None
+ if base_sha:
+ try:
+ on_main = source.base_sha_on_main(base_sha, repo_url)
+ except Exception as error: # noqa: BLE001 - surface any verification failure
+ if not allow_off_main:
+ problems.append(f"could not verify base_sha is on origin/main: {error}")
+ if on_main is False and not allow_off_main:
+ problems.append(
+ f"base_sha {base_sha[:12]} is not reachable from origin/main; "
+ "pass --allow-off-main to override"
+ )
+ elif not allow_off_main:
+ problems.append("build-info has no base_sha to verify against origin/main")
+
+ return problems, on_main
+
+
+def _print_refusal(run_id: str, problems: list[str]) -> None:
+ print(f"refusing to record baseline for {run_id}:")
+ for problem in problems:
+ print(f" - {problem}")
+
+
+def _print_recorded(record: dict[str, Any]) -> None:
+ print(
+ f"recorded baseline: {record.get('experiment')} / {record.get('model')} "
+ f"base_sha={str(record.get('base_sha'))[:12]} "
+ f"on_main={record.get('on_main')} clean={record.get('clean')} "
+ f"judge={record.get('judge')}"
+ )
+
+
+def command_baseline_record(args: argparse.Namespace) -> int:
+ namespace = default_namespace(args)
+ experiment_slug = source.sanitize_namespace(args.experiment_name)
+ repo_url = getattr(args, "repo_url", None) or source.DEFAULT_REPO_URL
+ read_run_provenance = deployed_function(args, "read_run_provenance")
+ record_baseline = deployed_function(args, "record_baseline")
+
+ exit_code = 0
+ for run_id in args.run_id:
+ provenance = read_run_provenance.remote(namespace, experiment_slug, run_id)
+ record, has_build_info = _record_from_provenance(
+ namespace=namespace,
+ fallback_experiment=args.experiment_name,
+ run_id=run_id,
+ provenance=provenance,
+ )
+ problems, on_main = _baseline_problems(
+ record=record,
+ has_build_info=has_build_info,
+ repo_url=repo_url,
+ allow_dirty=args.allow_dirty,
+ allow_off_main=args.allow_off_main,
+ )
+ record["on_main"] = on_main
+
+ if problems:
+ _print_refusal(run_id, problems)
+ exit_code = 1
+ continue
+
+ record_baseline.remote(record)
+ _print_recorded(record)
+ return exit_code
+
+
+def command_baseline_list(args: argparse.Namespace) -> int:
+ index = deployed_function(args, "read_baselines").remote(None, None, False)
+ rows = index.get("baselines") or []
+ if getattr(args, "json", False):
+ print(json.dumps(index, indent=2))
+ return 0
+ if not rows:
+ print("no baselines recorded")
+ return 0
+ header = (
+ f"{'EXPERIMENT':<20} {'MODEL':<32} {'BASE_REF':<14} "
+ f"{'SHA':<12} {'ON_MAIN':<8} {'JUDGE':<14} RUN"
+ )
+ print(header)
+ for row in rows:
+ print(
+ f"{(row.get('experiment') or ''):<20} "
+ f"{(row.get('model') or ''):<32} "
+ f"{(row.get('base_ref') or '') or '-':<14} "
+ f"{str(row.get('base_sha') or '')[:12]:<12} "
+ f"{str(row.get('on_main')):<8} "
+ f"{(row.get('judge') or '') or '-':<14} "
+ f"{row.get('run_id') or ''}"
+ )
+ return 0
+
+
+def command_baseline_show(args: argparse.Namespace) -> int:
+ result = deployed_function(args, "read_baselines").remote(
+ args.experiment_name,
+ _model_slug(args.model),
+ bool(getattr(args, "history", False)),
+ )
+ if not result or result.get("current") is None:
+ print(f"no baseline recorded for {args.experiment_name} / {args.model}")
+ return 1
+ print(json.dumps(result, indent=2))
+ return 0
diff --git a/crates/eval_cli/zed_eval/benchmarks.py b/crates/eval_cli/zed_eval/benchmarks.py
new file mode 100644
index 00000000000..489ac1baabf
--- /dev/null
+++ b/crates/eval_cli/zed_eval/benchmarks.py
@@ -0,0 +1,239 @@
+"""Benchmark registry.
+
+A `Benchmark` describes everything the orchestrator needs to run one benchmark
+end-to-end that is *not* about the build source or the model: which harness runs
+it (Harbor or its Pier fork), how its dataset is provisioned, how a trial is
+scored, whether it needs an LLM judge, and the per-task timeout.
+
+Adding a benchmark should be a data change here, not new control flow. The three
+benchmark families currently supported are all Harbor-family harnesses:
+
+ - SWE-Atlas (qna / rf / tw): Harbor, rubric LLM judge.
+ - Terminal-Bench 2.1: Harbor, test-script scoring, no judge.
+ - DeepSWE: Pier (a Harbor fork with per-agent network allowlists, required
+ because DeepSWE tasks run with `allow_internet = false`), test scoring.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+# How a benchmark's dataset is made available to the harness.
+# registry -> `-d <name>` pulled from the harness hub
+# path -> `-p <dir>` from a git repo cloned by the controller
+# pier_path -> path dataset run under Pier instead of Harbor
+DATASET_REGISTRY = "registry"
+DATASET_PATH = "path"
+DATASET_PIER_PATH = "pier_path"
+
+HARNESS_HARBOR = "harbor"
+HARNESS_PIER = "pier"
+
+SCORING_RUBRIC_JUDGE = "rubric-judge"
+SCORING_TESTS = "tests"
+
+
+@dataclass(frozen=True)
+class DatasetRef:
+ kind: str
+ # For registry datasets: the hub dataset name. For path datasets: the repo
+ # URL plus a ref and the sub-directory the tasks live in.
+ name: str | None = None
+ repo_url: str | None = None
+ repo_ref: str | None = None
+ data_dir: str | None = None
+
+
+@dataclass(frozen=True)
+class Benchmark:
+ id: str
+ label: str
+ harness: str
+ dataset: DatasetRef
+ default_timeout_secs: int
+ scoring: str
+ needs_judge: bool
+ default_judge: str | None = None
+ # Hosts the in-sandbox agent must reach even on air-gapped tasks. Required by
+ # Pier benchmarks (DeepSWE) so eval-cli can still call the model API.
+ network_allowlist: tuple[str, ...] = ()
+ env: dict[str, str] = field(default_factory=dict)
+
+
+SWE_ATLAS_REPO_URL = "https://github.com/scaleapi/SWE-Atlas.git"
+SWE_ATLAS_REPO_REF = "main"
+DEEPSWE_REPO_URL = "https://github.com/datacurve-ai/deep-swe.git"
+DEEPSWE_REPO_REF = "main"
+
+# Model/judge API hosts the agent needs even under `allow_internet = false`.
+# Pier grants these to the agent via its per-agent allowlist.
+AGENT_API_HOSTS: tuple[str, ...] = (
+ "api.anthropic.com",
+ "api.openai.com",
+ "inference.baseten.co",
+)
+
+
+BENCHMARKS: dict[str, Benchmark] = {
+ "swe-atlas-qna": Benchmark(
+ id="swe-atlas-qna",
+ label="SWE-Atlas Codebase Q&A",
+ harness=HARNESS_HARBOR,
+ dataset=DatasetRef(kind=DATASET_REGISTRY, name="scale-ai/swe-atlas-qna"),
+ default_timeout_secs=7200,
+ scoring=SCORING_RUBRIC_JUDGE,
+ needs_judge=True,
+ default_judge="deepseek-v4-pro",
+ ),
+ "swe-atlas-rf": Benchmark(
+ id="swe-atlas-rf",
+ label="SWE-Atlas Refactoring",
+ harness=HARNESS_HARBOR,
+ dataset=DatasetRef(kind=DATASET_REGISTRY, name="scale-ai/swe-atlas-rf"),
+ default_timeout_secs=3300,
+ scoring=SCORING_RUBRIC_JUDGE,
+ needs_judge=True,
+ default_judge="kimi-k2.7-code",
+ ),
+ "swe-atlas-tw": Benchmark(
+ id="swe-atlas-tw",
+ label="SWE-Atlas Test Writing",
+ harness=HARNESS_HARBOR,
+ # Test writing is not consistently published in the same registry shape as qna/rf.
+ dataset=DatasetRef(
+ kind=DATASET_PATH,
+ repo_url=SWE_ATLAS_REPO_URL,
+ repo_ref=SWE_ATLAS_REPO_REF,
+ data_dir="data/tw",
+ ),
+ default_timeout_secs=3300,
+ scoring=SCORING_RUBRIC_JUDGE,
+ needs_judge=True,
+ default_judge="kimi-k2.7-code",
+ ),
+ "terminal-bench-2.1": Benchmark(
+ id="terminal-bench-2.1",
+ label="Terminal-Bench 2.1",
+ harness=HARNESS_HARBOR,
+ dataset=DatasetRef(
+ kind=DATASET_REGISTRY, name="terminal-bench/terminal-bench-2-1"
+ ),
+ default_timeout_secs=3300,
+ scoring=SCORING_TESTS,
+ needs_judge=False,
+ # Air-gapped tasks: the fetch/web-search tools can't reach the network,
+ # so disable them (via the agent profile in eval-cli) to stop the agent
+ # wasting its budget on tools that can only fail.
+ env={"ZED_EVAL_DISABLE_TOOLS": "fetch,search_web"},
+ ),
+ "deepswe": Benchmark(
+ id="deepswe",
+ label="DeepSWE",
+ harness=HARNESS_PIER,
+ dataset=DatasetRef(
+ kind=DATASET_PIER_PATH,
+ repo_url=DEEPSWE_REPO_URL,
+ repo_ref=DEEPSWE_REPO_REF,
+ # DeepSWE keeps its Harbor-compatible tasks at the repo root.
+ data_dir="tasks",
+ ),
+ default_timeout_secs=7200,
+ scoring=SCORING_TESTS,
+ needs_judge=False,
+ network_allowlist=AGENT_API_HOSTS,
+ # Air-gapped except the model API allowlist; fetch/web-search are useless.
+ env={"ZED_EVAL_DISABLE_TOOLS": "fetch,search_web"},
+ ),
+}
+
+
+# Groups expand to multiple benchmarks in one launch.
+BENCHMARK_GROUPS: dict[str, tuple[str, ...]] = {
+ "swe-atlas": ("swe-atlas-qna", "swe-atlas-rf", "swe-atlas-tw"),
+}
+
+# One short alias per benchmark whose canonical id is verbose.
+BENCHMARK_ALIASES: dict[str, str] = {
+ "qna": "swe-atlas-qna",
+ "rf": "swe-atlas-rf",
+ "tw": "swe-atlas-tw",
+ "tb21": "terminal-bench-2.1",
+}
+
+SWE_ATLAS_PART_BENCHMARKS: dict[str, str] = {
+ "qna": "swe-atlas-qna",
+ "rf": "swe-atlas-rf",
+ "tw": "swe-atlas-tw",
+}
+
+
+def get_benchmark(benchmark_id: str) -> Benchmark:
+ try:
+ return BENCHMARKS[benchmark_id]
+ except KeyError as error:
+ valid = ", ".join(sorted(BENCHMARKS))
+ raise ValueError(
+ f"unknown benchmark '{benchmark_id}' (valid: {valid})"
+ ) from error
+
+
+def is_benchmark_selector(selector: str) -> bool:
+ """Whether `selector` names a known benchmark id, alias, or group.
+
+ Used by `run` to validate benchmark positionals before preparing builds."""
+ normalized = selector.strip().lower()
+ return (
+ normalized in BENCHMARK_GROUPS
+ or normalized in BENCHMARKS
+ or normalized in BENCHMARK_ALIASES
+ )
+
+
+def resolve_benchmark_selector(selector: str) -> list[str]:
+ """Expands a user-facing selector (benchmark id, alias, or group) into the
+ concrete benchmark ids it refers to, preserving order."""
+ normalized = selector.strip().lower()
+ if normalized in BENCHMARK_GROUPS:
+ return list(BENCHMARK_GROUPS[normalized])
+ if normalized in BENCHMARKS:
+ return [normalized]
+ if normalized in BENCHMARK_ALIASES:
+ return [BENCHMARK_ALIASES[normalized]]
+ valid = ", ".join(sorted({*BENCHMARKS, *BENCHMARK_GROUPS, *BENCHMARK_ALIASES}))
+ raise ValueError(f"unknown benchmark '{selector}' (valid: {valid})")
+
+
+def resolve_benchmarks(selectors: list[str]) -> list[str]:
+ resolved: list[str] = []
+ for selector in selectors:
+ for part in selector.split(","):
+ part = part.strip()
+ if not part:
+ continue
+ for benchmark_id in resolve_benchmark_selector(part):
+ if benchmark_id not in resolved:
+ resolved.append(benchmark_id)
+ return resolved
+
+
+def benchmark_metadata(benchmark: Benchmark) -> dict[str, object]:
+ """The self-describing block embedded in a run request so the controller and
+ harness-command builder need no separate registry lookup."""
+ return {
+ "id": benchmark.id,
+ "label": benchmark.label,
+ "harness": benchmark.harness,
+ "dataset": {
+ "kind": benchmark.dataset.kind,
+ "name": benchmark.dataset.name,
+ "repo_url": benchmark.dataset.repo_url,
+ "repo_ref": benchmark.dataset.repo_ref,
+ "data_dir": benchmark.dataset.data_dir,
+ },
+ "default_timeout_secs": benchmark.default_timeout_secs,
+ "scoring": benchmark.scoring,
+ "needs_judge": benchmark.needs_judge,
+ "default_judge": benchmark.default_judge,
+ "network_allowlist": list(benchmark.network_allowlist),
+ "env": dict(benchmark.env),
+ }
diff --git a/crates/eval_cli/zed_eval/builds.py b/crates/eval_cli/zed_eval/builds.py
new file mode 100644
index 00000000000..9ce2fdd409d
--- /dev/null
+++ b/crates/eval_cli/zed_eval/builds.py
@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+import argparse
+from typing import Any
+
+from . import source
+
+
+def validate_build_id(build_id: str | None) -> None:
+ if build_id and source.sanitize_namespace(build_id) != build_id:
+ raise ValueError(
+ "build ids may only contain lowercase letters, numbers, '.', '_', and '-'"
+ )
+
+
+def prepare_build_request(
+ *,
+ base_sha: str | None,
+ patch_path: str | None,
+ build_id: str | None,
+ allow_untracked: bool,
+ require_clean: bool,
+ repo_url: str | None,
+ clean_source: bool = False,
+ source_label: str | None = None,
+ pre_resolved_base_sha: str | None = None,
+) -> dict[str, Any]:
+ validate_build_id(build_id)
+ source_info, patch = source.prepare_build_source(
+ base_sha=base_sha,
+ patch_path=patch_path,
+ allow_untracked=allow_untracked,
+ require_clean=require_clean,
+ repo_url=repo_url,
+ clean=clean_source,
+ source_label=source_label,
+ pre_resolved_base_sha=pre_resolved_base_sha,
+ )
+ resolved_build_id = build_id or source.compute_build_id(source_info=source_info)
+ return {
+ "build_id": resolved_build_id,
+ "base_sha": source_info["base_sha"],
+ "patch": patch,
+ "patch_sha256": source_info.get("patch_sha256"),
+ "source": source_info,
+ "toolchain": source.build_toolchain_info(),
+ }
+
+
+def resolve_source(
+ args: argparse.Namespace,
+) -> tuple[str | None, bool, str | None, str | None]:
+ """Unify the source selectors into
+ `(base_sha, clean_source, source_label, pre_resolved_base_sha)`.
+
+ Precedence: `--from` wins over `--zed-version` / `--base-sha`.
+
+ --from local -> current HEAD + tracked patch (dev iteration)
+ --from <ref/tag/sha> -> clean build of that ref, resolved canonically
+ against the remote so everyone shares one build
+ """
+ repo_url = getattr(args, "repo_url", None) or source.DEFAULT_REPO_URL
+ from_source = getattr(args, "from_source", None)
+ if from_source:
+ if from_source.strip().lower() == "local":
+ return (
+ getattr(args, "base_sha", None),
+ bool(getattr(args, "clean_source", False)),
+ None,
+ None,
+ )
+ resolved = source.resolve_remote_ref(repo_url, from_source)
+ return None, True, from_source, resolved
+
+ zed_version = getattr(args, "zed_version", None)
+ if zed_version:
+ resolved = source.resolve_remote_ref(repo_url, zed_version)
+ return None, True, zed_version, resolved
+
+ return (
+ getattr(args, "base_sha", None),
+ bool(getattr(args, "clean_source", False)),
+ None,
+ None,
+ )
diff --git a/crates/eval_cli/zed_eval/cleanup.py b/crates/eval_cli/zed_eval/cleanup.py
new file mode 100644
index 00000000000..03470fe8b60
--- /dev/null
+++ b/crates/eval_cli/zed_eval/cleanup.py
@@ -0,0 +1,93 @@
+"""Prune stale build artifacts and build cache from the shared volume.
+
+Eval results are sacrosanct: this only ever touches the build-related top-level
+directories listed in `PRUNABLE_DIRS`. `runs/` (the eval outputs) is never in
+that set and is never walked, so a bug here cannot delete results.
+
+The logic is pure-filesystem (no Modal dependency) so it can be unit-tested
+against a temporary directory; `modal_app` wraps it with volume reload/commit and
+a daily schedule.
+"""
+
+from __future__ import annotations
+
+import datetime
+import shutil
+import time
+from pathlib import Path
+from typing import Any
+
+# The ONLY top-level volume directories cleanup may modify. Eval results live in
+# runs/, which is deliberately absent here.
+PRUNABLE_DIRS = frozenset({"builds", "build-locks", "tmp"})
+
+DEFAULT_BUILD_RETENTION_DAYS = 14.0
+DEFAULT_LOCK_TTL_HOURS = 6.0
+DEFAULT_TMP_MAX_AGE_HOURS = 24.0
+
+
+def _age_days(path: Path) -> float:
+ try:
+ return (time.time() - path.stat().st_mtime) / 86400.0
+ except OSError:
+ return 0.0
+
+
+def prune_artifacts(
+ data_root: Path | str,
+ *,
+ dry_run: bool = False,
+ build_retention_days: float = DEFAULT_BUILD_RETENTION_DAYS,
+ lock_ttl_hours: float = DEFAULT_LOCK_TTL_HOURS,
+ tmp_max_age_hours: float = DEFAULT_TMP_MAX_AGE_HOURS,
+) -> dict[str, Any]:
+ data_root = Path(data_root)
+ removed: dict[str, list[Any]] = {
+ "builds": [],
+ "locks": [],
+ "tmp": [],
+ }
+
+ # 1. Stale content-addressed builds, aged by their READY marker (or the dir
+ # itself for never-finished builds). Cheap to rebuild on demand.
+ builds_dir = data_root / "builds"
+ if builds_dir.is_dir():
+ for build_dir in sorted(builds_dir.iterdir()):
+ if not build_dir.is_dir():
+ continue
+ ready = build_dir / "READY"
+ age = _age_days(ready if ready.exists() else build_dir)
+ if age > build_retention_days:
+ removed["builds"].append(
+ {"build_id": build_dir.name, "age_days": round(age, 1)}
+ )
+ if not dry_run:
+ shutil.rmtree(build_dir, ignore_errors=True)
+
+ # 2. Orphaned single-flight build leases left by crashed builds.
+ locks_dir = data_root / "build-locks"
+ if locks_dir.is_dir():
+ for lock in sorted(locks_dir.glob("*.json")):
+ if _age_days(lock) * 24.0 > lock_ttl_hours:
+ removed["locks"].append(lock.name)
+ if not dry_run:
+ lock.unlink(missing_ok=True)
+
+ # 3. Orphaned temp build dirs from failed/raced builds.
+ tmp_builds = data_root / "tmp" / "builds"
+ if tmp_builds.is_dir():
+ for entry in sorted(tmp_builds.iterdir()):
+ if _age_days(entry) * 24.0 > tmp_max_age_hours:
+ removed["tmp"].append(entry.name)
+ if not dry_run:
+ if entry.is_dir():
+ shutil.rmtree(entry, ignore_errors=True)
+ else:
+ entry.unlink(missing_ok=True)
+
+ return {
+ "dry_run": dry_run,
+ "at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
+ "removed": removed,
+ "counts": {key: len(value) for key, value in removed.items()},
+ }
diff --git a/crates/eval_cli/zed_eval/cli.py b/crates/eval_cli/zed_eval/cli.py
new file mode 100644
index 00000000000..e93c701b9c8
--- /dev/null
+++ b/crates/eval_cli/zed_eval/cli.py
@@ -0,0 +1,787 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+from . import benchmarks, config, source
+from .baseline import (
+ command_baseline_list,
+ command_baseline_record,
+ command_baseline_show,
+)
+from .common import (
+ AppNotDeployedError,
+ command_exists,
+ default_namespace,
+ deploy_app,
+ deployed_function,
+ print_json,
+)
+from .launch import command_build, command_rejudge, command_run, command_swe_atlas
+from .report import command_report
+from .volume import (
+ command_builds,
+ command_fetch,
+ command_list,
+ command_logs,
+ command_runs,
+ command_status,
+ command_suite_fetch,
+ command_suite_logs,
+ command_suite_status,
+ ensure_volume_exists,
+)
+
+
+def modal_secret_names() -> set[str]:
+ result = subprocess.run(
+ ["modal", "secret", "list", "--json"],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ data = json.loads(result.stdout)
+ if not isinstance(data, list):
+ return set()
+ return {
+ item["name"] for item in data if isinstance(item, dict) and item.get("name")
+ }
+
+
+def command_doctor(args: argparse.Namespace) -> int:
+ namespace = default_namespace(args)
+ print("zed-eval doctor")
+ print(f" namespace: {namespace}")
+ print(f" app: {args.app_name}")
+ print(f" volume: {args.volume}")
+ print(f" api secret: {args.api_secret}")
+ print(f" modal token secret: {args.modal_token_secret}")
+ print(f" repo root: {source.repo_root()}")
+ print(f" base sha: {source.current_base_sha()}")
+ print(f" default model: {config.DEFAULT_MODEL}")
+ print(" default judges: qna=deepseek-v4-pro, rf/tw=kimi-k2.7-code")
+
+ missing = []
+ for executable in ("git", "modal", "harbor"):
+ if command_exists(executable):
+ print(f" {executable}: found")
+ else:
+ print(f" {executable}: missing")
+ missing.append(executable)
+
+ if missing:
+ print("\nMissing required executables: " + ", ".join(missing), file=sys.stderr)
+ return 1
+
+ try:
+ secrets = modal_secret_names()
+ except (subprocess.CalledProcessError, json.JSONDecodeError) as error:
+ print(
+ f" secrets: could not list Modal secrets: {error}",
+ file=sys.stderr,
+ )
+ else:
+ missing_secrets = [
+ secret_name
+ for secret_name in (args.api_secret, args.modal_token_secret)
+ if secret_name not in secrets
+ ]
+ if missing_secrets:
+ print(
+ "\nMissing Modal secret(s): " + ", ".join(missing_secrets),
+ file=sys.stderr,
+ )
+ print(
+ "Use --api-secret/--modal-token-secret to point at existing secrets, "
+ "or create the missing ones with `modal secret create`.",
+ file=sys.stderr,
+ )
+ return 1
+ print(" secrets: found")
+ print(
+ " controller token: use a dedicated Modal service-user token for production"
+ )
+
+ if args.create_volume:
+ return ensure_volume_exists(args)
+ return 0
+
+
+def command_deploy(args: argparse.Namespace) -> int:
+ deploy_app(args)
+ print(f"Deployed Modal app '{args.app_name}'.")
+ return 0
+
+
+def command_cleanup(args: argparse.Namespace) -> int:
+ request: dict[str, object] = {"dry_run": bool(args.dry_run)}
+ if args.build_retention_days is not None:
+ request["build_retention_days"] = args.build_retention_days
+ result = deployed_function(args, "cleanup_artifacts").remote(request)
+ print_json(result)
+ return 0
+
+
+def env_default(
+ variable_name: str, default: object, *, include_defaults: bool = True
+) -> object:
+ if not include_defaults:
+ return argparse.SUPPRESS
+ return os.environ.get(variable_name, default)
+
+
+def add_common_options(
+ parser: argparse.ArgumentParser, *, include_defaults: bool = True
+) -> None:
+ default_volume = env_default(
+ "AGENT_EVALS_VOLUME",
+ config.DEFAULT_VOLUME_NAME,
+ include_defaults=include_defaults,
+ )
+ default_namespace = None if include_defaults else argparse.SUPPRESS
+ default_api_secret = env_default(
+ "AGENT_EVALS_LLM_PROVIDERS_SECRET",
+ config.DEFAULT_LLM_PROVIDERS_SECRET_NAME,
+ include_defaults=include_defaults,
+ )
+ default_modal_token_secret = env_default(
+ "AGENT_EVALS_MODAL_TOKEN_SECRET",
+ config.DEFAULT_MODAL_TOKEN_SECRET_NAME,
+ include_defaults=include_defaults,
+ )
+ default_app_name = env_default(
+ "AGENT_EVALS_APP_NAME",
+ config.DEFAULT_APP_NAME,
+ include_defaults=include_defaults,
+ )
+ infra = parser.add_argument_group(
+ "infra options", "Modal app/volume/namespace and secret names (rarely changed)"
+ )
+ infra.add_argument(
+ "--app-name",
+ default=default_app_name,
+ help=f"Deployed Modal app name (default: {config.DEFAULT_APP_NAME})",
+ )
+ infra.add_argument(
+ "--volume",
+ default=default_volume,
+ help=f"Modal volume for builds/runs (default: {config.DEFAULT_VOLUME_NAME})",
+ )
+ infra.add_argument(
+ "--namespace",
+ default=default_namespace,
+ help="Run namespace inside the shared volume (default: env/git user/local user)",
+ )
+ infra.add_argument(
+ "--api-secret",
+ default=default_api_secret,
+ help=(
+ "Modal secret with the LLM provider keys mounted into trial sandboxes "
+ f"(default: {config.DEFAULT_LLM_PROVIDERS_SECRET_NAME})"
+ ),
+ )
+ infra.add_argument(
+ "--modal-token-secret",
+ default=default_modal_token_secret,
+ help=(
+ "Modal secret mounted into the controller with MODAL_TOKEN_ID and MODAL_TOKEN_SECRET; "
+ f"use a dedicated service-user token for production (default: {config.DEFAULT_MODAL_TOKEN_SECRET_NAME})"
+ ),
+ )
+
+
+def add_build_source_options(parser: argparse.ArgumentParser) -> None:
+ parser.add_argument(
+ "--from",
+ dest="from_source",
+ default=os.environ.get("AGENT_EVALS_FROM"),
+ help=(
+ "Unified build source: 'local' for current HEAD + tracked changes, "
+ "'main' for origin/main's tip, or a git ref/tag/SHA (e.g. v0.210.0) "
+ "for a clean build resolved canonically against the remote so "
+ "teammates share one build"
+ ),
+ )
+ parser.add_argument("--base-sha")
+ parser.add_argument("--patch-path")
+ parser.add_argument(
+ "--repo-url",
+ default=os.environ.get("AGENT_EVALS_REPO_URL", source.DEFAULT_REPO_URL),
+ help="Git remote the Modal builder fetches the base SHA from",
+ )
+ parser.add_argument(
+ "--allow-untracked",
+ action="store_true",
+ help="Proceed even when untracked files exist; they are not included in the build patch",
+ )
+ parser.add_argument(
+ "--require-clean",
+ action="store_true",
+ help="Fail if tracked changes are present",
+ )
+ parser.add_argument(
+ "--clean-source",
+ action="store_true",
+ help="Build exactly --base-sha/--zed-version with no local patch",
+ )
+ parser.add_argument(
+ "--zed-version",
+ help="Git ref/tag/SHA of Zed to build as a clean source snapshot",
+ )
+
+
+def add_model_options(parser: argparse.ArgumentParser) -> None:
+ parser.add_argument(
+ "-m",
+ "--model",
+ default=os.environ.get("AGENT_EVALS_MODEL", config.DEFAULT_MODEL),
+ help="Base model as provider/model, sonnet-4.6, baseten:kimi-k2.7-code, or baseten:<model-id>",
+ )
+ parser.add_argument(
+ "--extra-api-secret",
+ action="append",
+ help="Additional Modal secret name to mount into the trial sandbox",
+ )
+ advanced = parser.add_argument_group(
+ "advanced model options",
+ "Lower-level model routing. For Baseten, prefer --model baseten:<model-id>.",
+ )
+ advanced.add_argument(
+ "--model-provider",
+ choices=("zed", "baseten"),
+ default="zed",
+ help="Use 'zed' for built-in provider/model ids, or 'baseten' for Baseten Model APIs",
+ )
+ advanced.add_argument(
+ "--baseten-model",
+ help="Baseten model id when --model-provider baseten, e.g. moonshotai/Kimi-K2.7-Code",
+ )
+ advanced.add_argument("--baseten-model-display-name")
+ advanced.add_argument(
+ "--baseten-api-url",
+ default=config.BASETEN_API_URL,
+ help=f"Baseten OpenAI-compatible API URL (default: {config.BASETEN_API_URL})",
+ )
+ advanced.add_argument(
+ "--baseten-model-max-tokens",
+ type=int,
+ default=config.BASETEN_DEFAULT_MAX_TOKENS,
+ )
+ advanced.add_argument(
+ "--baseten-model-max-output-tokens",
+ type=int,
+ default=config.BASETEN_DEFAULT_MAX_OUTPUT_TOKENS,
+ )
+ advanced.add_argument(
+ "--openai-compatible-provider-json",
+ help="JSON object merged into language_models.openai_compatible before eval-cli resolves --model",
+ )
+ advanced.add_argument(
+ "--anthropic-available-models-json",
+ help="JSON array merged into language_models.anthropic.available_models before eval-cli resolves --model",
+ )
+
+
+def add_launch_options(parser: argparse.ArgumentParser) -> None:
+ add_model_options(parser)
+ parser.add_argument(
+ "-j",
+ "--judge",
+ default=os.environ.get("AGENT_EVALS_JUDGE", config.DEFAULT_JUDGE_PRESET),
+ choices=["auto", *sorted(config.JUDGES)],
+ help="Judge preset; auto uses qna=deepseek-v4-pro and rf/tw=kimi-k2.7-code",
+ )
+ parser.add_argument("--judge-model")
+ parser.add_argument(
+ "--build",
+ metavar="ID",
+ help="Reuse this build if it exists, otherwise create it with this id",
+ )
+ add_build_source_options(parser)
+ parser.add_argument("--build-wait-timeout-secs", type=int, default=7200)
+ parser.add_argument("--tasks", help="File containing one full task name per line")
+ parser.add_argument("--include-task-name", action="append")
+ parser.add_argument("-n", "--n-tasks", type=int, help="Forward harness --n-tasks")
+ parser.add_argument("--n-concurrent", type=int, default=config.DEFAULT_N_CONCURRENT)
+ parser.add_argument(
+ "--override-cpus", type=int, default=config.DEFAULT_OVERRIDE_CPUS
+ )
+ parser.add_argument(
+ "--override-memory-mb", type=int, default=config.DEFAULT_OVERRIDE_MEMORY_MB
+ )
+ parser.add_argument(
+ "--sandbox-timeout-secs", type=int, default=config.DEFAULT_SANDBOX_TIMEOUT_SECS
+ )
+ parser.add_argument(
+ "--sandbox-idle-timeout-secs",
+ type=int,
+ default=config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS,
+ )
+ parser.add_argument(
+ "--eval-cli-timeout",
+ type=int,
+ help="Override the per-task agent timeout (defaults to the benchmark's)",
+ )
+ parser.add_argument("--run-id")
+ parser.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Print full manifests and harness commands without launching",
+ )
+ parser.add_argument(
+ "--plan",
+ action="store_true",
+ help="Print a concise launch plan without launching",
+ )
+ parser.add_argument(
+ "--verbose",
+ action="store_true",
+ help="With --plan, include full manifests and harness commands",
+ )
+ parser.add_argument(
+ "--extra-harbor-arg",
+ action="append",
+ help="Append one raw argument to the harness run command",
+ )
+ parser.add_argument(
+ "--swe-atlas-repo-url",
+ default=os.environ.get(
+ "AGENT_EVALS_SWE_ATLAS_REPO_URL", benchmarks.SWE_ATLAS_REPO_URL
+ ),
+ help="SWE-Atlas repo URL used for path-backed datasets",
+ )
+ parser.add_argument(
+ "--swe-atlas-repo-ref",
+ default=os.environ.get(
+ "AGENT_EVALS_SWE_ATLAS_REPO_REF", benchmarks.SWE_ATLAS_REPO_REF
+ ),
+ help="SWE-Atlas repo ref used for path-backed datasets",
+ )
+
+
+def add_suite_options(parser: argparse.ArgumentParser) -> None:
+ add_launch_options(parser)
+ parser.add_argument(
+ "--parts",
+ help="Comma-separated SWE-Atlas parts to run: qna,rf,tw (or 'all')",
+ )
+ parser.add_argument("--experiment-prefix", default="swe-atlas")
+ parser.add_argument("--run-id-prefix")
+ parser.add_argument(
+ "--suite-id", help="Explicit suite id for grouping multi-part runs"
+ )
+ parser.add_argument("--interactive", action="store_true")
+ parser.add_argument(
+ "-y",
+ "--yes",
+ action="store_true",
+ help="Never prompt; require flags/defaults",
+ )
+
+
+def add_run_lookup_options(
+ parser: argparse.ArgumentParser,
+ *,
+ require_experiment: bool = False,
+ run_id_optional: bool = False,
+) -> None:
+ if run_id_optional:
+ parser.add_argument(
+ "run_id",
+ nargs="?",
+ help="Run id (defaults to the most recent run launched from this machine)",
+ )
+ else:
+ parser.add_argument("run_id")
+ parser.add_argument(
+ "-e",
+ "--experiment-name",
+ required=require_experiment,
+ help="Benchmark storage name (auto-resolved from the local run index when omitted)",
+ )
+
+
+def add_command_parser(
+ subparsers,
+ name: str,
+ *,
+ func: object | None = None,
+ **kwargs,
+) -> argparse.ArgumentParser:
+ parser = subparsers.add_parser(name, **kwargs)
+ add_common_options(parser, include_defaults=False)
+ if func is not None:
+ parser.set_defaults(func=func)
+ return parser
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ prog="zed-eval",
+ description="Launch, monitor, and fetch remote SWE-Atlas agent evals on Modal.",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=(
+ "Common workflows:\n"
+ " zed-eval run swe-atlas -m sonnet-4.6 # launch the SWE-Atlas suite\n"
+ " zed-eval run rf -m sonnet-4.6 # launch one benchmark\n"
+ " zed-eval runs # what did I launch recently?\n"
+ " zed-eval status # check my most recent run\n"
+ " zed-eval status <run-id> # check a specific run\n"
+ " zed-eval logs <run-id> # print controller logs\n"
+ " zed-eval report <run-id> --fetch # fetch + score a run\n"
+ "\n"
+ "After launching, a run id alone locates the run — namespace and\n"
+ "benchmark storage name are resolved from this machine's local run index."
+ ),
+ )
+ add_common_options(parser)
+ subparsers = parser.add_subparsers(dest="command", required=True)
+
+ doctor = add_command_parser(
+ subparsers,
+ "doctor",
+ func=command_doctor,
+ help="Check local prerequisites and defaults",
+ )
+ doctor.add_argument("--create-volume", action="store_true")
+
+ add_command_parser(
+ subparsers,
+ "deploy",
+ func=command_deploy,
+ help=(
+ "Publish harness code to Modal. This is the ONLY command that "
+ "deploys; deploying cancels in-flight runs. Run it once after "
+ "changing harness code — all other commands just invoke the "
+ "already-deployed functions and never deploy."
+ ),
+ )
+
+ build = add_command_parser(
+ subparsers,
+ "build",
+ func=command_build,
+ help="Build eval-cli on Modal into builds/<build-id>",
+ )
+ add_build_source_options(build)
+ build.add_argument(
+ "--build",
+ metavar="ID",
+ help="Reuse this build if it exists, otherwise create it with this id",
+ )
+ build.add_argument(
+ "--detach", action="store_true", help="Spawn the build and return immediately"
+ )
+
+ builds = add_command_parser(
+ subparsers,
+ "builds",
+ func=command_builds,
+ help="List content-addressed builds on the volume",
+ )
+ builds.add_argument("--details", action="store_true")
+ builds.add_argument("--json", action="store_true")
+ builds.add_argument("--limit", type=int, default=50)
+
+ run_cmd = add_command_parser(
+ subparsers,
+ "run",
+ func=command_run,
+ help="Run benchmarks — the everyday entry point",
+ description=(
+ "Launch runs by benchmark id, alias, or group. It picks the right "
+ "default judge for each benchmark and never prompts. The sibling "
+ "`swe-atlas` command launches SWE-Atlas parts (qna/rf/tw) with "
+ "interactive prompts."
+ ),
+ )
+ run_cmd.add_argument(
+ "benchmark",
+ nargs="+",
+ metavar="target",
+ help=(
+ "Benchmark id/alias/group (swe-atlas (= qna,rf,tw), swe-atlas-rf, "
+ "qna, rf, tw, terminal-bench-2.1 (tb21), deepswe). Comma-separated "
+ "and repeated values are combined."
+ ),
+ )
+ add_launch_options(run_cmd)
+
+ run_cmd.add_argument(
+ "--suite-id", help="Explicit suite id for multi-benchmark runs"
+ )
+ run_cmd.add_argument(
+ "--staff",
+ action="store_true",
+ help=(
+ "Run the agent with staff mode ON (default OFF). Staff mode enables "
+ "the sandboxed terminal, which hangs inside Modal sandboxes, so keep "
+ "it off for remote runs."
+ ),
+ )
+ run_cmd.add_argument(
+ "-y",
+ "--yes",
+ action="store_true",
+ help="Accepted for parity with swe-atlas; run never prompts",
+ )
+
+ report = add_command_parser(
+ subparsers,
+ "report",
+ func=command_report,
+ help="Success-conditioned metrics (rate, tokens, tool calls, steps)",
+ )
+ report.add_argument("run_id", nargs="?")
+ report.add_argument(
+ "-e",
+ "--experiment-name",
+ help="Benchmark storage name (auto-resolved from the local run index when omitted)",
+ )
+ report.add_argument("--job-dir", help="Analyze a local job directory directly")
+ report.add_argument("--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs"))
+ report.add_argument(
+ "--fetch", action="store_true", help="Fetch the run archive before reporting"
+ )
+ report.add_argument(
+ "--json",
+ dest="as_json",
+ action="store_true",
+ help="Emit JSON instead of a table",
+ )
+ report.add_argument(
+ "--timeouts-as-failures",
+ dest="timeouts_as_failures",
+ action=argparse.BooleanOptionalAction,
+ default=None,
+ help=(
+ "Count agent timeouts as failures rather than excluded errors "
+ "(default: auto — on for test-scored benchmarks like terminal-bench "
+ "and deepswe)"
+ ),
+ )
+
+ cleanup = add_command_parser(
+ subparsers,
+ "cleanup",
+ func=command_cleanup,
+ help="Prune stale builds and cold build cache (never eval results)",
+ )
+ cleanup.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Report what would be removed without deleting anything",
+ )
+ cleanup.add_argument(
+ "--build-retention-days",
+ type=float,
+ help="Remove builds older than this many days (default 14)",
+ )
+
+ suite = add_command_parser(
+ subparsers,
+ "swe-atlas",
+ func=command_swe_atlas,
+ help="Launch an interactive benchmark suite (SWE-Atlas, Terminal-Bench, DeepSWE)",
+ )
+ add_suite_options(suite)
+
+ list_runs = add_command_parser(
+ subparsers,
+ "list",
+ func=command_list,
+ help="List benchmark runs on the volume, optionally with metadata",
+ )
+ list_runs.add_argument("-e", "--experiment-name")
+ list_runs.add_argument("--details", action="store_true")
+ list_runs.add_argument("--json", action="store_true")
+ list_runs.add_argument("--limit", type=int, default=50)
+ list_runs.add_argument("--all-namespaces", action="store_true")
+
+ runs = add_command_parser(
+ subparsers,
+ "runs",
+ func=command_runs,
+ help="List recent runs launched from this machine (local, fast)",
+ description=(
+ "Show runs recorded in the local run index when you launched them "
+ "from this machine — newest first, no network call. Use this for a "
+ "quick 'what did I launch lately?'; use `list --details` to query "
+ "run state on the volume, or `status <run-id>` for one run's state."
+ ),
+ )
+ runs.add_argument("--json", action="store_true")
+ runs.add_argument("--limit", type=int, default=20)
+
+ status = add_command_parser(
+ subparsers,
+ "status",
+ func=command_status,
+ help="Print a run's state.json once (no run id = most recent run)",
+ )
+ add_run_lookup_options(status, run_id_optional=True)
+
+ logs = add_command_parser(
+ subparsers,
+ "logs",
+ func=command_logs,
+ help="Print a run's controller.log once (no run id = most recent run)",
+ )
+ add_run_lookup_options(logs, run_id_optional=True)
+
+ fetch = add_command_parser(
+ subparsers,
+ "fetch",
+ func=command_fetch,
+ help="Fetch and extract a run's Harbor job archive",
+ )
+ add_run_lookup_options(fetch)
+ fetch.add_argument("--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs"))
+
+ rejudge = add_command_parser(
+ subparsers,
+ "rejudge",
+ func=command_rejudge,
+ help="Re-grade a finished run with a different judge (new derived run, "
+ "reuses the agent work)",
+ )
+ add_run_lookup_options(rejudge)
+ rejudge.add_argument(
+ "-j",
+ "--judge",
+ required=True,
+ help="Judge preset to re-grade with (see config.JUDGES, e.g. "
+ "deepseek-v4-pro, kimi-k2.7-code, leaderboard)",
+ )
+ rejudge.add_argument(
+ "--judge-model",
+ dest="judge_model",
+ help="Override the judge preset's model id",
+ )
+ rejudge.add_argument(
+ "--parent-namespace",
+ dest="parent_namespace",
+ help="Namespace of the source run, if different from --namespace",
+ )
+ rejudge.add_argument(
+ "--new-run-id",
+ dest="new_run_id",
+ help="Explicit id for the derived run (default: <parent>-rejudge-<judge>-<rand>)",
+ )
+ rejudge.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Print the rejudge request without spawning the controller",
+ )
+
+ suite_group = add_command_parser(
+ subparsers,
+ "suite",
+ help="Inspect or fetch grouped benchmark suite runs",
+ )
+ suite_subparsers = suite_group.add_subparsers(dest="suite_command", required=True)
+ suite_status = add_command_parser(
+ suite_subparsers,
+ "status",
+ func=command_suite_status,
+ help="Show status for each run in a suite",
+ )
+ suite_status.add_argument("suite_id")
+ suite_status.add_argument("--json", action="store_true")
+ suite_logs = add_command_parser(
+ suite_subparsers,
+ "logs",
+ func=command_suite_logs,
+ help="Print logs for each run in a suite",
+ )
+ suite_logs.add_argument("suite_id")
+ suite_logs.add_argument("--follow", action="store_true")
+ suite_logs.add_argument("--interval", type=float, default=30.0)
+ suite_fetch = add_command_parser(
+ suite_subparsers,
+ "fetch",
+ func=command_suite_fetch,
+ help="Fetch all run archives in a suite",
+ )
+ suite_fetch.add_argument("suite_id")
+ suite_fetch.add_argument(
+ "--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs")
+ )
+
+ baseline_group = add_command_parser(
+ subparsers,
+ "baseline",
+ help="Record and inspect baseline-of-record results (clean commits on main)",
+ )
+ baseline_subparsers = baseline_group.add_subparsers(
+ dest="baseline_command", required=True
+ )
+ baseline_record = add_command_parser(
+ baseline_subparsers,
+ "record",
+ func=command_baseline_record,
+ help="Promote completed run(s) to the baseline of record for their (benchmark, model)",
+ )
+ baseline_record.add_argument("run_id", nargs="+")
+ baseline_record.add_argument("--experiment-name", required=True)
+ baseline_record.add_argument(
+ "--allow-dirty",
+ action="store_true",
+ help="Record even though the build carries a local patch (not a clean commit)",
+ )
+ baseline_record.add_argument(
+ "--allow-off-main",
+ action="store_true",
+ help="Record even though base_sha can't be verified as reachable from origin/main",
+ )
+ baseline_record.add_argument(
+ "--repo-url",
+ default=os.environ.get("AGENT_EVALS_REPO_URL"),
+ help="Git remote to resolve origin/main against (default: AGENT_EVALS_REPO_URL or the canonical repo)",
+ )
+
+ baseline_list = add_command_parser(
+ baseline_subparsers,
+ "list",
+ func=command_baseline_list,
+ help="List the current baseline of record for every (benchmark, model)",
+ )
+ baseline_list.add_argument("--json", action="store_true")
+
+ baseline_show = add_command_parser(
+ baseline_subparsers,
+ "show",
+ func=command_baseline_show,
+ help="Show the baseline record for one (benchmark, model)",
+ )
+ baseline_show.add_argument("experiment_name")
+ baseline_show.add_argument("--model", required=True)
+ baseline_show.add_argument(
+ "--history", action="store_true", help="Include superseded baselines"
+ )
+
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = build_parser()
+ args = parser.parse_args(argv)
+ try:
+ return args.func(args)
+ except (ValueError, AppNotDeployedError) as error:
+ print(f"error: {error}", file=sys.stderr)
+ return 1
+ except subprocess.CalledProcessError as error:
+ if error.stdout:
+ print(error.stdout, end="")
+ if error.stderr:
+ print(error.stderr, end="", file=sys.stderr)
+ return error.returncode
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/crates/eval_cli/zed_eval/common.py b/crates/eval_cli/zed_eval/common.py
new file mode 100644
index 00000000000..7ae6083231d
--- /dev/null
+++ b/crates/eval_cli/zed_eval/common.py
@@ -0,0 +1,220 @@
+from __future__ import annotations
+
+import argparse
+import datetime
+import json
+import os
+import pathlib
+import shutil
+import subprocess
+import tarfile
+from typing import Any
+
+from . import config, source
+
+ALL_SWE_ATLAS_PARTS = ["qna", "rf", "tw"]
+
+
+def utc_timestamp() -> str:
+ return datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+
+
+def utc_now() -> str:
+ return datetime.datetime.now(datetime.timezone.utc).isoformat()
+
+
+def print_json(data: Any) -> None:
+ print(json.dumps(data, indent=2, sort_keys=True))
+
+
+def write_json(path: pathlib.Path, data: dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n")
+
+
+def load_json(path: pathlib.Path) -> dict[str, Any] | None:
+ try:
+ data = json.loads(path.read_text())
+ except (OSError, json.JSONDecodeError):
+ return None
+ return data if isinstance(data, dict) else None
+
+
+def safe_extract_archive(archive: tarfile.TarFile, destination: pathlib.Path) -> None:
+ destination = destination.resolve()
+ members = archive.getmembers()
+ for member in members:
+ if member.issym() or member.islnk():
+ raise ValueError(f"archive links are not supported: {member.name}")
+ target = (destination / member.name).resolve()
+ if destination != target and destination not in target.parents:
+ raise ValueError(f"archive member escapes destination: {member.name}")
+ archive.extractall(destination, members=members)
+
+
+def command_exists(name: str) -> bool:
+ return shutil.which(name) is not None
+
+
+def run_command(
+ command: list[str], *, capture: bool = False
+) -> subprocess.CompletedProcess[str]:
+ return subprocess.run(command, check=True, text=True, capture_output=capture)
+
+
+def dedupe_preserving_order(values: list[str]) -> list[str]:
+ seen = set()
+ result = []
+ for value in values:
+ if value in seen:
+ continue
+ seen.add(value)
+ result.append(value)
+ return result
+
+
+def parse_comma_values(values: list[str] | None) -> list[str]:
+ result = []
+ for value in values or []:
+ result.extend(part.strip() for part in value.split(",") if part.strip())
+ return result
+
+
+def parse_parts(values: list[str] | None) -> list[str]:
+ requested = parse_comma_values(values)
+ if any(value.lower() == "all" for value in requested):
+ return list(ALL_SWE_ATLAS_PARTS)
+ return dedupe_preserving_order(
+ [config.canonical_part(value) for value in requested]
+ )
+
+
+def configure_modal_environment(args: argparse.Namespace) -> None:
+ os.environ["AGENT_EVALS_APP_NAME"] = args.app_name
+ os.environ["AGENT_EVALS_VOLUME"] = args.volume
+ os.environ["AGENT_EVALS_MODAL_TOKEN_SECRET"] = args.modal_token_secret
+ os.environ["AGENT_EVALS_LLM_PROVIDERS_SECRET"] = args.api_secret
+
+
+def import_modal_app(args: argparse.Namespace):
+ configure_modal_environment(args)
+ from . import modal_app
+
+ return modal_app
+
+
+class AppNotDeployedError(RuntimeError):
+ """Raised when a Modal function lookup fails because the app (or that
+ specific function) has not been deployed. Surfaced as a clean CLI error
+ pointing the user at `zed-eval deploy`.
+ """
+
+
+def deploy_app(args: argparse.Namespace) -> None:
+ """Publish the Modal app. This is the ONLY code path that deploys.
+
+ Deploying replaces the live app and cancels any in-flight eval runs, so it
+ must happen exclusively via the `deploy` subcommand. Every other command
+ just looks up already-deployed functions with `deployed_function`. After
+ changing harness code, run `zed-eval deploy` once.
+ """
+ modal_app = import_modal_app(args)
+ print(f"Deploying Modal app '{args.app_name}'...")
+ modal_app.app.deploy(name=args.app_name)
+
+
+def deployed_function(args: argparse.Namespace, function_name: str):
+ """Look up an already-deployed Modal function. Never deploys.
+
+ `modal.Function.from_name` resolves a published function without deploying,
+ which is exactly what we want: deploying would cancel in-flight runs. We
+ hydrate eagerly so a missing app/function fails here with an actionable
+ message instead of deep inside a later `.spawn()`/`.remote()` call.
+ """
+ import modal
+ from modal.exception import NotFoundError
+
+ configure_modal_environment(args)
+ function = modal.Function.from_name(args.app_name, function_name)
+ try:
+ function.hydrate()
+ except NotFoundError as error:
+ raise AppNotDeployedError(
+ f"app '{args.app_name}' not deployed (or function "
+ f"'{function_name}' missing) — run 'zed-eval deploy' first"
+ ) from error
+ return function
+
+
+def modal_call_id(call: Any) -> str:
+ for attribute in ("object_id", "id", "function_call_id"):
+ value = getattr(call, attribute, None)
+ if value:
+ return str(value)
+ return str(call)
+
+
+def print_table(rows: list[dict[str, Any]], columns: list[tuple[str, str]]) -> None:
+ if not rows:
+ print("No rows")
+ return
+ widths = []
+ for key, title in columns:
+ width = len(title)
+ for row in rows:
+ width = max(width, len(str(row.get(key) or "")))
+ widths.append(min(width, 80))
+ print(
+ " ".join(
+ title.ljust(widths[index]) for index, (_key, title) in enumerate(columns)
+ )
+ )
+ print(" ".join("-" * width for width in widths))
+ for row in rows:
+ cells = []
+ for index, (key, _title) in enumerate(columns):
+ text = str(row.get(key) or "")
+ if len(text) > widths[index]:
+ text = text[: widths[index] - 1] + "…"
+ cells.append(text.ljust(widths[index]))
+ print(" ".join(cells))
+
+
+def default_namespace(args: argparse.Namespace) -> str:
+ return source.sanitize_namespace(args.namespace or source.default_namespace())
+
+
+def resolve_run_location(args: argparse.Namespace, run_id: str) -> tuple[str, str]:
+ """Resolve (namespace, experiment_name) for a run-lookup command.
+
+ When `--experiment-name` is given, behaves exactly as before: namespace comes
+ from `--namespace` or the local git default. When it is omitted, the run is
+ located via the launch-time local run index (`run_index`) so a bare run id is
+ enough. Raises a friendly error pointing at the explicit flags when the run
+ can't be found.
+ """
+ from . import run_index
+
+ explicit_experiment = getattr(args, "experiment_name", None)
+ explicit_namespace = getattr(args, "namespace", None)
+ if explicit_experiment:
+ namespace = explicit_namespace or source.default_namespace()
+ return (
+ source.sanitize_namespace(namespace),
+ source.sanitize_namespace(explicit_experiment),
+ )
+ entry = run_index.lookup(run_id)
+ if not entry:
+ raise ValueError(
+ f"could not locate run '{run_id}' in the local run index "
+ f"({run_index.index_path()}). Pass --experiment-name (and "
+ "--namespace if it isn't yours). The index is written automatically "
+ "when you launch a run from this machine."
+ )
+ namespace = (
+ explicit_namespace or entry.get("namespace") or source.default_namespace()
+ )
+ return (
+ source.sanitize_namespace(namespace),
+ source.sanitize_namespace(entry["experiment_name"]),
+ )
diff --git a/crates/eval_cli/zed_eval/config.py b/crates/eval_cli/zed_eval/config.py
new file mode 100644
index 00000000000..94112db6b47
--- /dev/null
+++ b/crates/eval_cli/zed_eval/config.py
@@ -0,0 +1,164 @@
+from __future__ import annotations
+
+import shlex
+from dataclasses import dataclass
+
+DEFAULT_APP_NAME = "agent-evals"
+DEFAULT_VOLUME_NAME = "agent-evals"
+# Modal secret holding the LLM provider keys (ANTHROPIC_API_KEY, BASETEN_API_KEY,
+# ...) mounted into trial sandboxes.
+DEFAULT_LLM_PROVIDERS_SECRET_NAME = "agent-evals-llm-providers"
+# Modal secret holding the controller's Modal token (MODAL_TOKEN_ID/SECRET).
+DEFAULT_MODAL_TOKEN_SECRET_NAME = "agent-evals-modal-token"
+HARBOR_VERSION = "0.15.0"
+MODAL_VERSION = "1.5.0"
+CONTROLLER_IMAGE_RECIPE_VERSION = "agent-evals-controller-image-v1"
+DEFAULT_MODEL = "anthropic/claude-sonnet-4-6"
+DEFAULT_JUDGE_PRESET = "auto"
+BASETEN_API_URL = "https://inference.baseten.co/v1"
+BASETEN_PROVIDER_ID = "baseten"
+BASETEN_DEFAULT_MAX_TOKENS = 262_144
+BASETEN_DEFAULT_MAX_OUTPUT_TOKENS = 65_536
+
+# Kept modest on purpose: the agent model's API key is the real bottleneck,
+# not Modal capacity. Snickerdoodle-EAP is capped at 2M ITPM / 400K OTPM, and
+# ~150 concurrent agent loops (3 jobs x 50) blew past that. One job at this
+# concurrency stays well under the ceiling; raise only if you've confirmed the
+# model's rate limits have headroom.
+DEFAULT_N_CONCURRENT = 25
+DEFAULT_SANDBOX_TIMEOUT_SECS = 14_400
+DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS = 300
+# No per-task resource override by default. When these are None the
+# --override-cpus/--override-memory-mb flags are omitted entirely, so Harbor
+# applies each task's declared cpus/memory (SWE-Atlas tasks declare 16 CPU /
+# 16 GB). Overriding *below* the declared values OOM-kills memory-heavy tasks
+# (SIGKILL / exit 137) and, per the resource policy, changes the experimental
+# conditions. Pass --override-cpus/--override-memory-mb to opt into smaller
+# sandboxes deliberately.
+DEFAULT_OVERRIDE_CPUS = None
+DEFAULT_OVERRIDE_MEMORY_MB = None
+EVAL_BASE_URL_IN_SANDBOX = "http://127.0.0.1:8089/v1"
+JUDGE_PROXY_VERIFIER_IMPORT_PATH = "zed_eval.verifier:ZedJudgeProxyVerifier"
+
+
+@dataclass(frozen=True)
+class JudgeConfig:
+ model: str
+ upstream: str
+ auth_env: str
+ # Floor for the judge's max_completion_tokens, applied by the in-sandbox
+ # judge shim (ZED_JUDGE_MAX_TOKENS). Set for Baseten judges so the runtime
+ # judge matches the offline calibration (8192); left None for the
+ # opus/leaderboard path so it stays leaderboard-faithful at the verifier's
+ # hardcoded 2048.
+ max_tokens: int | None = None
+
+
+def judge_verifier_args(judge: JudgeConfig, judge_model: str) -> list[str]:
+ args = [
+ "--verifier-import-path",
+ JUDGE_PROXY_VERIFIER_IMPORT_PATH,
+ "--ve",
+ f"EVAL_MODEL={judge_model}",
+ "--ve",
+ f"EVAL_BASE_URL={EVAL_BASE_URL_IN_SANDBOX}",
+ "--ve",
+ "EVAL_API_KEY=unused-by-agent-evals-proxy",
+ "--ve",
+ f"ZED_JUDGE_UPSTREAM={judge.upstream}",
+ "--ve",
+ f"ZED_JUDGE_AUTH_ENV={judge.auth_env}",
+ ]
+ if judge.max_tokens is not None:
+ args.extend(["--ve", f"ZED_JUDGE_MAX_TOKENS={judge.max_tokens}"])
+ return args
+
+
+def redacted_command(command: list[str]) -> str:
+ return shlex.join(
+ "EVAL_API_KEY=<redacted>" if argument.startswith("EVAL_API_KEY=") else argument
+ for argument in command
+ )
+
+
+JUDGES: dict[str, JudgeConfig] = {
+ "leaderboard": JudgeConfig(
+ model="claude-opus-4-5-20251101",
+ upstream="https://api.anthropic.com/v1",
+ auth_env="ANTHROPIC_API_KEY",
+ ),
+ "opus-leaderboard": JudgeConfig(
+ model="claude-opus-4-5-20251101",
+ upstream="https://api.anthropic.com/v1",
+ auth_env="ANTHROPIC_API_KEY",
+ ),
+ "deepseek-v4-pro": JudgeConfig(
+ model="deepseek-ai/DeepSeek-V4-Pro",
+ upstream=BASETEN_API_URL,
+ auth_env="BASETEN_API_KEY",
+ max_tokens=8192,
+ ),
+ "deepseek": JudgeConfig(
+ model="deepseek-ai/DeepSeek-V4-Pro",
+ upstream=BASETEN_API_URL,
+ auth_env="BASETEN_API_KEY",
+ max_tokens=8192,
+ ),
+ "kimi-k2.7-code": JudgeConfig(
+ model="moonshotai/Kimi-K2.7-Code",
+ upstream=BASETEN_API_URL,
+ auth_env="BASETEN_API_KEY",
+ max_tokens=8192,
+ ),
+ "kimi": JudgeConfig(
+ model="moonshotai/Kimi-K2.7-Code",
+ upstream=BASETEN_API_URL,
+ auth_env="BASETEN_API_KEY",
+ max_tokens=8192,
+ ),
+ "gpt55": JudgeConfig(
+ model="gpt-5.5-2026-04-23",
+ upstream="https://api.openai.com/v1",
+ auth_env="OPENAI_API_KEY",
+ ),
+}
+
+
+MODEL_PRESETS: dict[str, str] = {
+ "sonnet-4.6": DEFAULT_MODEL,
+ "claude-sonnet-4.6": DEFAULT_MODEL,
+ "sonnet-4.6-latest": "anthropic/claude-sonnet-4-6-latest",
+ "opus-4.5": "anthropic/claude-opus-4-5",
+ "baseten:kimi-k2.7-code": "baseten/moonshotai/Kimi-K2.7-Code",
+ "baseten:kimi": "baseten/moonshotai/Kimi-K2.7-Code",
+ "baseten:deepseek-v4-pro": "baseten/deepseek-ai/DeepSeek-V4-Pro",
+ "baseten:deepseek": "baseten/deepseek-ai/DeepSeek-V4-Pro",
+}
+
+
+def orchestration_info() -> dict[str, str]:
+ return {
+ "controller_image_recipe_version": CONTROLLER_IMAGE_RECIPE_VERSION,
+ "harbor_version": HARBOR_VERSION,
+ "modal_client_version": MODAL_VERSION,
+ }
+
+
+def get_judge(name: str) -> JudgeConfig:
+ try:
+ return JUDGES[name]
+ except KeyError as error:
+ valid = ", ".join(sorted(JUDGES))
+ raise ValueError(f"unknown judge preset '{name}' (valid: {valid})") from error
+
+
+def canonical_part(part: str) -> str:
+ normalized = part.strip().lower()
+ if normalized in ("qna", "rf", "tw"):
+ return normalized
+ valid = ", ".join(("qna", "rf", "tw"))
+ raise ValueError(f"unknown SWE-Atlas part '{part}' (valid: {valid})")
+
+
+def resolve_model_preset(model: str) -> str:
+ return MODEL_PRESETS.get(model, model)
diff --git a/crates/eval_cli/zed_eval/harness_command.py b/crates/eval_cli/zed_eval/harness_command.py
new file mode 100644
index 00000000000..f6024d534f9
--- /dev/null
+++ b/crates/eval_cli/zed_eval/harness_command.py
@@ -0,0 +1,220 @@
+"""Build the harness command (Harbor or Pier) for a benchmark run request.
+
+The command is driven by the self-describing `benchmark` block embedded in a run
+request (see `benchmarks.benchmark_metadata`), so it works for any registered
+benchmark without a separate experiment registry lookup.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from . import benchmarks, config
+
+
+def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:
+ block = run_request.get("benchmark")
+ if not isinstance(block, dict):
+ raise ValueError("run request is missing a 'benchmark' block")
+ return block
+
+
+def dataset_args(benchmark: dict[str, Any]) -> list[str]:
+ dataset = benchmark.get("dataset") or {}
+ kind = dataset.get("kind")
+ if kind == benchmarks.DATASET_REGISTRY:
+ name = dataset.get("name")
+ if not name:
+ raise ValueError("registry dataset requires a name")
+ return ["-d", name]
+ if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):
+ data_dir = dataset.get("data_dir")
+ if not data_dir:
+ raise ValueError("path dataset requires data_dir")
+ return ["-p", dataset_path(benchmark)]
+ raise ValueError(f"unsupported dataset kind: {kind}")
+
+
+def dataset_clone_dir(benchmark: dict[str, Any]) -> str:
+ """Where the controller clones the dataset repo for path datasets."""
+ return f"/tmp/datasets/{benchmark['id']}"
+
+
+def dataset_path(benchmark: dict[str, Any]) -> str:
+ """Repo-relative task directory inside the cloned dataset repo."""
+ dataset = benchmark.get("dataset") or {}
+ return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"
+
+
+def harness_binary(benchmark: dict[str, Any]) -> str:
+ harness = benchmark.get("harness")
+ if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):
+ raise ValueError(f"unsupported harness: {harness}")
+ return harness
+
+
+def eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:
+ return int(
+ run_request.get("eval_cli_timeout")
+ or benchmark.get("default_timeout_secs")
+ or config.DEFAULT_SANDBOX_TIMEOUT_SECS
+ )
+
+
+def build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:
+ benchmark = _benchmark_block(run_request)
+ build_id = run_request.get("build_id")
+ if not build_id:
+ raise ValueError("zed benchmarks require build_id")
+ volume_name = run_request["volume_name"]
+ api_secret_name = run_request["api_secret_name"]
+ run_id = run_request["run_id"]
+ agent_model = run_request.get("agent_model") or config.DEFAULT_MODEL
+ n_concurrent = int(run_request.get("n_concurrent") or config.DEFAULT_N_CONCURRENT)
+ sandbox_timeout_secs = int(
+ run_request.get("sandbox_timeout_secs") or config.DEFAULT_SANDBOX_TIMEOUT_SECS
+ )
+ sandbox_idle_timeout_secs = int(
+ run_request.get("sandbox_idle_timeout_secs")
+ or config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS
+ )
+ override_cpus = run_request.get("override_cpus") or config.DEFAULT_OVERRIDE_CPUS
+ override_memory_mb = (
+ run_request.get("override_memory_mb") or config.DEFAULT_OVERRIDE_MEMORY_MB
+ )
+
+ secret_names = [api_secret_name]
+ for extra_secret_name in run_request.get("extra_api_secrets") or []:
+ if extra_secret_name not in secret_names:
+ secret_names.append(extra_secret_name)
+
+ command = [
+ harness_binary(benchmark),
+ "run",
+ *dataset_args(benchmark),
+ "-m",
+ agent_model,
+ "--env",
+ "modal",
+ "--n-concurrent",
+ str(n_concurrent),
+ "--job-name",
+ run_id,
+ "--jobs-dir",
+ jobs_dir,
+ "--ek",
+ f"volumes={json.dumps({'/data': volume_name})}",
+ "--ek",
+ f"secrets={json.dumps(secret_names)}",
+ "--ek",
+ f"sandbox_timeout_secs={sandbox_timeout_secs}",
+ "--ek",
+ f"sandbox_idle_timeout_secs={sandbox_idle_timeout_secs}",
+ # Harbor and Pier have different installed-agent interfaces, so each gets
+ # its own thin agent shell (the sanctioned cross-framework exception).
+ "--agent-import-path",
+ (
+ "zed_eval.pier_agent:ZedPierAgent"
+ if harness_binary(benchmark) == benchmarks.HARNESS_PIER
+ else "zed_eval.agent:ZedAgent"
+ ),
+ "--ae",
+ f"EVAL_CLI_CONTAINER_PATH=/data/builds/{build_id}/eval-cli",
+ "--ae",
+ f"EVAL_CLI_TIMEOUT={eval_cli_timeout(run_request, benchmark)}",
+ ]
+ # Omit --override-cpus/--override-memory-mb unless explicitly set, so Harbor
+ # applies each task's declared cpus/memory. Overriding below the declared
+ # values OOM-kills memory-heavy tasks (SIGKILL / exit 137).
+ if override_cpus is not None:
+ command += ["--override-cpus", str(int(override_cpus))]
+ if override_memory_mb is not None:
+ command += ["--override-memory-mb", str(int(override_memory_mb))]
+
+ openai_compatible_provider_json = run_request.get("openai_compatible_provider_json")
+ if openai_compatible_provider_json:
+ command.extend(
+ [
+ "--ae",
+ f"ZED_OPENAI_COMPATIBLE_PROVIDERS={openai_compatible_provider_json}",
+ ]
+ )
+ anthropic_available_models_json = run_request.get("anthropic_available_models_json")
+ if anthropic_available_models_json:
+ command.extend(
+ [
+ "--ae",
+ f"ZED_ANTHROPIC_AVAILABLE_MODELS={anthropic_available_models_json}",
+ ]
+ )
+
+ # NOTE: Pier's network allowlist for air-gapped DeepSWE tasks is declared by
+ # the agent (`network_allowlist()` on a Pier-native agent class), not via a
+ # CLI flag — see benchmarks.AGENT_API_HOSTS and the DeepSWE agent. There is
+ # no `--agent-allow-host` option on `pier run`.
+ if benchmark.get("needs_judge"):
+ judge = config.get_judge(run_request["judge_preset"])
+ judge_model = run_request.get("judge_model") or judge.model
+ command.extend(config.judge_verifier_args(judge, judge_model))
+
+ for key, value in (benchmark.get("env") or {}).items():
+ command.extend(["--ae", f"{key}={value}"])
+ for key, value in (run_request.get("extra_env") or {}).items():
+ command.extend(["--ae", f"{key}={value}"])
+
+ for task_name in run_request.get("task_names") or []:
+ command.extend(["--include-task-name", task_name])
+
+ n_tasks = run_request.get("n_tasks")
+ if n_tasks is not None:
+ command.extend(["--n-tasks", str(n_tasks)])
+
+ command.extend(run_request.get("extra_harbor_args") or [])
+ return command
+
+
+def run_metadata(run_request: dict[str, Any]) -> dict[str, Any]:
+ benchmark = _benchmark_block(run_request)
+ metadata: dict[str, Any] = {
+ "run_id": run_request["run_id"],
+ "namespace": run_request["namespace"],
+ "benchmark": benchmark,
+ "suite_id": run_request.get("suite_id"),
+ "harness": benchmark.get("harness"),
+ "scoring": benchmark.get("scoring"),
+ "build_id": run_request.get("build_id"),
+ "agent_model": run_request.get("agent_model") or config.DEFAULT_MODEL,
+ "orchestration": run_request.get("orchestration")
+ or config.orchestration_info(),
+ "build_toolchain": run_request.get("build_toolchain"),
+ "volume_name": run_request["volume_name"],
+ "api_secret_name": run_request["api_secret_name"],
+ "extra_api_secrets": run_request.get("extra_api_secrets") or [],
+ "openai_compatible_provider_json": run_request.get(
+ "openai_compatible_provider_json"
+ ),
+ "anthropic_available_models_json": run_request.get(
+ "anthropic_available_models_json"
+ ),
+ "n_concurrent": run_request.get("n_concurrent") or config.DEFAULT_N_CONCURRENT,
+ "override_cpus": run_request.get("override_cpus")
+ or config.DEFAULT_OVERRIDE_CPUS,
+ "override_memory_mb": run_request.get("override_memory_mb")
+ or config.DEFAULT_OVERRIDE_MEMORY_MB,
+ "eval_cli_timeout": eval_cli_timeout(run_request, benchmark),
+ "sandbox_timeout_secs": run_request.get("sandbox_timeout_secs")
+ or config.DEFAULT_SANDBOX_TIMEOUT_SECS,
+ "sandbox_idle_timeout_secs": run_request.get("sandbox_idle_timeout_secs")
+ or config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS,
+ "task_count": len(run_request.get("task_names") or []),
+ "n_tasks": run_request.get("n_tasks"),
+ }
+ if benchmark.get("needs_judge"):
+ judge = config.get_judge(run_request["judge_preset"])
+ metadata["judge_preset"] = run_request["judge_preset"]
+ metadata["judge_model"] = run_request.get("judge_model") or judge.model
+ metadata["judge_upstream"] = judge.upstream
+ metadata["judge_auth_env"] = judge.auth_env
+ metadata["judge_eval_base_url"] = config.EVAL_BASE_URL_IN_SANDBOX
+ return metadata
diff --git a/crates/eval_cli/zed_eval/interactive.py b/crates/eval_cli/zed_eval/interactive.py
new file mode 100644
index 00000000000..1829ab1e8b6
--- /dev/null
+++ b/crates/eval_cli/zed_eval/interactive.py
@@ -0,0 +1,163 @@
+from __future__ import annotations
+
+import argparse
+import sys
+
+from .common import dedupe_preserving_order, deployed_function
+
+INTERACTIVE_BENCHMARK_CHOICES = [
+ "qna",
+ "rf",
+ "tw",
+ "terminal-bench-2.1",
+ "deepswe",
+]
+INTERACTIVE_BENCHMARK_ALIASES = {
+ "terminal-bench": "terminal-bench-2.1",
+ "tb21": "terminal-bench-2.1",
+ "deep-swe": "deepswe",
+}
+INTERACTIVE_MODEL_CHOICES = [
+ "sonnet-4.6",
+ "opus-4.5",
+ "baseten:kimi-k2.7-code",
+ "baseten:deepseek-v4-pro",
+ "custom-zed-model",
+ "custom-baseten-model",
+]
+
+
+def prompt(label: str, default: str | None = None) -> str:
+ suffix = f" [{default}]" if default else ""
+ value = input(f"{label}{suffix}: ").strip()
+ return value or (default or "")
+
+
+def prompt_required(label: str) -> str:
+ while True:
+ value = prompt(label)
+ if value:
+ return value
+ print("Enter a value", file=sys.stderr)
+
+
+def prompt_choice(label: str, choices: list[str], default: str) -> str:
+ print(label)
+ for index, choice in enumerate(choices, start=1):
+ marker = " (default)" if choice == default else ""
+ print(f" {index}. {choice}{marker}")
+ while True:
+ value = prompt("Choose", default)
+ if value in choices:
+ return value
+ if value.isdigit():
+ index = int(value) - 1
+ if 0 <= index < len(choices):
+ return choices[index]
+ print(f"Enter one of: {', '.join(choices)}", file=sys.stderr)
+
+
+def prompt_multi(
+ label: str,
+ choices: list[str],
+ default: list[str],
+ aliases: dict[str, str] | None = None,
+) -> list[str]:
+ default_text = ",".join(default)
+ canonical_choices = {choice.lower(): choice for choice in choices}
+ aliases = aliases or {}
+ print(label)
+ for choice in choices:
+ print(f" - {choice}")
+ while True:
+ value = prompt("Comma-separated choices, or all", default_text)
+ selections = []
+ for selection in (part.strip().lower() for part in value.split(",")):
+ if not selection:
+ continue
+ if selection == "all":
+ return list(choices)
+ canonical = aliases.get(selection) or canonical_choices.get(selection)
+ if canonical is None:
+ valid = ", ".join([*choices, *sorted(aliases)])
+ print(f"unknown choice '{selection}' (valid: {valid})", file=sys.stderr)
+ break
+ selections.append(canonical)
+ else:
+ selections = dedupe_preserving_order(selections)
+ if selections:
+ return selections
+ print("Choose at least one benchmark", file=sys.stderr)
+
+
+def should_prompt(args: argparse.Namespace) -> bool:
+ if getattr(args, "interactive", False):
+ return True
+ if getattr(args, "yes", False):
+ return False
+ return sys.stdin.isatty() and not getattr(args, "parts", None)
+
+
+def choose_existing_build(args: argparse.Namespace) -> str:
+ try:
+ rows = deployed_function(args, "list_builds").remote(20)
+ except Exception as error:
+ print(
+ f"Could not list builds ({error}); falling back to manual entry.",
+ file=sys.stderr,
+ )
+ return prompt("Existing build id")
+ if not rows:
+ return prompt("No builds found; enter build id")
+ print("Existing builds")
+ for index, row in enumerate(rows, start=1):
+ build_id = row.get("build_id")
+ base = str(row.get("base_sha") or "")[:12]
+ patch = "dirty" if row.get("patch_sha256") else "clean"
+ ready = "ready" if row.get("ready") else "not-ready"
+ print(f" {index}. {build_id} {base} {patch} {ready}")
+ while True:
+ value = prompt("Choose build number or id", "1")
+ if value.isdigit():
+ index = int(value) - 1
+ if 0 <= index < len(rows):
+ return rows[index]["build_id"]
+ if value:
+ return value
+
+
+def configure_interactive_suite(args: argparse.Namespace) -> None:
+ if not should_prompt(args):
+ return
+
+ args.benchmark = prompt_multi(
+ "Which benchmarks should run?",
+ INTERACTIVE_BENCHMARK_CHOICES,
+ ["qna", "rf"],
+ aliases=INTERACTIVE_BENCHMARK_ALIASES,
+ )
+
+ if not getattr(args, "build", None):
+ build_choice = prompt_choice(
+ "Build selection",
+ ["auto", "existing-build"],
+ "auto",
+ )
+ if build_choice == "existing-build":
+ args.build = choose_existing_build(args)
+
+ model_choice = prompt_choice("Base model", INTERACTIVE_MODEL_CHOICES, "sonnet-4.6")
+ if model_choice == "custom-zed-model":
+ args.model = prompt("Zed model id (provider/model)", args.model)
+ elif model_choice == "custom-baseten-model":
+ args.model_provider = "baseten"
+ args.baseten_model = prompt_required("Baseten model id")
+ else:
+ args.model = model_choice
+
+ judge_choice = prompt_choice(
+ "Judge preset",
+ ["auto", "leaderboard", "deepseek-v4-pro", "kimi-k2.7-code", "gpt55"],
+ args.judge or "auto",
+ )
+ args.judge = judge_choice
diff --git a/crates/eval_cli/zed_eval/judge_proxy.py b/crates/eval_cli/zed_eval/judge_proxy.py
new file mode 100644
index 00000000000..0a8066b4f77
--- /dev/null
+++ b/crates/eval_cli/zed_eval/judge_proxy.py
@@ -0,0 +1,104 @@
+"""The in-sandbox judge proxy shim, shared by the Harbor verifier and the
+offline rejudge path so judge calls go through one identical code path
+everywhere.
+
+Stdlib-only and free of any `harbor`/`modal` imports on purpose: `verifier.py`
+imports these constants for the live in-sandbox install, and `rejudge.py` runs
+the very same proxy in a plain container to re-grade stored runs. Keeping the
+shim here means there is exactly one implementation of the upstream fixups
+(Anthropic `response_format` stripping, `max_tokens` -> `max_completion_tokens`,
+the `ZED_JUDGE_MAX_TOKENS` floor) rather than a copy per call site.
+"""
+
+from __future__ import annotations
+
+# Stdlib-only OpenAI-protocol shim. The SWE-Atlas RF judge hardcodes
+# `response_format={"type": "json_object"}`, which Anthropic's OpenAI-compatible
+# endpoint rejects (400: "response_format.type: Input should be 'json_schema'").
+# The shim listens on 127.0.0.1:PORT and forwards to ZED_JUDGE_UPSTREAM, applying
+# upstream-specific fixups:
+# - `response_format` is stripped ONLY for Anthropic (Baseten/OpenAI accept
+# json_object, and the Kimi/DeepSeek judges rely on it);
+# - `max_tokens` is renamed to `max_completion_tokens` (accepted by both
+# Anthropic-compat and OpenAI, including GPT-5.x reasoning models that reject
+# `max_tokens`) and raised to the ZED_JUDGE_MAX_TOKENS floor so the runtime
+# judge matches the offline-calibrated token budget.
+# When ZED_JUDGE_AUTH_ENV is set, it replaces verifier auth with the named
+# sandbox env var; otherwise it passes auth headers through unchanged.
+JUDGE_PROXY_SCRIPT = """\
+import json, os, urllib.request, urllib.error
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+UPSTREAM = os.environ.get("ZED_JUDGE_UPSTREAM", "https://api.anthropic.com/v1").rstrip("/")
+PORT = int(os.environ.get("ZED_JUDGE_PROXY_PORT", "8089"))
+AUTH_ENV = os.environ.get("ZED_JUDGE_AUTH_ENV")
+MAX_TOKENS_FLOOR = int(os.environ.get("ZED_JUDGE_MAX_TOKENS", "0"))
+
+class Handler(BaseHTTPRequestHandler):
+ def do_POST(self):
+ length = int(self.headers.get("Content-Length", 0))
+ body = self.rfile.read(length)
+ try:
+ payload = json.loads(body)
+ if "anthropic" in UPSTREAM:
+ payload.pop("response_format", None)
+ sent = payload.pop("max_tokens", None)
+ current = sent if sent is not None else payload.get("max_completion_tokens")
+ if current is not None or MAX_TOKENS_FLOOR:
+ payload["max_completion_tokens"] = max(current or 0, MAX_TOKENS_FLOOR)
+ body = json.dumps(payload).encode()
+ except Exception as error:
+ print(f"proxy: passing body through unmodified: {error}", flush=True)
+ path = self.path
+ if path.startswith("/v1/"):
+ path = path[len("/v1"):]
+ request = urllib.request.Request(UPSTREAM + path, data=body, method="POST")
+ request.add_header("Content-Type", "application/json")
+ auth_value = os.environ.get(AUTH_ENV, "") if AUTH_ENV else ""
+ if auth_value:
+ request.add_header("Authorization", f"Bearer {auth_value}")
+ else:
+ for header in ("Authorization", "x-api-key"):
+ value = self.headers.get(header)
+ if value:
+ request.add_header(header, value)
+ for header in ("anthropic-version",):
+ value = self.headers.get(header)
+ if value:
+ request.add_header(header, value)
+ try:
+ with urllib.request.urlopen(request, timeout=900) as response:
+ data = response.read()
+ status = response.status
+ except urllib.error.HTTPError as error:
+ data = error.read()
+ status = error.code
+ except Exception as error:
+ data = json.dumps({"error": {"message": str(error)}}).encode()
+ status = 502
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+
+ def log_message(self, format, *args):
+ print("proxy: " + format % args, flush=True)
+
+if __name__ == "__main__":
+ print(f"judge proxy listening on 127.0.0.1:{PORT} -> {UPSTREAM}", flush=True)
+ ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
+"""
+
+# Idempotent starter. Uses bash's /dev/tcp (pgrep isn't in every image) to check
+# the port; the daemon inherits ZED_JUDGE_UPSTREAM from the verifier setup exec
+# env. stdout/stderr/stdin are fully detached so the spawning exec doesn't hang
+# on open pipes.
+JUDGE_PROXY_ENSURE_SCRIPT = """\
+#!/bin/bash
+if ! (exec 3<>/dev/tcp/127.0.0.1/${ZED_JUDGE_PROXY_PORT:-8089}) 2>/dev/null; then
+ nohup python3 /usr/local/lib/zed_judge_proxy.py \\
+ >>/tmp/zed-judge-proxy.log 2>&1 </dev/null &
+ sleep 1
+fi
+"""
diff --git a/crates/eval_cli/zed_eval/launch.py b/crates/eval_cli/zed_eval/launch.py
new file mode 100644
index 00000000000..699cbe3bd39
--- /dev/null
+++ b/crates/eval_cli/zed_eval/launch.py
@@ -0,0 +1,670 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import uuid
+from pathlib import Path
+from typing import Any
+
+from . import benchmarks, config, harness_command, run_index, source
+from .builds import prepare_build_request, resolve_source, validate_build_id
+from .common import (
+ dedupe_preserving_order,
+ default_namespace,
+ deployed_function,
+ modal_call_id,
+ parse_parts,
+ print_json,
+ utc_now,
+ utc_timestamp,
+)
+from .volume import build_ready_on_volume
+
+# Local scratch dir used only for dry-run/plan previews of harness commands.
+PREVIEW_JOBS_DIR = "/tmp/agent-evals/harbor-jobs"
+
+
+def read_task_file(path: str | None) -> list[str]:
+ if not path:
+ return []
+ return [
+ task_name
+ for line in Path(path).read_text().splitlines()
+ if (task_name := line.strip()) and not task_name.startswith("#")
+ ]
+
+
+def baseten_provider_json(
+ *,
+ model_id: str,
+ api_url: str,
+ display_name: str | None,
+ max_tokens: int,
+ max_output_tokens: int,
+) -> str:
+ provider = {
+ config.BASETEN_PROVIDER_ID: {
+ "api_url": api_url,
+ "available_models": [
+ {
+ "name": model_id,
+ "display_name": display_name or model_id,
+ "max_tokens": max_tokens,
+ "max_output_tokens": max_output_tokens,
+ "capabilities": {
+ "tools": True,
+ "images": False,
+ "parallel_tool_calls": False,
+ "prompt_cache_key": False,
+ },
+ }
+ ],
+ }
+ }
+ return json.dumps(provider, separators=(",", ":"))
+
+
+def resolve_model_preset(model: str) -> str:
+ resolved = config.resolve_model_preset(model)
+ if resolved != model:
+ return resolved
+ if model.startswith("baseten:"):
+ return f"{config.BASETEN_PROVIDER_ID}/{model.split(':', 1)[1]}"
+ return model
+
+
+def resolve_model_options(
+ args: argparse.Namespace,
+) -> tuple[str, str | None, list[str]]:
+ raw_model = getattr(args, "model", None) or config.DEFAULT_MODEL
+ model = resolve_model_preset(raw_model)
+ openai_compatible_provider_json = getattr(
+ args, "openai_compatible_provider_json", None
+ )
+ extra_api_secrets = list(getattr(args, "extra_api_secret", None) or [])
+
+ if getattr(args, "model_provider", None) == "baseten":
+ baseten_model = getattr(args, "baseten_model", None) or model
+ if baseten_model.startswith(f"{config.BASETEN_PROVIDER_ID}/"):
+ baseten_model = baseten_model.split("/", 1)[1]
+ if baseten_model.startswith("baseten:"):
+ baseten_model = baseten_model.split(":", 1)[1]
+ model = f"{config.BASETEN_PROVIDER_ID}/{baseten_model}"
+
+ if model.startswith(f"{config.BASETEN_PROVIDER_ID}/"):
+ baseten_model = model.split("/", 1)[1]
+ if not openai_compatible_provider_json:
+ openai_compatible_provider_json = baseten_provider_json(
+ model_id=baseten_model,
+ api_url=getattr(args, "baseten_api_url", config.BASETEN_API_URL),
+ display_name=getattr(args, "baseten_model_display_name", None),
+ max_tokens=getattr(
+ args, "baseten_model_max_tokens", config.BASETEN_DEFAULT_MAX_TOKENS
+ ),
+ max_output_tokens=getattr(
+ args,
+ "baseten_model_max_output_tokens",
+ config.BASETEN_DEFAULT_MAX_OUTPUT_TOKENS,
+ ),
+ )
+
+ return model, openai_compatible_provider_json, extra_api_secrets
+
+
+def derive_run_id(
+ base_run_id: str | None, suite_id: str | None, suffix: str, index: int
+) -> str:
+ """Pick a run id for one leg of a (possibly multi-target) launch.
+
+ An explicit --run-id is used verbatim for the first leg and suffixed for the
+ rest; otherwise legs hang off the suite id, falling back to a timestamped id
+ for a lone run.
+ """
+ if base_run_id:
+ return base_run_id if index == 0 else f"{base_run_id}-{suffix}"
+ if suite_id:
+ return f"{suite_id}-{suffix}-{uuid.uuid4().hex[:6]}"
+ return f"{utc_timestamp()}-{uuid.uuid4().hex[:6]}"
+
+
+def mint_suite_id(args: argparse.Namespace, run_count: int) -> str | None:
+ """A suite id groups multiple runs from one invocation; None for a lone run."""
+ if run_count <= 1:
+ return None
+ explicit = getattr(args, "suite_id", None)
+ if explicit:
+ return explicit
+ prefix = getattr(args, "run_id", None) or "run"
+ return f"{source.sanitize_namespace(prefix)}-{utc_timestamp()}"
+
+
+def build_rejudge_request(args: argparse.Namespace) -> dict[str, Any]:
+ """Build the request for re-grading an existing run with a different judge.
+
+ The positional `run_id` is the *parent* run; the derived run gets a new id
+ under the same experiment so `report`/`list` group them together. Only the
+ judge differs — no build, no agent model, no task selection.
+ """
+ judge_preset = args.judge
+ config.get_judge(judge_preset) # validate before spawning anything remote
+ if getattr(args, "experiment_name", None):
+ experiment_name = source.sanitize_namespace(args.experiment_name)
+ namespace = default_namespace(args)
+ else:
+ entry = run_index.lookup(args.run_id)
+ if not entry:
+ raise ValueError(
+ f"could not locate run '{args.run_id}' in the local run index "
+ f"({run_index.index_path()}). Pass --experiment-name (and "
+ "--namespace if it isn't yours)."
+ )
+ experiment_name = source.sanitize_namespace(entry["experiment_name"])
+ namespace = source.sanitize_namespace(
+ getattr(args, "namespace", None) or entry["namespace"]
+ )
+ parent_namespace = (
+ source.sanitize_namespace(args.parent_namespace)
+ if getattr(args, "parent_namespace", None)
+ else namespace
+ )
+ parent_run_id = args.run_id
+ judge_slug = source.sanitize_namespace(judge_preset)
+ new_run_id = (
+ args.new_run_id
+ or f"{parent_run_id}-rejudge-{judge_slug}-{uuid.uuid4().hex[:6]}"
+ )
+ return {
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "run_id": new_run_id,
+ "parent": {
+ "namespace": parent_namespace,
+ "experiment_name": experiment_name,
+ "run_id": parent_run_id,
+ },
+ "judge_preset": judge_preset,
+ "judge_model": getattr(args, "judge_model", None),
+ "volume_name": args.volume,
+ "api_secret_name": args.api_secret,
+ "created_at": utc_now(),
+ }
+
+
+def command_rejudge(args: argparse.Namespace) -> int:
+ rejudge_request = build_rejudge_request(args)
+ if getattr(args, "dry_run", False) or getattr(args, "plan", False):
+ print_json(rejudge_request)
+ return 0
+
+ controller = deployed_function(args, "rejudge_controller")
+ call = controller.spawn(rejudge_request)
+ run_index.record_run(
+ {**rejudge_request, "volume_name": args.volume, "kind": "rejudge"}
+ )
+ parent = rejudge_request["parent"]
+ print(f"Namespace: {rejudge_request['namespace']}")
+ print(f"Experiment: {rejudge_request['experiment_name']}")
+ print(
+ f"Source run: {parent['namespace']}/{parent['experiment_name']}/"
+ f"{parent['run_id']}"
+ )
+ print(f"New run id: {rejudge_request['run_id']}")
+ print(f"Judge: {rejudge_request['judge_preset']}")
+ print(f"Spawned rejudge controller: {modal_call_id(call)}")
+ print("\nNext steps (run id alone is enough):")
+ print(f" zed-eval status {rejudge_request['run_id']}")
+ print(f" zed-eval report {rejudge_request['run_id']} --fetch")
+ return 0
+
+
+def common_run_request_fields(
+ args: argparse.Namespace,
+ *,
+ namespace: str,
+ run_id: str,
+ experiment_name: str,
+ judge_preset: str | None,
+ build_id: str | None,
+ suite_id: str | None,
+) -> dict[str, Any]:
+ """Fields shared by every benchmark run request."""
+ agent_model, openai_compatible_provider_json, extra_api_secrets = (
+ resolve_model_options(args)
+ )
+ task_names = dedupe_preserving_order(
+ read_task_file(getattr(args, "tasks", None))
+ + (getattr(args, "include_task_name", None) or [])
+ )
+ return {
+ "created_at": utc_now(),
+ "namespace": namespace,
+ "run_id": run_id,
+ "experiment_name": experiment_name,
+ "volume_name": args.volume,
+ "api_secret_name": args.api_secret,
+ "modal_token_secret_name": args.modal_token_secret,
+ "orchestration": config.orchestration_info(),
+ "agent_model": agent_model,
+ "judge_preset": judge_preset,
+ "judge_model": getattr(args, "judge_model", None),
+ "build_id": build_id,
+ "task_names": task_names,
+ "n_tasks": getattr(args, "n_tasks", None),
+ "n_concurrent": getattr(args, "n_concurrent", config.DEFAULT_N_CONCURRENT),
+ "override_cpus": getattr(args, "override_cpus", config.DEFAULT_OVERRIDE_CPUS),
+ "override_memory_mb": getattr(
+ args, "override_memory_mb", config.DEFAULT_OVERRIDE_MEMORY_MB
+ ),
+ "sandbox_timeout_secs": getattr(
+ args, "sandbox_timeout_secs", config.DEFAULT_SANDBOX_TIMEOUT_SECS
+ ),
+ "sandbox_idle_timeout_secs": getattr(
+ args, "sandbox_idle_timeout_secs", config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS
+ ),
+ "build_wait_timeout_secs": getattr(args, "build_wait_timeout_secs", 7200),
+ "extra_harbor_args": list(getattr(args, "extra_harbor_arg", None) or []),
+ "openai_compatible_provider_json": openai_compatible_provider_json,
+ "anthropic_available_models_json": getattr(
+ args, "anthropic_available_models_json", None
+ ),
+ "extra_api_secrets": dedupe_preserving_order(extra_api_secrets),
+ "suite_id": suite_id,
+ }
+
+
+def benchmark_plan_entry(
+ benchmark_id: str, run_request: dict[str, Any], build_request: dict[str, Any] | None
+) -> dict[str, Any]:
+ return {
+ "benchmark": benchmark_id,
+ "run_id": run_request["run_id"],
+ "harness": run_request["benchmark"]["harness"],
+ "model": run_request["agent_model"],
+ "judge": run_request.get("judge_preset"),
+ "build_id": run_request.get("build_id"),
+ "will_build": build_request is not None,
+ "n_tasks": run_request.get("n_tasks"),
+ }
+
+
+def print_plan(
+ summaries: list[dict[str, Any]],
+ details: list[tuple[str, dict[str, Any], dict[str, Any] | None]],
+ *,
+ verbose: bool,
+) -> None:
+ print("Plan:")
+ print_json(summaries)
+ if verbose:
+ for header, run_request, build_request in details:
+ print(f"\n=== {header} ===")
+ print_dry_run(run_request, build_request)
+
+
+def print_dry_run(
+ run_request: dict[str, Any], build_request: dict[str, Any] | None
+) -> None:
+ print("Run request:")
+ print_json(run_request)
+ if build_request:
+ print("\nBuild request:")
+ build_preview = dict(build_request)
+ patch = build_preview.pop("patch", "")
+ build_preview["patch_line_count"] = len(patch.splitlines())
+ build_preview["source"] = source.public_source_info(
+ build_request.get("source") or {}
+ )
+ print_json(build_preview)
+ command = config.redacted_command(
+ harness_command.build_harness_command(run_request, PREVIEW_JOBS_DIR)
+ )
+ print("\nHarness command:")
+ print(command)
+
+
+def execute_prepared_runs(
+ args: argparse.Namespace,
+ prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]],
+ plan_entry,
+) -> int:
+ if args.plan:
+ print_plan(
+ [
+ plan_entry(label, run_request, build_request)
+ for label, run_request, build_request in prepared
+ ],
+ prepared,
+ verbose=args.verbose,
+ )
+ return 0
+ if args.dry_run:
+ for label, run_request, build_request in prepared:
+ print(f"\n=== {label} ===")
+ print_dry_run(run_request, build_request)
+ return 0
+
+ spawned_builds: set[str] = set()
+ for label, run_request, build_request in prepared:
+ print(f"\n=== Launching {label} ===")
+ launch_prepared_run(args, run_request, build_request, spawned_builds)
+ return 0
+
+
+def launch_prepared_run(
+ args: argparse.Namespace,
+ run_request: dict[str, Any],
+ build_request: dict[str, Any] | None,
+ spawned_builds: set[str] | None = None,
+) -> None:
+ build_function = None
+ if build_request:
+ run_request["source"] = source.public_source_info(build_request["source"])
+ print_untracked_warning(build_request)
+ build_function = deployed_function(args, "build_eval_cli")
+ record_function = deployed_function(args, "create_run_record")
+ controller_function = deployed_function(args, "run_controller")
+
+ record_state = record_function.remote(run_request)
+ run_index.record_run(run_request)
+
+ print(f"Namespace: {run_request['namespace']}")
+ print(f"Experiment: {run_request['experiment_name']}")
+ print(f"Run id: {run_request['run_id']}")
+ print(f"Volume: {run_request['volume_name']}")
+ print(f"Run state: {record_state['status']}")
+ print(f"Model: {run_request['agent_model']}")
+ if run_request.get("judge_preset"):
+ print(f"Judge: {run_request['judge_preset']}")
+ if run_request.get("suite_id"):
+ print(f"Suite: {run_request['suite_id']}")
+ if run_request.get("build_id"):
+ print(f"Build id: {run_request['build_id']}")
+ if run_request.get("task_names"):
+ print(f"Tasks: {len(run_request['task_names'])} explicit task(s)")
+ elif run_request.get("n_tasks"):
+ print(f"Tasks: Harbor --n-tasks {run_request['n_tasks']}")
+ else:
+ print("Tasks: full dataset selection")
+
+ build_id = run_request.get("build_id")
+ if build_function is not None and build_id not in (spawned_builds or set()):
+ build_call = build_function.spawn(build_request)
+ print(f"Spawned build: {modal_call_id(build_call)}")
+ if spawned_builds is not None and build_id:
+ spawned_builds.add(build_id)
+ controller_call = controller_function.spawn(run_request)
+ print(f"Spawned controller: {modal_call_id(controller_call)}")
+
+ run_id = run_request["run_id"]
+ print(
+ "\nNext steps (run id alone is enough; namespace/experiment are resolved "
+ "from this machine's local run index):"
+ )
+ print(f" zed-eval status {run_id}")
+ print(f" zed-eval logs {run_id}")
+ print(f" zed-eval report {run_id} --fetch")
+
+
+def resolve_benchmark_judge(
+ args: argparse.Namespace, benchmark: benchmarks.Benchmark
+) -> str | None:
+ if not benchmark.needs_judge:
+ return None
+ judge = getattr(args, "judge", None) or config.DEFAULT_JUDGE_PRESET
+ if judge == "auto":
+ return benchmark.default_judge or "leaderboard"
+ config.get_judge(judge)
+ return judge
+
+
+def benchmark_metadata_for_run(
+ args: argparse.Namespace, benchmark: benchmarks.Benchmark
+) -> dict[str, object]:
+ metadata = benchmarks.benchmark_metadata(benchmark)
+ dataset = metadata.get("dataset")
+ if (
+ isinstance(dataset, dict)
+ and dataset.get("repo_url") == benchmarks.SWE_ATLAS_REPO_URL
+ ):
+ dataset["repo_url"] = (
+ getattr(args, "swe_atlas_repo_url", None) or benchmarks.SWE_ATLAS_REPO_URL
+ )
+ dataset["repo_ref"] = (
+ getattr(args, "swe_atlas_repo_ref", None) or benchmarks.SWE_ATLAS_REPO_REF
+ )
+ return metadata
+
+
+def print_untracked_warning(build_request: dict[str, Any]) -> None:
+ build_source = build_request.get("source") or {}
+ untracked_files = build_source.get("untracked_files") or []
+ if untracked_files:
+ print(
+ f"Warning: proceeding with {len(untracked_files)} untracked file(s) "
+ "not included in the build patch.",
+ file=sys.stderr,
+ )
+
+
+def prepare_shared_build(
+ args: argparse.Namespace,
+) -> tuple[str | None, dict[str, Any] | None]:
+ """Resolve the build once for a whole (possibly multi-benchmark) run.
+
+ Returns `(build_id, build_request)`. `build_request` is None when the target
+ build already exists; otherwise the caller should spawn the build.
+ """
+ explicit_build_id = getattr(args, "build", None)
+ validate_build_id(explicit_build_id)
+ if (
+ explicit_build_id
+ and not getattr(args, "plan", False)
+ and not getattr(args, "dry_run", False)
+ and build_ready_on_volume(args, explicit_build_id)
+ ):
+ print(f"Reusing existing build {explicit_build_id} (already on volume)")
+ return explicit_build_id, None
+
+ base_sha, clean_source, source_label, pre_resolved = resolve_source(args)
+
+ build_request = prepare_build_request(
+ base_sha=base_sha,
+ patch_path=getattr(args, "patch_path", None),
+ build_id=explicit_build_id,
+ allow_untracked=getattr(args, "allow_untracked", False),
+ require_clean=getattr(args, "require_clean", False),
+ repo_url=getattr(args, "repo_url", None),
+ clean_source=clean_source,
+ source_label=source_label,
+ pre_resolved_base_sha=pre_resolved,
+ )
+ build_id = build_request["build_id"]
+
+ if not getattr(args, "plan", False) and not getattr(args, "dry_run", False):
+ if build_ready_on_volume(args, build_id):
+ print(f"Reusing existing build {build_id} (already on volume)")
+ return build_id, None
+
+ return build_id, build_request
+
+
+def command_build(args: argparse.Namespace) -> int:
+ build_id, build_request = prepare_shared_build(args)
+ if build_request is None:
+ return 0
+
+ print(f"Build id: {build_id}")
+ print(f"Base sha: {build_request['base_sha']}")
+ print(f"Patch sha256: {build_request['patch_sha256'] or '(none)'}")
+ print_untracked_warning(build_request)
+
+ build_function = deployed_function(args, "build_eval_cli")
+ if args.detach:
+ call = build_function.spawn(build_request)
+ print(f"Spawned build: {modal_call_id(call)}")
+ else:
+ result = build_function.remote(build_request)
+ print_json(result)
+ return 0
+
+
+def build_benchmark_run_request(
+ args: argparse.Namespace,
+ *,
+ benchmark_id: str,
+ build_id: str | None,
+ suite_id: str | None,
+ index: int,
+ run_id_suffix: str | None = None,
+) -> dict[str, Any]:
+ benchmark = benchmarks.get_benchmark(benchmark_id)
+ run_id = derive_run_id(
+ getattr(args, "run_id", None), suite_id, run_id_suffix or benchmark_id, index
+ )
+ # Staff mode is off by default for remote runs: it enables the sandboxed
+ # terminal, which hangs inside Modal sandboxes.
+ extra_env = {"EVAL_CLI_STAFF": "true" if getattr(args, "staff", False) else "false"}
+
+ return {
+ **common_run_request_fields(
+ args,
+ namespace=default_namespace(args),
+ run_id=run_id,
+ # The benchmark id doubles as the experiment name for run storage
+ # paths (runs/<namespace>/<experiment_name>/<run_id>), keeping
+ # monitoring and fetch uniform across benchmarks.
+ experiment_name=source.sanitize_namespace(benchmark_id),
+ judge_preset=resolve_benchmark_judge(args, benchmark),
+ build_id=build_id,
+ suite_id=suite_id,
+ ),
+ "benchmark": benchmark_metadata_for_run(args, benchmark),
+ "eval_cli_timeout": getattr(args, "eval_cli_timeout", None),
+ "extra_env": extra_env,
+ }
+
+
+def prepare_runs_for_benchmarks(
+ args: argparse.Namespace,
+ benchmark_ids: list[str],
+ *,
+ suite_id: str | None,
+ label_for_benchmark,
+ mark_swe_atlas_parts: bool = False,
+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:
+ if not benchmark_ids:
+ raise ValueError("choose at least one benchmark to run")
+
+ build_id, build_request = prepare_shared_build(args)
+ prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]] = []
+ for index, benchmark_id in enumerate(benchmark_ids):
+ label = label_for_benchmark(benchmark_id)
+ run_request = build_benchmark_run_request(
+ args,
+ benchmark_id=benchmark_id,
+ build_id=build_id,
+ suite_id=suite_id,
+ index=index,
+ run_id_suffix=label,
+ )
+ if (
+ mark_swe_atlas_parts
+ and benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.values()
+ ):
+ run_request["suite_part"] = label
+ # Only the first run carries the build_request; launch_prepared_run
+ # dedups the actual spawn via spawned_builds anyway.
+ prepared.append((label, run_request, build_request if index == 0 else None))
+ return prepared
+
+
+def prepare_benchmark_runs(
+ args: argparse.Namespace,
+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:
+ benchmark_ids = benchmarks.resolve_benchmarks(
+ list(getattr(args, "benchmark", None) or [])
+ )
+ return prepare_runs_for_benchmarks(
+ args,
+ benchmark_ids,
+ suite_id=mint_suite_id(args, len(benchmark_ids)),
+ label_for_benchmark=lambda benchmark_id: benchmark_id,
+ )
+
+
+def command_run(args: argparse.Namespace) -> int:
+ return execute_prepared_runs(
+ args,
+ prepare_benchmark_runs(args),
+ benchmark_plan_entry,
+ )
+
+
+def resolve_suite_parts(args: argparse.Namespace) -> list[str]:
+ parts = parse_parts([args.parts] if getattr(args, "parts", None) else [])
+ if parts:
+ return parts
+ raise ValueError(
+ "choose at least one SWE-Atlas part with --parts (e.g. --parts rf,qna or --parts all)"
+ )
+
+
+def suite_plan_entry(
+ part: str, run_request: dict[str, Any], build_request: dict[str, Any] | None
+) -> dict[str, Any]:
+ return {
+ "part": part,
+ **benchmark_plan_entry(
+ run_request["benchmark"]["id"], run_request, build_request
+ ),
+ }
+
+
+def suite_entry_label(benchmark_id: str) -> str:
+ for part, part_benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.items():
+ if part_benchmark_id == benchmark_id:
+ return part
+ return benchmark_id
+
+
+def prepare_benchmark_suite(
+ args: argparse.Namespace, selectors: list[str]
+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:
+ benchmark_ids = benchmarks.resolve_benchmarks(selectors)
+ if not benchmark_ids:
+ raise ValueError("choose at least one benchmark to run")
+ if getattr(args, "zed_version", None):
+ args.clean_source = True
+ args.require_clean = True
+ timestamp = utc_timestamp()
+ prefix_seed = args.run_id_prefix or args.experiment_prefix or "swe-atlas"
+ if getattr(args, "zed_version", None):
+ prefix_seed = f"{prefix_seed}-{source.sanitize_namespace(args.zed_version)}"
+ suite_id = args.suite_id or f"{source.sanitize_namespace(prefix_seed)}-{timestamp}"
+
+ return prepare_runs_for_benchmarks(
+ args,
+ benchmark_ids,
+ suite_id=suite_id,
+ label_for_benchmark=suite_entry_label,
+ mark_swe_atlas_parts=True,
+ )
+
+
+def prepare_suite(
+ args: argparse.Namespace,
+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:
+ return prepare_benchmark_suite(args, resolve_suite_parts(args))
+
+
+def command_swe_atlas(args: argparse.Namespace) -> int:
+ from .interactive import configure_interactive_suite
+
+ configure_interactive_suite(args)
+ prepared = (
+ prepare_benchmark_suite(args, args.benchmark)
+ if getattr(args, "benchmark", None)
+ else prepare_suite(args)
+ )
+ return execute_prepared_runs(args, prepared, suite_plan_entry)
diff --git a/crates/eval_cli/zed_eval/modal_app.py b/crates/eval_cli/zed_eval/modal_app.py
new file mode 100644
index 00000000000..4ba1c478908
--- /dev/null
+++ b/crates/eval_cli/zed_eval/modal_app.py
@@ -0,0 +1,1008 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import pathlib
+import shlex
+import shutil
+import subprocess
+import tarfile
+import time
+import traceback
+import uuid
+from typing import Any
+
+import modal
+
+from . import cleanup, config, harness_command, rejudge, source
+from .common import load_json, safe_extract_archive, utc_now, write_json
+from .source import (
+ BUILD_TARGET,
+ CARGO_ZIGBUILD_VERSION,
+ RUST_IMAGE,
+ RUST_VERSION,
+ ZIG_VERSION,
+ build_toolchain_info,
+)
+
+APP_NAME = os.environ.get("AGENT_EVALS_APP_NAME", "agent-evals")
+VOLUME_NAME = os.environ.get("AGENT_EVALS_VOLUME", "agent-evals")
+MODAL_TOKEN_SECRET_NAME = os.environ.get(
+ "AGENT_EVALS_MODAL_TOKEN_SECRET", "agent-evals-modal-token"
+)
+# LLM-providers secret mounted into the rejudge controller so the in-controller
+# judge proxy can read the judge's API key (e.g. BASETEN_API_KEY) the same way a
+# trial sandbox does. Override at deploy time if the secret is named differently.
+LLM_PROVIDERS_SECRET_NAME = os.environ.get(
+ "AGENT_EVALS_LLM_PROVIDERS_SECRET", config.DEFAULT_LLM_PROVIDERS_SECRET_NAME
+)
+REPO_URL = os.environ.get(
+ "AGENT_EVALS_REPO_URL", "https://github.com/zed-industries/zed.git"
+)
+ZIG_URL = (
+ f"https://ziglang.org/download/{ZIG_VERSION}/zig-x86_64-linux-{ZIG_VERSION}.tar.xz"
+)
+CLEANUP_BUILD_RETENTION_DAYS = float(
+ os.environ.get("AGENT_EVALS_BUILD_RETENTION_DAYS", "14")
+)
+
+app = modal.App(APP_NAME)
+volume = modal.Volume.from_name(VOLUME_NAME, create_if_missing=True)
+
+
+def reload_volume() -> None:
+ try:
+ volume.reload()
+ except Exception:
+ pass
+
+
+build_image = (
+ modal.Image.from_registry(RUST_IMAGE, add_python="3.13")
+ .apt_install("cmake", "build-essential", "curl", "xz-utils", "git")
+ .run_commands(
+ f"rustup toolchain install {RUST_VERSION} --profile minimal"
+ " --component rustfmt --component clippy"
+ " --component rust-analyzer --component rust-src"
+ " --target wasm32-wasip2 --target wasm32-unknown-unknown"
+ " --target x86_64-unknown-linux-musl --target x86_64-unknown-linux-gnu",
+ f"mkdir -p /opt/zig && curl -fsSL {ZIG_URL}"
+ " | tar -xJ -C /opt/zig --strip-components=1"
+ " && ln -s /opt/zig/zig /usr/local/bin/zig",
+ f"cargo install --locked cargo-zigbuild --version {CARGO_ZIGBUILD_VERSION}",
+ )
+)
+
+controller_image = (
+ modal.Image.debian_slim(python_version="3.13")
+ .apt_install("bash", "ca-certificates", "curl", "git", "tar")
+ .pip_install(f"modal=={config.MODAL_VERSION}", "uv")
+ .run_commands(
+ f"uv tool install harbor=={config.HARBOR_VERSION}"
+ f" --with 'modal=={config.MODAL_VERSION}' && "
+ "ln -sf /root/.local/bin/harbor /usr/local/bin/harbor",
+ # Pier is a Harbor fork that adds per-agent network allowlists, required
+ # to run DeepSWE's air-gapped (`allow_internet = false`) tasks while
+ # still letting eval-cli reach the model API. Kept non-fatal so a Pier
+ # install problem can't break the Harbor benchmarks that share this image;
+ # DeepSWE runs surface a clear "pier not found" error at runtime instead.
+ "(uv tool install git+https://github.com/datacurve-ai/pier"
+ f" --with 'modal=={config.MODAL_VERSION}' && "
+ "ln -sf /root/.local/bin/pier /usr/local/bin/pier) || "
+ "echo 'WARNING: pier install failed; DeepSWE runs will not work'",
+ )
+)
+
+
+def provision_benchmark_dataset(run_request: dict[str, Any], log: Any) -> None:
+ """Clone a path-backed benchmark dataset (SWE-Atlas tw, DeepSWE) into the
+ location the harness command expects. Registry datasets need no provisioning;
+ the harness pulls them from its hub."""
+ benchmark = run_request["benchmark"]
+ dataset = benchmark.get("dataset") or {}
+ kind = dataset.get("kind")
+ if kind not in ("path", "pier_path"):
+ return
+
+ repo_url = dataset.get("repo_url")
+ repo_ref = dataset.get("repo_ref") or "main"
+ if not repo_url:
+ raise ValueError(f"benchmark {benchmark['id']} path dataset requires repo_url")
+
+ clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))
+ if clone_dir.exists():
+ shutil.rmtree(clone_dir)
+ clone_dir.mkdir(parents=True, exist_ok=True)
+ log(f"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}")
+ subprocess.run(["git", "init", "-q", str(clone_dir)], check=True)
+ subprocess.run(
+ ["git", "fetch", "--depth", "1", repo_url, repo_ref],
+ cwd=clone_dir,
+ check=True,
+ )
+ subprocess.run(["git", "checkout", "-q", "FETCH_HEAD"], cwd=clone_dir, check=True)
+ data_dir = pathlib.Path(harness_command.dataset_path(benchmark))
+ if not data_dir.exists():
+ raise FileNotFoundError(f"benchmark dataset directory not found: {data_dir}")
+
+
+def output_of(command: list[str]) -> str:
+ try:
+ return subprocess.run(
+ command,
+ check=True,
+ capture_output=True,
+ text=True,
+ ).stdout.strip()
+ except subprocess.CalledProcessError as error:
+ return f"unknown ({error})"
+
+
+ image=build_image,
+ # Right-sized from the original 16 cpu / 32 GB after observing peak usage of
+ # ~10 cores and ~10 GB. ephemeral_disk stays at Modal's 512 GiB floor (the
+ # minimum allowed for a function), so it isn't a tunable here.
+ cpu=12,
+ memory=24576,
+ ephemeral_disk=524288,
+ timeout=7200,
+ volumes={"/data": volume},
+)
+def build_eval_cli(build_request: dict[str, Any]) -> dict[str, Any]:
+ def run(command: str, **kwargs: Any) -> None:
+ print(f"+ {command}", flush=True)
+ subprocess.run(command, shell=True, check=True, **kwargs)
+
+ reload_volume()
+
+ build_id = build_request["build_id"]
+ source_info = build_request.get("source") or {"type": "git_patch"}
+ base_sha = source_info.get("base_sha") or build_request["base_sha"]
+ patch = build_request.get("patch") or ""
+ build_dir = pathlib.Path("/data/builds") / build_id
+ ready_path = build_dir / "READY"
+ binary_path = build_dir / "eval-cli"
+ build_info_path = build_dir / "build-info.json"
+ # The lease lives outside build_dir so it never interferes with the atomic
+ # move of the finished build directory below.
+ building_path = pathlib.Path("/data/build-locks") / f"{build_id}.json"
+ lease_ttl = int(build_request.get("build_wait_timeout_secs") or 7200)
+ owner = uuid.uuid4().hex
+
+ def reuse_existing() -> dict[str, Any]:
+ build_info = load_json(build_info_path) or {}
+ build_info.setdefault("build_id", build_id)
+ build_info["reused"] = True
+ volume.commit()
+ print(f"Reusing existing build {build_id}", flush=True)
+ return build_info
+
+ if ready_path.exists() and binary_path.exists():
+ return reuse_existing()
+ if build_dir.exists():
+ raise RuntimeError(
+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"
+ )
+
+ # Single-flight lease: if another invocation is already compiling this exact
+ # build, wait for it to finish rather than running a second multi-minute
+ # compile. The lease is best-effort (the volume has no atomic compare-and-swap);
+ # the atomic move below still guarantees correctness if two builds slip through.
+ deadline = time.time() + lease_ttl
+ while True:
+ reload_volume()
+ if ready_path.exists() and binary_path.exists():
+ return reuse_existing()
+ if build_dir.exists():
+ raise RuntimeError(
+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"
+ )
+ lease = load_json(building_path)
+ now = time.time()
+ held_by_other = (
+ isinstance(lease, dict)
+ and lease.get("owner") != owner
+ and (now - float(lease.get("epoch") or 0)) < lease_ttl
+ )
+ if held_by_other and time.time() < deadline:
+ print(
+ f"Build {build_id} is being compiled elsewhere; waiting",
+ flush=True,
+ )
+ time.sleep(15)
+ continue
+ break
+ # Claim the lease (write_json creates /data/build-locks). build_dir is
+ # intentionally NOT created here so the atomic move below still works.
+ write_json(building_path, {"owner": owner, "epoch": time.time(), "at": utc_now()})
+ volume.commit()
+
+ rustc_version = output_of(["rustc", "--version"])
+ zig_version = output_of(["zig", "version"])
+ cargo_zigbuild_version = output_of(["cargo-zigbuild", "--version"])
+
+ workdir = pathlib.Path("/build/zed")
+ if workdir.exists():
+ shutil.rmtree(workdir)
+ workdir.mkdir(parents=True, exist_ok=True)
+ os.chdir(workdir)
+
+ repo_url = source_info.get("repo_url") or REPO_URL
+ run("git init -q .")
+ run(f"git fetch --depth 1 {shlex.quote(repo_url)} {shlex.quote(base_sha)}")
+ run("git checkout -q FETCH_HEAD")
+
+ if patch.strip():
+ patch_file = pathlib.Path("/build/source.patch")
+ patch_file.write_text(patch)
+ run(f"git apply --stat {shlex.quote(str(patch_file))}")
+ run(f"git apply {shlex.quote(str(patch_file))}")
+
+ patch_sha256 = source_info.get("patch_sha256")
+
+ run(
+ "cargo zigbuild --release --package eval_cli --target x86_64-unknown-linux-musl"
+ )
+
+ built = workdir / "target/x86_64-unknown-linux-musl/release/eval-cli"
+ run(f"strip {built}")
+ binary_bytes = built.read_bytes()
+ binary_sha256 = hashlib.sha256(binary_bytes).hexdigest()
+
+ temporary_dir = pathlib.Path("/data/tmp/builds") / f"{build_id}-{uuid.uuid4().hex}"
+ temporary_dir.mkdir(parents=True, exist_ok=True)
+ shutil.copy(built, temporary_dir / "eval-cli")
+ (temporary_dir / "eval-cli").chmod(0o755)
+ if patch.strip():
+ (temporary_dir / "source.patch").write_text(patch)
+ write_json(temporary_dir / "source-info.json", source_info)
+
+ build_info = {
+ "build_id": build_id,
+ "base_sha": base_sha,
+ "patch_sha256": patch_sha256,
+ "rustc_version": rustc_version,
+ "zig_version": zig_version,
+ "cargo_zigbuild_version": cargo_zigbuild_version,
+ "built_at_utc": utc_now(),
+ "binary_size_bytes": len(binary_bytes),
+ "binary_sha256": binary_sha256,
+ "target": BUILD_TARGET,
+ "rust_image": RUST_IMAGE,
+ "toolchain": build_toolchain_info(),
+ "source": source_info,
+ }
+ write_json(temporary_dir / "build-info.json", build_info)
+ (temporary_dir / "READY").write_text(utc_now() + "\n")
+
+ if not build_dir.exists():
+ build_dir.parent.mkdir(parents=True, exist_ok=True)
+ shutil.move(str(temporary_dir), str(build_dir))
+ elif ready_path.exists() and binary_path.exists():
+ shutil.rmtree(temporary_dir, ignore_errors=True)
+ build_info = load_json(build_info_path) or build_info
+ build_info["reused_after_race"] = True
+ else:
+ shutil.rmtree(temporary_dir, ignore_errors=True)
+ raise RuntimeError(
+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"
+ )
+
+ # Release the single-flight lease now that READY exists.
+ try:
+ building_path.unlink()
+ except OSError:
+ pass
+ volume.commit()
+ print(f"Committed build {build_id} to volume '{VOLUME_NAME}'", flush=True)
+ return build_info
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def list_builds(limit: int = 50) -> list[dict[str, Any]]:
+ reload_volume()
+ builds_dir = pathlib.Path("/data/builds")
+ rows = []
+ for build_dir in builds_dir.iterdir() if builds_dir.exists() else []:
+ if not build_dir.is_dir():
+ continue
+ build_info = load_json(build_dir / "build-info.json") or {}
+ ready = (build_dir / "READY").exists() and (build_dir / "eval-cli").exists()
+ rows.append(
+ {
+ "build_id": build_dir.name,
+ "ready": ready,
+ "base_sha": build_info.get("base_sha"),
+ "patch_sha256": build_info.get("patch_sha256"),
+ "built_at_utc": build_info.get("built_at_utc"),
+ "binary_sha256": build_info.get("binary_sha256"),
+ "source": build_info.get("source"),
+ }
+ )
+ rows.sort(key=lambda row: row.get("built_at_utc") or "", reverse=True)
+ return rows[:limit]
+
+
+def run_row(
+ namespace_dir: pathlib.Path, experiment_dir: pathlib.Path, run_dir: pathlib.Path
+) -> dict[str, Any]:
+ state = load_json(run_dir / "state.json") or {}
+ request = load_json(run_dir / "request.json") or {}
+ metadata = load_json(run_dir / "run-metadata.json") or {}
+ summary = load_json(run_dir / "summary.json") or {}
+ benchmark = request.get("benchmark") or metadata.get("benchmark") or {}
+ benchmark_id = benchmark.get("id") if isinstance(benchmark, dict) else None
+ suite_id = request.get("suite_id") or metadata.get("suite_id")
+ return {
+ "namespace": namespace_dir.name,
+ "experiment_name": experiment_dir.name,
+ "run_id": run_dir.name,
+ "status": state.get("status"),
+ "updated_at": state.get("updated_at"),
+ "created_at": metadata.get("created_at")
+ or state.get("created_at")
+ or request.get("created_at"),
+ "agent_model": metadata.get("agent_model"),
+ "judge_preset": metadata.get("judge_preset"),
+ "judge_model": metadata.get("judge_model"),
+ "build_id": metadata.get("build_id") or state.get("build_id"),
+ "trial_count": summary.get("trial_count"),
+ "has_archive": summary.get("has_archive"),
+ "suite_id": suite_id,
+ "part": request.get("suite_part"),
+ "benchmark": benchmark_id,
+ }
+
+
+def scan_runs(
+ *,
+ namespace: str | None = None,
+ experiment_name: str | None = None,
+ suite_id: str | None = None,
+) -> list[dict[str, Any]]:
+ runs_root = pathlib.Path("/data/runs")
+ rows = []
+ if not runs_root.exists():
+ return []
+ namespaces = (
+ [runs_root / namespace] if namespace else [path for path in runs_root.iterdir()]
+ )
+ for namespace_dir in namespaces:
+ if not namespace_dir.is_dir():
+ continue
+ experiments = (
+ [namespace_dir / experiment_name]
+ if experiment_name
+ else [path for path in namespace_dir.iterdir()]
+ )
+ for experiment_dir in experiments:
+ if not experiment_dir.is_dir():
+ continue
+ for run_dir in experiment_dir.iterdir():
+ if not run_dir.is_dir():
+ continue
+ row = run_row(namespace_dir, experiment_dir, run_dir)
+ if suite_id and row.get("suite_id") != suite_id:
+ continue
+ rows.append(row)
+ rows.sort(
+ key=lambda row: row.get("updated_at") or row.get("created_at") or "",
+ reverse=True,
+ )
+ return rows
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def list_runs(
+ namespace: str | None = None,
+ experiment_name: str | None = None,
+ limit: int = 50,
+) -> list[dict[str, Any]]:
+ reload_volume()
+ rows = scan_runs(namespace=namespace, experiment_name=experiment_name)
+ return rows[:limit]
+
+
+def _run_cleanup(
+ *,
+ dry_run: bool,
+ build_retention_days: float | None = None,
+) -> dict[str, Any]:
+ reload_volume()
+ result = cleanup.prune_artifacts(
+ pathlib.Path("/data"),
+ dry_run=dry_run,
+ build_retention_days=(
+ build_retention_days
+ if build_retention_days is not None
+ else CLEANUP_BUILD_RETENTION_DAYS
+ ),
+ )
+ if not dry_run:
+ volume.commit()
+ print(json.dumps(result, indent=2), flush=True)
+ return result
+
+
+ image=controller_image,
+ cpu=1,
+ memory=2048,
+ timeout=1800,
+ volumes={"/data": volume},
+ schedule=modal.Period(days=1),
+)
+def cleanup_scheduled() -> dict[str, Any]:
+ """Daily prune of stale build artifacts. Never touches eval results."""
+ return _run_cleanup(dry_run=False)
+
+
+ image=controller_image,
+ cpu=1,
+ memory=2048,
+ timeout=1800,
+ volumes={"/data": volume},
+)
+def cleanup_artifacts(request: dict[str, Any]) -> dict[str, Any]:
+ """On-demand prune (the `zed-eval cleanup` command), supporting --dry-run
+ and retention overrides."""
+ return _run_cleanup(
+ dry_run=bool(request.get("dry_run")),
+ build_retention_days=request.get("build_retention_days"),
+ )
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def read_run_provenance(
+ namespace: str, experiment_name: str, run_id: str
+) -> dict[str, Any]:
+ """Provenance a baseline record needs for one run: the launch request
+ (model/judge/resources/build_id), the run summary (status), and the build's
+ build-info (base_sha/patch_sha256/source label)."""
+ reload_volume()
+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id
+ request = load_json(run_dir / "request.json") or {}
+ summary = load_json(run_dir / "summary.json") or {}
+ metadata = load_json(run_dir / "run-metadata.json") or {}
+ build_id = request.get("build_id") or metadata.get("build_id")
+ build_info: dict[str, Any] = {}
+ if build_id:
+ build_info = (
+ load_json(pathlib.Path("/data/builds") / build_id / "build-info.json") or {}
+ )
+ return {
+ "request": request,
+ "summary": summary,
+ "metadata": metadata,
+ "build_info": build_info,
+ }
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def record_baseline(record: dict[str, Any]) -> dict[str, Any]:
+ """Write a baseline-of-record keyed by (experiment, model). Supersedes any
+ existing current baseline (archived under history/) and refreshes the
+ discoverable index."""
+ reload_volume()
+ experiment_slug = source.sanitize_namespace(record["experiment"])
+ model_slug = source.sanitize_namespace(record.get("model_slug") or record["model"])
+ base_dir = pathlib.Path("/data/baselines") / experiment_slug / model_slug
+ current_path = base_dir / "current.json"
+
+ existing = load_json(current_path)
+ if existing:
+ previous_sha = source.sanitize_namespace(
+ str(existing.get("base_sha") or "unknown")
+ )
+ write_json(base_dir / "history" / f"{previous_sha}.json", existing)
+
+ payload = {**record, "updated_at": utc_now()}
+ write_json(current_path, payload)
+
+ index_path = pathlib.Path("/data/baselines/index.json")
+ index = load_json(index_path) or {}
+ entries = index.get("baselines")
+ if not isinstance(entries, list):
+ entries = []
+ entries = [
+ entry
+ for entry in entries
+ if not (
+ entry.get("experiment") == record["experiment"]
+ and entry.get("model") == record["model"]
+ )
+ ]
+ entries.append(
+ {
+ "experiment": record["experiment"],
+ "model": record["model"],
+ "base_sha": record.get("base_sha"),
+ "base_ref": record.get("base_ref"),
+ "on_main": record.get("on_main"),
+ "clean": record.get("clean"),
+ "judge": record.get("judge"),
+ "run_id": (record.get("run") or {}).get("run_id"),
+ "recorded_at": record.get("recorded_at"),
+ "path": f"baselines/{experiment_slug}/{model_slug}/current.json",
+ }
+ )
+ entries.sort(
+ key=lambda entry: (entry.get("experiment") or "", entry.get("model") or "")
+ )
+ write_json(index_path, {"baselines": entries})
+ volume.commit()
+ return payload
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def read_baselines(
+ experiment: str | None = None,
+ model_slug: str | None = None,
+ include_history: bool = False,
+) -> dict[str, Any]:
+ """List all current baselines (no args) or show one (experiment+model_slug),
+ optionally with superseded history."""
+ reload_volume()
+ root = pathlib.Path("/data/baselines")
+ if experiment and model_slug:
+ base_dir = (
+ root
+ / source.sanitize_namespace(experiment)
+ / source.sanitize_namespace(model_slug)
+ )
+ result: dict[str, Any] = {"current": load_json(base_dir / "current.json")}
+ if include_history:
+ history = []
+ history_dir = base_dir / "history"
+ if history_dir.is_dir():
+ for path in sorted(history_dir.glob("*.json")):
+ entry = load_json(path)
+ if entry:
+ history.append(entry)
+ result["history"] = history
+ return result
+ return load_json(root / "index.json") or {"baselines": []}
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def suite_status(namespace: str, suite_id: str) -> list[dict[str, Any]]:
+ reload_volume()
+ rows = scan_runs(namespace=namespace, suite_id=suite_id)
+ if not rows:
+ raise FileNotFoundError(f"suite not found: {namespace}/{suite_id}")
+ rows.sort(key=lambda row: row.get("created_at") or row.get("run_id") or "")
+ return rows
+
+
+def write_run_inputs(run_dir: pathlib.Path, run_request: dict[str, Any]) -> None:
+ write_json(run_dir / "request.json", run_request)
+ write_json(run_dir / "run-metadata.json", harness_command.run_metadata(run_request))
+ task_names = run_request.get("task_names") or []
+ (run_dir / "selected-tasks.txt").write_text(
+ "\n".join(task_names) + ("\n" if task_names else "")
+ )
+
+
+ image=controller_image,
+ cpu=1,
+ memory=512,
+ timeout=300,
+ volumes={"/data": volume},
+)
+def create_run_record(run_request: dict[str, Any]) -> dict[str, Any]:
+ namespace = run_request["namespace"]
+ experiment_name = run_request["experiment_name"]
+ run_id = run_request["run_id"]
+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id
+ state_path = run_dir / "state.json"
+
+ reload_volume()
+
+ if state_path.exists():
+ raise FileExistsError(
+ f"run record already exists: {namespace}/{experiment_name}/{run_id}"
+ )
+
+ run_dir.mkdir(parents=True, exist_ok=True)
+ write_run_inputs(run_dir, run_request)
+ state = {
+ "run_id": run_id,
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "status": "pending",
+ "created_at": run_request.get("created_at"),
+ "updated_at": utc_now(),
+ "build_id": run_request.get("build_id"),
+ }
+ write_json(state_path, state)
+ volume.commit()
+ return state
+
+
+ image=controller_image,
+ cpu=2,
+ memory=4096,
+ timeout=86_400,
+ volumes={"/data": volume},
+ secrets=[modal.Secret.from_name(MODAL_TOKEN_SECRET_NAME)],
+)
+def run_controller(run_request: dict[str, Any]) -> dict[str, Any]:
+ namespace = run_request["namespace"]
+ experiment_name = run_request["experiment_name"]
+ run_id = run_request["run_id"]
+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id
+ run_dir.mkdir(parents=True, exist_ok=True)
+ log_path = run_dir / "controller.log"
+ state_path = run_dir / "state.json"
+ started_at = utc_now()
+
+ def commit() -> None:
+ volume.commit()
+
+ def log(message: str) -> None:
+ line = f"[{utc_now()}] {message}"
+ print(line, flush=True)
+ with log_path.open("a") as log_file:
+ log_file.write(line + "\n")
+
+ def state(status: str, **extra: Any) -> None:
+ payload = {
+ "run_id": run_id,
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "status": status,
+ "updated_at": utc_now(),
+ "started_at": started_at,
+ **extra,
+ }
+ write_json(state_path, payload)
+ commit()
+
+ try:
+ write_run_inputs(run_dir, run_request)
+ state("starting", build_id=run_request.get("build_id"))
+ log(f"Starting run {namespace}/{experiment_name}/{run_id}")
+
+ build_id = run_request.get("build_id")
+ if build_id:
+ build_dir = pathlib.Path("/data/builds") / build_id
+ ready_path = build_dir / "READY"
+ build_info_path = build_dir / "build-info.json"
+ source_patch_path = build_dir / "source.patch"
+ source_info_path = build_dir / "source-info.json"
+ state("waiting_for_build", build_id=build_id)
+ deadline = time.time() + int(
+ run_request.get("build_wait_timeout_secs") or 7200
+ )
+ while not ready_path.exists():
+ if time.time() >= deadline:
+ raise TimeoutError(
+ f"build {build_id} was not ready before the wait timeout"
+ )
+ log(f"Waiting for build {build_id} to become ready")
+ time.sleep(30)
+ reload_volume()
+ shutil.copy(build_info_path, run_dir / "build-info.json")
+ if source_patch_path.exists():
+ shutil.copy(source_patch_path, run_dir / "source.patch")
+ if source_info_path.exists():
+ shutil.copy(source_info_path, run_dir / "source-info.json")
+ log(f"Using build {build_id}")
+
+ provision_benchmark_dataset(run_request, log)
+
+ jobs_parent = pathlib.Path("/tmp/agent-evals/harbor-jobs")
+ jobs_parent.mkdir(parents=True, exist_ok=True)
+ command = harness_command.build_harness_command(run_request, str(jobs_parent))
+ redacted = config.redacted_command(command)
+ (run_dir / "harbor-command.txt").write_text(redacted + "\n")
+ state(
+ "running",
+ build_id=build_id,
+ harness_command=redacted,
+ )
+ commit()
+ log("Launching Harbor")
+ log(redacted)
+
+ import zed_eval
+
+ package_file = zed_eval.__file__
+ if package_file is None:
+ raise RuntimeError("could not resolve zed_eval package location")
+ package_parent = str(pathlib.Path(package_file).resolve().parent.parent)
+ environment = os.environ.copy()
+ existing_pythonpath = environment.get("PYTHONPATH")
+ environment["PYTHONPATH"] = (
+ package_parent
+ if not existing_pythonpath
+ else f"{package_parent}:{existing_pythonpath}"
+ )
+
+ process = subprocess.Popen(
+ command,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ text=True,
+ env=environment,
+ )
+ last_commit = time.time()
+ assert process.stdout is not None
+ with log_path.open("a") as log_file:
+ for line in process.stdout:
+ print(line, end="", flush=True)
+ log_file.write(line)
+ if time.time() - last_commit >= 10:
+ log_file.flush()
+ commit()
+ last_commit = time.time()
+ return_code = process.wait()
+ commit()
+ log(f"Harbor exited with status {return_code}")
+
+ job_dir = jobs_parent / run_id
+ archive_path = run_dir / "harbor-job.tar.gz"
+ if job_dir.exists():
+ with tarfile.open(archive_path, "w:gz") as archive:
+ archive.add(job_dir, arcname=job_dir.name)
+ # Count trial dirs by the presence of a trial result.json rather than
+ # a name prefix: Harbor names them `task-<id>__<suffix>` while Pier
+ # uses `<task-id>__<suffix>`.
+ trial_count = sum(
+ 1
+ for path in job_dir.iterdir()
+ if path.is_dir() and (path / "result.json").exists()
+ )
+ job_result = load_json(job_dir / "result.json")
+ else:
+ trial_count = 0
+ job_result = None
+ log(f"Harbor job dir was not found at {job_dir}")
+
+ summary = {
+ "run_id": run_id,
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "status": "completed" if return_code == 0 else "failed",
+ "harbor_return_code": return_code,
+ "trial_count": trial_count,
+ "has_archive": archive_path.exists(),
+ "job_result": job_result,
+ "started_at": started_at,
+ "completed_at": utc_now(),
+ }
+ write_json(run_dir / "summary.json", summary)
+
+ if return_code == 0:
+ (run_dir / "READY").write_text(utc_now() + "\n")
+ state("completed", summary=summary)
+ else:
+ (run_dir / "FAILED").write_text(utc_now() + "\n")
+ state("failed", summary=summary)
+ commit()
+ return summary
+ except Exception as error:
+ formatted = traceback.format_exc()
+ log(f"FAILED: {error}\n{formatted}")
+ (run_dir / "FAILED").write_text(utc_now() + "\n")
+ state("failed", error=str(error), traceback=formatted)
+ commit()
+ raise
+
+
+def rejudge_run_metadata(rejudge_request: dict[str, Any]) -> dict[str, Any]:
+ judge = config.get_judge(rejudge_request["judge_preset"])
+ parent = rejudge_request["parent"]
+ return {
+ "kind": "rejudge",
+ "run_id": rejudge_request["run_id"],
+ "namespace": rejudge_request["namespace"],
+ "experiment_name": rejudge_request["experiment_name"],
+ "source_run": parent,
+ "judge_preset": rejudge_request["judge_preset"],
+ "judge_model": rejudge_request.get("judge_model") or judge.model,
+ "judge_upstream": judge.upstream,
+ "judge_auth_env": judge.auth_env,
+ "orchestration": config.orchestration_info(),
+ "volume_name": rejudge_request["volume_name"],
+ "api_secret_name": rejudge_request.get("api_secret_name"),
+ "created_at": rejudge_request.get("created_at"),
+ }
+
+
+ image=controller_image,
+ cpu=2,
+ memory=4096,
+ timeout=86_400,
+ volumes={"/data": volume},
+ secrets=[
+ modal.Secret.from_name(MODAL_TOKEN_SECRET_NAME),
+ modal.Secret.from_name(LLM_PROVIDERS_SECRET_NAME),
+ ],
+)
+def rejudge_controller(rejudge_request: dict[str, Any]) -> dict[str, Any]:
+ """Re-grade a finished parent run with a different judge, producing a new
+ derived run. The parent is read only: its stored agent outputs are re-scored
+ by the real cached verifier through the judge proxy, and only the verdicts
+ change. See `rejudge.py` for the per-trial grading."""
+ namespace = rejudge_request["namespace"]
+ experiment_name = rejudge_request["experiment_name"]
+ run_id = rejudge_request["run_id"]
+ parent = rejudge_request["parent"]
+ judge_preset = rejudge_request["judge_preset"]
+ judge = config.get_judge(judge_preset)
+ judge_model = rejudge_request.get("judge_model") or judge.model
+
+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id
+ run_dir.mkdir(parents=True, exist_ok=True)
+ log_path = run_dir / "controller.log"
+ state_path = run_dir / "state.json"
+ started_at = utc_now()
+
+ def log(message: str) -> None:
+ line = f"[{utc_now()}] {message}"
+ print(line, flush=True)
+ with log_path.open("a") as log_file:
+ log_file.write(line + "\n")
+
+ def state(status: str, **extra: Any) -> None:
+ write_json(
+ state_path,
+ {
+ "run_id": run_id,
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "kind": "rejudge",
+ "status": status,
+ "updated_at": utc_now(),
+ "started_at": started_at,
+ **extra,
+ },
+ )
+ volume.commit()
+
+ try:
+ write_json(run_dir / "request.json", rejudge_request)
+ write_json(run_dir / "run-metadata.json", rejudge_run_metadata(rejudge_request))
+ state("starting", source_run=parent)
+ log(
+ f"Rejudging {parent['namespace']}/{parent['experiment_name']}/"
+ f"{parent['run_id']} as {namespace}/{experiment_name}/{run_id} "
+ f"with judge {judge_preset} ({judge_model})"
+ )
+
+ reload_volume()
+ parent_dir = (
+ pathlib.Path("/data/runs")
+ / parent["namespace"]
+ / parent["experiment_name"]
+ / parent["run_id"]
+ )
+ parent_archive = parent_dir / "harbor-job.tar.gz"
+ if not parent_archive.exists():
+ raise FileNotFoundError(
+ f"parent run has no harbor-job.tar.gz: {parent_archive}. "
+ "Rejudge needs a completed parent run with a stored job archive."
+ )
+
+ metadata = load_json(parent_dir / "run-metadata.json") or {}
+ benchmark = metadata.get("benchmark") or {}
+ dataset = benchmark.get("dataset") if isinstance(benchmark, dict) else {}
+ dataset_kind = dataset.get("kind") if isinstance(dataset, dict) else None
+ dataset_name = dataset.get("name") if isinstance(dataset, dict) else None
+ if dataset_kind != "registry" or not isinstance(dataset_name, str):
+ raise NotImplementedError(
+ "rejudge currently supports registry datasets (SWE-Atlas rf/qna). "
+ f"Parent run dataset_kind={dataset_kind!r} is not yet supported."
+ )
+
+ work = pathlib.Path("/tmp/agent-evals/rejudge") / run_id
+ if work.exists():
+ shutil.rmtree(work)
+ extract_root = work / "parent"
+ extract_root.mkdir(parents=True, exist_ok=True)
+ log(f"Extracting parent archive {parent_archive}")
+ with tarfile.open(parent_archive, "r:gz") as archive:
+ safe_extract_archive(archive, extract_root)
+ parent_job_dir = extract_root / parent["run_id"]
+ if not parent_job_dir.is_dir():
+ # Fall back to the single top-level dir if the arcname differs.
+ candidates = [p for p in extract_root.iterdir() if p.is_dir()]
+ if len(candidates) != 1:
+ raise FileNotFoundError(
+ f"could not locate job dir inside {parent_archive}"
+ )
+ parent_job_dir = candidates[0]
+
+ log(f"Downloading task packages for dataset {dataset_name}")
+ subprocess.run(["harbor", "datasets", "download", dataset_name], check=True)
+ tasks_root = pathlib.Path.home() / ".cache" / "harbor" / "tasks" / "packages"
+
+ state("running", source_run=parent, judge_model=judge_model)
+ out_job_dir = work / "job" / run_id
+ summary = rejudge.rejudge_job(
+ parent_job_dir=parent_job_dir,
+ out_job_dir=out_job_dir,
+ tasks_root=tasks_root,
+ judge=judge,
+ judge_model=judge_model,
+ log=log,
+ )
+ volume.commit()
+
+ archive_path = run_dir / "harbor-job.tar.gz"
+ with tarfile.open(archive_path, "w:gz") as archive:
+ archive.add(out_job_dir, arcname=out_job_dir.name)
+
+ summary = {
+ **summary,
+ "run_id": run_id,
+ "namespace": namespace,
+ "experiment_name": experiment_name,
+ "kind": "rejudge",
+ "source_run": parent,
+ "judge_preset": judge_preset,
+ "status": "completed",
+ "has_archive": archive_path.exists(),
+ "started_at": started_at,
+ "completed_at": utc_now(),
+ }
+ write_json(run_dir / "summary.json", summary)
+ (run_dir / "READY").write_text(utc_now() + "\n")
+ log(
+ f"Rejudge complete: {summary['passed_count']}/"
+ f"{summary['rejudged_count']} passed "
+ f"({summary['failed_count']} trials could not be rejudged)"
+ )
+ state("completed", summary=summary)
+ volume.commit()
+ return summary
+ except Exception as error:
+ formatted = traceback.format_exc()
+ log(f"FAILED: {error}\n{formatted}")
+ (run_dir / "FAILED").write_text(utc_now() + "\n")
+ state("failed", error=str(error), traceback=formatted)
+ volume.commit()
+ raise
diff --git a/crates/eval_cli/zed_eval/pier_agent.py b/crates/eval_cli/zed_eval/pier_agent.py
new file mode 100644
index 00000000000..0a5c07631fa
--- /dev/null
+++ b/crates/eval_cli/zed_eval/pier_agent.py
@@ -0,0 +1,226 @@
+"""Pier-native agent wrapper for Zed's eval-cli binary.
+
+Pier (a Harbor fork) is the harness for DeepSWE, whose tasks run air-gapped
+(`allow_internet = false`). Pier's installed-agent interface differs from
+Harbor's — it requires `install_spec()` and, crucially, a `network_allowlist()`
+the egress proxy honors so the agent can still reach the model API. Harbor's
+`BaseInstalledAgent` has neither, so this is a separate, Pier-specific class
+(the sanctioned cross-framework exception): everything else in the project has a
+single canonical implementation, but two genuinely different harness SDKs need
+two thin agent shells.
+
+Air-gap notes:
+ - The eval-cli binary is a static musl executable copied from the mounted Modal
+ volume (`/data`, local — not network), so installing it needs no egress.
+ - Network installs (Node, LSPs) are intentionally skipped: there's no egress to
+ fetch them, and eval-cli's core tools (read/edit/terminal/...) don't need
+ them. Only the model API host is allowlisted.
+"""
+
+from __future__ import annotations
+
+import json
+import shlex
+from urllib.parse import urlparse
+
+from pier.agents.installed.base import BaseInstalledAgent, with_prompt_template
+from pier.environments.base import BaseEnvironment
+from pier.models.agent.context import AgentContext
+from pier.models.agent.install import AgentInstallSpec, InstallStep
+from pier.models.agent.network import NetworkAllowlist
+from pier.models.trial.paths import EnvironmentPaths
+
+from .agent_common import (
+ add_anthropic_available_models_env,
+ add_openai_compatible_provider_env,
+ add_zed_eval_env,
+ detect_workdir,
+ eval_cli_with_log_command,
+ patch_command,
+ populate_context_from_result,
+ provider_api_env,
+)
+
+# Default model-provider API hosts the agent must reach even on air-gapped tasks.
+# Mirrors benchmarks.AGENT_API_HOSTS; duplicated here so this module has no
+# dependency on the orchestration package when loaded inside a sandbox.
+DEFAULT_PROVIDER_DOMAINS: dict[str, list[str]] = {
+ "anthropic": ["api.anthropic.com"],
+ "openai": ["api.openai.com"],
+ "google": [".googleapis.com"],
+ "gemini": [".googleapis.com"],
+ "deepseek": ["api.deepseek.com"],
+ "baseten": ["inference.baseten.co"],
+}
+# Base-URL env vars Pier-style allowlist resolution should also honor.
+BASE_URL_ENV_VARS = (
+ "ANTHROPIC_BASE_URL",
+ "OPENAI_BASE_URL",
+ "OPENAI_API_BASE",
+ "GEMINI_API_BASE",
+)
+
+
+class ZedPierAgent(BaseInstalledAgent):
+ """Runs Zed's headless eval-cli binary under Pier."""
+
+ SUPPORTS_ATIF: bool = False
+
+ @staticmethod
+ def name() -> str:
+ return "zed"
+
+ def _container_binary_path(self) -> str:
+ path = self._get_env("EVAL_CLI_CONTAINER_PATH")
+ if not path:
+ raise ValueError(
+ "ZedPierAgent requires EVAL_CLI_CONTAINER_PATH (the eval-cli binary "
+ "on the mounted volume, e.g. /data/builds/<id>/eval-cli)"
+ )
+ return path
+
+ def install_spec(self) -> AgentInstallSpec:
+ # The binary lives on the runtime-mounted volume, which isn't available at
+ # image-build time, so the real placement happens in run() via
+ # exec_as_root. install_spec only needs a non-empty, no-egress step.
+ return AgentInstallSpec(
+ agent_name=self.name(),
+ version=self._version,
+ steps=[
+ InstallStep(
+ user="root",
+ run="true # eval-cli is copied from the volume at run time",
+ )
+ ],
+ )
+
+ def network_allowlist(self) -> NetworkAllowlist:
+ """Hosts the egress proxy must allow so eval-cli can reach the model API
+ on air-gapped DeepSWE tasks."""
+ domains: list[str] = []
+
+ provider = None
+ if self.model_name and "/" in self.model_name:
+ provider = self.model_name.split("/", 1)[0]
+ domains.extend(DEFAULT_PROVIDER_DOMAINS.get(provider or "", []))
+
+ # Any explicitly configured base URL (e.g. an OpenAI-compatible gateway
+ # such as Baseten) wins/adds alongside the provider default.
+ for env_var in BASE_URL_ENV_VARS:
+ value = self._get_env(env_var)
+ if value:
+ host = urlparse(value).hostname
+ if host:
+ domains.append(host)
+
+ # A Baseten (or other OpenAI-compatible) provider is wired via this JSON;
+ # allow its api_url host too.
+ providers_json = self._get_env("ZED_OPENAI_COMPATIBLE_PROVIDERS")
+ if providers_json:
+ try:
+ parsed = json.loads(providers_json)
+ except json.JSONDecodeError:
+ parsed = {}
+ for provider_config in (parsed or {}).values():
+ api_url = (
+ provider_config.get("api_url")
+ if isinstance(provider_config, dict)
+ else None
+ )
+ host = urlparse(api_url).hostname if api_url else None
+ if host:
+ domains.append(host)
+
+ if not domains:
+ # Never return an empty allowlist: without the model host the agent
+ # can do nothing on an air-gapped task. Default to Anthropic.
+ domains = ["api.anthropic.com"]
+ return NetworkAllowlist(domains=domains)
+
+ def _api_env(self) -> dict[str, str]:
+ env = provider_api_env(self.model_name, self._get_env)
+ add_openai_compatible_provider_env(
+ env, self._get_env("ZED_OPENAI_COMPATIBLE_PROVIDERS")
+ )
+ add_anthropic_available_models_env(
+ env, self._get_env("ZED_ANTHROPIC_AVAILABLE_MODELS")
+ )
+ add_zed_eval_env(
+ env, self._extra_env, exclude={"ZED_EVAL_INSTRUCTION_SUFFIX_FILE"}
+ )
+ return env
+
+ async def _detect_workdir(self, environment: BaseEnvironment) -> str:
+ return await detect_workdir(
+ environment,
+ self.exec_as_agent,
+ self._get_env,
+ "Could not detect a working directory; set EVAL_CLI_WORKDIR via --ae",
+ )
+
+ @with_prompt_template
+ async def run(
+ self, instruction: str, environment: BaseEnvironment, context: AgentContext
+ ) -> None:
+ # Place the static binary from the mounted volume (no egress needed).
+ await self.exec_as_root(
+ environment,
+ command=(
+ f"install -m755 {shlex.quote(self._container_binary_path())} "
+ "/usr/local/bin/eval-cli && eval-cli --help >/dev/null"
+ ),
+ )
+
+ workdir = await self._detect_workdir(environment)
+ agent_dir = str(EnvironmentPaths.agent_dir)
+ env = self.build_process_env(self._api_env())
+
+ parts = [
+ "eval-cli",
+ f"--workdir {shlex.quote(workdir)}",
+ f"--output-dir {shlex.quote(agent_dir)}",
+ ]
+ if self.model_name:
+ parts.append(f"--model {shlex.quote(self.model_name)}")
+ timeout = self._get_env("EVAL_CLI_TIMEOUT")
+ if timeout:
+ parts.append(f"--timeout {shlex.quote(timeout)}")
+ instruction_suffix_file = self._get_env("ZED_EVAL_INSTRUCTION_SUFFIX_FILE")
+ if instruction_suffix_file:
+ parts.append(
+ f"--instruction-suffix-file {shlex.quote(instruction_suffix_file)}"
+ )
+ staff = self._get_env("EVAL_CLI_STAFF")
+ if staff and staff.lower() == "false":
+ parts.append("--no-staff")
+ reasoning_effort = self._get_env("EVAL_CLI_REASONING_EFFORT")
+ if reasoning_effort:
+ parts.append(f"--reasoning-effort {shlex.quote(reasoning_effort)}")
+ enable_thinking = self._get_env("EVAL_CLI_ENABLE_THINKING")
+ if enable_thinking:
+ if enable_thinking.lower() == "true":
+ parts.append("--thinking true")
+ elif enable_thinking.lower() == "false":
+ parts.append("--thinking false")
+ parts.append(f"--instruction {shlex.quote(instruction)}")
+
+ # Tolerate exit 2 (timeout): deliver whatever the agent produced.
+ await self.exec_as_agent(
+ environment,
+ command=eval_cli_with_log_command(
+ parts,
+ agent_dir + "/eval-cli.txt",
+ timeout_message=None,
+ ),
+ env=env,
+ )
+
+ # SWE-bench-style patch for the verifier (DeepSWE tasks are git repos).
+ await self.exec_as_agent(
+ environment,
+ command=patch_command(agent_dir),
+ cwd=workdir,
+ )
+
+ def populate_context_post_run(self, context: AgentContext) -> None:
+ populate_context_from_result(self.logs_dir, context, self.logger)
diff --git a/crates/eval_cli/zed_eval/pyproject.toml b/crates/eval_cli/zed_eval/pyproject.toml
index e25ec9825c0..c521651c924 100644
--- a/crates/eval_cli/zed_eval/pyproject.toml
+++ b/crates/eval_cli/zed_eval/pyproject.toml
@@ -1,9 +1,16 @@
[project]
name = "zed-eval"
-version = "0.1.0"
-description = "Harbor agent wrapper for Zed's eval-cli"
+version = "0.2.0"
+description = "Harbor agent wrapper and remote orchestration CLI for Zed's eval-cli"
requires-python = ">=3.12"
-dependencies = ["harbor==0.13.0"]
+dependencies = ["harbor==0.15.0", "modal==1.5.0"]
+
+[project.scripts]
+zed-eval = "zed_eval.cli:main"
+
+[tool.setuptools]
+packages = ["zed_eval"]
+package-dir = { zed_eval = "." }
[build-system]
requires = ["setuptools"]
diff --git a/crates/eval_cli/zed_eval/rejudge.py b/crates/eval_cli/zed_eval/rejudge.py
new file mode 100644
index 00000000000..42c0b477125
--- /dev/null
+++ b/crates/eval_cli/zed_eval/rejudge.py
@@ -0,0 +1,359 @@
+"""Re-grade a finished run with a different judge, without redoing agent work.
+
+A rejudge takes the agent output already stored in a parent run (the expensive
+part: patches/answers the agent produced) and re-runs only the *judge* step with
+a new judge model, producing a brand-new derived run. The original run is read
+only and never modified.
+
+This is deliberately not a re-implementation of the grader. It runs the exact
+SWE-Atlas verifier script shipped in each task package (`evaluate_rubrics.py` for
+RF, `evaluate_answer.py` for QnA) through the same judge proxy shim the live
+in-sandbox path uses (`judge_proxy.JUDGE_PROXY_SCRIPT`), so a rejudged verdict is
+produced identically to a first-pass verdict — only the judge model differs.
+
+The verifier scripts read their inputs from fixed absolute paths (`/tests`,
+`/logs/...`) and pick the judge up from `EVAL_MODEL`/`EVAL_BASE_URL`/
+`EVAL_API_KEY`, so this module is meant to run inside a throwaway container (the
+Modal `rejudge_controller`), where staging those absolute paths is safe. The pure
+helpers (resolution, reward recombination, result patching) have no such
+requirement and are unit-tested directly.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from . import config
+from .judge_proxy import JUDGE_PROXY_SCRIPT
+
+# Each task package ships exactly one of these verifier scripts; its presence is
+# how we tell an RF task from a QnA one without trusting the experiment config.
+RF_VERIFIER = "evaluate_rubrics.py"
+QNA_VERIFIER = "evaluate_answer.py"
+
+PART_RF = "rf"
+PART_QNA = "qna"
+
+# Absolute paths the cached verifier scripts hardcode. Staged per trial.
+TESTS_DIR = Path("/tests")
+LOGS_AGENT_DIR = Path("/logs/agent")
+LOGS_VERIFIER_DIR = Path("/logs/verifier")
+RF_PATCH_PATH = LOGS_VERIFIER_DIR / "agent.patch"
+RF_RESULTS_PATH = LOGS_VERIFIER_DIR / "rubrics_results.json"
+QNA_ANSWER_PATH = LOGS_AGENT_DIR / "answer.txt"
+QNA_RESULTS_PATH = LOGS_VERIFIER_DIR / "evaluation_results.json"
+
+
+def load_json(path: Path) -> Any:
+ try:
+ return json.loads(Path(path).read_text())
+ except (OSError, json.JSONDecodeError):
+ return None
+
+
+def task_tests_dir(task_name: str, task_ref: str, tasks_root: Path) -> Path:
+ """Resolve a trial's cached task package `tests/` dir from its name + ref.
+
+ Mirrors Harbor's content-addressed layout:
+ `<tasks_root>/<org>/<task-id>/<ref-hash>/tests`.
+ """
+ org, task_id = task_name.split("/", 1)
+ ref_hash = task_ref.split(":", 1)[1] if ":" in task_ref else task_ref
+ return tasks_root / org / task_id / ref_hash / "tests"
+
+
+def tests_dir_for_trial(trial_dir: Path, tasks_root: Path) -> Path | None:
+ config_data = load_json(trial_dir / "config.json")
+ if not isinstance(config_data, dict):
+ return None
+ task = config_data.get("task") or {}
+ name, ref = task.get("name"), task.get("ref")
+ if not (name and ref):
+ return None
+ tests_dir = task_tests_dir(name, ref, tasks_root)
+ return tests_dir if tests_dir.is_dir() else None
+
+
+def detect_part(tests_dir: Path) -> str | None:
+ if (tests_dir / RF_VERIFIER).exists():
+ return PART_RF
+ if (tests_dir / QNA_VERIFIER).exists():
+ return PART_QNA
+ return None
+
+
+def recompute_rf_rewards(
+ rubrics_results: dict[str, Any], prior_rewards: dict[str, Any]
+) -> dict[str, float]:
+ """Combine the freshly re-judged rubric verdict with the *unchanged* saved
+ deterministic test reward. The agent patch didn't change, so `tests_reward`
+ is invariant under rejudge; only the rubric (`must_have_pass`/agg) moves.
+
+ The overall RF reward is `must_have_pass AND tests_reward == 1`, verified to
+ hold across every cached RF trial.
+ """
+ must_have_pass = bool(rubrics_results.get("must_have_pass"))
+ agg_score = rubrics_results.get("agg_score")
+ tests_reward = float(prior_rewards.get("tests_reward") or 0.0)
+ overall_pass = 1.0 if (must_have_pass and tests_reward >= 1.0) else 0.0
+ return {
+ "reward": overall_pass,
+ "overall_pass": overall_pass,
+ "tests_reward": tests_reward,
+ "must_have_pass": 1.0 if must_have_pass else 0.0,
+ "rubrics_agg_score": float(agg_score) if agg_score is not None else 0.0,
+ }
+
+
+def recompute_qna_rewards(evaluation_results: dict[str, Any]) -> dict[str, float]:
+ """QnA has no deterministic test component; the reward is just the judged
+ pass verdict, matching the single-key `{"reward": ...}` production shape."""
+ return {"reward": 1.0 if bool(evaluation_results.get("pass")) else 0.0}
+
+
+def proxy_environment(judge: config.JudgeConfig) -> dict[str, str]:
+ """Env the judge proxy subprocess reads. The auth key it forwards
+ (`judge.auth_env`, e.g. BASETEN_API_KEY) must already be present in the
+ process environment via the mounted LLM-providers secret."""
+ env = {
+ "ZED_JUDGE_UPSTREAM": judge.upstream,
+ "ZED_JUDGE_AUTH_ENV": judge.auth_env,
+ }
+ if judge.max_tokens is not None:
+ env["ZED_JUDGE_MAX_TOKENS"] = str(judge.max_tokens)
+ return env
+
+
+def verifier_environment(judge_model: str, port: int) -> dict[str, str]:
+ """Env the cached verifier reads to find the judge.
+
+ This mirrors the verifier env injected by the benchmark harness command,
+ pointed at the local rejudge proxy.
+ """
+ return {
+ "EVAL_MODEL": judge_model,
+ "EVAL_BASE_URL": f"http://127.0.0.1:{port}/v1",
+ "EVAL_API_KEY": "unused-by-agent-evals-proxy",
+ }
+
+
+class JudgeProxy:
+ """Runs `judge_proxy.JUDGE_PROXY_SCRIPT` as a local subprocess for the life
+ of a rejudge job. One proxy serves every trial (they all target the same
+ judge), mirroring the single in-sandbox daemon."""
+
+ def __init__(self, judge: config.JudgeConfig, port: int = 8089) -> None:
+ self._judge = judge
+ self._port = port
+ self._process: subprocess.Popen[bytes] | None = None
+ self._script_path = Path("/tmp/zed_judge_proxy.py")
+
+ @property
+ def port(self) -> int:
+ return self._port
+
+ def __enter__(self) -> JudgeProxy:
+ self._script_path.write_text(JUDGE_PROXY_SCRIPT)
+ env = os.environ.copy()
+ env.update(proxy_environment(self._judge))
+ env["ZED_JUDGE_PROXY_PORT"] = str(self._port)
+ self._process = subprocess.Popen(
+ [sys.executable, str(self._script_path)],
+ env=env,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ )
+ self._wait_for_port()
+ return self
+
+ def __exit__(self, *_exc: object) -> None:
+ if self._process is not None:
+ self._process.terminate()
+ try:
+ self._process.wait(timeout=10)
+ except subprocess.TimeoutExpired:
+ self._process.kill()
+
+ def _wait_for_port(self, timeout_secs: float = 30.0) -> None:
+ deadline = time.time() + timeout_secs
+ while time.time() < deadline:
+ if self._process is not None and self._process.poll() is not None:
+ raise RuntimeError("judge proxy exited before becoming ready")
+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:
+ probe.settimeout(0.5)
+ if probe.connect_ex(("127.0.0.1", self._port)) == 0:
+ return
+ time.sleep(0.25)
+ raise TimeoutError(f"judge proxy did not open port {self._port} in time")
+
+
+def _stage_tests_dir(tests_dir: Path) -> None:
+ """Point the absolute `/tests` path at this trial's package. Reset per trial
+ so each verifier reads its own rubrics/prompts."""
+ if TESTS_DIR.is_symlink() or TESTS_DIR.exists():
+ if TESTS_DIR.is_dir() and not TESTS_DIR.is_symlink():
+ shutil.rmtree(TESTS_DIR)
+ else:
+ TESTS_DIR.unlink()
+ TESTS_DIR.symlink_to(tests_dir)
+
+
+def _run_verifier(
+ part: str, tests_dir: Path, judge_model: str, port: int
+) -> dict[str, Any]:
+ """Stage the verifier's expected inputs and run the real cached script."""
+ LOGS_AGENT_DIR.mkdir(parents=True, exist_ok=True)
+ LOGS_VERIFIER_DIR.mkdir(parents=True, exist_ok=True)
+ _stage_tests_dir(tests_dir)
+
+ env = os.environ.copy()
+ env.update(verifier_environment(judge_model, port))
+ script = tests_dir / (RF_VERIFIER if part == PART_RF else QNA_VERIFIER)
+ results_path = RF_RESULTS_PATH if part == PART_RF else QNA_RESULTS_PATH
+ if results_path.exists():
+ results_path.unlink()
+
+ completed = subprocess.run(
+ [sys.executable, str(script)],
+ env=env,
+ capture_output=True,
+ text=True,
+ )
+ results = load_json(results_path)
+ if not isinstance(results, dict):
+ raise RuntimeError(
+ f"verifier produced no results (exit {completed.returncode}): "
+ f"{completed.stderr.strip()[:500]}"
+ )
+ return results
+
+
+def rejudge_trial(
+ trial_dir: Path,
+ tasks_root: Path,
+ judge: config.JudgeConfig,
+ judge_model: str,
+ port: int,
+) -> dict[str, Any]:
+ """Re-grade one trial in place (the trial dir is a copy in the derived run).
+
+ Returns a per-trial record; on any recoverable problem it returns
+ `{"ok": False, "error": ...}` so one bad trial can't sink the whole job.
+ """
+ task_name = trial_dir.name
+ tests_dir = tests_dir_for_trial(trial_dir, tasks_root)
+ if tests_dir is None:
+ return {"task": task_name, "ok": False, "error": "task package not found"}
+ part = detect_part(tests_dir)
+ if part is None:
+ return {"task": task_name, "ok": False, "error": "no verifier in task package"}
+
+ # The agent output is already on disk from the parent run; copy it to the
+ # path the verifier expects. RF reads the patch, QnA the delivered answer.
+ if part == PART_RF:
+ source = trial_dir / "verifier" / "agent.patch"
+ if not source.exists():
+ source = trial_dir / "verifier" / "agent_source_only.patch"
+ if not source.exists():
+ return {"task": task_name, "ok": False, "error": "no agent.patch"}
+ RF_PATCH_PATH.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(source, RF_PATCH_PATH)
+ else:
+ source = trial_dir / "agent" / "answer.txt"
+ if not source.exists():
+ return {"task": task_name, "ok": False, "error": "no answer.txt"}
+ QNA_ANSWER_PATH.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(source, QNA_ANSWER_PATH)
+
+ try:
+ results = _run_verifier(part, tests_dir, judge_model, port)
+ except (RuntimeError, OSError) as error:
+ return {"task": task_name, "ok": False, "error": str(error)}
+
+ prior_rewards = (
+ (load_json(trial_dir / "result.json") or {}).get("verifier_result") or {}
+ ).get("rewards") or {}
+ if part == PART_RF:
+ new_rewards = recompute_rf_rewards(results, prior_rewards)
+ verifier_results_name = "rubrics_results.json"
+ else:
+ new_rewards = recompute_qna_rewards(results)
+ verifier_results_name = "evaluation_results.json"
+
+ # Persist the fresh verifier output and verdict into the derived trial dir so
+ # report.py reads the new judge's result with no special-casing.
+ (trial_dir / "verifier").mkdir(parents=True, exist_ok=True)
+ (trial_dir / "verifier" / verifier_results_name).write_text(
+ json.dumps(results, indent=2)
+ )
+ patch_result_rewards(trial_dir / "result.json", new_rewards)
+ return {"task": task_name, "ok": True, "part": part, "rewards": new_rewards}
+
+
+def patch_result_rewards(result_path: Path, new_rewards: dict[str, float]) -> None:
+ """Overwrite only `verifier_result.rewards` in a trial's `result.json`,
+ leaving the agent's resource metrics (tokens/steps/tool calls) untouched —
+ the agent didn't re-run, only the judge did."""
+ result = load_json(result_path)
+ if not isinstance(result, dict):
+ result = {}
+ verifier_result = result.get("verifier_result")
+ if not isinstance(verifier_result, dict):
+ verifier_result = {}
+ verifier_result["rewards"] = new_rewards
+ result["verifier_result"] = verifier_result
+ Path(result_path).write_text(json.dumps(result, indent=2))
+
+
+def rejudge_job(
+ parent_job_dir: Path,
+ out_job_dir: Path,
+ tasks_root: Path,
+ judge: config.JudgeConfig,
+ judge_model: str,
+ port: int = 8089,
+ log: Any = print,
+) -> dict[str, Any]:
+ """Copy a parent run's Harbor job dir into the derived run and re-judge every
+ trial in place. Returns a summary suitable for the run's `summary.json`."""
+ if out_job_dir.exists():
+ shutil.rmtree(out_job_dir)
+ shutil.copytree(parent_job_dir, out_job_dir)
+
+ trial_dirs = sorted(
+ path
+ for path in out_job_dir.iterdir()
+ if path.is_dir() and (path / "result.json").exists()
+ )
+ records: list[dict[str, Any]] = []
+ with JudgeProxy(judge, port=port) as proxy:
+ for index, trial_dir in enumerate(trial_dirs, start=1):
+ record = rejudge_trial(
+ trial_dir, tasks_root, judge, judge_model, proxy.port
+ )
+ records.append(record)
+ status = "ok" if record["ok"] else f"FAILED: {record.get('error')}"
+ log(f"[{index}/{len(trial_dirs)}] {trial_dir.name}: {status}")
+
+ rejudged = sum(1 for record in records if record["ok"])
+ passed = sum(
+ 1
+ for record in records
+ if record["ok"] and record["rewards"].get("reward", 0.0) >= 1.0
+ )
+ return {
+ "trial_count": len(trial_dirs),
+ "rejudged_count": rejudged,
+ "failed_count": len(trial_dirs) - rejudged,
+ "passed_count": passed,
+ "judge_model": judge_model,
+ "trials": records,
+ }
diff --git a/crates/eval_cli/zed_eval/report.py b/crates/eval_cli/zed_eval/report.py
new file mode 100644
index 00000000000..bd809e3927f
--- /dev/null
+++ b/crates/eval_cli/zed_eval/report.py
@@ -0,0 +1,484 @@
+"""Success-conditioned metrics for fetched benchmark runs.
+
+This reads a fetched Harbor/Pier job directory and reports, for each run:
+
+ - success rate (with SEM when there are repeated attempts)
+ - token consumption *for the passing subset* (and overall, for comparison)
+ - tool calls *for the passing subset*, aggregate and broken down by tool
+ - total steps (agent turns) *for the passing subset*
+
+The per-trial record has two sources, one canonical each, with no fallbacks:
+
+ - The verdict (pass/fail/errored) comes from the harness' own trial result
+ (verifier rewards) — rubric judge for SWE-Atlas, test scripts for
+ Terminal-Bench and DeepSWE.
+ - Resource usage (tokens / steps / tool calls) comes from the Zed agent's
+ `result.json`, which eval-cli emits identically on every harness. There is
+ deliberately no secondary parser: if a metric isn't in `result.json`, it's
+ reported as absent rather than reconstructed from another file.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import statistics
+import sys
+from collections import Counter, defaultdict
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+# Effective-token weights (cache reads are ~10x cheaper, output ~4x input).
+ET_INPUT = 1.0
+ET_CACHE_READ = 0.1
+ET_CACHE_CREATION = 1.0
+ET_OUTPUT = 4.0
+
+HARBOR_TRIAL_KEYS = {"task_name", "trial_name", "verifier_result"}
+ZED_RESULT_KEYS = {"status", "duration_secs"}
+
+
+def load_json(path: Path) -> Any:
+ try:
+ return json.loads(path.read_text())
+ except (OSError, json.JSONDecodeError) as error:
+ print(f"warning: could not parse {path}: {error}", file=sys.stderr)
+ return None
+
+
+def find_trial_dirs(job_dir: Path) -> list[Path]:
+ trial_dirs: set[Path] = set()
+ for name in ("result.json", "results.json"):
+ for path in job_dir.rglob(name):
+ if path.parent == job_dir:
+ continue
+ data = load_json(path)
+ if isinstance(data, dict) and HARBOR_TRIAL_KEYS & data.keys():
+ trial_dirs.add(path.parent)
+ return sorted(trial_dirs)
+
+
+def load_harbor_trial_result(trial_dir: Path) -> dict | None:
+ for name in ("result.json", "results.json"):
+ path = trial_dir / name
+ if path.exists():
+ data = load_json(path)
+ if isinstance(data, dict) and HARBOR_TRIAL_KEYS & data.keys():
+ return data
+ return None
+
+
+def load_zed_result(trial_dir: Path) -> dict | None:
+ for path in sorted(trial_dir.rglob("result.json")):
+ if path.parent == trial_dir:
+ continue
+ data = load_json(path)
+ if isinstance(data, dict) and ZED_RESULT_KEYS & data.keys():
+ return data
+ return None
+
+
+def trial_verdict(
+ harbor_result: dict, *, timeout_is_failure: bool = False
+) -> tuple[bool | None, str | None]:
+ """Returns (passed, error_reason). `passed` is None when the trial errored
+ (harness/infra exception or no verifier reward); infra errors are not
+ failures.
+
+ `timeout_is_failure` reclassifies an agent timeout as a real failure
+ (passed=False) rather than an excluded error: on test-scored benchmarks an
+ `AgentTimeoutError` means the agent didn't solve the task within its intended
+ time budget, which is an agent failure, not infra noise."""
+ exception_info = harbor_result.get("exception_info")
+ if exception_info:
+ reason = "exception"
+ if isinstance(exception_info, dict):
+ reason = exception_info.get("exception_type") or "exception"
+ if timeout_is_failure and reason == "AgentTimeoutError":
+ return False, reason
+ return None, reason
+
+ verifier = harbor_result.get("verifier_result")
+ if not isinstance(verifier, dict):
+ return None, "no verifier result"
+ rewards = verifier.get("rewards")
+ if not isinstance(rewards, dict) or not rewards:
+ return None, "no rewards"
+ try:
+ # Prefer the strict leaderboard metric. max(rewards) would wrongly count
+ # a trial as passed when only a component reward (e.g. tests_reward) hit
+ # 1.0 on SWE-Atlas RF.
+ if "reward" in rewards:
+ return float(rewards["reward"]) >= 1.0, None
+ return max(float(value) for value in rewards.values()) >= 1.0, None
+ except (TypeError, ValueError):
+ return None, "unparsable rewards"
+
+
+@dataclass
+class TrialRecord:
+ task_name: str
+ started_at: str | None
+ passed: bool | None
+ error_reason: str | None
+ input_tokens: int = 0
+ output_tokens: int = 0
+ cache_read_input_tokens: int = 0
+ cache_creation_input_tokens: int = 0
+ step_count: int | None = None
+ total_tool_calls: int | None = None
+ tool_calls: dict[str, int] = field(default_factory=dict)
+ duration_secs: float | None = None
+ status: str | None = None
+ model: str | None = None
+ # True when the agent's result.json was found (i.e. the trial produced
+ # metrics). False for trials hard-killed on timeout before writing it.
+ has_result: bool = False
+
+ @property
+ def effective_tokens(self) -> float:
+ return (
+ ET_INPUT * self.input_tokens
+ + ET_CACHE_READ * self.cache_read_input_tokens
+ + ET_CACHE_CREATION * self.cache_creation_input_tokens
+ + ET_OUTPUT * self.output_tokens
+ )
+
+ @property
+ def total_tokens(self) -> int:
+ return (
+ self.input_tokens
+ + self.output_tokens
+ + self.cache_read_input_tokens
+ + self.cache_creation_input_tokens
+ )
+
+
+def extract_trial(
+ trial_dir: Path, *, timeout_is_failure: bool = False
+) -> TrialRecord | None:
+ harbor_result = load_harbor_trial_result(trial_dir)
+ if harbor_result is None:
+ return None
+
+ passed, error_reason = trial_verdict(
+ harbor_result, timeout_is_failure=timeout_is_failure
+ )
+ record = TrialRecord(
+ task_name=harbor_result.get("task_name") or trial_dir.name,
+ started_at=harbor_result.get("started_at"),
+ passed=passed,
+ error_reason=error_reason,
+ )
+
+ zed_result = load_zed_result(trial_dir)
+ if zed_result:
+ record.has_result = True
+ record.status = zed_result.get("status")
+ record.model = zed_result.get("model")
+ record.input_tokens = int(zed_result.get("input_tokens") or 0)
+ record.output_tokens = int(zed_result.get("output_tokens") or 0)
+ record.cache_read_input_tokens = int(
+ zed_result.get("cache_read_input_tokens") or 0
+ )
+ record.cache_creation_input_tokens = int(
+ zed_result.get("cache_creation_input_tokens") or 0
+ )
+ duration = zed_result.get("duration_secs")
+ if isinstance(duration, (int, float)):
+ record.duration_secs = float(duration)
+ # The canonical metrics source: counts eval-cli writes into result.json.
+ if isinstance(zed_result.get("step_count"), int):
+ record.step_count = zed_result["step_count"]
+ if isinstance(zed_result.get("tool_call_count"), int):
+ record.total_tool_calls = zed_result["tool_call_count"]
+ if isinstance(zed_result.get("tool_calls"), dict):
+ record.tool_calls = {
+ str(name): int(count)
+ for name, count in zed_result["tool_calls"].items()
+ if isinstance(count, int)
+ }
+
+ return record
+
+
+def sample_sem(values: list[float]) -> float | None:
+ n = len(values)
+ if n < 2:
+ return None
+ mean = sum(values) / n
+ return math.sqrt(sum((x - mean) ** 2 for x in values) / (n * (n - 1)))
+
+
+def pass_rate_with_sem(
+ scored: list[TrialRecord],
+) -> tuple[float | None, float | None, int]:
+ """Pass rate grouped by task. With >=2 attempts per task the rate is the
+ mean of per-attempt pass rates with sample SEM across attempts; otherwise
+ it's the plain rate (SEM undefined)."""
+ if not scored:
+ return None, None, 0
+ by_task: dict[str, list[TrialRecord]] = defaultdict(list)
+ for trial in scored:
+ by_task[trial.task_name].append(trial)
+ for attempts in by_task.values():
+ attempts.sort(key=lambda t: t.started_at or "")
+
+ max_attempts = max(len(attempts) for attempts in by_task.values())
+ if max_attempts < 2:
+ rate = sum(1 for t in scored if t.passed) / len(scored)
+ return rate, None, 1
+
+ attempt_rates: list[float] = []
+ for i in range(max_attempts):
+ attempt_trials = [
+ attempts[i] for attempts in by_task.values() if len(attempts) > i
+ ]
+ if attempt_trials:
+ attempt_rates.append(
+ sum(1 for t in attempt_trials if t.passed) / len(attempt_trials)
+ )
+ if not attempt_rates:
+ return None, None, max_attempts
+ mean = sum(attempt_rates) / len(attempt_rates)
+ return mean, sample_sem(attempt_rates), max_attempts
+
+
+def _mean(values: list[float]) -> float | None:
+ return statistics.mean(values) if values else None
+
+
+def slice_metrics(records: list[TrialRecord]) -> dict[str, Any]:
+ """Mean resource usage over a set of trials (used for both the whole run
+ and the passing subset)."""
+ empty = {
+ "n": 0,
+ "n_with_metrics": 0,
+ "mean_steps": None,
+ "mean_total_tokens": None,
+ "mean_effective_tokens": None,
+ "mean_tool_calls": None,
+ "mean_tool_calls_by_tool": {},
+ "mean_duration_secs": None,
+ }
+ if not records:
+ return empty
+ # Resource means are taken only over trials that actually produced metrics
+ # (result.json present). Trials hard-killed on timeout count toward the pass
+ # rate but must not drag token/step means toward zero.
+ with_result = [r for r in records if r.has_result]
+ steps = [float(r.step_count) for r in records if r.step_count is not None]
+ tool_totals = [
+ float(r.total_tool_calls) for r in records if r.total_tool_calls is not None
+ ]
+ per_tool_sum: Counter[str] = Counter()
+ for record in with_result:
+ per_tool_sum.update(record.tool_calls)
+ per_tool_n = len(with_result)
+ durations = [r.duration_secs for r in records if r.duration_secs is not None]
+ return {
+ "n": len(records),
+ "n_with_metrics": len(with_result),
+ "mean_steps": _mean(steps),
+ "mean_total_tokens": _mean([float(r.total_tokens) for r in with_result]),
+ "mean_effective_tokens": _mean([r.effective_tokens for r in with_result]),
+ "mean_tool_calls": _mean(tool_totals),
+ "mean_tool_calls_by_tool": (
+ {
+ name: per_tool_sum[name] / per_tool_n
+ for name in sorted(per_tool_sum, key=lambda n: -per_tool_sum[n])
+ }
+ if per_tool_n
+ else {}
+ ),
+ "mean_duration_secs": _mean(durations),
+ }
+
+
+def build_report(
+ job_dir: Path,
+ *,
+ label: str | None = None,
+ timeout_is_failure: bool = False,
+) -> dict[str, Any]:
+ records = [
+ record
+ for trial_dir in find_trial_dirs(job_dir)
+ if (record := extract_trial(trial_dir, timeout_is_failure=timeout_is_failure))
+ is not None
+ ]
+ scored = [r for r in records if r.passed is not None]
+ errored = [r for r in records if r.passed is None]
+ passing = [r for r in scored if r.passed]
+ pass_rate, pass_sem, n_attempts = pass_rate_with_sem(scored)
+
+ models = Counter(r.model for r in records if r.model)
+ statuses = Counter(r.status for r in records if r.status)
+
+ return {
+ "label": label or job_dir.name,
+ "job_dir": str(job_dir),
+ "n_trials": len(records),
+ "n_scored": len(scored),
+ "n_passed": len(passing),
+ "n_failed": len(scored) - len(passing),
+ "n_errored": len(errored),
+ "n_attempts": n_attempts,
+ "pass_rate": pass_rate,
+ "pass_sem": pass_sem,
+ "resolved_models": dict(models),
+ "agent_statuses": dict(statuses),
+ # The headline numbers the user asked for: resource usage conditioned on
+ # success, with the overall slice alongside for comparison.
+ "on_success": slice_metrics(passing),
+ "overall": slice_metrics(scored),
+ "errored_trials": [
+ {"task_name": r.task_name, "reason": r.error_reason} for r in errored
+ ],
+ }
+
+
+def _fmt(value: float | None, places: int = 1) -> str:
+ return "n/a" if value is None else f"{value:,.{places}f}"
+
+
+def _fmt_rate(rate: float | None, sem: float | None) -> str:
+ if rate is None:
+ return "n/a"
+ if sem is None:
+ return f"{rate * 100:.1f}%"
+ return f"{rate * 100:.1f}% ± {sem * 100:.1f}"
+
+
+def format_report(report: dict[str, Any]) -> str:
+ lines: list[str] = []
+ lines.append(f"# {report['label']}")
+ lines.append("")
+ lines.append(
+ f"- Success rate: {_fmt_rate(report['pass_rate'], report['pass_sem'])} "
+ f"({report['n_passed']}/{report['n_scored']} scored; "
+ f"{report['n_errored']} errored)"
+ )
+ if report["resolved_models"]:
+ models = ", ".join(
+ f"{name} x{count}" for name, count in report["resolved_models"].items()
+ )
+ lines.append(f"- Models: {models}")
+ lines.append("")
+
+ on_success = report["on_success"]
+ overall = report["overall"]
+ lines.append("| Metric (mean) | On success | Overall (scored) |")
+ lines.append("| --- | --- | --- |")
+ lines.append(
+ f"| Steps | {_fmt(on_success['mean_steps'])} | {_fmt(overall['mean_steps'])} |"
+ )
+ lines.append(
+ f"| Total tokens | {_fmt(on_success['mean_total_tokens'], 0)} "
+ f"| {_fmt(overall['mean_total_tokens'], 0)} |"
+ )
+ lines.append(
+ f"| Effective tokens | {_fmt(on_success['mean_effective_tokens'], 0)} "
+ f"| {_fmt(overall['mean_effective_tokens'], 0)} |"
+ )
+ lines.append(
+ f"| Tool calls | {_fmt(on_success['mean_tool_calls'])} "
+ f"| {_fmt(overall['mean_tool_calls'])} |"
+ )
+ lines.append(
+ f"| Duration (s) | {_fmt(on_success['mean_duration_secs'])} "
+ f"| {_fmt(overall['mean_duration_secs'])} |"
+ )
+ lines.append("")
+
+ if on_success["mean_tool_calls_by_tool"]:
+ lines.append("Tool calls by tool (mean per passing trial):")
+ lines.append("")
+ lines.append("| Tool | On success | Overall (scored) |")
+ lines.append("| --- | --- | --- |")
+ overall_by_tool = overall["mean_tool_calls_by_tool"]
+ for name, value in on_success["mean_tool_calls_by_tool"].items():
+ lines.append(
+ f"| {name} | {_fmt(value, 2)} | {_fmt(overall_by_tool.get(name), 2)} |"
+ )
+ lines.append("")
+
+ if report["errored_trials"]:
+ lines.append(f"Errored trials ({len(report['errored_trials'])}):")
+ for entry in report["errored_trials"][:20]:
+ lines.append(f" - {entry['task_name']}: {entry['reason']}")
+ if len(report["errored_trials"]) > 20:
+ lines.append(f" ... and {len(report['errored_trials']) - 20} more")
+ lines.append("")
+
+ return "\n".join(lines)
+
+
+def locate_job_dir(args: Any) -> Path:
+ """Resolve the fetched Harbor/Pier job directory for a `report` invocation,
+ fetching it first when asked."""
+ job_dir = getattr(args, "job_dir", None)
+ if job_dir:
+ return Path(job_dir)
+
+ run_id = getattr(args, "run_id", None)
+ if not run_id:
+ raise ValueError("provide a run_id or --job-dir")
+ jobs_dir = Path(getattr(args, "jobs_dir", Path.home() / ".cache/harbor/jobs"))
+ candidate = jobs_dir / run_id
+
+ if getattr(args, "fetch", False) or not candidate.exists():
+ # Reuse the existing fetch command to download + extract the archive.
+ from .volume import command_fetch
+
+ command_fetch(args)
+ if not candidate.exists():
+ raise ValueError(
+ f"no fetched job at {candidate}. Run `zed-eval fetch {run_id} "
+ f"--experiment-name <benchmark>` first, or pass --job-dir."
+ )
+ return candidate
+
+
+def _timeout_is_failure(args: Any) -> bool:
+ """Agent timeouts count as failures on test-scored benchmarks. Derived from
+ the benchmark's scoring (via --experiment-name), with an explicit override."""
+ override = getattr(args, "timeouts_as_failures", None)
+ if override is not None:
+ return bool(override)
+ from . import benchmarks
+
+ experiment = getattr(args, "experiment_name", None)
+ benchmark = benchmarks.BENCHMARKS.get(experiment) if experiment else None
+ return bool(benchmark and benchmark.scoring == benchmarks.SCORING_TESTS)
+
+
+def _fill_run_location_from_index(args: Any) -> None:
+ """Backfill --experiment-name/--namespace from the local run index so a bare
+ run id is enough for `report`. Best-effort and only fills what's missing."""
+ run_id = getattr(args, "run_id", None)
+ if not run_id or getattr(args, "experiment_name", None):
+ return
+ from . import run_index
+
+ entry = run_index.lookup(run_id)
+ if not entry:
+ return
+ args.experiment_name = entry.get("experiment_name")
+ if not getattr(args, "namespace", None):
+ args.namespace = entry.get("namespace")
+
+
+def command_report(args: Any) -> int:
+ _fill_run_location_from_index(args)
+ job_dir = locate_job_dir(args)
+ label = getattr(args, "experiment_name", None) or getattr(args, "run_id", None)
+ report = build_report(
+ job_dir, label=label, timeout_is_failure=_timeout_is_failure(args)
+ )
+ if getattr(args, "as_json", False):
+ print(json.dumps(report, indent=2))
+ else:
+ print(format_report(report))
+ return 0
diff --git a/crates/eval_cli/zed_eval/run_index.py b/crates/eval_cli/zed_eval/run_index.py
new file mode 100644
index 00000000000..7fcd6864c07
--- /dev/null
+++ b/crates/eval_cli/zed_eval/run_index.py
@@ -0,0 +1,168 @@
+"""Local, launch-time index of runs started from this machine.
+
+Every launch writes a tiny record mapping a run id to where it lives on the
+volume (namespace + experiment) plus a little metadata. This lets `status`,
+`logs`, `fetch`, `report`, and `rejudge` locate a run from just its id, so the
+operator no longer has to repeat `--namespace`/`--experiment-name`.
+
+It is deliberately best-effort: a missing, unreadable, or unwritable index never
+blocks a launch and never crashes a lookup. Commands that can't find a run in
+the index fall back to the explicit `--experiment-name`/`--namespace` flags with
+a friendly hint.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+from pathlib import Path
+from typing import Any
+
+INDEX_VERSION = 1
+MAX_ENTRIES = 500
+REQUIRED_FIELDS = ("run_id", "namespace", "experiment_name")
+OPTIONAL_FIELDS = (
+ "volume",
+ "agent_model",
+ "judge_preset",
+ "build_id",
+ "suite_id",
+ "kind",
+ "created_at",
+)
+
+
+def index_path() -> Path:
+ override = os.environ.get("AGENT_EVALS_RUN_INDEX")
+ if override:
+ return Path(override).expanduser()
+ cache_home = os.environ.get("XDG_CACHE_HOME")
+ base = Path(cache_home).expanduser() if cache_home else Path.home() / ".cache"
+ return base / "agent-evals" / "run-index.json"
+
+
+def _string_field(data: dict[str, Any], field: str) -> str | None:
+ value = data.get(field)
+ if value is None:
+ return None
+ value = str(value)
+ return value if value else None
+
+
+def _normalize_entry(data: dict[str, Any]) -> dict[str, Any] | None:
+ entry = {}
+ for field in REQUIRED_FIELDS:
+ value = _string_field(data, field)
+ if value is None:
+ return None
+ entry[field] = value
+ for field in OPTIONAL_FIELDS:
+ value = data.get(field)
+ if value is not None:
+ entry[field] = value
+ return entry
+
+
+def _load_entries() -> list[dict[str, Any]]:
+ try:
+ data = json.loads(index_path().read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ return []
+ raw_entries = data.get("runs") if isinstance(data, dict) else data
+ if not isinstance(raw_entries, list):
+ return []
+
+ entries = []
+ for raw_entry in raw_entries:
+ if not isinstance(raw_entry, dict):
+ continue
+ entry = _normalize_entry(raw_entry)
+ if entry is not None:
+ entries.append(entry)
+ return entries
+
+
+def _entry_from_request(run_request: dict[str, Any]) -> dict[str, Any] | None:
+ entry = _normalize_entry(
+ {
+ "run_id": run_request.get("run_id"),
+ "namespace": run_request.get("namespace"),
+ "experiment_name": run_request.get("experiment_name"),
+ "volume": run_request.get("volume_name"),
+ "agent_model": run_request.get("agent_model"),
+ "judge_preset": run_request.get("judge_preset"),
+ "build_id": run_request.get("build_id"),
+ "suite_id": run_request.get("suite_id"),
+ "kind": run_request.get("kind"),
+ "created_at": run_request.get("created_at"),
+ }
+ )
+ return entry
+
+
+def _write_entries(entries: list[dict[str, Any]]) -> None:
+ path = index_path()
+ payload = json.dumps({"version": INDEX_VERSION, "runs": entries}, indent=2) + "\n"
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary_path: Path | None = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ "w",
+ encoding="utf-8",
+ dir=path.parent,
+ prefix=f".{path.name}.",
+ suffix=".tmp",
+ delete=False,
+ ) as temporary_file:
+ temporary_path = Path(temporary_file.name)
+ temporary_file.write(payload)
+ os.replace(temporary_path, path)
+ except OSError:
+ if temporary_path is not None:
+ temporary_path.unlink(missing_ok=True)
+ raise
+
+
+def record_run(run_request: dict[str, Any]) -> None:
+ """Append (or refresh) the index entry for a launched run.
+
+ Accepts the same request dict the controller is spawned with. Silently does
+ nothing if the essential fields are missing or the index can't be written —
+ bookkeeping must never abort a launch.
+ """
+ entry = _entry_from_request(run_request)
+ if entry is None:
+ return
+
+ entries = [
+ item for item in _load_entries() if item.get("run_id") != entry["run_id"]
+ ]
+ entries.append(entry)
+ entries = entries[-MAX_ENTRIES:]
+ try:
+ _write_entries(entries)
+ except OSError:
+ pass
+
+
+def lookup(run_id: str | None) -> dict[str, Any] | None:
+ """Return the most recently recorded entry for `run_id`, if any."""
+ if not run_id:
+ return None
+ for entry in reversed(_load_entries()):
+ if entry["run_id"] == run_id:
+ return entry
+ return None
+
+
+def recent(limit: int = 20) -> list[dict[str, Any]]:
+ """Return up to `limit` most-recent entries, newest first."""
+ if limit <= 0:
+ return []
+ return list(reversed(_load_entries()[-limit:]))
+
+
+def most_recent() -> dict[str, Any] | None:
+ entries = recent(1)
+ return entries[0] if entries else None
diff --git a/crates/eval_cli/zed_eval/source.py b/crates/eval_cli/zed_eval/source.py
new file mode 100644
index 00000000000..dbdb054604d
--- /dev/null
+++ b/crates/eval_cli/zed_eval/source.py
@@ -0,0 +1,332 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import subprocess
+from pathlib import Path
+from typing import Any
+
+BUILD_SCHEMA_VERSION = "agent-evals-build-v3"
+SOURCE_SCHEMA_VERSION = "agent-evals-source-v1"
+BUILD_TARGET = "x86_64-unknown-linux-musl"
+BUILD_IMAGE_RECIPE_VERSION = "agent-evals-build-image-v1"
+RUST_VERSION = "1.95.0"
+RUST_IMAGE_TAG = f"rust:{RUST_VERSION}"
+RUST_IMAGE_DIGEST = (
+ "sha256:f49565f188ee00bc2a18dd418183f2c5f23ef7d6e691890517ed341a598f67c3"
+)
+RUST_IMAGE = f"{RUST_IMAGE_TAG}@{RUST_IMAGE_DIGEST}"
+ZIG_VERSION = "0.15.2"
+CARGO_ZIGBUILD_VERSION = "0.22.3"
+DEFAULT_REPO_URL = os.environ.get(
+ "AGENT_EVALS_REPO_URL", "https://github.com/zed-industries/zed.git"
+)
+
+
+def package_root() -> Path:
+ return Path(__file__).resolve().parent
+
+
+def eval_cli_root() -> Path:
+ return package_root().parent
+
+
+def repo_root() -> Path:
+ try:
+ result = subprocess.run(
+ ["git", "rev-parse", "--show-toplevel"],
+ cwd=eval_cli_root(),
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return Path(result.stdout.strip())
+ except (OSError, subprocess.CalledProcessError):
+ return eval_cli_root().parents[1]
+
+
+def git_output(args: list[str], cwd: Path | None = None) -> str:
+ result = subprocess.run(
+ ["git", *args],
+ cwd=cwd or repo_root(),
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return result.stdout.strip()
+
+
+def git_bytes(args: list[str], cwd: Path | None = None) -> bytes:
+ result = subprocess.run(
+ ["git", *args],
+ cwd=cwd or repo_root(),
+ check=True,
+ capture_output=True,
+ )
+ return result.stdout
+
+
+def git_path_list(args: list[str]) -> list[str]:
+ data = git_bytes([*args, "-z"])
+ return [path.decode("utf-8") for path in data.split(b"\0") if path]
+
+
+def current_base_sha() -> str:
+ return git_output(["rev-parse", "HEAD"])
+
+
+def commit_present(sha: str) -> bool:
+ """Whether `sha` resolves to a commit object in the local repo."""
+ result = subprocess.run(
+ ["git", "-C", repo_root(), "cat-file", "-e", f"{sha}^{{commit}}"],
+ capture_output=True,
+ )
+ return result.returncode == 0
+
+
+def base_sha_on_main(base_sha: str, repo_url: str) -> bool:
+ """Whether `base_sha` is reachable from origin/main.
+
+ Resolves origin/main's tip against the remote (so it doesn't depend on the
+ caller's possibly-stale local refs), fetching main if the objects aren't
+ present locally, then checks ancestry. Raises only when the main tip can't be
+ made available locally (a network/remote problem), so callers can
+ distinguish that from a definitive "not on main".
+ """
+ main_sha = resolve_remote_ref(repo_url, "main")
+ if base_sha.lower() == main_sha.lower():
+ return True
+ if not (commit_present(base_sha) and commit_present(main_sha)):
+ subprocess.run(
+ ["git", "-C", repo_root(), "fetch", "--quiet", repo_url, "main"],
+ capture_output=True,
+ )
+ if not commit_present(main_sha):
+ raise RuntimeError(
+ f"origin/main tip {main_sha[:12]} unavailable locally after fetch"
+ )
+ if not commit_present(base_sha):
+ # Not present even after pulling main's history -> not reachable from main.
+ return False
+ ancestry = subprocess.run(
+ ["git", "-C", repo_root(), "merge-base", "--is-ancestor", base_sha, main_sha],
+ capture_output=True,
+ )
+ if ancestry.returncode in (0, 1):
+ return ancestry.returncode == 0
+ raise RuntimeError(
+ (ancestry.stderr or b"").decode(errors="replace").strip()
+ or "git merge-base --is-ancestor failed"
+ )
+
+
+def resolve_git_ref(ref: str) -> str:
+ return git_output(["rev-parse", "--verify", f"{ref}^{{commit}}"])
+
+
+_FULL_SHA_RE = re.compile(r"[0-9a-fA-F]{40}")
+
+
+def resolve_remote_ref(repo_url: str, ref: str) -> str:
+ """Resolve a git ref/tag/branch/SHA to a canonical commit SHA against the
+ *remote* repo, so the resulting build id is identical for every launcher
+ regardless of what they happen to have fetched locally.
+
+ A full 40-char SHA is already canonical and is returned (lowercased) without
+ a network call. For named refs and tags this uses `git ls-remote`, preferring
+ the peeled commit of an annotated tag.
+ """
+ if _FULL_SHA_RE.fullmatch(ref):
+ return ref.lower()
+ output = git_output(
+ [
+ "ls-remote",
+ repo_url,
+ ref,
+ f"refs/tags/{ref}",
+ f"refs/heads/{ref}",
+ ]
+ )
+ entries: list[tuple[str, str]] = []
+ for line in output.splitlines():
+ sha, _, name = line.partition("\t")
+ if sha and name:
+ entries.append((name, sha))
+ # An annotated tag yields both the tag object and a peeled "<ref>^{}" entry
+ # pointing at the underlying commit; the commit is what we want to build.
+ for name, sha in entries:
+ if name.endswith("^{}"):
+ return sha
+ if entries:
+ return entries[0][1]
+ raise ValueError(
+ f"could not resolve ref '{ref}' against {repo_url}. "
+ "Push the commit/tag, or pass a full commit SHA."
+ )
+
+
+def current_ref_name() -> str | None:
+ try:
+ ref_name = git_output(["rev-parse", "--abbrev-ref", "HEAD"])
+ except (OSError, subprocess.CalledProcessError):
+ return None
+ return None if ref_name == "HEAD" else ref_name
+
+
+def current_tracked_patch(base_sha: str) -> str:
+ result = subprocess.run(
+ ["git", "diff", "--binary", base_sha],
+ cwd=repo_root(),
+ check=True,
+ capture_output=True,
+ )
+ return result.stdout.decode("utf-8")
+
+
+def read_patch(patch_path: str | None, base_sha: str, *, clean: bool = False) -> str:
+ if clean:
+ return ""
+ if not patch_path:
+ return current_tracked_patch(base_sha)
+ return Path(patch_path).read_text()
+
+
+def untracked_files() -> list[str]:
+ return git_path_list(["ls-files", "--others", "--exclude-standard"])
+
+
+def sha256_text(text: str) -> str:
+ return hashlib.sha256(text.encode()).hexdigest()
+
+
+def format_untracked_warning(files: list[str]) -> str:
+ preview = "\n".join(f" - {path}" for path in files[:20])
+ if len(files) > 20:
+ preview += f"\n ... and {len(files) - 20} more"
+ return (
+ "untracked files are present and will NOT be included in the build patch.\n"
+ "Commit/remove them, or pass --allow-untracked to proceed anyway.\n"
+ f"{preview}"
+ )
+
+
+def prepare_build_source(
+ *,
+ base_sha: str | None,
+ patch_path: str | None,
+ allow_untracked: bool,
+ require_clean: bool,
+ repo_url: str | None,
+ clean: bool = False,
+ source_label: str | None = None,
+ pre_resolved_base_sha: str | None = None,
+) -> tuple[dict[str, Any], str]:
+ # `pre_resolved_base_sha` is a canonical SHA already resolved against the
+ # remote (see `resolve_remote_ref`); it must be used verbatim because the
+ # launcher may not have that commit fetched locally to `rev-parse`.
+ if pre_resolved_base_sha:
+ resolved_base_sha = pre_resolved_base_sha
+ elif base_sha:
+ resolved_base_sha = resolve_git_ref(base_sha)
+ else:
+ resolved_base_sha = current_base_sha()
+ resolved_repo_url = repo_url or DEFAULT_REPO_URL
+ patch = read_patch(patch_path, resolved_base_sha, clean=clean)
+ patch_sha256 = sha256_text(patch) if patch.strip() else None
+ untracked = [] if clean else untracked_files()
+ is_dirty = bool(patch.strip())
+
+ if untracked and not allow_untracked:
+ raise ValueError(format_untracked_warning(untracked))
+
+ if require_clean and is_dirty:
+ raise ValueError(
+ "tracked changes are present. Commit or stash them, or omit --require-clean."
+ )
+
+ return {
+ "schema": SOURCE_SCHEMA_VERSION,
+ "type": "git_patch",
+ "repo_url": resolved_repo_url,
+ "base_sha": resolved_base_sha,
+ "base_ref": source_label or (base_sha if clean else current_ref_name()),
+ "patch_sha256": patch_sha256,
+ "is_dirty": is_dirty,
+ "patch_path": "source.patch" if patch.strip() else None,
+ "untracked_files": untracked,
+ "untracked_files_included": False,
+ "allow_untracked": allow_untracked,
+ "require_clean": require_clean,
+ "clean_source": clean,
+ }, patch
+
+
+def public_source_info(source_info: dict[str, Any]) -> dict[str, Any]:
+ return dict(source_info)
+
+
+def build_toolchain_info(
+ *,
+ target: str = BUILD_TARGET,
+ rust_image: str = RUST_IMAGE,
+ zig_version: str = ZIG_VERSION,
+ cargo_zigbuild_version: str = CARGO_ZIGBUILD_VERSION,
+) -> dict[str, str]:
+ return {
+ "build_image_recipe_version": BUILD_IMAGE_RECIPE_VERSION,
+ "target": target,
+ "rust_version": RUST_VERSION,
+ "rust_image": rust_image,
+ "rust_image_tag": RUST_IMAGE_TAG,
+ "rust_image_digest": RUST_IMAGE_DIGEST,
+ "zig_version": zig_version,
+ "cargo_zigbuild_version": cargo_zigbuild_version,
+ }
+
+
+def compute_build_id(
+ *,
+ source_info: dict[str, Any],
+ target: str = BUILD_TARGET,
+ rust_image: str = RUST_IMAGE,
+ zig_version: str = ZIG_VERSION,
+ cargo_zigbuild_version: str = CARGO_ZIGBUILD_VERSION,
+) -> str:
+ payload = {
+ "schema": BUILD_SCHEMA_VERSION,
+ "source_type": "git_patch",
+ "base_sha": source_info["base_sha"],
+ "patch_sha256": source_info["patch_sha256"],
+ **build_toolchain_info(
+ target=target,
+ rust_image=rust_image,
+ zig_version=zig_version,
+ cargo_zigbuild_version=cargo_zigbuild_version,
+ ),
+ }
+ digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
+ return f"bld-{digest[:20]}"
+
+
+def sanitize_namespace(value: str) -> str:
+ sanitized = re.sub(r"[^A-Za-z0-9_.-]+", "-", value.strip().lower()).strip("-.")
+ return sanitized or "default"
+
+
+def default_namespace() -> str:
+ configured = os.environ.get("AGENT_EVALS_NAMESPACE")
+ if configured:
+ return sanitize_namespace(configured)
+
+ try:
+ email = git_output(["config", "user.email"])
+ if email:
+ return sanitize_namespace(email.split("@", 1)[0])
+ except (OSError, subprocess.CalledProcessError):
+ pass
+
+ return sanitize_namespace(
+ os.environ.get("USER") or os.environ.get("LOGNAME") or "default"
+ )
diff --git a/crates/eval_cli/zed_eval/tests/test_agent_common.py b/crates/eval_cli/zed_eval/tests/test_agent_common.py
new file mode 100644
index 00000000000..d2c25a20e34
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_agent_common.py
@@ -0,0 +1,139 @@
+from __future__ import annotations
+
+import argparse
+import io
+import os
+import shlex
+import subprocess
+import tarfile
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+from zed_eval import common
+from zed_eval.agent_common import (
+ add_anthropic_available_models_env,
+ eval_cli_with_log_command,
+)
+
+
+class EvalCliWithLogCommandTests(unittest.TestCase):
+ def run_logged_command(self, script: str, *, timeout_message: str | None = None):
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ log_path = Path(temporary_directory) / "eval-cli.txt"
+ command = eval_cli_with_log_command(
+ ["sh", "-c", shlex.quote(script)],
+ str(log_path),
+ timeout_message=timeout_message,
+ )
+ completed = subprocess.run(
+ command,
+ shell=True,
+ capture_output=True,
+ text=True,
+ )
+ log = log_path.read_text() if log_path.exists() else ""
+ return completed, log
+
+ def test_preserves_non_timeout_exit_status_through_tee(self) -> None:
+ completed, log = self.run_logged_command("echo before-failure; exit 7")
+
+ self.assertEqual(completed.returncode, 7)
+ self.assertIn("before-failure", completed.stdout)
+ self.assertIn("before-failure", log)
+
+ def test_maps_eval_cli_timeout_exit_to_success(self) -> None:
+ completed, log = self.run_logged_command(
+ "echo partial-output; exit 2",
+ timeout_message="timeout converted",
+ )
+
+ self.assertEqual(completed.returncode, 0)
+ self.assertIn("partial-output", completed.stdout)
+ self.assertIn("timeout converted", completed.stdout)
+ self.assertIn("partial-output", log)
+ self.assertIn("timeout converted", log)
+
+
+class EnvForwardingTests(unittest.TestCase):
+ def test_anthropic_available_models_env_is_added(self) -> None:
+ env: dict[str, str] = {}
+ add_anthropic_available_models_env(env, '[{"name":"model"}]')
+
+ self.assertEqual(env["ZED_ANTHROPIC_AVAILABLE_MODELS"], '[{"name":"model"}]')
+
+ def test_configure_modal_environment_sets_documented_secret_env(self) -> None:
+ args = argparse.Namespace(
+ app_name="app",
+ volume="volume",
+ modal_token_secret="modal-secret",
+ api_secret="llm-secret",
+ )
+
+ with patch.dict(os.environ, {}, clear=False):
+ common.configure_modal_environment(args)
+
+ self.assertEqual(os.environ["AGENT_EVALS_APP_NAME"], "app")
+ self.assertEqual(os.environ["AGENT_EVALS_VOLUME"], "volume")
+ self.assertEqual(
+ os.environ["AGENT_EVALS_MODAL_TOKEN_SECRET"], "modal-secret"
+ )
+ self.assertEqual(
+ os.environ["AGENT_EVALS_LLM_PROVIDERS_SECRET"], "llm-secret"
+ )
+
+
+class SafeExtractArchiveTests(unittest.TestCase):
+ def make_archive(
+ self, archive_path: Path, member_name: str, data: bytes = b"data"
+ ) -> None:
+ with tarfile.open(archive_path, "w:gz") as archive:
+ info = tarfile.TarInfo(member_name)
+ info.size = len(data)
+ archive.addfile(info, io.BytesIO(data))
+
+ def test_extracts_normal_members(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ root = Path(temporary_directory)
+ archive_path = root / "archive.tar.gz"
+ destination = root / "out"
+ destination.mkdir()
+ self.make_archive(archive_path, "job/result.json", b"{}")
+
+ with tarfile.open(archive_path, "r:gz") as archive:
+ common.safe_extract_archive(archive, destination)
+
+ self.assertEqual((destination / "job" / "result.json").read_text(), "{}")
+
+ def test_rejects_path_traversal_members(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ root = Path(temporary_directory)
+ archive_path = root / "archive.tar.gz"
+ destination = root / "out"
+ destination.mkdir()
+ self.make_archive(archive_path, "../evil.txt")
+
+ with tarfile.open(archive_path, "r:gz") as archive:
+ with self.assertRaises(ValueError):
+ common.safe_extract_archive(archive, destination)
+
+ def test_rejects_links(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ root = Path(temporary_directory)
+ archive_path = root / "archive.tar.gz"
+ destination = root / "out"
+ destination.mkdir()
+ with tarfile.open(archive_path, "w:gz") as archive:
+ info = tarfile.TarInfo("link")
+ info.type = tarfile.SYMTYPE
+ info.linkname = "/tmp/target"
+ archive.addfile(info)
+
+ with tarfile.open(archive_path, "r:gz") as archive:
+ with self.assertRaises(ValueError):
+ common.safe_extract_archive(archive, destination)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_baseline.py b/crates/eval_cli/zed_eval/tests/test_baseline.py
new file mode 100644
index 00000000000..d1652897b36
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_baseline.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+import unittest
+
+from zed_eval import baseline
+
+
+class BaselineRecordTests(unittest.TestCase):
+ def test_record_uses_base_ref_from_source_info(self) -> None:
+ record, has_build_info = baseline._record_from_provenance(
+ namespace="alice",
+ fallback_experiment="swe-atlas-rf",
+ run_id="run-1",
+ provenance={
+ "request": {
+ "experiment_name": "swe-atlas-rf",
+ "agent_model": "anthropic/claude-sonnet-4-6",
+ "judge_preset": "kimi-k2.7-code",
+ "build_id": "bld-test",
+ },
+ "summary": {"status": "completed", "trial_count": 10},
+ "build_info": {
+ "build_id": "bld-test",
+ "base_sha": "a" * 40,
+ "patch_sha256": None,
+ "source": {"base_ref": "v0.210.0"},
+ },
+ },
+ )
+
+ self.assertTrue(has_build_info)
+ self.assertEqual(record["base_ref"], "v0.210.0")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_benchmarks.py b/crates/eval_cli/zed_eval/tests/test_benchmarks.py
new file mode 100644
index 00000000000..c366855cb4c
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_benchmarks.py
@@ -0,0 +1,137 @@
+from __future__ import annotations
+
+import unittest
+
+from zed_eval import benchmarks, config, harness_command
+
+
+def make_run_request(benchmark_id: str, **overrides: object) -> dict[str, object]:
+ benchmark = benchmarks.get_benchmark(benchmark_id)
+ request: dict[str, object] = {
+ "run_id": "run-1",
+ "namespace": "tester",
+ "experiment_name": benchmark_id,
+ "benchmark": benchmarks.benchmark_metadata(benchmark),
+ "volume_name": "agent-evals",
+ "api_secret_name": "agent-evals-llm-providers",
+ "build_id": "bld-test",
+ "agent_model": "anthropic/claude-sonnet-4-6",
+ "judge_preset": benchmark.default_judge or "leaderboard",
+ }
+ request.update(overrides)
+ return request
+
+
+def option_values(command: list[str], option: str) -> list[str]:
+ return [
+ command[index + 1]
+ for index, argument in enumerate(command[:-1])
+ if argument == option
+ ]
+
+
+class BenchmarkResolutionTests(unittest.TestCase):
+ def test_group_expands_to_parts(self) -> None:
+ self.assertEqual(
+ benchmarks.resolve_benchmarks(["swe-atlas"]),
+ ["swe-atlas-qna", "swe-atlas-rf", "swe-atlas-tw"],
+ )
+
+ def test_aliases_and_dedup(self) -> None:
+ self.assertEqual(
+ benchmarks.resolve_benchmarks(["qna", "tb21", "rf,tw"]),
+ [
+ "swe-atlas-qna",
+ "terminal-bench-2.1",
+ "swe-atlas-rf",
+ "swe-atlas-tw",
+ ],
+ )
+
+
+class HarnessCommandTests(unittest.TestCase):
+ def test_terminal_bench_is_harbor_registry_no_judge(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("terminal-bench-2.1"), "/tmp/jobs"
+ )
+ self.assertEqual(command[0], "harbor")
+ self.assertIn("terminal-bench/terminal-bench-2-1", command)
+ self.assertIn("-d", command)
+ # Test-scored benchmarks must not wire up an LLM judge.
+ self.assertFalse(any(arg.startswith("EVAL_MODEL=") for arg in command))
+ self.assertNotIn("--verifier-import-path", command)
+
+ def test_swe_atlas_rf_wires_judge(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("swe-atlas-rf"), "/tmp/jobs"
+ )
+ verifier_env = option_values(command, "--ve")
+ agent_env = option_values(command, "--ae")
+
+ self.assertEqual(command[0], "harbor")
+ self.assertIn("scale-ai/swe-atlas-rf", command)
+ self.assertIn(
+ config.JUDGE_PROXY_VERIFIER_IMPORT_PATH,
+ option_values(command, "--verifier-import-path"),
+ )
+ self.assertTrue(any(arg.startswith("EVAL_MODEL=") for arg in verifier_env))
+ self.assertTrue(
+ any(arg.startswith("ZED_JUDGE_UPSTREAM=") for arg in verifier_env)
+ )
+ self.assertTrue(
+ any(arg.startswith("ZED_JUDGE_AUTH_ENV=") for arg in verifier_env)
+ )
+ self.assertIn("ZED_JUDGE_MAX_TOKENS=8192", verifier_env)
+ self.assertFalse(any(arg.startswith("ZED_JUDGE_") for arg in agent_env))
+
+ def test_deepswe_uses_pier_path_without_cli_allowlist(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("deepswe"), "/tmp/jobs"
+ )
+ self.assertEqual(command[0], "pier")
+ self.assertIn("-p", command)
+ self.assertIn("/tmp/datasets/deepswe/tasks", command)
+ # Pier uses the Pier-native agent class.
+ self.assertIn("zed_eval.pier_agent:ZedPierAgent", command)
+ # The Pier network allowlist is declared by the agent, not the CLI;
+ # `pier run` has no --agent-allow-host option.
+ self.assertNotIn("--agent-allow-host", command)
+
+ def test_harbor_benchmarks_use_harbor_agent(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("swe-atlas-rf"), "/tmp/jobs"
+ )
+ self.assertIn("zed_eval.agent:ZedAgent", command)
+
+ def test_eval_cli_timeout_override(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("swe-atlas-rf", eval_cli_timeout=123), "/tmp/jobs"
+ )
+ self.assertIn("EVAL_CLI_TIMEOUT=123", command)
+
+ def test_extra_env_is_forwarded_to_harness_agent(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request(
+ "swe-atlas-rf",
+ extra_env={
+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE": "/data/prompts/checklist.md"
+ },
+ ),
+ "/tmp/jobs",
+ )
+
+ self.assertIn(
+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE=/data/prompts/checklist.md",
+ option_values(command, "--ae"),
+ )
+
+ def test_swe_atlas_tw_uses_path_dataset(self) -> None:
+ command = harness_command.build_harness_command(
+ make_run_request("swe-atlas-tw"), "/tmp/jobs"
+ )
+ self.assertIn("-p", command)
+ self.assertIn("/tmp/datasets/swe-atlas-tw/data/tw", command)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_cleanup.py b/crates/eval_cli/zed_eval/tests/test_cleanup.py
new file mode 100644
index 00000000000..da8a5cf7e20
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_cleanup.py
@@ -0,0 +1,79 @@
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+import unittest
+from pathlib import Path
+
+from zed_eval import cleanup
+
+
+def age_path(path: Path, days: float) -> None:
+ past = time.time() - days * 86400.0
+ os.utime(path, (past, past))
+
+
+class PruneArtifactsTests(unittest.TestCase):
+ def setUp(self) -> None:
+ self._tmp = tempfile.TemporaryDirectory()
+ self.root = Path(self._tmp.name)
+ self.addCleanup(self._tmp.cleanup)
+
+ def _make_build(self, build_id: str, age_days: float) -> Path:
+ build_dir = self.root / "builds" / build_id
+ build_dir.mkdir(parents=True)
+ (build_dir / "eval-cli").write_text("binary")
+ ready = build_dir / "READY"
+ ready.write_text("done")
+ age_path(ready, age_days)
+ return build_dir
+
+ def test_eval_results_are_never_touched(self) -> None:
+ runs = self.root / "runs" / "anant" / "swe-atlas-rf" / "run-1"
+ runs.mkdir(parents=True)
+ result_file = runs / "summary.json"
+ result_file.write_text("{}")
+ age_path(result_file, 999)
+
+ cleanup.prune_artifacts(self.root, dry_run=False, build_retention_days=14)
+
+ self.assertTrue(result_file.exists())
+
+ def test_removes_old_builds_keeps_recent(self) -> None:
+ self._make_build("bld-old", age_days=30)
+ self._make_build("bld-new", age_days=1)
+
+ result = cleanup.prune_artifacts(
+ self.root, dry_run=False, build_retention_days=14
+ )
+
+ self.assertFalse((self.root / "builds" / "bld-old").exists())
+ self.assertTrue((self.root / "builds" / "bld-new").exists())
+ self.assertEqual(result["counts"]["builds"], 1)
+
+ def test_dry_run_deletes_nothing(self) -> None:
+ self._make_build("bld-old", age_days=30)
+ result = cleanup.prune_artifacts(
+ self.root, dry_run=True, build_retention_days=14
+ )
+ self.assertTrue((self.root / "builds" / "bld-old").exists())
+ self.assertEqual(result["counts"]["builds"], 1)
+
+ def test_stale_lock_removed(self) -> None:
+ locks = self.root / "build-locks"
+ locks.mkdir(parents=True)
+ stale = locks / "bld-x.json"
+ stale.write_text("{}")
+ age_path(stale, 1) # 1 day > 6h ttl
+ fresh = locks / "bld-y.json"
+ fresh.write_text("{}")
+
+ cleanup.prune_artifacts(self.root, dry_run=False, lock_ttl_hours=6)
+
+ self.assertFalse(stale.exists())
+ self.assertTrue(fresh.exists())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_launch.py b/crates/eval_cli/zed_eval/tests/test_launch.py
new file mode 100644
index 00000000000..7c1ee906b50
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_launch.py
@@ -0,0 +1,146 @@
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import unittest
+from unittest.mock import patch
+
+from zed_eval import config
+from zed_eval.cli import build_parser
+from zed_eval.launch import prepare_benchmark_suite, prepare_shared_build, prepare_suite
+
+
+class LaunchPlanTests(unittest.TestCase):
+ def parse(self, argv: list[str]) -> argparse.Namespace:
+ return build_parser().parse_args(argv)
+
+ def test_swe_atlas_defaults_judges_by_part(self) -> None:
+ args = self.parse(
+ [
+ "swe-atlas",
+ "--parts",
+ "qna,rf",
+ "--build",
+ "bld-test",
+ "--plan",
+ "--allow-untracked",
+ "--n-tasks",
+ "1",
+ "--yes",
+ ]
+ )
+
+ prepared = prepare_suite(args)
+
+ self.assertEqual([part for part, *_ in prepared], ["qna", "rf"])
+ self.assertEqual(prepared[0][1]["build_id"], "bld-test")
+ self.assertEqual(prepared[0][2]["build_id"], "bld-test")
+ self.assertEqual(prepared[1][1]["build_id"], "bld-test")
+ self.assertIsNone(prepared[1][2])
+ self.assertEqual(prepared[0][1]["judge_preset"], "deepseek-v4-pro")
+ self.assertEqual(prepared[1][1]["judge_preset"], "kimi-k2.7-code")
+ self.assertEqual(prepared[0][1]["agent_model"], config.DEFAULT_MODEL)
+
+ def test_baseten_model_preset_generates_provider_json(self) -> None:
+ args = self.parse(
+ [
+ "swe-atlas",
+ "--parts",
+ "tw",
+ "--build",
+ "bld-test",
+ "--plan",
+ "--allow-untracked",
+ "--model",
+ "baseten:kimi-k2.7-code",
+ "--yes",
+ ]
+ )
+
+ prepared = prepare_suite(args)
+ run_request = prepared[0][1]
+
+ self.assertEqual([part for part, *_ in prepared], ["tw"])
+ self.assertEqual(run_request["benchmark"]["id"], "swe-atlas-tw")
+ self.assertEqual(
+ run_request["agent_model"], "baseten/moonshotai/Kimi-K2.7-Code"
+ )
+ self.assertIn(
+ "moonshotai/Kimi-K2.7-Code", run_request["openai_compatible_provider_json"]
+ )
+ self.assertIn(
+ "https://inference.baseten.co/v1",
+ run_request["openai_compatible_provider_json"],
+ )
+
+ def test_interactive_suite_can_include_non_swe_atlas_benchmarks(self) -> None:
+ args = self.parse(
+ [
+ "swe-atlas",
+ "--build",
+ "bld-test",
+ "--plan",
+ "--allow-untracked",
+ "--yes",
+ ]
+ )
+
+ prepared = prepare_benchmark_suite(
+ args, ["qna", "terminal-bench-2.1", "deepswe"]
+ )
+
+ self.assertEqual(
+ [
+ run_request["benchmark"]["id"]
+ for _label, run_request, _build in prepared
+ ],
+ ["swe-atlas-qna", "terminal-bench-2.1", "deepswe"],
+ )
+ self.assertEqual(
+ [label for label, *_ in prepared], ["qna", "terminal-bench-2.1", "deepswe"]
+ )
+ self.assertEqual(prepared[0][1]["judge_preset"], "deepseek-v4-pro")
+ self.assertIsNone(prepared[1][1]["judge_preset"])
+ self.assertIsNone(prepared[2][1]["judge_preset"])
+ self.assertEqual(prepared[0][1]["build_id"], "bld-test")
+ self.assertEqual(prepared[1][1]["build_id"], "bld-test")
+ self.assertEqual(prepared[2][1]["build_id"], "bld-test")
+ self.assertEqual(prepared[0][2]["build_id"], "bld-test")
+ self.assertIsNone(prepared[1][2])
+ self.assertIsNone(prepared[2][2])
+
+ def test_explicit_existing_build_skips_source_preparation(self) -> None:
+ args = self.parse(["run", "rf", "--build", "custom-build"])
+
+ with (
+ contextlib.redirect_stdout(io.StringIO()),
+ patch("zed_eval.launch.build_ready_on_volume", return_value=True),
+ patch("zed_eval.launch.resolve_source") as resolve_source,
+ ):
+ build_id, build_request = prepare_shared_build(args)
+
+ self.assertEqual(build_id, "custom-build")
+ self.assertIsNone(build_request)
+ resolve_source.assert_not_called()
+
+ def test_explicit_missing_build_uses_from_source(self) -> None:
+ args = self.parse(
+ ["run", "rf", "--build", "custom-build", "--from", "v0.210.0"]
+ )
+
+ with (
+ patch("zed_eval.launch.build_ready_on_volume", return_value=False),
+ patch("zed_eval.source.resolve_remote_ref", return_value="a" * 40),
+ ):
+ build_id, build_request = prepare_shared_build(args)
+
+ self.assertEqual(build_id, "custom-build")
+ self.assertEqual(build_request["build_id"], "custom-build")
+ self.assertEqual(build_request["source"]["base_sha"], "a" * 40)
+ self.assertEqual(build_request["source"]["base_ref"], "v0.210.0")
+ self.assertTrue(build_request["source"]["clean_source"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_rejudge.py b/crates/eval_cli/zed_eval/tests/test_rejudge.py
new file mode 100644
index 00000000000..49c815fbfdd
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_rejudge.py
@@ -0,0 +1,123 @@
+from __future__ import annotations
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from zed_eval import config, rejudge
+
+
+class TaskResolutionTests(unittest.TestCase):
+ def test_task_tests_dir_layout(self) -> None:
+ root = Path("/cache")
+ self.assertEqual(
+ rejudge.task_tests_dir("scale-ai/task-abc", "sha256:deadbeef", root),
+ root / "scale-ai" / "task-abc" / "deadbeef" / "tests",
+ )
+
+ def test_task_tests_dir_without_hash_prefix(self) -> None:
+ root = Path("/cache")
+ self.assertEqual(
+ rejudge.task_tests_dir("org/task-1", "rawref", root),
+ root / "org" / "task-1" / "rawref" / "tests",
+ )
+
+ def test_detect_part(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ tests = Path(tmp)
+ self.assertIsNone(rejudge.detect_part(tests))
+ (tests / rejudge.RF_VERIFIER).write_text("")
+ self.assertEqual(rejudge.detect_part(tests), rejudge.PART_RF)
+
+ def test_detect_part_qna(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ tests = Path(tmp)
+ (tests / rejudge.QNA_VERIFIER).write_text("")
+ self.assertEqual(rejudge.detect_part(tests), rejudge.PART_QNA)
+
+
+class RewardRecombinationTests(unittest.TestCase):
+ def test_rf_pass_requires_rubric_and_tests(self) -> None:
+ # rubric passes, tests passed -> overall pass
+ rewards = rejudge.recompute_rf_rewards(
+ {"must_have_pass": True, "agg_score": 0.9},
+ {"tests_reward": 1.0},
+ )
+ self.assertEqual(rewards["reward"], 1.0)
+ self.assertEqual(rewards["overall_pass"], 1.0)
+ self.assertEqual(rewards["must_have_pass"], 1.0)
+ self.assertEqual(rewards["tests_reward"], 1.0)
+
+ def test_rf_rubric_pass_but_tests_fail_is_fail(self) -> None:
+ rewards = rejudge.recompute_rf_rewards(
+ {"must_have_pass": True, "agg_score": 0.9},
+ {"tests_reward": 0.0},
+ )
+ self.assertEqual(rewards["reward"], 0.0)
+
+ def test_rf_tests_pass_but_rubric_fail_is_fail(self) -> None:
+ rewards = rejudge.recompute_rf_rewards(
+ {"must_have_pass": False, "agg_score": 0.2},
+ {"tests_reward": 1.0},
+ )
+ self.assertEqual(rewards["reward"], 0.0)
+ self.assertEqual(rewards["rubrics_agg_score"], 0.2)
+
+ def test_rf_preserves_tests_reward_from_parent(self) -> None:
+ # tests_reward is invariant under rejudge (patch unchanged).
+ rewards = rejudge.recompute_rf_rewards(
+ {"must_have_pass": False, "agg_score": 0.0},
+ {"tests_reward": 1.0, "reward": 1.0},
+ )
+ self.assertEqual(rewards["tests_reward"], 1.0)
+
+ def test_qna_reward_is_pass_verdict(self) -> None:
+ self.assertEqual(rejudge.recompute_qna_rewards({"pass": True})["reward"], 1.0)
+ self.assertEqual(rejudge.recompute_qna_rewards({"pass": False})["reward"], 0.0)
+
+
+class ResultPatchingTests(unittest.TestCase):
+ def test_patch_preserves_agent_metrics(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ result_path = Path(tmp) / "result.json"
+ result_path.write_text(
+ json.dumps(
+ {
+ "task_name": "scale-ai/task-1",
+ "agent_result": {"tokens": 12345},
+ "verifier_result": {"rewards": {"reward": 0.0}},
+ }
+ )
+ )
+ rejudge.patch_result_rewards(result_path, {"reward": 1.0})
+ patched = json.loads(result_path.read_text())
+ # Verdict updated...
+ self.assertEqual(patched["verifier_result"]["rewards"], {"reward": 1.0})
+ # ...but agent metrics and other fields untouched.
+ self.assertEqual(patched["agent_result"], {"tokens": 12345})
+ self.assertEqual(patched["task_name"], "scale-ai/task-1")
+
+
+class JudgeEnvironmentTests(unittest.TestCase):
+ def test_proxy_environment_from_judge_config(self) -> None:
+ judge = config.get_judge("deepseek-v4-pro")
+ env = rejudge.proxy_environment(judge)
+ self.assertEqual(env["ZED_JUDGE_UPSTREAM"], judge.upstream)
+ self.assertEqual(env["ZED_JUDGE_AUTH_ENV"], judge.auth_env)
+ self.assertEqual(env["ZED_JUDGE_MAX_TOKENS"], str(judge.max_tokens))
+
+ def test_proxy_environment_omits_unset_max_tokens(self) -> None:
+ judge = config.get_judge("leaderboard")
+ env = rejudge.proxy_environment(judge)
+ self.assertNotIn("ZED_JUDGE_MAX_TOKENS", env)
+
+ def test_verifier_environment_points_at_proxy(self) -> None:
+ env = rejudge.verifier_environment("some/model", 8089)
+ self.assertEqual(env["EVAL_MODEL"], "some/model")
+ self.assertEqual(env["EVAL_BASE_URL"], "http://127.0.0.1:8089/v1")
+ self.assertTrue(env["EVAL_API_KEY"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_report.py b/crates/eval_cli/zed_eval/tests/test_report.py
new file mode 100644
index 00000000000..52a5282cf53
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_report.py
@@ -0,0 +1,113 @@
+from __future__ import annotations
+
+import unittest
+
+from zed_eval import report
+from zed_eval.report import TrialRecord
+
+
+def trial(
+ task: str,
+ passed: bool | None,
+ *,
+ steps: int,
+ tool_calls: dict[str, int],
+ input_tokens: int = 0,
+ output_tokens: int = 0,
+) -> TrialRecord:
+ return TrialRecord(
+ task_name=task,
+ started_at=None,
+ passed=passed,
+ error_reason=None,
+ input_tokens=input_tokens,
+ output_tokens=output_tokens,
+ step_count=steps,
+ total_tool_calls=sum(tool_calls.values()),
+ tool_calls=dict(tool_calls),
+ has_result=True,
+ )
+
+
+class SliceMetricsTests(unittest.TestCase):
+ def test_means_over_subset(self) -> None:
+ records = [
+ trial("a", True, steps=4, tool_calls={"read_file": 2, "edit_file": 1}),
+ trial("b", True, steps=6, tool_calls={"read_file": 4, "edit_file": 3}),
+ ]
+ metrics = report.slice_metrics(records)
+ self.assertEqual(metrics["n"], 2)
+ self.assertEqual(metrics["mean_steps"], 5.0)
+ self.assertEqual(metrics["mean_tool_calls"], 5.0)
+ self.assertEqual(metrics["mean_tool_calls_by_tool"]["read_file"], 3.0)
+ self.assertEqual(metrics["mean_tool_calls_by_tool"]["edit_file"], 2.0)
+
+ def test_empty_slice(self) -> None:
+ metrics = report.slice_metrics([])
+ self.assertEqual(metrics["n"], 0)
+ self.assertIsNone(metrics["mean_steps"])
+
+
+class ReportConditioningTests(unittest.TestCase):
+ def test_on_success_excludes_failures_and_errors(self) -> None:
+ records = [
+ trial(
+ "pass1", True, steps=3, tool_calls={"read_file": 1}, input_tokens=100
+ ),
+ trial(
+ "fail1", False, steps=9, tool_calls={"read_file": 9}, input_tokens=900
+ ),
+ trial("err1", None, steps=1, tool_calls={"read_file": 1}),
+ ]
+ scored = [r for r in records if r.passed is not None]
+ passing = [r for r in scored if r.passed]
+
+ on_success = report.slice_metrics(passing)
+ overall = report.slice_metrics(scored)
+
+ # Conditioned on success: only the passing trial counts.
+ self.assertEqual(on_success["n"], 1)
+ self.assertEqual(on_success["mean_steps"], 3.0)
+ # Overall (scored) includes the failure, inflating the mean.
+ self.assertEqual(overall["n"], 2)
+ self.assertEqual(overall["mean_steps"], 6.0)
+
+ def test_pass_rate_single_attempt(self) -> None:
+ scored = [
+ trial("a", True, steps=1, tool_calls={}),
+ trial("b", False, steps=1, tool_calls={}),
+ trial("c", True, steps=1, tool_calls={}),
+ trial("d", True, steps=1, tool_calls={}),
+ ]
+ rate, sem, attempts = report.pass_rate_with_sem(scored)
+ self.assertAlmostEqual(rate, 0.75)
+ self.assertIsNone(sem)
+ self.assertEqual(attempts, 1)
+
+
+class TimeoutVerdictTests(unittest.TestCase):
+ def _result(self, exc_type: str) -> dict:
+ return {"exception_info": {"exception_type": exc_type}}
+
+ def test_timeout_excluded_by_default(self) -> None:
+ passed, reason = report.trial_verdict(self._result("AgentTimeoutError"))
+ self.assertIsNone(passed)
+ self.assertEqual(reason, "AgentTimeoutError")
+
+ def test_timeout_counts_as_failure_when_enabled(self) -> None:
+ passed, reason = report.trial_verdict(
+ self._result("AgentTimeoutError"), timeout_is_failure=True
+ )
+ self.assertIs(passed, False)
+ self.assertEqual(reason, "AgentTimeoutError")
+
+ def test_other_exceptions_stay_errored_even_when_enabled(self) -> None:
+ # A genuine harness/infra exception is never a scored failure.
+ passed, _ = report.trial_verdict(
+ self._result("SandboxCreateError"), timeout_is_failure=True
+ )
+ self.assertIsNone(passed)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_run_index.py b/crates/eval_cli/zed_eval/tests/test_run_index.py
new file mode 100644
index 00000000000..1cef1db2ff2
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_run_index.py
@@ -0,0 +1,132 @@
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+from zed_eval import run_index
+
+
+class RunIndexTests(unittest.TestCase):
+ def with_index_path(self, path: Path):
+ return patch.dict(os.environ, {"AGENT_EVALS_RUN_INDEX": str(path)})
+
+ def test_record_lookup_and_recent(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ path = Path(temporary_directory) / "nested" / "run-index.json"
+ with self.with_index_path(path):
+ run_index.record_run(
+ {
+ "run_id": "run-1",
+ "namespace": "alice",
+ "experiment_name": "rf",
+ "volume_name": "custom-volume",
+ "agent_model": "sonnet-4.6",
+ "created_at": "2026-01-01T00:00:00+00:00",
+ }
+ )
+ run_index.record_run(
+ {
+ "run_id": "run-2",
+ "namespace": "alice",
+ "experiment_name": "qna",
+ "volume_name": None,
+ }
+ )
+
+ self.assertEqual(run_index.lookup("run-1")["volume"], "custom-volume")
+ self.assertEqual(
+ [entry["run_id"] for entry in run_index.recent(2)],
+ ["run-2", "run-1"],
+ )
+ self.assertEqual(run_index.most_recent()["run_id"], "run-2")
+ self.assertNotIn("volume", run_index.lookup("run-2"))
+
+ def test_record_refreshes_existing_run(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ path = Path(temporary_directory) / "run-index.json"
+ with self.with_index_path(path):
+ run_index.record_run(
+ {
+ "run_id": "run-1",
+ "namespace": "alice",
+ "experiment_name": "rf",
+ }
+ )
+ run_index.record_run(
+ {
+ "run_id": "run-1",
+ "namespace": "bob",
+ "experiment_name": "tw",
+ }
+ )
+
+ entries = run_index.recent(10)
+ self.assertEqual(len(entries), 1)
+ self.assertEqual(entries[0]["namespace"], "bob")
+ self.assertEqual(entries[0]["experiment_name"], "tw")
+
+ def test_corrupt_or_malformed_index_is_ignored(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ path = Path(temporary_directory) / "run-index.json"
+ with self.with_index_path(path):
+ path.write_text("not json")
+ self.assertIsNone(run_index.lookup("run-1"))
+ self.assertEqual(run_index.recent(), [])
+
+ path.write_text(
+ json.dumps(
+ {
+ "runs": [
+ {"run_id": "missing-location"},
+ ["not", "a", "dict"],
+ {
+ "run_id": "run-1",
+ "namespace": "alice",
+ "experiment_name": "rf",
+ },
+ ]
+ }
+ )
+ )
+ self.assertIsNone(run_index.lookup("missing-location"))
+ self.assertEqual(run_index.lookup("run-1")["experiment_name"], "rf")
+
+ def test_caps_entries(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ path = Path(temporary_directory) / "run-index.json"
+ with self.with_index_path(path), patch.object(run_index, "MAX_ENTRIES", 2):
+ for index in range(3):
+ run_index.record_run(
+ {
+ "run_id": f"run-{index}",
+ "namespace": "alice",
+ "experiment_name": "rf",
+ }
+ )
+
+ self.assertEqual(
+ [entry["run_id"] for entry in run_index.recent(10)],
+ ["run-2", "run-1"],
+ )
+ self.assertIsNone(run_index.lookup("run-0"))
+
+ def test_zero_limit_returns_no_entries(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary_directory:
+ path = Path(temporary_directory) / "run-index.json"
+ with self.with_index_path(path):
+ run_index.record_run(
+ {
+ "run_id": "run-1",
+ "namespace": "alice",
+ "experiment_name": "rf",
+ }
+ )
+ self.assertEqual(run_index.recent(0), [])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/tests/test_source.py b/crates/eval_cli/zed_eval/tests/test_source.py
new file mode 100644
index 00000000000..233e1e397c7
--- /dev/null
+++ b/crates/eval_cli/zed_eval/tests/test_source.py
@@ -0,0 +1,68 @@
+from __future__ import annotations
+
+import argparse
+import unittest
+from unittest.mock import patch
+
+from zed_eval import source
+from zed_eval.builds import prepare_build_request, resolve_source
+
+
+class ResolveRemoteRefTests(unittest.TestCase):
+ def test_full_sha_needs_no_network(self) -> None:
+ sha = "a" * 40
+ with patch("zed_eval.source.git_output") as git_output:
+ self.assertEqual(source.resolve_remote_ref("url", sha), sha)
+ git_output.assert_not_called()
+
+ def test_annotated_tag_prefers_peeled_commit(self) -> None:
+ ls_remote = (
+ "1111111111111111111111111111111111111111\trefs/tags/v0.210.0\n"
+ "2222222222222222222222222222222222222222\trefs/tags/v0.210.0^{}"
+ )
+ with patch("zed_eval.source.git_output", return_value=ls_remote):
+ self.assertEqual(source.resolve_remote_ref("url", "v0.210.0"), "2" * 40)
+
+ def test_unresolvable_ref_raises(self) -> None:
+ with patch("zed_eval.source.git_output", return_value=""):
+ with self.assertRaises(ValueError):
+ source.resolve_remote_ref("url", "missing")
+
+
+class BuildIdDedupTests(unittest.TestCase):
+ """A clean tag and the SHA it resolves to must yield the *same* build id, so
+ a build done by one teammate is reused by another."""
+
+ def _build_id_for(self, from_source: str, resolved_sha: str) -> str:
+ args = argparse.Namespace(
+ from_source=from_source,
+ base_sha=None,
+ clean_source=False,
+ zed_version=None,
+ repo_url=source.DEFAULT_REPO_URL,
+ )
+ with patch("zed_eval.source.resolve_remote_ref", return_value=resolved_sha):
+ base_sha, clean, label, pre_resolved = resolve_source(args)
+ request = prepare_build_request(
+ base_sha=base_sha,
+ patch_path=None,
+ build_id=None,
+ allow_untracked=False,
+ require_clean=False,
+ repo_url=source.DEFAULT_REPO_URL,
+ clean_source=clean,
+ source_label=label,
+ pre_resolved_base_sha=pre_resolved,
+ )
+ return request["build_id"]
+
+ def test_tag_and_sha_share_build_id(self) -> None:
+ sha = "c" * 40
+ from_tag = self._build_id_for("v0.210.0", sha)
+ from_sha = self._build_id_for(sha, sha)
+ self.assertEqual(from_tag, from_sha)
+ self.assertTrue(from_tag.startswith("bld-"))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/eval_cli/zed_eval/verifier.py b/crates/eval_cli/zed_eval/verifier.py
new file mode 100644
index 00000000000..f5bf5eeb03a
--- /dev/null
+++ b/crates/eval_cli/zed_eval/verifier.py
@@ -0,0 +1,59 @@
+from __future__ import annotations
+
+from harbor.models.verifier.result import VerifierResult
+from harbor.utils.env import resolve_env_vars
+from harbor.verifier.verifier import Verifier
+
+from .judge_proxy import JUDGE_PROXY_ENSURE_SCRIPT, JUDGE_PROXY_SCRIPT
+
+
+class ZedJudgeProxyVerifier(Verifier):
+ async def verify(self) -> VerifierResult:
+ await self._ensure_judge_proxy()
+ return await super().verify()
+
+ def _merged_verifier_env(self) -> dict[str, str]:
+ return {
+ **self.task.config.verifier.env,
+ **(self.verifier_env or {}),
+ **self.override_env,
+ }
+
+ async def _ensure_judge_proxy(self) -> None:
+ merged_env = self._merged_verifier_env()
+ if not merged_env.get("ZED_JUDGE_UPSTREAM"):
+ return
+
+ env = resolve_env_vars(merged_env)
+ result = await self.environment.exec(
+ command=(
+ "set -e\n"
+ "mkdir -p /usr/local/lib /usr/local/bin\n"
+ "cat > /usr/local/lib/zed_judge_proxy.py <<'ZED_PROXY_EOF'\n"
+ f"{JUDGE_PROXY_SCRIPT}"
+ "ZED_PROXY_EOF\n"
+ "cat > /usr/local/bin/ensure-judge-proxy <<'ZED_ENSURE_EOF'\n"
+ f"{JUDGE_PROXY_ENSURE_SCRIPT}"
+ "ZED_ENSURE_EOF\n"
+ "chmod +x /usr/local/bin/ensure-judge-proxy\n"
+ "ensure-judge-proxy\n"
+ "python3 - <<'ZED_PROXY_CHECK_EOF'\n"
+ "import os, socket\n"
+ "port = int(os.environ.get('ZED_JUDGE_PROXY_PORT', '8089'))\n"
+ "with socket.create_connection(('127.0.0.1', port), timeout=5):\n"
+ " pass\n"
+ "ZED_PROXY_CHECK_EOF\n"
+ "if [ -f /tmp/zed-judge-proxy.log ]; then "
+ "tail -2 /tmp/zed-judge-proxy.log; fi\n"
+ ),
+ user="root",
+ env=env,
+ )
+ if result.return_code != 0:
+ output = "\n".join(
+ part for part in (result.stdout, result.stderr) if part
+ ).strip()
+ detail = f": {output}" if output else ""
+ raise RuntimeError(
+ f"Failed to install or start judge proxy (exit {result.return_code}){detail}"
+ )
diff --git a/crates/eval_cli/zed_eval/volume.py b/crates/eval_cli/zed_eval/volume.py
new file mode 100644
index 00000000000..c0bdebfe704
--- /dev/null
+++ b/crates/eval_cli/zed_eval/volume.py
@@ -0,0 +1,254 @@
+from __future__ import annotations
+
+import argparse
+import subprocess
+import sys
+import tarfile
+import time
+from pathlib import Path
+
+from . import run_index, source
+from .common import (
+ default_namespace,
+ deployed_function,
+ print_json,
+ print_table,
+ resolve_run_location,
+ run_command,
+ safe_extract_archive,
+)
+
+RECENT_RUN_COLUMNS = [
+ ("created_at", "created"),
+ ("run_id", "run"),
+ ("experiment_name", "experiment"),
+ ("namespace", "namespace"),
+ ("agent_model", "model"),
+ ("judge_preset", "judge"),
+ ("build_id", "build"),
+ ("suite_id", "suite"),
+]
+REMOTE_RUN_COLUMNS = [
+ ("updated_at", "updated"),
+ ("namespace", "namespace"),
+ ("experiment_name", "experiment"),
+ ("run_id", "run"),
+ ("status", "status"),
+ ("agent_model", "model"),
+ ("judge_preset", "judge"),
+ ("build_id", "build"),
+ ("suite_id", "suite"),
+]
+BUILD_COLUMNS = [
+ ("built_at_utc", "built"),
+ ("build_id", "build"),
+ ("ready", "ready"),
+ ("base_sha", "base"),
+ ("patch_sha256", "patch"),
+]
+SUITE_COLUMNS = [
+ ("part", "part"),
+ ("experiment_name", "experiment"),
+ ("run_id", "run"),
+ ("status", "status"),
+ ("updated_at", "updated"),
+ ("trial_count", "trials"),
+]
+
+
+def volume_get(
+ args: argparse.Namespace, remote_path: str, local_path: str
+) -> subprocess.CompletedProcess[str]:
+ return run_command(
+ ["modal", "volume", "get", args.volume, remote_path, local_path],
+ capture=local_path == "-",
+ )
+
+
+def read_volume_text(args: argparse.Namespace, remote_path: str) -> str:
+ result = volume_get(args, remote_path, "-")
+ return result.stdout
+
+
+def build_ready_on_volume(args: argparse.Namespace, build_id: str) -> bool:
+ try:
+ read_volume_text(args, f"builds/{build_id}/READY")
+ return True
+ except (subprocess.CalledProcessError, OSError):
+ return False
+
+
+def ensure_volume_exists(args: argparse.Namespace) -> int:
+ print(f"\nEnsuring Modal volume '{args.volume}' exists...")
+ try:
+ result = subprocess.run(
+ ["modal", "volume", "create", args.volume],
+ text=True,
+ capture_output=True,
+ check=True,
+ )
+ if result.stdout:
+ print(result.stdout, end="")
+ except subprocess.CalledProcessError as error:
+ output = (error.stdout or "") + (error.stderr or "")
+ if "already exists" in output.lower():
+ print(f"Volume '{args.volume}' already exists; continuing.")
+ else:
+ if output:
+ print(output, end="", file=sys.stderr)
+ return error.returncode
+ return 0
+
+
+def print_rows(
+ rows: list[dict], columns: list[tuple[str, str]], *, as_json: bool = False
+) -> None:
+ if as_json:
+ print_json(rows)
+ else:
+ print_table(rows, columns)
+
+
+def run_remote_prefix(args: argparse.Namespace, run_id: str) -> str:
+ namespace, experiment_name = resolve_run_location(args, run_id)
+ return f"runs/{namespace}/{experiment_name}/{run_id}"
+
+
+def read_run_file(args: argparse.Namespace, file_name: str) -> str:
+ run_id = resolve_run_id(args)
+ return read_volume_text(args, f"{run_remote_prefix(args, run_id)}/{file_name}")
+
+
+def suite_rows(args: argparse.Namespace) -> list[dict]:
+ namespace = default_namespace(args)
+ return deployed_function(args, "suite_status").remote(namespace, args.suite_id)
+
+
+def suite_member_args(args: argparse.Namespace, row: dict) -> argparse.Namespace:
+ run_args = argparse.Namespace(**vars(args))
+ run_args.experiment_name = row["experiment_name"]
+ run_args.run_id = row["run_id"]
+ return run_args
+
+
+def resolve_run_id(args: argparse.Namespace) -> str:
+ """Return the run id to act on, defaulting to the most recent local run.
+
+ Commands like `status`/`logs` accept no run id at all, in which case we use
+ the most recently launched run recorded in the local index so a bare
+ `zed-eval status` answers "how's my latest run doing?".
+ """
+ run_id = getattr(args, "run_id", None)
+ if run_id:
+ return run_id
+ entry = run_index.most_recent()
+ if not entry:
+ raise ValueError(
+ "no run id given and the local run index is empty. Launch a run "
+ "first, or pass a run id explicitly."
+ )
+ args.run_id = entry["run_id"]
+ print(
+ f"(no run id given; using most recent: {entry['run_id']} / "
+ f"{entry.get('experiment_name')})",
+ file=sys.stderr,
+ )
+ return entry["run_id"]
+
+
+def command_runs(args: argparse.Namespace) -> int:
+ entries = run_index.recent(args.limit)
+ if getattr(args, "json", False):
+ print_json(entries)
+ return 0
+ if not entries:
+ print(
+ "No runs recorded locally yet. Launch a run, or use "
+ "`zed-eval list --details` to query runs on the volume."
+ )
+ return 0
+ print_table(entries, RECENT_RUN_COLUMNS)
+ return 0
+
+
+def command_list(args: argparse.Namespace) -> int:
+ namespace = None if args.all_namespaces else default_namespace(args)
+ experiment_name = getattr(args, "experiment_name", None)
+ if experiment_name:
+ experiment_name = source.sanitize_namespace(experiment_name)
+ if args.details or args.json:
+ rows = deployed_function(args, "list_runs").remote(
+ namespace, experiment_name, args.limit
+ )
+ print_rows(rows, REMOTE_RUN_COLUMNS, as_json=args.json)
+ return 0
+ remote_path = (
+ f"runs/{namespace}/{experiment_name}/"
+ if experiment_name
+ else f"runs/{namespace}/"
+ )
+ run_command(["modal", "volume", "ls", args.volume, remote_path])
+ return 0
+
+
+def command_builds(args: argparse.Namespace) -> int:
+ if args.details or args.json:
+ rows = deployed_function(args, "list_builds").remote(args.limit)
+ print_rows(rows, BUILD_COLUMNS, as_json=args.json)
+ return 0
+ run_command(["modal", "volume", "ls", args.volume, "builds/"])
+ return 0
+
+
+def command_status(args: argparse.Namespace) -> int:
+ # One-shot status ping: the controller's state.json only carries a coarse
+ # status (pending -> running -> completed/failed), so there is no per-trial
+ # progress worth following. Print it once and return.
+ print(read_run_file(args, "state.json"), end="")
+ return 0
+
+
+def command_logs(args: argparse.Namespace) -> int:
+ print(read_run_file(args, "controller.log"), end="")
+ return 0
+
+
+def command_fetch(args: argparse.Namespace) -> int:
+ jobs_dir = Path(args.jobs_dir).expanduser()
+ jobs_dir.mkdir(parents=True, exist_ok=True)
+ temporary_archive = jobs_dir / f".{args.run_id}.tar.gz"
+ remote_path = f"{run_remote_prefix(args, args.run_id)}/harbor-job.tar.gz"
+ print(f"Fetching {args.volume}:/{remote_path} -> {temporary_archive}")
+ volume_get(args, remote_path, str(temporary_archive))
+ with tarfile.open(temporary_archive, "r:gz") as archive:
+ safe_extract_archive(archive, jobs_dir)
+ temporary_archive.unlink(missing_ok=True)
+ print(f"Extracted Harbor job under {jobs_dir / args.run_id}")
+ return 0
+
+
+def command_suite_status(args: argparse.Namespace) -> int:
+ print_rows(suite_rows(args), SUITE_COLUMNS, as_json=args.json)
+ return 0
+
+
+def command_suite_logs(args: argparse.Namespace) -> int:
+ rows = suite_rows(args)
+ while True:
+ for row in rows:
+ print(
+ f"\n=== {row['part']} / {row['experiment_name']} / {row['run_id']} ==="
+ )
+ run_args = suite_member_args(args, row)
+ run_args.follow = False
+ command_logs(run_args)
+ if not args.follow:
+ return 0
+ time.sleep(args.interval)
+
+
+def command_suite_fetch(args: argparse.Namespace) -> int:
+ result = 0
+ for row in suite_rows(args):
+ result = command_fetch(suite_member_args(args, row)) or result
+ return result