Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
eval_cli: Add remote benchmark orchestration (#59802)
Anant Goel10f501d7007c
diff --git a/crates/eval_cli/.gitignore b/crates/eval_cli/.gitignoreindex a6317b892a1..6b8a8da509d 100644--- a/crates/eval_cli/.gitignore+++ b/crates/eval_cli/.gitignore@@ -1,4 +1,10 @@**/jobs**/*.egg-info**/__pycache__+**/.venv+**/.pytest_cache+**/.ruff_cacheuv.lock++# Experiment artifacts, that do not need to be committed+experiments/*diff --git a/crates/eval_cli/Dockerfile b/crates/eval_cli/Dockerfileindex 9782e5982b9..d67453f1faa 100644--- a/crates/eval_cli/Dockerfile+++ b/crates/eval_cli/Dockerfile@@ -7,11 +7,13 @@# Or use the helper script:# crates/eval_cli/script/build-linux-FROM rust:1.95.0 AS builder+FROM rust:1.95.0@sha256:f49565f188ee00bc2a18dd418183f2c5f23ef7d6e691890517ed341a598f67c3 AS builder++ARG CARGO_ZIGBUILD_VERSION=0.22.3WORKDIR /app- # Pre-install the toolchain specified in rust-toolchain.toml so it is cached.+# Pre-install the toolchain specified in rust-toolchain.toml so it is cached.RUN rustup toolchain install 1.95.0 --profile minimal \--component rustfmt --component clippy --component rust-analyzer --component rust-src \--target wasm32-wasip2 --target wasm32-unknown-unknown --target x86_64-unknown-linux-musl --target x86_64-unknown-linux-gnu@@ -34,7 +36,7 @@ RUN mkdir -p /opt/zig \| tar -xJ -C /opt/zig --strip-components=1 \&& ln -s /opt/zig/zig /usr/local/bin/zig-RUN cargo install --locked cargo-zigbuild+RUN cargo install --locked cargo-zigbuild --version ${CARGO_ZIGBUILD_VERSION}COPY . .diff --git a/crates/eval_cli/README.md b/crates/eval_cli/README.mdindex a9952bbf4fe..3cb54d55376 100644--- a/crates/eval_cli/README.md+++ b/crates/eval_cli/README.md@@ -1,108 +1,81 @@# eval-cli-Headless CLI binary for running Zed's agent in evaluation/benchmark-environments. Designed to work inside containerized environments like-[Harbor](https://harborframework.com/) where the repository is already-checked out and API keys are provided via environment variables.+Headless Rust binary for running Zed's agent in evaluation and benchmark+environments. It is designed for containerized harnesses such as+[Harbor](https://harborframework.com/) and Pier, where the repository is already+checked out and model API keys are provided via environment variables.-Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed-editor — full agentic loop with tool calls, subagents, and retries, just-without a GUI.+`eval-cli` uses the same `NativeAgent` + `AcpThread` pipeline as the production+Zed editor: a full agentic loop with tool calls, subagents, and retries, without+a GUI.++This directory also contains `zed_eval/`, the Python `zed-eval` package used to+build this binary, launch remote benchmark runs on Modal/Harbor/Pier, and fetch+results. For normal benchmark orchestration, start with+[`zed_eval/README.md`](zed_eval/README.md).## Building-### Native (for local testing on the same OS)+### Native, for local testing on the same OS-```+```shcargo build --release -p eval_cli```-### Cross-compile for Linux x86_64 (from macOS or other hosts)+### Linux x86_64, for Harbor/Pier sandboxes-Harbor containers run Linux x86_64. Use the Docker-based build script:+Harbor and Pier containers run Linux x86_64. From the repository root, use the+Docker-based build script:-```+```shcrates/eval_cli/script/build-linux```-This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can-also specify a custom output path:+This produces `target/eval-cli`, an x86_64 Linux ELF binary. You can also+specify a custom output path:-```+```shcrates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux```## Standalone usage-```+```sheval-cli \--workdir /testbed \- --model anthropic/claude-sonnet-4-6-latest \+ --model anthropic/claude-sonnet-4-6 \--instruction "Fix the bug described in..." \--timeout 600 \--output-dir /logs/agent```-Reads API keys from environment variables (`ANTHROPIC_API_KEY`,-`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and-`thread.json` to the output directory.+`eval-cli` reads provider API keys from environment variables such as+`ANTHROPIC_API_KEY` and `OPENAI_API_KEY`. It writes `result.json`, `thread.md`,+and `thread.json` to the output directory.### Exit codes-| Code | Meaning |-| ---- | ---------------------------------- |-| 0 | Agent finished |-| 1 | Error (model/auth/runtime failure) |-| 2 | Timeout |-| 3 | Interrupted (SIGTERM/SIGINT) |--## Harbor integration+| Code | Meaning |+| --- | --- |+| 0 | Agent finished |+| 1 | Error, such as model/auth/runtime failure |+| 2 | Timeout |+| 3 | Interrupted by SIGTERM or SIGINT |-The `zed_eval/` directory contains a Python package that-implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to-be used with `--agent-import-path` without modifying Harbor's source code.+## Running benchmarks-### Setup+Most benchmark runs should use the Python `zed-eval` CLI instead of invoking+`eval-cli` directly. From the repository root:+```sh+crates/eval_cli/script/install-zed-eval+zed-eval doctor --create-volume+zed-eval run rf --from local --n-tasks 2```-pip install -e crates/eval_cli/harbor/-```--### Running with a local binary--Build for Linux first, then pass the binary path:-```-crates/eval_cli/script/build-linux+For one-off source runs without installing the tool globally, use+`crates/eval_cli/script/zed-eval <args>`.-harbor run -d "swebench_verified@latest" \- --agent-import-path zed_eval.agent:ZedAgent \- --ae binary_path=target/eval-cli \- -m anthropic/claude-sonnet-4-6-latest-```--The agent uploads the binary into the container during setup — no-download URL needed during local iteration.--### Running with a download URL--For CI or when the binary is hosted somewhere:--```-harbor run -d "swebench_verified@latest" \- --agent-import-path zed_eval.agent:ZedAgent \- --ak download_url=https://example.com/eval-cli \- -m anthropic/claude-sonnet-4-6-latest-```--### Setting a timeout--Pass `EVAL_CLI_TIMEOUT` via `--ae`:--```-harbor run -d "swebench_verified@latest" \- --agent-import-path zed_eval.agent:ZedAgent \- --ak binary_path=target/eval-cli \- --ae EVAL_CLI_TIMEOUT=600 \- -m anthropic/claude-sonnet-4-6-latest-```+See [`zed_eval/README.md`](zed_eval/README.md) for supported benchmarks, remote+builds, Modal setup, reporting, rejudging, baselines, and Harbor/Pier installed+agent usage.diff --git a/crates/eval_cli/script/install-zed-eval b/crates/eval_cli/script/install-zed-evalnew file mode 100755index 00000000000..dd4bfaea506--- /dev/null+++ b/crates/eval_cli/script/install-zed-eval@@ -0,0 +1,27 @@+#!/usr/bin/env bash+#+# Install or update the zed-eval Python CLI as an editable uv tool.+#+# Usage:+# crates/eval_cli/script/install-zed-eval+#+# After this, `zed-eval` is available on PATH (assuming uv's tool bin directory+# is on PATH) and tracks changes in crates/eval_cli/zed_eval without reinstalling.++set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"+PACKAGE_DIR="$REPO_ROOT/crates/eval_cli/zed_eval"++if ! command -v uv >/dev/null 2>&1; then+ echo "error: uv is required to install zed-eval" >&2+ echo "Install uv from https://docs.astral.sh/uv/getting-started/installation/" >&2+ exit 1+fi++uv tool install --editable "$PACKAGE_DIR" --force++echo ""+echo "Installed zed-eval from $PACKAGE_DIR"+echo "If your shell cannot find 'zed-eval', run: uv tool update-shell"diff --git a/crates/eval_cli/script/zed-eval b/crates/eval_cli/script/zed-evalnew file mode 100755index 00000000000..53ac35de521--- /dev/null+++ b/crates/eval_cli/script/zed-eval@@ -0,0 +1,21 @@+#!/usr/bin/env bash+#+# Run the zed-eval Python CLI from this checkout without manually setting+# PYTHONPATH or installing the tool globally.+#+# Usage:+# crates/eval_cli/script/zed-eval doctor++set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"+PACKAGE_DIR="$REPO_ROOT/crates/eval_cli/zed_eval"++if ! command -v uv >/dev/null 2>&1; then+ echo "error: uv is required to run zed-eval from source" >&2+ echo "Install uv from https://docs.astral.sh/uv/getting-started/installation/" >&2+ exit 1+fi++exec uv run --project "$PACKAGE_DIR" zed-eval "$@"diff --git a/crates/eval_cli/src/main.rs b/crates/eval_cli/src/main.rsindex 11e36914664..ef00fc87e24 100644--- a/crates/eval_cli/src/main.rs+++ b/crates/eval_cli/src/main.rs@@ -15,7 +15,8 @@//! ## Output//!//! Writes to `--output-dir` (default `/logs/agent/`):-//! - `result.json` — structured result with status, timing, and token usage+//! - `result.json` — structured result with status, timing, token usage,+//! step count, and tool-call counts (total and per tool)//! - `thread.md` — full conversation as markdown//! - `thread.json` — raw thread state as JSON//!@@ -72,10 +73,14 @@ struct Args {#[arg(long, default_value = ".")]workdir: PathBuf,- /// Instruction/prompt text. If omitted, read from --instruction-file or stdin.+ /// Instruction/prompt text. If omitted, read from stdin.#[arg(long, allow_hyphen_values = true)]instruction: Option<String>,+ /// File containing additional instruction text appended after the task prompt.+ #[arg(long)]+ instruction_suffix_file: Option<PathBuf>,+/// Language model to use, in `provider/model` format.#[arg(long, default_value = "anthropic/claude-sonnet-4-6-latest")]model: String,@@ -125,6 +130,26 @@ struct EvalResult {cache_creation_input_tokens: Option<u64>,#[serde(skip_serializing_if = "Option::is_none")]cache_read_input_tokens: Option<u64>,+ /// Number of agent (assistant) turns, i.e. model round-trips in the agentic+ /// loop. Reported as "steps" by the eval harness.+ #[serde(skip_serializing_if = "Option::is_none")]+ step_count: Option<u64>,+ /// Total number of tool calls across all steps.+ #[serde(skip_serializing_if = "Option::is_none")]+ tool_call_count: Option<u64>,+ /// Tool calls broken down by tool name.+ #[serde(skip_serializing_if = "Option::is_none")]+ tool_calls: Option<std::collections::BTreeMap<String, u64>>,+}++/// Per-run statistics collected from the finished thread, written into+/// `result.json` so the post-hoc report can compute success-conditioned metrics.+#[derive(Default)]+struct RunStats {+ token_usage: Option<language_model::TokenUsage>,+ step_count: Option<u64>,+ tool_call_count: Option<u64>,+ tool_calls: Option<std::collections::BTreeMap<String, u64>>,}const EXIT_OK: i32 = 0;@@ -166,6 +191,10 @@ fn main() {eprintln!("Error creating output dir {}: {e}", output_dir.display());process::exit(EXIT_ERROR);}+ let output_dir = output_dir.canonicalize().unwrap_or_else(|e| {+ eprintln!("Invalid --output-dir {:?}: {e}", output_dir);+ process::exit(EXIT_ERROR);+ });let http_client = Arc::new(reqwest_client::ReqwestClient::new());let app = gpui_platform::headless().with_http_client(http_client);@@ -174,13 +203,23 @@ fn main() {let app_state = headless::init(cx);cx.set_staff(!args.no_staff);- let auth_tasks = LanguageModelRegistry::global(cx).update(cx, |registry, cx| {- registry- .providers()- .iter()- .map(|p| p.authenticate(cx))- .collect::<Vec<_>>()- });+ // Eval hook: enable additional feature-flag-gated tools (e.g. the LSP+ // navigation tools behind `lsp-tool` / `rename-tool`) so experiments can+ // measure the agent with tools that aren't yet GA. Comma-separated flag+ // names; unset in production.+ if let Ok(raw_flags) = std::env::var("ZED_EVAL_ENABLE_FLAGS") {+ let flags: Vec<String> = raw_flags+ .split(',')+ .map(|flag| flag.trim().to_string())+ .filter(|flag| !flag.is_empty())+ .collect();+ if !flags.is_empty() {+ cx.update_flags(!args.no_staff, flags);+ }+ }++ let openai_compatible_providers_json = openai_compatible_providers_override();+ let anthropic_available_models_json = anthropic_available_models_override();let model_name = args.model.clone();let timeout = args.timeout;@@ -188,11 +227,44 @@ fn main() {let reasoning_effort = args.reasoning_effort.clone();cx.spawn(async move |cx| {+ // Each settings change below is applied in its own `cx.update` call (rather than+ // inline in the synchronous body of `app.run`) so that GPUI flushes the resulting+ // `NotifyGlobalObservers` effect before we move on. Without this, the+ // openai_compatible/anthropic provider registration (driven by an+ // `observe_global::<SettingsStore>` callback in language_models::init) wouldn't+ // have run yet by the time we collect `auth_tasks`, so a newly-added provider's+ // `authenticate()` would never get called and it would be permanently stuck+ // unauthenticated.+ if let Some(providers_json) = &openai_compatible_providers_json {+ let result = cx.update(|cx| apply_openai_compatible_providers(providers_json, cx));+ if let Err(e) = result {+ eprintln!("Error applying {OPENAI_COMPATIBLE_PROVIDERS_ENV}: {e:#}");+ process::exit(EXIT_ERROR);+ }+ }++ if let Some(models_json) = &anthropic_available_models_json {+ let result = cx.update(|cx| apply_anthropic_available_models(models_json, cx));+ if let Err(e) = result {+ eprintln!("Error applying {ANTHROPIC_AVAILABLE_MODELS_ENV}: {e:#}");+ process::exit(EXIT_ERROR);+ }+ }++ let auth_tasks = cx.update(|cx| {+ LanguageModelRegistry::global(cx).update(cx, |registry, cx| {+ registry+ .providers()+ .iter()+ .map(|p| p.authenticate(cx))+ .collect::<Vec<_>>()+ })+ });futures::future::join_all(auth_tasks).await;let start = Instant::now();- let (outcome, token_usage) = run_agent(+ let (outcome, stats) = run_agent(&app_state,&workdir,&instruction,@@ -201,6 +273,8 @@ fn main() {thinking_override,reasoning_effort.as_deref(),Some(&output_dir),+ openai_compatible_providers_json.as_deref(),+ anthropic_available_models_json.as_deref(),cx,).await;@@ -223,6 +297,7 @@ fn main() {}};+ let token_usage = stats.token_usage;let result = EvalResult {status,error,@@ -239,6 +314,9 @@ fn main() {.as_ref().filter(|u| u.cache_read_input_tokens > 0).map(|u| u.cache_read_input_tokens),+ step_count: stats.step_count,+ tool_call_count: stats.tool_call_count,+ tool_calls: stats.tool_calls,};match serde_json::to_string_pretty(&result) {@@ -258,8 +336,60 @@ fn main() {});}+/// Name of the env var carrying a JSON object to merge into+/// `language_models.openai_compatible` user settings before model discovery, in+/// the same shape as Zed's `openai_compatible` settings key (provider id ->+/// `{ "api_url": ..., "available_models": [...] }`). Lets zed-eval route the+/// agent itself through an OpenAI-compatible endpoint (e.g. Baseten) that isn't+/// one of Zed's built-in providers, without hardcoding it into eval-cli.+const OPENAI_COMPATIBLE_PROVIDERS_ENV: &str = "ZED_OPENAI_COMPATIBLE_PROVIDERS";++fn openai_compatible_providers_override() -> Option<String> {+ let raw = std::env::var(OPENAI_COMPATIBLE_PROVIDERS_ENV).ok()?;+ if raw.trim().is_empty() {+ return None;+ }+ Some(raw)+}++fn apply_openai_compatible_providers(providers_json: &str, cx: &mut gpui::App) -> Result<()> {+ let settings = format!(r#"{{"language_models": {{"openai_compatible": {providers_json}}}}}"#);+ SettingsStore::update_global(cx, |store, cx| {+ store.set_user_settings(&settings, cx).result()+ })+ .context("applying openai_compatible provider settings")?;+ Ok(())+}++/// Name of the env var carrying a JSON array to merge into+/// `language_models.anthropic.available_models` user settings before model+/// discovery, in the same shape as Zed's `anthropic.available_models` settings+/// key (a list of `{ "name": ..., "max_tokens": ..., ... }` entries). Lets+/// zed-eval run models that exist on the Anthropic API for the configured+/// key (e.g. early-access-program models) but aren't returned by the live+/// `/v1/models` listing, without hardcoding them into eval-cli.+const ANTHROPIC_AVAILABLE_MODELS_ENV: &str = "ZED_ANTHROPIC_AVAILABLE_MODELS";++fn anthropic_available_models_override() -> Option<String> {+ let raw = std::env::var(ANTHROPIC_AVAILABLE_MODELS_ENV).ok()?;+ if raw.trim().is_empty() {+ return None;+ }+ Some(raw)+}++fn apply_anthropic_available_models(models_json: &str, cx: &mut gpui::App) -> Result<()> {+ let settings =+ format!(r#"{{"language_models": {{"anthropic": {{"available_models": {models_json}}}}}}}"#);+ SettingsStore::update_global(cx, |store, cx| {+ store.set_user_settings(&settings, cx).result()+ })+ .context("applying anthropic available_models settings")?;+ Ok(())+}+fn read_instruction(args: &Args) -> Result<String> {- let text = if let Some(text) = &args.instruction {+ let mut text = if let Some(text) = &args.instruction {text.clone()} else {use std::io::Read;@@ -270,9 +400,23 @@ fn read_instruction(args: &Args) -> Result<String> {buf};anyhow::ensure!(!text.trim().is_empty(), "instruction is empty");++ if let Some(path) = &args.instruction_suffix_file {+ let suffix = read_instruction_suffix_file(path)?;+ text.push_str("\n\n");+ text.push_str(&suffix);+ }Ok(text)}+fn read_instruction_suffix_file(path: &PathBuf) -> Result<String> {+ let suffix = std::fs::read_to_string(path)+ .with_context(|| format!("reading instruction suffix file {}", path.display()))?;+ let suffix = suffix.trim().to_string();+ anyhow::ensure!(!suffix.is_empty(), "instruction suffix file is empty");+ Ok(suffix)+}+async fn wait_for_model(selected: &SelectedModel, cx: &mut AsyncApp) -> Result<()> {let started_at = Instant::now();@@ -458,15 +602,17 @@ async fn run_agent(thinking_override: Option<bool>,reasoning_effort: Option<&str>,output_dir: Option<&std::path::Path>,+ openai_compatible_providers_json: Option<&str>,+ anthropic_available_models_json: Option<&str>,cx: &mut AsyncApp,-) -> (Result<AgentOutcome>, Option<language_model::TokenUsage>) {+) -> (Result<AgentOutcome>, RunStats) {let selected = match SelectedModel::from_str(model_name).map_err(|e| anyhow::anyhow!("{e}")) {Ok(selected) => selected,- Err(e) => return (Err(e), None),+ Err(e) => return (Err(e), RunStats::default()),};if let Err(e) = wait_for_model(&selected, cx).await {- return (Err(e), None);+ return (Err(e), RunStats::default());}let setup_result: Result<()> = cx.update(|cx| {@@ -498,9 +644,79 @@ async fn run_agent("null".to_string()};let provider_id = selected.provider.0.to_string();+ // set_user_settings replaces the whole user settings buffer, so the+ // openai_compatible/anthropic blocks applied earlier (before model+ // discovery) have to be folded back in here, or they would be dropped+ // by this call.+ let mut language_models_fields = Vec::new();+ if let Some(providers_json) = openai_compatible_providers_json {+ language_models_fields.push(format!(r#""openai_compatible": {providers_json}"#));+ }+ if let Some(models_json) = anthropic_available_models_json {+ language_models_fields.push(format!(+ r#""anthropic": {{"available_models": {models_json}}}"#+ ));+ }+ let language_models_settings = format!("{{{}}}", language_models_fields.join(","));+ // Disable specific tools (e.g. `fetch`/`search_web` on air-gapped+ // benchmarks) the canonical way: via the agent profile. The model only+ // sees a built-in tool when the active profile enables it (see+ // Thread::enabled_tools), so we define a dedicated "eval" profile that+ // mirrors the built-in "write" profile minus the disabled tools, and make+ // it the default profile. A fresh profile key is NOT deep-merged against+ // the defaults, so it can't inherit "write"'s tools — the full set is+ // listed explicitly here. Keep WRITE_TOOLS in sync with the "write"+ // profile in assets/settings/default.json.+ let profile_field = {+ let raw = std::env::var("ZED_EVAL_DISABLE_TOOLS").unwrap_or_default();+ let disabled = raw+ .split(',')+ .map(|name| name.trim().to_string())+ .filter(|name| !name.is_empty())+ .collect::<std::collections::HashSet<String>>();+ if disabled.is_empty() {+ String::new()+ } else {+ const WRITE_TOOLS: &[&str] = &[+ "copy_path",+ "create_directory",+ "create_thread",+ "delete_path",+ "diagnostics",+ "apply_code_action",+ "edit_file",+ "write_file",+ "fetch",+ "find_path",+ "find_references",+ "get_code_actions",+ "go_to_definition",+ "list_agents_and_models",+ "list_directory",+ "move_path",+ "rename_symbol",+ "read_file",+ "grep",+ "skill",+ "spawn_agent",+ "terminal",+ "search_web",+ ];+ let tools = WRITE_TOOLS+ .iter()+ .filter(|tool| !disabled.contains(**tool))+ .map(|tool| format!(r#""{tool}": true"#))+ .collect::<Vec<_>>()+ .join(", ");+ format!(+ r#","default_profile": "eval", "profiles": {{"eval": {{"name": "Eval", "enable_all_context_servers": true, "tools": {{{tools}}}}}}}"#+ )+ }+ };SettingsStore::update_global(cx, |store, cx| {let settings = format!(r#"{{+ "language_models": {language_models_settings},"agent": {{"tool_permissions": {{"default": "allow"}},"default_model": {{@@ -508,7 +724,7 @@ async fn run_agent("model": "{model_id}","enable_thinking": {enable_thinking},"effort": {effort}- }}+ }}{profile_field}}},"autosave": "off","format_on_save": "off"@@ -523,7 +739,7 @@ async fn run_agent(});if let Err(e) = setup_result {- return (Err(e), None);+ return (Err(e), RunStats::default());}let project = cx.update(|cx| {@@ -545,7 +761,7 @@ async fn run_agent(let worktree = project.update(cx, |project, cx| project.create_worktree(workdir, true, cx));let worktree = match worktree.await {Ok(w) => w,- Err(e) => return (Err(e).context("creating worktree"), None),+ Err(e) => return (Err(e).context("creating worktree"), RunStats::default()),};let scan_result = worktree.update(cx, |tree, _cx| {@@ -555,9 +771,39 @@ async fn run_agent(});match scan_result {Ok(future) => future.await,- Err(e) => return (Err(e), None),+ Err(e) => return (Err(e), RunStats::default()),+ };++ let output_worktree = match output_dir {+ Some(output_dir) if !output_dir.starts_with(workdir) => {+ let output_worktree = project.update(cx, |project, cx| {+ project.create_worktree(output_dir, true, cx)+ });+ match output_worktree.await {+ Ok(worktree) => Some(worktree),+ Err(e) => {+ return (+ Err(e).context("creating output worktree"),+ RunStats::default(),+ );+ }+ }+ }+ _ => None,};+ if let Some(output_worktree) = output_worktree {+ let scan_result = output_worktree.update(cx, |tree, _cx| {+ tree.as_local()+ .context("expected local output worktree")+ .map(|local| local.scan_complete())+ });+ match scan_result {+ Ok(future) => future.await,+ Err(e) => return (Err(e), RunStats::default()),+ };+ }+let agent = cx.update(|cx| {let thread_store = cx.new(|cx| ThreadStore::new(cx));NativeAgent::new(thread_store, Templates::new(), app_state.fs.clone(), cx)@@ -573,7 +819,7 @@ async fn run_agent(.await{Ok(t) => t,- Err(e) => return (Err(e).context("creating ACP session"), None),+ Err(e) => return (Err(e).context("creating ACP session"), RunStats::default()),};let _subscription = cx.subscribe(&acp_thread, |acp_thread, event, cx| {@@ -638,9 +884,30 @@ async fn run_agent(connection.thread(&session_id, cx)});+ let mut step_count = None;+ let mut tool_call_count = None;+ let mut tool_calls = None;let cumulative_usage = if let Some(thread) = &thread {let db_thread = thread.read_with(cx, |thread, cx| thread.to_db(cx));let db_thread = db_thread.await;+ let mut counts = std::collections::BTreeMap::<String, u64>::new();+ let mut agent_turn_count = 0;+ for message in &db_thread.messages {+ let Some(agent_message) = message.as_agent_message() else {+ continue;+ };+ agent_turn_count += 1;+ for request_message in agent_message.to_request() {+ for content in request_message.content {+ if let language_model::MessageContent::ToolUse(tool_use) = content {+ *counts.entry(tool_use.name.to_string()).or_default() += 1;+ }+ }+ }+ }+ step_count = Some(agent_turn_count);+ tool_call_count = Some(counts.values().sum());+ tool_calls = Some(counts);let usage = db_thread.cumulative_token_usage;if usage.input_tokens > 0 || usage.output_tokens > 0 {Some(usage)@@ -682,7 +949,15 @@ async fn run_agent(}}- (outcome, final_usage)+ (+ outcome,+ RunStats {+ token_usage: final_usage,+ step_count,+ tool_call_count,+ tool_calls,+ },+ )}fn log_acp_thread_event(diff --git a/crates/eval_cli/zed_eval/README.md b/crates/eval_cli/zed_eval/README.mdnew file mode 100644index 00000000000..01dfca777c2--- /dev/null+++ b/crates/eval_cli/zed_eval/README.md@@ -0,0 +1,348 @@+# zed-eval++Python CLI and installed-agent package for building Zed's headless `eval-cli`+binary, launching benchmark runs on Modal/Harbor/Pier, and fetching results.++This README is for `crates/eval_cli/zed_eval/`. The Rust `eval-cli` binary is+documented in [`../README.md`](../README.md). Commands below assume they are run+from the repository root unless otherwise noted.++Most benchmark users should start with `zed-eval`.++## Quick start++Install the Python CLI as an editable tool:++```sh+crates/eval_cli/script/install-zed-eval+```++This wraps `uv tool install --editable crates/eval_cli/zed_eval --force`, so the+installed `zed-eval` command tracks checkout changes without needing+`PYTHONPATH`. If you prefer not to install globally, run from the checkout with:++```sh+crates/eval_cli/script/zed-eval doctor+```++Check your setup and create the Modal volume if needed:++```sh+zed-eval doctor --create-volume+```++Deploy the Modal app once before launching runs, and again after changing the+Python orchestration code:++```sh+zed-eval deploy+```++Deploying replaces the live app and can cancel in-flight runs, so avoid running+it while evals are active.++Launch a small benchmark run from your current checkout:++```sh+zed-eval run rf --from local --n-tasks 2+```++Monitor and report:++```sh+zed-eval runs # recent runs launched from this machine+zed-eval status # status for the most recent run+zed-eval logs <run-id> # controller log+zed-eval report <run-id> --fetch+```++After a launch, the local run index remembers the run's namespace and benchmark,+so most commands only need the run id. If the run was launched elsewhere, pass+`--experiment-name` and, if needed, `--namespace`.++## Development++Run the package tests without manually setting `PYTHONPATH`:++```sh+uv run --project crates/eval_cli/zed_eval python -m unittest discover -s crates/eval_cli/zed_eval/tests+```++## Prerequisites++You need:++- access to the Modal workspace used for evals+- a Modal token secret for the controller, default: `agent-evals-modal-token`+- an LLM-provider secret for agent/judge API keys, default:+ `agent-evals-llm-providers`++The controller secret should contain:++- `MODAL_TOKEN_ID`+- `MODAL_TOKEN_SECRET`++The LLM-provider secret should contain the keys your selected models need, such as:++- `ANTHROPIC_API_KEY`+- `OPENAI_API_KEY`+- `BASETEN_API_KEY`++Defaults can be overridden globally:++```sh+zed-eval \+ --volume agent-evals \+ --api-secret agent-evals-llm-providers \+ --modal-token-secret agent-evals-modal-token \+ doctor+```++## Launch benchmarks++Use `zed-eval run` for non-interactive launches:++```sh+# One SWE-Atlas part+zed-eval run rf --from local --n-tasks 5++# Multiple benchmarks share one build when possible+zed-eval run swe-atlas terminal-bench-2.1 --from v0.210.0 --n-tasks 5 --yes++# DeepSWE runs under Pier automatically+zed-eval run deepswe --from local --n-tasks 10++# Inspect what would happen without launching+zed-eval run swe-atlas --from v0.210.0 --plan+zed-eval run deepswe --from local --dry-run+```++Supported benchmark selectors:++| Selector | Meaning | Scoring |+| --- | --- | --- |+| `swe-atlas` | `qna`, `rf`, and `tw` | LLM judge |+| `qna` / `swe-atlas-qna` | SWE-Atlas Codebase Q&A | LLM judge |+| `rf` / `swe-atlas-rf` | SWE-Atlas Refactoring | LLM judge |+| `tw` / `swe-atlas-tw` | SWE-Atlas Test Writing | LLM judge |+| `terminal-bench-2.1` / `tb21` | Terminal-Bench 2.1 | tests |+| `deepswe` | DeepSWE | tests |++For an interactive prompt, use:++```sh+zed-eval swe-atlas --interactive+```++Despite the command name, the interactive flow can launch SWE-Atlas parts,+Terminal-Bench, and DeepSWE.++## Choose source and builds++`--from` controls what source is built into `eval-cli`:++- `--from local` builds current `HEAD` plus tracked changes.+- `--from <ref/tag/sha>` builds a clean git ref, tag, or SHA.++Builds are content-addressed and reused when possible. You can also name or reuse+a build explicitly:++```sh+zed-eval build --from local+zed-eval run rf --build bld-abc123 --n-tasks 2+zed-eval builds --details+```++Untracked files are not included in builds. If they are irrelevant, opt in:++```sh+zed-eval run rf --from local --allow-untracked --n-tasks 2+```++For reproducible runs, use a clean ref/tag/SHA or require a clean tracked tree:++```sh+zed-eval run rf --from v0.210.0 --n-tasks 2+zed-eval run rf --from local --require-clean --n-tasks 2+```++## Choose models and judges++Default agent model:++- `sonnet-4.6` → `anthropic/claude-sonnet-4-6`++Common agent model examples:++```sh+zed-eval run rf --model sonnet-4.6 --n-tasks 2+zed-eval run rf --model opus-4.5 --n-tasks 2+zed-eval run rf --model baseten:kimi-k2.7-code --n-tasks 2+zed-eval run rf --model baseten:deepseek-v4-pro --n-tasks 2+```++For SWE-Atlas judge defaults, `--judge auto` uses:++- `qna`: `deepseek-v4-pro`+- `rf` and `tw`: `kimi-k2.7-code`++Override the judge when needed:++```sh+zed-eval run rf --judge leaderboard --n-tasks 2+zed-eval run rf --judge deepseek-v4-pro --judge-model deepseek-ai/DeepSeek-V4-Pro+```++Baseten presets generate the OpenAI-compatible provider settings automatically.+The sandbox secret must include `BASETEN_API_KEY`.++## Monitor, fetch, and report++Useful commands:++```sh+zed-eval runs # local, fast recent-run list+zed-eval list --details # query runs on the Modal volume+zed-eval list -e swe-atlas-rf --details+zed-eval status <run-id>+zed-eval logs <run-id>+zed-eval fetch <run-id>+zed-eval report <run-id> --fetch+```++`fetch` downloads the run archive and extracts it to:++```text+~/.cache/harbor/jobs/<run-id>/+```++`report` prints pass rate plus resource usage such as tokens, tool calls, and+agent steps. Use `--json` for machine-readable output.++## Multi-benchmark suites++Launching more than one benchmark in one command groups the runs under a+`suite_id` stored on each run.++```sh+zed-eval run qna rf terminal-bench-2.1 --from local --n-tasks 2 --yes+```++Suite commands operate on all runs in that group:++```sh+zed-eval suite status <suite-id>+zed-eval suite logs <suite-id>+zed-eval suite fetch <suite-id>+```++## Rejudge a finished run++`rejudge` creates a new derived run by re-running only the judge. It does not redo+the agent's work or modify the parent run.++```sh+zed-eval rejudge <parent-run-id> --judge deepseek-v4-pro+zed-eval rejudge <parent-run-id> --judge kimi-k2.7-code --dry-run+zed-eval report <derived-run-id> --fetch+```++If the parent run is not in your local run index, pass `--experiment-name` and+possibly `--namespace`.++## Baselines++Baseline commands record and inspect baseline-of-record results for completed+clean-commit runs:++```sh+zed-eval baseline record <run-id> --experiment-name swe-atlas-rf+zed-eval baseline list+zed-eval baseline show swe-atlas-rf --model anthropic/claude-sonnet-4-6+```++## Storage model++Runs and builds live on the shared Modal volume:++```text+builds/<build-id>/+ eval-cli+ build-info.json+ source-info.json+ source.patch++runs/<namespace>/<experiment-name>/<run-id>/+ request.json+ state.json+ controller.log+ harbor-command.txt+ run-metadata.json+ harbor-job.tar.gz+ summary.json+```++Namespaces prevent accidental collisions but are not access control. Anyone with+access to the Modal workspace/volume can read run manifests, logs, patches, and+archives.++## Standalone eval-cli++For local or custom harness usage, you can build and invoke the Rust binary+directly.++Build for the current platform:++```sh+cargo build --release -p eval_cli+```++Build the Linux binary used by Harbor/Pier sandboxes:++```sh+crates/eval_cli/script/build-linux+```++Run directly:++```sh+eval-cli \+ --workdir /testbed \+ --model anthropic/claude-sonnet-4-6 \+ --instruction "Fix the bug described in..." \+ --timeout 600 \+ --output-dir /logs/agent+```++`eval-cli` reads provider API keys from environment variables and writes+`result.json`, `thread.md`, and `thread.json` to the output directory.++Exit codes:++| Code | Meaning |+| --- | --- |+| 0 | Agent finished |+| 1 | Error, such as model/auth/runtime failure |+| 2 | Timeout |+| 3 | Interrupted by SIGTERM or SIGINT |++## Harbor/Pier installed agent++The Python package also exposes installed-agent classes used by the remote+orchestrator:++- `zed_eval.agent:ZedAgent` for Harbor+- `zed_eval.pier_agent:ZedPierAgent` for Pier++For manual Harbor experiments with a locally built Linux binary:++```sh+pip install -e crates/eval_cli/zed_eval/+crates/eval_cli/script/build-linux++harbor run -d "swebench_verified@latest" \+ --agent-import-path zed_eval.agent:ZedAgent \+ --ae binary_path=target/eval-cli \+ --ae EVAL_CLI_TIMEOUT=600 \+ -m anthropic/claude-sonnet-4-6+```diff --git a/crates/eval_cli/zed_eval/__init__.py b/crates/eval_cli/zed_eval/__init__.pyindex 8cf07a06883..8b137891791 100644--- a/crates/eval_cli/zed_eval/__init__.py+++ b/crates/eval_cli/zed_eval/__init__.py@@ -1,3 +1 @@-from zed_eval.agent import ZedAgent-__all__ = ["ZedAgent"]diff --git a/crates/eval_cli/zed_eval/agent.py b/crates/eval_cli/zed_eval/agent.pyindex 4720a7dbc13..50135583f78 100644--- a/crates/eval_cli/zed_eval/agent.py+++ b/crates/eval_cli/zed_eval/agent.py@@ -1,40 +1,33 @@-"""Harbor agent wrapper for Zed's eval-cli binary.--Usage:- # Build eval-cli locally first:- cargo build --release -p eval_cli-- # Run via Harbor with a local binary:- harbor run -d "dataset@version" \- --agent-import-path zed_eval.agent:ZedAgent \- --ae binary_path=/path/to/target/release/eval-cli \- --agent-model anthropic/claude-sonnet-4-6-latest-- # Or with a download URL (for CI):- harbor run -d "dataset@version" \- --agent-import-path zed_eval.agent:ZedAgent \- --ae download_url=https://example.com/eval-cli \- --agent-model anthropic/claude-sonnet-4-6-latest-"""+"""Harbor installed-agent adapter for Zed's eval-cli binary."""import jsonimport osimport shlex+import tomllibfrom pathlib import Pathfrom harbor.agents.installed.base import BaseInstalledAgent, with_prompt_templatefrom harbor.environments.base import BaseEnvironmentfrom harbor.models.agent.context import AgentContext+from .agent_common import (+ add_anthropic_available_models_env,+ add_openai_compatible_provider_env,+ add_zed_eval_env,+ detect_workdir,+ eval_cli_with_log_command,+ patch_command,+ populate_context_from_result,+ provider_api_env,+)-class ZedAgent(BaseInstalledAgent):- """Runs Zed's headless AI agent (eval-cli) to solve tasks.+# Leave eval-cli time to exit cleanly and flush logs before Harbor's outer+# asyncio.wait_for can kill the agent coroutine.+EVAL_CLI_FINALIZE_BUFFER_SEC = 45+EVAL_CLI_MIN_TIMEOUT_SEC = 60- The eval-cli binary boots a headless GPUI application and uses the same- NativeAgent + AcpThread pipeline as the production Zed editor, driving- the full agentic loop (tool calls, subagents, retries) without a GUI.- """+class ZedAgent(BaseInstalledAgent):def __init__(self,logs_dir: Path,@@ -52,52 +45,12 @@ class ZedAgent(BaseInstalledAgent):return "zed"async def _detect_workdir(self, environment: BaseEnvironment) -> str:- """Detect the working directory inside the container.-- Checks, in order:- 1. Explicit ``EVAL_CLI_WORKDIR`` extra-env override- 2. Well-known dirs with a ``.git`` subdirectory (SWE-bench style)- 3. First git repo found under ``/`` (max depth 3)- 4. Well-known dirs that exist at all (terminal-bench style)- 5. The container's default working directory (``pwd``)- """- override = self._extra_env.get("EVAL_CLI_WORKDIR")- if override:- return override-- # First: try to find a git repo (SWE-bench, etc.)- result = await self.exec_as_agent(- environment,- command=(- "for d in /app /testbed /repo; do "- ' if [ -d "$d/.git" ]; then echo "$d"; exit 0; fi; '- "done; "- "find / -maxdepth 3 -name .git -type d 2>/dev/null "- '| head -1 | sed "s|/.git$||"'- ),- )- workdir = (result.stdout or "").strip()- if workdir:- return workdir-- # Fallback: use the first well-known directory that exists,- # even without .git (terminal-bench containers aren't git repos).- result = await self.exec_as_agent(+ return await detect_workdir(environment,- command=(- "for d in /app /testbed /repo /root /home; do "- ' if [ -d "$d" ]; then echo "$d"; exit 0; fi; '- "done; "- "pwd"- ),- )- workdir = (result.stdout or "").strip()- if workdir:- return workdir-- raise RuntimeError(+ self.exec_as_agent,+ self._extra_env.get,"Could not detect a working directory in the container. "- "Set EVAL_CLI_WORKDIR explicitly via --ae EVAL_CLI_WORKDIR=/path/to/repo"+ "Set EVAL_CLI_WORKDIR explicitly via --ae EVAL_CLI_WORKDIR=/path/to/repo",)async def install(self, environment: BaseEnvironment) -> None:@@ -123,16 +76,25 @@ class ZedAgent(BaseInstalledAgent):env={"DEBIAN_FRONTEND": "noninteractive"},)- # ── Non-essential tooling ─────────────────────────────────────- # Everything below here (Node.js, LSPs, uv/ruff) is nice-to-have.- # If any step fails (e.g. musl incompatibility, network issues),- # log a warning and continue — the agent can still work without- # pre-installed language servers.-+ # Tooling setup is best-effort; benchmark images vary enough that one+ # unsupported runtime should not fail the trial before eval-cli starts.await self._install_node(environment)await self._install_lsps(environment)await self._install_uv_and_ruff(environment)+ # Modal can mount a prebuilt binary inside each sandbox, avoiding uploads.+ container_path = self._extra_env.get("EVAL_CLI_CONTAINER_PATH")+ if container_path:+ await self.exec_as_root(+ environment,+ command=(+ f"cp {shlex.quote(container_path)} /usr/local/bin/eval-cli && "+ "chmod +x /usr/local/bin/eval-cli && "+ "eval-cli --help"+ ),+ )+ return+if self._binary_path:binary = Path(self._binary_path)if not binary.exists():@@ -164,8 +126,9 @@ class ZedAgent(BaseInstalledAgent):raise ValueError("No eval-cli binary provided. "- "Either pass binary_path=/path/to/target/release/eval-cli "- "or set download_url=/EVAL_CLI_DOWNLOAD_URL."+ "Either pass binary_path=/path/to/target/release/eval-cli, "+ "set download_url=/EVAL_CLI_DOWNLOAD_URL, "+ "or set --ae EVAL_CLI_CONTAINER_PATH=/path/inside/container.")async def _install_node(self, environment: BaseEnvironment) -> None:@@ -346,53 +309,100 @@ class ZedAgent(BaseInstalledAgent):self.logger.warning("uv/ruff installation failed (non-fatal): %s", exc)def populate_context_post_run(self, context: AgentContext) -> None:- result_data = None- for json_file in self.logs_dir.rglob("result.json"):- try:- result_data = json.loads(json_file.read_text())- break- except (json.JSONDecodeError, OSError):- continue-- if result_data is None:- self.logger.warning("Could not find or parse result.json from eval-cli")- return+ populate_context_from_result(self.logs_dir, context, self.logger)- if result_data.get("input_tokens") is not None:- context.n_input_tokens = result_data["input_tokens"]- if result_data.get("output_tokens") is not None:- context.n_output_tokens = result_data["output_tokens"]- if result_data.get("cache_read_input_tokens") is not None:- context.n_cache_tokens = result_data["cache_read_input_tokens"]+ def _get_api_env(self) -> dict[str, str]:+ env = provider_api_env(self.model_name)+ add_openai_compatible_provider_env(+ env, self._extra_env.get("ZED_OPENAI_COMPATIBLE_PROVIDERS")+ )+ add_anthropic_available_models_env(+ env, self._extra_env.get("ZED_ANTHROPIC_AVAILABLE_MODELS")+ )+ return env- context.metadata = {- "status": result_data.get("status"),- "duration_secs": result_data.get("duration_secs"),- "model": result_data.get("model"),- }+ def _harbor_agent_budget_sec(self) -> float | None:+ """Recover Harbor's hidden outer timeout so eval-cli can exit first.- def _get_api_env(self) -> dict[str, str]:- env: dict[str, str] = {}- if not self.model_name or "/" not in self.model_name:- return env-- provider = self.model_name.split("/", 1)[0]- provider_env_map = {- "anthropic": "ANTHROPIC_API_KEY",- "openai": "OPENAI_API_KEY",- "google": "GEMINI_API_KEY",- "gemini": "GEMINI_API_KEY",- "deepseek": "DEEPSEEK_API_KEY",- "mistral": "MISTRAL_API_KEY",- }-- env_var = provider_env_map.get(provider)- if env_var:- api_key = os.environ.get(env_var, "")- if api_key:- env[env_var] = api_key+ Harbor 0.15.0 computes this on ``Trial`` but does not pass it to custom+ installed agents. Replaying the calculation host-side lets us preserve+ trajectories instead of losing them to Harbor's coroutine kill.+ """+ config_path = Path(self.logs_dir).parent / "config.json"+ try:+ trial_config = json.loads(config_path.read_text())+ except (OSError, ValueError) as exc:+ print(+ f"[zed-eval] agent budget: could not read {config_path}: {exc!r}",+ flush=True,+ )+ return None- return env+ agent_cfg = trial_config.get("agent") or {}+ base_sec = agent_cfg.get("override_timeout_sec")+ if base_sec is None:+ base_sec = self._task_declared_agent_timeout_sec(+ trial_config.get("task") or {}+ )+ if base_sec is None:+ return None++ max_sec = agent_cfg.get("max_timeout_sec")+ if max_sec is not None:+ base_sec = min(base_sec, max_sec)+ multiplier = trial_config.get("agent_timeout_multiplier")+ if multiplier is None:+ multiplier = trial_config.get("timeout_multiplier", 1.0)+ return float(base_sec) * float(multiplier)++ def _task_declared_agent_timeout_sec(self, task_cfg: dict) -> float | None:+ task_dir = self._resolve_task_dir(task_cfg)+ if task_dir is None:+ return None+ task_toml = task_dir / "task.toml"+ try:+ data = tomllib.loads(task_toml.read_text())+ except (OSError, ValueError) as exc:+ print(+ f"[zed-eval] agent budget: could not parse {task_toml}: {exc!r}",+ flush=True,+ )+ return None+ timeout_sec = (data.get("agent") or {}).get("timeout_sec")+ return float(timeout_sec) if timeout_sec is not None else None++ def _resolve_task_dir(self, task_cfg: dict) -> Path | None:+ path = task_cfg.get("path")+ if path:+ return Path(path).expanduser().resolve()++ # config.json omits Harbor's package content hash; use the newest cached+ # digest that contains a task.toml.+ name = task_cfg.get("name")+ if name and "/" in name:+ try:+ from harbor.constants import PACKAGE_CACHE_DIR+ except ImportError as exc:+ print(+ f"[zed-eval] agent budget: harbor PACKAGE_CACHE_DIR "+ f"unavailable: {exc!r}",+ flush=True,+ )+ return None+ download_dir = task_cfg.get("download_dir")+ base_dir = Path(download_dir) if download_dir else PACKAGE_CACHE_DIR+ org, package = name.split("/", 1)+ package_root = base_dir / org / package+ candidates = [+ digest_dir+ for digest_dir in package_root.glob("*")+ if (digest_dir / "task.toml").is_file()+ ]+ if not candidates:+ return None+ return max(candidates, key=lambda p: p.stat().st_mtime)++ return None@with_prompt_templateasync def run(@@ -401,6 +411,10 @@ class ZedAgent(BaseInstalledAgent):escaped_instruction = shlex.quote(instruction)env = self._get_api_env()+ add_zed_eval_env(+ env, self._extra_env, exclude={"ZED_EVAL_INSTRUCTION_SUFFIX_FILE"}+ )+workdir = await self._detect_workdir(environment)parts = [@@ -412,9 +426,58 @@ class ZedAgent(BaseInstalledAgent):if self.model_name:parts.append(f"--model {shlex.quote(self.model_name)}")- timeout = self._extra_env.get("EVAL_CLI_TIMEOUT")- if timeout:- parts.append(f"--timeout {shlex.quote(timeout)}")+ instruction_suffix_file = self._extra_env.get(+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE"+ )+ if instruction_suffix_file:+ parts.append(+ f"--instruction-suffix-file {shlex.quote(instruction_suffix_file)}"+ )++ # Prefer eval-cli's own timeout over Harbor's coroutine kill so logs and+ # partial answers are still available for delivery.+ configured_timeout_raw = self._extra_env.get("EVAL_CLI_TIMEOUT")+ configured_timeout: int | None+ if configured_timeout_raw:+ try:+ configured_timeout = int(configured_timeout_raw)+ except ValueError:+ print(+ "[zed-eval] ignoring non-integer EVAL_CLI_TIMEOUT="+ f"{configured_timeout_raw!r}",+ flush=True,+ )+ configured_timeout = None+ else:+ configured_timeout = None++ budget_sec = self._harbor_agent_budget_sec()+ timeout_arg: int | None = None+ if budget_sec is not None:+ ceiling = (+ configured_timeout+ if configured_timeout is not None+ else int(budget_sec)+ )+ timeout_arg = min(ceiling, int(budget_sec - EVAL_CLI_FINALIZE_BUFFER_SEC))+ timeout_arg = max(timeout_arg, EVAL_CLI_MIN_TIMEOUT_SEC)+ print(+ f"[zed-eval] eval-cli --timeout {timeout_arg}s (harbor agent "+ f"budget {int(budget_sec)}s - {EVAL_CLI_FINALIZE_BUFFER_SEC}s "+ f"finalize buffer; EVAL_CLI_TIMEOUT="+ f"{configured_timeout if configured_timeout is not None else 'unset'})",+ flush=True,+ )+ elif configured_timeout is not None:+ timeout_arg = configured_timeout+ print(+ f"[zed-eval] eval-cli --timeout {timeout_arg}s (harbor agent "+ "budget unavailable; using EVAL_CLI_TIMEOUT)",+ flush=True,+ )++ if timeout_arg is not None:+ parts.append(f"--timeout {timeout_arg}")staff = self._extra_env.get("EVAL_CLI_STAFF")if staff and staff.lower() == "false":@@ -427,39 +490,41 @@ class ZedAgent(BaseInstalledAgent):enable_thinking = self._extra_env.get("EVAL_CLI_ENABLE_THINKING")if enable_thinking:if enable_thinking.lower() == "true":- parts.append("--enable-thinking")+ parts.append("--thinking true")elif enable_thinking.lower() == "false":- parts.append("--disable-thinking")+ parts.append("--thinking false")parts.append(f"--instruction {escaped_instruction}")+ # Exit 2 is eval-cli's timeout, not a crash; keep the trajectory judgeable.await self.exec_as_agent(environment,- command=(- " ".join(parts) + " 2>&1 | if command -v stdbuf >/dev/null 2>&1;"- " then stdbuf -oL tee /logs/agent/eval-cli.txt;"- " else tee /logs/agent/eval-cli.txt; fi"+ command=eval_cli_with_log_command(+ parts,+ "/logs/agent/eval-cli.txt",+ timeout_message=(+ "[zed-eval] eval-cli timed out (exit 2); continuing to "+ "delivery/verification"+ ),+ line_buffered=True,),env=env,)- # Only generate a patch if the workdir is a git repo with a valid HEAD- # (SWE-bench style). Terminal-bench containers aren't git repos, and- # some harnesses mount an initialized repo before creating the first commit.+ # Modal-style remote runs can differ in agent-log collection behavior.await self.exec_as_agent(environment,command=(- "if git rev-parse --git-dir >/dev/null 2>&1; then "- "git add -A && "- "if git rev-parse --verify HEAD >/dev/null 2>&1; then "- "git diff --cached HEAD -- > /logs/agent/patch.diff && "- 'echo "Patch size: $(wc -c < /logs/agent/patch.diff) bytes"; '- "else "- 'echo "Git repo has no valid HEAD, skipping patch generation"; '- "fi; "- "else "- 'echo "No git repo found, skipping patch generation"; '- "fi"+ "mkdir -p /logs/artifacts && "+ "for f in result.json thread.json thread.md eval-cli.txt; do "+ ' cp "/logs/agent/$f" /logs/artifacts/ 2>/dev/null || true; '+ "done; true"),+ )++ # Some harnesses mount an initialized repo before creating the first commit.+ await self.exec_as_agent(+ environment,+ command=patch_command("/logs/agent"),cwd=workdir,)diff --git a/crates/eval_cli/zed_eval/agent_common.py b/crates/eval_cli/zed_eval/agent_common.pynew file mode 100644index 00000000000..48079f39ca2--- /dev/null+++ b/crates/eval_cli/zed_eval/agent_common.py@@ -0,0 +1,184 @@+from __future__ import annotations++import json+import os+import shlex+from collections.abc import Callable+from pathlib import Path+from typing import Any++PROVIDER_ENV_MAP = {+ "anthropic": "ANTHROPIC_API_KEY",+ "openai": "OPENAI_API_KEY",+ "google": "GEMINI_API_KEY",+ "gemini": "GEMINI_API_KEY",+ "deepseek": "DEEPSEEK_API_KEY",+ "mistral": "MISTRAL_API_KEY",+}+++def provider_api_env(+ model_name: str | None,+ get_env: Callable[[str], str | None] = os.environ.get,+) -> dict[str, str]:+ env: dict[str, str] = {}+ if not model_name or "/" not in model_name:+ return env++ provider = model_name.split("/", 1)[0]+ env_var = PROVIDER_ENV_MAP.get(provider) or (+ f"{provider}_API_KEY".upper().replace("-", "_")+ )+ api_key = get_env(env_var)+ if api_key:+ env[env_var] = api_key+ return env+++def add_openai_compatible_provider_env(+ env: dict[str, str], providers_json: str | None+) -> None:+ if providers_json:+ env["ZED_OPENAI_COMPATIBLE_PROVIDERS"] = providers_json+++def add_anthropic_available_models_env(+ env: dict[str, str], models_json: str | None+) -> None:+ if models_json:+ env["ZED_ANTHROPIC_AVAILABLE_MODELS"] = models_json+++def add_zed_eval_env(+ env: dict[str, str], extra_env: dict[str, str], *, exclude: set[str] | None = None+) -> None:+ exclude = exclude or set()+ for key, value in extra_env.items():+ if key.startswith("ZED_EVAL_") and key not in exclude:+ env[key] = value+++async def detect_workdir(+ environment: Any,+ exec_as_agent: Callable[..., Any],+ get_env: Callable[[str], str | None],+ error_message: str,+) -> str:+ override = get_env("EVAL_CLI_WORKDIR")+ if override:+ return override++ result = await exec_as_agent(+ environment,+ command=(+ "for d in /app /testbed /repo; do "+ ' if [ -d "$d/.git" ]; then echo "$d"; exit 0; fi; '+ "done; "+ "find / -maxdepth 3 -name .git -type d 2>/dev/null "+ '| head -1 | sed "s|/.git$||"'+ ),+ )+ workdir = (result.stdout or "").strip()+ if workdir:+ return workdir++ result = await exec_as_agent(+ environment,+ command=(+ "for d in /app /testbed /repo /root /home; do "+ ' if [ -d "$d" ]; then echo "$d"; exit 0; fi; '+ "done; pwd"+ ),+ )+ workdir = (result.stdout or "").strip()+ if workdir:+ return workdir+ raise RuntimeError(error_message)+++def populate_context_from_result(logs_dir: Path, context: Any, logger: Any) -> None:+ result_data = None+ for json_file in logs_dir.rglob("result.json"):+ try:+ result_data = json.loads(json_file.read_text())+ break+ except (json.JSONDecodeError, OSError):+ continue++ if result_data is None:+ logger.warning("Could not find or parse result.json from eval-cli")+ return++ if result_data.get("input_tokens") is not None:+ context.n_input_tokens = result_data["input_tokens"]+ if result_data.get("output_tokens") is not None:+ context.n_output_tokens = result_data["output_tokens"]+ if result_data.get("cache_read_input_tokens") is not None:+ context.n_cache_tokens = result_data["cache_read_input_tokens"]+ if isinstance(result_data.get("step_count"), int):+ context.n_agent_steps = result_data["step_count"]++ context.metadata = {+ "status": result_data.get("status"),+ "duration_secs": result_data.get("duration_secs"),+ "model": result_data.get("model"),+ "tool_call_count": result_data.get("tool_call_count"),+ }+++def eval_cli_with_log_command(+ parts: list[str],+ log_path: str,+ *,+ timeout_message: str | None = None,+ line_buffered: bool = False,+) -> str:+ """Run eval-cli, tee output, and preserve eval-cli's exit status.++ POSIX shells return the last command's status for a pipeline, so `cmd | tee`+ would otherwise hide eval-cli failures whenever `tee` succeeds.+ """+ status_file = "/tmp/zed-eval-eval-cli-status"+ quoted_status_file = shlex.quote(status_file)+ quoted_log_path = shlex.quote(log_path)+ timeout_handler = 'if [ "$ec" -eq 2 ]; then ec=0; fi; '+ if timeout_message:+ timeout_handler = (+ f'if [ "$ec" -eq 2 ]; then echo {shlex.quote(timeout_message)}; ec=0; fi; '+ )+ tee_command = f"tee {quoted_log_path}"+ if line_buffered:+ tee_command = (+ "if command -v stdbuf >/dev/null 2>&1; "+ f"then stdbuf -oL tee {quoted_log_path}; "+ f"else tee {quoted_log_path}; fi"+ )+ return (+ f'status_file={quoted_status_file}; rm -f "$status_file"; '+ "( "+ + " ".join(parts)+ + "; ec=$?; "+ + timeout_handler+ + 'printf "%s\\n" "$ec" > "$status_file"; '+ + 'exit "$ec" ) 2>&1 | '+ + tee_command+ + '; ec=1; if [ -s "$status_file" ]; then read ec < "$status_file"; fi; '+ + 'rm -f "$status_file"; exit "$ec"'+ )+++def patch_command(agent_dir: str) -> str:+ patch_path = shlex.quote(f"{agent_dir}/patch.diff")+ return (+ "if git rev-parse --git-dir >/dev/null 2>&1; then "+ "git add -A && "+ "if git rev-parse --verify HEAD >/dev/null 2>&1; then "+ f"git diff --cached HEAD -- > {patch_path} && "+ f'echo "Patch size: $(wc -c < {patch_path}) bytes"; '+ "else "+ 'echo "Git repo has no valid HEAD, skipping patch generation"; '+ "fi; "+ "else "+ 'echo "No git repo found, skipping patch generation"; '+ "fi"+ )diff --git a/crates/eval_cli/zed_eval/baseline.py b/crates/eval_cli/zed_eval/baseline.pynew file mode 100644index 00000000000..a1dcf547443--- /dev/null+++ b/crates/eval_cli/zed_eval/baseline.py@@ -0,0 +1,205 @@+"""Baseline-of-record management.++A "baseline" is a completed run promoted to the canonical reference for its+`(experiment, model)` pair. It only qualifies if its build came from a *clean*+commit reachable from `origin/main` (no local patch) — so baselines are both+obviously discoverable (the `baselines/` tree + `zed-eval baseline list`) and+provably pinned to a commit on main. The judge preset is recorded but not gated,+so non-leaderboard baselines are allowed.++Records live on the shared volume at:++ baselines/index.json # roll-up for discovery+ baselines/<experiment>/<model-slug>/current.json # canonical pointer+ baselines/<experiment>/<model-slug>/history/<sha>.json # superseded baselines+"""++from __future__ import annotations++import argparse+import json+from typing import Any++from . import config, source+from .common import default_namespace, deployed_function, utc_now+++def _model_slug(model: str) -> str:+ return source.sanitize_namespace(model)+++def _record_from_provenance(+ *,+ namespace: str,+ fallback_experiment: str,+ run_id: str,+ provenance: dict[str, Any],+) -> tuple[dict[str, Any], bool]:+ request = provenance["request"]+ summary = provenance["summary"]+ build_info = provenance["build_info"]+ source_info = build_info.get("source") or {}++ experiment = request.get("experiment_name") or fallback_experiment+ model = request.get("agent_model") or config.DEFAULT_MODEL+ base_sha = build_info.get("base_sha")++ record = {+ "experiment": experiment,+ "model": model,+ "model_slug": _model_slug(model),+ "judge": request.get("judge_preset") or request.get("judge"),+ "base_sha": base_sha,+ "base_ref": source_info.get("base_ref"),+ "on_main": None,+ "clean": not bool(build_info.get("patch_sha256")),+ "build_id": build_info.get("build_id") or request.get("build_id"),+ "run": {+ "namespace": namespace,+ "experiment_name": experiment,+ "run_id": run_id,+ },+ "resources": {+ "override_cpus": request.get("override_cpus"),+ "override_memory_mb": request.get("override_memory_mb"),+ },+ "summary": {+ "status": summary.get("status"),+ "trial_count": summary.get("trial_count"),+ },+ "recorded_at": utc_now(),+ "recorded_by": namespace,+ }+ return record, bool(build_info)+++def _baseline_problems(+ *,+ record: dict[str, Any],+ has_build_info: bool,+ repo_url: str,+ allow_dirty: bool,+ allow_off_main: bool,+) -> tuple[list[str], bool | None]:+ problems: list[str] = []+ run_id = (record.get("run") or {}).get("run_id")+ status = (record.get("summary") or {}).get("status")+ base_sha = record.get("base_sha")++ if not has_build_info:+ problems.append(+ f"no build-info found for run {run_id}; cannot verify provenance"+ )+ if status != "completed":+ problems.append(f"run status is {status!r}, expected 'completed'")+ if not record.get("clean") and not allow_dirty:+ problems.append(+ "build carries a local patch (not a clean commit); pass --allow-dirty to override"+ )++ on_main: bool | None = None+ if base_sha:+ try:+ on_main = source.base_sha_on_main(base_sha, repo_url)+ except Exception as error: # noqa: BLE001 - surface any verification failure+ if not allow_off_main:+ problems.append(f"could not verify base_sha is on origin/main: {error}")+ if on_main is False and not allow_off_main:+ problems.append(+ f"base_sha {base_sha[:12]} is not reachable from origin/main; "+ "pass --allow-off-main to override"+ )+ elif not allow_off_main:+ problems.append("build-info has no base_sha to verify against origin/main")++ return problems, on_main+++def _print_refusal(run_id: str, problems: list[str]) -> None:+ print(f"refusing to record baseline for {run_id}:")+ for problem in problems:+ print(f" - {problem}")+++def _print_recorded(record: dict[str, Any]) -> None:+ print(+ f"recorded baseline: {record.get('experiment')} / {record.get('model')} "+ f"base_sha={str(record.get('base_sha'))[:12]} "+ f"on_main={record.get('on_main')} clean={record.get('clean')} "+ f"judge={record.get('judge')}"+ )+++def command_baseline_record(args: argparse.Namespace) -> int:+ namespace = default_namespace(args)+ experiment_slug = source.sanitize_namespace(args.experiment_name)+ repo_url = getattr(args, "repo_url", None) or source.DEFAULT_REPO_URL+ read_run_provenance = deployed_function(args, "read_run_provenance")+ record_baseline = deployed_function(args, "record_baseline")++ exit_code = 0+ for run_id in args.run_id:+ provenance = read_run_provenance.remote(namespace, experiment_slug, run_id)+ record, has_build_info = _record_from_provenance(+ namespace=namespace,+ fallback_experiment=args.experiment_name,+ run_id=run_id,+ provenance=provenance,+ )+ problems, on_main = _baseline_problems(+ record=record,+ has_build_info=has_build_info,+ repo_url=repo_url,+ allow_dirty=args.allow_dirty,+ allow_off_main=args.allow_off_main,+ )+ record["on_main"] = on_main++ if problems:+ _print_refusal(run_id, problems)+ exit_code = 1+ continue++ record_baseline.remote(record)+ _print_recorded(record)+ return exit_code+++def command_baseline_list(args: argparse.Namespace) -> int:+ index = deployed_function(args, "read_baselines").remote(None, None, False)+ rows = index.get("baselines") or []+ if getattr(args, "json", False):+ print(json.dumps(index, indent=2))+ return 0+ if not rows:+ print("no baselines recorded")+ return 0+ header = (+ f"{'EXPERIMENT':<20} {'MODEL':<32} {'BASE_REF':<14} "+ f"{'SHA':<12} {'ON_MAIN':<8} {'JUDGE':<14} RUN"+ )+ print(header)+ for row in rows:+ print(+ f"{(row.get('experiment') or ''):<20} "+ f"{(row.get('model') or ''):<32} "+ f"{(row.get('base_ref') or '') or '-':<14} "+ f"{str(row.get('base_sha') or '')[:12]:<12} "+ f"{str(row.get('on_main')):<8} "+ f"{(row.get('judge') or '') or '-':<14} "+ f"{row.get('run_id') or ''}"+ )+ return 0+++def command_baseline_show(args: argparse.Namespace) -> int:+ result = deployed_function(args, "read_baselines").remote(+ args.experiment_name,+ _model_slug(args.model),+ bool(getattr(args, "history", False)),+ )+ if not result or result.get("current") is None:+ print(f"no baseline recorded for {args.experiment_name} / {args.model}")+ return 1+ print(json.dumps(result, indent=2))+ return 0diff --git a/crates/eval_cli/zed_eval/benchmarks.py b/crates/eval_cli/zed_eval/benchmarks.pynew file mode 100644index 00000000000..489ac1baabf--- /dev/null+++ b/crates/eval_cli/zed_eval/benchmarks.py@@ -0,0 +1,239 @@+"""Benchmark registry.++A `Benchmark` describes everything the orchestrator needs to run one benchmark+end-to-end that is *not* about the build source or the model: which harness runs+it (Harbor or its Pier fork), how its dataset is provisioned, how a trial is+scored, whether it needs an LLM judge, and the per-task timeout.++Adding a benchmark should be a data change here, not new control flow. The three+benchmark families currently supported are all Harbor-family harnesses:++ - SWE-Atlas (qna / rf / tw): Harbor, rubric LLM judge.+ - Terminal-Bench 2.1: Harbor, test-script scoring, no judge.+ - DeepSWE: Pier (a Harbor fork with per-agent network allowlists, required+ because DeepSWE tasks run with `allow_internet = false`), test scoring.+"""++from __future__ import annotations++from dataclasses import dataclass, field++# How a benchmark's dataset is made available to the harness.+# registry -> `-d <name>` pulled from the harness hub+# path -> `-p <dir>` from a git repo cloned by the controller+# pier_path -> path dataset run under Pier instead of Harbor+DATASET_REGISTRY = "registry"+DATASET_PATH = "path"+DATASET_PIER_PATH = "pier_path"++HARNESS_HARBOR = "harbor"+HARNESS_PIER = "pier"++SCORING_RUBRIC_JUDGE = "rubric-judge"+SCORING_TESTS = "tests"+++@dataclass(frozen=True)+class DatasetRef:+ kind: str+ # For registry datasets: the hub dataset name. For path datasets: the repo+ # URL plus a ref and the sub-directory the tasks live in.+ name: str | None = None+ repo_url: str | None = None+ repo_ref: str | None = None+ data_dir: str | None = None+++@dataclass(frozen=True)+class Benchmark:+ id: str+ label: str+ harness: str+ dataset: DatasetRef+ default_timeout_secs: int+ scoring: str+ needs_judge: bool+ default_judge: str | None = None+ # Hosts the in-sandbox agent must reach even on air-gapped tasks. Required by+ # Pier benchmarks (DeepSWE) so eval-cli can still call the model API.+ network_allowlist: tuple[str, ...] = ()+ env: dict[str, str] = field(default_factory=dict)+++SWE_ATLAS_REPO_URL = "https://github.com/scaleapi/SWE-Atlas.git"+SWE_ATLAS_REPO_REF = "main"+DEEPSWE_REPO_URL = "https://github.com/datacurve-ai/deep-swe.git"+DEEPSWE_REPO_REF = "main"++# Model/judge API hosts the agent needs even under `allow_internet = false`.+# Pier grants these to the agent via its per-agent allowlist.+AGENT_API_HOSTS: tuple[str, ...] = (+ "api.anthropic.com",+ "api.openai.com",+ "inference.baseten.co",+)+++BENCHMARKS: dict[str, Benchmark] = {+ "swe-atlas-qna": Benchmark(+ id="swe-atlas-qna",+ label="SWE-Atlas Codebase Q&A",+ harness=HARNESS_HARBOR,+ dataset=DatasetRef(kind=DATASET_REGISTRY, name="scale-ai/swe-atlas-qna"),+ default_timeout_secs=7200,+ scoring=SCORING_RUBRIC_JUDGE,+ needs_judge=True,+ default_judge="deepseek-v4-pro",+ ),+ "swe-atlas-rf": Benchmark(+ id="swe-atlas-rf",+ label="SWE-Atlas Refactoring",+ harness=HARNESS_HARBOR,+ dataset=DatasetRef(kind=DATASET_REGISTRY, name="scale-ai/swe-atlas-rf"),+ default_timeout_secs=3300,+ scoring=SCORING_RUBRIC_JUDGE,+ needs_judge=True,+ default_judge="kimi-k2.7-code",+ ),+ "swe-atlas-tw": Benchmark(+ id="swe-atlas-tw",+ label="SWE-Atlas Test Writing",+ harness=HARNESS_HARBOR,+ # Test writing is not consistently published in the same registry shape as qna/rf.+ dataset=DatasetRef(+ kind=DATASET_PATH,+ repo_url=SWE_ATLAS_REPO_URL,+ repo_ref=SWE_ATLAS_REPO_REF,+ data_dir="data/tw",+ ),+ default_timeout_secs=3300,+ scoring=SCORING_RUBRIC_JUDGE,+ needs_judge=True,+ default_judge="kimi-k2.7-code",+ ),+ "terminal-bench-2.1": Benchmark(+ id="terminal-bench-2.1",+ label="Terminal-Bench 2.1",+ harness=HARNESS_HARBOR,+ dataset=DatasetRef(+ kind=DATASET_REGISTRY, name="terminal-bench/terminal-bench-2-1"+ ),+ default_timeout_secs=3300,+ scoring=SCORING_TESTS,+ needs_judge=False,+ # Air-gapped tasks: the fetch/web-search tools can't reach the network,+ # so disable them (via the agent profile in eval-cli) to stop the agent+ # wasting its budget on tools that can only fail.+ env={"ZED_EVAL_DISABLE_TOOLS": "fetch,search_web"},+ ),+ "deepswe": Benchmark(+ id="deepswe",+ label="DeepSWE",+ harness=HARNESS_PIER,+ dataset=DatasetRef(+ kind=DATASET_PIER_PATH,+ repo_url=DEEPSWE_REPO_URL,+ repo_ref=DEEPSWE_REPO_REF,+ # DeepSWE keeps its Harbor-compatible tasks at the repo root.+ data_dir="tasks",+ ),+ default_timeout_secs=7200,+ scoring=SCORING_TESTS,+ needs_judge=False,+ network_allowlist=AGENT_API_HOSTS,+ # Air-gapped except the model API allowlist; fetch/web-search are useless.+ env={"ZED_EVAL_DISABLE_TOOLS": "fetch,search_web"},+ ),+}+++# Groups expand to multiple benchmarks in one launch.+BENCHMARK_GROUPS: dict[str, tuple[str, ...]] = {+ "swe-atlas": ("swe-atlas-qna", "swe-atlas-rf", "swe-atlas-tw"),+}++# One short alias per benchmark whose canonical id is verbose.+BENCHMARK_ALIASES: dict[str, str] = {+ "qna": "swe-atlas-qna",+ "rf": "swe-atlas-rf",+ "tw": "swe-atlas-tw",+ "tb21": "terminal-bench-2.1",+}++SWE_ATLAS_PART_BENCHMARKS: dict[str, str] = {+ "qna": "swe-atlas-qna",+ "rf": "swe-atlas-rf",+ "tw": "swe-atlas-tw",+}+++def get_benchmark(benchmark_id: str) -> Benchmark:+ try:+ return BENCHMARKS[benchmark_id]+ except KeyError as error:+ valid = ", ".join(sorted(BENCHMARKS))+ raise ValueError(+ f"unknown benchmark '{benchmark_id}' (valid: {valid})"+ ) from error+++def is_benchmark_selector(selector: str) -> bool:+ """Whether `selector` names a known benchmark id, alias, or group.++ Used by `run` to validate benchmark positionals before preparing builds."""+ normalized = selector.strip().lower()+ return (+ normalized in BENCHMARK_GROUPS+ or normalized in BENCHMARKS+ or normalized in BENCHMARK_ALIASES+ )+++def resolve_benchmark_selector(selector: str) -> list[str]:+ """Expands a user-facing selector (benchmark id, alias, or group) into the+ concrete benchmark ids it refers to, preserving order."""+ normalized = selector.strip().lower()+ if normalized in BENCHMARK_GROUPS:+ return list(BENCHMARK_GROUPS[normalized])+ if normalized in BENCHMARKS:+ return [normalized]+ if normalized in BENCHMARK_ALIASES:+ return [BENCHMARK_ALIASES[normalized]]+ valid = ", ".join(sorted({*BENCHMARKS, *BENCHMARK_GROUPS, *BENCHMARK_ALIASES}))+ raise ValueError(f"unknown benchmark '{selector}' (valid: {valid})")+++def resolve_benchmarks(selectors: list[str]) -> list[str]:+ resolved: list[str] = []+ for selector in selectors:+ for part in selector.split(","):+ part = part.strip()+ if not part:+ continue+ for benchmark_id in resolve_benchmark_selector(part):+ if benchmark_id not in resolved:+ resolved.append(benchmark_id)+ return resolved+++def benchmark_metadata(benchmark: Benchmark) -> dict[str, object]:+ """The self-describing block embedded in a run request so the controller and+ harness-command builder need no separate registry lookup."""+ return {+ "id": benchmark.id,+ "label": benchmark.label,+ "harness": benchmark.harness,+ "dataset": {+ "kind": benchmark.dataset.kind,+ "name": benchmark.dataset.name,+ "repo_url": benchmark.dataset.repo_url,+ "repo_ref": benchmark.dataset.repo_ref,+ "data_dir": benchmark.dataset.data_dir,+ },+ "default_timeout_secs": benchmark.default_timeout_secs,+ "scoring": benchmark.scoring,+ "needs_judge": benchmark.needs_judge,+ "default_judge": benchmark.default_judge,+ "network_allowlist": list(benchmark.network_allowlist),+ "env": dict(benchmark.env),+ }diff --git a/crates/eval_cli/zed_eval/builds.py b/crates/eval_cli/zed_eval/builds.pynew file mode 100644index 00000000000..9ce2fdd409d--- /dev/null+++ b/crates/eval_cli/zed_eval/builds.py@@ -0,0 +1,85 @@+from __future__ import annotations++import argparse+from typing import Any++from . import source+++def validate_build_id(build_id: str | None) -> None:+ if build_id and source.sanitize_namespace(build_id) != build_id:+ raise ValueError(+ "build ids may only contain lowercase letters, numbers, '.', '_', and '-'"+ )+++def prepare_build_request(+ *,+ base_sha: str | None,+ patch_path: str | None,+ build_id: str | None,+ allow_untracked: bool,+ require_clean: bool,+ repo_url: str | None,+ clean_source: bool = False,+ source_label: str | None = None,+ pre_resolved_base_sha: str | None = None,+) -> dict[str, Any]:+ validate_build_id(build_id)+ source_info, patch = source.prepare_build_source(+ base_sha=base_sha,+ patch_path=patch_path,+ allow_untracked=allow_untracked,+ require_clean=require_clean,+ repo_url=repo_url,+ clean=clean_source,+ source_label=source_label,+ pre_resolved_base_sha=pre_resolved_base_sha,+ )+ resolved_build_id = build_id or source.compute_build_id(source_info=source_info)+ return {+ "build_id": resolved_build_id,+ "base_sha": source_info["base_sha"],+ "patch": patch,+ "patch_sha256": source_info.get("patch_sha256"),+ "source": source_info,+ "toolchain": source.build_toolchain_info(),+ }+++def resolve_source(+ args: argparse.Namespace,+) -> tuple[str | None, bool, str | None, str | None]:+ """Unify the source selectors into+ `(base_sha, clean_source, source_label, pre_resolved_base_sha)`.++ Precedence: `--from` wins over `--zed-version` / `--base-sha`.++ --from local -> current HEAD + tracked patch (dev iteration)+ --from <ref/tag/sha> -> clean build of that ref, resolved canonically+ against the remote so everyone shares one build+ """+ repo_url = getattr(args, "repo_url", None) or source.DEFAULT_REPO_URL+ from_source = getattr(args, "from_source", None)+ if from_source:+ if from_source.strip().lower() == "local":+ return (+ getattr(args, "base_sha", None),+ bool(getattr(args, "clean_source", False)),+ None,+ None,+ )+ resolved = source.resolve_remote_ref(repo_url, from_source)+ return None, True, from_source, resolved++ zed_version = getattr(args, "zed_version", None)+ if zed_version:+ resolved = source.resolve_remote_ref(repo_url, zed_version)+ return None, True, zed_version, resolved++ return (+ getattr(args, "base_sha", None),+ bool(getattr(args, "clean_source", False)),+ None,+ None,+ )diff --git a/crates/eval_cli/zed_eval/cleanup.py b/crates/eval_cli/zed_eval/cleanup.pynew file mode 100644index 00000000000..03470fe8b60--- /dev/null+++ b/crates/eval_cli/zed_eval/cleanup.py@@ -0,0 +1,93 @@+"""Prune stale build artifacts and build cache from the shared volume.++Eval results are sacrosanct: this only ever touches the build-related top-level+directories listed in `PRUNABLE_DIRS`. `runs/` (the eval outputs) is never in+that set and is never walked, so a bug here cannot delete results.++The logic is pure-filesystem (no Modal dependency) so it can be unit-tested+against a temporary directory; `modal_app` wraps it with volume reload/commit and+a daily schedule.+"""++from __future__ import annotations++import datetime+import shutil+import time+from pathlib import Path+from typing import Any++# The ONLY top-level volume directories cleanup may modify. Eval results live in+# runs/, which is deliberately absent here.+PRUNABLE_DIRS = frozenset({"builds", "build-locks", "tmp"})++DEFAULT_BUILD_RETENTION_DAYS = 14.0+DEFAULT_LOCK_TTL_HOURS = 6.0+DEFAULT_TMP_MAX_AGE_HOURS = 24.0+++def _age_days(path: Path) -> float:+ try:+ return (time.time() - path.stat().st_mtime) / 86400.0+ except OSError:+ return 0.0+++def prune_artifacts(+ data_root: Path | str,+ *,+ dry_run: bool = False,+ build_retention_days: float = DEFAULT_BUILD_RETENTION_DAYS,+ lock_ttl_hours: float = DEFAULT_LOCK_TTL_HOURS,+ tmp_max_age_hours: float = DEFAULT_TMP_MAX_AGE_HOURS,+) -> dict[str, Any]:+ data_root = Path(data_root)+ removed: dict[str, list[Any]] = {+ "builds": [],+ "locks": [],+ "tmp": [],+ }++ # 1. Stale content-addressed builds, aged by their READY marker (or the dir+ # itself for never-finished builds). Cheap to rebuild on demand.+ builds_dir = data_root / "builds"+ if builds_dir.is_dir():+ for build_dir in sorted(builds_dir.iterdir()):+ if not build_dir.is_dir():+ continue+ ready = build_dir / "READY"+ age = _age_days(ready if ready.exists() else build_dir)+ if age > build_retention_days:+ removed["builds"].append(+ {"build_id": build_dir.name, "age_days": round(age, 1)}+ )+ if not dry_run:+ shutil.rmtree(build_dir, ignore_errors=True)++ # 2. Orphaned single-flight build leases left by crashed builds.+ locks_dir = data_root / "build-locks"+ if locks_dir.is_dir():+ for lock in sorted(locks_dir.glob("*.json")):+ if _age_days(lock) * 24.0 > lock_ttl_hours:+ removed["locks"].append(lock.name)+ if not dry_run:+ lock.unlink(missing_ok=True)++ # 3. Orphaned temp build dirs from failed/raced builds.+ tmp_builds = data_root / "tmp" / "builds"+ if tmp_builds.is_dir():+ for entry in sorted(tmp_builds.iterdir()):+ if _age_days(entry) * 24.0 > tmp_max_age_hours:+ removed["tmp"].append(entry.name)+ if not dry_run:+ if entry.is_dir():+ shutil.rmtree(entry, ignore_errors=True)+ else:+ entry.unlink(missing_ok=True)++ return {+ "dry_run": dry_run,+ "at": datetime.datetime.now(datetime.timezone.utc).isoformat(),+ "removed": removed,+ "counts": {key: len(value) for key, value in removed.items()},+ }diff --git a/crates/eval_cli/zed_eval/cli.py b/crates/eval_cli/zed_eval/cli.pynew file mode 100644index 00000000000..e93c701b9c8--- /dev/null+++ b/crates/eval_cli/zed_eval/cli.py@@ -0,0 +1,787 @@+from __future__ import annotations++import argparse+import json+import os+import subprocess+import sys+from pathlib import Path++from . import benchmarks, config, source+from .baseline import (+ command_baseline_list,+ command_baseline_record,+ command_baseline_show,+)+from .common import (+ AppNotDeployedError,+ command_exists,+ default_namespace,+ deploy_app,+ deployed_function,+ print_json,+)+from .launch import command_build, command_rejudge, command_run, command_swe_atlas+from .report import command_report+from .volume import (+ command_builds,+ command_fetch,+ command_list,+ command_logs,+ command_runs,+ command_status,+ command_suite_fetch,+ command_suite_logs,+ command_suite_status,+ ensure_volume_exists,+)+++def modal_secret_names() -> set[str]:+ result = subprocess.run(+ ["modal", "secret", "list", "--json"],+ check=True,+ capture_output=True,+ text=True,+ )+ data = json.loads(result.stdout)+ if not isinstance(data, list):+ return set()+ return {+ item["name"] for item in data if isinstance(item, dict) and item.get("name")+ }+++def command_doctor(args: argparse.Namespace) -> int:+ namespace = default_namespace(args)+ print("zed-eval doctor")+ print(f" namespace: {namespace}")+ print(f" app: {args.app_name}")+ print(f" volume: {args.volume}")+ print(f" api secret: {args.api_secret}")+ print(f" modal token secret: {args.modal_token_secret}")+ print(f" repo root: {source.repo_root()}")+ print(f" base sha: {source.current_base_sha()}")+ print(f" default model: {config.DEFAULT_MODEL}")+ print(" default judges: qna=deepseek-v4-pro, rf/tw=kimi-k2.7-code")++ missing = []+ for executable in ("git", "modal", "harbor"):+ if command_exists(executable):+ print(f" {executable}: found")+ else:+ print(f" {executable}: missing")+ missing.append(executable)++ if missing:+ print("\nMissing required executables: " + ", ".join(missing), file=sys.stderr)+ return 1++ try:+ secrets = modal_secret_names()+ except (subprocess.CalledProcessError, json.JSONDecodeError) as error:+ print(+ f" secrets: could not list Modal secrets: {error}",+ file=sys.stderr,+ )+ else:+ missing_secrets = [+ secret_name+ for secret_name in (args.api_secret, args.modal_token_secret)+ if secret_name not in secrets+ ]+ if missing_secrets:+ print(+ "\nMissing Modal secret(s): " + ", ".join(missing_secrets),+ file=sys.stderr,+ )+ print(+ "Use --api-secret/--modal-token-secret to point at existing secrets, "+ "or create the missing ones with `modal secret create`.",+ file=sys.stderr,+ )+ return 1+ print(" secrets: found")+ print(+ " controller token: use a dedicated Modal service-user token for production"+ )++ if args.create_volume:+ return ensure_volume_exists(args)+ return 0+++def command_deploy(args: argparse.Namespace) -> int:+ deploy_app(args)+ print(f"Deployed Modal app '{args.app_name}'.")+ return 0+++def command_cleanup(args: argparse.Namespace) -> int:+ request: dict[str, object] = {"dry_run": bool(args.dry_run)}+ if args.build_retention_days is not None:+ request["build_retention_days"] = args.build_retention_days+ result = deployed_function(args, "cleanup_artifacts").remote(request)+ print_json(result)+ return 0+++def env_default(+ variable_name: str, default: object, *, include_defaults: bool = True+) -> object:+ if not include_defaults:+ return argparse.SUPPRESS+ return os.environ.get(variable_name, default)+++def add_common_options(+ parser: argparse.ArgumentParser, *, include_defaults: bool = True+) -> None:+ default_volume = env_default(+ "AGENT_EVALS_VOLUME",+ config.DEFAULT_VOLUME_NAME,+ include_defaults=include_defaults,+ )+ default_namespace = None if include_defaults else argparse.SUPPRESS+ default_api_secret = env_default(+ "AGENT_EVALS_LLM_PROVIDERS_SECRET",+ config.DEFAULT_LLM_PROVIDERS_SECRET_NAME,+ include_defaults=include_defaults,+ )+ default_modal_token_secret = env_default(+ "AGENT_EVALS_MODAL_TOKEN_SECRET",+ config.DEFAULT_MODAL_TOKEN_SECRET_NAME,+ include_defaults=include_defaults,+ )+ default_app_name = env_default(+ "AGENT_EVALS_APP_NAME",+ config.DEFAULT_APP_NAME,+ include_defaults=include_defaults,+ )+ infra = parser.add_argument_group(+ "infra options", "Modal app/volume/namespace and secret names (rarely changed)"+ )+ infra.add_argument(+ "--app-name",+ default=default_app_name,+ help=f"Deployed Modal app name (default: {config.DEFAULT_APP_NAME})",+ )+ infra.add_argument(+ "--volume",+ default=default_volume,+ help=f"Modal volume for builds/runs (default: {config.DEFAULT_VOLUME_NAME})",+ )+ infra.add_argument(+ "--namespace",+ default=default_namespace,+ help="Run namespace inside the shared volume (default: env/git user/local user)",+ )+ infra.add_argument(+ "--api-secret",+ default=default_api_secret,+ help=(+ "Modal secret with the LLM provider keys mounted into trial sandboxes "+ f"(default: {config.DEFAULT_LLM_PROVIDERS_SECRET_NAME})"+ ),+ )+ infra.add_argument(+ "--modal-token-secret",+ default=default_modal_token_secret,+ help=(+ "Modal secret mounted into the controller with MODAL_TOKEN_ID and MODAL_TOKEN_SECRET; "+ f"use a dedicated service-user token for production (default: {config.DEFAULT_MODAL_TOKEN_SECRET_NAME})"+ ),+ )+++def add_build_source_options(parser: argparse.ArgumentParser) -> None:+ parser.add_argument(+ "--from",+ dest="from_source",+ default=os.environ.get("AGENT_EVALS_FROM"),+ help=(+ "Unified build source: 'local' for current HEAD + tracked changes, "+ "'main' for origin/main's tip, or a git ref/tag/SHA (e.g. v0.210.0) "+ "for a clean build resolved canonically against the remote so "+ "teammates share one build"+ ),+ )+ parser.add_argument("--base-sha")+ parser.add_argument("--patch-path")+ parser.add_argument(+ "--repo-url",+ default=os.environ.get("AGENT_EVALS_REPO_URL", source.DEFAULT_REPO_URL),+ help="Git remote the Modal builder fetches the base SHA from",+ )+ parser.add_argument(+ "--allow-untracked",+ action="store_true",+ help="Proceed even when untracked files exist; they are not included in the build patch",+ )+ parser.add_argument(+ "--require-clean",+ action="store_true",+ help="Fail if tracked changes are present",+ )+ parser.add_argument(+ "--clean-source",+ action="store_true",+ help="Build exactly --base-sha/--zed-version with no local patch",+ )+ parser.add_argument(+ "--zed-version",+ help="Git ref/tag/SHA of Zed to build as a clean source snapshot",+ )+++def add_model_options(parser: argparse.ArgumentParser) -> None:+ parser.add_argument(+ "-m",+ "--model",+ default=os.environ.get("AGENT_EVALS_MODEL", config.DEFAULT_MODEL),+ help="Base model as provider/model, sonnet-4.6, baseten:kimi-k2.7-code, or baseten:<model-id>",+ )+ parser.add_argument(+ "--extra-api-secret",+ action="append",+ help="Additional Modal secret name to mount into the trial sandbox",+ )+ advanced = parser.add_argument_group(+ "advanced model options",+ "Lower-level model routing. For Baseten, prefer --model baseten:<model-id>.",+ )+ advanced.add_argument(+ "--model-provider",+ choices=("zed", "baseten"),+ default="zed",+ help="Use 'zed' for built-in provider/model ids, or 'baseten' for Baseten Model APIs",+ )+ advanced.add_argument(+ "--baseten-model",+ help="Baseten model id when --model-provider baseten, e.g. moonshotai/Kimi-K2.7-Code",+ )+ advanced.add_argument("--baseten-model-display-name")+ advanced.add_argument(+ "--baseten-api-url",+ default=config.BASETEN_API_URL,+ help=f"Baseten OpenAI-compatible API URL (default: {config.BASETEN_API_URL})",+ )+ advanced.add_argument(+ "--baseten-model-max-tokens",+ type=int,+ default=config.BASETEN_DEFAULT_MAX_TOKENS,+ )+ advanced.add_argument(+ "--baseten-model-max-output-tokens",+ type=int,+ default=config.BASETEN_DEFAULT_MAX_OUTPUT_TOKENS,+ )+ advanced.add_argument(+ "--openai-compatible-provider-json",+ help="JSON object merged into language_models.openai_compatible before eval-cli resolves --model",+ )+ advanced.add_argument(+ "--anthropic-available-models-json",+ help="JSON array merged into language_models.anthropic.available_models before eval-cli resolves --model",+ )+++def add_launch_options(parser: argparse.ArgumentParser) -> None:+ add_model_options(parser)+ parser.add_argument(+ "-j",+ "--judge",+ default=os.environ.get("AGENT_EVALS_JUDGE", config.DEFAULT_JUDGE_PRESET),+ choices=["auto", *sorted(config.JUDGES)],+ help="Judge preset; auto uses qna=deepseek-v4-pro and rf/tw=kimi-k2.7-code",+ )+ parser.add_argument("--judge-model")+ parser.add_argument(+ "--build",+ metavar="ID",+ help="Reuse this build if it exists, otherwise create it with this id",+ )+ add_build_source_options(parser)+ parser.add_argument("--build-wait-timeout-secs", type=int, default=7200)+ parser.add_argument("--tasks", help="File containing one full task name per line")+ parser.add_argument("--include-task-name", action="append")+ parser.add_argument("-n", "--n-tasks", type=int, help="Forward harness --n-tasks")+ parser.add_argument("--n-concurrent", type=int, default=config.DEFAULT_N_CONCURRENT)+ parser.add_argument(+ "--override-cpus", type=int, default=config.DEFAULT_OVERRIDE_CPUS+ )+ parser.add_argument(+ "--override-memory-mb", type=int, default=config.DEFAULT_OVERRIDE_MEMORY_MB+ )+ parser.add_argument(+ "--sandbox-timeout-secs", type=int, default=config.DEFAULT_SANDBOX_TIMEOUT_SECS+ )+ parser.add_argument(+ "--sandbox-idle-timeout-secs",+ type=int,+ default=config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS,+ )+ parser.add_argument(+ "--eval-cli-timeout",+ type=int,+ help="Override the per-task agent timeout (defaults to the benchmark's)",+ )+ parser.add_argument("--run-id")+ parser.add_argument(+ "--dry-run",+ action="store_true",+ help="Print full manifests and harness commands without launching",+ )+ parser.add_argument(+ "--plan",+ action="store_true",+ help="Print a concise launch plan without launching",+ )+ parser.add_argument(+ "--verbose",+ action="store_true",+ help="With --plan, include full manifests and harness commands",+ )+ parser.add_argument(+ "--extra-harbor-arg",+ action="append",+ help="Append one raw argument to the harness run command",+ )+ parser.add_argument(+ "--swe-atlas-repo-url",+ default=os.environ.get(+ "AGENT_EVALS_SWE_ATLAS_REPO_URL", benchmarks.SWE_ATLAS_REPO_URL+ ),+ help="SWE-Atlas repo URL used for path-backed datasets",+ )+ parser.add_argument(+ "--swe-atlas-repo-ref",+ default=os.environ.get(+ "AGENT_EVALS_SWE_ATLAS_REPO_REF", benchmarks.SWE_ATLAS_REPO_REF+ ),+ help="SWE-Atlas repo ref used for path-backed datasets",+ )+++def add_suite_options(parser: argparse.ArgumentParser) -> None:+ add_launch_options(parser)+ parser.add_argument(+ "--parts",+ help="Comma-separated SWE-Atlas parts to run: qna,rf,tw (or 'all')",+ )+ parser.add_argument("--experiment-prefix", default="swe-atlas")+ parser.add_argument("--run-id-prefix")+ parser.add_argument(+ "--suite-id", help="Explicit suite id for grouping multi-part runs"+ )+ parser.add_argument("--interactive", action="store_true")+ parser.add_argument(+ "-y",+ "--yes",+ action="store_true",+ help="Never prompt; require flags/defaults",+ )+++def add_run_lookup_options(+ parser: argparse.ArgumentParser,+ *,+ require_experiment: bool = False,+ run_id_optional: bool = False,+) -> None:+ if run_id_optional:+ parser.add_argument(+ "run_id",+ nargs="?",+ help="Run id (defaults to the most recent run launched from this machine)",+ )+ else:+ parser.add_argument("run_id")+ parser.add_argument(+ "-e",+ "--experiment-name",+ required=require_experiment,+ help="Benchmark storage name (auto-resolved from the local run index when omitted)",+ )+++def add_command_parser(+ subparsers,+ name: str,+ *,+ func: object | None = None,+ **kwargs,+) -> argparse.ArgumentParser:+ parser = subparsers.add_parser(name, **kwargs)+ add_common_options(parser, include_defaults=False)+ if func is not None:+ parser.set_defaults(func=func)+ return parser+++def build_parser() -> argparse.ArgumentParser:+ parser = argparse.ArgumentParser(+ prog="zed-eval",+ description="Launch, monitor, and fetch remote SWE-Atlas agent evals on Modal.",+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog=(+ "Common workflows:\n"+ " zed-eval run swe-atlas -m sonnet-4.6 # launch the SWE-Atlas suite\n"+ " zed-eval run rf -m sonnet-4.6 # launch one benchmark\n"+ " zed-eval runs # what did I launch recently?\n"+ " zed-eval status # check my most recent run\n"+ " zed-eval status <run-id> # check a specific run\n"+ " zed-eval logs <run-id> # print controller logs\n"+ " zed-eval report <run-id> --fetch # fetch + score a run\n"+ "\n"+ "After launching, a run id alone locates the run — namespace and\n"+ "benchmark storage name are resolved from this machine's local run index."+ ),+ )+ add_common_options(parser)+ subparsers = parser.add_subparsers(dest="command", required=True)++ doctor = add_command_parser(+ subparsers,+ "doctor",+ func=command_doctor,+ help="Check local prerequisites and defaults",+ )+ doctor.add_argument("--create-volume", action="store_true")++ add_command_parser(+ subparsers,+ "deploy",+ func=command_deploy,+ help=(+ "Publish harness code to Modal. This is the ONLY command that "+ "deploys; deploying cancels in-flight runs. Run it once after "+ "changing harness code — all other commands just invoke the "+ "already-deployed functions and never deploy."+ ),+ )++ build = add_command_parser(+ subparsers,+ "build",+ func=command_build,+ help="Build eval-cli on Modal into builds/<build-id>",+ )+ add_build_source_options(build)+ build.add_argument(+ "--build",+ metavar="ID",+ help="Reuse this build if it exists, otherwise create it with this id",+ )+ build.add_argument(+ "--detach", action="store_true", help="Spawn the build and return immediately"+ )++ builds = add_command_parser(+ subparsers,+ "builds",+ func=command_builds,+ help="List content-addressed builds on the volume",+ )+ builds.add_argument("--details", action="store_true")+ builds.add_argument("--json", action="store_true")+ builds.add_argument("--limit", type=int, default=50)++ run_cmd = add_command_parser(+ subparsers,+ "run",+ func=command_run,+ help="Run benchmarks — the everyday entry point",+ description=(+ "Launch runs by benchmark id, alias, or group. It picks the right "+ "default judge for each benchmark and never prompts. The sibling "+ "`swe-atlas` command launches SWE-Atlas parts (qna/rf/tw) with "+ "interactive prompts."+ ),+ )+ run_cmd.add_argument(+ "benchmark",+ nargs="+",+ metavar="target",+ help=(+ "Benchmark id/alias/group (swe-atlas (= qna,rf,tw), swe-atlas-rf, "+ "qna, rf, tw, terminal-bench-2.1 (tb21), deepswe). Comma-separated "+ "and repeated values are combined."+ ),+ )+ add_launch_options(run_cmd)++ run_cmd.add_argument(+ "--suite-id", help="Explicit suite id for multi-benchmark runs"+ )+ run_cmd.add_argument(+ "--staff",+ action="store_true",+ help=(+ "Run the agent with staff mode ON (default OFF). Staff mode enables "+ "the sandboxed terminal, which hangs inside Modal sandboxes, so keep "+ "it off for remote runs."+ ),+ )+ run_cmd.add_argument(+ "-y",+ "--yes",+ action="store_true",+ help="Accepted for parity with swe-atlas; run never prompts",+ )++ report = add_command_parser(+ subparsers,+ "report",+ func=command_report,+ help="Success-conditioned metrics (rate, tokens, tool calls, steps)",+ )+ report.add_argument("run_id", nargs="?")+ report.add_argument(+ "-e",+ "--experiment-name",+ help="Benchmark storage name (auto-resolved from the local run index when omitted)",+ )+ report.add_argument("--job-dir", help="Analyze a local job directory directly")+ report.add_argument("--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs"))+ report.add_argument(+ "--fetch", action="store_true", help="Fetch the run archive before reporting"+ )+ report.add_argument(+ "--json",+ dest="as_json",+ action="store_true",+ help="Emit JSON instead of a table",+ )+ report.add_argument(+ "--timeouts-as-failures",+ dest="timeouts_as_failures",+ action=argparse.BooleanOptionalAction,+ default=None,+ help=(+ "Count agent timeouts as failures rather than excluded errors "+ "(default: auto — on for test-scored benchmarks like terminal-bench "+ "and deepswe)"+ ),+ )++ cleanup = add_command_parser(+ subparsers,+ "cleanup",+ func=command_cleanup,+ help="Prune stale builds and cold build cache (never eval results)",+ )+ cleanup.add_argument(+ "--dry-run",+ action="store_true",+ help="Report what would be removed without deleting anything",+ )+ cleanup.add_argument(+ "--build-retention-days",+ type=float,+ help="Remove builds older than this many days (default 14)",+ )++ suite = add_command_parser(+ subparsers,+ "swe-atlas",+ func=command_swe_atlas,+ help="Launch an interactive benchmark suite (SWE-Atlas, Terminal-Bench, DeepSWE)",+ )+ add_suite_options(suite)++ list_runs = add_command_parser(+ subparsers,+ "list",+ func=command_list,+ help="List benchmark runs on the volume, optionally with metadata",+ )+ list_runs.add_argument("-e", "--experiment-name")+ list_runs.add_argument("--details", action="store_true")+ list_runs.add_argument("--json", action="store_true")+ list_runs.add_argument("--limit", type=int, default=50)+ list_runs.add_argument("--all-namespaces", action="store_true")++ runs = add_command_parser(+ subparsers,+ "runs",+ func=command_runs,+ help="List recent runs launched from this machine (local, fast)",+ description=(+ "Show runs recorded in the local run index when you launched them "+ "from this machine — newest first, no network call. Use this for a "+ "quick 'what did I launch lately?'; use `list --details` to query "+ "run state on the volume, or `status <run-id>` for one run's state."+ ),+ )+ runs.add_argument("--json", action="store_true")+ runs.add_argument("--limit", type=int, default=20)++ status = add_command_parser(+ subparsers,+ "status",+ func=command_status,+ help="Print a run's state.json once (no run id = most recent run)",+ )+ add_run_lookup_options(status, run_id_optional=True)++ logs = add_command_parser(+ subparsers,+ "logs",+ func=command_logs,+ help="Print a run's controller.log once (no run id = most recent run)",+ )+ add_run_lookup_options(logs, run_id_optional=True)++ fetch = add_command_parser(+ subparsers,+ "fetch",+ func=command_fetch,+ help="Fetch and extract a run's Harbor job archive",+ )+ add_run_lookup_options(fetch)+ fetch.add_argument("--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs"))++ rejudge = add_command_parser(+ subparsers,+ "rejudge",+ func=command_rejudge,+ help="Re-grade a finished run with a different judge (new derived run, "+ "reuses the agent work)",+ )+ add_run_lookup_options(rejudge)+ rejudge.add_argument(+ "-j",+ "--judge",+ required=True,+ help="Judge preset to re-grade with (see config.JUDGES, e.g. "+ "deepseek-v4-pro, kimi-k2.7-code, leaderboard)",+ )+ rejudge.add_argument(+ "--judge-model",+ dest="judge_model",+ help="Override the judge preset's model id",+ )+ rejudge.add_argument(+ "--parent-namespace",+ dest="parent_namespace",+ help="Namespace of the source run, if different from --namespace",+ )+ rejudge.add_argument(+ "--new-run-id",+ dest="new_run_id",+ help="Explicit id for the derived run (default: <parent>-rejudge-<judge>-<rand>)",+ )+ rejudge.add_argument(+ "--dry-run",+ action="store_true",+ help="Print the rejudge request without spawning the controller",+ )++ suite_group = add_command_parser(+ subparsers,+ "suite",+ help="Inspect or fetch grouped benchmark suite runs",+ )+ suite_subparsers = suite_group.add_subparsers(dest="suite_command", required=True)+ suite_status = add_command_parser(+ suite_subparsers,+ "status",+ func=command_suite_status,+ help="Show status for each run in a suite",+ )+ suite_status.add_argument("suite_id")+ suite_status.add_argument("--json", action="store_true")+ suite_logs = add_command_parser(+ suite_subparsers,+ "logs",+ func=command_suite_logs,+ help="Print logs for each run in a suite",+ )+ suite_logs.add_argument("suite_id")+ suite_logs.add_argument("--follow", action="store_true")+ suite_logs.add_argument("--interval", type=float, default=30.0)+ suite_fetch = add_command_parser(+ suite_subparsers,+ "fetch",+ func=command_suite_fetch,+ help="Fetch all run archives in a suite",+ )+ suite_fetch.add_argument("suite_id")+ suite_fetch.add_argument(+ "--jobs-dir", default=str(Path.home() / ".cache/harbor/jobs")+ )++ baseline_group = add_command_parser(+ subparsers,+ "baseline",+ help="Record and inspect baseline-of-record results (clean commits on main)",+ )+ baseline_subparsers = baseline_group.add_subparsers(+ dest="baseline_command", required=True+ )+ baseline_record = add_command_parser(+ baseline_subparsers,+ "record",+ func=command_baseline_record,+ help="Promote completed run(s) to the baseline of record for their (benchmark, model)",+ )+ baseline_record.add_argument("run_id", nargs="+")+ baseline_record.add_argument("--experiment-name", required=True)+ baseline_record.add_argument(+ "--allow-dirty",+ action="store_true",+ help="Record even though the build carries a local patch (not a clean commit)",+ )+ baseline_record.add_argument(+ "--allow-off-main",+ action="store_true",+ help="Record even though base_sha can't be verified as reachable from origin/main",+ )+ baseline_record.add_argument(+ "--repo-url",+ default=os.environ.get("AGENT_EVALS_REPO_URL"),+ help="Git remote to resolve origin/main against (default: AGENT_EVALS_REPO_URL or the canonical repo)",+ )++ baseline_list = add_command_parser(+ baseline_subparsers,+ "list",+ func=command_baseline_list,+ help="List the current baseline of record for every (benchmark, model)",+ )+ baseline_list.add_argument("--json", action="store_true")++ baseline_show = add_command_parser(+ baseline_subparsers,+ "show",+ func=command_baseline_show,+ help="Show the baseline record for one (benchmark, model)",+ )+ baseline_show.add_argument("experiment_name")+ baseline_show.add_argument("--model", required=True)+ baseline_show.add_argument(+ "--history", action="store_true", help="Include superseded baselines"+ )++ return parser+++def main(argv: list[str] | None = None) -> int:+ parser = build_parser()+ args = parser.parse_args(argv)+ try:+ return args.func(args)+ except (ValueError, AppNotDeployedError) as error:+ print(f"error: {error}", file=sys.stderr)+ return 1+ except subprocess.CalledProcessError as error:+ if error.stdout:+ print(error.stdout, end="")+ if error.stderr:+ print(error.stderr, end="", file=sys.stderr)+ return error.returncode+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/crates/eval_cli/zed_eval/common.py b/crates/eval_cli/zed_eval/common.pynew file mode 100644index 00000000000..7ae6083231d--- /dev/null+++ b/crates/eval_cli/zed_eval/common.py@@ -0,0 +1,220 @@+from __future__ import annotations++import argparse+import datetime+import json+import os+import pathlib+import shutil+import subprocess+import tarfile+from typing import Any++from . import config, source++ALL_SWE_ATLAS_PARTS = ["qna", "rf", "tw"]+++def utc_timestamp() -> str:+ return datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")+++def utc_now() -> str:+ return datetime.datetime.now(datetime.timezone.utc).isoformat()+++def print_json(data: Any) -> None:+ print(json.dumps(data, indent=2, sort_keys=True))+++def write_json(path: pathlib.Path, data: dict[str, Any]) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n")+++def load_json(path: pathlib.Path) -> dict[str, Any] | None:+ try:+ data = json.loads(path.read_text())+ except (OSError, json.JSONDecodeError):+ return None+ return data if isinstance(data, dict) else None+++def safe_extract_archive(archive: tarfile.TarFile, destination: pathlib.Path) -> None:+ destination = destination.resolve()+ members = archive.getmembers()+ for member in members:+ if member.issym() or member.islnk():+ raise ValueError(f"archive links are not supported: {member.name}")+ target = (destination / member.name).resolve()+ if destination != target and destination not in target.parents:+ raise ValueError(f"archive member escapes destination: {member.name}")+ archive.extractall(destination, members=members)+++def command_exists(name: str) -> bool:+ return shutil.which(name) is not None+++def run_command(+ command: list[str], *, capture: bool = False+) -> subprocess.CompletedProcess[str]:+ return subprocess.run(command, check=True, text=True, capture_output=capture)+++def dedupe_preserving_order(values: list[str]) -> list[str]:+ seen = set()+ result = []+ for value in values:+ if value in seen:+ continue+ seen.add(value)+ result.append(value)+ return result+++def parse_comma_values(values: list[str] | None) -> list[str]:+ result = []+ for value in values or []:+ result.extend(part.strip() for part in value.split(",") if part.strip())+ return result+++def parse_parts(values: list[str] | None) -> list[str]:+ requested = parse_comma_values(values)+ if any(value.lower() == "all" for value in requested):+ return list(ALL_SWE_ATLAS_PARTS)+ return dedupe_preserving_order(+ [config.canonical_part(value) for value in requested]+ )+++def configure_modal_environment(args: argparse.Namespace) -> None:+ os.environ["AGENT_EVALS_APP_NAME"] = args.app_name+ os.environ["AGENT_EVALS_VOLUME"] = args.volume+ os.environ["AGENT_EVALS_MODAL_TOKEN_SECRET"] = args.modal_token_secret+ os.environ["AGENT_EVALS_LLM_PROVIDERS_SECRET"] = args.api_secret+++def import_modal_app(args: argparse.Namespace):+ configure_modal_environment(args)+ from . import modal_app++ return modal_app+++class AppNotDeployedError(RuntimeError):+ """Raised when a Modal function lookup fails because the app (or that+ specific function) has not been deployed. Surfaced as a clean CLI error+ pointing the user at `zed-eval deploy`.+ """+++def deploy_app(args: argparse.Namespace) -> None:+ """Publish the Modal app. This is the ONLY code path that deploys.++ Deploying replaces the live app and cancels any in-flight eval runs, so it+ must happen exclusively via the `deploy` subcommand. Every other command+ just looks up already-deployed functions with `deployed_function`. After+ changing harness code, run `zed-eval deploy` once.+ """+ modal_app = import_modal_app(args)+ print(f"Deploying Modal app '{args.app_name}'...")+ modal_app.app.deploy(name=args.app_name)+++def deployed_function(args: argparse.Namespace, function_name: str):+ """Look up an already-deployed Modal function. Never deploys.++ `modal.Function.from_name` resolves a published function without deploying,+ which is exactly what we want: deploying would cancel in-flight runs. We+ hydrate eagerly so a missing app/function fails here with an actionable+ message instead of deep inside a later `.spawn()`/`.remote()` call.+ """+ import modal+ from modal.exception import NotFoundError++ configure_modal_environment(args)+ function = modal.Function.from_name(args.app_name, function_name)+ try:+ function.hydrate()+ except NotFoundError as error:+ raise AppNotDeployedError(+ f"app '{args.app_name}' not deployed (or function "+ f"'{function_name}' missing) — run 'zed-eval deploy' first"+ ) from error+ return function+++def modal_call_id(call: Any) -> str:+ for attribute in ("object_id", "id", "function_call_id"):+ value = getattr(call, attribute, None)+ if value:+ return str(value)+ return str(call)+++def print_table(rows: list[dict[str, Any]], columns: list[tuple[str, str]]) -> None:+ if not rows:+ print("No rows")+ return+ widths = []+ for key, title in columns:+ width = len(title)+ for row in rows:+ width = max(width, len(str(row.get(key) or "")))+ widths.append(min(width, 80))+ print(+ " ".join(+ title.ljust(widths[index]) for index, (_key, title) in enumerate(columns)+ )+ )+ print(" ".join("-" * width for width in widths))+ for row in rows:+ cells = []+ for index, (key, _title) in enumerate(columns):+ text = str(row.get(key) or "")+ if len(text) > widths[index]:+ text = text[: widths[index] - 1] + "…"+ cells.append(text.ljust(widths[index]))+ print(" ".join(cells))+++def default_namespace(args: argparse.Namespace) -> str:+ return source.sanitize_namespace(args.namespace or source.default_namespace())+++def resolve_run_location(args: argparse.Namespace, run_id: str) -> tuple[str, str]:+ """Resolve (namespace, experiment_name) for a run-lookup command.++ When `--experiment-name` is given, behaves exactly as before: namespace comes+ from `--namespace` or the local git default. When it is omitted, the run is+ located via the launch-time local run index (`run_index`) so a bare run id is+ enough. Raises a friendly error pointing at the explicit flags when the run+ can't be found.+ """+ from . import run_index++ explicit_experiment = getattr(args, "experiment_name", None)+ explicit_namespace = getattr(args, "namespace", None)+ if explicit_experiment:+ namespace = explicit_namespace or source.default_namespace()+ return (+ source.sanitize_namespace(namespace),+ source.sanitize_namespace(explicit_experiment),+ )+ entry = run_index.lookup(run_id)+ if not entry:+ raise ValueError(+ f"could not locate run '{run_id}' in the local run index "+ f"({run_index.index_path()}). Pass --experiment-name (and "+ "--namespace if it isn't yours). The index is written automatically "+ "when you launch a run from this machine."+ )+ namespace = (+ explicit_namespace or entry.get("namespace") or source.default_namespace()+ )+ return (+ source.sanitize_namespace(namespace),+ source.sanitize_namespace(entry["experiment_name"]),+ )diff --git a/crates/eval_cli/zed_eval/config.py b/crates/eval_cli/zed_eval/config.pynew file mode 100644index 00000000000..94112db6b47--- /dev/null+++ b/crates/eval_cli/zed_eval/config.py@@ -0,0 +1,164 @@+from __future__ import annotations++import shlex+from dataclasses import dataclass++DEFAULT_APP_NAME = "agent-evals"+DEFAULT_VOLUME_NAME = "agent-evals"+# Modal secret holding the LLM provider keys (ANTHROPIC_API_KEY, BASETEN_API_KEY,+# ...) mounted into trial sandboxes.+DEFAULT_LLM_PROVIDERS_SECRET_NAME = "agent-evals-llm-providers"+# Modal secret holding the controller's Modal token (MODAL_TOKEN_ID/SECRET).+DEFAULT_MODAL_TOKEN_SECRET_NAME = "agent-evals-modal-token"+HARBOR_VERSION = "0.15.0"+MODAL_VERSION = "1.5.0"+CONTROLLER_IMAGE_RECIPE_VERSION = "agent-evals-controller-image-v1"+DEFAULT_MODEL = "anthropic/claude-sonnet-4-6"+DEFAULT_JUDGE_PRESET = "auto"+BASETEN_API_URL = "https://inference.baseten.co/v1"+BASETEN_PROVIDER_ID = "baseten"+BASETEN_DEFAULT_MAX_TOKENS = 262_144+BASETEN_DEFAULT_MAX_OUTPUT_TOKENS = 65_536++# Kept modest on purpose: the agent model's API key is the real bottleneck,+# not Modal capacity. Snickerdoodle-EAP is capped at 2M ITPM / 400K OTPM, and+# ~150 concurrent agent loops (3 jobs x 50) blew past that. One job at this+# concurrency stays well under the ceiling; raise only if you've confirmed the+# model's rate limits have headroom.+DEFAULT_N_CONCURRENT = 25+DEFAULT_SANDBOX_TIMEOUT_SECS = 14_400+DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS = 300+# No per-task resource override by default. When these are None the+# --override-cpus/--override-memory-mb flags are omitted entirely, so Harbor+# applies each task's declared cpus/memory (SWE-Atlas tasks declare 16 CPU /+# 16 GB). Overriding *below* the declared values OOM-kills memory-heavy tasks+# (SIGKILL / exit 137) and, per the resource policy, changes the experimental+# conditions. Pass --override-cpus/--override-memory-mb to opt into smaller+# sandboxes deliberately.+DEFAULT_OVERRIDE_CPUS = None+DEFAULT_OVERRIDE_MEMORY_MB = None+EVAL_BASE_URL_IN_SANDBOX = "http://127.0.0.1:8089/v1"+JUDGE_PROXY_VERIFIER_IMPORT_PATH = "zed_eval.verifier:ZedJudgeProxyVerifier"+++@dataclass(frozen=True)+class JudgeConfig:+ model: str+ upstream: str+ auth_env: str+ # Floor for the judge's max_completion_tokens, applied by the in-sandbox+ # judge shim (ZED_JUDGE_MAX_TOKENS). Set for Baseten judges so the runtime+ # judge matches the offline calibration (8192); left None for the+ # opus/leaderboard path so it stays leaderboard-faithful at the verifier's+ # hardcoded 2048.+ max_tokens: int | None = None+++def judge_verifier_args(judge: JudgeConfig, judge_model: str) -> list[str]:+ args = [+ "--verifier-import-path",+ JUDGE_PROXY_VERIFIER_IMPORT_PATH,+ "--ve",+ f"EVAL_MODEL={judge_model}",+ "--ve",+ f"EVAL_BASE_URL={EVAL_BASE_URL_IN_SANDBOX}",+ "--ve",+ "EVAL_API_KEY=unused-by-agent-evals-proxy",+ "--ve",+ f"ZED_JUDGE_UPSTREAM={judge.upstream}",+ "--ve",+ f"ZED_JUDGE_AUTH_ENV={judge.auth_env}",+ ]+ if judge.max_tokens is not None:+ args.extend(["--ve", f"ZED_JUDGE_MAX_TOKENS={judge.max_tokens}"])+ return args+++def redacted_command(command: list[str]) -> str:+ return shlex.join(+ "EVAL_API_KEY=<redacted>" if argument.startswith("EVAL_API_KEY=") else argument+ for argument in command+ )+++JUDGES: dict[str, JudgeConfig] = {+ "leaderboard": JudgeConfig(+ model="claude-opus-4-5-20251101",+ upstream="https://api.anthropic.com/v1",+ auth_env="ANTHROPIC_API_KEY",+ ),+ "opus-leaderboard": JudgeConfig(+ model="claude-opus-4-5-20251101",+ upstream="https://api.anthropic.com/v1",+ auth_env="ANTHROPIC_API_KEY",+ ),+ "deepseek-v4-pro": JudgeConfig(+ model="deepseek-ai/DeepSeek-V4-Pro",+ upstream=BASETEN_API_URL,+ auth_env="BASETEN_API_KEY",+ max_tokens=8192,+ ),+ "deepseek": JudgeConfig(+ model="deepseek-ai/DeepSeek-V4-Pro",+ upstream=BASETEN_API_URL,+ auth_env="BASETEN_API_KEY",+ max_tokens=8192,+ ),+ "kimi-k2.7-code": JudgeConfig(+ model="moonshotai/Kimi-K2.7-Code",+ upstream=BASETEN_API_URL,+ auth_env="BASETEN_API_KEY",+ max_tokens=8192,+ ),+ "kimi": JudgeConfig(+ model="moonshotai/Kimi-K2.7-Code",+ upstream=BASETEN_API_URL,+ auth_env="BASETEN_API_KEY",+ max_tokens=8192,+ ),+ "gpt55": JudgeConfig(+ model="gpt-5.5-2026-04-23",+ upstream="https://api.openai.com/v1",+ auth_env="OPENAI_API_KEY",+ ),+}+++MODEL_PRESETS: dict[str, str] = {+ "sonnet-4.6": DEFAULT_MODEL,+ "claude-sonnet-4.6": DEFAULT_MODEL,+ "sonnet-4.6-latest": "anthropic/claude-sonnet-4-6-latest",+ "opus-4.5": "anthropic/claude-opus-4-5",+ "baseten:kimi-k2.7-code": "baseten/moonshotai/Kimi-K2.7-Code",+ "baseten:kimi": "baseten/moonshotai/Kimi-K2.7-Code",+ "baseten:deepseek-v4-pro": "baseten/deepseek-ai/DeepSeek-V4-Pro",+ "baseten:deepseek": "baseten/deepseek-ai/DeepSeek-V4-Pro",+}+++def orchestration_info() -> dict[str, str]:+ return {+ "controller_image_recipe_version": CONTROLLER_IMAGE_RECIPE_VERSION,+ "harbor_version": HARBOR_VERSION,+ "modal_client_version": MODAL_VERSION,+ }+++def get_judge(name: str) -> JudgeConfig:+ try:+ return JUDGES[name]+ except KeyError as error:+ valid = ", ".join(sorted(JUDGES))+ raise ValueError(f"unknown judge preset '{name}' (valid: {valid})") from error+++def canonical_part(part: str) -> str:+ normalized = part.strip().lower()+ if normalized in ("qna", "rf", "tw"):+ return normalized+ valid = ", ".join(("qna", "rf", "tw"))+ raise ValueError(f"unknown SWE-Atlas part '{part}' (valid: {valid})")+++def resolve_model_preset(model: str) -> str:+ return MODEL_PRESETS.get(model, model)diff --git a/crates/eval_cli/zed_eval/harness_command.py b/crates/eval_cli/zed_eval/harness_command.pynew file mode 100644index 00000000000..f6024d534f9--- /dev/null+++ b/crates/eval_cli/zed_eval/harness_command.py@@ -0,0 +1,220 @@+"""Build the harness command (Harbor or Pier) for a benchmark run request.++The command is driven by the self-describing `benchmark` block embedded in a run+request (see `benchmarks.benchmark_metadata`), so it works for any registered+benchmark without a separate experiment registry lookup.+"""++from __future__ import annotations++import json+from typing import Any++from . import benchmarks, config+++def _benchmark_block(run_request: dict[str, Any]) -> dict[str, Any]:+ block = run_request.get("benchmark")+ if not isinstance(block, dict):+ raise ValueError("run request is missing a 'benchmark' block")+ return block+++def dataset_args(benchmark: dict[str, Any]) -> list[str]:+ dataset = benchmark.get("dataset") or {}+ kind = dataset.get("kind")+ if kind == benchmarks.DATASET_REGISTRY:+ name = dataset.get("name")+ if not name:+ raise ValueError("registry dataset requires a name")+ return ["-d", name]+ if kind in (benchmarks.DATASET_PATH, benchmarks.DATASET_PIER_PATH):+ data_dir = dataset.get("data_dir")+ if not data_dir:+ raise ValueError("path dataset requires data_dir")+ return ["-p", dataset_path(benchmark)]+ raise ValueError(f"unsupported dataset kind: {kind}")+++def dataset_clone_dir(benchmark: dict[str, Any]) -> str:+ """Where the controller clones the dataset repo for path datasets."""+ return f"/tmp/datasets/{benchmark['id']}"+++def dataset_path(benchmark: dict[str, Any]) -> str:+ """Repo-relative task directory inside the cloned dataset repo."""+ dataset = benchmark.get("dataset") or {}+ return f"{dataset_clone_dir(benchmark)}/{dataset.get('data_dir')}"+++def harness_binary(benchmark: dict[str, Any]) -> str:+ harness = benchmark.get("harness")+ if harness not in (benchmarks.HARNESS_HARBOR, benchmarks.HARNESS_PIER):+ raise ValueError(f"unsupported harness: {harness}")+ return harness+++def eval_cli_timeout(run_request: dict[str, Any], benchmark: dict[str, Any]) -> int:+ return int(+ run_request.get("eval_cli_timeout")+ or benchmark.get("default_timeout_secs")+ or config.DEFAULT_SANDBOX_TIMEOUT_SECS+ )+++def build_harness_command(run_request: dict[str, Any], jobs_dir: str) -> list[str]:+ benchmark = _benchmark_block(run_request)+ build_id = run_request.get("build_id")+ if not build_id:+ raise ValueError("zed benchmarks require build_id")+ volume_name = run_request["volume_name"]+ api_secret_name = run_request["api_secret_name"]+ run_id = run_request["run_id"]+ agent_model = run_request.get("agent_model") or config.DEFAULT_MODEL+ n_concurrent = int(run_request.get("n_concurrent") or config.DEFAULT_N_CONCURRENT)+ sandbox_timeout_secs = int(+ run_request.get("sandbox_timeout_secs") or config.DEFAULT_SANDBOX_TIMEOUT_SECS+ )+ sandbox_idle_timeout_secs = int(+ run_request.get("sandbox_idle_timeout_secs")+ or config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS+ )+ override_cpus = run_request.get("override_cpus") or config.DEFAULT_OVERRIDE_CPUS+ override_memory_mb = (+ run_request.get("override_memory_mb") or config.DEFAULT_OVERRIDE_MEMORY_MB+ )++ secret_names = [api_secret_name]+ for extra_secret_name in run_request.get("extra_api_secrets") or []:+ if extra_secret_name not in secret_names:+ secret_names.append(extra_secret_name)++ command = [+ harness_binary(benchmark),+ "run",+ *dataset_args(benchmark),+ "-m",+ agent_model,+ "--env",+ "modal",+ "--n-concurrent",+ str(n_concurrent),+ "--job-name",+ run_id,+ "--jobs-dir",+ jobs_dir,+ "--ek",+ f"volumes={json.dumps({'/data': volume_name})}",+ "--ek",+ f"secrets={json.dumps(secret_names)}",+ "--ek",+ f"sandbox_timeout_secs={sandbox_timeout_secs}",+ "--ek",+ f"sandbox_idle_timeout_secs={sandbox_idle_timeout_secs}",+ # Harbor and Pier have different installed-agent interfaces, so each gets+ # its own thin agent shell (the sanctioned cross-framework exception).+ "--agent-import-path",+ (+ "zed_eval.pier_agent:ZedPierAgent"+ if harness_binary(benchmark) == benchmarks.HARNESS_PIER+ else "zed_eval.agent:ZedAgent"+ ),+ "--ae",+ f"EVAL_CLI_CONTAINER_PATH=/data/builds/{build_id}/eval-cli",+ "--ae",+ f"EVAL_CLI_TIMEOUT={eval_cli_timeout(run_request, benchmark)}",+ ]+ # Omit --override-cpus/--override-memory-mb unless explicitly set, so Harbor+ # applies each task's declared cpus/memory. Overriding below the declared+ # values OOM-kills memory-heavy tasks (SIGKILL / exit 137).+ if override_cpus is not None:+ command += ["--override-cpus", str(int(override_cpus))]+ if override_memory_mb is not None:+ command += ["--override-memory-mb", str(int(override_memory_mb))]++ openai_compatible_provider_json = run_request.get("openai_compatible_provider_json")+ if openai_compatible_provider_json:+ command.extend(+ [+ "--ae",+ f"ZED_OPENAI_COMPATIBLE_PROVIDERS={openai_compatible_provider_json}",+ ]+ )+ anthropic_available_models_json = run_request.get("anthropic_available_models_json")+ if anthropic_available_models_json:+ command.extend(+ [+ "--ae",+ f"ZED_ANTHROPIC_AVAILABLE_MODELS={anthropic_available_models_json}",+ ]+ )++ # NOTE: Pier's network allowlist for air-gapped DeepSWE tasks is declared by+ # the agent (`network_allowlist()` on a Pier-native agent class), not via a+ # CLI flag — see benchmarks.AGENT_API_HOSTS and the DeepSWE agent. There is+ # no `--agent-allow-host` option on `pier run`.+ if benchmark.get("needs_judge"):+ judge = config.get_judge(run_request["judge_preset"])+ judge_model = run_request.get("judge_model") or judge.model+ command.extend(config.judge_verifier_args(judge, judge_model))++ for key, value in (benchmark.get("env") or {}).items():+ command.extend(["--ae", f"{key}={value}"])+ for key, value in (run_request.get("extra_env") or {}).items():+ command.extend(["--ae", f"{key}={value}"])++ for task_name in run_request.get("task_names") or []:+ command.extend(["--include-task-name", task_name])++ n_tasks = run_request.get("n_tasks")+ if n_tasks is not None:+ command.extend(["--n-tasks", str(n_tasks)])++ command.extend(run_request.get("extra_harbor_args") or [])+ return command+++def run_metadata(run_request: dict[str, Any]) -> dict[str, Any]:+ benchmark = _benchmark_block(run_request)+ metadata: dict[str, Any] = {+ "run_id": run_request["run_id"],+ "namespace": run_request["namespace"],+ "benchmark": benchmark,+ "suite_id": run_request.get("suite_id"),+ "harness": benchmark.get("harness"),+ "scoring": benchmark.get("scoring"),+ "build_id": run_request.get("build_id"),+ "agent_model": run_request.get("agent_model") or config.DEFAULT_MODEL,+ "orchestration": run_request.get("orchestration")+ or config.orchestration_info(),+ "build_toolchain": run_request.get("build_toolchain"),+ "volume_name": run_request["volume_name"],+ "api_secret_name": run_request["api_secret_name"],+ "extra_api_secrets": run_request.get("extra_api_secrets") or [],+ "openai_compatible_provider_json": run_request.get(+ "openai_compatible_provider_json"+ ),+ "anthropic_available_models_json": run_request.get(+ "anthropic_available_models_json"+ ),+ "n_concurrent": run_request.get("n_concurrent") or config.DEFAULT_N_CONCURRENT,+ "override_cpus": run_request.get("override_cpus")+ or config.DEFAULT_OVERRIDE_CPUS,+ "override_memory_mb": run_request.get("override_memory_mb")+ or config.DEFAULT_OVERRIDE_MEMORY_MB,+ "eval_cli_timeout": eval_cli_timeout(run_request, benchmark),+ "sandbox_timeout_secs": run_request.get("sandbox_timeout_secs")+ or config.DEFAULT_SANDBOX_TIMEOUT_SECS,+ "sandbox_idle_timeout_secs": run_request.get("sandbox_idle_timeout_secs")+ or config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS,+ "task_count": len(run_request.get("task_names") or []),+ "n_tasks": run_request.get("n_tasks"),+ }+ if benchmark.get("needs_judge"):+ judge = config.get_judge(run_request["judge_preset"])+ metadata["judge_preset"] = run_request["judge_preset"]+ metadata["judge_model"] = run_request.get("judge_model") or judge.model+ metadata["judge_upstream"] = judge.upstream+ metadata["judge_auth_env"] = judge.auth_env+ metadata["judge_eval_base_url"] = config.EVAL_BASE_URL_IN_SANDBOX+ return metadatadiff --git a/crates/eval_cli/zed_eval/interactive.py b/crates/eval_cli/zed_eval/interactive.pynew file mode 100644index 00000000000..1829ab1e8b6--- /dev/null+++ b/crates/eval_cli/zed_eval/interactive.py@@ -0,0 +1,163 @@+from __future__ import annotations++import argparse+import sys++from .common import dedupe_preserving_order, deployed_function++INTERACTIVE_BENCHMARK_CHOICES = [+ "qna",+ "rf",+ "tw",+ "terminal-bench-2.1",+ "deepswe",+]+INTERACTIVE_BENCHMARK_ALIASES = {+ "terminal-bench": "terminal-bench-2.1",+ "tb21": "terminal-bench-2.1",+ "deep-swe": "deepswe",+}+INTERACTIVE_MODEL_CHOICES = [+ "sonnet-4.6",+ "opus-4.5",+ "baseten:kimi-k2.7-code",+ "baseten:deepseek-v4-pro",+ "custom-zed-model",+ "custom-baseten-model",+]+++def prompt(label: str, default: str | None = None) -> str:+ suffix = f" [{default}]" if default else ""+ value = input(f"{label}{suffix}: ").strip()+ return value or (default or "")+++def prompt_required(label: str) -> str:+ while True:+ value = prompt(label)+ if value:+ return value+ print("Enter a value", file=sys.stderr)+++def prompt_choice(label: str, choices: list[str], default: str) -> str:+ print(label)+ for index, choice in enumerate(choices, start=1):+ marker = " (default)" if choice == default else ""+ print(f" {index}. {choice}{marker}")+ while True:+ value = prompt("Choose", default)+ if value in choices:+ return value+ if value.isdigit():+ index = int(value) - 1+ if 0 <= index < len(choices):+ return choices[index]+ print(f"Enter one of: {', '.join(choices)}", file=sys.stderr)+++def prompt_multi(+ label: str,+ choices: list[str],+ default: list[str],+ aliases: dict[str, str] | None = None,+) -> list[str]:+ default_text = ",".join(default)+ canonical_choices = {choice.lower(): choice for choice in choices}+ aliases = aliases or {}+ print(label)+ for choice in choices:+ print(f" - {choice}")+ while True:+ value = prompt("Comma-separated choices, or all", default_text)+ selections = []+ for selection in (part.strip().lower() for part in value.split(",")):+ if not selection:+ continue+ if selection == "all":+ return list(choices)+ canonical = aliases.get(selection) or canonical_choices.get(selection)+ if canonical is None:+ valid = ", ".join([*choices, *sorted(aliases)])+ print(f"unknown choice '{selection}' (valid: {valid})", file=sys.stderr)+ break+ selections.append(canonical)+ else:+ selections = dedupe_preserving_order(selections)+ if selections:+ return selections+ print("Choose at least one benchmark", file=sys.stderr)+++def should_prompt(args: argparse.Namespace) -> bool:+ if getattr(args, "interactive", False):+ return True+ if getattr(args, "yes", False):+ return False+ return sys.stdin.isatty() and not getattr(args, "parts", None)+++def choose_existing_build(args: argparse.Namespace) -> str:+ try:+ rows = deployed_function(args, "list_builds").remote(20)+ except Exception as error:+ print(+ f"Could not list builds ({error}); falling back to manual entry.",+ file=sys.stderr,+ )+ return prompt("Existing build id")+ if not rows:+ return prompt("No builds found; enter build id")+ print("Existing builds")+ for index, row in enumerate(rows, start=1):+ build_id = row.get("build_id")+ base = str(row.get("base_sha") or "")[:12]+ patch = "dirty" if row.get("patch_sha256") else "clean"+ ready = "ready" if row.get("ready") else "not-ready"+ print(f" {index}. {build_id} {base} {patch} {ready}")+ while True:+ value = prompt("Choose build number or id", "1")+ if value.isdigit():+ index = int(value) - 1+ if 0 <= index < len(rows):+ return rows[index]["build_id"]+ if value:+ return value+++def configure_interactive_suite(args: argparse.Namespace) -> None:+ if not should_prompt(args):+ return++ args.benchmark = prompt_multi(+ "Which benchmarks should run?",+ INTERACTIVE_BENCHMARK_CHOICES,+ ["qna", "rf"],+ aliases=INTERACTIVE_BENCHMARK_ALIASES,+ )++ if not getattr(args, "build", None):+ build_choice = prompt_choice(+ "Build selection",+ ["auto", "existing-build"],+ "auto",+ )+ if build_choice == "existing-build":+ args.build = choose_existing_build(args)++ model_choice = prompt_choice("Base model", INTERACTIVE_MODEL_CHOICES, "sonnet-4.6")+ if model_choice == "custom-zed-model":+ args.model = prompt("Zed model id (provider/model)", args.model)+ elif model_choice == "custom-baseten-model":+ args.model_provider = "baseten"+ args.baseten_model = prompt_required("Baseten model id")+ else:+ args.model = model_choice++ judge_choice = prompt_choice(+ "Judge preset",+ ["auto", "leaderboard", "deepseek-v4-pro", "kimi-k2.7-code", "gpt55"],+ args.judge or "auto",+ )+ args.judge = judge_choicediff --git a/crates/eval_cli/zed_eval/judge_proxy.py b/crates/eval_cli/zed_eval/judge_proxy.pynew file mode 100644index 00000000000..0a8066b4f77--- /dev/null+++ b/crates/eval_cli/zed_eval/judge_proxy.py@@ -0,0 +1,104 @@+"""The in-sandbox judge proxy shim, shared by the Harbor verifier and the+offline rejudge path so judge calls go through one identical code path+everywhere.++Stdlib-only and free of any `harbor`/`modal` imports on purpose: `verifier.py`+imports these constants for the live in-sandbox install, and `rejudge.py` runs+the very same proxy in a plain container to re-grade stored runs. Keeping the+shim here means there is exactly one implementation of the upstream fixups+(Anthropic `response_format` stripping, `max_tokens` -> `max_completion_tokens`,+the `ZED_JUDGE_MAX_TOKENS` floor) rather than a copy per call site.+"""++from __future__ import annotations++# Stdlib-only OpenAI-protocol shim. The SWE-Atlas RF judge hardcodes+# `response_format={"type": "json_object"}`, which Anthropic's OpenAI-compatible+# endpoint rejects (400: "response_format.type: Input should be 'json_schema'").+# The shim listens on 127.0.0.1:PORT and forwards to ZED_JUDGE_UPSTREAM, applying+# upstream-specific fixups:+# - `response_format` is stripped ONLY for Anthropic (Baseten/OpenAI accept+# json_object, and the Kimi/DeepSeek judges rely on it);+# - `max_tokens` is renamed to `max_completion_tokens` (accepted by both+# Anthropic-compat and OpenAI, including GPT-5.x reasoning models that reject+# `max_tokens`) and raised to the ZED_JUDGE_MAX_TOKENS floor so the runtime+# judge matches the offline-calibrated token budget.+# When ZED_JUDGE_AUTH_ENV is set, it replaces verifier auth with the named+# sandbox env var; otherwise it passes auth headers through unchanged.+JUDGE_PROXY_SCRIPT = """\+import json, os, urllib.request, urllib.error+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++UPSTREAM = os.environ.get("ZED_JUDGE_UPSTREAM", "https://api.anthropic.com/v1").rstrip("/")+PORT = int(os.environ.get("ZED_JUDGE_PROXY_PORT", "8089"))+AUTH_ENV = os.environ.get("ZED_JUDGE_AUTH_ENV")+MAX_TOKENS_FLOOR = int(os.environ.get("ZED_JUDGE_MAX_TOKENS", "0"))++class Handler(BaseHTTPRequestHandler):+ def do_POST(self):+ length = int(self.headers.get("Content-Length", 0))+ body = self.rfile.read(length)+ try:+ payload = json.loads(body)+ if "anthropic" in UPSTREAM:+ payload.pop("response_format", None)+ sent = payload.pop("max_tokens", None)+ current = sent if sent is not None else payload.get("max_completion_tokens")+ if current is not None or MAX_TOKENS_FLOOR:+ payload["max_completion_tokens"] = max(current or 0, MAX_TOKENS_FLOOR)+ body = json.dumps(payload).encode()+ except Exception as error:+ print(f"proxy: passing body through unmodified: {error}", flush=True)+ path = self.path+ if path.startswith("/v1/"):+ path = path[len("/v1"):]+ request = urllib.request.Request(UPSTREAM + path, data=body, method="POST")+ request.add_header("Content-Type", "application/json")+ auth_value = os.environ.get(AUTH_ENV, "") if AUTH_ENV else ""+ if auth_value:+ request.add_header("Authorization", f"Bearer {auth_value}")+ else:+ for header in ("Authorization", "x-api-key"):+ value = self.headers.get(header)+ if value:+ request.add_header(header, value)+ for header in ("anthropic-version",):+ value = self.headers.get(header)+ if value:+ request.add_header(header, value)+ try:+ with urllib.request.urlopen(request, timeout=900) as response:+ data = response.read()+ status = response.status+ except urllib.error.HTTPError as error:+ data = error.read()+ status = error.code+ except Exception as error:+ data = json.dumps({"error": {"message": str(error)}}).encode()+ status = 502+ self.send_response(status)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(data)))+ self.end_headers()+ self.wfile.write(data)++ def log_message(self, format, *args):+ print("proxy: " + format % args, flush=True)++if __name__ == "__main__":+ print(f"judge proxy listening on 127.0.0.1:{PORT} -> {UPSTREAM}", flush=True)+ ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()+"""++# Idempotent starter. Uses bash's /dev/tcp (pgrep isn't in every image) to check+# the port; the daemon inherits ZED_JUDGE_UPSTREAM from the verifier setup exec+# env. stdout/stderr/stdin are fully detached so the spawning exec doesn't hang+# on open pipes.+JUDGE_PROXY_ENSURE_SCRIPT = """\+#!/bin/bash+if ! (exec 3<>/dev/tcp/127.0.0.1/${ZED_JUDGE_PROXY_PORT:-8089}) 2>/dev/null; then+ nohup python3 /usr/local/lib/zed_judge_proxy.py \\+ >>/tmp/zed-judge-proxy.log 2>&1 </dev/null &+ sleep 1+fi+"""diff --git a/crates/eval_cli/zed_eval/launch.py b/crates/eval_cli/zed_eval/launch.pynew file mode 100644index 00000000000..699cbe3bd39--- /dev/null+++ b/crates/eval_cli/zed_eval/launch.py@@ -0,0 +1,670 @@+from __future__ import annotations++import argparse+import json+import sys+import uuid+from pathlib import Path+from typing import Any++from . import benchmarks, config, harness_command, run_index, source+from .builds import prepare_build_request, resolve_source, validate_build_id+from .common import (+ dedupe_preserving_order,+ default_namespace,+ deployed_function,+ modal_call_id,+ parse_parts,+ print_json,+ utc_now,+ utc_timestamp,+)+from .volume import build_ready_on_volume++# Local scratch dir used only for dry-run/plan previews of harness commands.+PREVIEW_JOBS_DIR = "/tmp/agent-evals/harbor-jobs"+++def read_task_file(path: str | None) -> list[str]:+ if not path:+ return []+ return [+ task_name+ for line in Path(path).read_text().splitlines()+ if (task_name := line.strip()) and not task_name.startswith("#")+ ]+++def baseten_provider_json(+ *,+ model_id: str,+ api_url: str,+ display_name: str | None,+ max_tokens: int,+ max_output_tokens: int,+) -> str:+ provider = {+ config.BASETEN_PROVIDER_ID: {+ "api_url": api_url,+ "available_models": [+ {+ "name": model_id,+ "display_name": display_name or model_id,+ "max_tokens": max_tokens,+ "max_output_tokens": max_output_tokens,+ "capabilities": {+ "tools": True,+ "images": False,+ "parallel_tool_calls": False,+ "prompt_cache_key": False,+ },+ }+ ],+ }+ }+ return json.dumps(provider, separators=(",", ":"))+++def resolve_model_preset(model: str) -> str:+ resolved = config.resolve_model_preset(model)+ if resolved != model:+ return resolved+ if model.startswith("baseten:"):+ return f"{config.BASETEN_PROVIDER_ID}/{model.split(':', 1)[1]}"+ return model+++def resolve_model_options(+ args: argparse.Namespace,+) -> tuple[str, str | None, list[str]]:+ raw_model = getattr(args, "model", None) or config.DEFAULT_MODEL+ model = resolve_model_preset(raw_model)+ openai_compatible_provider_json = getattr(+ args, "openai_compatible_provider_json", None+ )+ extra_api_secrets = list(getattr(args, "extra_api_secret", None) or [])++ if getattr(args, "model_provider", None) == "baseten":+ baseten_model = getattr(args, "baseten_model", None) or model+ if baseten_model.startswith(f"{config.BASETEN_PROVIDER_ID}/"):+ baseten_model = baseten_model.split("/", 1)[1]+ if baseten_model.startswith("baseten:"):+ baseten_model = baseten_model.split(":", 1)[1]+ model = f"{config.BASETEN_PROVIDER_ID}/{baseten_model}"++ if model.startswith(f"{config.BASETEN_PROVIDER_ID}/"):+ baseten_model = model.split("/", 1)[1]+ if not openai_compatible_provider_json:+ openai_compatible_provider_json = baseten_provider_json(+ model_id=baseten_model,+ api_url=getattr(args, "baseten_api_url", config.BASETEN_API_URL),+ display_name=getattr(args, "baseten_model_display_name", None),+ max_tokens=getattr(+ args, "baseten_model_max_tokens", config.BASETEN_DEFAULT_MAX_TOKENS+ ),+ max_output_tokens=getattr(+ args,+ "baseten_model_max_output_tokens",+ config.BASETEN_DEFAULT_MAX_OUTPUT_TOKENS,+ ),+ )++ return model, openai_compatible_provider_json, extra_api_secrets+++def derive_run_id(+ base_run_id: str | None, suite_id: str | None, suffix: str, index: int+) -> str:+ """Pick a run id for one leg of a (possibly multi-target) launch.++ An explicit --run-id is used verbatim for the first leg and suffixed for the+ rest; otherwise legs hang off the suite id, falling back to a timestamped id+ for a lone run.+ """+ if base_run_id:+ return base_run_id if index == 0 else f"{base_run_id}-{suffix}"+ if suite_id:+ return f"{suite_id}-{suffix}-{uuid.uuid4().hex[:6]}"+ return f"{utc_timestamp()}-{uuid.uuid4().hex[:6]}"+++def mint_suite_id(args: argparse.Namespace, run_count: int) -> str | None:+ """A suite id groups multiple runs from one invocation; None for a lone run."""+ if run_count <= 1:+ return None+ explicit = getattr(args, "suite_id", None)+ if explicit:+ return explicit+ prefix = getattr(args, "run_id", None) or "run"+ return f"{source.sanitize_namespace(prefix)}-{utc_timestamp()}"+++def build_rejudge_request(args: argparse.Namespace) -> dict[str, Any]:+ """Build the request for re-grading an existing run with a different judge.++ The positional `run_id` is the *parent* run; the derived run gets a new id+ under the same experiment so `report`/`list` group them together. Only the+ judge differs — no build, no agent model, no task selection.+ """+ judge_preset = args.judge+ config.get_judge(judge_preset) # validate before spawning anything remote+ if getattr(args, "experiment_name", None):+ experiment_name = source.sanitize_namespace(args.experiment_name)+ namespace = default_namespace(args)+ else:+ entry = run_index.lookup(args.run_id)+ if not entry:+ raise ValueError(+ f"could not locate run '{args.run_id}' in the local run index "+ f"({run_index.index_path()}). Pass --experiment-name (and "+ "--namespace if it isn't yours)."+ )+ experiment_name = source.sanitize_namespace(entry["experiment_name"])+ namespace = source.sanitize_namespace(+ getattr(args, "namespace", None) or entry["namespace"]+ )+ parent_namespace = (+ source.sanitize_namespace(args.parent_namespace)+ if getattr(args, "parent_namespace", None)+ else namespace+ )+ parent_run_id = args.run_id+ judge_slug = source.sanitize_namespace(judge_preset)+ new_run_id = (+ args.new_run_id+ or f"{parent_run_id}-rejudge-{judge_slug}-{uuid.uuid4().hex[:6]}"+ )+ return {+ "namespace": namespace,+ "experiment_name": experiment_name,+ "run_id": new_run_id,+ "parent": {+ "namespace": parent_namespace,+ "experiment_name": experiment_name,+ "run_id": parent_run_id,+ },+ "judge_preset": judge_preset,+ "judge_model": getattr(args, "judge_model", None),+ "volume_name": args.volume,+ "api_secret_name": args.api_secret,+ "created_at": utc_now(),+ }+++def command_rejudge(args: argparse.Namespace) -> int:+ rejudge_request = build_rejudge_request(args)+ if getattr(args, "dry_run", False) or getattr(args, "plan", False):+ print_json(rejudge_request)+ return 0++ controller = deployed_function(args, "rejudge_controller")+ call = controller.spawn(rejudge_request)+ run_index.record_run(+ {**rejudge_request, "volume_name": args.volume, "kind": "rejudge"}+ )+ parent = rejudge_request["parent"]+ print(f"Namespace: {rejudge_request['namespace']}")+ print(f"Experiment: {rejudge_request['experiment_name']}")+ print(+ f"Source run: {parent['namespace']}/{parent['experiment_name']}/"+ f"{parent['run_id']}"+ )+ print(f"New run id: {rejudge_request['run_id']}")+ print(f"Judge: {rejudge_request['judge_preset']}")+ print(f"Spawned rejudge controller: {modal_call_id(call)}")+ print("\nNext steps (run id alone is enough):")+ print(f" zed-eval status {rejudge_request['run_id']}")+ print(f" zed-eval report {rejudge_request['run_id']} --fetch")+ return 0+++def common_run_request_fields(+ args: argparse.Namespace,+ *,+ namespace: str,+ run_id: str,+ experiment_name: str,+ judge_preset: str | None,+ build_id: str | None,+ suite_id: str | None,+) -> dict[str, Any]:+ """Fields shared by every benchmark run request."""+ agent_model, openai_compatible_provider_json, extra_api_secrets = (+ resolve_model_options(args)+ )+ task_names = dedupe_preserving_order(+ read_task_file(getattr(args, "tasks", None))+ + (getattr(args, "include_task_name", None) or [])+ )+ return {+ "created_at": utc_now(),+ "namespace": namespace,+ "run_id": run_id,+ "experiment_name": experiment_name,+ "volume_name": args.volume,+ "api_secret_name": args.api_secret,+ "modal_token_secret_name": args.modal_token_secret,+ "orchestration": config.orchestration_info(),+ "agent_model": agent_model,+ "judge_preset": judge_preset,+ "judge_model": getattr(args, "judge_model", None),+ "build_id": build_id,+ "task_names": task_names,+ "n_tasks": getattr(args, "n_tasks", None),+ "n_concurrent": getattr(args, "n_concurrent", config.DEFAULT_N_CONCURRENT),+ "override_cpus": getattr(args, "override_cpus", config.DEFAULT_OVERRIDE_CPUS),+ "override_memory_mb": getattr(+ args, "override_memory_mb", config.DEFAULT_OVERRIDE_MEMORY_MB+ ),+ "sandbox_timeout_secs": getattr(+ args, "sandbox_timeout_secs", config.DEFAULT_SANDBOX_TIMEOUT_SECS+ ),+ "sandbox_idle_timeout_secs": getattr(+ args, "sandbox_idle_timeout_secs", config.DEFAULT_SANDBOX_IDLE_TIMEOUT_SECS+ ),+ "build_wait_timeout_secs": getattr(args, "build_wait_timeout_secs", 7200),+ "extra_harbor_args": list(getattr(args, "extra_harbor_arg", None) or []),+ "openai_compatible_provider_json": openai_compatible_provider_json,+ "anthropic_available_models_json": getattr(+ args, "anthropic_available_models_json", None+ ),+ "extra_api_secrets": dedupe_preserving_order(extra_api_secrets),+ "suite_id": suite_id,+ }+++def benchmark_plan_entry(+ benchmark_id: str, run_request: dict[str, Any], build_request: dict[str, Any] | None+) -> dict[str, Any]:+ return {+ "benchmark": benchmark_id,+ "run_id": run_request["run_id"],+ "harness": run_request["benchmark"]["harness"],+ "model": run_request["agent_model"],+ "judge": run_request.get("judge_preset"),+ "build_id": run_request.get("build_id"),+ "will_build": build_request is not None,+ "n_tasks": run_request.get("n_tasks"),+ }+++def print_plan(+ summaries: list[dict[str, Any]],+ details: list[tuple[str, dict[str, Any], dict[str, Any] | None]],+ *,+ verbose: bool,+) -> None:+ print("Plan:")+ print_json(summaries)+ if verbose:+ for header, run_request, build_request in details:+ print(f"\n=== {header} ===")+ print_dry_run(run_request, build_request)+++def print_dry_run(+ run_request: dict[str, Any], build_request: dict[str, Any] | None+) -> None:+ print("Run request:")+ print_json(run_request)+ if build_request:+ print("\nBuild request:")+ build_preview = dict(build_request)+ patch = build_preview.pop("patch", "")+ build_preview["patch_line_count"] = len(patch.splitlines())+ build_preview["source"] = source.public_source_info(+ build_request.get("source") or {}+ )+ print_json(build_preview)+ command = config.redacted_command(+ harness_command.build_harness_command(run_request, PREVIEW_JOBS_DIR)+ )+ print("\nHarness command:")+ print(command)+++def execute_prepared_runs(+ args: argparse.Namespace,+ prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]],+ plan_entry,+) -> int:+ if args.plan:+ print_plan(+ [+ plan_entry(label, run_request, build_request)+ for label, run_request, build_request in prepared+ ],+ prepared,+ verbose=args.verbose,+ )+ return 0+ if args.dry_run:+ for label, run_request, build_request in prepared:+ print(f"\n=== {label} ===")+ print_dry_run(run_request, build_request)+ return 0++ spawned_builds: set[str] = set()+ for label, run_request, build_request in prepared:+ print(f"\n=== Launching {label} ===")+ launch_prepared_run(args, run_request, build_request, spawned_builds)+ return 0+++def launch_prepared_run(+ args: argparse.Namespace,+ run_request: dict[str, Any],+ build_request: dict[str, Any] | None,+ spawned_builds: set[str] | None = None,+) -> None:+ build_function = None+ if build_request:+ run_request["source"] = source.public_source_info(build_request["source"])+ print_untracked_warning(build_request)+ build_function = deployed_function(args, "build_eval_cli")+ record_function = deployed_function(args, "create_run_record")+ controller_function = deployed_function(args, "run_controller")++ record_state = record_function.remote(run_request)+ run_index.record_run(run_request)++ print(f"Namespace: {run_request['namespace']}")+ print(f"Experiment: {run_request['experiment_name']}")+ print(f"Run id: {run_request['run_id']}")+ print(f"Volume: {run_request['volume_name']}")+ print(f"Run state: {record_state['status']}")+ print(f"Model: {run_request['agent_model']}")+ if run_request.get("judge_preset"):+ print(f"Judge: {run_request['judge_preset']}")+ if run_request.get("suite_id"):+ print(f"Suite: {run_request['suite_id']}")+ if run_request.get("build_id"):+ print(f"Build id: {run_request['build_id']}")+ if run_request.get("task_names"):+ print(f"Tasks: {len(run_request['task_names'])} explicit task(s)")+ elif run_request.get("n_tasks"):+ print(f"Tasks: Harbor --n-tasks {run_request['n_tasks']}")+ else:+ print("Tasks: full dataset selection")++ build_id = run_request.get("build_id")+ if build_function is not None and build_id not in (spawned_builds or set()):+ build_call = build_function.spawn(build_request)+ print(f"Spawned build: {modal_call_id(build_call)}")+ if spawned_builds is not None and build_id:+ spawned_builds.add(build_id)+ controller_call = controller_function.spawn(run_request)+ print(f"Spawned controller: {modal_call_id(controller_call)}")++ run_id = run_request["run_id"]+ print(+ "\nNext steps (run id alone is enough; namespace/experiment are resolved "+ "from this machine's local run index):"+ )+ print(f" zed-eval status {run_id}")+ print(f" zed-eval logs {run_id}")+ print(f" zed-eval report {run_id} --fetch")+++def resolve_benchmark_judge(+ args: argparse.Namespace, benchmark: benchmarks.Benchmark+) -> str | None:+ if not benchmark.needs_judge:+ return None+ judge = getattr(args, "judge", None) or config.DEFAULT_JUDGE_PRESET+ if judge == "auto":+ return benchmark.default_judge or "leaderboard"+ config.get_judge(judge)+ return judge+++def benchmark_metadata_for_run(+ args: argparse.Namespace, benchmark: benchmarks.Benchmark+) -> dict[str, object]:+ metadata = benchmarks.benchmark_metadata(benchmark)+ dataset = metadata.get("dataset")+ if (+ isinstance(dataset, dict)+ and dataset.get("repo_url") == benchmarks.SWE_ATLAS_REPO_URL+ ):+ dataset["repo_url"] = (+ getattr(args, "swe_atlas_repo_url", None) or benchmarks.SWE_ATLAS_REPO_URL+ )+ dataset["repo_ref"] = (+ getattr(args, "swe_atlas_repo_ref", None) or benchmarks.SWE_ATLAS_REPO_REF+ )+ return metadata+++def print_untracked_warning(build_request: dict[str, Any]) -> None:+ build_source = build_request.get("source") or {}+ untracked_files = build_source.get("untracked_files") or []+ if untracked_files:+ print(+ f"Warning: proceeding with {len(untracked_files)} untracked file(s) "+ "not included in the build patch.",+ file=sys.stderr,+ )+++def prepare_shared_build(+ args: argparse.Namespace,+) -> tuple[str | None, dict[str, Any] | None]:+ """Resolve the build once for a whole (possibly multi-benchmark) run.++ Returns `(build_id, build_request)`. `build_request` is None when the target+ build already exists; otherwise the caller should spawn the build.+ """+ explicit_build_id = getattr(args, "build", None)+ validate_build_id(explicit_build_id)+ if (+ explicit_build_id+ and not getattr(args, "plan", False)+ and not getattr(args, "dry_run", False)+ and build_ready_on_volume(args, explicit_build_id)+ ):+ print(f"Reusing existing build {explicit_build_id} (already on volume)")+ return explicit_build_id, None++ base_sha, clean_source, source_label, pre_resolved = resolve_source(args)++ build_request = prepare_build_request(+ base_sha=base_sha,+ patch_path=getattr(args, "patch_path", None),+ build_id=explicit_build_id,+ allow_untracked=getattr(args, "allow_untracked", False),+ require_clean=getattr(args, "require_clean", False),+ repo_url=getattr(args, "repo_url", None),+ clean_source=clean_source,+ source_label=source_label,+ pre_resolved_base_sha=pre_resolved,+ )+ build_id = build_request["build_id"]++ if not getattr(args, "plan", False) and not getattr(args, "dry_run", False):+ if build_ready_on_volume(args, build_id):+ print(f"Reusing existing build {build_id} (already on volume)")+ return build_id, None++ return build_id, build_request+++def command_build(args: argparse.Namespace) -> int:+ build_id, build_request = prepare_shared_build(args)+ if build_request is None:+ return 0++ print(f"Build id: {build_id}")+ print(f"Base sha: {build_request['base_sha']}")+ print(f"Patch sha256: {build_request['patch_sha256'] or '(none)'}")+ print_untracked_warning(build_request)++ build_function = deployed_function(args, "build_eval_cli")+ if args.detach:+ call = build_function.spawn(build_request)+ print(f"Spawned build: {modal_call_id(call)}")+ else:+ result = build_function.remote(build_request)+ print_json(result)+ return 0+++def build_benchmark_run_request(+ args: argparse.Namespace,+ *,+ benchmark_id: str,+ build_id: str | None,+ suite_id: str | None,+ index: int,+ run_id_suffix: str | None = None,+) -> dict[str, Any]:+ benchmark = benchmarks.get_benchmark(benchmark_id)+ run_id = derive_run_id(+ getattr(args, "run_id", None), suite_id, run_id_suffix or benchmark_id, index+ )+ # Staff mode is off by default for remote runs: it enables the sandboxed+ # terminal, which hangs inside Modal sandboxes.+ extra_env = {"EVAL_CLI_STAFF": "true" if getattr(args, "staff", False) else "false"}++ return {+ **common_run_request_fields(+ args,+ namespace=default_namespace(args),+ run_id=run_id,+ # The benchmark id doubles as the experiment name for run storage+ # paths (runs/<namespace>/<experiment_name>/<run_id>), keeping+ # monitoring and fetch uniform across benchmarks.+ experiment_name=source.sanitize_namespace(benchmark_id),+ judge_preset=resolve_benchmark_judge(args, benchmark),+ build_id=build_id,+ suite_id=suite_id,+ ),+ "benchmark": benchmark_metadata_for_run(args, benchmark),+ "eval_cli_timeout": getattr(args, "eval_cli_timeout", None),+ "extra_env": extra_env,+ }+++def prepare_runs_for_benchmarks(+ args: argparse.Namespace,+ benchmark_ids: list[str],+ *,+ suite_id: str | None,+ label_for_benchmark,+ mark_swe_atlas_parts: bool = False,+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:+ if not benchmark_ids:+ raise ValueError("choose at least one benchmark to run")++ build_id, build_request = prepare_shared_build(args)+ prepared: list[tuple[str, dict[str, Any], dict[str, Any] | None]] = []+ for index, benchmark_id in enumerate(benchmark_ids):+ label = label_for_benchmark(benchmark_id)+ run_request = build_benchmark_run_request(+ args,+ benchmark_id=benchmark_id,+ build_id=build_id,+ suite_id=suite_id,+ index=index,+ run_id_suffix=label,+ )+ if (+ mark_swe_atlas_parts+ and benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.values()+ ):+ run_request["suite_part"] = label+ # Only the first run carries the build_request; launch_prepared_run+ # dedups the actual spawn via spawned_builds anyway.+ prepared.append((label, run_request, build_request if index == 0 else None))+ return prepared+++def prepare_benchmark_runs(+ args: argparse.Namespace,+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:+ benchmark_ids = benchmarks.resolve_benchmarks(+ list(getattr(args, "benchmark", None) or [])+ )+ return prepare_runs_for_benchmarks(+ args,+ benchmark_ids,+ suite_id=mint_suite_id(args, len(benchmark_ids)),+ label_for_benchmark=lambda benchmark_id: benchmark_id,+ )+++def command_run(args: argparse.Namespace) -> int:+ return execute_prepared_runs(+ args,+ prepare_benchmark_runs(args),+ benchmark_plan_entry,+ )+++def resolve_suite_parts(args: argparse.Namespace) -> list[str]:+ parts = parse_parts([args.parts] if getattr(args, "parts", None) else [])+ if parts:+ return parts+ raise ValueError(+ "choose at least one SWE-Atlas part with --parts (e.g. --parts rf,qna or --parts all)"+ )+++def suite_plan_entry(+ part: str, run_request: dict[str, Any], build_request: dict[str, Any] | None+) -> dict[str, Any]:+ return {+ "part": part,+ **benchmark_plan_entry(+ run_request["benchmark"]["id"], run_request, build_request+ ),+ }+++def suite_entry_label(benchmark_id: str) -> str:+ for part, part_benchmark_id in benchmarks.SWE_ATLAS_PART_BENCHMARKS.items():+ if part_benchmark_id == benchmark_id:+ return part+ return benchmark_id+++def prepare_benchmark_suite(+ args: argparse.Namespace, selectors: list[str]+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:+ benchmark_ids = benchmarks.resolve_benchmarks(selectors)+ if not benchmark_ids:+ raise ValueError("choose at least one benchmark to run")+ if getattr(args, "zed_version", None):+ args.clean_source = True+ args.require_clean = True+ timestamp = utc_timestamp()+ prefix_seed = args.run_id_prefix or args.experiment_prefix or "swe-atlas"+ if getattr(args, "zed_version", None):+ prefix_seed = f"{prefix_seed}-{source.sanitize_namespace(args.zed_version)}"+ suite_id = args.suite_id or f"{source.sanitize_namespace(prefix_seed)}-{timestamp}"++ return prepare_runs_for_benchmarks(+ args,+ benchmark_ids,+ suite_id=suite_id,+ label_for_benchmark=suite_entry_label,+ mark_swe_atlas_parts=True,+ )+++def prepare_suite(+ args: argparse.Namespace,+) -> list[tuple[str, dict[str, Any], dict[str, Any] | None]]:+ return prepare_benchmark_suite(args, resolve_suite_parts(args))+++def command_swe_atlas(args: argparse.Namespace) -> int:+ from .interactive import configure_interactive_suite++ configure_interactive_suite(args)+ prepared = (+ prepare_benchmark_suite(args, args.benchmark)+ if getattr(args, "benchmark", None)+ else prepare_suite(args)+ )+ return execute_prepared_runs(args, prepared, suite_plan_entry)diff --git a/crates/eval_cli/zed_eval/modal_app.py b/crates/eval_cli/zed_eval/modal_app.pynew file mode 100644index 00000000000..4ba1c478908--- /dev/null+++ b/crates/eval_cli/zed_eval/modal_app.py@@ -0,0 +1,1008 @@+from __future__ import annotations++import hashlib+import json+import os+import pathlib+import shlex+import shutil+import subprocess+import tarfile+import time+import traceback+import uuid+from typing import Any++import modal++from . import cleanup, config, harness_command, rejudge, source+from .common import load_json, safe_extract_archive, utc_now, write_json+from .source import (+ BUILD_TARGET,+ CARGO_ZIGBUILD_VERSION,+ RUST_IMAGE,+ RUST_VERSION,+ ZIG_VERSION,+ build_toolchain_info,+)++APP_NAME = os.environ.get("AGENT_EVALS_APP_NAME", "agent-evals")+VOLUME_NAME = os.environ.get("AGENT_EVALS_VOLUME", "agent-evals")+MODAL_TOKEN_SECRET_NAME = os.environ.get(+ "AGENT_EVALS_MODAL_TOKEN_SECRET", "agent-evals-modal-token"+)+# LLM-providers secret mounted into the rejudge controller so the in-controller+# judge proxy can read the judge's API key (e.g. BASETEN_API_KEY) the same way a+# trial sandbox does. Override at deploy time if the secret is named differently.+LLM_PROVIDERS_SECRET_NAME = os.environ.get(+ "AGENT_EVALS_LLM_PROVIDERS_SECRET", config.DEFAULT_LLM_PROVIDERS_SECRET_NAME+)+REPO_URL = os.environ.get(+ "AGENT_EVALS_REPO_URL", "https://github.com/zed-industries/zed.git"+)+ZIG_URL = (+ f"https://ziglang.org/download/{ZIG_VERSION}/zig-x86_64-linux-{ZIG_VERSION}.tar.xz"+)+CLEANUP_BUILD_RETENTION_DAYS = float(+ os.environ.get("AGENT_EVALS_BUILD_RETENTION_DAYS", "14")+)++app = modal.App(APP_NAME)+volume = modal.Volume.from_name(VOLUME_NAME, create_if_missing=True)+++def reload_volume() -> None:+ try:+ volume.reload()+ except Exception:+ pass+++build_image = (+ modal.Image.from_registry(RUST_IMAGE, add_python="3.13")+ .apt_install("cmake", "build-essential", "curl", "xz-utils", "git")+ .run_commands(+ f"rustup toolchain install {RUST_VERSION} --profile minimal"+ " --component rustfmt --component clippy"+ " --component rust-analyzer --component rust-src"+ " --target wasm32-wasip2 --target wasm32-unknown-unknown"+ " --target x86_64-unknown-linux-musl --target x86_64-unknown-linux-gnu",+ f"mkdir -p /opt/zig && curl -fsSL {ZIG_URL}"+ " | tar -xJ -C /opt/zig --strip-components=1"+ " && ln -s /opt/zig/zig /usr/local/bin/zig",+ f"cargo install --locked cargo-zigbuild --version {CARGO_ZIGBUILD_VERSION}",+ )+)++controller_image = (+ modal.Image.debian_slim(python_version="3.13")+ .apt_install("bash", "ca-certificates", "curl", "git", "tar")+ .pip_install(f"modal=={config.MODAL_VERSION}", "uv")+ .run_commands(+ f"uv tool install harbor=={config.HARBOR_VERSION}"+ f" --with 'modal=={config.MODAL_VERSION}' && "+ "ln -sf /root/.local/bin/harbor /usr/local/bin/harbor",+ # Pier is a Harbor fork that adds per-agent network allowlists, required+ # to run DeepSWE's air-gapped (`allow_internet = false`) tasks while+ # still letting eval-cli reach the model API. Kept non-fatal so a Pier+ # install problem can't break the Harbor benchmarks that share this image;+ # DeepSWE runs surface a clear "pier not found" error at runtime instead.+ "(uv tool install git+https://github.com/datacurve-ai/pier"+ f" --with 'modal=={config.MODAL_VERSION}' && "+ "ln -sf /root/.local/bin/pier /usr/local/bin/pier) || "+ "echo 'WARNING: pier install failed; DeepSWE runs will not work'",+ )+)+++def provision_benchmark_dataset(run_request: dict[str, Any], log: Any) -> None:+ """Clone a path-backed benchmark dataset (SWE-Atlas tw, DeepSWE) into the+ location the harness command expects. Registry datasets need no provisioning;+ the harness pulls them from its hub."""+ benchmark = run_request["benchmark"]+ dataset = benchmark.get("dataset") or {}+ kind = dataset.get("kind")+ if kind not in ("path", "pier_path"):+ return++ repo_url = dataset.get("repo_url")+ repo_ref = dataset.get("repo_ref") or "main"+ if not repo_url:+ raise ValueError(f"benchmark {benchmark['id']} path dataset requires repo_url")++ clone_dir = pathlib.Path(harness_command.dataset_clone_dir(benchmark))+ if clone_dir.exists():+ shutil.rmtree(clone_dir)+ clone_dir.mkdir(parents=True, exist_ok=True)+ log(f"Fetching {benchmark['id']} dataset {repo_url}@{repo_ref}")+ subprocess.run(["git", "init", "-q", str(clone_dir)], check=True)+ subprocess.run(+ ["git", "fetch", "--depth", "1", repo_url, repo_ref],+ cwd=clone_dir,+ check=True,+ )+ subprocess.run(["git", "checkout", "-q", "FETCH_HEAD"], cwd=clone_dir, check=True)+ data_dir = pathlib.Path(harness_command.dataset_path(benchmark))+ if not data_dir.exists():+ raise FileNotFoundError(f"benchmark dataset directory not found: {data_dir}")+++def output_of(command: list[str]) -> str:+ try:+ return subprocess.run(+ command,+ check=True,+ capture_output=True,+ text=True,+ ).stdout.strip()+ except subprocess.CalledProcessError as error:+ return f"unknown ({error})"+++ image=build_image,+ # Right-sized from the original 16 cpu / 32 GB after observing peak usage of+ # ~10 cores and ~10 GB. ephemeral_disk stays at Modal's 512 GiB floor (the+ # minimum allowed for a function), so it isn't a tunable here.+ cpu=12,+ memory=24576,+ ephemeral_disk=524288,+ timeout=7200,+ volumes={"/data": volume},+)+def build_eval_cli(build_request: dict[str, Any]) -> dict[str, Any]:+ def run(command: str, **kwargs: Any) -> None:+ print(f"+ {command}", flush=True)+ subprocess.run(command, shell=True, check=True, **kwargs)++ reload_volume()++ build_id = build_request["build_id"]+ source_info = build_request.get("source") or {"type": "git_patch"}+ base_sha = source_info.get("base_sha") or build_request["base_sha"]+ patch = build_request.get("patch") or ""+ build_dir = pathlib.Path("/data/builds") / build_id+ ready_path = build_dir / "READY"+ binary_path = build_dir / "eval-cli"+ build_info_path = build_dir / "build-info.json"+ # The lease lives outside build_dir so it never interferes with the atomic+ # move of the finished build directory below.+ building_path = pathlib.Path("/data/build-locks") / f"{build_id}.json"+ lease_ttl = int(build_request.get("build_wait_timeout_secs") or 7200)+ owner = uuid.uuid4().hex++ def reuse_existing() -> dict[str, Any]:+ build_info = load_json(build_info_path) or {}+ build_info.setdefault("build_id", build_id)+ build_info["reused"] = True+ volume.commit()+ print(f"Reusing existing build {build_id}", flush=True)+ return build_info++ if ready_path.exists() and binary_path.exists():+ return reuse_existing()+ if build_dir.exists():+ raise RuntimeError(+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"+ )++ # Single-flight lease: if another invocation is already compiling this exact+ # build, wait for it to finish rather than running a second multi-minute+ # compile. The lease is best-effort (the volume has no atomic compare-and-swap);+ # the atomic move below still guarantees correctness if two builds slip through.+ deadline = time.time() + lease_ttl+ while True:+ reload_volume()+ if ready_path.exists() and binary_path.exists():+ return reuse_existing()+ if build_dir.exists():+ raise RuntimeError(+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"+ )+ lease = load_json(building_path)+ now = time.time()+ held_by_other = (+ isinstance(lease, dict)+ and lease.get("owner") != owner+ and (now - float(lease.get("epoch") or 0)) < lease_ttl+ )+ if held_by_other and time.time() < deadline:+ print(+ f"Build {build_id} is being compiled elsewhere; waiting",+ flush=True,+ )+ time.sleep(15)+ continue+ break+ # Claim the lease (write_json creates /data/build-locks). build_dir is+ # intentionally NOT created here so the atomic move below still works.+ write_json(building_path, {"owner": owner, "epoch": time.time(), "at": utc_now()})+ volume.commit()++ rustc_version = output_of(["rustc", "--version"])+ zig_version = output_of(["zig", "version"])+ cargo_zigbuild_version = output_of(["cargo-zigbuild", "--version"])++ workdir = pathlib.Path("/build/zed")+ if workdir.exists():+ shutil.rmtree(workdir)+ workdir.mkdir(parents=True, exist_ok=True)+ os.chdir(workdir)++ repo_url = source_info.get("repo_url") or REPO_URL+ run("git init -q .")+ run(f"git fetch --depth 1 {shlex.quote(repo_url)} {shlex.quote(base_sha)}")+ run("git checkout -q FETCH_HEAD")++ if patch.strip():+ patch_file = pathlib.Path("/build/source.patch")+ patch_file.write_text(patch)+ run(f"git apply --stat {shlex.quote(str(patch_file))}")+ run(f"git apply {shlex.quote(str(patch_file))}")++ patch_sha256 = source_info.get("patch_sha256")++ run(+ "cargo zigbuild --release --package eval_cli --target x86_64-unknown-linux-musl"+ )++ built = workdir / "target/x86_64-unknown-linux-musl/release/eval-cli"+ run(f"strip {built}")+ binary_bytes = built.read_bytes()+ binary_sha256 = hashlib.sha256(binary_bytes).hexdigest()++ temporary_dir = pathlib.Path("/data/tmp/builds") / f"{build_id}-{uuid.uuid4().hex}"+ temporary_dir.mkdir(parents=True, exist_ok=True)+ shutil.copy(built, temporary_dir / "eval-cli")+ (temporary_dir / "eval-cli").chmod(0o755)+ if patch.strip():+ (temporary_dir / "source.patch").write_text(patch)+ write_json(temporary_dir / "source-info.json", source_info)++ build_info = {+ "build_id": build_id,+ "base_sha": base_sha,+ "patch_sha256": patch_sha256,+ "rustc_version": rustc_version,+ "zig_version": zig_version,+ "cargo_zigbuild_version": cargo_zigbuild_version,+ "built_at_utc": utc_now(),+ "binary_size_bytes": len(binary_bytes),+ "binary_sha256": binary_sha256,+ "target": BUILD_TARGET,+ "rust_image": RUST_IMAGE,+ "toolchain": build_toolchain_info(),+ "source": source_info,+ }+ write_json(temporary_dir / "build-info.json", build_info)+ (temporary_dir / "READY").write_text(utc_now() + "\n")++ if not build_dir.exists():+ build_dir.parent.mkdir(parents=True, exist_ok=True)+ shutil.move(str(temporary_dir), str(build_dir))+ elif ready_path.exists() and binary_path.exists():+ shutil.rmtree(temporary_dir, ignore_errors=True)+ build_info = load_json(build_info_path) or build_info+ build_info["reused_after_race"] = True+ else:+ shutil.rmtree(temporary_dir, ignore_errors=True)+ raise RuntimeError(+ f"build directory already exists but is not ready; refusing to overwrite {build_id}"+ )++ # Release the single-flight lease now that READY exists.+ try:+ building_path.unlink()+ except OSError:+ pass+ volume.commit()+ print(f"Committed build {build_id} to volume '{VOLUME_NAME}'", flush=True)+ return build_info+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def list_builds(limit: int = 50) -> list[dict[str, Any]]:+ reload_volume()+ builds_dir = pathlib.Path("/data/builds")+ rows = []+ for build_dir in builds_dir.iterdir() if builds_dir.exists() else []:+ if not build_dir.is_dir():+ continue+ build_info = load_json(build_dir / "build-info.json") or {}+ ready = (build_dir / "READY").exists() and (build_dir / "eval-cli").exists()+ rows.append(+ {+ "build_id": build_dir.name,+ "ready": ready,+ "base_sha": build_info.get("base_sha"),+ "patch_sha256": build_info.get("patch_sha256"),+ "built_at_utc": build_info.get("built_at_utc"),+ "binary_sha256": build_info.get("binary_sha256"),+ "source": build_info.get("source"),+ }+ )+ rows.sort(key=lambda row: row.get("built_at_utc") or "", reverse=True)+ return rows[:limit]+++def run_row(+ namespace_dir: pathlib.Path, experiment_dir: pathlib.Path, run_dir: pathlib.Path+) -> dict[str, Any]:+ state = load_json(run_dir / "state.json") or {}+ request = load_json(run_dir / "request.json") or {}+ metadata = load_json(run_dir / "run-metadata.json") or {}+ summary = load_json(run_dir / "summary.json") or {}+ benchmark = request.get("benchmark") or metadata.get("benchmark") or {}+ benchmark_id = benchmark.get("id") if isinstance(benchmark, dict) else None+ suite_id = request.get("suite_id") or metadata.get("suite_id")+ return {+ "namespace": namespace_dir.name,+ "experiment_name": experiment_dir.name,+ "run_id": run_dir.name,+ "status": state.get("status"),+ "updated_at": state.get("updated_at"),+ "created_at": metadata.get("created_at")+ or state.get("created_at")+ or request.get("created_at"),+ "agent_model": metadata.get("agent_model"),+ "judge_preset": metadata.get("judge_preset"),+ "judge_model": metadata.get("judge_model"),+ "build_id": metadata.get("build_id") or state.get("build_id"),+ "trial_count": summary.get("trial_count"),+ "has_archive": summary.get("has_archive"),+ "suite_id": suite_id,+ "part": request.get("suite_part"),+ "benchmark": benchmark_id,+ }+++def scan_runs(+ *,+ namespace: str | None = None,+ experiment_name: str | None = None,+ suite_id: str | None = None,+) -> list[dict[str, Any]]:+ runs_root = pathlib.Path("/data/runs")+ rows = []+ if not runs_root.exists():+ return []+ namespaces = (+ [runs_root / namespace] if namespace else [path for path in runs_root.iterdir()]+ )+ for namespace_dir in namespaces:+ if not namespace_dir.is_dir():+ continue+ experiments = (+ [namespace_dir / experiment_name]+ if experiment_name+ else [path for path in namespace_dir.iterdir()]+ )+ for experiment_dir in experiments:+ if not experiment_dir.is_dir():+ continue+ for run_dir in experiment_dir.iterdir():+ if not run_dir.is_dir():+ continue+ row = run_row(namespace_dir, experiment_dir, run_dir)+ if suite_id and row.get("suite_id") != suite_id:+ continue+ rows.append(row)+ rows.sort(+ key=lambda row: row.get("updated_at") or row.get("created_at") or "",+ reverse=True,+ )+ return rows+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def list_runs(+ namespace: str | None = None,+ experiment_name: str | None = None,+ limit: int = 50,+) -> list[dict[str, Any]]:+ reload_volume()+ rows = scan_runs(namespace=namespace, experiment_name=experiment_name)+ return rows[:limit]+++def _run_cleanup(+ *,+ dry_run: bool,+ build_retention_days: float | None = None,+) -> dict[str, Any]:+ reload_volume()+ result = cleanup.prune_artifacts(+ pathlib.Path("/data"),+ dry_run=dry_run,+ build_retention_days=(+ build_retention_days+ if build_retention_days is not None+ else CLEANUP_BUILD_RETENTION_DAYS+ ),+ )+ if not dry_run:+ volume.commit()+ print(json.dumps(result, indent=2), flush=True)+ return result+++ image=controller_image,+ cpu=1,+ memory=2048,+ timeout=1800,+ volumes={"/data": volume},+ schedule=modal.Period(days=1),+)+def cleanup_scheduled() -> dict[str, Any]:+ """Daily prune of stale build artifacts. Never touches eval results."""+ return _run_cleanup(dry_run=False)+++ image=controller_image,+ cpu=1,+ memory=2048,+ timeout=1800,+ volumes={"/data": volume},+)+def cleanup_artifacts(request: dict[str, Any]) -> dict[str, Any]:+ """On-demand prune (the `zed-eval cleanup` command), supporting --dry-run+ and retention overrides."""+ return _run_cleanup(+ dry_run=bool(request.get("dry_run")),+ build_retention_days=request.get("build_retention_days"),+ )+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def read_run_provenance(+ namespace: str, experiment_name: str, run_id: str+) -> dict[str, Any]:+ """Provenance a baseline record needs for one run: the launch request+ (model/judge/resources/build_id), the run summary (status), and the build's+ build-info (base_sha/patch_sha256/source label)."""+ reload_volume()+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id+ request = load_json(run_dir / "request.json") or {}+ summary = load_json(run_dir / "summary.json") or {}+ metadata = load_json(run_dir / "run-metadata.json") or {}+ build_id = request.get("build_id") or metadata.get("build_id")+ build_info: dict[str, Any] = {}+ if build_id:+ build_info = (+ load_json(pathlib.Path("/data/builds") / build_id / "build-info.json") or {}+ )+ return {+ "request": request,+ "summary": summary,+ "metadata": metadata,+ "build_info": build_info,+ }+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def record_baseline(record: dict[str, Any]) -> dict[str, Any]:+ """Write a baseline-of-record keyed by (experiment, model). Supersedes any+ existing current baseline (archived under history/) and refreshes the+ discoverable index."""+ reload_volume()+ experiment_slug = source.sanitize_namespace(record["experiment"])+ model_slug = source.sanitize_namespace(record.get("model_slug") or record["model"])+ base_dir = pathlib.Path("/data/baselines") / experiment_slug / model_slug+ current_path = base_dir / "current.json"++ existing = load_json(current_path)+ if existing:+ previous_sha = source.sanitize_namespace(+ str(existing.get("base_sha") or "unknown")+ )+ write_json(base_dir / "history" / f"{previous_sha}.json", existing)++ payload = {**record, "updated_at": utc_now()}+ write_json(current_path, payload)++ index_path = pathlib.Path("/data/baselines/index.json")+ index = load_json(index_path) or {}+ entries = index.get("baselines")+ if not isinstance(entries, list):+ entries = []+ entries = [+ entry+ for entry in entries+ if not (+ entry.get("experiment") == record["experiment"]+ and entry.get("model") == record["model"]+ )+ ]+ entries.append(+ {+ "experiment": record["experiment"],+ "model": record["model"],+ "base_sha": record.get("base_sha"),+ "base_ref": record.get("base_ref"),+ "on_main": record.get("on_main"),+ "clean": record.get("clean"),+ "judge": record.get("judge"),+ "run_id": (record.get("run") or {}).get("run_id"),+ "recorded_at": record.get("recorded_at"),+ "path": f"baselines/{experiment_slug}/{model_slug}/current.json",+ }+ )+ entries.sort(+ key=lambda entry: (entry.get("experiment") or "", entry.get("model") or "")+ )+ write_json(index_path, {"baselines": entries})+ volume.commit()+ return payload+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def read_baselines(+ experiment: str | None = None,+ model_slug: str | None = None,+ include_history: bool = False,+) -> dict[str, Any]:+ """List all current baselines (no args) or show one (experiment+model_slug),+ optionally with superseded history."""+ reload_volume()+ root = pathlib.Path("/data/baselines")+ if experiment and model_slug:+ base_dir = (+ root+ / source.sanitize_namespace(experiment)+ / source.sanitize_namespace(model_slug)+ )+ result: dict[str, Any] = {"current": load_json(base_dir / "current.json")}+ if include_history:+ history = []+ history_dir = base_dir / "history"+ if history_dir.is_dir():+ for path in sorted(history_dir.glob("*.json")):+ entry = load_json(path)+ if entry:+ history.append(entry)+ result["history"] = history+ return result+ return load_json(root / "index.json") or {"baselines": []}+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def suite_status(namespace: str, suite_id: str) -> list[dict[str, Any]]:+ reload_volume()+ rows = scan_runs(namespace=namespace, suite_id=suite_id)+ if not rows:+ raise FileNotFoundError(f"suite not found: {namespace}/{suite_id}")+ rows.sort(key=lambda row: row.get("created_at") or row.get("run_id") or "")+ return rows+++def write_run_inputs(run_dir: pathlib.Path, run_request: dict[str, Any]) -> None:+ write_json(run_dir / "request.json", run_request)+ write_json(run_dir / "run-metadata.json", harness_command.run_metadata(run_request))+ task_names = run_request.get("task_names") or []+ (run_dir / "selected-tasks.txt").write_text(+ "\n".join(task_names) + ("\n" if task_names else "")+ )+++ image=controller_image,+ cpu=1,+ memory=512,+ timeout=300,+ volumes={"/data": volume},+)+def create_run_record(run_request: dict[str, Any]) -> dict[str, Any]:+ namespace = run_request["namespace"]+ experiment_name = run_request["experiment_name"]+ run_id = run_request["run_id"]+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id+ state_path = run_dir / "state.json"++ reload_volume()++ if state_path.exists():+ raise FileExistsError(+ f"run record already exists: {namespace}/{experiment_name}/{run_id}"+ )++ run_dir.mkdir(parents=True, exist_ok=True)+ write_run_inputs(run_dir, run_request)+ state = {+ "run_id": run_id,+ "namespace": namespace,+ "experiment_name": experiment_name,+ "status": "pending",+ "created_at": run_request.get("created_at"),+ "updated_at": utc_now(),+ "build_id": run_request.get("build_id"),+ }+ write_json(state_path, state)+ volume.commit()+ return state+++ image=controller_image,+ cpu=2,+ memory=4096,+ timeout=86_400,+ volumes={"/data": volume},+ secrets=[modal.Secret.from_name(MODAL_TOKEN_SECRET_NAME)],+)+def run_controller(run_request: dict[str, Any]) -> dict[str, Any]:+ namespace = run_request["namespace"]+ experiment_name = run_request["experiment_name"]+ run_id = run_request["run_id"]+ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id+ run_dir.mkdir(parents=True, exist_ok=True)+ log_path = run_dir / "controller.log"+ state_path = run_dir / "state.json"+ started_at = utc_now()++ def commit() -> None:+ volume.commit()++ def log(message: str) -> None:+ line = f"[{utc_now()}] {message}"+ print(line, flush=True)+ with log_path.open("a") as log_file:+ log_file.write(line + "\n")++ def state(status: str, **extra: Any) -> None:+ payload = {+ "run_id": run_id,+ "namespace": namespace,+ "experiment_name": experiment_name,+ "status": status,+ "updated_at": utc_now(),+ "started_at": started_at,+ **extra,+ }+ write_json(state_path, payload)+ commit()++ try:+ write_run_inputs(run_dir, run_request)+ state("starting", build_id=run_request.get("build_id"))+ log(f"Starting run {namespace}/{experiment_name}/{run_id}")++ build_id = run_request.get("build_id")+ if build_id:+ build_dir = pathlib.Path("/data/builds") / build_id+ ready_path = build_dir / "READY"+ build_info_path = build_dir / "build-info.json"+ source_patch_path = build_dir / "source.patch"+ source_info_path = build_dir / "source-info.json"+ state("waiting_for_build", build_id=build_id)+ deadline = time.time() + int(+ run_request.get("build_wait_timeout_secs") or 7200+ )+ while not ready_path.exists():+ if time.time() >= deadline:+ raise TimeoutError(+ f"build {build_id} was not ready before the wait timeout"+ )+ log(f"Waiting for build {build_id} to become ready")+ time.sleep(30)+ reload_volume()+ shutil.copy(build_info_path, run_dir / "build-info.json")+ if source_patch_path.exists():+ shutil.copy(source_patch_path, run_dir / "source.patch")+ if source_info_path.exists():+ shutil.copy(source_info_path, run_dir / "source-info.json")+ log(f"Using build {build_id}")++ provision_benchmark_dataset(run_request, log)++ jobs_parent = pathlib.Path("/tmp/agent-evals/harbor-jobs")+ jobs_parent.mkdir(parents=True, exist_ok=True)+ command = harness_command.build_harness_command(run_request, str(jobs_parent))+ redacted = config.redacted_command(command)+ (run_dir / "harbor-command.txt").write_text(redacted + "\n")+ state(+ "running",+ build_id=build_id,+ harness_command=redacted,+ )+ commit()+ log("Launching Harbor")+ log(redacted)++ import zed_eval++ package_file = zed_eval.__file__+ if package_file is None:+ raise RuntimeError("could not resolve zed_eval package location")+ package_parent = str(pathlib.Path(package_file).resolve().parent.parent)+ environment = os.environ.copy()+ existing_pythonpath = environment.get("PYTHONPATH")+ environment["PYTHONPATH"] = (+ package_parent+ if not existing_pythonpath+ else f"{package_parent}:{existing_pythonpath}"+ )++ process = subprocess.Popen(+ command,+ stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT,+ text=True,+ env=environment,+ )+ last_commit = time.time()+ assert process.stdout is not None+ with log_path.open("a") as log_file:+ for line in process.stdout:+ print(line, end="", flush=True)+ log_file.write(line)+ if time.time() - last_commit >= 10:+ log_file.flush()+ commit()+ last_commit = time.time()+ return_code = process.wait()+ commit()+ log(f"Harbor exited with status {return_code}")++ job_dir = jobs_parent / run_id+ archive_path = run_dir / "harbor-job.tar.gz"+ if job_dir.exists():+ with tarfile.open(archive_path, "w:gz") as archive:+ archive.add(job_dir, arcname=job_dir.name)+ # Count trial dirs by the presence of a trial result.json rather than+ # a name prefix: Harbor names them `task-<id>__<suffix>` while Pier+ # uses `<task-id>__<suffix>`.+ trial_count = sum(+ 1+ for path in job_dir.iterdir()+ if path.is_dir() and (path / "result.json").exists()+ )+ job_result = load_json(job_dir / "result.json")+ else:+ trial_count = 0+ job_result = None+ log(f"Harbor job dir was not found at {job_dir}")++ summary = {+ "run_id": run_id,+ "namespace": namespace,+ "experiment_name": experiment_name,+ "status": "completed" if return_code == 0 else "failed",+ "harbor_return_code": return_code,+ "trial_count": trial_count,+ "has_archive": archive_path.exists(),+ "job_result": job_result,+ "started_at": started_at,+ "completed_at": utc_now(),+ }+ write_json(run_dir / "summary.json", summary)++ if return_code == 0:+ (run_dir / "READY").write_text(utc_now() + "\n")+ state("completed", summary=summary)+ else:+ (run_dir / "FAILED").write_text(utc_now() + "\n")+ state("failed", summary=summary)+ commit()+ return summary+ except Exception as error:+ formatted = traceback.format_exc()+ log(f"FAILED: {error}\n{formatted}")+ (run_dir / "FAILED").write_text(utc_now() + "\n")+ state("failed", error=str(error), traceback=formatted)+ commit()+ raise+++def rejudge_run_metadata(rejudge_request: dict[str, Any]) -> dict[str, Any]:+ judge = config.get_judge(rejudge_request["judge_preset"])+ parent = rejudge_request["parent"]+ return {+ "kind": "rejudge",+ "run_id": rejudge_request["run_id"],+ "namespace": rejudge_request["namespace"],+ "experiment_name": rejudge_request["experiment_name"],+ "source_run": parent,+ "judge_preset": rejudge_request["judge_preset"],+ "judge_model": rejudge_request.get("judge_model") or judge.model,+ "judge_upstream": judge.upstream,+ "judge_auth_env": judge.auth_env,+ "orchestration": config.orchestration_info(),+ "volume_name": rejudge_request["volume_name"],+ "api_secret_name": rejudge_request.get("api_secret_name"),+ "created_at": rejudge_request.get("created_at"),+ }+++ image=controller_image,+ cpu=2,+ memory=4096,+ timeout=86_400,+ volumes={"/data": volume},+ secrets=[+ modal.Secret.from_name(MODAL_TOKEN_SECRET_NAME),+ modal.Secret.from_name(LLM_PROVIDERS_SECRET_NAME),+ ],+)+def rejudge_controller(rejudge_request: dict[str, Any]) -> dict[str, Any]:+ """Re-grade a finished parent run with a different judge, producing a new+ derived run. The parent is read only: its stored agent outputs are re-scored+ by the real cached verifier through the judge proxy, and only the verdicts+ change. See `rejudge.py` for the per-trial grading."""+ namespace = rejudge_request["namespace"]+ experiment_name = rejudge_request["experiment_name"]+ run_id = rejudge_request["run_id"]+ parent = rejudge_request["parent"]+ judge_preset = rejudge_request["judge_preset"]+ judge = config.get_judge(judge_preset)+ judge_model = rejudge_request.get("judge_model") or judge.model++ run_dir = pathlib.Path("/data/runs") / namespace / experiment_name / run_id+ run_dir.mkdir(parents=True, exist_ok=True)+ log_path = run_dir / "controller.log"+ state_path = run_dir / "state.json"+ started_at = utc_now()++ def log(message: str) -> None:+ line = f"[{utc_now()}] {message}"+ print(line, flush=True)+ with log_path.open("a") as log_file:+ log_file.write(line + "\n")++ def state(status: str, **extra: Any) -> None:+ write_json(+ state_path,+ {+ "run_id": run_id,+ "namespace": namespace,+ "experiment_name": experiment_name,+ "kind": "rejudge",+ "status": status,+ "updated_at": utc_now(),+ "started_at": started_at,+ **extra,+ },+ )+ volume.commit()++ try:+ write_json(run_dir / "request.json", rejudge_request)+ write_json(run_dir / "run-metadata.json", rejudge_run_metadata(rejudge_request))+ state("starting", source_run=parent)+ log(+ f"Rejudging {parent['namespace']}/{parent['experiment_name']}/"+ f"{parent['run_id']} as {namespace}/{experiment_name}/{run_id} "+ f"with judge {judge_preset} ({judge_model})"+ )++ reload_volume()+ parent_dir = (+ pathlib.Path("/data/runs")+ / parent["namespace"]+ / parent["experiment_name"]+ / parent["run_id"]+ )+ parent_archive = parent_dir / "harbor-job.tar.gz"+ if not parent_archive.exists():+ raise FileNotFoundError(+ f"parent run has no harbor-job.tar.gz: {parent_archive}. "+ "Rejudge needs a completed parent run with a stored job archive."+ )++ metadata = load_json(parent_dir / "run-metadata.json") or {}+ benchmark = metadata.get("benchmark") or {}+ dataset = benchmark.get("dataset") if isinstance(benchmark, dict) else {}+ dataset_kind = dataset.get("kind") if isinstance(dataset, dict) else None+ dataset_name = dataset.get("name") if isinstance(dataset, dict) else None+ if dataset_kind != "registry" or not isinstance(dataset_name, str):+ raise NotImplementedError(+ "rejudge currently supports registry datasets (SWE-Atlas rf/qna). "+ f"Parent run dataset_kind={dataset_kind!r} is not yet supported."+ )++ work = pathlib.Path("/tmp/agent-evals/rejudge") / run_id+ if work.exists():+ shutil.rmtree(work)+ extract_root = work / "parent"+ extract_root.mkdir(parents=True, exist_ok=True)+ log(f"Extracting parent archive {parent_archive}")+ with tarfile.open(parent_archive, "r:gz") as archive:+ safe_extract_archive(archive, extract_root)+ parent_job_dir = extract_root / parent["run_id"]+ if not parent_job_dir.is_dir():+ # Fall back to the single top-level dir if the arcname differs.+ candidates = [p for p in extract_root.iterdir() if p.is_dir()]+ if len(candidates) != 1:+ raise FileNotFoundError(+ f"could not locate job dir inside {parent_archive}"+ )+ parent_job_dir = candidates[0]++ log(f"Downloading task packages for dataset {dataset_name}")+ subprocess.run(["harbor", "datasets", "download", dataset_name], check=True)+ tasks_root = pathlib.Path.home() / ".cache" / "harbor" / "tasks" / "packages"++ state("running", source_run=parent, judge_model=judge_model)+ out_job_dir = work / "job" / run_id+ summary = rejudge.rejudge_job(+ parent_job_dir=parent_job_dir,+ out_job_dir=out_job_dir,+ tasks_root=tasks_root,+ judge=judge,+ judge_model=judge_model,+ log=log,+ )+ volume.commit()++ archive_path = run_dir / "harbor-job.tar.gz"+ with tarfile.open(archive_path, "w:gz") as archive:+ archive.add(out_job_dir, arcname=out_job_dir.name)++ summary = {+ **summary,+ "run_id": run_id,+ "namespace": namespace,+ "experiment_name": experiment_name,+ "kind": "rejudge",+ "source_run": parent,+ "judge_preset": judge_preset,+ "status": "completed",+ "has_archive": archive_path.exists(),+ "started_at": started_at,+ "completed_at": utc_now(),+ }+ write_json(run_dir / "summary.json", summary)+ (run_dir / "READY").write_text(utc_now() + "\n")+ log(+ f"Rejudge complete: {summary['passed_count']}/"+ f"{summary['rejudged_count']} passed "+ f"({summary['failed_count']} trials could not be rejudged)"+ )+ state("completed", summary=summary)+ volume.commit()+ return summary+ except Exception as error:+ formatted = traceback.format_exc()+ log(f"FAILED: {error}\n{formatted}")+ (run_dir / "FAILED").write_text(utc_now() + "\n")+ state("failed", error=str(error), traceback=formatted)+ volume.commit()+ raisediff --git a/crates/eval_cli/zed_eval/pier_agent.py b/crates/eval_cli/zed_eval/pier_agent.pynew file mode 100644index 00000000000..0a5c07631fa--- /dev/null+++ b/crates/eval_cli/zed_eval/pier_agent.py@@ -0,0 +1,226 @@+"""Pier-native agent wrapper for Zed's eval-cli binary.++Pier (a Harbor fork) is the harness for DeepSWE, whose tasks run air-gapped+(`allow_internet = false`). Pier's installed-agent interface differs from+Harbor's — it requires `install_spec()` and, crucially, a `network_allowlist()`+the egress proxy honors so the agent can still reach the model API. Harbor's+`BaseInstalledAgent` has neither, so this is a separate, Pier-specific class+(the sanctioned cross-framework exception): everything else in the project has a+single canonical implementation, but two genuinely different harness SDKs need+two thin agent shells.++Air-gap notes:+ - The eval-cli binary is a static musl executable copied from the mounted Modal+ volume (`/data`, local — not network), so installing it needs no egress.+ - Network installs (Node, LSPs) are intentionally skipped: there's no egress to+ fetch them, and eval-cli's core tools (read/edit/terminal/...) don't need+ them. Only the model API host is allowlisted.+"""++from __future__ import annotations++import json+import shlex+from urllib.parse import urlparse++from pier.agents.installed.base import BaseInstalledAgent, with_prompt_template+from pier.environments.base import BaseEnvironment+from pier.models.agent.context import AgentContext+from pier.models.agent.install import AgentInstallSpec, InstallStep+from pier.models.agent.network import NetworkAllowlist+from pier.models.trial.paths import EnvironmentPaths++from .agent_common import (+ add_anthropic_available_models_env,+ add_openai_compatible_provider_env,+ add_zed_eval_env,+ detect_workdir,+ eval_cli_with_log_command,+ patch_command,+ populate_context_from_result,+ provider_api_env,+)++# Default model-provider API hosts the agent must reach even on air-gapped tasks.+# Mirrors benchmarks.AGENT_API_HOSTS; duplicated here so this module has no+# dependency on the orchestration package when loaded inside a sandbox.+DEFAULT_PROVIDER_DOMAINS: dict[str, list[str]] = {+ "anthropic": ["api.anthropic.com"],+ "openai": ["api.openai.com"],+ "google": [".googleapis.com"],+ "gemini": [".googleapis.com"],+ "deepseek": ["api.deepseek.com"],+ "baseten": ["inference.baseten.co"],+}+# Base-URL env vars Pier-style allowlist resolution should also honor.+BASE_URL_ENV_VARS = (+ "ANTHROPIC_BASE_URL",+ "OPENAI_BASE_URL",+ "OPENAI_API_BASE",+ "GEMINI_API_BASE",+)+++class ZedPierAgent(BaseInstalledAgent):+ """Runs Zed's headless eval-cli binary under Pier."""++ SUPPORTS_ATIF: bool = False++ @staticmethod+ def name() -> str:+ return "zed"++ def _container_binary_path(self) -> str:+ path = self._get_env("EVAL_CLI_CONTAINER_PATH")+ if not path:+ raise ValueError(+ "ZedPierAgent requires EVAL_CLI_CONTAINER_PATH (the eval-cli binary "+ "on the mounted volume, e.g. /data/builds/<id>/eval-cli)"+ )+ return path++ def install_spec(self) -> AgentInstallSpec:+ # The binary lives on the runtime-mounted volume, which isn't available at+ # image-build time, so the real placement happens in run() via+ # exec_as_root. install_spec only needs a non-empty, no-egress step.+ return AgentInstallSpec(+ agent_name=self.name(),+ version=self._version,+ steps=[+ InstallStep(+ user="root",+ run="true # eval-cli is copied from the volume at run time",+ )+ ],+ )++ def network_allowlist(self) -> NetworkAllowlist:+ """Hosts the egress proxy must allow so eval-cli can reach the model API+ on air-gapped DeepSWE tasks."""+ domains: list[str] = []++ provider = None+ if self.model_name and "/" in self.model_name:+ provider = self.model_name.split("/", 1)[0]+ domains.extend(DEFAULT_PROVIDER_DOMAINS.get(provider or "", []))++ # Any explicitly configured base URL (e.g. an OpenAI-compatible gateway+ # such as Baseten) wins/adds alongside the provider default.+ for env_var in BASE_URL_ENV_VARS:+ value = self._get_env(env_var)+ if value:+ host = urlparse(value).hostname+ if host:+ domains.append(host)++ # A Baseten (or other OpenAI-compatible) provider is wired via this JSON;+ # allow its api_url host too.+ providers_json = self._get_env("ZED_OPENAI_COMPATIBLE_PROVIDERS")+ if providers_json:+ try:+ parsed = json.loads(providers_json)+ except json.JSONDecodeError:+ parsed = {}+ for provider_config in (parsed or {}).values():+ api_url = (+ provider_config.get("api_url")+ if isinstance(provider_config, dict)+ else None+ )+ host = urlparse(api_url).hostname if api_url else None+ if host:+ domains.append(host)++ if not domains:+ # Never return an empty allowlist: without the model host the agent+ # can do nothing on an air-gapped task. Default to Anthropic.+ domains = ["api.anthropic.com"]+ return NetworkAllowlist(domains=domains)++ def _api_env(self) -> dict[str, str]:+ env = provider_api_env(self.model_name, self._get_env)+ add_openai_compatible_provider_env(+ env, self._get_env("ZED_OPENAI_COMPATIBLE_PROVIDERS")+ )+ add_anthropic_available_models_env(+ env, self._get_env("ZED_ANTHROPIC_AVAILABLE_MODELS")+ )+ add_zed_eval_env(+ env, self._extra_env, exclude={"ZED_EVAL_INSTRUCTION_SUFFIX_FILE"}+ )+ return env++ async def _detect_workdir(self, environment: BaseEnvironment) -> str:+ return await detect_workdir(+ environment,+ self.exec_as_agent,+ self._get_env,+ "Could not detect a working directory; set EVAL_CLI_WORKDIR via --ae",+ )++ @with_prompt_template+ async def run(+ self, instruction: str, environment: BaseEnvironment, context: AgentContext+ ) -> None:+ # Place the static binary from the mounted volume (no egress needed).+ await self.exec_as_root(+ environment,+ command=(+ f"install -m755 {shlex.quote(self._container_binary_path())} "+ "/usr/local/bin/eval-cli && eval-cli --help >/dev/null"+ ),+ )++ workdir = await self._detect_workdir(environment)+ agent_dir = str(EnvironmentPaths.agent_dir)+ env = self.build_process_env(self._api_env())++ parts = [+ "eval-cli",+ f"--workdir {shlex.quote(workdir)}",+ f"--output-dir {shlex.quote(agent_dir)}",+ ]+ if self.model_name:+ parts.append(f"--model {shlex.quote(self.model_name)}")+ timeout = self._get_env("EVAL_CLI_TIMEOUT")+ if timeout:+ parts.append(f"--timeout {shlex.quote(timeout)}")+ instruction_suffix_file = self._get_env("ZED_EVAL_INSTRUCTION_SUFFIX_FILE")+ if instruction_suffix_file:+ parts.append(+ f"--instruction-suffix-file {shlex.quote(instruction_suffix_file)}"+ )+ staff = self._get_env("EVAL_CLI_STAFF")+ if staff and staff.lower() == "false":+ parts.append("--no-staff")+ reasoning_effort = self._get_env("EVAL_CLI_REASONING_EFFORT")+ if reasoning_effort:+ parts.append(f"--reasoning-effort {shlex.quote(reasoning_effort)}")+ enable_thinking = self._get_env("EVAL_CLI_ENABLE_THINKING")+ if enable_thinking:+ if enable_thinking.lower() == "true":+ parts.append("--thinking true")+ elif enable_thinking.lower() == "false":+ parts.append("--thinking false")+ parts.append(f"--instruction {shlex.quote(instruction)}")++ # Tolerate exit 2 (timeout): deliver whatever the agent produced.+ await self.exec_as_agent(+ environment,+ command=eval_cli_with_log_command(+ parts,+ agent_dir + "/eval-cli.txt",+ timeout_message=None,+ ),+ env=env,+ )++ # SWE-bench-style patch for the verifier (DeepSWE tasks are git repos).+ await self.exec_as_agent(+ environment,+ command=patch_command(agent_dir),+ cwd=workdir,+ )++ def populate_context_post_run(self, context: AgentContext) -> None:+ populate_context_from_result(self.logs_dir, context, self.logger)diff --git a/crates/eval_cli/zed_eval/pyproject.toml b/crates/eval_cli/zed_eval/pyproject.tomlindex e25ec9825c0..c521651c924 100644--- a/crates/eval_cli/zed_eval/pyproject.toml+++ b/crates/eval_cli/zed_eval/pyproject.toml@@ -1,9 +1,16 @@[project]name = "zed-eval"-version = "0.1.0"-description = "Harbor agent wrapper for Zed's eval-cli"+version = "0.2.0"+description = "Harbor agent wrapper and remote orchestration CLI for Zed's eval-cli"requires-python = ">=3.12"-dependencies = ["harbor==0.13.0"]+dependencies = ["harbor==0.15.0", "modal==1.5.0"]++[project.scripts]+zed-eval = "zed_eval.cli:main"++[tool.setuptools]+packages = ["zed_eval"]+package-dir = { zed_eval = "." }[build-system]requires = ["setuptools"]diff --git a/crates/eval_cli/zed_eval/rejudge.py b/crates/eval_cli/zed_eval/rejudge.pynew file mode 100644index 00000000000..42c0b477125--- /dev/null+++ b/crates/eval_cli/zed_eval/rejudge.py@@ -0,0 +1,359 @@+"""Re-grade a finished run with a different judge, without redoing agent work.++A rejudge takes the agent output already stored in a parent run (the expensive+part: patches/answers the agent produced) and re-runs only the *judge* step with+a new judge model, producing a brand-new derived run. The original run is read+only and never modified.++This is deliberately not a re-implementation of the grader. It runs the exact+SWE-Atlas verifier script shipped in each task package (`evaluate_rubrics.py` for+RF, `evaluate_answer.py` for QnA) through the same judge proxy shim the live+in-sandbox path uses (`judge_proxy.JUDGE_PROXY_SCRIPT`), so a rejudged verdict is+produced identically to a first-pass verdict — only the judge model differs.++The verifier scripts read their inputs from fixed absolute paths (`/tests`,+`/logs/...`) and pick the judge up from `EVAL_MODEL`/`EVAL_BASE_URL`/+`EVAL_API_KEY`, so this module is meant to run inside a throwaway container (the+Modal `rejudge_controller`), where staging those absolute paths is safe. The pure+helpers (resolution, reward recombination, result patching) have no such+requirement and are unit-tested directly.+"""++from __future__ import annotations++import json+import os+import shutil+import socket+import subprocess+import sys+import time+from pathlib import Path+from typing import Any++from . import config+from .judge_proxy import JUDGE_PROXY_SCRIPT++# Each task package ships exactly one of these verifier scripts; its presence is+# how we tell an RF task from a QnA one without trusting the experiment config.+RF_VERIFIER = "evaluate_rubrics.py"+QNA_VERIFIER = "evaluate_answer.py"++PART_RF = "rf"+PART_QNA = "qna"++# Absolute paths the cached verifier scripts hardcode. Staged per trial.+TESTS_DIR = Path("/tests")+LOGS_AGENT_DIR = Path("/logs/agent")+LOGS_VERIFIER_DIR = Path("/logs/verifier")+RF_PATCH_PATH = LOGS_VERIFIER_DIR / "agent.patch"+RF_RESULTS_PATH = LOGS_VERIFIER_DIR / "rubrics_results.json"+QNA_ANSWER_PATH = LOGS_AGENT_DIR / "answer.txt"+QNA_RESULTS_PATH = LOGS_VERIFIER_DIR / "evaluation_results.json"+++def load_json(path: Path) -> Any:+ try:+ return json.loads(Path(path).read_text())+ except (OSError, json.JSONDecodeError):+ return None+++def task_tests_dir(task_name: str, task_ref: str, tasks_root: Path) -> Path:+ """Resolve a trial's cached task package `tests/` dir from its name + ref.++ Mirrors Harbor's content-addressed layout:+ `<tasks_root>/<org>/<task-id>/<ref-hash>/tests`.+ """+ org, task_id = task_name.split("/", 1)+ ref_hash = task_ref.split(":", 1)[1] if ":" in task_ref else task_ref+ return tasks_root / org / task_id / ref_hash / "tests"+++def tests_dir_for_trial(trial_dir: Path, tasks_root: Path) -> Path | None:+ config_data = load_json(trial_dir / "config.json")+ if not isinstance(config_data, dict):+ return None+ task = config_data.get("task") or {}+ name, ref = task.get("name"), task.get("ref")+ if not (name and ref):+ return None+ tests_dir = task_tests_dir(name, ref, tasks_root)+ return tests_dir if tests_dir.is_dir() else None+++def detect_part(tests_dir: Path) -> str | None:+ if (tests_dir / RF_VERIFIER).exists():+ return PART_RF+ if (tests_dir / QNA_VERIFIER).exists():+ return PART_QNA+ return None+++def recompute_rf_rewards(+ rubrics_results: dict[str, Any], prior_rewards: dict[str, Any]+) -> dict[str, float]:+ """Combine the freshly re-judged rubric verdict with the *unchanged* saved+ deterministic test reward. The agent patch didn't change, so `tests_reward`+ is invariant under rejudge; only the rubric (`must_have_pass`/agg) moves.++ The overall RF reward is `must_have_pass AND tests_reward == 1`, verified to+ hold across every cached RF trial.+ """+ must_have_pass = bool(rubrics_results.get("must_have_pass"))+ agg_score = rubrics_results.get("agg_score")+ tests_reward = float(prior_rewards.get("tests_reward") or 0.0)+ overall_pass = 1.0 if (must_have_pass and tests_reward >= 1.0) else 0.0+ return {+ "reward": overall_pass,+ "overall_pass": overall_pass,+ "tests_reward": tests_reward,+ "must_have_pass": 1.0 if must_have_pass else 0.0,+ "rubrics_agg_score": float(agg_score) if agg_score is not None else 0.0,+ }+++def recompute_qna_rewards(evaluation_results: dict[str, Any]) -> dict[str, float]:+ """QnA has no deterministic test component; the reward is just the judged+ pass verdict, matching the single-key `{"reward": ...}` production shape."""+ return {"reward": 1.0 if bool(evaluation_results.get("pass")) else 0.0}+++def proxy_environment(judge: config.JudgeConfig) -> dict[str, str]:+ """Env the judge proxy subprocess reads. The auth key it forwards+ (`judge.auth_env`, e.g. BASETEN_API_KEY) must already be present in the+ process environment via the mounted LLM-providers secret."""+ env = {+ "ZED_JUDGE_UPSTREAM": judge.upstream,+ "ZED_JUDGE_AUTH_ENV": judge.auth_env,+ }+ if judge.max_tokens is not None:+ env["ZED_JUDGE_MAX_TOKENS"] = str(judge.max_tokens)+ return env+++def verifier_environment(judge_model: str, port: int) -> dict[str, str]:+ """Env the cached verifier reads to find the judge.++ This mirrors the verifier env injected by the benchmark harness command,+ pointed at the local rejudge proxy.+ """+ return {+ "EVAL_MODEL": judge_model,+ "EVAL_BASE_URL": f"http://127.0.0.1:{port}/v1",+ "EVAL_API_KEY": "unused-by-agent-evals-proxy",+ }+++class JudgeProxy:+ """Runs `judge_proxy.JUDGE_PROXY_SCRIPT` as a local subprocess for the life+ of a rejudge job. One proxy serves every trial (they all target the same+ judge), mirroring the single in-sandbox daemon."""++ def __init__(self, judge: config.JudgeConfig, port: int = 8089) -> None:+ self._judge = judge+ self._port = port+ self._process: subprocess.Popen[bytes] | None = None+ self._script_path = Path("/tmp/zed_judge_proxy.py")++ @property+ def port(self) -> int:+ return self._port++ def __enter__(self) -> JudgeProxy:+ self._script_path.write_text(JUDGE_PROXY_SCRIPT)+ env = os.environ.copy()+ env.update(proxy_environment(self._judge))+ env["ZED_JUDGE_PROXY_PORT"] = str(self._port)+ self._process = subprocess.Popen(+ [sys.executable, str(self._script_path)],+ env=env,+ stdout=subprocess.DEVNULL,+ stderr=subprocess.DEVNULL,+ )+ self._wait_for_port()+ return self++ def __exit__(self, *_exc: object) -> None:+ if self._process is not None:+ self._process.terminate()+ try:+ self._process.wait(timeout=10)+ except subprocess.TimeoutExpired:+ self._process.kill()++ def _wait_for_port(self, timeout_secs: float = 30.0) -> None:+ deadline = time.time() + timeout_secs+ while time.time() < deadline:+ if self._process is not None and self._process.poll() is not None:+ raise RuntimeError("judge proxy exited before becoming ready")+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:+ probe.settimeout(0.5)+ if probe.connect_ex(("127.0.0.1", self._port)) == 0:+ return+ time.sleep(0.25)+ raise TimeoutError(f"judge proxy did not open port {self._port} in time")+++def _stage_tests_dir(tests_dir: Path) -> None:+ """Point the absolute `/tests` path at this trial's package. Reset per trial+ so each verifier reads its own rubrics/prompts."""+ if TESTS_DIR.is_symlink() or TESTS_DIR.exists():+ if TESTS_DIR.is_dir() and not TESTS_DIR.is_symlink():+ shutil.rmtree(TESTS_DIR)+ else:+ TESTS_DIR.unlink()+ TESTS_DIR.symlink_to(tests_dir)+++def _run_verifier(+ part: str, tests_dir: Path, judge_model: str, port: int+) -> dict[str, Any]:+ """Stage the verifier's expected inputs and run the real cached script."""+ LOGS_AGENT_DIR.mkdir(parents=True, exist_ok=True)+ LOGS_VERIFIER_DIR.mkdir(parents=True, exist_ok=True)+ _stage_tests_dir(tests_dir)++ env = os.environ.copy()+ env.update(verifier_environment(judge_model, port))+ script = tests_dir / (RF_VERIFIER if part == PART_RF else QNA_VERIFIER)+ results_path = RF_RESULTS_PATH if part == PART_RF else QNA_RESULTS_PATH+ if results_path.exists():+ results_path.unlink()++ completed = subprocess.run(+ [sys.executable, str(script)],+ env=env,+ capture_output=True,+ text=True,+ )+ results = load_json(results_path)+ if not isinstance(results, dict):+ raise RuntimeError(+ f"verifier produced no results (exit {completed.returncode}): "+ f"{completed.stderr.strip()[:500]}"+ )+ return results+++def rejudge_trial(+ trial_dir: Path,+ tasks_root: Path,+ judge: config.JudgeConfig,+ judge_model: str,+ port: int,+) -> dict[str, Any]:+ """Re-grade one trial in place (the trial dir is a copy in the derived run).++ Returns a per-trial record; on any recoverable problem it returns+ `{"ok": False, "error": ...}` so one bad trial can't sink the whole job.+ """+ task_name = trial_dir.name+ tests_dir = tests_dir_for_trial(trial_dir, tasks_root)+ if tests_dir is None:+ return {"task": task_name, "ok": False, "error": "task package not found"}+ part = detect_part(tests_dir)+ if part is None:+ return {"task": task_name, "ok": False, "error": "no verifier in task package"}++ # The agent output is already on disk from the parent run; copy it to the+ # path the verifier expects. RF reads the patch, QnA the delivered answer.+ if part == PART_RF:+ source = trial_dir / "verifier" / "agent.patch"+ if not source.exists():+ source = trial_dir / "verifier" / "agent_source_only.patch"+ if not source.exists():+ return {"task": task_name, "ok": False, "error": "no agent.patch"}+ RF_PATCH_PATH.parent.mkdir(parents=True, exist_ok=True)+ shutil.copyfile(source, RF_PATCH_PATH)+ else:+ source = trial_dir / "agent" / "answer.txt"+ if not source.exists():+ return {"task": task_name, "ok": False, "error": "no answer.txt"}+ QNA_ANSWER_PATH.parent.mkdir(parents=True, exist_ok=True)+ shutil.copyfile(source, QNA_ANSWER_PATH)++ try:+ results = _run_verifier(part, tests_dir, judge_model, port)+ except (RuntimeError, OSError) as error:+ return {"task": task_name, "ok": False, "error": str(error)}++ prior_rewards = (+ (load_json(trial_dir / "result.json") or {}).get("verifier_result") or {}+ ).get("rewards") or {}+ if part == PART_RF:+ new_rewards = recompute_rf_rewards(results, prior_rewards)+ verifier_results_name = "rubrics_results.json"+ else:+ new_rewards = recompute_qna_rewards(results)+ verifier_results_name = "evaluation_results.json"++ # Persist the fresh verifier output and verdict into the derived trial dir so+ # report.py reads the new judge's result with no special-casing.+ (trial_dir / "verifier").mkdir(parents=True, exist_ok=True)+ (trial_dir / "verifier" / verifier_results_name).write_text(+ json.dumps(results, indent=2)+ )+ patch_result_rewards(trial_dir / "result.json", new_rewards)+ return {"task": task_name, "ok": True, "part": part, "rewards": new_rewards}+++def patch_result_rewards(result_path: Path, new_rewards: dict[str, float]) -> None:+ """Overwrite only `verifier_result.rewards` in a trial's `result.json`,+ leaving the agent's resource metrics (tokens/steps/tool calls) untouched —+ the agent didn't re-run, only the judge did."""+ result = load_json(result_path)+ if not isinstance(result, dict):+ result = {}+ verifier_result = result.get("verifier_result")+ if not isinstance(verifier_result, dict):+ verifier_result = {}+ verifier_result["rewards"] = new_rewards+ result["verifier_result"] = verifier_result+ Path(result_path).write_text(json.dumps(result, indent=2))+++def rejudge_job(+ parent_job_dir: Path,+ out_job_dir: Path,+ tasks_root: Path,+ judge: config.JudgeConfig,+ judge_model: str,+ port: int = 8089,+ log: Any = print,+) -> dict[str, Any]:+ """Copy a parent run's Harbor job dir into the derived run and re-judge every+ trial in place. Returns a summary suitable for the run's `summary.json`."""+ if out_job_dir.exists():+ shutil.rmtree(out_job_dir)+ shutil.copytree(parent_job_dir, out_job_dir)++ trial_dirs = sorted(+ path+ for path in out_job_dir.iterdir()+ if path.is_dir() and (path / "result.json").exists()+ )+ records: list[dict[str, Any]] = []+ with JudgeProxy(judge, port=port) as proxy:+ for index, trial_dir in enumerate(trial_dirs, start=1):+ record = rejudge_trial(+ trial_dir, tasks_root, judge, judge_model, proxy.port+ )+ records.append(record)+ status = "ok" if record["ok"] else f"FAILED: {record.get('error')}"+ log(f"[{index}/{len(trial_dirs)}] {trial_dir.name}: {status}")++ rejudged = sum(1 for record in records if record["ok"])+ passed = sum(+ 1+ for record in records+ if record["ok"] and record["rewards"].get("reward", 0.0) >= 1.0+ )+ return {+ "trial_count": len(trial_dirs),+ "rejudged_count": rejudged,+ "failed_count": len(trial_dirs) - rejudged,+ "passed_count": passed,+ "judge_model": judge_model,+ "trials": records,+ }diff --git a/crates/eval_cli/zed_eval/report.py b/crates/eval_cli/zed_eval/report.pynew file mode 100644index 00000000000..bd809e3927f--- /dev/null+++ b/crates/eval_cli/zed_eval/report.py@@ -0,0 +1,484 @@+"""Success-conditioned metrics for fetched benchmark runs.++This reads a fetched Harbor/Pier job directory and reports, for each run:++ - success rate (with SEM when there are repeated attempts)+ - token consumption *for the passing subset* (and overall, for comparison)+ - tool calls *for the passing subset*, aggregate and broken down by tool+ - total steps (agent turns) *for the passing subset*++The per-trial record has two sources, one canonical each, with no fallbacks:++ - The verdict (pass/fail/errored) comes from the harness' own trial result+ (verifier rewards) — rubric judge for SWE-Atlas, test scripts for+ Terminal-Bench and DeepSWE.+ - Resource usage (tokens / steps / tool calls) comes from the Zed agent's+ `result.json`, which eval-cli emits identically on every harness. There is+ deliberately no secondary parser: if a metric isn't in `result.json`, it's+ reported as absent rather than reconstructed from another file.+"""++from __future__ import annotations++import json+import math+import statistics+import sys+from collections import Counter, defaultdict+from dataclasses import dataclass, field+from pathlib import Path+from typing import Any++# Effective-token weights (cache reads are ~10x cheaper, output ~4x input).+ET_INPUT = 1.0+ET_CACHE_READ = 0.1+ET_CACHE_CREATION = 1.0+ET_OUTPUT = 4.0++HARBOR_TRIAL_KEYS = {"task_name", "trial_name", "verifier_result"}+ZED_RESULT_KEYS = {"status", "duration_secs"}+++def load_json(path: Path) -> Any:+ try:+ return json.loads(path.read_text())+ except (OSError, json.JSONDecodeError) as error:+ print(f"warning: could not parse {path}: {error}", file=sys.stderr)+ return None+++def find_trial_dirs(job_dir: Path) -> list[Path]:+ trial_dirs: set[Path] = set()+ for name in ("result.json", "results.json"):+ for path in job_dir.rglob(name):+ if path.parent == job_dir:+ continue+ data = load_json(path)+ if isinstance(data, dict) and HARBOR_TRIAL_KEYS & data.keys():+ trial_dirs.add(path.parent)+ return sorted(trial_dirs)+++def load_harbor_trial_result(trial_dir: Path) -> dict | None:+ for name in ("result.json", "results.json"):+ path = trial_dir / name+ if path.exists():+ data = load_json(path)+ if isinstance(data, dict) and HARBOR_TRIAL_KEYS & data.keys():+ return data+ return None+++def load_zed_result(trial_dir: Path) -> dict | None:+ for path in sorted(trial_dir.rglob("result.json")):+ if path.parent == trial_dir:+ continue+ data = load_json(path)+ if isinstance(data, dict) and ZED_RESULT_KEYS & data.keys():+ return data+ return None+++def trial_verdict(+ harbor_result: dict, *, timeout_is_failure: bool = False+) -> tuple[bool | None, str | None]:+ """Returns (passed, error_reason). `passed` is None when the trial errored+ (harness/infra exception or no verifier reward); infra errors are not+ failures.++ `timeout_is_failure` reclassifies an agent timeout as a real failure+ (passed=False) rather than an excluded error: on test-scored benchmarks an+ `AgentTimeoutError` means the agent didn't solve the task within its intended+ time budget, which is an agent failure, not infra noise."""+ exception_info = harbor_result.get("exception_info")+ if exception_info:+ reason = "exception"+ if isinstance(exception_info, dict):+ reason = exception_info.get("exception_type") or "exception"+ if timeout_is_failure and reason == "AgentTimeoutError":+ return False, reason+ return None, reason++ verifier = harbor_result.get("verifier_result")+ if not isinstance(verifier, dict):+ return None, "no verifier result"+ rewards = verifier.get("rewards")+ if not isinstance(rewards, dict) or not rewards:+ return None, "no rewards"+ try:+ # Prefer the strict leaderboard metric. max(rewards) would wrongly count+ # a trial as passed when only a component reward (e.g. tests_reward) hit+ # 1.0 on SWE-Atlas RF.+ if "reward" in rewards:+ return float(rewards["reward"]) >= 1.0, None+ return max(float(value) for value in rewards.values()) >= 1.0, None+ except (TypeError, ValueError):+ return None, "unparsable rewards"+++@dataclass+class TrialRecord:+ task_name: str+ started_at: str | None+ passed: bool | None+ error_reason: str | None+ input_tokens: int = 0+ output_tokens: int = 0+ cache_read_input_tokens: int = 0+ cache_creation_input_tokens: int = 0+ step_count: int | None = None+ total_tool_calls: int | None = None+ tool_calls: dict[str, int] = field(default_factory=dict)+ duration_secs: float | None = None+ status: str | None = None+ model: str | None = None+ # True when the agent's result.json was found (i.e. the trial produced+ # metrics). False for trials hard-killed on timeout before writing it.+ has_result: bool = False++ @property+ def effective_tokens(self) -> float:+ return (+ ET_INPUT * self.input_tokens+ + ET_CACHE_READ * self.cache_read_input_tokens+ + ET_CACHE_CREATION * self.cache_creation_input_tokens+ + ET_OUTPUT * self.output_tokens+ )++ @property+ def total_tokens(self) -> int:+ return (+ self.input_tokens+ + self.output_tokens+ + self.cache_read_input_tokens+ + self.cache_creation_input_tokens+ )+++def extract_trial(+ trial_dir: Path, *, timeout_is_failure: bool = False+) -> TrialRecord | None:+ harbor_result = load_harbor_trial_result(trial_dir)+ if harbor_result is None:+ return None++ passed, error_reason = trial_verdict(+ harbor_result, timeout_is_failure=timeout_is_failure+ )+ record = TrialRecord(+ task_name=harbor_result.get("task_name") or trial_dir.name,+ started_at=harbor_result.get("started_at"),+ passed=passed,+ error_reason=error_reason,+ )++ zed_result = load_zed_result(trial_dir)+ if zed_result:+ record.has_result = True+ record.status = zed_result.get("status")+ record.model = zed_result.get("model")+ record.input_tokens = int(zed_result.get("input_tokens") or 0)+ record.output_tokens = int(zed_result.get("output_tokens") or 0)+ record.cache_read_input_tokens = int(+ zed_result.get("cache_read_input_tokens") or 0+ )+ record.cache_creation_input_tokens = int(+ zed_result.get("cache_creation_input_tokens") or 0+ )+ duration = zed_result.get("duration_secs")+ if isinstance(duration, (int, float)):+ record.duration_secs = float(duration)+ # The canonical metrics source: counts eval-cli writes into result.json.+ if isinstance(zed_result.get("step_count"), int):+ record.step_count = zed_result["step_count"]+ if isinstance(zed_result.get("tool_call_count"), int):+ record.total_tool_calls = zed_result["tool_call_count"]+ if isinstance(zed_result.get("tool_calls"), dict):+ record.tool_calls = {+ str(name): int(count)+ for name, count in zed_result["tool_calls"].items()+ if isinstance(count, int)+ }++ return record+++def sample_sem(values: list[float]) -> float | None:+ n = len(values)+ if n < 2:+ return None+ mean = sum(values) / n+ return math.sqrt(sum((x - mean) ** 2 for x in values) / (n * (n - 1)))+++def pass_rate_with_sem(+ scored: list[TrialRecord],+) -> tuple[float | None, float | None, int]:+ """Pass rate grouped by task. With >=2 attempts per task the rate is the+ mean of per-attempt pass rates with sample SEM across attempts; otherwise+ it's the plain rate (SEM undefined)."""+ if not scored:+ return None, None, 0+ by_task: dict[str, list[TrialRecord]] = defaultdict(list)+ for trial in scored:+ by_task[trial.task_name].append(trial)+ for attempts in by_task.values():+ attempts.sort(key=lambda t: t.started_at or "")++ max_attempts = max(len(attempts) for attempts in by_task.values())+ if max_attempts < 2:+ rate = sum(1 for t in scored if t.passed) / len(scored)+ return rate, None, 1++ attempt_rates: list[float] = []+ for i in range(max_attempts):+ attempt_trials = [+ attempts[i] for attempts in by_task.values() if len(attempts) > i+ ]+ if attempt_trials:+ attempt_rates.append(+ sum(1 for t in attempt_trials if t.passed) / len(attempt_trials)+ )+ if not attempt_rates:+ return None, None, max_attempts+ mean = sum(attempt_rates) / len(attempt_rates)+ return mean, sample_sem(attempt_rates), max_attempts+++def _mean(values: list[float]) -> float | None:+ return statistics.mean(values) if values else None+++def slice_metrics(records: list[TrialRecord]) -> dict[str, Any]:+ """Mean resource usage over a set of trials (used for both the whole run+ and the passing subset)."""+ empty = {+ "n": 0,+ "n_with_metrics": 0,+ "mean_steps": None,+ "mean_total_tokens": None,+ "mean_effective_tokens": None,+ "mean_tool_calls": None,+ "mean_tool_calls_by_tool": {},+ "mean_duration_secs": None,+ }+ if not records:+ return empty+ # Resource means are taken only over trials that actually produced metrics+ # (result.json present). Trials hard-killed on timeout count toward the pass+ # rate but must not drag token/step means toward zero.+ with_result = [r for r in records if r.has_result]+ steps = [float(r.step_count) for r in records if r.step_count is not None]+ tool_totals = [+ float(r.total_tool_calls) for r in records if r.total_tool_calls is not None+ ]+ per_tool_sum: Counter[str] = Counter()+ for record in with_result:+ per_tool_sum.update(record.tool_calls)+ per_tool_n = len(with_result)+ durations = [r.duration_secs for r in records if r.duration_secs is not None]+ return {+ "n": len(records),+ "n_with_metrics": len(with_result),+ "mean_steps": _mean(steps),+ "mean_total_tokens": _mean([float(r.total_tokens) for r in with_result]),+ "mean_effective_tokens": _mean([r.effective_tokens for r in with_result]),+ "mean_tool_calls": _mean(tool_totals),+ "mean_tool_calls_by_tool": (+ {+ name: per_tool_sum[name] / per_tool_n+ for name in sorted(per_tool_sum, key=lambda n: -per_tool_sum[n])+ }+ if per_tool_n+ else {}+ ),+ "mean_duration_secs": _mean(durations),+ }+++def build_report(+ job_dir: Path,+ *,+ label: str | None = None,+ timeout_is_failure: bool = False,+) -> dict[str, Any]:+ records = [+ record+ for trial_dir in find_trial_dirs(job_dir)+ if (record := extract_trial(trial_dir, timeout_is_failure=timeout_is_failure))+ is not None+ ]+ scored = [r for r in records if r.passed is not None]+ errored = [r for r in records if r.passed is None]+ passing = [r for r in scored if r.passed]+ pass_rate, pass_sem, n_attempts = pass_rate_with_sem(scored)++ models = Counter(r.model for r in records if r.model)+ statuses = Counter(r.status for r in records if r.status)++ return {+ "label": label or job_dir.name,+ "job_dir": str(job_dir),+ "n_trials": len(records),+ "n_scored": len(scored),+ "n_passed": len(passing),+ "n_failed": len(scored) - len(passing),+ "n_errored": len(errored),+ "n_attempts": n_attempts,+ "pass_rate": pass_rate,+ "pass_sem": pass_sem,+ "resolved_models": dict(models),+ "agent_statuses": dict(statuses),+ # The headline numbers the user asked for: resource usage conditioned on+ # success, with the overall slice alongside for comparison.+ "on_success": slice_metrics(passing),+ "overall": slice_metrics(scored),+ "errored_trials": [+ {"task_name": r.task_name, "reason": r.error_reason} for r in errored+ ],+ }+++def _fmt(value: float | None, places: int = 1) -> str:+ return "n/a" if value is None else f"{value:,.{places}f}"+++def _fmt_rate(rate: float | None, sem: float | None) -> str:+ if rate is None:+ return "n/a"+ if sem is None:+ return f"{rate * 100:.1f}%"+ return f"{rate * 100:.1f}% ± {sem * 100:.1f}"+++def format_report(report: dict[str, Any]) -> str:+ lines: list[str] = []+ lines.append(f"# {report['label']}")+ lines.append("")+ lines.append(+ f"- Success rate: {_fmt_rate(report['pass_rate'], report['pass_sem'])} "+ f"({report['n_passed']}/{report['n_scored']} scored; "+ f"{report['n_errored']} errored)"+ )+ if report["resolved_models"]:+ models = ", ".join(+ f"{name} x{count}" for name, count in report["resolved_models"].items()+ )+ lines.append(f"- Models: {models}")+ lines.append("")++ on_success = report["on_success"]+ overall = report["overall"]+ lines.append("| Metric (mean) | On success | Overall (scored) |")+ lines.append("| --- | --- | --- |")+ lines.append(+ f"| Steps | {_fmt(on_success['mean_steps'])} | {_fmt(overall['mean_steps'])} |"+ )+ lines.append(+ f"| Total tokens | {_fmt(on_success['mean_total_tokens'], 0)} "+ f"| {_fmt(overall['mean_total_tokens'], 0)} |"+ )+ lines.append(+ f"| Effective tokens | {_fmt(on_success['mean_effective_tokens'], 0)} "+ f"| {_fmt(overall['mean_effective_tokens'], 0)} |"+ )+ lines.append(+ f"| Tool calls | {_fmt(on_success['mean_tool_calls'])} "+ f"| {_fmt(overall['mean_tool_calls'])} |"+ )+ lines.append(+ f"| Duration (s) | {_fmt(on_success['mean_duration_secs'])} "+ f"| {_fmt(overall['mean_duration_secs'])} |"+ )+ lines.append("")++ if on_success["mean_tool_calls_by_tool"]:+ lines.append("Tool calls by tool (mean per passing trial):")+ lines.append("")+ lines.append("| Tool | On success | Overall (scored) |")+ lines.append("| --- | --- | --- |")+ overall_by_tool = overall["mean_tool_calls_by_tool"]+ for name, value in on_success["mean_tool_calls_by_tool"].items():+ lines.append(+ f"| {name} | {_fmt(value, 2)} | {_fmt(overall_by_tool.get(name), 2)} |"+ )+ lines.append("")++ if report["errored_trials"]:+ lines.append(f"Errored trials ({len(report['errored_trials'])}):")+ for entry in report["errored_trials"][:20]:+ lines.append(f" - {entry['task_name']}: {entry['reason']}")+ if len(report["errored_trials"]) > 20:+ lines.append(f" ... and {len(report['errored_trials']) - 20} more")+ lines.append("")++ return "\n".join(lines)+++def locate_job_dir(args: Any) -> Path:+ """Resolve the fetched Harbor/Pier job directory for a `report` invocation,+ fetching it first when asked."""+ job_dir = getattr(args, "job_dir", None)+ if job_dir:+ return Path(job_dir)++ run_id = getattr(args, "run_id", None)+ if not run_id:+ raise ValueError("provide a run_id or --job-dir")+ jobs_dir = Path(getattr(args, "jobs_dir", Path.home() / ".cache/harbor/jobs"))+ candidate = jobs_dir / run_id++ if getattr(args, "fetch", False) or not candidate.exists():+ # Reuse the existing fetch command to download + extract the archive.+ from .volume import command_fetch++ command_fetch(args)+ if not candidate.exists():+ raise ValueError(+ f"no fetched job at {candidate}. Run `zed-eval fetch {run_id} "+ f"--experiment-name <benchmark>` first, or pass --job-dir."+ )+ return candidate+++def _timeout_is_failure(args: Any) -> bool:+ """Agent timeouts count as failures on test-scored benchmarks. Derived from+ the benchmark's scoring (via --experiment-name), with an explicit override."""+ override = getattr(args, "timeouts_as_failures", None)+ if override is not None:+ return bool(override)+ from . import benchmarks++ experiment = getattr(args, "experiment_name", None)+ benchmark = benchmarks.BENCHMARKS.get(experiment) if experiment else None+ return bool(benchmark and benchmark.scoring == benchmarks.SCORING_TESTS)+++def _fill_run_location_from_index(args: Any) -> None:+ """Backfill --experiment-name/--namespace from the local run index so a bare+ run id is enough for `report`. Best-effort and only fills what's missing."""+ run_id = getattr(args, "run_id", None)+ if not run_id or getattr(args, "experiment_name", None):+ return+ from . import run_index++ entry = run_index.lookup(run_id)+ if not entry:+ return+ args.experiment_name = entry.get("experiment_name")+ if not getattr(args, "namespace", None):+ args.namespace = entry.get("namespace")+++def command_report(args: Any) -> int:+ _fill_run_location_from_index(args)+ job_dir = locate_job_dir(args)+ label = getattr(args, "experiment_name", None) or getattr(args, "run_id", None)+ report = build_report(+ job_dir, label=label, timeout_is_failure=_timeout_is_failure(args)+ )+ if getattr(args, "as_json", False):+ print(json.dumps(report, indent=2))+ else:+ print(format_report(report))+ return 0diff --git a/crates/eval_cli/zed_eval/run_index.py b/crates/eval_cli/zed_eval/run_index.pynew file mode 100644index 00000000000..7fcd6864c07--- /dev/null+++ b/crates/eval_cli/zed_eval/run_index.py@@ -0,0 +1,168 @@+"""Local, launch-time index of runs started from this machine.++Every launch writes a tiny record mapping a run id to where it lives on the+volume (namespace + experiment) plus a little metadata. This lets `status`,+`logs`, `fetch`, `report`, and `rejudge` locate a run from just its id, so the+operator no longer has to repeat `--namespace`/`--experiment-name`.++It is deliberately best-effort: a missing, unreadable, or unwritable index never+blocks a launch and never crashes a lookup. Commands that can't find a run in+the index fall back to the explicit `--experiment-name`/`--namespace` flags with+a friendly hint.+"""++from __future__ import annotations++import json+import os+import tempfile+from pathlib import Path+from typing import Any++INDEX_VERSION = 1+MAX_ENTRIES = 500+REQUIRED_FIELDS = ("run_id", "namespace", "experiment_name")+OPTIONAL_FIELDS = (+ "volume",+ "agent_model",+ "judge_preset",+ "build_id",+ "suite_id",+ "kind",+ "created_at",+)+++def index_path() -> Path:+ override = os.environ.get("AGENT_EVALS_RUN_INDEX")+ if override:+ return Path(override).expanduser()+ cache_home = os.environ.get("XDG_CACHE_HOME")+ base = Path(cache_home).expanduser() if cache_home else Path.home() / ".cache"+ return base / "agent-evals" / "run-index.json"+++def _string_field(data: dict[str, Any], field: str) -> str | None:+ value = data.get(field)+ if value is None:+ return None+ value = str(value)+ return value if value else None+++def _normalize_entry(data: dict[str, Any]) -> dict[str, Any] | None:+ entry = {}+ for field in REQUIRED_FIELDS:+ value = _string_field(data, field)+ if value is None:+ return None+ entry[field] = value+ for field in OPTIONAL_FIELDS:+ value = data.get(field)+ if value is not None:+ entry[field] = value+ return entry+++def _load_entries() -> list[dict[str, Any]]:+ try:+ data = json.loads(index_path().read_text(encoding="utf-8"))+ except (OSError, json.JSONDecodeError):+ return []+ raw_entries = data.get("runs") if isinstance(data, dict) else data+ if not isinstance(raw_entries, list):+ return []++ entries = []+ for raw_entry in raw_entries:+ if not isinstance(raw_entry, dict):+ continue+ entry = _normalize_entry(raw_entry)+ if entry is not None:+ entries.append(entry)+ return entries+++def _entry_from_request(run_request: dict[str, Any]) -> dict[str, Any] | None:+ entry = _normalize_entry(+ {+ "run_id": run_request.get("run_id"),+ "namespace": run_request.get("namespace"),+ "experiment_name": run_request.get("experiment_name"),+ "volume": run_request.get("volume_name"),+ "agent_model": run_request.get("agent_model"),+ "judge_preset": run_request.get("judge_preset"),+ "build_id": run_request.get("build_id"),+ "suite_id": run_request.get("suite_id"),+ "kind": run_request.get("kind"),+ "created_at": run_request.get("created_at"),+ }+ )+ return entry+++def _write_entries(entries: list[dict[str, Any]]) -> None:+ path = index_path()+ payload = json.dumps({"version": INDEX_VERSION, "runs": entries}, indent=2) + "\n"+ path.parent.mkdir(parents=True, exist_ok=True)+ temporary_path: Path | None = None+ try:+ with tempfile.NamedTemporaryFile(+ "w",+ encoding="utf-8",+ dir=path.parent,+ prefix=f".{path.name}.",+ suffix=".tmp",+ delete=False,+ ) as temporary_file:+ temporary_path = Path(temporary_file.name)+ temporary_file.write(payload)+ os.replace(temporary_path, path)+ except OSError:+ if temporary_path is not None:+ temporary_path.unlink(missing_ok=True)+ raise+++def record_run(run_request: dict[str, Any]) -> None:+ """Append (or refresh) the index entry for a launched run.++ Accepts the same request dict the controller is spawned with. Silently does+ nothing if the essential fields are missing or the index can't be written —+ bookkeeping must never abort a launch.+ """+ entry = _entry_from_request(run_request)+ if entry is None:+ return++ entries = [+ item for item in _load_entries() if item.get("run_id") != entry["run_id"]+ ]+ entries.append(entry)+ entries = entries[-MAX_ENTRIES:]+ try:+ _write_entries(entries)+ except OSError:+ pass+++def lookup(run_id: str | None) -> dict[str, Any] | None:+ """Return the most recently recorded entry for `run_id`, if any."""+ if not run_id:+ return None+ for entry in reversed(_load_entries()):+ if entry["run_id"] == run_id:+ return entry+ return None+++def recent(limit: int = 20) -> list[dict[str, Any]]:+ """Return up to `limit` most-recent entries, newest first."""+ if limit <= 0:+ return []+ return list(reversed(_load_entries()[-limit:]))+++def most_recent() -> dict[str, Any] | None:+ entries = recent(1)+ return entries[0] if entries else Nonediff --git a/crates/eval_cli/zed_eval/source.py b/crates/eval_cli/zed_eval/source.pynew file mode 100644index 00000000000..dbdb054604d--- /dev/null+++ b/crates/eval_cli/zed_eval/source.py@@ -0,0 +1,332 @@+from __future__ import annotations++import hashlib+import json+import os+import re+import subprocess+from pathlib import Path+from typing import Any++BUILD_SCHEMA_VERSION = "agent-evals-build-v3"+SOURCE_SCHEMA_VERSION = "agent-evals-source-v1"+BUILD_TARGET = "x86_64-unknown-linux-musl"+BUILD_IMAGE_RECIPE_VERSION = "agent-evals-build-image-v1"+RUST_VERSION = "1.95.0"+RUST_IMAGE_TAG = f"rust:{RUST_VERSION}"+RUST_IMAGE_DIGEST = (+ "sha256:f49565f188ee00bc2a18dd418183f2c5f23ef7d6e691890517ed341a598f67c3"+)+RUST_IMAGE = f"{RUST_IMAGE_TAG}@{RUST_IMAGE_DIGEST}"+ZIG_VERSION = "0.15.2"+CARGO_ZIGBUILD_VERSION = "0.22.3"+DEFAULT_REPO_URL = os.environ.get(+ "AGENT_EVALS_REPO_URL", "https://github.com/zed-industries/zed.git"+)+++def package_root() -> Path:+ return Path(__file__).resolve().parent+++def eval_cli_root() -> Path:+ return package_root().parent+++def repo_root() -> Path:+ try:+ result = subprocess.run(+ ["git", "rev-parse", "--show-toplevel"],+ cwd=eval_cli_root(),+ check=True,+ capture_output=True,+ text=True,+ )+ return Path(result.stdout.strip())+ except (OSError, subprocess.CalledProcessError):+ return eval_cli_root().parents[1]+++def git_output(args: list[str], cwd: Path | None = None) -> str:+ result = subprocess.run(+ ["git", *args],+ cwd=cwd or repo_root(),+ check=True,+ capture_output=True,+ text=True,+ )+ return result.stdout.strip()+++def git_bytes(args: list[str], cwd: Path | None = None) -> bytes:+ result = subprocess.run(+ ["git", *args],+ cwd=cwd or repo_root(),+ check=True,+ capture_output=True,+ )+ return result.stdout+++def git_path_list(args: list[str]) -> list[str]:+ data = git_bytes([*args, "-z"])+ return [path.decode("utf-8") for path in data.split(b"\0") if path]+++def current_base_sha() -> str:+ return git_output(["rev-parse", "HEAD"])+++def commit_present(sha: str) -> bool:+ """Whether `sha` resolves to a commit object in the local repo."""+ result = subprocess.run(+ ["git", "-C", repo_root(), "cat-file", "-e", f"{sha}^{{commit}}"],+ capture_output=True,+ )+ return result.returncode == 0+++def base_sha_on_main(base_sha: str, repo_url: str) -> bool:+ """Whether `base_sha` is reachable from origin/main.++ Resolves origin/main's tip against the remote (so it doesn't depend on the+ caller's possibly-stale local refs), fetching main if the objects aren't+ present locally, then checks ancestry. Raises only when the main tip can't be+ made available locally (a network/remote problem), so callers can+ distinguish that from a definitive "not on main".+ """+ main_sha = resolve_remote_ref(repo_url, "main")+ if base_sha.lower() == main_sha.lower():+ return True+ if not (commit_present(base_sha) and commit_present(main_sha)):+ subprocess.run(+ ["git", "-C", repo_root(), "fetch", "--quiet", repo_url, "main"],+ capture_output=True,+ )+ if not commit_present(main_sha):+ raise RuntimeError(+ f"origin/main tip {main_sha[:12]} unavailable locally after fetch"+ )+ if not commit_present(base_sha):+ # Not present even after pulling main's history -> not reachable from main.+ return False+ ancestry = subprocess.run(+ ["git", "-C", repo_root(), "merge-base", "--is-ancestor", base_sha, main_sha],+ capture_output=True,+ )+ if ancestry.returncode in (0, 1):+ return ancestry.returncode == 0+ raise RuntimeError(+ (ancestry.stderr or b"").decode(errors="replace").strip()+ or "git merge-base --is-ancestor failed"+ )+++def resolve_git_ref(ref: str) -> str:+ return git_output(["rev-parse", "--verify", f"{ref}^{{commit}}"])+++_FULL_SHA_RE = re.compile(r"[0-9a-fA-F]{40}")+++def resolve_remote_ref(repo_url: str, ref: str) -> str:+ """Resolve a git ref/tag/branch/SHA to a canonical commit SHA against the+ *remote* repo, so the resulting build id is identical for every launcher+ regardless of what they happen to have fetched locally.++ A full 40-char SHA is already canonical and is returned (lowercased) without+ a network call. For named refs and tags this uses `git ls-remote`, preferring+ the peeled commit of an annotated tag.+ """+ if _FULL_SHA_RE.fullmatch(ref):+ return ref.lower()+ output = git_output(+ [+ "ls-remote",+ repo_url,+ ref,+ f"refs/tags/{ref}",+ f"refs/heads/{ref}",+ ]+ )+ entries: list[tuple[str, str]] = []+ for line in output.splitlines():+ sha, _, name = line.partition("\t")+ if sha and name:+ entries.append((name, sha))+ # An annotated tag yields both the tag object and a peeled "<ref>^{}" entry+ # pointing at the underlying commit; the commit is what we want to build.+ for name, sha in entries:+ if name.endswith("^{}"):+ return sha+ if entries:+ return entries[0][1]+ raise ValueError(+ f"could not resolve ref '{ref}' against {repo_url}. "+ "Push the commit/tag, or pass a full commit SHA."+ )+++def current_ref_name() -> str | None:+ try:+ ref_name = git_output(["rev-parse", "--abbrev-ref", "HEAD"])+ except (OSError, subprocess.CalledProcessError):+ return None+ return None if ref_name == "HEAD" else ref_name+++def current_tracked_patch(base_sha: str) -> str:+ result = subprocess.run(+ ["git", "diff", "--binary", base_sha],+ cwd=repo_root(),+ check=True,+ capture_output=True,+ )+ return result.stdout.decode("utf-8")+++def read_patch(patch_path: str | None, base_sha: str, *, clean: bool = False) -> str:+ if clean:+ return ""+ if not patch_path:+ return current_tracked_patch(base_sha)+ return Path(patch_path).read_text()+++def untracked_files() -> list[str]:+ return git_path_list(["ls-files", "--others", "--exclude-standard"])+++def sha256_text(text: str) -> str:+ return hashlib.sha256(text.encode()).hexdigest()+++def format_untracked_warning(files: list[str]) -> str:+ preview = "\n".join(f" - {path}" for path in files[:20])+ if len(files) > 20:+ preview += f"\n ... and {len(files) - 20} more"+ return (+ "untracked files are present and will NOT be included in the build patch.\n"+ "Commit/remove them, or pass --allow-untracked to proceed anyway.\n"+ f"{preview}"+ )+++def prepare_build_source(+ *,+ base_sha: str | None,+ patch_path: str | None,+ allow_untracked: bool,+ require_clean: bool,+ repo_url: str | None,+ clean: bool = False,+ source_label: str | None = None,+ pre_resolved_base_sha: str | None = None,+) -> tuple[dict[str, Any], str]:+ # `pre_resolved_base_sha` is a canonical SHA already resolved against the+ # remote (see `resolve_remote_ref`); it must be used verbatim because the+ # launcher may not have that commit fetched locally to `rev-parse`.+ if pre_resolved_base_sha:+ resolved_base_sha = pre_resolved_base_sha+ elif base_sha:+ resolved_base_sha = resolve_git_ref(base_sha)+ else:+ resolved_base_sha = current_base_sha()+ resolved_repo_url = repo_url or DEFAULT_REPO_URL+ patch = read_patch(patch_path, resolved_base_sha, clean=clean)+ patch_sha256 = sha256_text(patch) if patch.strip() else None+ untracked = [] if clean else untracked_files()+ is_dirty = bool(patch.strip())++ if untracked and not allow_untracked:+ raise ValueError(format_untracked_warning(untracked))++ if require_clean and is_dirty:+ raise ValueError(+ "tracked changes are present. Commit or stash them, or omit --require-clean."+ )++ return {+ "schema": SOURCE_SCHEMA_VERSION,+ "type": "git_patch",+ "repo_url": resolved_repo_url,+ "base_sha": resolved_base_sha,+ "base_ref": source_label or (base_sha if clean else current_ref_name()),+ "patch_sha256": patch_sha256,+ "is_dirty": is_dirty,+ "patch_path": "source.patch" if patch.strip() else None,+ "untracked_files": untracked,+ "untracked_files_included": False,+ "allow_untracked": allow_untracked,+ "require_clean": require_clean,+ "clean_source": clean,+ }, patch+++def public_source_info(source_info: dict[str, Any]) -> dict[str, Any]:+ return dict(source_info)+++def build_toolchain_info(+ *,+ target: str = BUILD_TARGET,+ rust_image: str = RUST_IMAGE,+ zig_version: str = ZIG_VERSION,+ cargo_zigbuild_version: str = CARGO_ZIGBUILD_VERSION,+) -> dict[str, str]:+ return {+ "build_image_recipe_version": BUILD_IMAGE_RECIPE_VERSION,+ "target": target,+ "rust_version": RUST_VERSION,+ "rust_image": rust_image,+ "rust_image_tag": RUST_IMAGE_TAG,+ "rust_image_digest": RUST_IMAGE_DIGEST,+ "zig_version": zig_version,+ "cargo_zigbuild_version": cargo_zigbuild_version,+ }+++def compute_build_id(+ *,+ source_info: dict[str, Any],+ target: str = BUILD_TARGET,+ rust_image: str = RUST_IMAGE,+ zig_version: str = ZIG_VERSION,+ cargo_zigbuild_version: str = CARGO_ZIGBUILD_VERSION,+) -> str:+ payload = {+ "schema": BUILD_SCHEMA_VERSION,+ "source_type": "git_patch",+ "base_sha": source_info["base_sha"],+ "patch_sha256": source_info["patch_sha256"],+ **build_toolchain_info(+ target=target,+ rust_image=rust_image,+ zig_version=zig_version,+ cargo_zigbuild_version=cargo_zigbuild_version,+ ),+ }+ digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()+ return f"bld-{digest[:20]}"+++def sanitize_namespace(value: str) -> str:+ sanitized = re.sub(r"[^A-Za-z0-9_.-]+", "-", value.strip().lower()).strip("-.")+ return sanitized or "default"+++def default_namespace() -> str:+ configured = os.environ.get("AGENT_EVALS_NAMESPACE")+ if configured:+ return sanitize_namespace(configured)++ try:+ email = git_output(["config", "user.email"])+ if email:+ return sanitize_namespace(email.split("@", 1)[0])+ except (OSError, subprocess.CalledProcessError):+ pass++ return sanitize_namespace(+ os.environ.get("USER") or os.environ.get("LOGNAME") or "default"+ )diff --git a/crates/eval_cli/zed_eval/tests/test_agent_common.py b/crates/eval_cli/zed_eval/tests/test_agent_common.pynew file mode 100644index 00000000000..d2c25a20e34--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_agent_common.py@@ -0,0 +1,139 @@+from __future__ import annotations++import argparse+import io+import os+import shlex+import subprocess+import tarfile+import tempfile+import unittest+from pathlib import Path+from unittest.mock import patch++from zed_eval import common+from zed_eval.agent_common import (+ add_anthropic_available_models_env,+ eval_cli_with_log_command,+)+++class EvalCliWithLogCommandTests(unittest.TestCase):+ def run_logged_command(self, script: str, *, timeout_message: str | None = None):+ with tempfile.TemporaryDirectory() as temporary_directory:+ log_path = Path(temporary_directory) / "eval-cli.txt"+ command = eval_cli_with_log_command(+ ["sh", "-c", shlex.quote(script)],+ str(log_path),+ timeout_message=timeout_message,+ )+ completed = subprocess.run(+ command,+ shell=True,+ capture_output=True,+ text=True,+ )+ log = log_path.read_text() if log_path.exists() else ""+ return completed, log++ def test_preserves_non_timeout_exit_status_through_tee(self) -> None:+ completed, log = self.run_logged_command("echo before-failure; exit 7")++ self.assertEqual(completed.returncode, 7)+ self.assertIn("before-failure", completed.stdout)+ self.assertIn("before-failure", log)++ def test_maps_eval_cli_timeout_exit_to_success(self) -> None:+ completed, log = self.run_logged_command(+ "echo partial-output; exit 2",+ timeout_message="timeout converted",+ )++ self.assertEqual(completed.returncode, 0)+ self.assertIn("partial-output", completed.stdout)+ self.assertIn("timeout converted", completed.stdout)+ self.assertIn("partial-output", log)+ self.assertIn("timeout converted", log)+++class EnvForwardingTests(unittest.TestCase):+ def test_anthropic_available_models_env_is_added(self) -> None:+ env: dict[str, str] = {}+ add_anthropic_available_models_env(env, '[{"name":"model"}]')++ self.assertEqual(env["ZED_ANTHROPIC_AVAILABLE_MODELS"], '[{"name":"model"}]')++ def test_configure_modal_environment_sets_documented_secret_env(self) -> None:+ args = argparse.Namespace(+ app_name="app",+ volume="volume",+ modal_token_secret="modal-secret",+ api_secret="llm-secret",+ )++ with patch.dict(os.environ, {}, clear=False):+ common.configure_modal_environment(args)++ self.assertEqual(os.environ["AGENT_EVALS_APP_NAME"], "app")+ self.assertEqual(os.environ["AGENT_EVALS_VOLUME"], "volume")+ self.assertEqual(+ os.environ["AGENT_EVALS_MODAL_TOKEN_SECRET"], "modal-secret"+ )+ self.assertEqual(+ os.environ["AGENT_EVALS_LLM_PROVIDERS_SECRET"], "llm-secret"+ )+++class SafeExtractArchiveTests(unittest.TestCase):+ def make_archive(+ self, archive_path: Path, member_name: str, data: bytes = b"data"+ ) -> None:+ with tarfile.open(archive_path, "w:gz") as archive:+ info = tarfile.TarInfo(member_name)+ info.size = len(data)+ archive.addfile(info, io.BytesIO(data))++ def test_extracts_normal_members(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ root = Path(temporary_directory)+ archive_path = root / "archive.tar.gz"+ destination = root / "out"+ destination.mkdir()+ self.make_archive(archive_path, "job/result.json", b"{}")++ with tarfile.open(archive_path, "r:gz") as archive:+ common.safe_extract_archive(archive, destination)++ self.assertEqual((destination / "job" / "result.json").read_text(), "{}")++ def test_rejects_path_traversal_members(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ root = Path(temporary_directory)+ archive_path = root / "archive.tar.gz"+ destination = root / "out"+ destination.mkdir()+ self.make_archive(archive_path, "../evil.txt")++ with tarfile.open(archive_path, "r:gz") as archive:+ with self.assertRaises(ValueError):+ common.safe_extract_archive(archive, destination)++ def test_rejects_links(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ root = Path(temporary_directory)+ archive_path = root / "archive.tar.gz"+ destination = root / "out"+ destination.mkdir()+ with tarfile.open(archive_path, "w:gz") as archive:+ info = tarfile.TarInfo("link")+ info.type = tarfile.SYMTYPE+ info.linkname = "/tmp/target"+ archive.addfile(info)++ with tarfile.open(archive_path, "r:gz") as archive:+ with self.assertRaises(ValueError):+ common.safe_extract_archive(archive, destination)+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_baseline.py b/crates/eval_cli/zed_eval/tests/test_baseline.pynew file mode 100644index 00000000000..d1652897b36--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_baseline.py@@ -0,0 +1,36 @@+from __future__ import annotations++import unittest++from zed_eval import baseline+++class BaselineRecordTests(unittest.TestCase):+ def test_record_uses_base_ref_from_source_info(self) -> None:+ record, has_build_info = baseline._record_from_provenance(+ namespace="alice",+ fallback_experiment="swe-atlas-rf",+ run_id="run-1",+ provenance={+ "request": {+ "experiment_name": "swe-atlas-rf",+ "agent_model": "anthropic/claude-sonnet-4-6",+ "judge_preset": "kimi-k2.7-code",+ "build_id": "bld-test",+ },+ "summary": {"status": "completed", "trial_count": 10},+ "build_info": {+ "build_id": "bld-test",+ "base_sha": "a" * 40,+ "patch_sha256": None,+ "source": {"base_ref": "v0.210.0"},+ },+ },+ )++ self.assertTrue(has_build_info)+ self.assertEqual(record["base_ref"], "v0.210.0")+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_benchmarks.py b/crates/eval_cli/zed_eval/tests/test_benchmarks.pynew file mode 100644index 00000000000..c366855cb4c--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_benchmarks.py@@ -0,0 +1,137 @@+from __future__ import annotations++import unittest++from zed_eval import benchmarks, config, harness_command+++def make_run_request(benchmark_id: str, **overrides: object) -> dict[str, object]:+ benchmark = benchmarks.get_benchmark(benchmark_id)+ request: dict[str, object] = {+ "run_id": "run-1",+ "namespace": "tester",+ "experiment_name": benchmark_id,+ "benchmark": benchmarks.benchmark_metadata(benchmark),+ "volume_name": "agent-evals",+ "api_secret_name": "agent-evals-llm-providers",+ "build_id": "bld-test",+ "agent_model": "anthropic/claude-sonnet-4-6",+ "judge_preset": benchmark.default_judge or "leaderboard",+ }+ request.update(overrides)+ return request+++def option_values(command: list[str], option: str) -> list[str]:+ return [+ command[index + 1]+ for index, argument in enumerate(command[:-1])+ if argument == option+ ]+++class BenchmarkResolutionTests(unittest.TestCase):+ def test_group_expands_to_parts(self) -> None:+ self.assertEqual(+ benchmarks.resolve_benchmarks(["swe-atlas"]),+ ["swe-atlas-qna", "swe-atlas-rf", "swe-atlas-tw"],+ )++ def test_aliases_and_dedup(self) -> None:+ self.assertEqual(+ benchmarks.resolve_benchmarks(["qna", "tb21", "rf,tw"]),+ [+ "swe-atlas-qna",+ "terminal-bench-2.1",+ "swe-atlas-rf",+ "swe-atlas-tw",+ ],+ )+++class HarnessCommandTests(unittest.TestCase):+ def test_terminal_bench_is_harbor_registry_no_judge(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("terminal-bench-2.1"), "/tmp/jobs"+ )+ self.assertEqual(command[0], "harbor")+ self.assertIn("terminal-bench/terminal-bench-2-1", command)+ self.assertIn("-d", command)+ # Test-scored benchmarks must not wire up an LLM judge.+ self.assertFalse(any(arg.startswith("EVAL_MODEL=") for arg in command))+ self.assertNotIn("--verifier-import-path", command)++ def test_swe_atlas_rf_wires_judge(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("swe-atlas-rf"), "/tmp/jobs"+ )+ verifier_env = option_values(command, "--ve")+ agent_env = option_values(command, "--ae")++ self.assertEqual(command[0], "harbor")+ self.assertIn("scale-ai/swe-atlas-rf", command)+ self.assertIn(+ config.JUDGE_PROXY_VERIFIER_IMPORT_PATH,+ option_values(command, "--verifier-import-path"),+ )+ self.assertTrue(any(arg.startswith("EVAL_MODEL=") for arg in verifier_env))+ self.assertTrue(+ any(arg.startswith("ZED_JUDGE_UPSTREAM=") for arg in verifier_env)+ )+ self.assertTrue(+ any(arg.startswith("ZED_JUDGE_AUTH_ENV=") for arg in verifier_env)+ )+ self.assertIn("ZED_JUDGE_MAX_TOKENS=8192", verifier_env)+ self.assertFalse(any(arg.startswith("ZED_JUDGE_") for arg in agent_env))++ def test_deepswe_uses_pier_path_without_cli_allowlist(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("deepswe"), "/tmp/jobs"+ )+ self.assertEqual(command[0], "pier")+ self.assertIn("-p", command)+ self.assertIn("/tmp/datasets/deepswe/tasks", command)+ # Pier uses the Pier-native agent class.+ self.assertIn("zed_eval.pier_agent:ZedPierAgent", command)+ # The Pier network allowlist is declared by the agent, not the CLI;+ # `pier run` has no --agent-allow-host option.+ self.assertNotIn("--agent-allow-host", command)++ def test_harbor_benchmarks_use_harbor_agent(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("swe-atlas-rf"), "/tmp/jobs"+ )+ self.assertIn("zed_eval.agent:ZedAgent", command)++ def test_eval_cli_timeout_override(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("swe-atlas-rf", eval_cli_timeout=123), "/tmp/jobs"+ )+ self.assertIn("EVAL_CLI_TIMEOUT=123", command)++ def test_extra_env_is_forwarded_to_harness_agent(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request(+ "swe-atlas-rf",+ extra_env={+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE": "/data/prompts/checklist.md"+ },+ ),+ "/tmp/jobs",+ )++ self.assertIn(+ "ZED_EVAL_INSTRUCTION_SUFFIX_FILE=/data/prompts/checklist.md",+ option_values(command, "--ae"),+ )++ def test_swe_atlas_tw_uses_path_dataset(self) -> None:+ command = harness_command.build_harness_command(+ make_run_request("swe-atlas-tw"), "/tmp/jobs"+ )+ self.assertIn("-p", command)+ self.assertIn("/tmp/datasets/swe-atlas-tw/data/tw", command)+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_cleanup.py b/crates/eval_cli/zed_eval/tests/test_cleanup.pynew file mode 100644index 00000000000..da8a5cf7e20--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_cleanup.py@@ -0,0 +1,79 @@+from __future__ import annotations++import os+import tempfile+import time+import unittest+from pathlib import Path++from zed_eval import cleanup+++def age_path(path: Path, days: float) -> None:+ past = time.time() - days * 86400.0+ os.utime(path, (past, past))+++class PruneArtifactsTests(unittest.TestCase):+ def setUp(self) -> None:+ self._tmp = tempfile.TemporaryDirectory()+ self.root = Path(self._tmp.name)+ self.addCleanup(self._tmp.cleanup)++ def _make_build(self, build_id: str, age_days: float) -> Path:+ build_dir = self.root / "builds" / build_id+ build_dir.mkdir(parents=True)+ (build_dir / "eval-cli").write_text("binary")+ ready = build_dir / "READY"+ ready.write_text("done")+ age_path(ready, age_days)+ return build_dir++ def test_eval_results_are_never_touched(self) -> None:+ runs = self.root / "runs" / "anant" / "swe-atlas-rf" / "run-1"+ runs.mkdir(parents=True)+ result_file = runs / "summary.json"+ result_file.write_text("{}")+ age_path(result_file, 999)++ cleanup.prune_artifacts(self.root, dry_run=False, build_retention_days=14)++ self.assertTrue(result_file.exists())++ def test_removes_old_builds_keeps_recent(self) -> None:+ self._make_build("bld-old", age_days=30)+ self._make_build("bld-new", age_days=1)++ result = cleanup.prune_artifacts(+ self.root, dry_run=False, build_retention_days=14+ )++ self.assertFalse((self.root / "builds" / "bld-old").exists())+ self.assertTrue((self.root / "builds" / "bld-new").exists())+ self.assertEqual(result["counts"]["builds"], 1)++ def test_dry_run_deletes_nothing(self) -> None:+ self._make_build("bld-old", age_days=30)+ result = cleanup.prune_artifacts(+ self.root, dry_run=True, build_retention_days=14+ )+ self.assertTrue((self.root / "builds" / "bld-old").exists())+ self.assertEqual(result["counts"]["builds"], 1)++ def test_stale_lock_removed(self) -> None:+ locks = self.root / "build-locks"+ locks.mkdir(parents=True)+ stale = locks / "bld-x.json"+ stale.write_text("{}")+ age_path(stale, 1) # 1 day > 6h ttl+ fresh = locks / "bld-y.json"+ fresh.write_text("{}")++ cleanup.prune_artifacts(self.root, dry_run=False, lock_ttl_hours=6)++ self.assertFalse(stale.exists())+ self.assertTrue(fresh.exists())+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_launch.py b/crates/eval_cli/zed_eval/tests/test_launch.pynew file mode 100644index 00000000000..7c1ee906b50--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_launch.py@@ -0,0 +1,146 @@+from __future__ import annotations++import argparse+import contextlib+import io+import unittest+from unittest.mock import patch++from zed_eval import config+from zed_eval.cli import build_parser+from zed_eval.launch import prepare_benchmark_suite, prepare_shared_build, prepare_suite+++class LaunchPlanTests(unittest.TestCase):+ def parse(self, argv: list[str]) -> argparse.Namespace:+ return build_parser().parse_args(argv)++ def test_swe_atlas_defaults_judges_by_part(self) -> None:+ args = self.parse(+ [+ "swe-atlas",+ "--parts",+ "qna,rf",+ "--build",+ "bld-test",+ "--plan",+ "--allow-untracked",+ "--n-tasks",+ "1",+ "--yes",+ ]+ )++ prepared = prepare_suite(args)++ self.assertEqual([part for part, *_ in prepared], ["qna", "rf"])+ self.assertEqual(prepared[0][1]["build_id"], "bld-test")+ self.assertEqual(prepared[0][2]["build_id"], "bld-test")+ self.assertEqual(prepared[1][1]["build_id"], "bld-test")+ self.assertIsNone(prepared[1][2])+ self.assertEqual(prepared[0][1]["judge_preset"], "deepseek-v4-pro")+ self.assertEqual(prepared[1][1]["judge_preset"], "kimi-k2.7-code")+ self.assertEqual(prepared[0][1]["agent_model"], config.DEFAULT_MODEL)++ def test_baseten_model_preset_generates_provider_json(self) -> None:+ args = self.parse(+ [+ "swe-atlas",+ "--parts",+ "tw",+ "--build",+ "bld-test",+ "--plan",+ "--allow-untracked",+ "--model",+ "baseten:kimi-k2.7-code",+ "--yes",+ ]+ )++ prepared = prepare_suite(args)+ run_request = prepared[0][1]++ self.assertEqual([part for part, *_ in prepared], ["tw"])+ self.assertEqual(run_request["benchmark"]["id"], "swe-atlas-tw")+ self.assertEqual(+ run_request["agent_model"], "baseten/moonshotai/Kimi-K2.7-Code"+ )+ self.assertIn(+ "moonshotai/Kimi-K2.7-Code", run_request["openai_compatible_provider_json"]+ )+ self.assertIn(+ "https://inference.baseten.co/v1",+ run_request["openai_compatible_provider_json"],+ )++ def test_interactive_suite_can_include_non_swe_atlas_benchmarks(self) -> None:+ args = self.parse(+ [+ "swe-atlas",+ "--build",+ "bld-test",+ "--plan",+ "--allow-untracked",+ "--yes",+ ]+ )++ prepared = prepare_benchmark_suite(+ args, ["qna", "terminal-bench-2.1", "deepswe"]+ )++ self.assertEqual(+ [+ run_request["benchmark"]["id"]+ for _label, run_request, _build in prepared+ ],+ ["swe-atlas-qna", "terminal-bench-2.1", "deepswe"],+ )+ self.assertEqual(+ [label for label, *_ in prepared], ["qna", "terminal-bench-2.1", "deepswe"]+ )+ self.assertEqual(prepared[0][1]["judge_preset"], "deepseek-v4-pro")+ self.assertIsNone(prepared[1][1]["judge_preset"])+ self.assertIsNone(prepared[2][1]["judge_preset"])+ self.assertEqual(prepared[0][1]["build_id"], "bld-test")+ self.assertEqual(prepared[1][1]["build_id"], "bld-test")+ self.assertEqual(prepared[2][1]["build_id"], "bld-test")+ self.assertEqual(prepared[0][2]["build_id"], "bld-test")+ self.assertIsNone(prepared[1][2])+ self.assertIsNone(prepared[2][2])++ def test_explicit_existing_build_skips_source_preparation(self) -> None:+ args = self.parse(["run", "rf", "--build", "custom-build"])++ with (+ contextlib.redirect_stdout(io.StringIO()),+ patch("zed_eval.launch.build_ready_on_volume", return_value=True),+ patch("zed_eval.launch.resolve_source") as resolve_source,+ ):+ build_id, build_request = prepare_shared_build(args)++ self.assertEqual(build_id, "custom-build")+ self.assertIsNone(build_request)+ resolve_source.assert_not_called()++ def test_explicit_missing_build_uses_from_source(self) -> None:+ args = self.parse(+ ["run", "rf", "--build", "custom-build", "--from", "v0.210.0"]+ )++ with (+ patch("zed_eval.launch.build_ready_on_volume", return_value=False),+ patch("zed_eval.source.resolve_remote_ref", return_value="a" * 40),+ ):+ build_id, build_request = prepare_shared_build(args)++ self.assertEqual(build_id, "custom-build")+ self.assertEqual(build_request["build_id"], "custom-build")+ self.assertEqual(build_request["source"]["base_sha"], "a" * 40)+ self.assertEqual(build_request["source"]["base_ref"], "v0.210.0")+ self.assertTrue(build_request["source"]["clean_source"])+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_rejudge.py b/crates/eval_cli/zed_eval/tests/test_rejudge.pynew file mode 100644index 00000000000..49c815fbfdd--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_rejudge.py@@ -0,0 +1,123 @@+from __future__ import annotations++import json+import tempfile+import unittest+from pathlib import Path++from zed_eval import config, rejudge+++class TaskResolutionTests(unittest.TestCase):+ def test_task_tests_dir_layout(self) -> None:+ root = Path("/cache")+ self.assertEqual(+ rejudge.task_tests_dir("scale-ai/task-abc", "sha256:deadbeef", root),+ root / "scale-ai" / "task-abc" / "deadbeef" / "tests",+ )++ def test_task_tests_dir_without_hash_prefix(self) -> None:+ root = Path("/cache")+ self.assertEqual(+ rejudge.task_tests_dir("org/task-1", "rawref", root),+ root / "org" / "task-1" / "rawref" / "tests",+ )++ def test_detect_part(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ tests = Path(tmp)+ self.assertIsNone(rejudge.detect_part(tests))+ (tests / rejudge.RF_VERIFIER).write_text("")+ self.assertEqual(rejudge.detect_part(tests), rejudge.PART_RF)++ def test_detect_part_qna(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ tests = Path(tmp)+ (tests / rejudge.QNA_VERIFIER).write_text("")+ self.assertEqual(rejudge.detect_part(tests), rejudge.PART_QNA)+++class RewardRecombinationTests(unittest.TestCase):+ def test_rf_pass_requires_rubric_and_tests(self) -> None:+ # rubric passes, tests passed -> overall pass+ rewards = rejudge.recompute_rf_rewards(+ {"must_have_pass": True, "agg_score": 0.9},+ {"tests_reward": 1.0},+ )+ self.assertEqual(rewards["reward"], 1.0)+ self.assertEqual(rewards["overall_pass"], 1.0)+ self.assertEqual(rewards["must_have_pass"], 1.0)+ self.assertEqual(rewards["tests_reward"], 1.0)++ def test_rf_rubric_pass_but_tests_fail_is_fail(self) -> None:+ rewards = rejudge.recompute_rf_rewards(+ {"must_have_pass": True, "agg_score": 0.9},+ {"tests_reward": 0.0},+ )+ self.assertEqual(rewards["reward"], 0.0)++ def test_rf_tests_pass_but_rubric_fail_is_fail(self) -> None:+ rewards = rejudge.recompute_rf_rewards(+ {"must_have_pass": False, "agg_score": 0.2},+ {"tests_reward": 1.0},+ )+ self.assertEqual(rewards["reward"], 0.0)+ self.assertEqual(rewards["rubrics_agg_score"], 0.2)++ def test_rf_preserves_tests_reward_from_parent(self) -> None:+ # tests_reward is invariant under rejudge (patch unchanged).+ rewards = rejudge.recompute_rf_rewards(+ {"must_have_pass": False, "agg_score": 0.0},+ {"tests_reward": 1.0, "reward": 1.0},+ )+ self.assertEqual(rewards["tests_reward"], 1.0)++ def test_qna_reward_is_pass_verdict(self) -> None:+ self.assertEqual(rejudge.recompute_qna_rewards({"pass": True})["reward"], 1.0)+ self.assertEqual(rejudge.recompute_qna_rewards({"pass": False})["reward"], 0.0)+++class ResultPatchingTests(unittest.TestCase):+ def test_patch_preserves_agent_metrics(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ result_path = Path(tmp) / "result.json"+ result_path.write_text(+ json.dumps(+ {+ "task_name": "scale-ai/task-1",+ "agent_result": {"tokens": 12345},+ "verifier_result": {"rewards": {"reward": 0.0}},+ }+ )+ )+ rejudge.patch_result_rewards(result_path, {"reward": 1.0})+ patched = json.loads(result_path.read_text())+ # Verdict updated...+ self.assertEqual(patched["verifier_result"]["rewards"], {"reward": 1.0})+ # ...but agent metrics and other fields untouched.+ self.assertEqual(patched["agent_result"], {"tokens": 12345})+ self.assertEqual(patched["task_name"], "scale-ai/task-1")+++class JudgeEnvironmentTests(unittest.TestCase):+ def test_proxy_environment_from_judge_config(self) -> None:+ judge = config.get_judge("deepseek-v4-pro")+ env = rejudge.proxy_environment(judge)+ self.assertEqual(env["ZED_JUDGE_UPSTREAM"], judge.upstream)+ self.assertEqual(env["ZED_JUDGE_AUTH_ENV"], judge.auth_env)+ self.assertEqual(env["ZED_JUDGE_MAX_TOKENS"], str(judge.max_tokens))++ def test_proxy_environment_omits_unset_max_tokens(self) -> None:+ judge = config.get_judge("leaderboard")+ env = rejudge.proxy_environment(judge)+ self.assertNotIn("ZED_JUDGE_MAX_TOKENS", env)++ def test_verifier_environment_points_at_proxy(self) -> None:+ env = rejudge.verifier_environment("some/model", 8089)+ self.assertEqual(env["EVAL_MODEL"], "some/model")+ self.assertEqual(env["EVAL_BASE_URL"], "http://127.0.0.1:8089/v1")+ self.assertTrue(env["EVAL_API_KEY"])+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_report.py b/crates/eval_cli/zed_eval/tests/test_report.pynew file mode 100644index 00000000000..52a5282cf53--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_report.py@@ -0,0 +1,113 @@+from __future__ import annotations++import unittest++from zed_eval import report+from zed_eval.report import TrialRecord+++def trial(+ task: str,+ passed: bool | None,+ *,+ steps: int,+ tool_calls: dict[str, int],+ input_tokens: int = 0,+ output_tokens: int = 0,+) -> TrialRecord:+ return TrialRecord(+ task_name=task,+ started_at=None,+ passed=passed,+ error_reason=None,+ input_tokens=input_tokens,+ output_tokens=output_tokens,+ step_count=steps,+ total_tool_calls=sum(tool_calls.values()),+ tool_calls=dict(tool_calls),+ has_result=True,+ )+++class SliceMetricsTests(unittest.TestCase):+ def test_means_over_subset(self) -> None:+ records = [+ trial("a", True, steps=4, tool_calls={"read_file": 2, "edit_file": 1}),+ trial("b", True, steps=6, tool_calls={"read_file": 4, "edit_file": 3}),+ ]+ metrics = report.slice_metrics(records)+ self.assertEqual(metrics["n"], 2)+ self.assertEqual(metrics["mean_steps"], 5.0)+ self.assertEqual(metrics["mean_tool_calls"], 5.0)+ self.assertEqual(metrics["mean_tool_calls_by_tool"]["read_file"], 3.0)+ self.assertEqual(metrics["mean_tool_calls_by_tool"]["edit_file"], 2.0)++ def test_empty_slice(self) -> None:+ metrics = report.slice_metrics([])+ self.assertEqual(metrics["n"], 0)+ self.assertIsNone(metrics["mean_steps"])+++class ReportConditioningTests(unittest.TestCase):+ def test_on_success_excludes_failures_and_errors(self) -> None:+ records = [+ trial(+ "pass1", True, steps=3, tool_calls={"read_file": 1}, input_tokens=100+ ),+ trial(+ "fail1", False, steps=9, tool_calls={"read_file": 9}, input_tokens=900+ ),+ trial("err1", None, steps=1, tool_calls={"read_file": 1}),+ ]+ scored = [r for r in records if r.passed is not None]+ passing = [r for r in scored if r.passed]++ on_success = report.slice_metrics(passing)+ overall = report.slice_metrics(scored)++ # Conditioned on success: only the passing trial counts.+ self.assertEqual(on_success["n"], 1)+ self.assertEqual(on_success["mean_steps"], 3.0)+ # Overall (scored) includes the failure, inflating the mean.+ self.assertEqual(overall["n"], 2)+ self.assertEqual(overall["mean_steps"], 6.0)++ def test_pass_rate_single_attempt(self) -> None:+ scored = [+ trial("a", True, steps=1, tool_calls={}),+ trial("b", False, steps=1, tool_calls={}),+ trial("c", True, steps=1, tool_calls={}),+ trial("d", True, steps=1, tool_calls={}),+ ]+ rate, sem, attempts = report.pass_rate_with_sem(scored)+ self.assertAlmostEqual(rate, 0.75)+ self.assertIsNone(sem)+ self.assertEqual(attempts, 1)+++class TimeoutVerdictTests(unittest.TestCase):+ def _result(self, exc_type: str) -> dict:+ return {"exception_info": {"exception_type": exc_type}}++ def test_timeout_excluded_by_default(self) -> None:+ passed, reason = report.trial_verdict(self._result("AgentTimeoutError"))+ self.assertIsNone(passed)+ self.assertEqual(reason, "AgentTimeoutError")++ def test_timeout_counts_as_failure_when_enabled(self) -> None:+ passed, reason = report.trial_verdict(+ self._result("AgentTimeoutError"), timeout_is_failure=True+ )+ self.assertIs(passed, False)+ self.assertEqual(reason, "AgentTimeoutError")++ def test_other_exceptions_stay_errored_even_when_enabled(self) -> None:+ # A genuine harness/infra exception is never a scored failure.+ passed, _ = report.trial_verdict(+ self._result("SandboxCreateError"), timeout_is_failure=True+ )+ self.assertIsNone(passed)+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_run_index.py b/crates/eval_cli/zed_eval/tests/test_run_index.pynew file mode 100644index 00000000000..1cef1db2ff2--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_run_index.py@@ -0,0 +1,132 @@+from __future__ import annotations++import json+import os+import tempfile+import unittest+from pathlib import Path+from unittest.mock import patch++from zed_eval import run_index+++class RunIndexTests(unittest.TestCase):+ def with_index_path(self, path: Path):+ return patch.dict(os.environ, {"AGENT_EVALS_RUN_INDEX": str(path)})++ def test_record_lookup_and_recent(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ path = Path(temporary_directory) / "nested" / "run-index.json"+ with self.with_index_path(path):+ run_index.record_run(+ {+ "run_id": "run-1",+ "namespace": "alice",+ "experiment_name": "rf",+ "volume_name": "custom-volume",+ "agent_model": "sonnet-4.6",+ "created_at": "2026-01-01T00:00:00+00:00",+ }+ )+ run_index.record_run(+ {+ "run_id": "run-2",+ "namespace": "alice",+ "experiment_name": "qna",+ "volume_name": None,+ }+ )++ self.assertEqual(run_index.lookup("run-1")["volume"], "custom-volume")+ self.assertEqual(+ [entry["run_id"] for entry in run_index.recent(2)],+ ["run-2", "run-1"],+ )+ self.assertEqual(run_index.most_recent()["run_id"], "run-2")+ self.assertNotIn("volume", run_index.lookup("run-2"))++ def test_record_refreshes_existing_run(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ path = Path(temporary_directory) / "run-index.json"+ with self.with_index_path(path):+ run_index.record_run(+ {+ "run_id": "run-1",+ "namespace": "alice",+ "experiment_name": "rf",+ }+ )+ run_index.record_run(+ {+ "run_id": "run-1",+ "namespace": "bob",+ "experiment_name": "tw",+ }+ )++ entries = run_index.recent(10)+ self.assertEqual(len(entries), 1)+ self.assertEqual(entries[0]["namespace"], "bob")+ self.assertEqual(entries[0]["experiment_name"], "tw")++ def test_corrupt_or_malformed_index_is_ignored(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ path = Path(temporary_directory) / "run-index.json"+ with self.with_index_path(path):+ path.write_text("not json")+ self.assertIsNone(run_index.lookup("run-1"))+ self.assertEqual(run_index.recent(), [])++ path.write_text(+ json.dumps(+ {+ "runs": [+ {"run_id": "missing-location"},+ ["not", "a", "dict"],+ {+ "run_id": "run-1",+ "namespace": "alice",+ "experiment_name": "rf",+ },+ ]+ }+ )+ )+ self.assertIsNone(run_index.lookup("missing-location"))+ self.assertEqual(run_index.lookup("run-1")["experiment_name"], "rf")++ def test_caps_entries(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ path = Path(temporary_directory) / "run-index.json"+ with self.with_index_path(path), patch.object(run_index, "MAX_ENTRIES", 2):+ for index in range(3):+ run_index.record_run(+ {+ "run_id": f"run-{index}",+ "namespace": "alice",+ "experiment_name": "rf",+ }+ )++ self.assertEqual(+ [entry["run_id"] for entry in run_index.recent(10)],+ ["run-2", "run-1"],+ )+ self.assertIsNone(run_index.lookup("run-0"))++ def test_zero_limit_returns_no_entries(self) -> None:+ with tempfile.TemporaryDirectory() as temporary_directory:+ path = Path(temporary_directory) / "run-index.json"+ with self.with_index_path(path):+ run_index.record_run(+ {+ "run_id": "run-1",+ "namespace": "alice",+ "experiment_name": "rf",+ }+ )+ self.assertEqual(run_index.recent(0), [])+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/tests/test_source.py b/crates/eval_cli/zed_eval/tests/test_source.pynew file mode 100644index 00000000000..233e1e397c7--- /dev/null+++ b/crates/eval_cli/zed_eval/tests/test_source.py@@ -0,0 +1,68 @@+from __future__ import annotations++import argparse+import unittest+from unittest.mock import patch++from zed_eval import source+from zed_eval.builds import prepare_build_request, resolve_source+++class ResolveRemoteRefTests(unittest.TestCase):+ def test_full_sha_needs_no_network(self) -> None:+ sha = "a" * 40+ with patch("zed_eval.source.git_output") as git_output:+ self.assertEqual(source.resolve_remote_ref("url", sha), sha)+ git_output.assert_not_called()++ def test_annotated_tag_prefers_peeled_commit(self) -> None:+ ls_remote = (+ "1111111111111111111111111111111111111111\trefs/tags/v0.210.0\n"+ "2222222222222222222222222222222222222222\trefs/tags/v0.210.0^{}"+ )+ with patch("zed_eval.source.git_output", return_value=ls_remote):+ self.assertEqual(source.resolve_remote_ref("url", "v0.210.0"), "2" * 40)++ def test_unresolvable_ref_raises(self) -> None:+ with patch("zed_eval.source.git_output", return_value=""):+ with self.assertRaises(ValueError):+ source.resolve_remote_ref("url", "missing")+++class BuildIdDedupTests(unittest.TestCase):+ """A clean tag and the SHA it resolves to must yield the *same* build id, so+ a build done by one teammate is reused by another."""++ def _build_id_for(self, from_source: str, resolved_sha: str) -> str:+ args = argparse.Namespace(+ from_source=from_source,+ base_sha=None,+ clean_source=False,+ zed_version=None,+ repo_url=source.DEFAULT_REPO_URL,+ )+ with patch("zed_eval.source.resolve_remote_ref", return_value=resolved_sha):+ base_sha, clean, label, pre_resolved = resolve_source(args)+ request = prepare_build_request(+ base_sha=base_sha,+ patch_path=None,+ build_id=None,+ allow_untracked=False,+ require_clean=False,+ repo_url=source.DEFAULT_REPO_URL,+ clean_source=clean,+ source_label=label,+ pre_resolved_base_sha=pre_resolved,+ )+ return request["build_id"]++ def test_tag_and_sha_share_build_id(self) -> None:+ sha = "c" * 40+ from_tag = self._build_id_for("v0.210.0", sha)+ from_sha = self._build_id_for(sha, sha)+ self.assertEqual(from_tag, from_sha)+ self.assertTrue(from_tag.startswith("bld-"))+++if __name__ == "__main__":+ unittest.main()diff --git a/crates/eval_cli/zed_eval/verifier.py b/crates/eval_cli/zed_eval/verifier.pynew file mode 100644index 00000000000..f5bf5eeb03a--- /dev/null+++ b/crates/eval_cli/zed_eval/verifier.py@@ -0,0 +1,59 @@+from __future__ import annotations++from harbor.models.verifier.result import VerifierResult+from harbor.utils.env import resolve_env_vars+from harbor.verifier.verifier import Verifier++from .judge_proxy import JUDGE_PROXY_ENSURE_SCRIPT, JUDGE_PROXY_SCRIPT+++class ZedJudgeProxyVerifier(Verifier):+ async def verify(self) -> VerifierResult:+ await self._ensure_judge_proxy()+ return await super().verify()++ def _merged_verifier_env(self) -> dict[str, str]:+ return {+ **self.task.config.verifier.env,+ **(self.verifier_env or {}),+ **self.override_env,+ }++ async def _ensure_judge_proxy(self) -> None:+ merged_env = self._merged_verifier_env()+ if not merged_env.get("ZED_JUDGE_UPSTREAM"):+ return++ env = resolve_env_vars(merged_env)+ result = await self.environment.exec(+ command=(+ "set -e\n"+ "mkdir -p /usr/local/lib /usr/local/bin\n"+ "cat > /usr/local/lib/zed_judge_proxy.py <<'ZED_PROXY_EOF'\n"+ f"{JUDGE_PROXY_SCRIPT}"+ "ZED_PROXY_EOF\n"+ "cat > /usr/local/bin/ensure-judge-proxy <<'ZED_ENSURE_EOF'\n"+ f"{JUDGE_PROXY_ENSURE_SCRIPT}"+ "ZED_ENSURE_EOF\n"+ "chmod +x /usr/local/bin/ensure-judge-proxy\n"+ "ensure-judge-proxy\n"+ "python3 - <<'ZED_PROXY_CHECK_EOF'\n"+ "import os, socket\n"+ "port = int(os.environ.get('ZED_JUDGE_PROXY_PORT', '8089'))\n"+ "with socket.create_connection(('127.0.0.1', port), timeout=5):\n"+ " pass\n"+ "ZED_PROXY_CHECK_EOF\n"+ "if [ -f /tmp/zed-judge-proxy.log ]; then "+ "tail -2 /tmp/zed-judge-proxy.log; fi\n"+ ),+ user="root",+ env=env,+ )+ if result.return_code != 0:+ output = "\n".join(+ part for part in (result.stdout, result.stderr) if part+ ).strip()+ detail = f": {output}" if output else ""+ raise RuntimeError(+ f"Failed to install or start judge proxy (exit {result.return_code}){detail}"+ )diff --git a/crates/eval_cli/zed_eval/volume.py b/crates/eval_cli/zed_eval/volume.pynew file mode 100644index 00000000000..c0bdebfe704--- /dev/null+++ b/crates/eval_cli/zed_eval/volume.py@@ -0,0 +1,254 @@+from __future__ import annotations++import argparse+import subprocess+import sys+import tarfile+import time+from pathlib import Path++from . import run_index, source+from .common import (+ default_namespace,+ deployed_function,+ print_json,+ print_table,+ resolve_run_location,+ run_command,+ safe_extract_archive,+)++RECENT_RUN_COLUMNS = [+ ("created_at", "created"),+ ("run_id", "run"),+ ("experiment_name", "experiment"),+ ("namespace", "namespace"),+ ("agent_model", "model"),+ ("judge_preset", "judge"),+ ("build_id", "build"),+ ("suite_id", "suite"),+]+REMOTE_RUN_COLUMNS = [+ ("updated_at", "updated"),+ ("namespace", "namespace"),+ ("experiment_name", "experiment"),+ ("run_id", "run"),+ ("status", "status"),+ ("agent_model", "model"),+ ("judge_preset", "judge"),+ ("build_id", "build"),+ ("suite_id", "suite"),+]+BUILD_COLUMNS = [+ ("built_at_utc", "built"),+ ("build_id", "build"),+ ("ready", "ready"),+ ("base_sha", "base"),+ ("patch_sha256", "patch"),+]+SUITE_COLUMNS = [+ ("part", "part"),+ ("experiment_name", "experiment"),+ ("run_id", "run"),+ ("status", "status"),+ ("updated_at", "updated"),+ ("trial_count", "trials"),+]+++def volume_get(+ args: argparse.Namespace, remote_path: str, local_path: str+) -> subprocess.CompletedProcess[str]:+ return run_command(+ ["modal", "volume", "get", args.volume, remote_path, local_path],+ capture=local_path == "-",+ )+++def read_volume_text(args: argparse.Namespace, remote_path: str) -> str:+ result = volume_get(args, remote_path, "-")+ return result.stdout+++def build_ready_on_volume(args: argparse.Namespace, build_id: str) -> bool:+ try:+ read_volume_text(args, f"builds/{build_id}/READY")+ return True+ except (subprocess.CalledProcessError, OSError):+ return False+++def ensure_volume_exists(args: argparse.Namespace) -> int:+ print(f"\nEnsuring Modal volume '{args.volume}' exists...")+ try:+ result = subprocess.run(+ ["modal", "volume", "create", args.volume],+ text=True,+ capture_output=True,+ check=True,+ )+ if result.stdout:+ print(result.stdout, end="")+ except subprocess.CalledProcessError as error:+ output = (error.stdout or "") + (error.stderr or "")+ if "already exists" in output.lower():+ print(f"Volume '{args.volume}' already exists; continuing.")+ else:+ if output:+ print(output, end="", file=sys.stderr)+ return error.returncode+ return 0+++def print_rows(+ rows: list[dict], columns: list[tuple[str, str]], *, as_json: bool = False+) -> None:+ if as_json:+ print_json(rows)+ else:+ print_table(rows, columns)+++def run_remote_prefix(args: argparse.Namespace, run_id: str) -> str:+ namespace, experiment_name = resolve_run_location(args, run_id)+ return f"runs/{namespace}/{experiment_name}/{run_id}"+++def read_run_file(args: argparse.Namespace, file_name: str) -> str:+ run_id = resolve_run_id(args)+ return read_volume_text(args, f"{run_remote_prefix(args, run_id)}/{file_name}")+++def suite_rows(args: argparse.Namespace) -> list[dict]:+ namespace = default_namespace(args)+ return deployed_function(args, "suite_status").remote(namespace, args.suite_id)+++def suite_member_args(args: argparse.Namespace, row: dict) -> argparse.Namespace:+ run_args = argparse.Namespace(**vars(args))+ run_args.experiment_name = row["experiment_name"]+ run_args.run_id = row["run_id"]+ return run_args+++def resolve_run_id(args: argparse.Namespace) -> str:+ """Return the run id to act on, defaulting to the most recent local run.++ Commands like `status`/`logs` accept no run id at all, in which case we use+ the most recently launched run recorded in the local index so a bare+ `zed-eval status` answers "how's my latest run doing?".+ """+ run_id = getattr(args, "run_id", None)+ if run_id:+ return run_id+ entry = run_index.most_recent()+ if not entry:+ raise ValueError(+ "no run id given and the local run index is empty. Launch a run "+ "first, or pass a run id explicitly."+ )+ args.run_id = entry["run_id"]+ print(+ f"(no run id given; using most recent: {entry['run_id']} / "+ f"{entry.get('experiment_name')})",+ file=sys.stderr,+ )+ return entry["run_id"]+++def command_runs(args: argparse.Namespace) -> int:+ entries = run_index.recent(args.limit)+ if getattr(args, "json", False):+ print_json(entries)+ return 0+ if not entries:+ print(+ "No runs recorded locally yet. Launch a run, or use "+ "`zed-eval list --details` to query runs on the volume."+ )+ return 0+ print_table(entries, RECENT_RUN_COLUMNS)+ return 0+++def command_list(args: argparse.Namespace) -> int:+ namespace = None if args.all_namespaces else default_namespace(args)+ experiment_name = getattr(args, "experiment_name", None)+ if experiment_name:+ experiment_name = source.sanitize_namespace(experiment_name)+ if args.details or args.json:+ rows = deployed_function(args, "list_runs").remote(+ namespace, experiment_name, args.limit+ )+ print_rows(rows, REMOTE_RUN_COLUMNS, as_json=args.json)+ return 0+ remote_path = (+ f"runs/{namespace}/{experiment_name}/"+ if experiment_name+ else f"runs/{namespace}/"+ )+ run_command(["modal", "volume", "ls", args.volume, remote_path])+ return 0+++def command_builds(args: argparse.Namespace) -> int:+ if args.details or args.json:+ rows = deployed_function(args, "list_builds").remote(args.limit)+ print_rows(rows, BUILD_COLUMNS, as_json=args.json)+ return 0+ run_command(["modal", "volume", "ls", args.volume, "builds/"])+ return 0+++def command_status(args: argparse.Namespace) -> int:+ # One-shot status ping: the controller's state.json only carries a coarse+ # status (pending -> running -> completed/failed), so there is no per-trial+ # progress worth following. Print it once and return.+ print(read_run_file(args, "state.json"), end="")+ return 0+++def command_logs(args: argparse.Namespace) -> int:+ print(read_run_file(args, "controller.log"), end="")+ return 0+++def command_fetch(args: argparse.Namespace) -> int:+ jobs_dir = Path(args.jobs_dir).expanduser()+ jobs_dir.mkdir(parents=True, exist_ok=True)+ temporary_archive = jobs_dir / f".{args.run_id}.tar.gz"+ remote_path = f"{run_remote_prefix(args, args.run_id)}/harbor-job.tar.gz"+ print(f"Fetching {args.volume}:/{remote_path} -> {temporary_archive}")+ volume_get(args, remote_path, str(temporary_archive))+ with tarfile.open(temporary_archive, "r:gz") as archive:+ safe_extract_archive(archive, jobs_dir)+ temporary_archive.unlink(missing_ok=True)+ print(f"Extracted Harbor job under {jobs_dir / args.run_id}")+ return 0+++def command_suite_status(args: argparse.Namespace) -> int:+ print_rows(suite_rows(args), SUITE_COLUMNS, as_json=args.json)+ return 0+++def command_suite_logs(args: argparse.Namespace) -> int:+ rows = suite_rows(args)+ while True:+ for row in rows:+ print(+ f"\n=== {row['part']} / {row['experiment_name']} / {row['run_id']} ==="+ )+ run_args = suite_member_args(args, row)+ run_args.follow = False+ command_logs(run_args)+ if not args.follow:+ return 0+ time.sleep(args.interval)+++def command_suite_fetch(args: argparse.Namespace) -> int:+ result = 0+ for row in suite_rows(args):+ result = command_fetch(suite_member_args(args, row)) or result+ return result