Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
chore(evals): add switchyard-server wiring to benchmarks (#177)
Ayush Agarwald9e90ee5b92e
diff --git a/benchmark/DATASETS.md b/benchmark/DATASETS.mdindex a60d6b7ea..3e8596dfa 100644--- a/benchmark/DATASETS.md+++ b/benchmark/DATASETS.md@@ -97,7 +97,7 @@ Pass the generated directory to the runner:```bashbash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/terminal-bench-2-closed-book \- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \--model switchyard \--agent codex \--n-tasks 1 \@@ -105,8 +105,8 @@ bash benchmark/run-baseline.sh \--max-retries 0```-Use a routing profile and `--model` that match the route key you want Harbor agents to call. Omit-`--routing-profiles` for a direct-upstream comparison, as described in the main README.+Use a server config and `--model` that match the route id you want Harbor agents to call. Omit+`--server-config` for a direct-upstream comparison, as described in the main README.Before reporting numbers for a new dataset, inspect:@@ -117,7 +117,8 @@ run_manifest.jsonjobs/<job-name>/<task-id>/agent/trajectory.json```-For Switchyard runs, also inspect `routing_stats_final.json`.+For Switchyard runs, also inspect `server_metrics_final.prom` and+`routing_stats_final.json`.## What The Rewriter Changesdiff --git a/benchmark/README.md b/benchmark/README.mdindex 72e340ac8..3a78eeaf0 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -10,7 +10,7 @@ It covers the two smoke paths most people need first:- **Switchyard routing:** Harbor calls Switchyard, and Switchyard routes across two model tiers.Both paths use the same generated dataset, task proxy, pinned agent versions, and run artifact-layout. Passing `--routing-profiles` starts Switchyard; omitting it disables Switchyard and points+layout. Passing `--server-config` starts the Rust server; omitting it disables Switchyard and pointsHarbor directly at the upstream provider.## Prerequisites@@ -22,7 +22,7 @@ uv sync```You also need Docker with Compose support, because baseline runs launch task containers and use the-generated benchmark proxy topology. Runs with `--routing-profiles` also start Switchyard inside+generated benchmark proxy topology. Runs with `--server-config` also start Switchyard insideDocker.Harbor is installed as a dev dependency. Check that the CLI resolves from the uv environment:@@ -55,8 +55,8 @@ bash benchmark/run-baseline.sh \...```-Route-profile YAML files expand `${OPENROUTER_API_KEY}` by default. For another provider, copy a-profile and update its `api_key`, `base_url`, and model ids to match that provider.+Rust server TOML files refer to the credential through `api_key_env`. For another provider, copy a+config and update its `api_key_env`, `base_url`, and model ids to match that provider.## One-Time Setup@@ -141,7 +141,7 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \## Run Without Switchyard-Omit `--routing-profiles` to fully disable Switchyard. The runner still creates the benchmark+Omit `--server-config` to fully disable Switchyard. The runner still creates the benchmarkDocker network for the generated proxy sidecar, but Harbor sends model calls straight to`${UPSTREAM_BASE_URL:-https://openrouter.ai/api/v1}` using `OPENROUTER_API_KEY` by default:@@ -162,14 +162,14 @@ Switchyard translation is disabled.## Run With Switchyard Routing-Pass `--routing-profiles` to start Switchyard and route Harbor traffic through it. This smoke test-uses `benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml`, an-LLM-as-classifier profile for coding-agent tasks:+Pass `--server-config` to start `switchyard-server` and route Harbor traffic through it. This smoke+test uses `benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml`, a Rust+task-classifier configuration for coding-agent tasks:```bashbash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \--model switchyard \--agent codex \--reasoning-effort xhigh \@@ -178,8 +178,8 @@ bash benchmark/run-baseline.sh \--max-retries 0```-Use the route key from the YAML as `--model`. In this profile, the Gemini classifier selects the-target tier for each request, then Switchyard routes to one of:+Use the route `id` from the TOML as `--model`. In this config, the Gemini classifier selects the+target tier for the task, then Switchyard routes to one of:- strong: `anthropic/claude-opus-4.7`- weak: `moonshotai/kimi-k2.7-code`@@ -189,8 +189,8 @@ Classifier model: `google/gemini-3.5-flash`.To smoke-test a single-model Switchyard path instead, use one of:```text-benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml-benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml+benchmark/server-configs/tb-lite-single-gpt-5-5.toml+benchmark/server-configs/tb-lite-single-opus-4-7.toml```By default, the runner starts in the background and prints the PID, log path, and kill command.@@ -198,14 +198,14 @@ By default, the runner starts in the background and prints the PID, log path, an## Book ModesBoth book modes use the same generated `--harbor-path` dataset, prebaked agent images, and proxy-sidecar topology. Switchyard is Dockerized only when `--routing-profiles` is provided.+sidecar topology. Switchyard is Dockerized only when `--server-config` is provided.Closed-book mode is the default:```bashbash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \--model switchyard \--agent codex \--n-tasks 1@@ -221,14 +221,14 @@ Open-book mode keeps the same proxy path but broadens egress:bash benchmark/run-baseline.sh \--book-mode open \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \--model switchyard \--agent codex \--n-tasks 1```Use open-book mode only when the evaluation intentionally allows internet access. The manifest-records the mode, the local dataset digest, any copied routing profile, proxy metadata, upstream+records the mode, the local dataset digest, a snapshot of the server config, proxy metadata, upstreambase URL for direct runs, and agent version pins in both modes.## Run A Full TB Lite Pass@@ -253,7 +253,7 @@ Switchyard LLM-classifier routing:```bashbash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \--model switchyard \--agent codex \--reasoning-effort xhigh \@@ -272,30 +272,21 @@ Run directories are created under `benchmark/tb_runs/`. The most useful artifactrun_manifest.jsonserver.logharbor.log+server_metrics_final.promrouting_stats_final.json-routing_requests.jsonl-routing_stats_by_task.jsonjobs/<job-name>/result.jsonjobs/<job-name>/<task-id>/agent/trajectory.json```The manifest records the command, git state, Harbor patch provenance, local dataset digest, copied-routing profile when present, direct-upstream metadata when Switchyard is disabled, book-mode-settings, agent version pins, log paths, and final Harbor status. When the routing profile contains-deterministic LLM-classifier routes, `server.classifier_prompts` records each route's effective-prompt, prompt SHA-256, `max_request_chars`, and `recent_turn_window` for reproducibility. Direct-runs mark routing stats as `not-requested`.--Switchyard runs also write `routing_requests.jsonl` (one record per router request: task, trial,-session, served model, tier, and the six token fields from the global schema) and roll it up into-`routing_stats_by_task.json` at finalize. The per-task proxy sidecar stamps the identity headers on-model-bound requests: `x-switchyard-intake-task`, `x-switchyard-trial-id` (Harbor's trial name), and-a per-attempt `proxy_x_session_id`.--Each task in `routing_stats_by_task.json` splits into `final` and `retries`. Requests are grouped by-trial, and within a trial the latest attempt is the one Harbor kept (it retries sequentially and-discards earlier attempts), so tokens spent on retried-away attempts land in `retries` and can be-netted out of cost.+server config, direct-upstream metadata when Switchyard is disabled, book-mode settings, agent+version pins, log paths, and final Harbor status.++`server_metrics_final.prom` is the final `/metrics` snapshot. `routing_stats_final.json` is the+final aggregate `/v1/stats` snapshot, including model and tier calls, errors, tokens, and latency.+Neither artifact provides task or trial attribution. The runner writes them only after Harbor exits+and while the Rust server is still reachable; otherwise the manifest records them as missing.+`routing_requests.jsonl` and `routing_stats_by_task.json` are not produced by the Rust server.### Replay Stage-Router Scores@@ -316,7 +307,8 @@ fall-open behavior.## Docker Image Notes-Baseline runs build `switchyard-baseline:local` from `benchmark/switchyard-server.Dockerfile`.+Baseline runs build `switchyard-baseline:local` from+`benchmark/switchyard-rust-server.Dockerfile`.The default is to rebuild before each run so the container matches the current checkout.To reuse an already built image:@@ -325,7 +317,7 @@ To reuse an already built image:SWITCHYARD_DOCKER_BUILD=0 bash benchmark/run-baseline.sh ...```-Only reuse the image when you know it already contains the current Switchyard code and Rust extension.+Only reuse the image when you know it already contains the current Rust `switchyard-server` binary.## Troubleshootingdiff --git a/benchmark/resolve_classifier_prompts.py b/benchmark/resolve_classifier_prompts.pydeleted file mode 100644index cb7e14925..000000000--- a/benchmark/resolve_classifier_prompts.py+++ /dev/null@@ -1,114 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--"""Resolve deterministic-route classifier prompts for benchmark manifests."""--from __future__ import annotations--import json-import os-import re-import sys-from pathlib import Path-from typing import Any--import yaml # type: ignore[import-untyped]--from switchyard.lib.processors.llm_classifier import DEFAULT_MAX_REQUEST_CHARS-from switchyard.lib.processors.llm_classifier.presets import (- classifier_prompt_sha256,- profile_default_prompt,-)--_DETERMINISTIC_TYPES = {"deterministic", "llm_classifier", "llm_classifier_routing"}-_ENV_REF_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")-_MIN_MAX_REQUEST_CHARS = 256---def _expand_env_string(value: str) -> str:- missing = [name for name in _ENV_REF_RE.findall(value) if name not in os.environ]- if missing:- raise ValueError(- f"missing environment variable(s): {', '.join(sorted(set(missing)))}"- )- return os.path.expandvars(value)---def _int_or_default(value: object, default: int) -> int:- if isinstance(value, bool) or not isinstance(value, int):- return default- return value---def _bounded_int_or_default(value: object, default: int, *, minimum: int) -> int:- if isinstance(value, bool) or not isinstance(value, int):- return default- if value < minimum:- return default- return value---def resolve_classifier_prompts(path: Path) -> dict[str, dict[str, Any]]:- """Return effective classifier-prompt metadata for deterministic routes."""- loaded = yaml.safe_load(path.read_text())- routes = loaded.get("routes") if isinstance(loaded, dict) else None- if not isinstance(routes, dict):- return {}-- result: dict[str, dict[str, Any]] = {}- for route_key, route in routes.items():- if not isinstance(route_key, str) or not isinstance(route, dict):- continue- route_type = str(route.get("type", "")).lower().replace("-", "_")- if route_type not in _DETERMINISTIC_TYPES:- continue-- profile = route.get("profile") or "general"- if not isinstance(profile, str):- raise ValueError(f"route {route_key!r}: profile must be a string")-- classifier = route.get("classifier")- classifier_map = classifier if isinstance(classifier, dict) else {}- raw_prompt = classifier_map.get("prompt")- if isinstance(raw_prompt, str) and raw_prompt.strip():- prompt = _expand_env_string(raw_prompt)- source = "override"- else:- prompt = profile_default_prompt(profile)- source = "profile_default"-- result[route_key] = {- "profile": profile,- "source": source,- "classifier_prompt": prompt,- "classifier_prompt_sha256": classifier_prompt_sha256(prompt),- "max_request_chars": _bounded_int_or_default(- classifier_map.get("max_request_chars"),- DEFAULT_MAX_REQUEST_CHARS,- minimum=_MIN_MAX_REQUEST_CHARS,- ),- "recent_turn_window": _int_or_default(- classifier_map.get("recent_turn_window"),- 4,- ),- }- return result---def main(argv: list[str] | None = None) -> int:- """Print classifier-prompt metadata JSON; never fail a benchmark run."""- args = argv if argv is not None else sys.argv[1:]- if not args:- print("WARNING: no routing-profiles path supplied", file=sys.stderr)- print("{}")- return 0- try:- print(json.dumps(resolve_classifier_prompts(Path(args[0]))))- except Exception as exc: # noqa: BLE001- print(f"WARNING: failed to resolve classifier prompts: {exc}", file=sys.stderr)- print("{}")- return 0---if __name__ == "__main__":- raise SystemExit(main())diff --git a/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml b/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yamldeleted file mode 100644index 4ab7e2d72..000000000--- a/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml+++ /dev/null@@ -1,38 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# LLM-classifier GPT-5.5/DeepSeek/Gemini profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml \-# --model switchyard--routes:- switchyard:- type: deterministic- profile: coding_agent- enable_stats: true- fallback_target_on_evict: strong- strong:- model: openai/gpt-5.5- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: responses- timeout_secs: 600- extra_body:- reasoning_effort: high- weak:- model: deepseek/deepseek-v4-pro- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- classifier:- model: google/gemini-3.5-flash- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 30- min_confidence: 0.0- fail_open: true- recent_turn_window: 4diff --git a/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml b/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yamldeleted file mode 100644index 690c14106..000000000--- a/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml+++ /dev/null@@ -1,39 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# LLM-classifier GPT-5.5/Kimi/Gemini profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml \-# --model switchyard--routes:- switchyard:- type: deterministic- profile: coding_agent- enable_stats: true- fallback_target_on_evict: strong- strong:- model: openai/gpt-5.5- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: responses- timeout_secs: 600- extra_body:- reasoning:- effort: high- weak:- model: moonshotai/kimi-k2.7-code- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- classifier:- model: google/gemini-3.5-flash- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 30- min_confidence: 0.0- fail_open: true- recent_turn_window: 4diff --git a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml b/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yamldeleted file mode 100644index 8b08ea791..000000000--- a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml+++ /dev/null@@ -1,39 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# LLM-classifier Opus/DeepSeek/Gemini profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml \-# --model switchyard--routes:- switchyard:- type: deterministic- profile: coding_agent- enable_stats: true- fallback_target_on_evict: strong- strong:- model: anthropic/claude-opus-4.7- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- extra_body:- reasoning:- effort: high- weak:- model: deepseek/deepseek-v4-pro- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- classifier:- model: google/gemini-3.5-flash- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 30- min_confidence: 0.0- fail_open: true- recent_turn_window: 4diff --git a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml b/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yamldeleted file mode 100644index 6428f4c75..000000000--- a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml+++ /dev/null@@ -1,39 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# LLM-classifier Opus/Kimi/Gemini profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \-# --model switchyard--routes:- switchyard:- type: deterministic- profile: coding_agent- enable_stats: true- fallback_target_on_evict: strong- strong:- model: anthropic/claude-opus-4.7- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- extra_body:- reasoning:- effort: high- weak:- model: moonshotai/kimi-k2.7-code- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- format: openai- timeout_secs: 600- classifier:- model: google/gemini-3.5-flash- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 30- min_confidence: 0- fail_open: true- recent_turn_window: 4diff --git a/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml b/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yamldeleted file mode 100644index 72c327eb2..000000000--- a/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml+++ /dev/null@@ -1,23 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# Serve-backed single-model GPT-5.5 profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml \-# --model tb-lite-single-gpt-5-5-# Harbor uses openai/tb-lite-single-gpt-5-5; LiteLLM strips the provider-# prefix and sends this route key to Switchyard.--defaults:- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 600.0--routes:- tb-lite-single-gpt-5-5:- type: model- target:- model: openai/gpt-5.5- format: responsesdiff --git a/benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml b/benchmark/routing-profiles/tb-lite-single-opus-4-7.yamldeleted file mode 100644index f7321d5b3..000000000--- a/benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml+++ /dev/null@@ -1,26 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-#-# Serve-backed single-model Opus 4.7 profile for Harbor Terminal-Bench Lite.-# Run with:-# benchmark/run-baseline.sh \-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \-# --routing-profiles benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml \-# --model tb-lite-single-opus-4-7-# Harbor uses openai/tb-lite-single-opus-4-7; LiteLLM strips the provider-# prefix and sends this route key to Switchyard.-#-# Note: provider support for agent sampling fields varies. If a provider rejects-# a field, strip it in the benchmark agent config before running a full pass.--defaults:- api_key: ${OPENROUTER_API_KEY}- base_url: https://openrouter.ai/api/v1- timeout_secs: 600.0--routes:- tb-lite-single-opus-4-7:- type: model- target:- model: anthropic/claude-opus-4.7- format: openaidiff --git a/benchmark/run-baseline.sh b/benchmark/run-baseline.shindex bc4caa830..fa0881658 100755--- a/benchmark/run-baseline.sh+++ b/benchmark/run-baseline.sh@@ -34,7 +34,7 @@ OUTPUT_DIR="${REPO_ROOT}/benchmark/tb_runs"PORT=4000SERVER_URL=""HARBOR_SERVER_URL=""-ROUTING_PROFILES=""+SERVER_CONFIG=""MODEL=""ROUTE_MODEL=""AGENT="terminus-2"@@ -47,7 +47,7 @@ BOOK_MODE="closed"PROXY_STRIP_ARTIFACT="/etc/proxy-public/strip.jsonl"VERIFIER_PROXY_TEMPLATE='${SWITCHYARD_VERIFIER_HTTP_PROXY}'SWITCHYARD_DOCKER_IMAGE="${SWITCHYARD_DOCKER_IMAGE:-switchyard-baseline:local}"-SWITCHYARD_DOCKERFILE="${SWITCHYARD_DOCKERFILE:-benchmark/switchyard-server.Dockerfile}"+SWITCHYARD_DOCKERFILE="${SWITCHYARD_DOCKERFILE:-benchmark/switchyard-rust-server.Dockerfile}"SWITCHYARD_DOCKER_SERVICE_NAME="${SWITCHYARD_DOCKER_SERVICE_NAME:-switchyard}"SWITCHYARD_DOCKER_BUILD="${SWITCHYARD_DOCKER_BUILD:-1}"SWITCHYARD_DOCKER_NETWORK="${SWITCHYARD_DOCKER_NETWORK:-}"@@ -91,10 +91,10 @@ Main options:--server-url URL Host URL used for health and stats checks.--harbor-server-url URL Server URL as seen from Harbor task containers.Defaults to the Dockerized Switchyard service.- --routing-profiles PATH Route bundle YAML for switchyard serve. When+ --server-config PATH Rust switchyard-server TOML configuration. Whenomitted, Harbor connects directly upstream.--model MODEL Model Harbor should request. With- --routing-profiles this is a Switchyard route+ --server-config this is a Switchyard routekey; without it, this is the upstream model.Defaults --harbor-model to nvidia/MODEL foropencode, openai/MODEL for other OpenAI-style@@ -248,12 +248,17 @@ merge_claude_closed_book_tools() {}harbor_extra_has_agent_version() {+ harbor_extra_has_agent_kwarg version+}++harbor_extra_has_agent_kwarg() {+ local key="$1"local i=0 token nextwhile [[ "${i}" -lt "${#HARBOR_EXTRA[@]}" ]]; dotoken="${HARBOR_EXTRA[$i]}"if [[ "${token}" == "--ak" || "${token}" == "--agent-kwarg" ]]; thennext="${HARBOR_EXTRA[$((i + 1))]:-}"- [[ "${next}" == version=* ]] && return 0+ [[ "${next}" == "${key}="* ]] && return 0i=$((i + 2))continuefi@@ -293,7 +298,10 @@ while [[ $# -gt 0 ]]; do--port) PORT="$2"; shift 2 ;;--server-url) SERVER_URL="$2"; shift 2 ;;--harbor-server-url) HARBOR_SERVER_URL="$2"; shift 2 ;;- --routing-profiles) ROUTING_PROFILES="$2"; shift 2 ;;+ --server-config) SERVER_CONFIG="$2"; shift 2 ;;+ --routing-profiles)+ die "--routing-profiles is no longer supported by the benchmark runner; use --server-config with a Rust server TOML file"+ ;;--model) MODEL="$2"; shift 2 ;;--route-model) ROUTE_MODEL="$2"; shift 2 ;;--agent) AGENT="$2"; shift 2 ;;@@ -314,7 +322,7 @@ while [[ $# -gt 0 ]]; do--harbor-extra) HARBOR_EXTRA+=("$2"); shift 2 ;;--server-extra) SERVER_EXTRA+=("$2"); shift 2 ;;--strong-model|--weak-model|--classifier-model|--profile|--strong-probability)- die "$1 belongs to the removed legacy benchmark launchers; encode server routing in --routing-profiles instead"+ die "$1 belongs to the removed legacy benchmark launchers; encode server routing in --server-config instead";;--dry-run) DRY_RUN=1; shift ;;--foreground) FOREGROUND=1; shift ;;@@ -350,9 +358,9 @@ resolve_harbor() {resolve_harbor-if [[ -n "${ROUTING_PROFILES}" ]]; then- [[ -f "${ROUTING_PROFILES}" ]] || die "--routing-profiles not found: ${ROUTING_PROFILES}"- ROUTING_PROFILES="$(cd "$(dirname "${ROUTING_PROFILES}")" && pwd)/$(basename "${ROUTING_PROFILES}")"+if [[ -n "${SERVER_CONFIG}" ]]; then+ [[ -f "${SERVER_CONFIG}" ]] || die "--server-config not found: ${SERVER_CONFIG}"+ SERVER_CONFIG="$(cd "$(dirname "${SERVER_CONFIG}")" && pwd)/$(basename "${SERVER_CONFIG}")"fiUPSTREAM_BASE_URL="${UPSTREAM_BASE_URL:-https://openrouter.ai/api/v1}"if [[ -z "${UPSTREAM_API_KEY_ENV}" ]]; then@@ -364,7 +372,7 @@ if [[ -z "${UPSTREAM_API_KEY_ENV}" ]]; thenfi[[ "${UPSTREAM_API_KEY_ENV}" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || die "--upstream-api-key-env must be a valid environment variable name"-if [[ -n "${ROUTING_PROFILES}" ]]; then+if [[ -n "${SERVER_CONFIG}" ]]; thenif [[ -n "${MODEL}" && -n "${ROUTE_MODEL}" && "${MODEL}" != "${ROUTE_MODEL}" ]]; thendie "--model and --route-model must name the same model when both are provided"fi@@ -374,18 +382,18 @@ if [[ -n "${ROUTING_PROFILES}" ]]; thenSWITCHYARD_ENABLED=1SERVER_PRESET="serve"[[ -z "${MODE}" ]] && MODE="serve"- [[ -n "${MODEL}" ]] || die "--model is required when using --routing-profiles"+ [[ -n "${MODEL}" ]] || die "--model is required when using --server-config"ROUTE_MODEL="${MODEL}"elseSWITCHYARD_ENABLED=0SERVER_PRESET="direct"[[ -z "${MODE}" ]] && MODE="direct"- [[ -z "${ROUTE_MODEL}" ]] || die "--route-model requires --routing-profiles; use --model for direct upstream"+ [[ -z "${ROUTE_MODEL}" ]] || die "--route-model requires --server-config; use --model for direct upstream"[[ -n "${MODEL}" ]] || die "--model is required when running direct upstream"ROUTE_MODEL=""- [[ -z "${SERVER_URL}" ]] || die "--server-url requires --routing-profiles"- [[ -z "${HARBOR_SERVER_URL}" ]] || die "--harbor-server-url requires --routing-profiles"- [[ "${#SERVER_EXTRA[@]}" -eq 0 ]] || die "--server-extra requires --routing-profiles"+ [[ -z "${SERVER_URL}" ]] || die "--server-url requires --server-config"+ [[ -z "${HARBOR_SERVER_URL}" ]] || die "--harbor-server-url requires --server-config"+ [[ "${#SERVER_EXTRA[@]}" -eq 0 ]] || die "--server-extra requires --server-config"[[ -n "${!UPSTREAM_API_KEY_ENV:-}" ]] || die "direct upstream requires \$${UPSTREAM_API_KEY_ENV} to be set"fiif [[ "${HARBOR_MODEL_SET}" -eq 0 ]]; then@@ -412,17 +420,15 @@ check_harbor_available() {}if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then- SERVER_CMD=(uv run --no-sync switchyard- --routing-profiles "${ROUTING_PROFILES}"- serve+ SERVER_CMD=(switchyard-server+ --config "${SERVER_CONFIG}"--host 0.0.0.0--port "${PORT}")- SERVER_DOCKER_CMD=(--routing-profiles "${ROUTING_PROFILES}"- serve+ SERVER_DOCKER_CMD=(--config "${SERVER_CONFIG}"--host 0.0.0.0--port "${PORT}")SERVER_CONFIG_JSON="$(json_object_from_pairs \- mode "${MODE}" routing_profiles "${ROUTING_PROFILES}" route_model "${ROUTE_MODEL}")"+ mode "${MODE}" server_config "${SERVER_CONFIG}" route_model "${ROUTE_MODEL}")"if [[ "${#SERVER_EXTRA[@]}" -gt 0 ]]; thenSERVER_CMD+=("${SERVER_EXTRA[@]}")SERVER_DOCKER_CMD+=("${SERVER_EXTRA[@]}")@@ -468,7 +474,8 @@ if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; thenSERVER_ROOT_URL="${SERVER_CONTROL_URL}"fiSERVER_HEALTH_URL="${SERVER_ROOT_URL}/health"- SERVER_STATS_URL="${SERVER_ROOT_URL}/v1/routing/stats"+ SERVER_METRICS_URL="${SERVER_ROOT_URL}/metrics"+ SERVER_STATS_URL="${SERVER_ROOT_URL}/v1/stats"if [[ -z "${HARBOR_SERVER_URL}" ]]; thenHARBOR_SERVER_URL="http://${SWITCHYARD_DOCKER_SERVICE_NAME}:${PORT}"@@ -484,6 +491,7 @@ if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; thenelseSERVER_ROOT_URL=""SERVER_HEALTH_URL=""+ SERVER_METRICS_URL=""SERVER_STATS_URL=""HARBOR_BASE_URL="${UPSTREAM_BASE_URL}"if [[ "${HARBOR_BASE_URL}" == */v1 ]]; then@@ -502,13 +510,8 @@ LOG_PATH="${RUN_DIR}/${JOB_NAME}.log"SERVER_LOG="${RUN_DIR}/server.log"HARBOR_LOG="${RUN_DIR}/harbor.log"HARBOR_RESULT_JSON="${RUN_DIR}/harbor_result.json"+SERVER_METRICS_PROM="${RUN_DIR}/server_metrics_final.prom"ROUTING_STATS_JSON="${RUN_DIR}/routing_stats_final.json"-ROUTING_LOG_JSONL="${RUN_DIR}/routing_requests.jsonl"-ROUTING_STATS_BY_TASK_JSON="${RUN_DIR}/routing_stats_by_task.json"-if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then- SERVER_CMD+=(--routing-log-file "${ROUTING_LOG_JSONL}")- SERVER_DOCKER_CMD+=(--routing-log-file "${ROUTING_LOG_JSONL}")-fiDOCKER_RUN_ID="$(printf '%s-%s' "${TS##*_}" "$$" | tr -c '[:alnum:]_.-' '-')"SWITCHYARD_DOCKER_NETWORK="${SWITCHYARD_DOCKER_NETWORK:-switchyard-${DOCKER_RUN_ID}}"SWITCHYARD_DOCKER_CONTAINER="${SWITCHYARD_DOCKER_CONTAINER:-switchyard-${DOCKER_RUN_ID}}"@@ -528,17 +531,9 @@ if [[ "${BOOK_MODE}" == "closed" || "${BOOK_MODE}" == "open" ]]; thendocker_network "${SWITCHYARD_DOCKER_NETWORK}")"fifi-ROUTING_PROFILES_DIR=""-if [[ -n "${ROUTING_PROFILES}" ]]; then- ROUTING_PROFILES_DIR="$(cd "$(dirname "${ROUTING_PROFILES}")" && pwd)"-fi-CLASSIFIER_PROMPTS_JSON="{}"-if [[ -n "${ROUTING_PROFILES}" ]]; then- CLASSIFIER_PROMPTS_JSON="$(- uv run --no-sync python "${SCRIPT_DIR}/resolve_classifier_prompts.py" "${ROUTING_PROFILES}" \- 2>/dev/null || echo '{}'- )"- [[ -n "${CLASSIFIER_PROMPTS_JSON}" ]] || CLASSIFIER_PROMPTS_JSON="{}"+SERVER_CONFIG_DIR=""+if [[ -n "${SERVER_CONFIG}" ]]; then+ SERVER_CONFIG_DIR="$(cd "$(dirname "${SERVER_CONFIG}")" && pwd)"fiWRAPPER="${RUN_DIR}/run-background.sh"HARBOR_JOB_DIR="${RUN_DIR}/jobs/${JOB_NAME}"@@ -579,6 +574,10 @@ add_agent_version_kwargif [[ -n "${REASONING_EFFORT}" ]]; thenHARBOR_CMD+=(--ak "reasoning_effort=${REASONING_EFFORT}")+ # Claude code does not support thinking=true for now, so we use adaptive instead+ if [[ "${AGENT}" == "claude-code" ]] && ! harbor_extra_has_agent_kwarg thinking; then+ HARBOR_CMD+=(--ak "thinking=adaptive")+ fifiif [[ -n "${N_TASKS}" ]]; then@@ -701,8 +700,8 @@ print_resolved() {if [[ "${BOOK_MODE}" == "closed" ]]; thenecho " verifier_net: authenticated proxy egress"fi- if [[ -n "${ROUTING_PROFILES}" ]]; then- echo " route_profile: ${ROUTING_PROFILES}"+ if [[ -n "${SERVER_CONFIG}" ]]; then+ echo " server_config: ${SERVER_CONFIG}"echo " route_model: ${ROUTE_MODEL}"fiprintf ' harbor_cmd: '@@ -764,7 +763,6 @@ MANIFEST_CMD=(python3 "${MANIFEST_HELPER}" write--server-port "${PORT}"--server-argv-json "$(json_array "${SERVER_CMD[@]}")"--server-config-json "${SERVER_CONFIG_JSON}"- --classifier-prompts-json "${CLASSIFIER_PROMPTS_JSON}"--harbor-server-url "${HARBOR_SERVER_ROOT_URL}"--harbor-base-url "${HARBOR_BASE_URL}"--upstream-base-url "$([[ "${SWITCHYARD_ENABLED}" -eq 0 ]] && echo "${UPSTREAM_BASE_URL}")"@@ -790,6 +788,8 @@ MANIFEST_CMD=(python3 "${MANIFEST_HELPER}" write--run-dir "${RUN_DIR}"--log-path "${LOG_PATH}"--harbor-result-json "${HARBOR_RESULT_JSON}"+ --server-metrics-prom "${SERVER_METRICS_PROM}"+ --server-metrics-status "$([[ "${SWITCHYARD_ENABLED}" -eq 1 ]] && echo predicted || echo not-requested)"--routing-stats-json "${ROUTING_STATS_JSON}"--routing-stats-status "$([[ "${SWITCHYARD_ENABLED}" -eq 1 ]] && echo predicted || echo not-requested)")if [[ -n "${HARBOR_PATH}" ]]; then@@ -798,8 +798,8 @@ fiif [[ -n "${TASK_LIST_FILE}" ]]; thenMANIFEST_CMD+=(--task-list-file "${TASK_LIST_FILE}")fi-if [[ -n "${ROUTING_PROFILES}" ]]; then- MANIFEST_CMD+=(--routing-profiles "${ROUTING_PROFILES}")+if [[ -n "${SERVER_CONFIG}" ]]; then+ MANIFEST_CMD+=(--server-config "${SERVER_CONFIG}")fiif [[ -n "${CODEX_MODEL_CATALOG_HOST}" ]]; thenMANIFEST_CMD+=(--codex-model-catalog "${CODEX_MODEL_CATALOG_HOST}")@@ -820,6 +820,7 @@ UPSTREAM_API_KEY_ENV=$(q "${UPSTREAM_API_KEY_ENV}")SERVER_ROOT_URL=$(q "${SERVER_ROOT_URL}")HARBOR_SERVER_ROOT_URL=$(q "${HARBOR_SERVER_ROOT_URL}")SERVER_HEALTH_URL=$(q "${SERVER_HEALTH_URL}")+SERVER_METRICS_URL=$(q "${SERVER_METRICS_URL}")SERVER_STATS_URL=$(q "${SERVER_STATS_URL}")HARBOR_BASE_URL=$(q "${HARBOR_BASE_URL}")MANIFEST_PATH=$(q "${MANIFEST_PATH}")@@ -827,9 +828,8 @@ RUN_DIR=$(q "${RUN_DIR}")REPO_ROOT=$(q "${REPO_ROOT}")SERVER_LOG=$(q "${SERVER_LOG}")HARBOR_LOG=$(q "${HARBOR_LOG}")+SERVER_METRICS_PROM=$(q "${SERVER_METRICS_PROM}")ROUTING_STATS_JSON=$(q "${ROUTING_STATS_JSON}")-ROUTING_LOG_JSONL=$(q "${ROUTING_LOG_JSONL}")-ROUTING_STATS_BY_TASK_JSON=$(q "${ROUTING_STATS_BY_TASK_JSON}")HARBOR_JOB_DIR=$(q "${HARBOR_JOB_DIR}")SKIP_HEALTH_CHECK=$(q "${SKIP_HEALTH_CHECK}")BOOK_MODE=$(q "${BOOK_MODE}")@@ -839,7 +839,7 @@ SWITCHYARD_DOCKER_BUILD=$(q "${SWITCHYARD_DOCKER_BUILD}")SWITCHYARD_DOCKER_NETWORK=$(q "${SWITCHYARD_DOCKER_NETWORK}")SWITCHYARD_DOCKER_CONTAINER=$(q "${SWITCHYARD_DOCKER_CONTAINER}")SWITCHYARD_DOCKER_SERVICE_NAME=$(q "${SWITCHYARD_DOCKER_SERVICE_NAME}")-ROUTING_PROFILES_DIR=$(q "${ROUTING_PROFILES_DIR}")+SERVER_CONFIG_DIR=$(q "${SERVER_CONFIG_DIR}")SERVER_CMD=($(shell_join "${SERVER_CMD[@]}"))SERVER_DOCKER_CMD=($(shell_join "${SERVER_DOCKER_CMD[@]}"))@@ -869,7 +869,7 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; thenecho "Starting Dockerized Switchyard service"echo " image: \${SWITCHYARD_DOCKER_IMAGE}"echo " network: \${SWITCHYARD_DOCKER_NETWORK}"- echo " command: switchyard \${SERVER_DOCKER_CMD[*]}"+ echo " command: switchyard-server \${SERVER_DOCKER_CMD[*]}"if [[ "\${SWITCHYARD_DOCKER_BUILD}" != "0" ]]; thendocker build -f "\${SWITCHYARD_DOCKERFILE}" -t "\${SWITCHYARD_DOCKER_IMAGE}" .fi@@ -887,8 +887,8 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; then-v "\${REPO_ROOT}:\${REPO_ROOT}:ro"-v "\${RUN_DIR}:\${RUN_DIR}")- if [[ -n "\${ROUTING_PROFILES_DIR}" && "\${ROUTING_PROFILES_DIR}" != "\${REPO_ROOT}"* ]]; then- DOCKER_RUN_ARGS+=(-v "\${ROUTING_PROFILES_DIR}:\${ROUTING_PROFILES_DIR}:ro")+ if [[ -n "\${SERVER_CONFIG_DIR}" && "\${SERVER_CONFIG_DIR}" != "\${REPO_ROOT}"* ]]; then+ DOCKER_RUN_ARGS+=(-v "\${SERVER_CONFIG_DIR}:\${SERVER_CONFIG_DIR}:ro")fiif [[ -n "\${UPSTREAM_API_KEY_ENV}" ]]; thenDOCKER_RUN_ARGS+=(--env "\${UPSTREAM_API_KEY_ENV}")@@ -898,6 +898,7 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; then--env PYTHONHASHSEED=0 \\--env LC_ALL=C.UTF-8 \\--env OPENAI_API_KEY \\+ --env NVIDIA_API_KEY \\--env OPENROUTER_API_KEY \\--env UPSTREAM_API_KEY \\--env ANTHROPIC_API_KEY \\@@ -943,8 +944,7 @@ if [[ "\${SERVER_ENABLED}" == "1" && "\${SKIP_HEALTH_CHECK}" != "1" ]]; thenpython3 benchmark/run_manifest.py finalize \\--manifest "\${MANIFEST_PATH}" \\--harbor-rc 124 \\- --harbor-job-dir "\${HARBOR_JOB_DIR}" \\- --routing-stats "\${ROUTING_STATS_JSON}" || true+ --harbor-job-dir "\${HARBOR_JOB_DIR}" || trueexit 124fifi@@ -961,8 +961,7 @@ if [[ "\${SERVER_ENABLED}" == "1" && "\${SWITCHYARD_CLOSED_BOOK_PREFLIGHT:-1}" !python3 benchmark/run_manifest.py finalize \\--manifest "\${MANIFEST_PATH}" \\--harbor-rc 125 \\- --harbor-job-dir "\${HARBOR_JOB_DIR}" \\- --routing-stats "\${ROUTING_STATS_JSON}" || true+ --harbor-job-dir "\${HARBOR_JOB_DIR}" || trueexit 125fifi@@ -1037,10 +1036,18 @@ FINALIZE_CMD=(python3 benchmark/run_manifest.py finalize \\--harbor-rc "\${HARBOR_RC}" \\--harbor-job-dir "\${HARBOR_JOB_DIR}")if [[ "\${SERVER_ENABLED}" == "1" ]]; then- curl -fsS "\${SERVER_STATS_URL}" -o "\${ROUTING_STATS_JSON}" >/dev/null 2>&1 || true- FINALIZE_CMD+=(--routing-stats "\${ROUTING_STATS_JSON}"- --routing-log "\${ROUTING_LOG_JSONL}"- --routing-stats-by-task "\${ROUTING_STATS_BY_TASK_JSON}")+ if curl -fsS "\${SERVER_METRICS_URL}" -o "\${SERVER_METRICS_PROM}" >/dev/null 2>&1; then+ echo "Captured aggregate Rust server metrics"+ else+ echo "WARNING: Rust server metrics could not be collected from \${SERVER_METRICS_URL}" >&2+ fi+ FINALIZE_CMD+=(--server-metrics "\${SERVER_METRICS_PROM}")+ if curl -fsS "\${SERVER_STATS_URL}" -o "\${ROUTING_STATS_JSON}" >/dev/null 2>&1; then+ echo "Captured aggregate Rust server routing stats"+ else+ echo "WARNING: Rust server stats could not be collected from \${SERVER_STATS_URL}" >&2+ fi+ FINALIZE_CMD+=(--routing-stats "\${ROUTING_STATS_JSON}")fi"\${FINALIZE_CMD[@]}" || truediff --git a/benchmark/run_cost.py b/benchmark/run_cost.pydeleted file mode 100644index 3b7f3250f..000000000--- a/benchmark/run_cost.py+++ /dev/null@@ -1,105 +0,0 @@-#!/usr/bin/env python3-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0-"""Per-model cost breakdown for a benchmark run's routing log.--Aggregates ``routing_requests.jsonl`` by served model and prices each bucket with-:func:`switchyard.lib.cost_estimator.estimate_model_cost`, which splits input into-base / cache-read / cache-write tiers. That split matters: agentic runs are-cache-dominated (90%+ of prompt tokens are cache reads is normal), so a flat-input rate overstates cost several-fold.--On a run that routes sub-agent work to a different tier, the served-model column-doubles as the parent/child attribution key — the routing log records no-``agent_id``, so which model answered is the only role signal available.--Models absent from the price table contribute $0 and are listed separately, so an-unpriced model reads as a gap rather than as a free one.--Usage:- uv run python benchmark/run_cost.py --run benchmark/tb_runs/<run-name>-"""--from __future__ import annotations--import argparse-import collections-import json-from pathlib import Path--from switchyard.lib.cost_estimator import MODEL_PRICING, estimate_model_cost--_TOKEN_FIELDS = (- "prompt_tokens",- "completion_tokens",- "cached_tokens",- "cache_creation_tokens",-)---def _aggregate(routing_log: Path) -> dict[str, collections.Counter[str]]:- """Sum request counts and token fields per served model."""- totals: dict[str, collections.Counter[str]] = collections.defaultdict(collections.Counter)- for line in routing_log.read_text().splitlines():- if not line.strip():- continue- record = json.loads(line)- bucket = totals[record.get("model") or "<unknown>"]- bucket["reqs"] += 1- for field in _TOKEN_FIELDS:- bucket[field] += record.get(field) or 0- return totals---def _mean_agg_score(run: Path) -> tuple[float, int] | None:- """Mean rubric ``agg_score`` across the run's scored tasks, if any."""- scores = [- json.loads(path.read_text())["agg_score"]- for path in run.glob("jobs/*/task-*/verifier/evaluation_results.json")- ]- return (sum(scores) / len(scores), len(scores)) if scores else None---def main() -> None:- parser = argparse.ArgumentParser(description=__doc__)- parser.add_argument("--run", required=True, type=Path, help="Run directory under tb_runs/")- args = parser.parse_args()-- totals = _aggregate(args.run / "routing_requests.jsonl")- if not totals:- print("no routing records found")- return-- unpriced = [model for model in totals if model not in MODEL_PRICING]- total_cost = 0.0- header = f"{'model':46s} {'reqs':>5s} {'in':>11s} {'out':>9s} {'cached':>11s} {'USD':>9s}"- print(header)- for model, bucket in sorted(totals.items(), key=lambda item: -item[1]["reqs"]):- cost = estimate_model_cost(- model,- bucket["prompt_tokens"],- bucket["completion_tokens"],- bucket["cached_tokens"],- bucket["cache_creation_tokens"],- )["total_cost"]- total_cost += cost- print(- f"{model:46s} {bucket['reqs']:5d} {bucket['prompt_tokens']:11,} "- f"{bucket['completion_tokens']:9,} {bucket['cached_tokens']:11,} {cost:9.3f}"- )-- requests = sum(bucket["reqs"] for bucket in totals.values())- print(f"\nTOTAL: ${total_cost:.2f} over {requests} requests")- if unpriced:- # Priced at zero by estimate_model_cost — surface it so the total is not- # mistaken for complete.- print(f"NOTE: no price entry for {', '.join(sorted(unpriced))} — excluded from the total")-- scored = _mean_agg_score(args.run)- if scored:- mean, count = scored- print(f"mean agg_score: {mean:.3f} over {count} tasks → ${total_cost / count:.2f}/task")---if __name__ == "__main__":- main()diff --git a/benchmark/run_manifest.py b/benchmark/run_manifest.pyindex 92879b5bf..30ed596e1 100644--- a/benchmark/run_manifest.py+++ b/benchmark/run_manifest.py@@ -117,14 +117,14 @@ def dataset_fingerprint(return f"sha256:{hashlib.sha256(encoded).hexdigest()}"-def snapshot_routing_profiles(source: Path | None, run_dir: Path) -> Path | None:- """Copy the route bundle used for the run into the run directory."""+def snapshot_server_config(source: Path | None, run_dir: Path) -> Path | None:+ """Copy the Rust server configuration used for the run."""if source is None:return Noneif not source.is_file():raise FileNotFoundError(source)- dest = run_dir.resolve() / "routing_profiles" / source.name+ dest = run_dir.resolve() / "server_config" / source.namedest.parent.mkdir(parents=True, exist_ok=True)shutil.copyfile(source, dest)return dest@@ -223,116 +223,15 @@ def _copy_if_present(source: Path | None, dest: Path | None) -> str:return "present"-_TOKEN_FIELDS = (- "prompt_tokens",- "cached_tokens",- "cache_creation_tokens",- "completion_tokens",- "reasoning_tokens",- "total_tokens",-)---def _new_section() -> dict[str, Any]:- return {"calls": 0, "totals": dict.fromkeys(_TOKEN_FIELDS, 0), "_buckets": {}}---def _accumulate(section: dict[str, Any], record: dict[str, Any]) -> None:- section["calls"] += 1- model = record.get("model") or "unknown"- tier = record.get("tier") or ""- bucket = section["_buckets"].setdefault(- (model, tier),- {"model": model, "tier": tier, "calls": 0, **dict.fromkeys(_TOKEN_FIELDS, 0)},- )- bucket["calls"] += 1- for field in _TOKEN_FIELDS:- value = record.get(field)- value = value if isinstance(value, int) else 0- bucket[field] += value- section["totals"][field] += value---def _finalize_section(section: dict[str, Any]) -> None:- buckets = section.pop("_buckets")- section["models"] = [buckets[key] for key in sorted(buckets)]---def _kept_sessions(records: list[dict[str, Any]]) -> set[str]:- """The kept attempt of each trial is its latest-timestamped session.-- Harbor retries sequentially and keeps only the final attempt, so within a- trial the session with the newest timestamp is the one that survived.- Records without a ``trial_id`` are all treated as kept (no netting).- """- latest: dict[str, tuple[str, str]] = {}- for record in records:- trial = record.get("trial_id")- session = record.get("session_id")- if not trial or not session:- continue- ts = record.get("ts") or ""- if trial not in latest or ts > latest[trial][0]:- latest[trial] = (ts, session)- return {session for _, session in latest.values()}---def summarize_routing_log(log_path: Path) -> dict[str, Any]:- """Roll the routing log into per-task ``final`` and ``retries`` sections.-- Requests are grouped by trial; the latest attempt in each trial is ``final``- and earlier attempts are ``retries`` (see :func:`_kept_sessions`).- """- by_task: dict[str, list[dict[str, Any]]] = {}- total_requests = 0- for raw in log_path.read_text().splitlines():- raw = raw.strip()- if not raw:- continue- try:- record = json.loads(raw)- except json.JSONDecodeError:- continue- if not isinstance(record, dict):- continue- total_requests += 1- by_task.setdefault(record.get("task") or "unattributed", []).append(record)-- tasks: dict[str, dict[str, Any]] = {}- for task, records in by_task.items():- kept = _kept_sessions(records)- entry = {- "requests": len(records),- "n_retries": 0,- "final": _new_section(),- "retries": _new_section(),- }- retry_sessions: set[str] = set()- for record in records:- session = record.get("session_id")- # A record is a retry only when it has a trial_id and its session was- # not the kept one; untagged records default to final.- is_retry = bool(record.get("trial_id")) and session not in kept- _accumulate(entry["retries" if is_retry else "final"], record)- if is_retry and session:- retry_sessions.add(session)- entry["n_retries"] = len(retry_sessions)- _finalize_section(entry["final"])- _finalize_section(entry["retries"])- tasks[task] = entry- return {"total_requests": total_requests, "tasks": tasks}--def finalize_manifest(path: Path,*,harbor_rc: int | None,harbor_job_dir: Path | None = None,+ server_metrics: Path | None = None,routing_stats: Path | None = None,- routing_log: Path | None = None,- routing_stats_by_task: Path | None = None,) -> int:- """Copy run artifacts into place, roll up the routing log, and record outcomes.+ """Copy run artifacts into place and record outcomes.Returns 0 on success, 1 when the manifest is missing."""@@ -349,6 +248,14 @@ def finalize_manifest(if harbor_job_dir is not None:outcomes["harbor_job_dir"] = str(harbor_job_dir.resolve())+ metrics_dest = (+ Path(outcomes["server_metrics_prom"]) if outcomes.get("server_metrics_prom") else None+ )+ if outcomes.get("server_metrics_prom_status") != "not-requested":+ if server_metrics is None and metrics_dest is not None:+ server_metrics = metrics_dest+ outcomes["server_metrics_prom_status"] = _copy_if_present(server_metrics, metrics_dest)+stats_dest = (Path(outcomes["routing_stats_json"]) if outcomes.get("routing_stats_json") else None)@@ -370,20 +277,6 @@ def finalize_manifest(breakclosed_book["proxy_strip_log_status"] = _copy_if_present(strip_source, strip_dest)- if routing_stats_by_task is not None:- status = "missing"- if routing_log is not None and routing_log.is_file():- try:- summary = summarize_routing_log(routing_log)- routing_stats_by_task.write_text(- json.dumps(summary, indent=2, sort_keys=False) + "\n"- )- status = "present"- except OSError:- status = "missing"- outcomes["routing_stats_by_task_json"] = str(routing_stats_by_task.resolve())- outcomes["routing_stats_by_task_json_status"] = status-outcomes["harbor_rc"] = harbor_rcoutcomes["completed_at"] = _iso_timestamp()path.write_text(json.dumps(manifest, indent=2, sort_keys=False) + "\n")@@ -422,12 +315,11 @@ def _cli_main(argv: list[str] | None = None) -> int:write.add_argument("--server-port", type=int, default=0)write.add_argument("--server-argv-json", default="[]")write.add_argument("--server-config-json", default="{}")- write.add_argument("--classifier-prompts-json", default="{}")write.add_argument("--harbor-server-url", default="")write.add_argument("--harbor-base-url", default="")write.add_argument("--upstream-base-url", default="")write.add_argument("--upstream-api-key-env", default="")- write.add_argument("--routing-profiles", type=Path, default=None)+ write.add_argument("--server-config", type=Path, default=None)write.add_argument("--route-model", default="")write.add_argument("--harbor-command-json", default="[]")write.add_argument("--dataset-label", default="")@@ -452,6 +344,8 @@ def _cli_main(argv: list[str] | None = None) -> int:write.add_argument("--run-dir", type=Path, required=True)write.add_argument("--log-path", type=Path, required=True)write.add_argument("--harbor-result-json", type=Path, required=True)+ write.add_argument("--server-metrics-prom", type=Path, default=None)+ write.add_argument("--server-metrics-status", default="not-requested")write.add_argument("--routing-stats-json", type=Path, required=True)write.add_argument("--routing-stats-status", default="predicted")write.add_argument("--extra", action="append", default=[])@@ -460,9 +354,8 @@ def _cli_main(argv: list[str] | None = None) -> int:finalize.add_argument("--manifest", type=Path, required=True)finalize.add_argument("--harbor-rc", type=int, default=None)finalize.add_argument("--harbor-job-dir", type=Path, default=None)+ finalize.add_argument("--server-metrics", type=Path, default=None)finalize.add_argument("--routing-stats", type=Path, default=None)- finalize.add_argument("--routing-log", type=Path, default=None)- finalize.add_argument("--routing-stats-by-task", type=Path, default=None)ns = parser.parse_args(argv)if ns.command == "finalize":@@ -470,27 +363,22 @@ def _cli_main(argv: list[str] | None = None) -> int:ns.manifest,harbor_rc=ns.harbor_rc,harbor_job_dir=ns.harbor_job_dir,+ server_metrics=ns.server_metrics,routing_stats=ns.routing_stats,- routing_log=ns.routing_log,- routing_stats_by_task=ns.routing_stats_by_task,)if ns.command != "write":parser.print_help()return 2task_list = ns.task_list_file.resolve() if ns.task_list_file else None- routing_profiles = ns.routing_profiles.resolve() if ns.routing_profiles else None+ server_config = ns.server_config.resolve() if ns.server_config else Noneharbor_path = ns.harbor_path.resolve() if ns.harbor_path else Nonecodex_model_catalog = (ns.codex_model_catalog.resolve() if ns.codex_model_catalog else None)run_dir = ns.run_dir.resolve()- classifier_prompts = _json_arg(ns.classifier_prompts_json, {})- if not isinstance(classifier_prompts, dict):- print("ERROR: --classifier-prompts-json must decode to a JSON object")- return 2try:- routing_profiles_snapshot = snapshot_routing_profiles(routing_profiles, run_dir)+ server_config_snapshot = snapshot_server_config(server_config, run_dir)dataset_manifest_snapshot = snapshot_dataset_manifest(harbor_path, run_dir)except OSError as exc:print(f"ERROR: failed to snapshot run inputs: {exc}")@@ -550,20 +438,17 @@ def _cli_main(argv: list[str] | None = None) -> int:"port": ns.server_port or None,"argv": _json_arg(ns.server_argv_json, []),"config": _json_arg(ns.server_config_json, {}),- "classifier_prompts": classifier_prompts,"harbor_server_url": _opt(ns.harbor_server_url),"harbor_base_url": _opt(ns.harbor_base_url),"upstream_base_url": _opt(ns.upstream_base_url),"upstream_api_key_env": _opt(ns.upstream_api_key_env),- "routing_profiles": str(routing_profiles) if routing_profiles else None,- "routing_profiles_digest": (- path_digest(routing_profiles) if routing_profiles else None+ "server_config": str(server_config) if server_config else None,+ "server_config_digest": path_digest(server_config) if server_config else None,+ "server_config_snapshot": (+ str(server_config_snapshot) if server_config_snapshot else None),- "routing_profiles_snapshot": (- str(routing_profiles_snapshot) if routing_profiles_snapshot else None- ),- "routing_profiles_snapshot_digest": (- path_digest(routing_profiles_snapshot) if routing_profiles_snapshot else None+ "server_config_snapshot_digest": (+ path_digest(server_config_snapshot) if server_config_snapshot else None),"route_model": _opt(ns.route_model),},@@ -579,6 +464,10 @@ def _cli_main(argv: list[str] | None = None) -> int:"log_path": str(ns.log_path.resolve()),"harbor_result_json": str(ns.harbor_result_json.resolve()),"harbor_result_json_status": "predicted",+ "server_metrics_prom": (+ str(ns.server_metrics_prom.resolve()) if ns.server_metrics_prom else None+ ),+ "server_metrics_prom_status": ns.server_metrics_status,"routing_stats_json": str(ns.routing_stats_json.resolve()),"routing_stats_json_status": ns.routing_stats_status,"harbor_rc": None,diff --git a/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml b/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.tomlnew file mode 100644index 000000000..0ddfd9823--- /dev/null+++ b/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml@@ -0,0 +1,35 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+#+# Rust task-classifier configuration for Harbor Terminal-Bench Lite.+# Harbor selects this route with --model switchyard.++schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.classifier]+id = "google/gemini-3.5-flash"+llm_client = "openrouter"++[targets.strong]+id = "anthropic/claude-opus-4.7"+llm_client = "openrouter"++[targets.weak]+id = "moonshotai/kimi-k2.7-code"+llm_client = "openrouter"++[routes.switchyard]+id = "switchyard"+type = "llm_classifier"+classifier_target = "classifier"+strong_target = "strong"+weak_target = "weak"+base_threshold = 0.5+min_confidence = 0.0+session_affinity = true+message_hash_fallback = truediff --git a/benchmark/server-configs/tb-lite-single-gpt-5-5.toml b/benchmark/server-configs/tb-lite-single-gpt-5-5.tomlnew file mode 100644index 000000000..5f1898ae4--- /dev/null+++ b/benchmark/server-configs/tb-lite-single-gpt-5-5.toml@@ -0,0 +1,18 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++schema_version = 1++[llm_clients.openrouter]+format = "openai_responses"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.primary]+id = "openai/gpt-5.5"+llm_client = "openrouter"++[routes.primary]+id = "tb-lite-single-gpt-5-5"+type = "passthrough"+target = "primary"diff --git a/benchmark/server-configs/tb-lite-single-opus-4-7.toml b/benchmark/server-configs/tb-lite-single-opus-4-7.tomlnew file mode 100644index 000000000..ab3532a77--- /dev/null+++ b/benchmark/server-configs/tb-lite-single-opus-4-7.toml@@ -0,0 +1,18 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++schema_version = 1++[llm_clients.openrouter]+format = "openai_chat"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"++[targets.primary]+id = "anthropic/claude-opus-4.7"+llm_client = "openrouter"++[routes.primary]+id = "tb-lite-single-opus-4-7"+type = "passthrough"+target = "primary"diff --git a/docs/cli_reference.md b/docs/cli_reference.mdindex bc4df7274..9950631a5 100644--- a/docs/cli_reference.md+++ b/docs/cli_reference.md@@ -82,7 +82,7 @@ For `type: deterministic` routes, the `classifier:` block also accepts:| `max_request_chars` | Optional cap on the serialized request summary sent to the classifier before truncation. Defaults to `16000`; minimum `256`. || `recent_turn_window` | Number of trailing conversation turns included alongside the stable system/first-user anchors. Defaults to `4`. |-Benchmark runs started through `benchmark/run-baseline.sh --routing-profiles` record the effective classifier prompt, prompt SHA-256, `max_request_chars`, and `recent_turn_window` in `run_manifest.json` under `server.classifier_prompts`.+Benchmark runs started through `benchmark/run-baseline.sh --server-config` copy the server TOML into the run directory and record its path and SHA-256 in `run_manifest.json` under `server.server_config_snapshot` and `server.server_config_snapshot_digest`.Each route name becomes a model ID on `GET /v1/models`. Clients select a routeby setting the request's `model` field to that name.@@ -115,8 +115,9 @@ The Intake sink posts live model-call captures to nemo-platformderives queryable token fields from `response.usage` and queryable cost fieldsfrom top-level `cost_usd`, `cost_input_usd`, `cost_output_usd`, and`cost_details`. Switchyard emits cost fields only when the served model has a-known pricing entry; `routing_stats_final.json` remains the run-level source-for aggregate routing/model cost estimates. When a session ID is present,+known pricing entry. `routing_stats_final.json` carries the run-level aggregate+`/v1/stats` snapshot (per-model and per-tier calls, errors, tokens, and+latency); it does not include cost estimates. When a session ID is present,Switchyard also maps the Intake app/task labels into top-level`evaluation_context.dataset_*` and `evaluation_context.test_case_id` for spanqueries while keeping the original labels under `request.switchyard`.diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.mdindex 9ffc90763..23b4196e0 100644--- a/examples/experimental/litellm/README.md+++ b/examples/experimental/litellm/README.md@@ -182,7 +182,7 @@ Then run the three-task smoke benchmark in the foreground:SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \bash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles examples/experimental/litellm/benchmark-route.yaml \+ --server-config examples/experimental/litellm/benchmark-route.toml \--task-list-file examples/experimental/litellm/benchmark-tasks.txt \--model litellm-random \--agent codex \@@ -224,7 +224,7 @@ PATH="$CODEX_SHIM_DIR:$PATH" \SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \bash benchmark/run-baseline.sh \--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \- --routing-profiles examples/experimental/litellm/benchmark-route.yaml \+ --server-config examples/experimental/litellm/benchmark-route.toml \--task-list-file examples/experimental/litellm/benchmark-tasks.txt \--model litellm-random \--agent codex \diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.tomlnew file mode 100644index 000000000..1ef063bd6--- /dev/null+++ b/examples/experimental/litellm/benchmark-route.toml@@ -0,0 +1,23 @@+schema_version = 1++[llm_clients.litellm]+format = "openai_chat"+base_url = "http://litellm:4000/v1"++[llm_clients.litellm.extra_headers]+authorization = "Bearer not-needed"++[targets.strong]+id = "strong"+llm_client = "litellm"++[targets.fast]+id = "fast"+llm_client = "litellm"++[routes.litellm_random]+id = "litellm-random"+type = "random"+targets = ["strong", "fast"]+weights = [0.5, 0.5]+seed = 42diff --git a/examples/experimental/litellm/benchmark-route.yaml b/examples/experimental/litellm/benchmark-route.yamldeleted file mode 100644index f7a9d4ca8..000000000--- a/examples/experimental/litellm/benchmark-route.yaml+++ /dev/null@@ -1,19 +0,0 @@-defaults:- api_key: not-needed- base_url: http://litellm:4000/v1- format: openai- timeout_secs: 600.0--routes:- litellm-random:- type: random_routing- strong:- id: strong- model: strong- weak:- id: fast- model: fast- strong_probability: 0.5- rng_seed: 42- fallback_target_on_evict: strong- enable_stats: truediff --git a/tests/test_resolve_classifier_prompts.py b/tests/test_resolve_classifier_prompts.pydeleted file mode 100644index f65b72884..000000000--- a/tests/test_resolve_classifier_prompts.py+++ /dev/null@@ -1,181 +0,0 @@-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.-# SPDX-License-Identifier: Apache-2.0--from __future__ import annotations--import hashlib-import importlib.util-from pathlib import Path-from types import ModuleType--import pytest-import yaml--from switchyard.cli.route_bundle import build_route_bundle_table-from switchyard.lib.processors.llm_classifier import (- DEFAULT_MAX_REQUEST_CHARS,- LLMClassifierRequestProcessor,-)-from switchyard.lib.processors.llm_classifier.presets import profile_default_prompt--RESOLVER = Path(__file__).resolve().parents[1] / "benchmark" / "resolve_classifier_prompts.py"---def _load_resolver_module() -> ModuleType:- spec = importlib.util.spec_from_file_location(- "switchyard_benchmark_resolve_classifier_prompts",- RESOLVER,- )- assert spec is not None- assert spec.loader is not None- module = importlib.util.module_from_spec(spec)- spec.loader.exec_module(module)- return module---def _write(tmp_path: Path, bundle: dict) -> Path:- path = tmp_path / "routes.yaml"- path.write_text(yaml.safe_dump(bundle))- return path---def _bundle(prompt: str | None = None) -> dict:- classifier: dict[str, object] = {- "model": "c",- "api_key": "k",- "base_url": "https://example.invalid/v1",- "recent_turn_window": 2,- "max_request_chars": 1024,- }- if prompt is not None:- classifier["prompt"] = prompt- return {- "routes": {- "r/llm": {- "type": "deterministic",- "fallback_target_on_evict": "strong",- "profile": "coding_agent",- "classifier": classifier,- "strong": {- "model": "s",- "api_key": "k",- "base_url": "https://example.invalid/v1",- },- "weak": {- "model": "w",- "api_key": "k",- "base_url": "https://example.invalid/v1",- },- },- },- }---def test_explicit_prompt_override_is_recorded(tmp_path: Path) -> None:- module = _load_resolver_module()- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle("MY PROMPT")))-- assert out["r/llm"]["source"] == "override"- assert out["r/llm"]["classifier_prompt"] == "MY PROMPT"- assert out["r/llm"]["classifier_prompt_sha256"] == hashlib.sha256(- b"MY PROMPT"- ).hexdigest()- assert out["r/llm"]["profile"] == "coding_agent"- assert out["r/llm"]["max_request_chars"] == 1024- assert out["r/llm"]["recent_turn_window"] == 2---def test_profile_default_is_resolved(tmp_path: Path) -> None:- module = _load_resolver_module()- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle()))-- assert out["r/llm"]["source"] == "profile_default"- assert out["r/llm"]["classifier_prompt"] == profile_default_prompt("coding_agent")---def test_default_profile_is_general_when_absent(tmp_path: Path) -> None:- module = _load_resolver_module()- bundle = _bundle()- del bundle["routes"]["r/llm"]["profile"]-- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))-- assert out["r/llm"]["profile"] == "general"---def test_non_deterministic_routes_are_skipped(tmp_path: Path) -> None:- module = _load_resolver_module()- bundle = {- "routes": {- "r/pass": {"type": "model", "target": "some/model"},- "r/rand": {"type": "random_routing"},- },- }-- assert module.resolve_classifier_prompts(_write(tmp_path, bundle)) == {}---def test_whitespace_only_prompt_uses_profile_default(tmp_path: Path) -> None:- module = _load_resolver_module()- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle(" ")))-- assert out["r/llm"]["source"] == "profile_default"---def test_context_defaults_are_recorded(tmp_path: Path) -> None:- module = _load_resolver_module()- bundle = _bundle()- classifier = bundle["routes"]["r/llm"]["classifier"]- del classifier["recent_turn_window"]- del classifier["max_request_chars"]-- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))-- assert out["r/llm"]["recent_turn_window"] == 4- assert out["r/llm"]["max_request_chars"] == DEFAULT_MAX_REQUEST_CHARS---def test_subminimum_max_request_chars_uses_default(tmp_path: Path) -> None:- module = _load_resolver_module()- bundle = _bundle()- classifier = bundle["routes"]["r/llm"]["classifier"]- classifier["max_request_chars"] = 255-- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))-- assert out["r/llm"]["max_request_chars"] == DEFAULT_MAX_REQUEST_CHARS---def test_resolver_matches_server_effective_prompt_override(- tmp_path: Path,- monkeypatch: pytest.MonkeyPatch,-) -> None:- module = _load_resolver_module()- bundle = _bundle("CUSTOM SERVER PROMPT")- monkeypatch.setattr("switchyard.cli.route_bundle.fetch_model_ids", lambda *_: [])-- table = build_route_bundle_table(bundle)- switchyard = table.lookup_switchyard("r/llm")- classifier = next(- c for c in switchyard.iter_components()- if isinstance(c, LLMClassifierRequestProcessor)- )-- resolved = module.resolve_classifier_prompts(_write(tmp_path, bundle))["r/llm"]- assert resolved["classifier_prompt"] == classifier._config.system_prompt- assert resolved["max_request_chars"] == classifier._config.max_request_chars- assert resolved["recent_turn_window"] == classifier._config.recent_turn_window---def test_prompt_env_vars_match_route_bundle_expansion(- tmp_path: Path,- monkeypatch: pytest.MonkeyPatch,-) -> None:- module = _load_resolver_module()- monkeypatch.setenv("CUSTOM_CLASSIFIER_PROMPT", "expanded prompt")-- out = module.resolve_classifier_prompts(- _write(tmp_path, _bundle("${CUSTOM_CLASSIFIER_PROMPT}")),- )-- assert out["r/llm"]["classifier_prompt"] == "expanded prompt"diff --git a/tests/test_run_baseline_script.py b/tests/test_run_baseline_script.pyindex 85550f743..86e19e97e 100644--- a/tests/test_run_baseline_script.py+++ b/tests/test_run_baseline_script.py@@ -62,6 +62,50 @@ exit 2return harbor+def _write_fake_docker(bin_dir: Path) -> Path:+ docker = bin_dir / "docker"+ docker.write_text(+ "#!/usr/bin/env bash\n"+ "if [[ -n \"${FAKE_DOCKER_LOG:-}\" ]]; then\n"+ " printf '%s\\n' \"$*\" >> \"${FAKE_DOCKER_LOG}\"\n"+ "fi\n"+ "exit 0\n"+ )+ docker.chmod(0o755)+ return docker+++def _write_fake_curl(bin_dir: Path) -> Path:+ curl = bin_dir / "curl"+ curl.write_text(+ """+#!/usr/bin/env bash+set -euo pipefail+output=""+url=""+while [[ $# -gt 0 ]]; do+ case "$1" in+ -o) output="$2"; shift 2 ;;+ *) url="$1"; shift ;;+ esac+done+case "${url}" in+ */health) body='{"status":"ok"}' ;;+ */metrics) body='switchyard_total_requests 2' ;;+ */v1/stats) body='{"total_requests":2,"total_errors":0,"models":{"strong":{"calls":2}}}' ;;+ *) exit 22 ;;+esac+if [[ -n "${output}" ]]; then+ printf '%s\\n' "${body}" > "${output}"+else+ printf '%s\\n' "${body}"+fi+""".lstrip()+ )+ curl.chmod(0o755)+ return curl++def _write_fake_harbor_patch(tmp_path: Path) -> Path:patch_file = tmp_path / "fake-harbor-agent-patches.diff"patch_file.write_text(@@ -106,10 +150,14 @@ def _run_baseline(*args: str,env: dict[str, str] | None = None,include_dataset: bool = True,+ fake_server: bool = False,):fake_bin = tmp_path / "default-bin"fake_bin.mkdir(exist_ok=True)_write_fake_harbor(fake_bin)+ if fake_server:+ _write_fake_docker(fake_bin)+ _write_fake_curl(fake_bin)fake_patch = _write_fake_harbor_patch(tmp_path)fake_harbor_python = _write_fake_harbor_python(tmp_path)base_path = env.get("PATH", os.environ["PATH"]) if env else os.environ["PATH"]@@ -158,21 +206,29 @@ def _option_values(argv: list[str], option: str) -> list[str]:return [argv[idx + 1] for idx, value in enumerate(argv[:-1]) if value == option]-def _write_route_profile(path: Path) -> Path:+def _write_server_config(path: Path) -> Path:path.write_text("""-routes:- tb-lite-random-routing:- type: random_routing- strong_probability: 0.5- rng_seed: 444- fallback_target_on_evict: strong- strong:- id: strong- model: strong-model- weak:- id: weak- model: weak-model+schema_version = 1++[llm_clients.test]+format = "openai_chat"+base_url = "https://example.test/v1"++[targets.strong]+id = "strong-model"+llm_client = "test"++[targets.weak]+id = "weak-model"+llm_client = "test"++[routes.random]+id = "tb-lite-random-routing"+type = "random"+targets = ["strong", "weak"]+weights = [1, 1]+seed = 444""".lstrip())return path@@ -222,8 +278,8 @@ def test_direct_mode_defaults_to_openrouter_upstream_without_switchyard(tmp_pathassert "upstream_url: https://openrouter.ai/api/v1" in result.stdoutassert "api_key_env: OPENROUTER_API_KEY" in result.stdoutassert "SERVER_CMD: <direct-upstream>" in result.stdout- assert "switchyard serve" not in result.stdout- assert "route_profile:" not in result.stdout+ assert "switchyard-server" not in result.stdout+ assert "server_config:" not in result.stdoutdef test_direct_mode_uses_generic_upstream_key_when_set(tmp_path: Path) -> None:@@ -269,7 +325,7 @@ def test_direct_mode_rejects_server_only_options(tmp_path: Path) -> None:)assert result.returncode != 0- assert "--server-extra requires --routing-profiles" in result.stderr+ assert "--server-extra requires --server-config" in result.stderrresult = _run_baseline(tmp_path,@@ -279,14 +335,29 @@ def test_direct_mode_rejects_server_only_options(tmp_path: Path) -> None:)assert result.returncode != 0- assert "--route-model requires --routing-profiles" in result.stderr+ assert "--route-model requires --server-config" in result.stderr-def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> None:+def test_legacy_routing_profiles_are_rejected(tmp_path: Path) -> None:result = _run_baseline(tmp_path,"--routing-profiles",- str(REPO / "benchmark" / "routing-profiles" / "tb-lite-single-opus-4-7.yaml"),+ str(tmp_path / "routes.yaml"),+ "--model",+ "switchyard",+ "--dry-run",+ )++ assert result.returncode != 0+ assert "--routing-profiles is no longer supported" in result.stderr+ assert "use --server-config" in result.stderr+++def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> None:+ result = _run_baseline(+ tmp_path,+ "--server-config",+ str(REPO / "benchmark" / "server-configs" / "tb-lite-single-opus-4-7.toml"),"--route-model","tb-lite-single-opus-4-7","--agent",@@ -299,6 +370,7 @@ def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> Nonassert _option_value(harbor, "--model") == "tb-lite-single-opus-4-7"assert "version=2.1.211" in _option_values(harbor, "--ak")assert "reasoning_effort=high" in _option_values(harbor, "--ak")+ assert "thinking=adaptive" in _option_values(harbor, "--ak")assert "reasoning: high" in result.stdoutassert "harbor_server: http://switchyard:4000" in result.stdout@@ -306,8 +378,8 @@ def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> Nondef test_dry_run_codex_gpt_defaults_high_reasoning(tmp_path: Path) -> None:result = _run_baseline(tmp_path,- "--routing-profiles",- str(REPO / "benchmark" / "routing-profiles" / "tb-lite-single-gpt-5-5.yaml"),+ "--server-config",+ str(REPO / "benchmark" / "server-configs" / "tb-lite-single-gpt-5-5.toml"),"--route-model","tb-lite-single-gpt-5-5","--agent",@@ -330,11 +402,11 @@ def test_dry_run_codex_gpt_defaults_high_reasoning(tmp_path: Path) -> None:def test_dry_run_explicit_empty_reasoning_omits_kwarg(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -348,15 +420,16 @@ def test_dry_run_explicit_empty_reasoning_omits_kwarg(tmp_path: Path) -> None:assert result.returncode == 0, result.stderrharbor = _line_argv(result.stdout, "HARBOR_CMD: ")assert not any(value.startswith("reasoning_effort=") for value in _option_values(harbor, "--ak"))+ assert not any(value.startswith("thinking=") for value in _option_values(harbor, "--ak"))assert "reasoning: unset" in result.stdoutdef test_dry_run_explicit_reasoning_overrides_default(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -370,14 +443,41 @@ def test_dry_run_explicit_reasoning_overrides_default(tmp_path: Path) -> None:assert result.returncode == 0, result.stderrharbor = _line_argv(result.stdout, "HARBOR_CMD: ")assert "reasoning_effort=medium" in _option_values(harbor, "--ak")+ assert "thinking=adaptive" in _option_values(harbor, "--ak")+++def test_dry_run_explicit_claude_thinking_overrides_adaptive_default(tmp_path: Path) -> None:+ profile = _write_server_config(tmp_path / "routes.toml")++ result = _run_baseline(+ tmp_path,+ "--server-config",+ str(profile),+ "--route-model",+ "tb-lite-random-routing",+ "--agent",+ "claude-code",+ "--reasoning-effort",+ "high",+ "--harbor-extra",+ "--ak",+ "--harbor-extra",+ "thinking=disabled",+ "--dry-run",+ )++ assert result.returncode == 0, result.stderr+ harbor = _line_argv(result.stdout, "HARBOR_CMD: ")+ thinking = [value for value in _option_values(harbor, "--ak") if value.startswith("thinking=")]+ assert thinking == ["thinking=disabled"]def test_harbor_path_is_required(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -391,14 +491,14 @@ def test_harbor_path_is_required(tmp_path: Path) -> None:def test_closed_book_dataset_rejects_unpatched_harbor(tmp_path: Path) -> None:dataset = _write_closed_book_dataset(tmp_path / "dataset")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")fake_python = _write_fake_harbor_python(tmp_path / "unpatched", patched=False)result = _run_baseline(tmp_path,"--harbor-path",str(dataset),- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -415,12 +515,12 @@ def test_closed_book_dataset_rejects_unpatched_harbor(tmp_path: Path) -> None:assert "patch -p1 <" in result.stderr-def test_dry_run_routing_profile_uses_switchyard_serve(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+def test_dry_run_server_config_uses_rust_switchyard_server(tmp_path: Path) -> None:+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--model","tb-lite-random-routing",@@ -430,23 +530,47 @@ def test_dry_run_routing_profile_uses_switchyard_serve(tmp_path: Path) -> None:assert result.returncode == 0, result.stderrserver = _line_argv(result.stdout, "SERVER_CMD: ")harbor = _line_argv(result.stdout, "HARBOR_CMD: ")- # New form: uv run --no-sync switchyard --routing-profiles FILE -- serve ...- assert server[:4] == ["uv", "run", "--no-sync", "switchyard"]- assert _option_value(server, "--routing-profiles") == str(profile)- assert "serve" in server+ assert server[0] == "switchyard-server"+ assert _option_value(server, "--config") == str(profile)assert _option_value(server, "--host") == "0.0.0.0"assert _option_value(server, "--port") == "4000"assert _option_value(harbor, "--model") == "openai/tb-lite-random-routing"- assert "route_profile:" in result.stdout+ assert "server_config:" in result.stdoutassert "route_model: tb-lite-random-routing" in result.stdout-def test_dry_run_route_model_alias_uses_switchyard_serve(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+def test_foreground_server_run_collects_rust_stats_endpoint(tmp_path: Path) -> None:+ profile = _write_server_config(tmp_path / "routes.toml")+ docker_log = tmp_path / "docker.log"result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",+ str(profile),+ "--route-model",+ "tb-lite-random-routing",+ "--foreground",+ fake_server=True,+ env={"FAKE_DOCKER_LOG": str(docker_log)},+ )++ assert result.returncode == 0, result.stderr+ run_dir = next((tmp_path / "out").glob("baseline-serve-serve-*"))+ manifest = json.loads((run_dir / "run_manifest.json").read_text())+ stats = json.loads((run_dir / "routing_stats_final.json").read_text())++ assert stats["total_requests"] == 2+ assert manifest["outcomes"]["routing_stats_json_status"] == "present"+ assert manifest["outcomes"]["server_metrics_prom_status"] == "present"+ assert "--env NVIDIA_API_KEY" in docker_log.read_text()+++def test_dry_run_route_model_alias_uses_rust_switchyard_server(tmp_path: Path) -> None:+ profile = _write_server_config(tmp_path / "routes.toml")++ result = _run_baseline(+ tmp_path,+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -456,18 +580,18 @@ def test_dry_run_route_model_alias_uses_switchyard_serve(tmp_path: Path) -> Noneassert result.returncode == 0, result.stderrserver = _line_argv(result.stdout, "SERVER_CMD: ")harbor = _line_argv(result.stdout, "HARBOR_CMD: ")- assert server[:4] == ["uv", "run", "--no-sync", "switchyard"]- assert "serve" in server+ assert server[0] == "switchyard-server"+ assert _option_value(server, "--config") == str(profile)assert _option_value(harbor, "--model") == "openai/tb-lite-random-routing"assert "route_model: tb-lite-random-routing" in result.stdout-def test_dry_run_routing_profile_honors_explicit_harbor_model(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+def test_dry_run_server_config_honors_explicit_harbor_model(tmp_path: Path) -> None:+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -481,33 +605,12 @@ def test_dry_run_routing_profile_honors_explicit_harbor_model(tmp_path: Path) ->assert _option_value(harbor, "--model") == "openai/custom-route-label"-def test_checked_in_routing_profiles_load(monkeypatch) -> None:- monkeypatch.setenv("OPENROUTER_API_KEY", "or-test")-- import switchyard.cli.route_bundle as route_bundle-- profiles = {- "gpt-5-5-deepseek-gemini.yaml": "switchyard",- "gpt-5-5-kimi-gemini.yaml": "switchyard",- "tb-lite-llm-classifier-opus-deepseek-gemini.yaml": "switchyard",- "tb-lite-llm-classifier-opus-kimi-gemini.yaml": "switchyard",- "tb-lite-single-opus-4-7.yaml": "tb-lite-single-opus-4-7",- "tb-lite-single-gpt-5-5.yaml": "tb-lite-single-gpt-5-5",- }- monkeypatch.setattr(route_bundle, "fetch_model_ids", lambda _base_url, _api_key: [])- for filename, model in profiles.items():- table = route_bundle.load_route_bundle_table(- REPO / "benchmark" / "routing-profiles" / filename- )- assert model in table.registered_models()--def test_dry_run_harbor_extra(tmp_path: Path) -> None:- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -529,13 +632,13 @@ def test_dry_run_harbor_path_uses_local_dataset_and_closed_book_artifact(tmp_path: Path,) -> None:dataset = _write_closed_book_dataset(tmp_path / "dataset")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,"--harbor-path",str(dataset),- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -572,13 +675,13 @@ def test_dry_run_harbor_path_uses_local_dataset_and_closed_book_artifact(def test_dry_run_open_book_uses_proxy_topology_without_tool_disables(tmp_path: Path) -> None:dataset = _write_closed_book_dataset(tmp_path / "dataset")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,"--harbor-path",str(dataset),- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -604,13 +707,13 @@ def test_dry_run_open_book_uses_proxy_topology_without_tool_disables(tmp_path: Pdef test_dry_run_claude_closed_book_merges_disallowed_tools(tmp_path: Path) -> None:dataset = _write_closed_book_dataset(tmp_path / "dataset")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,"--harbor-path",str(dataset),- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -632,13 +735,13 @@ def test_dry_run_claude_closed_book_merges_disallowed_tools(tmp_path: Path) -> Ndef test_dry_run_opencode_closed_book_disables_webfetch(tmp_path: Path) -> None:dataset = _write_closed_book_dataset(tmp_path / "dataset")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,"--harbor-path",str(dataset),- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -658,11 +761,11 @@ def test_dry_run_opencode_closed_book_disables_webfetch(tmp_path: Path) -> None:def test_task_list_file_expands_to_include_task_name(tmp_path: Path) -> None:task_list = tmp_path / "tasks.txt"task_list.write_text("# comment\nalpha\n\nbeta # inline\n")- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",@@ -680,11 +783,11 @@ def test_dry_run_uses_harbor_bin_override(tmp_path: Path) -> None:override_bin = tmp_path / "override-bin"override_bin.mkdir()expected_harbor = _write_fake_harbor(override_bin)- profile = _write_route_profile(tmp_path / "routes.yaml")+ profile = _write_server_config(tmp_path / "routes.toml")result = _run_baseline(tmp_path,- "--routing-profiles",+ "--server-config",str(profile),"--route-model","tb-lite-random-routing",diff --git a/tests/test_run_manifest.py b/tests/test_run_manifest.pyindex e920ac26d..d107f825c 100644--- a/tests/test_run_manifest.py+++ b/tests/test_run_manifest.py@@ -117,14 +117,12 @@ def test_cli_write_applies_extra_fields(tmp_path: Path) -> None:assert manifest["harbor"]["dataset_fingerprint"].startswith("sha256:")-def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:+def test_cli_write_records_server_config_digest(tmp_path: Path) -> None:module = _load_manifest_module()out = tmp_path / "run_manifest.json"run_dir = tmp_path / "run"- profile = tmp_path / "routes.yaml"- profile.write_text(- "routes:\n tb-lite-random-routing:\n type: model\n target: some/model\n"- )+ config = tmp_path / "routes.toml"+ config.write_text('schema_version = 1\n\n[routes.noop]\nid = "noop"\ntype = "noop"\n')dataset = tmp_path / "dataset"dataset.mkdir()@@ -135,12 +133,10 @@ def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:str(out),"--server-preset","serve",- "--routing-profiles",- str(profile),+ "--server-config",+ str(config),"--route-model",- "tb-lite-random-routing",- "--classifier-prompts-json",- '{"tb-lite-random-routing":{"classifier_prompt_sha256":"abc"}}',+ "noop","--harbor-path",str(dataset),"--agent",@@ -166,55 +162,13 @@ def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:assert rc == 0manifest = json.loads(out.read_text())- assert manifest["server"]["routing_profiles"] == str(profile)- assert manifest["server"]["routing_profiles_digest"] == module.path_digest(profile)- snapshot = run_dir / "routing_profiles" / profile.name- assert manifest["server"]["routing_profiles_snapshot"] == str(snapshot)- assert manifest["server"]["routing_profiles_snapshot_digest"] == module.path_digest(snapshot)- assert snapshot.read_bytes() == profile.read_bytes()- assert manifest["server"]["route_model"] == "tb-lite-random-routing"- assert manifest["server"]["classifier_prompts"] == {- "tb-lite-random-routing": {"classifier_prompt_sha256": "abc"}- }---def test_cli_write_rejects_non_object_classifier_prompts(tmp_path: Path) -> None:- module = _load_manifest_module()- out = tmp_path / "run_manifest.json"- run_dir = tmp_path / "run"-- rc = module._cli_main(- [- "write",- "--output",- str(out),- "--server-preset",- "serve",- "--classifier-prompts-json",- "[]",- "--agent",- "terminus-2",- "--harbor-model",- "tb-lite-random-routing",- "--n-concurrent",- "1",- "--max-retries",- "0",- "--agent-timeout-multiplier",- "1.0",- "--run-dir",- str(run_dir),- "--log-path",- str(run_dir / "run.log"),- "--harbor-result-json",- str(run_dir / "harbor_result.json"),- "--routing-stats-json",- str(run_dir / "routing_stats_final.json"),- ]- )-- assert rc == 2- assert not out.exists()+ assert manifest["server"]["server_config"] == str(config)+ assert manifest["server"]["server_config_digest"] == module.path_digest(config)+ snapshot = run_dir / "server_config" / config.name+ assert manifest["server"]["server_config_snapshot"] == str(snapshot)+ assert manifest["server"]["server_config_snapshot_digest"] == module.path_digest(snapshot)+ assert snapshot.read_bytes() == config.read_bytes()+ assert manifest["server"]["route_model"] == "noop"def test_cli_write_records_direct_upstream_mode_without_routing_stats(tmp_path: Path) -> None:@@ -272,7 +226,7 @@ def test_cli_write_records_direct_upstream_mode_without_routing_stats(tmp_path:assert manifest["server"]["mode"] == "direct"assert manifest["server"]["upstream_base_url"] == "https://inference-api.nvidia.com/v1"assert manifest["server"]["upstream_api_key_env"] == "NVIDIA_API_KEY"- assert manifest["server"]["routing_profiles"] is None+ assert manifest["server"]["server_config"] is Noneassert manifest["outcomes"]["routing_stats_json_status"] == "not-requested"rc = module.finalize_manifest(out, harbor_rc=0, harbor_job_dir=run_dir / "jobs" / "job")@@ -417,24 +371,35 @@ def test_finalize_copies_harbor_result_and_marks_stats(tmp_path: Path) -> None:stats = run_dir / "routing_stats_final.json"stats.parent.mkdir(parents=True, exist_ok=True)stats.write_text('{"total_requests":1}\n')+ metrics = run_dir / "server_metrics_final.prom"+ metrics.write_text("switchyard_requests_total 1\n")module.write_manifest(out,outcomes={"harbor_result_json": str(run_dir / "harbor_result.json"),"harbor_result_json_status": "predicted",+ "server_metrics_prom": str(metrics),+ "server_metrics_prom_status": "predicted","routing_stats_json": str(stats),"routing_stats_json_status": "predicted","harbor_rc": None,},)- rc = module.finalize_manifest(out, harbor_rc=0, harbor_job_dir=job_dir, routing_stats=stats)+ rc = module.finalize_manifest(+ out,+ harbor_rc=0,+ harbor_job_dir=job_dir,+ server_metrics=metrics,+ routing_stats=stats,+ )assert rc == 0manifest = json.loads(out.read_text())assert manifest["outcomes"]["harbor_rc"] == 0assert manifest["outcomes"]["harbor_result_json_status"] == "present"+ assert manifest["outcomes"]["server_metrics_prom_status"] == "present"assert manifest["outcomes"]["routing_stats_json_status"] == "present"assert (run_dir / "harbor_result.json").is_file()@@ -469,149 +434,3 @@ def test_finalize_copies_proxy_strip_log_artifact(tmp_path: Path) -> None:manifest = json.loads(out.read_text())assert manifest["closed_book"]["proxy_strip_log_status"] == "present"assert (run_dir / "proxy_strip_log.jsonl").read_text() == '{"removed":["web_search"]}\n'---def _record(task: str, ts: str, model: str, tier: str, *, trial: str = "t", session: str = "s",- **tokens: int) -> str:- base = {"prompt_tokens": 0, "cached_tokens": 0, "cache_creation_tokens": 0,- "completion_tokens": 0, "reasoning_tokens": 0, "total_tokens": 0}- base.update(tokens)- return json.dumps({"task": task, "ts": ts, "trial_id": trial, "session_id": session,- "model": model, "tier": tier, **base})---def test_summarize_routing_log_groups_by_task_and_model(tmp_path: Path) -> None:- module = _load_manifest_module()- log = tmp_path / "routing_requests.jsonl"- log.write_text(- "\n".join([- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",- prompt_tokens=10, cached_tokens=6, completion_tokens=2,- reasoning_tokens=1, total_tokens=12),- _record("task-a", "2026-01-01T00:00:02Z", "kimi", "weak",- prompt_tokens=4, completion_tokens=1, total_tokens=5),- "not json",- json.dumps(["not", "a", "dict"]),- ]) + "\n"- )-- summary = module.summarize_routing_log(log)-- assert summary["total_requests"] == 2- task_a = summary["tasks"]["task-a"]- assert task_a["requests"] == 2- assert task_a["n_retries"] == 0- assert task_a["retries"]["calls"] == 0- assert task_a["final"]["calls"] == 2- assert task_a["final"]["totals"] == {- "prompt_tokens": 14, "cached_tokens": 6, "cache_creation_tokens": 0,- "completion_tokens": 3, "reasoning_tokens": 1, "total_tokens": 17,- }- assert [(b["model"], b["tier"], b["calls"]) for b in task_a["final"]["models"]] == [- ("kimi", "weak", 1), ("opus", "strong", 1),- ]---def test_summarize_routing_log_nets_out_retried_attempt(tmp_path: Path) -> None:- module = _load_manifest_module()- log = tmp_path / "routing_requests.jsonl"- # Same trial, two attempts: the earlier session was retried away.- log.write_text(- "\n".join([- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",- trial="trial-1", session="retry-sess",- prompt_tokens=100, completion_tokens=10, total_tokens=110),- _record("task-a", "2026-01-01T00:05:01Z", "opus", "strong",- trial="trial-1", session="final-sess",- prompt_tokens=20, completion_tokens=3, total_tokens=23),- ]) + "\n"- )-- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]-- assert task_a["n_retries"] == 1- assert task_a["final"]["totals"]["total_tokens"] == 23- assert task_a["retries"]["totals"]["total_tokens"] == 110- assert task_a["final"]["calls"] == 1- assert task_a["retries"]["calls"] == 1---def test_summarize_routing_log_keeps_all_concurrent_trials(tmp_path: Path) -> None:- module = _load_manifest_module()- log = tmp_path / "routing_requests.jsonl"- # Two trials of the same task (k>1). Neither retried, so both are final even- # though their calls interleave in time.- log.write_text(- "\n".join([- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",- trial="trial-1", session="s1", total_tokens=10),- _record("task-a", "2026-01-01T00:00:02Z", "opus", "strong",- trial="trial-2", session="s2", total_tokens=20),- ]) + "\n"- )-- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]-- assert task_a["n_retries"] == 0- assert task_a["final"]["calls"] == 2- assert task_a["final"]["totals"]["total_tokens"] == 30- assert task_a["retries"]["calls"] == 0---def test_summarize_routing_log_untagged_records_all_final(tmp_path: Path) -> None:- module = _load_manifest_module()- log = tmp_path / "routing_requests.jsonl"- # No trial_id -> no netting, everything counts as final.- log.write_text(json.dumps({- "task": "task-a", "ts": "2026-01-01T00:00:01Z", "session_id": "s1",- "model": "opus", "tier": "strong", "total_tokens": 5,- }) + "\n")-- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]-- assert task_a["final"]["calls"] == 1- assert task_a["retries"]["calls"] == 0- assert task_a["n_retries"] == 0---def test_finalize_writes_routing_stats_by_task(tmp_path: Path) -> None:- module = _load_manifest_module()- out = tmp_path / "run_manifest.json"- run_dir = tmp_path / "run"- run_dir.mkdir()- log = run_dir / "routing_requests.jsonl"- log.write_text(json.dumps({- "task": "task-a", "session_id": "s1", "model": "opus", "tier": "strong",- "prompt_tokens": 10, "cached_tokens": 3, "cache_creation_tokens": 1,- "completion_tokens": 2, "total_tokens": 12,- }) + "\n")- by_task = run_dir / "routing_stats_by_task.json"-- module.write_manifest(out, outcomes={"harbor_rc": None})- rc = module.finalize_manifest(- out, harbor_rc=0, routing_log=log, routing_stats_by_task=by_task,- )-- assert rc == 0- summary = json.loads(by_task.read_text())- assert summary["tasks"]["task-a"]["final"]["models"][0]["calls"] == 1- assert summary["tasks"]["task-a"]["final"]["models"][0]["cached_tokens"] == 3- manifest = json.loads(out.read_text())- assert manifest["outcomes"]["routing_stats_by_task_json_status"] == "present"---def test_finalize_marks_routing_stats_by_task_missing_without_log(tmp_path: Path) -> None:- module = _load_manifest_module()- out = tmp_path / "run_manifest.json"- by_task = tmp_path / "routing_stats_by_task.json"-- module.write_manifest(out, outcomes={"harbor_rc": None})- rc = module.finalize_manifest(- out, harbor_rc=1,- routing_log=tmp_path / "absent.jsonl", routing_stats_by_task=by_task,- )-- assert rc == 0- assert not by_task.exists()- manifest = json.loads(out.read_text())- assert manifest["outcomes"]["routing_stats_by_task_json_status"] == "missing"