Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

chore(evals): add switchyard-server wiring to benchmarks (#177)

Ayush Agarwald9e90ee5b92e
diff --git a/benchmark/DATASETS.md b/benchmark/DATASETS.md
index a60d6b7ea..3e8596dfa 100644
--- a/benchmark/DATASETS.md
+++ b/benchmark/DATASETS.md
@@ -97,7 +97,7 @@ Pass the generated directory to the runner:
```bash
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/terminal-bench-2-closed-book \
- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \
--model switchyard \
--agent codex \
--n-tasks 1 \
@@ -105,8 +105,8 @@ bash benchmark/run-baseline.sh \
--max-retries 0
```
-Use a routing profile and `--model` that match the route key you want Harbor agents to call. Omit
-`--routing-profiles` for a direct-upstream comparison, as described in the main README.
+Use a server config and `--model` that match the route id you want Harbor agents to call. Omit
+`--server-config` for a direct-upstream comparison, as described in the main README.
Before reporting numbers for a new dataset, inspect:
@@ -117,7 +117,8 @@ run_manifest.json
jobs/<job-name>/<task-id>/agent/trajectory.json
```
-For Switchyard runs, also inspect `routing_stats_final.json`.
+For Switchyard runs, also inspect `server_metrics_final.prom` and
+`routing_stats_final.json`.
## What The Rewriter Changes
diff --git a/benchmark/README.md b/benchmark/README.md
index 72e340ac8..3a78eeaf0 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -10,7 +10,7 @@ It covers the two smoke paths most people need first:
- **Switchyard routing:** Harbor calls Switchyard, and Switchyard routes across two model tiers.
Both paths use the same generated dataset, task proxy, pinned agent versions, and run artifact
-layout. Passing `--routing-profiles` starts Switchyard; omitting it disables Switchyard and points
+layout. Passing `--server-config` starts the Rust server; omitting it disables Switchyard and points
Harbor directly at the upstream provider.
## Prerequisites
@@ -22,7 +22,7 @@ uv sync
```
You also need Docker with Compose support, because baseline runs launch task containers and use the
-generated benchmark proxy topology. Runs with `--routing-profiles` also start Switchyard inside
+generated benchmark proxy topology. Runs with `--server-config` also start Switchyard inside
Docker.
Harbor is installed as a dev dependency. Check that the CLI resolves from the uv environment:
@@ -55,8 +55,8 @@ bash benchmark/run-baseline.sh \
...
```
-Route-profile YAML files expand `${OPENROUTER_API_KEY}` by default. For another provider, copy a
-profile and update its `api_key`, `base_url`, and model ids to match that provider.
+Rust server TOML files refer to the credential through `api_key_env`. For another provider, copy a
+config and update its `api_key_env`, `base_url`, and model ids to match that provider.
## One-Time Setup
@@ -141,7 +141,7 @@ uv run --no-sync python benchmark/prepare_harbor_dataset.py \
## Run Without Switchyard
-Omit `--routing-profiles` to fully disable Switchyard. The runner still creates the benchmark
+Omit `--server-config` to fully disable Switchyard. The runner still creates the benchmark
Docker network for the generated proxy sidecar, but Harbor sends model calls straight to
`${UPSTREAM_BASE_URL:-https://openrouter.ai/api/v1}` using `OPENROUTER_API_KEY` by default:
@@ -162,14 +162,14 @@ Switchyard translation is disabled.
## Run With Switchyard Routing
-Pass `--routing-profiles` to start Switchyard and route Harbor traffic through it. This smoke test
-uses `benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml`, an
-LLM-as-classifier profile for coding-agent tasks:
+Pass `--server-config` to start `switchyard-server` and route Harbor traffic through it. This smoke
+test uses `benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml`, a Rust
+task-classifier configuration for coding-agent tasks:
```bash
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \
--model switchyard \
--agent codex \
--reasoning-effort xhigh \
@@ -178,8 +178,8 @@ bash benchmark/run-baseline.sh \
--max-retries 0
```
-Use the route key from the YAML as `--model`. In this profile, the Gemini classifier selects the
-target tier for each request, then Switchyard routes to one of:
+Use the route `id` from the TOML as `--model`. In this config, the Gemini classifier selects the
+target tier for the task, then Switchyard routes to one of:
- strong: `anthropic/claude-opus-4.7`
- weak: `moonshotai/kimi-k2.7-code`
@@ -189,8 +189,8 @@ Classifier model: `google/gemini-3.5-flash`.
To smoke-test a single-model Switchyard path instead, use one of:
```text
-benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml
-benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml
+benchmark/server-configs/tb-lite-single-gpt-5-5.toml
+benchmark/server-configs/tb-lite-single-opus-4-7.toml
```
By default, the runner starts in the background and prints the PID, log path, and kill command.
@@ -198,14 +198,14 @@ By default, the runner starts in the background and prints the PID, log path, an
## Book Modes
Both book modes use the same generated `--harbor-path` dataset, prebaked agent images, and proxy
-sidecar topology. Switchyard is Dockerized only when `--routing-profiles` is provided.
+sidecar topology. Switchyard is Dockerized only when `--server-config` is provided.
Closed-book mode is the default:
```bash
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \
--model switchyard \
--agent codex \
--n-tasks 1
@@ -221,14 +221,14 @@ Open-book mode keeps the same proxy path but broadens egress:
bash benchmark/run-baseline.sh \
--book-mode open \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \
--model switchyard \
--agent codex \
--n-tasks 1
```
Use open-book mode only when the evaluation intentionally allows internet access. The manifest
-records the mode, the local dataset digest, any copied routing profile, proxy metadata, upstream
+records the mode, the local dataset digest, a snapshot of the server config, proxy metadata, upstream
base URL for direct runs, and agent version pins in both modes.
## Run A Full TB Lite Pass
@@ -253,7 +253,7 @@ Switchyard LLM-classifier routing:
```bash
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
+ --server-config benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml \
--model switchyard \
--agent codex \
--reasoning-effort xhigh \
@@ -272,30 +272,21 @@ Run directories are created under `benchmark/tb_runs/`. The most useful artifact
run_manifest.json
server.log
harbor.log
+server_metrics_final.prom
routing_stats_final.json
-routing_requests.jsonl
-routing_stats_by_task.json
jobs/<job-name>/result.json
jobs/<job-name>/<task-id>/agent/trajectory.json
```
The manifest records the command, git state, Harbor patch provenance, local dataset digest, copied
-routing profile when present, direct-upstream metadata when Switchyard is disabled, book-mode
-settings, agent version pins, log paths, and final Harbor status. When the routing profile contains
-deterministic LLM-classifier routes, `server.classifier_prompts` records each route's effective
-prompt, prompt SHA-256, `max_request_chars`, and `recent_turn_window` for reproducibility. Direct
-runs mark routing stats as `not-requested`.
-
-Switchyard runs also write `routing_requests.jsonl` (one record per router request: task, trial,
-session, served model, tier, and the six token fields from the global schema) and roll it up into
-`routing_stats_by_task.json` at finalize. The per-task proxy sidecar stamps the identity headers on
-model-bound requests: `x-switchyard-intake-task`, `x-switchyard-trial-id` (Harbor's trial name), and
-a per-attempt `proxy_x_session_id`.
-
-Each task in `routing_stats_by_task.json` splits into `final` and `retries`. Requests are grouped by
-trial, and within a trial the latest attempt is the one Harbor kept (it retries sequentially and
-discards earlier attempts), so tokens spent on retried-away attempts land in `retries` and can be
-netted out of cost.
+server config, direct-upstream metadata when Switchyard is disabled, book-mode settings, agent
+version pins, log paths, and final Harbor status.
+
+`server_metrics_final.prom` is the final `/metrics` snapshot. `routing_stats_final.json` is the
+final aggregate `/v1/stats` snapshot, including model and tier calls, errors, tokens, and latency.
+Neither artifact provides task or trial attribution. The runner writes them only after Harbor exits
+and while the Rust server is still reachable; otherwise the manifest records them as missing.
+`routing_requests.jsonl` and `routing_stats_by_task.json` are not produced by the Rust server.
### Replay Stage-Router Scores
@@ -316,7 +307,8 @@ fall-open behavior.
## Docker Image Notes
-Baseline runs build `switchyard-baseline:local` from `benchmark/switchyard-server.Dockerfile`.
+Baseline runs build `switchyard-baseline:local` from
+`benchmark/switchyard-rust-server.Dockerfile`.
The default is to rebuild before each run so the container matches the current checkout.
To reuse an already built image:
@@ -325,7 +317,7 @@ To reuse an already built image:
SWITCHYARD_DOCKER_BUILD=0 bash benchmark/run-baseline.sh ...
```
-Only reuse the image when you know it already contains the current Switchyard code and Rust extension.
+Only reuse the image when you know it already contains the current Rust `switchyard-server` binary.
## Troubleshooting
diff --git a/benchmark/resolve_classifier_prompts.py b/benchmark/resolve_classifier_prompts.py
deleted file mode 100644
index cb7e14925..000000000
--- a/benchmark/resolve_classifier_prompts.py
+++ /dev/null
@@ -1,114 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-"""Resolve deterministic-route classifier prompts for benchmark manifests."""
-
-from __future__ import annotations
-
-import json
-import os
-import re
-import sys
-from pathlib import Path
-from typing import Any
-
-import yaml # type: ignore[import-untyped]
-
-from switchyard.lib.processors.llm_classifier import DEFAULT_MAX_REQUEST_CHARS
-from switchyard.lib.processors.llm_classifier.presets import (
- classifier_prompt_sha256,
- profile_default_prompt,
-)
-
-_DETERMINISTIC_TYPES = {"deterministic", "llm_classifier", "llm_classifier_routing"}
-_ENV_REF_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
-_MIN_MAX_REQUEST_CHARS = 256
-
-
-def _expand_env_string(value: str) -> str:
- missing = [name for name in _ENV_REF_RE.findall(value) if name not in os.environ]
- if missing:
- raise ValueError(
- f"missing environment variable(s): {', '.join(sorted(set(missing)))}"
- )
- return os.path.expandvars(value)
-
-
-def _int_or_default(value: object, default: int) -> int:
- if isinstance(value, bool) or not isinstance(value, int):
- return default
- return value
-
-
-def _bounded_int_or_default(value: object, default: int, *, minimum: int) -> int:
- if isinstance(value, bool) or not isinstance(value, int):
- return default
- if value < minimum:
- return default
- return value
-
-
-def resolve_classifier_prompts(path: Path) -> dict[str, dict[str, Any]]:
- """Return effective classifier-prompt metadata for deterministic routes."""
- loaded = yaml.safe_load(path.read_text())
- routes = loaded.get("routes") if isinstance(loaded, dict) else None
- if not isinstance(routes, dict):
- return {}
-
- result: dict[str, dict[str, Any]] = {}
- for route_key, route in routes.items():
- if not isinstance(route_key, str) or not isinstance(route, dict):
- continue
- route_type = str(route.get("type", "")).lower().replace("-", "_")
- if route_type not in _DETERMINISTIC_TYPES:
- continue
-
- profile = route.get("profile") or "general"
- if not isinstance(profile, str):
- raise ValueError(f"route {route_key!r}: profile must be a string")
-
- classifier = route.get("classifier")
- classifier_map = classifier if isinstance(classifier, dict) else {}
- raw_prompt = classifier_map.get("prompt")
- if isinstance(raw_prompt, str) and raw_prompt.strip():
- prompt = _expand_env_string(raw_prompt)
- source = "override"
- else:
- prompt = profile_default_prompt(profile)
- source = "profile_default"
-
- result[route_key] = {
- "profile": profile,
- "source": source,
- "classifier_prompt": prompt,
- "classifier_prompt_sha256": classifier_prompt_sha256(prompt),
- "max_request_chars": _bounded_int_or_default(
- classifier_map.get("max_request_chars"),
- DEFAULT_MAX_REQUEST_CHARS,
- minimum=_MIN_MAX_REQUEST_CHARS,
- ),
- "recent_turn_window": _int_or_default(
- classifier_map.get("recent_turn_window"),
- 4,
- ),
- }
- return result
-
-
-def main(argv: list[str] | None = None) -> int:
- """Print classifier-prompt metadata JSON; never fail a benchmark run."""
- args = argv if argv is not None else sys.argv[1:]
- if not args:
- print("WARNING: no routing-profiles path supplied", file=sys.stderr)
- print("{}")
- return 0
- try:
- print(json.dumps(resolve_classifier_prompts(Path(args[0]))))
- except Exception as exc: # noqa: BLE001
- print(f"WARNING: failed to resolve classifier prompts: {exc}", file=sys.stderr)
- print("{}")
- return 0
-
-
-if __name__ == "__main__":
- raise SystemExit(main())
diff --git a/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml b/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml
deleted file mode 100644
index 4ab7e2d72..000000000
--- a/benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# LLM-classifier GPT-5.5/DeepSeek/Gemini profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/gpt-5-5-deepseek-gemini.yaml \
-# --model switchyard
-
-routes:
- switchyard:
- type: deterministic
- profile: coding_agent
- enable_stats: true
- fallback_target_on_evict: strong
- strong:
- model: openai/gpt-5.5
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: responses
- timeout_secs: 600
- extra_body:
- reasoning_effort: high
- weak:
- model: deepseek/deepseek-v4-pro
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- classifier:
- model: google/gemini-3.5-flash
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 30
- min_confidence: 0.0
- fail_open: true
- recent_turn_window: 4
diff --git a/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml b/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml
deleted file mode 100644
index 690c14106..000000000
--- a/benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# LLM-classifier GPT-5.5/Kimi/Gemini profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/gpt-5-5-kimi-gemini.yaml \
-# --model switchyard
-
-routes:
- switchyard:
- type: deterministic
- profile: coding_agent
- enable_stats: true
- fallback_target_on_evict: strong
- strong:
- model: openai/gpt-5.5
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: responses
- timeout_secs: 600
- extra_body:
- reasoning:
- effort: high
- weak:
- model: moonshotai/kimi-k2.7-code
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- classifier:
- model: google/gemini-3.5-flash
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 30
- min_confidence: 0.0
- fail_open: true
- recent_turn_window: 4
diff --git a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml b/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml
deleted file mode 100644
index 8b08ea791..000000000
--- a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# LLM-classifier Opus/DeepSeek/Gemini profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-deepseek-gemini.yaml \
-# --model switchyard
-
-routes:
- switchyard:
- type: deterministic
- profile: coding_agent
- enable_stats: true
- fallback_target_on_evict: strong
- strong:
- model: anthropic/claude-opus-4.7
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- extra_body:
- reasoning:
- effort: high
- weak:
- model: deepseek/deepseek-v4-pro
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- classifier:
- model: google/gemini-3.5-flash
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 30
- min_confidence: 0.0
- fail_open: true
- recent_turn_window: 4
diff --git a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml b/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml
deleted file mode 100644
index 6428f4c75..000000000
--- a/benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# LLM-classifier Opus/Kimi/Gemini profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/tb-lite-llm-classifier-opus-kimi-gemini.yaml \
-# --model switchyard
-
-routes:
- switchyard:
- type: deterministic
- profile: coding_agent
- enable_stats: true
- fallback_target_on_evict: strong
- strong:
- model: anthropic/claude-opus-4.7
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- extra_body:
- reasoning:
- effort: high
- weak:
- model: moonshotai/kimi-k2.7-code
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- format: openai
- timeout_secs: 600
- classifier:
- model: google/gemini-3.5-flash
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 30
- min_confidence: 0
- fail_open: true
- recent_turn_window: 4
diff --git a/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml b/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml
deleted file mode 100644
index 72c327eb2..000000000
--- a/benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# Serve-backed single-model GPT-5.5 profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/tb-lite-single-gpt-5-5.yaml \
-# --model tb-lite-single-gpt-5-5
-# Harbor uses openai/tb-lite-single-gpt-5-5; LiteLLM strips the provider
-# prefix and sends this route key to Switchyard.
-
-defaults:
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 600.0
-
-routes:
- tb-lite-single-gpt-5-5:
- type: model
- target:
- model: openai/gpt-5.5
- format: responses
diff --git a/benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml b/benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml
deleted file mode 100644
index f7321d5b3..000000000
--- a/benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml
+++ /dev/null
@@ -1,26 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-#
-# Serve-backed single-model Opus 4.7 profile for Harbor Terminal-Bench Lite.
-# Run with:
-# benchmark/run-baseline.sh \
-# --harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
-# --routing-profiles benchmark/routing-profiles/tb-lite-single-opus-4-7.yaml \
-# --model tb-lite-single-opus-4-7
-# Harbor uses openai/tb-lite-single-opus-4-7; LiteLLM strips the provider
-# prefix and sends this route key to Switchyard.
-#
-# Note: provider support for agent sampling fields varies. If a provider rejects
-# a field, strip it in the benchmark agent config before running a full pass.
-
-defaults:
- api_key: ${OPENROUTER_API_KEY}
- base_url: https://openrouter.ai/api/v1
- timeout_secs: 600.0
-
-routes:
- tb-lite-single-opus-4-7:
- type: model
- target:
- model: anthropic/claude-opus-4.7
- format: openai
diff --git a/benchmark/run-baseline.sh b/benchmark/run-baseline.sh
index bc4caa830..fa0881658 100755
--- a/benchmark/run-baseline.sh
+++ b/benchmark/run-baseline.sh
@@ -34,7 +34,7 @@ OUTPUT_DIR="${REPO_ROOT}/benchmark/tb_runs"
PORT=4000
SERVER_URL=""
HARBOR_SERVER_URL=""
-ROUTING_PROFILES=""
+SERVER_CONFIG=""
MODEL=""
ROUTE_MODEL=""
AGENT="terminus-2"
@@ -47,7 +47,7 @@ BOOK_MODE="closed"
PROXY_STRIP_ARTIFACT="/etc/proxy-public/strip.jsonl"
VERIFIER_PROXY_TEMPLATE='${SWITCHYARD_VERIFIER_HTTP_PROXY}'
SWITCHYARD_DOCKER_IMAGE="${SWITCHYARD_DOCKER_IMAGE:-switchyard-baseline:local}"
-SWITCHYARD_DOCKERFILE="${SWITCHYARD_DOCKERFILE:-benchmark/switchyard-server.Dockerfile}"
+SWITCHYARD_DOCKERFILE="${SWITCHYARD_DOCKERFILE:-benchmark/switchyard-rust-server.Dockerfile}"
SWITCHYARD_DOCKER_SERVICE_NAME="${SWITCHYARD_DOCKER_SERVICE_NAME:-switchyard}"
SWITCHYARD_DOCKER_BUILD="${SWITCHYARD_DOCKER_BUILD:-1}"
SWITCHYARD_DOCKER_NETWORK="${SWITCHYARD_DOCKER_NETWORK:-}"
@@ -91,10 +91,10 @@ Main options:
--server-url URL Host URL used for health and stats checks.
--harbor-server-url URL Server URL as seen from Harbor task containers.
Defaults to the Dockerized Switchyard service.
- --routing-profiles PATH Route bundle YAML for switchyard serve. When
+ --server-config PATH Rust switchyard-server TOML configuration. When
omitted, Harbor connects directly upstream.
--model MODEL Model Harbor should request. With
- --routing-profiles this is a Switchyard route
+ --server-config this is a Switchyard route
key; without it, this is the upstream model.
Defaults --harbor-model to nvidia/MODEL for
opencode, openai/MODEL for other OpenAI-style
@@ -248,12 +248,17 @@ merge_claude_closed_book_tools() {
}
harbor_extra_has_agent_version() {
+ harbor_extra_has_agent_kwarg version
+}
+
+harbor_extra_has_agent_kwarg() {
+ local key="$1"
local i=0 token next
while [[ "${i}" -lt "${#HARBOR_EXTRA[@]}" ]]; do
token="${HARBOR_EXTRA[$i]}"
if [[ "${token}" == "--ak" || "${token}" == "--agent-kwarg" ]]; then
next="${HARBOR_EXTRA[$((i + 1))]:-}"
- [[ "${next}" == version=* ]] && return 0
+ [[ "${next}" == "${key}="* ]] && return 0
i=$((i + 2))
continue
fi
@@ -293,7 +298,10 @@ while [[ $# -gt 0 ]]; do
--port) PORT="$2"; shift 2 ;;
--server-url) SERVER_URL="$2"; shift 2 ;;
--harbor-server-url) HARBOR_SERVER_URL="$2"; shift 2 ;;
- --routing-profiles) ROUTING_PROFILES="$2"; shift 2 ;;
+ --server-config) SERVER_CONFIG="$2"; shift 2 ;;
+ --routing-profiles)
+ die "--routing-profiles is no longer supported by the benchmark runner; use --server-config with a Rust server TOML file"
+ ;;
--model) MODEL="$2"; shift 2 ;;
--route-model) ROUTE_MODEL="$2"; shift 2 ;;
--agent) AGENT="$2"; shift 2 ;;
@@ -314,7 +322,7 @@ while [[ $# -gt 0 ]]; do
--harbor-extra) HARBOR_EXTRA+=("$2"); shift 2 ;;
--server-extra) SERVER_EXTRA+=("$2"); shift 2 ;;
--strong-model|--weak-model|--classifier-model|--profile|--strong-probability)
- die "$1 belongs to the removed legacy benchmark launchers; encode server routing in --routing-profiles instead"
+ die "$1 belongs to the removed legacy benchmark launchers; encode server routing in --server-config instead"
;;
--dry-run) DRY_RUN=1; shift ;;
--foreground) FOREGROUND=1; shift ;;
@@ -350,9 +358,9 @@ resolve_harbor() {
resolve_harbor
-if [[ -n "${ROUTING_PROFILES}" ]]; then
- [[ -f "${ROUTING_PROFILES}" ]] || die "--routing-profiles not found: ${ROUTING_PROFILES}"
- ROUTING_PROFILES="$(cd "$(dirname "${ROUTING_PROFILES}")" && pwd)/$(basename "${ROUTING_PROFILES}")"
+if [[ -n "${SERVER_CONFIG}" ]]; then
+ [[ -f "${SERVER_CONFIG}" ]] || die "--server-config not found: ${SERVER_CONFIG}"
+ SERVER_CONFIG="$(cd "$(dirname "${SERVER_CONFIG}")" && pwd)/$(basename "${SERVER_CONFIG}")"
fi
UPSTREAM_BASE_URL="${UPSTREAM_BASE_URL:-https://openrouter.ai/api/v1}"
if [[ -z "${UPSTREAM_API_KEY_ENV}" ]]; then
@@ -364,7 +372,7 @@ if [[ -z "${UPSTREAM_API_KEY_ENV}" ]]; then
fi
[[ "${UPSTREAM_API_KEY_ENV}" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || die "--upstream-api-key-env must be a valid environment variable name"
-if [[ -n "${ROUTING_PROFILES}" ]]; then
+if [[ -n "${SERVER_CONFIG}" ]]; then
if [[ -n "${MODEL}" && -n "${ROUTE_MODEL}" && "${MODEL}" != "${ROUTE_MODEL}" ]]; then
die "--model and --route-model must name the same model when both are provided"
fi
@@ -374,18 +382,18 @@ if [[ -n "${ROUTING_PROFILES}" ]]; then
SWITCHYARD_ENABLED=1
SERVER_PRESET="serve"
[[ -z "${MODE}" ]] && MODE="serve"
- [[ -n "${MODEL}" ]] || die "--model is required when using --routing-profiles"
+ [[ -n "${MODEL}" ]] || die "--model is required when using --server-config"
ROUTE_MODEL="${MODEL}"
else
SWITCHYARD_ENABLED=0
SERVER_PRESET="direct"
[[ -z "${MODE}" ]] && MODE="direct"
- [[ -z "${ROUTE_MODEL}" ]] || die "--route-model requires --routing-profiles; use --model for direct upstream"
+ [[ -z "${ROUTE_MODEL}" ]] || die "--route-model requires --server-config; use --model for direct upstream"
[[ -n "${MODEL}" ]] || die "--model is required when running direct upstream"
ROUTE_MODEL=""
- [[ -z "${SERVER_URL}" ]] || die "--server-url requires --routing-profiles"
- [[ -z "${HARBOR_SERVER_URL}" ]] || die "--harbor-server-url requires --routing-profiles"
- [[ "${#SERVER_EXTRA[@]}" -eq 0 ]] || die "--server-extra requires --routing-profiles"
+ [[ -z "${SERVER_URL}" ]] || die "--server-url requires --server-config"
+ [[ -z "${HARBOR_SERVER_URL}" ]] || die "--harbor-server-url requires --server-config"
+ [[ "${#SERVER_EXTRA[@]}" -eq 0 ]] || die "--server-extra requires --server-config"
[[ -n "${!UPSTREAM_API_KEY_ENV:-}" ]] || die "direct upstream requires \$${UPSTREAM_API_KEY_ENV} to be set"
fi
if [[ "${HARBOR_MODEL_SET}" -eq 0 ]]; then
@@ -412,17 +420,15 @@ check_harbor_available() {
}
if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then
- SERVER_CMD=(uv run --no-sync switchyard
- --routing-profiles "${ROUTING_PROFILES}"
- serve
+ SERVER_CMD=(switchyard-server
+ --config "${SERVER_CONFIG}"
--host 0.0.0.0
--port "${PORT}")
- SERVER_DOCKER_CMD=(--routing-profiles "${ROUTING_PROFILES}"
- serve
+ SERVER_DOCKER_CMD=(--config "${SERVER_CONFIG}"
--host 0.0.0.0
--port "${PORT}")
SERVER_CONFIG_JSON="$(json_object_from_pairs \
- mode "${MODE}" routing_profiles "${ROUTING_PROFILES}" route_model "${ROUTE_MODEL}")"
+ mode "${MODE}" server_config "${SERVER_CONFIG}" route_model "${ROUTE_MODEL}")"
if [[ "${#SERVER_EXTRA[@]}" -gt 0 ]]; then
SERVER_CMD+=("${SERVER_EXTRA[@]}")
SERVER_DOCKER_CMD+=("${SERVER_EXTRA[@]}")
@@ -468,7 +474,8 @@ if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then
SERVER_ROOT_URL="${SERVER_CONTROL_URL}"
fi
SERVER_HEALTH_URL="${SERVER_ROOT_URL}/health"
- SERVER_STATS_URL="${SERVER_ROOT_URL}/v1/routing/stats"
+ SERVER_METRICS_URL="${SERVER_ROOT_URL}/metrics"
+ SERVER_STATS_URL="${SERVER_ROOT_URL}/v1/stats"
if [[ -z "${HARBOR_SERVER_URL}" ]]; then
HARBOR_SERVER_URL="http://${SWITCHYARD_DOCKER_SERVICE_NAME}:${PORT}"
@@ -484,6 +491,7 @@ if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then
else
SERVER_ROOT_URL=""
SERVER_HEALTH_URL=""
+ SERVER_METRICS_URL=""
SERVER_STATS_URL=""
HARBOR_BASE_URL="${UPSTREAM_BASE_URL}"
if [[ "${HARBOR_BASE_URL}" == */v1 ]]; then
@@ -502,13 +510,8 @@ LOG_PATH="${RUN_DIR}/${JOB_NAME}.log"
SERVER_LOG="${RUN_DIR}/server.log"
HARBOR_LOG="${RUN_DIR}/harbor.log"
HARBOR_RESULT_JSON="${RUN_DIR}/harbor_result.json"
+SERVER_METRICS_PROM="${RUN_DIR}/server_metrics_final.prom"
ROUTING_STATS_JSON="${RUN_DIR}/routing_stats_final.json"
-ROUTING_LOG_JSONL="${RUN_DIR}/routing_requests.jsonl"
-ROUTING_STATS_BY_TASK_JSON="${RUN_DIR}/routing_stats_by_task.json"
-if [[ "${SWITCHYARD_ENABLED}" -eq 1 ]]; then
- SERVER_CMD+=(--routing-log-file "${ROUTING_LOG_JSONL}")
- SERVER_DOCKER_CMD+=(--routing-log-file "${ROUTING_LOG_JSONL}")
-fi
DOCKER_RUN_ID="$(printf '%s-%s' "${TS##*_}" "$$" | tr -c '[:alnum:]_.-' '-')"
SWITCHYARD_DOCKER_NETWORK="${SWITCHYARD_DOCKER_NETWORK:-switchyard-${DOCKER_RUN_ID}}"
SWITCHYARD_DOCKER_CONTAINER="${SWITCHYARD_DOCKER_CONTAINER:-switchyard-${DOCKER_RUN_ID}}"
@@ -528,17 +531,9 @@ if [[ "${BOOK_MODE}" == "closed" || "${BOOK_MODE}" == "open" ]]; then
docker_network "${SWITCHYARD_DOCKER_NETWORK}")"
fi
fi
-ROUTING_PROFILES_DIR=""
-if [[ -n "${ROUTING_PROFILES}" ]]; then
- ROUTING_PROFILES_DIR="$(cd "$(dirname "${ROUTING_PROFILES}")" && pwd)"
-fi
-CLASSIFIER_PROMPTS_JSON="{}"
-if [[ -n "${ROUTING_PROFILES}" ]]; then
- CLASSIFIER_PROMPTS_JSON="$(
- uv run --no-sync python "${SCRIPT_DIR}/resolve_classifier_prompts.py" "${ROUTING_PROFILES}" \
- 2>/dev/null || echo '{}'
- )"
- [[ -n "${CLASSIFIER_PROMPTS_JSON}" ]] || CLASSIFIER_PROMPTS_JSON="{}"
+SERVER_CONFIG_DIR=""
+if [[ -n "${SERVER_CONFIG}" ]]; then
+ SERVER_CONFIG_DIR="$(cd "$(dirname "${SERVER_CONFIG}")" && pwd)"
fi
WRAPPER="${RUN_DIR}/run-background.sh"
HARBOR_JOB_DIR="${RUN_DIR}/jobs/${JOB_NAME}"
@@ -579,6 +574,10 @@ add_agent_version_kwarg
if [[ -n "${REASONING_EFFORT}" ]]; then
HARBOR_CMD+=(--ak "reasoning_effort=${REASONING_EFFORT}")
+ # Claude code does not support thinking=true for now, so we use adaptive instead
+ if [[ "${AGENT}" == "claude-code" ]] && ! harbor_extra_has_agent_kwarg thinking; then
+ HARBOR_CMD+=(--ak "thinking=adaptive")
+ fi
fi
if [[ -n "${N_TASKS}" ]]; then
@@ -701,8 +700,8 @@ print_resolved() {
if [[ "${BOOK_MODE}" == "closed" ]]; then
echo " verifier_net: authenticated proxy egress"
fi
- if [[ -n "${ROUTING_PROFILES}" ]]; then
- echo " route_profile: ${ROUTING_PROFILES}"
+ if [[ -n "${SERVER_CONFIG}" ]]; then
+ echo " server_config: ${SERVER_CONFIG}"
echo " route_model: ${ROUTE_MODEL}"
fi
printf ' harbor_cmd: '
@@ -764,7 +763,6 @@ MANIFEST_CMD=(python3 "${MANIFEST_HELPER}" write
--server-port "${PORT}"
--server-argv-json "$(json_array "${SERVER_CMD[@]}")"
--server-config-json "${SERVER_CONFIG_JSON}"
- --classifier-prompts-json "${CLASSIFIER_PROMPTS_JSON}"
--harbor-server-url "${HARBOR_SERVER_ROOT_URL}"
--harbor-base-url "${HARBOR_BASE_URL}"
--upstream-base-url "$([[ "${SWITCHYARD_ENABLED}" -eq 0 ]] && echo "${UPSTREAM_BASE_URL}")"
@@ -790,6 +788,8 @@ MANIFEST_CMD=(python3 "${MANIFEST_HELPER}" write
--run-dir "${RUN_DIR}"
--log-path "${LOG_PATH}"
--harbor-result-json "${HARBOR_RESULT_JSON}"
+ --server-metrics-prom "${SERVER_METRICS_PROM}"
+ --server-metrics-status "$([[ "${SWITCHYARD_ENABLED}" -eq 1 ]] && echo predicted || echo not-requested)"
--routing-stats-json "${ROUTING_STATS_JSON}"
--routing-stats-status "$([[ "${SWITCHYARD_ENABLED}" -eq 1 ]] && echo predicted || echo not-requested)")
if [[ -n "${HARBOR_PATH}" ]]; then
@@ -798,8 +798,8 @@ fi
if [[ -n "${TASK_LIST_FILE}" ]]; then
MANIFEST_CMD+=(--task-list-file "${TASK_LIST_FILE}")
fi
-if [[ -n "${ROUTING_PROFILES}" ]]; then
- MANIFEST_CMD+=(--routing-profiles "${ROUTING_PROFILES}")
+if [[ -n "${SERVER_CONFIG}" ]]; then
+ MANIFEST_CMD+=(--server-config "${SERVER_CONFIG}")
fi
if [[ -n "${CODEX_MODEL_CATALOG_HOST}" ]]; then
MANIFEST_CMD+=(--codex-model-catalog "${CODEX_MODEL_CATALOG_HOST}")
@@ -820,6 +820,7 @@ UPSTREAM_API_KEY_ENV=$(q "${UPSTREAM_API_KEY_ENV}")
SERVER_ROOT_URL=$(q "${SERVER_ROOT_URL}")
HARBOR_SERVER_ROOT_URL=$(q "${HARBOR_SERVER_ROOT_URL}")
SERVER_HEALTH_URL=$(q "${SERVER_HEALTH_URL}")
+SERVER_METRICS_URL=$(q "${SERVER_METRICS_URL}")
SERVER_STATS_URL=$(q "${SERVER_STATS_URL}")
HARBOR_BASE_URL=$(q "${HARBOR_BASE_URL}")
MANIFEST_PATH=$(q "${MANIFEST_PATH}")
@@ -827,9 +828,8 @@ RUN_DIR=$(q "${RUN_DIR}")
REPO_ROOT=$(q "${REPO_ROOT}")
SERVER_LOG=$(q "${SERVER_LOG}")
HARBOR_LOG=$(q "${HARBOR_LOG}")
+SERVER_METRICS_PROM=$(q "${SERVER_METRICS_PROM}")
ROUTING_STATS_JSON=$(q "${ROUTING_STATS_JSON}")
-ROUTING_LOG_JSONL=$(q "${ROUTING_LOG_JSONL}")
-ROUTING_STATS_BY_TASK_JSON=$(q "${ROUTING_STATS_BY_TASK_JSON}")
HARBOR_JOB_DIR=$(q "${HARBOR_JOB_DIR}")
SKIP_HEALTH_CHECK=$(q "${SKIP_HEALTH_CHECK}")
BOOK_MODE=$(q "${BOOK_MODE}")
@@ -839,7 +839,7 @@ SWITCHYARD_DOCKER_BUILD=$(q "${SWITCHYARD_DOCKER_BUILD}")
SWITCHYARD_DOCKER_NETWORK=$(q "${SWITCHYARD_DOCKER_NETWORK}")
SWITCHYARD_DOCKER_CONTAINER=$(q "${SWITCHYARD_DOCKER_CONTAINER}")
SWITCHYARD_DOCKER_SERVICE_NAME=$(q "${SWITCHYARD_DOCKER_SERVICE_NAME}")
-ROUTING_PROFILES_DIR=$(q "${ROUTING_PROFILES_DIR}")
+SERVER_CONFIG_DIR=$(q "${SERVER_CONFIG_DIR}")
SERVER_CMD=($(shell_join "${SERVER_CMD[@]}"))
SERVER_DOCKER_CMD=($(shell_join "${SERVER_DOCKER_CMD[@]}"))
@@ -869,7 +869,7 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; then
echo "Starting Dockerized Switchyard service"
echo " image: \${SWITCHYARD_DOCKER_IMAGE}"
echo " network: \${SWITCHYARD_DOCKER_NETWORK}"
- echo " command: switchyard \${SERVER_DOCKER_CMD[*]}"
+ echo " command: switchyard-server \${SERVER_DOCKER_CMD[*]}"
if [[ "\${SWITCHYARD_DOCKER_BUILD}" != "0" ]]; then
docker build -f "\${SWITCHYARD_DOCKERFILE}" -t "\${SWITCHYARD_DOCKER_IMAGE}" .
fi
@@ -887,8 +887,8 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; then
-v "\${REPO_ROOT}:\${REPO_ROOT}:ro"
-v "\${RUN_DIR}:\${RUN_DIR}"
)
- if [[ -n "\${ROUTING_PROFILES_DIR}" && "\${ROUTING_PROFILES_DIR}" != "\${REPO_ROOT}"* ]]; then
- DOCKER_RUN_ARGS+=(-v "\${ROUTING_PROFILES_DIR}:\${ROUTING_PROFILES_DIR}:ro")
+ if [[ -n "\${SERVER_CONFIG_DIR}" && "\${SERVER_CONFIG_DIR}" != "\${REPO_ROOT}"* ]]; then
+ DOCKER_RUN_ARGS+=(-v "\${SERVER_CONFIG_DIR}:\${SERVER_CONFIG_DIR}:ro")
fi
if [[ -n "\${UPSTREAM_API_KEY_ENV}" ]]; then
DOCKER_RUN_ARGS+=(--env "\${UPSTREAM_API_KEY_ENV}")
@@ -898,6 +898,7 @@ if [[ "\${SERVER_ENABLED}" == "1" ]]; then
--env PYTHONHASHSEED=0 \\
--env LC_ALL=C.UTF-8 \\
--env OPENAI_API_KEY \\
+ --env NVIDIA_API_KEY \\
--env OPENROUTER_API_KEY \\
--env UPSTREAM_API_KEY \\
--env ANTHROPIC_API_KEY \\
@@ -943,8 +944,7 @@ if [[ "\${SERVER_ENABLED}" == "1" && "\${SKIP_HEALTH_CHECK}" != "1" ]]; then
python3 benchmark/run_manifest.py finalize \\
--manifest "\${MANIFEST_PATH}" \\
--harbor-rc 124 \\
- --harbor-job-dir "\${HARBOR_JOB_DIR}" \\
- --routing-stats "\${ROUTING_STATS_JSON}" || true
+ --harbor-job-dir "\${HARBOR_JOB_DIR}" || true
exit 124
fi
fi
@@ -961,8 +961,7 @@ if [[ "\${SERVER_ENABLED}" == "1" && "\${SWITCHYARD_CLOSED_BOOK_PREFLIGHT:-1}" !
python3 benchmark/run_manifest.py finalize \\
--manifest "\${MANIFEST_PATH}" \\
--harbor-rc 125 \\
- --harbor-job-dir "\${HARBOR_JOB_DIR}" \\
- --routing-stats "\${ROUTING_STATS_JSON}" || true
+ --harbor-job-dir "\${HARBOR_JOB_DIR}" || true
exit 125
fi
fi
@@ -1037,10 +1036,18 @@ FINALIZE_CMD=(python3 benchmark/run_manifest.py finalize \\
--harbor-rc "\${HARBOR_RC}" \\
--harbor-job-dir "\${HARBOR_JOB_DIR}")
if [[ "\${SERVER_ENABLED}" == "1" ]]; then
- curl -fsS "\${SERVER_STATS_URL}" -o "\${ROUTING_STATS_JSON}" >/dev/null 2>&1 || true
- FINALIZE_CMD+=(--routing-stats "\${ROUTING_STATS_JSON}"
- --routing-log "\${ROUTING_LOG_JSONL}"
- --routing-stats-by-task "\${ROUTING_STATS_BY_TASK_JSON}")
+ if curl -fsS "\${SERVER_METRICS_URL}" -o "\${SERVER_METRICS_PROM}" >/dev/null 2>&1; then
+ echo "Captured aggregate Rust server metrics"
+ else
+ echo "WARNING: Rust server metrics could not be collected from \${SERVER_METRICS_URL}" >&2
+ fi
+ FINALIZE_CMD+=(--server-metrics "\${SERVER_METRICS_PROM}")
+ if curl -fsS "\${SERVER_STATS_URL}" -o "\${ROUTING_STATS_JSON}" >/dev/null 2>&1; then
+ echo "Captured aggregate Rust server routing stats"
+ else
+ echo "WARNING: Rust server stats could not be collected from \${SERVER_STATS_URL}" >&2
+ fi
+ FINALIZE_CMD+=(--routing-stats "\${ROUTING_STATS_JSON}")
fi
"\${FINALIZE_CMD[@]}" || true
diff --git a/benchmark/run_cost.py b/benchmark/run_cost.py
deleted file mode 100644
index 3b7f3250f..000000000
--- a/benchmark/run_cost.py
+++ /dev/null
@@ -1,105 +0,0 @@
-#!/usr/bin/env python3
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-"""Per-model cost breakdown for a benchmark run's routing log.
-
-Aggregates ``routing_requests.jsonl`` by served model and prices each bucket with
-:func:`switchyard.lib.cost_estimator.estimate_model_cost`, which splits input into
-base / cache-read / cache-write tiers. That split matters: agentic runs are
-cache-dominated (90%+ of prompt tokens are cache reads is normal), so a flat
-input rate overstates cost several-fold.
-
-On a run that routes sub-agent work to a different tier, the served-model column
-doubles as the parent/child attribution key — the routing log records no
-``agent_id``, so which model answered is the only role signal available.
-
-Models absent from the price table contribute $0 and are listed separately, so an
-unpriced model reads as a gap rather than as a free one.
-
-Usage:
- uv run python benchmark/run_cost.py --run benchmark/tb_runs/<run-name>
-"""
-
-from __future__ import annotations
-
-import argparse
-import collections
-import json
-from pathlib import Path
-
-from switchyard.lib.cost_estimator import MODEL_PRICING, estimate_model_cost
-
-_TOKEN_FIELDS = (
- "prompt_tokens",
- "completion_tokens",
- "cached_tokens",
- "cache_creation_tokens",
-)
-
-
-def _aggregate(routing_log: Path) -> dict[str, collections.Counter[str]]:
- """Sum request counts and token fields per served model."""
- totals: dict[str, collections.Counter[str]] = collections.defaultdict(collections.Counter)
- for line in routing_log.read_text().splitlines():
- if not line.strip():
- continue
- record = json.loads(line)
- bucket = totals[record.get("model") or "<unknown>"]
- bucket["reqs"] += 1
- for field in _TOKEN_FIELDS:
- bucket[field] += record.get(field) or 0
- return totals
-
-
-def _mean_agg_score(run: Path) -> tuple[float, int] | None:
- """Mean rubric ``agg_score`` across the run's scored tasks, if any."""
- scores = [
- json.loads(path.read_text())["agg_score"]
- for path in run.glob("jobs/*/task-*/verifier/evaluation_results.json")
- ]
- return (sum(scores) / len(scores), len(scores)) if scores else None
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(description=__doc__)
- parser.add_argument("--run", required=True, type=Path, help="Run directory under tb_runs/")
- args = parser.parse_args()
-
- totals = _aggregate(args.run / "routing_requests.jsonl")
- if not totals:
- print("no routing records found")
- return
-
- unpriced = [model for model in totals if model not in MODEL_PRICING]
- total_cost = 0.0
- header = f"{'model':46s} {'reqs':>5s} {'in':>11s} {'out':>9s} {'cached':>11s} {'USD':>9s}"
- print(header)
- for model, bucket in sorted(totals.items(), key=lambda item: -item[1]["reqs"]):
- cost = estimate_model_cost(
- model,
- bucket["prompt_tokens"],
- bucket["completion_tokens"],
- bucket["cached_tokens"],
- bucket["cache_creation_tokens"],
- )["total_cost"]
- total_cost += cost
- print(
- f"{model:46s} {bucket['reqs']:5d} {bucket['prompt_tokens']:11,} "
- f"{bucket['completion_tokens']:9,} {bucket['cached_tokens']:11,} {cost:9.3f}"
- )
-
- requests = sum(bucket["reqs"] for bucket in totals.values())
- print(f"\nTOTAL: ${total_cost:.2f} over {requests} requests")
- if unpriced:
- # Priced at zero by estimate_model_cost — surface it so the total is not
- # mistaken for complete.
- print(f"NOTE: no price entry for {', '.join(sorted(unpriced))} — excluded from the total")
-
- scored = _mean_agg_score(args.run)
- if scored:
- mean, count = scored
- print(f"mean agg_score: {mean:.3f} over {count} tasks → ${total_cost / count:.2f}/task")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/run_manifest.py b/benchmark/run_manifest.py
index 92879b5bf..30ed596e1 100644
--- a/benchmark/run_manifest.py
+++ b/benchmark/run_manifest.py
@@ -117,14 +117,14 @@ def dataset_fingerprint(
return f"sha256:{hashlib.sha256(encoded).hexdigest()}"
-def snapshot_routing_profiles(source: Path | None, run_dir: Path) -> Path | None:
- """Copy the route bundle used for the run into the run directory."""
+def snapshot_server_config(source: Path | None, run_dir: Path) -> Path | None:
+ """Copy the Rust server configuration used for the run."""
if source is None:
return None
if not source.is_file():
raise FileNotFoundError(source)
- dest = run_dir.resolve() / "routing_profiles" / source.name
+ dest = run_dir.resolve() / "server_config" / source.name
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source, dest)
return dest
@@ -223,116 +223,15 @@ def _copy_if_present(source: Path | None, dest: Path | None) -> str:
return "present"
-_TOKEN_FIELDS = (
- "prompt_tokens",
- "cached_tokens",
- "cache_creation_tokens",
- "completion_tokens",
- "reasoning_tokens",
- "total_tokens",
-)
-
-
-def _new_section() -> dict[str, Any]:
- return {"calls": 0, "totals": dict.fromkeys(_TOKEN_FIELDS, 0), "_buckets": {}}
-
-
-def _accumulate(section: dict[str, Any], record: dict[str, Any]) -> None:
- section["calls"] += 1
- model = record.get("model") or "unknown"
- tier = record.get("tier") or ""
- bucket = section["_buckets"].setdefault(
- (model, tier),
- {"model": model, "tier": tier, "calls": 0, **dict.fromkeys(_TOKEN_FIELDS, 0)},
- )
- bucket["calls"] += 1
- for field in _TOKEN_FIELDS:
- value = record.get(field)
- value = value if isinstance(value, int) else 0
- bucket[field] += value
- section["totals"][field] += value
-
-
-def _finalize_section(section: dict[str, Any]) -> None:
- buckets = section.pop("_buckets")
- section["models"] = [buckets[key] for key in sorted(buckets)]
-
-
-def _kept_sessions(records: list[dict[str, Any]]) -> set[str]:
- """The kept attempt of each trial is its latest-timestamped session.
-
- Harbor retries sequentially and keeps only the final attempt, so within a
- trial the session with the newest timestamp is the one that survived.
- Records without a ``trial_id`` are all treated as kept (no netting).
- """
- latest: dict[str, tuple[str, str]] = {}
- for record in records:
- trial = record.get("trial_id")
- session = record.get("session_id")
- if not trial or not session:
- continue
- ts = record.get("ts") or ""
- if trial not in latest or ts > latest[trial][0]:
- latest[trial] = (ts, session)
- return {session for _, session in latest.values()}
-
-
-def summarize_routing_log(log_path: Path) -> dict[str, Any]:
- """Roll the routing log into per-task ``final`` and ``retries`` sections.
-
- Requests are grouped by trial; the latest attempt in each trial is ``final``
- and earlier attempts are ``retries`` (see :func:`_kept_sessions`).
- """
- by_task: dict[str, list[dict[str, Any]]] = {}
- total_requests = 0
- for raw in log_path.read_text().splitlines():
- raw = raw.strip()
- if not raw:
- continue
- try:
- record = json.loads(raw)
- except json.JSONDecodeError:
- continue
- if not isinstance(record, dict):
- continue
- total_requests += 1
- by_task.setdefault(record.get("task") or "unattributed", []).append(record)
-
- tasks: dict[str, dict[str, Any]] = {}
- for task, records in by_task.items():
- kept = _kept_sessions(records)
- entry = {
- "requests": len(records),
- "n_retries": 0,
- "final": _new_section(),
- "retries": _new_section(),
- }
- retry_sessions: set[str] = set()
- for record in records:
- session = record.get("session_id")
- # A record is a retry only when it has a trial_id and its session was
- # not the kept one; untagged records default to final.
- is_retry = bool(record.get("trial_id")) and session not in kept
- _accumulate(entry["retries" if is_retry else "final"], record)
- if is_retry and session:
- retry_sessions.add(session)
- entry["n_retries"] = len(retry_sessions)
- _finalize_section(entry["final"])
- _finalize_section(entry["retries"])
- tasks[task] = entry
- return {"total_requests": total_requests, "tasks": tasks}
-
-
def finalize_manifest(
path: Path,
*,
harbor_rc: int | None,
harbor_job_dir: Path | None = None,
+ server_metrics: Path | None = None,
routing_stats: Path | None = None,
- routing_log: Path | None = None,
- routing_stats_by_task: Path | None = None,
) -> int:
- """Copy run artifacts into place, roll up the routing log, and record outcomes.
+ """Copy run artifacts into place and record outcomes.
Returns 0 on success, 1 when the manifest is missing.
"""
@@ -349,6 +248,14 @@ def finalize_manifest(
if harbor_job_dir is not None:
outcomes["harbor_job_dir"] = str(harbor_job_dir.resolve())
+ metrics_dest = (
+ Path(outcomes["server_metrics_prom"]) if outcomes.get("server_metrics_prom") else None
+ )
+ if outcomes.get("server_metrics_prom_status") != "not-requested":
+ if server_metrics is None and metrics_dest is not None:
+ server_metrics = metrics_dest
+ outcomes["server_metrics_prom_status"] = _copy_if_present(server_metrics, metrics_dest)
+
stats_dest = (
Path(outcomes["routing_stats_json"]) if outcomes.get("routing_stats_json") else None
)
@@ -370,20 +277,6 @@ def finalize_manifest(
break
closed_book["proxy_strip_log_status"] = _copy_if_present(strip_source, strip_dest)
- if routing_stats_by_task is not None:
- status = "missing"
- if routing_log is not None and routing_log.is_file():
- try:
- summary = summarize_routing_log(routing_log)
- routing_stats_by_task.write_text(
- json.dumps(summary, indent=2, sort_keys=False) + "\n"
- )
- status = "present"
- except OSError:
- status = "missing"
- outcomes["routing_stats_by_task_json"] = str(routing_stats_by_task.resolve())
- outcomes["routing_stats_by_task_json_status"] = status
-
outcomes["harbor_rc"] = harbor_rc
outcomes["completed_at"] = _iso_timestamp()
path.write_text(json.dumps(manifest, indent=2, sort_keys=False) + "\n")
@@ -422,12 +315,11 @@ def _cli_main(argv: list[str] | None = None) -> int:
write.add_argument("--server-port", type=int, default=0)
write.add_argument("--server-argv-json", default="[]")
write.add_argument("--server-config-json", default="{}")
- write.add_argument("--classifier-prompts-json", default="{}")
write.add_argument("--harbor-server-url", default="")
write.add_argument("--harbor-base-url", default="")
write.add_argument("--upstream-base-url", default="")
write.add_argument("--upstream-api-key-env", default="")
- write.add_argument("--routing-profiles", type=Path, default=None)
+ write.add_argument("--server-config", type=Path, default=None)
write.add_argument("--route-model", default="")
write.add_argument("--harbor-command-json", default="[]")
write.add_argument("--dataset-label", default="")
@@ -452,6 +344,8 @@ def _cli_main(argv: list[str] | None = None) -> int:
write.add_argument("--run-dir", type=Path, required=True)
write.add_argument("--log-path", type=Path, required=True)
write.add_argument("--harbor-result-json", type=Path, required=True)
+ write.add_argument("--server-metrics-prom", type=Path, default=None)
+ write.add_argument("--server-metrics-status", default="not-requested")
write.add_argument("--routing-stats-json", type=Path, required=True)
write.add_argument("--routing-stats-status", default="predicted")
write.add_argument("--extra", action="append", default=[])
@@ -460,9 +354,8 @@ def _cli_main(argv: list[str] | None = None) -> int:
finalize.add_argument("--manifest", type=Path, required=True)
finalize.add_argument("--harbor-rc", type=int, default=None)
finalize.add_argument("--harbor-job-dir", type=Path, default=None)
+ finalize.add_argument("--server-metrics", type=Path, default=None)
finalize.add_argument("--routing-stats", type=Path, default=None)
- finalize.add_argument("--routing-log", type=Path, default=None)
- finalize.add_argument("--routing-stats-by-task", type=Path, default=None)
ns = parser.parse_args(argv)
if ns.command == "finalize":
@@ -470,27 +363,22 @@ def _cli_main(argv: list[str] | None = None) -> int:
ns.manifest,
harbor_rc=ns.harbor_rc,
harbor_job_dir=ns.harbor_job_dir,
+ server_metrics=ns.server_metrics,
routing_stats=ns.routing_stats,
- routing_log=ns.routing_log,
- routing_stats_by_task=ns.routing_stats_by_task,
)
if ns.command != "write":
parser.print_help()
return 2
task_list = ns.task_list_file.resolve() if ns.task_list_file else None
- routing_profiles = ns.routing_profiles.resolve() if ns.routing_profiles else None
+ server_config = ns.server_config.resolve() if ns.server_config else None
harbor_path = ns.harbor_path.resolve() if ns.harbor_path else None
codex_model_catalog = (
ns.codex_model_catalog.resolve() if ns.codex_model_catalog else None
)
run_dir = ns.run_dir.resolve()
- classifier_prompts = _json_arg(ns.classifier_prompts_json, {})
- if not isinstance(classifier_prompts, dict):
- print("ERROR: --classifier-prompts-json must decode to a JSON object")
- return 2
try:
- routing_profiles_snapshot = snapshot_routing_profiles(routing_profiles, run_dir)
+ server_config_snapshot = snapshot_server_config(server_config, run_dir)
dataset_manifest_snapshot = snapshot_dataset_manifest(harbor_path, run_dir)
except OSError as exc:
print(f"ERROR: failed to snapshot run inputs: {exc}")
@@ -550,20 +438,17 @@ def _cli_main(argv: list[str] | None = None) -> int:
"port": ns.server_port or None,
"argv": _json_arg(ns.server_argv_json, []),
"config": _json_arg(ns.server_config_json, {}),
- "classifier_prompts": classifier_prompts,
"harbor_server_url": _opt(ns.harbor_server_url),
"harbor_base_url": _opt(ns.harbor_base_url),
"upstream_base_url": _opt(ns.upstream_base_url),
"upstream_api_key_env": _opt(ns.upstream_api_key_env),
- "routing_profiles": str(routing_profiles) if routing_profiles else None,
- "routing_profiles_digest": (
- path_digest(routing_profiles) if routing_profiles else None
+ "server_config": str(server_config) if server_config else None,
+ "server_config_digest": path_digest(server_config) if server_config else None,
+ "server_config_snapshot": (
+ str(server_config_snapshot) if server_config_snapshot else None
),
- "routing_profiles_snapshot": (
- str(routing_profiles_snapshot) if routing_profiles_snapshot else None
- ),
- "routing_profiles_snapshot_digest": (
- path_digest(routing_profiles_snapshot) if routing_profiles_snapshot else None
+ "server_config_snapshot_digest": (
+ path_digest(server_config_snapshot) if server_config_snapshot else None
),
"route_model": _opt(ns.route_model),
},
@@ -579,6 +464,10 @@ def _cli_main(argv: list[str] | None = None) -> int:
"log_path": str(ns.log_path.resolve()),
"harbor_result_json": str(ns.harbor_result_json.resolve()),
"harbor_result_json_status": "predicted",
+ "server_metrics_prom": (
+ str(ns.server_metrics_prom.resolve()) if ns.server_metrics_prom else None
+ ),
+ "server_metrics_prom_status": ns.server_metrics_status,
"routing_stats_json": str(ns.routing_stats_json.resolve()),
"routing_stats_json_status": ns.routing_stats_status,
"harbor_rc": None,
diff --git a/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml b/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml
new file mode 100644
index 000000000..0ddfd9823
--- /dev/null
+++ b/benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml
@@ -0,0 +1,35 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Rust task-classifier configuration for Harbor Terminal-Bench Lite.
+# Harbor selects this route with --model switchyard.
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.classifier]
+id = "google/gemini-3.5-flash"
+llm_client = "openrouter"
+
+[targets.strong]
+id = "anthropic/claude-opus-4.7"
+llm_client = "openrouter"
+
+[targets.weak]
+id = "moonshotai/kimi-k2.7-code"
+llm_client = "openrouter"
+
+[routes.switchyard]
+id = "switchyard"
+type = "llm_classifier"
+classifier_target = "classifier"
+strong_target = "strong"
+weak_target = "weak"
+base_threshold = 0.5
+min_confidence = 0.0
+session_affinity = true
+message_hash_fallback = true
diff --git a/benchmark/server-configs/tb-lite-single-gpt-5-5.toml b/benchmark/server-configs/tb-lite-single-gpt-5-5.toml
new file mode 100644
index 000000000..5f1898ae4
--- /dev/null
+++ b/benchmark/server-configs/tb-lite-single-gpt-5-5.toml
@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_responses"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.primary]
+id = "openai/gpt-5.5"
+llm_client = "openrouter"
+
+[routes.primary]
+id = "tb-lite-single-gpt-5-5"
+type = "passthrough"
+target = "primary"
diff --git a/benchmark/server-configs/tb-lite-single-opus-4-7.toml b/benchmark/server-configs/tb-lite-single-opus-4-7.toml
new file mode 100644
index 000000000..ab3532a77
--- /dev/null
+++ b/benchmark/server-configs/tb-lite-single-opus-4-7.toml
@@ -0,0 +1,18 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+schema_version = 1
+
+[llm_clients.openrouter]
+format = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[targets.primary]
+id = "anthropic/claude-opus-4.7"
+llm_client = "openrouter"
+
+[routes.primary]
+id = "tb-lite-single-opus-4-7"
+type = "passthrough"
+target = "primary"
diff --git a/docs/cli_reference.md b/docs/cli_reference.md
index bc4df7274..9950631a5 100644
--- a/docs/cli_reference.md
+++ b/docs/cli_reference.md
@@ -82,7 +82,7 @@ For `type: deterministic` routes, the `classifier:` block also accepts:
| `max_request_chars` | Optional cap on the serialized request summary sent to the classifier before truncation. Defaults to `16000`; minimum `256`. |
| `recent_turn_window` | Number of trailing conversation turns included alongside the stable system/first-user anchors. Defaults to `4`. |
-Benchmark runs started through `benchmark/run-baseline.sh --routing-profiles` record the effective classifier prompt, prompt SHA-256, `max_request_chars`, and `recent_turn_window` in `run_manifest.json` under `server.classifier_prompts`.
+Benchmark runs started through `benchmark/run-baseline.sh --server-config` copy the server TOML into the run directory and record its path and SHA-256 in `run_manifest.json` under `server.server_config_snapshot` and `server.server_config_snapshot_digest`.
Each route name becomes a model ID on `GET /v1/models`. Clients select a route
by setting the request's `model` field to that name.
@@ -115,8 +115,9 @@ The Intake sink posts live model-call captures to nemo-platform
derives queryable token fields from `response.usage` and queryable cost fields
from top-level `cost_usd`, `cost_input_usd`, `cost_output_usd`, and
`cost_details`. Switchyard emits cost fields only when the served model has a
-known pricing entry; `routing_stats_final.json` remains the run-level source
-for aggregate routing/model cost estimates. When a session ID is present,
+known pricing entry. `routing_stats_final.json` carries the run-level aggregate
+`/v1/stats` snapshot (per-model and per-tier calls, errors, tokens, and
+latency); it does not include cost estimates. When a session ID is present,
Switchyard also maps the Intake app/task labels into top-level
`evaluation_context.dataset_*` and `evaluation_context.test_case_id` for span
queries while keeping the original labels under `request.switchyard`.
diff --git a/examples/experimental/litellm/README.md b/examples/experimental/litellm/README.md
index 9ffc90763..23b4196e0 100644
--- a/examples/experimental/litellm/README.md
+++ b/examples/experimental/litellm/README.md
@@ -182,7 +182,7 @@ Then run the three-task smoke benchmark in the foreground:
SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles examples/experimental/litellm/benchmark-route.yaml \
+ --server-config examples/experimental/litellm/benchmark-route.toml \
--task-list-file examples/experimental/litellm/benchmark-tasks.txt \
--model litellm-random \
--agent codex \
@@ -224,7 +224,7 @@ PATH="$CODEX_SHIM_DIR:$PATH" \
SWITCHYARD_DOCKER_NETWORK=switchyard-litellm \
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
- --routing-profiles examples/experimental/litellm/benchmark-route.yaml \
+ --server-config examples/experimental/litellm/benchmark-route.toml \
--task-list-file examples/experimental/litellm/benchmark-tasks.txt \
--model litellm-random \
--agent codex \
diff --git a/examples/experimental/litellm/benchmark-route.toml b/examples/experimental/litellm/benchmark-route.toml
new file mode 100644
index 000000000..1ef063bd6
--- /dev/null
+++ b/examples/experimental/litellm/benchmark-route.toml
@@ -0,0 +1,23 @@
+schema_version = 1
+
+[llm_clients.litellm]
+format = "openai_chat"
+base_url = "http://litellm:4000/v1"
+
+[llm_clients.litellm.extra_headers]
+authorization = "Bearer not-needed"
+
+[targets.strong]
+id = "strong"
+llm_client = "litellm"
+
+[targets.fast]
+id = "fast"
+llm_client = "litellm"
+
+[routes.litellm_random]
+id = "litellm-random"
+type = "random"
+targets = ["strong", "fast"]
+weights = [0.5, 0.5]
+seed = 42
diff --git a/examples/experimental/litellm/benchmark-route.yaml b/examples/experimental/litellm/benchmark-route.yaml
deleted file mode 100644
index f7a9d4ca8..000000000
--- a/examples/experimental/litellm/benchmark-route.yaml
+++ /dev/null
@@ -1,19 +0,0 @@
-defaults:
- api_key: not-needed
- base_url: http://litellm:4000/v1
- format: openai
- timeout_secs: 600.0
-
-routes:
- litellm-random:
- type: random_routing
- strong:
- id: strong
- model: strong
- weak:
- id: fast
- model: fast
- strong_probability: 0.5
- rng_seed: 42
- fallback_target_on_evict: strong
- enable_stats: true
diff --git a/tests/test_resolve_classifier_prompts.py b/tests/test_resolve_classifier_prompts.py
deleted file mode 100644
index f65b72884..000000000
--- a/tests/test_resolve_classifier_prompts.py
+++ /dev/null
@@ -1,181 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-from __future__ import annotations
-
-import hashlib
-import importlib.util
-from pathlib import Path
-from types import ModuleType
-
-import pytest
-import yaml
-
-from switchyard.cli.route_bundle import build_route_bundle_table
-from switchyard.lib.processors.llm_classifier import (
- DEFAULT_MAX_REQUEST_CHARS,
- LLMClassifierRequestProcessor,
-)
-from switchyard.lib.processors.llm_classifier.presets import profile_default_prompt
-
-RESOLVER = Path(__file__).resolve().parents[1] / "benchmark" / "resolve_classifier_prompts.py"
-
-
-def _load_resolver_module() -> ModuleType:
- spec = importlib.util.spec_from_file_location(
- "switchyard_benchmark_resolve_classifier_prompts",
- RESOLVER,
- )
- assert spec is not None
- assert spec.loader is not None
- module = importlib.util.module_from_spec(spec)
- spec.loader.exec_module(module)
- return module
-
-
-def _write(tmp_path: Path, bundle: dict) -> Path:
- path = tmp_path / "routes.yaml"
- path.write_text(yaml.safe_dump(bundle))
- return path
-
-
-def _bundle(prompt: str | None = None) -> dict:
- classifier: dict[str, object] = {
- "model": "c",
- "api_key": "k",
- "base_url": "https://example.invalid/v1",
- "recent_turn_window": 2,
- "max_request_chars": 1024,
- }
- if prompt is not None:
- classifier["prompt"] = prompt
- return {
- "routes": {
- "r/llm": {
- "type": "deterministic",
- "fallback_target_on_evict": "strong",
- "profile": "coding_agent",
- "classifier": classifier,
- "strong": {
- "model": "s",
- "api_key": "k",
- "base_url": "https://example.invalid/v1",
- },
- "weak": {
- "model": "w",
- "api_key": "k",
- "base_url": "https://example.invalid/v1",
- },
- },
- },
- }
-
-
-def test_explicit_prompt_override_is_recorded(tmp_path: Path) -> None:
- module = _load_resolver_module()
- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle("MY PROMPT")))
-
- assert out["r/llm"]["source"] == "override"
- assert out["r/llm"]["classifier_prompt"] == "MY PROMPT"
- assert out["r/llm"]["classifier_prompt_sha256"] == hashlib.sha256(
- b"MY PROMPT"
- ).hexdigest()
- assert out["r/llm"]["profile"] == "coding_agent"
- assert out["r/llm"]["max_request_chars"] == 1024
- assert out["r/llm"]["recent_turn_window"] == 2
-
-
-def test_profile_default_is_resolved(tmp_path: Path) -> None:
- module = _load_resolver_module()
- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle()))
-
- assert out["r/llm"]["source"] == "profile_default"
- assert out["r/llm"]["classifier_prompt"] == profile_default_prompt("coding_agent")
-
-
-def test_default_profile_is_general_when_absent(tmp_path: Path) -> None:
- module = _load_resolver_module()
- bundle = _bundle()
- del bundle["routes"]["r/llm"]["profile"]
-
- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))
-
- assert out["r/llm"]["profile"] == "general"
-
-
-def test_non_deterministic_routes_are_skipped(tmp_path: Path) -> None:
- module = _load_resolver_module()
- bundle = {
- "routes": {
- "r/pass": {"type": "model", "target": "some/model"},
- "r/rand": {"type": "random_routing"},
- },
- }
-
- assert module.resolve_classifier_prompts(_write(tmp_path, bundle)) == {}
-
-
-def test_whitespace_only_prompt_uses_profile_default(tmp_path: Path) -> None:
- module = _load_resolver_module()
- out = module.resolve_classifier_prompts(_write(tmp_path, _bundle(" ")))
-
- assert out["r/llm"]["source"] == "profile_default"
-
-
-def test_context_defaults_are_recorded(tmp_path: Path) -> None:
- module = _load_resolver_module()
- bundle = _bundle()
- classifier = bundle["routes"]["r/llm"]["classifier"]
- del classifier["recent_turn_window"]
- del classifier["max_request_chars"]
-
- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))
-
- assert out["r/llm"]["recent_turn_window"] == 4
- assert out["r/llm"]["max_request_chars"] == DEFAULT_MAX_REQUEST_CHARS
-
-
-def test_subminimum_max_request_chars_uses_default(tmp_path: Path) -> None:
- module = _load_resolver_module()
- bundle = _bundle()
- classifier = bundle["routes"]["r/llm"]["classifier"]
- classifier["max_request_chars"] = 255
-
- out = module.resolve_classifier_prompts(_write(tmp_path, bundle))
-
- assert out["r/llm"]["max_request_chars"] == DEFAULT_MAX_REQUEST_CHARS
-
-
-def test_resolver_matches_server_effective_prompt_override(
- tmp_path: Path,
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- module = _load_resolver_module()
- bundle = _bundle("CUSTOM SERVER PROMPT")
- monkeypatch.setattr("switchyard.cli.route_bundle.fetch_model_ids", lambda *_: [])
-
- table = build_route_bundle_table(bundle)
- switchyard = table.lookup_switchyard("r/llm")
- classifier = next(
- c for c in switchyard.iter_components()
- if isinstance(c, LLMClassifierRequestProcessor)
- )
-
- resolved = module.resolve_classifier_prompts(_write(tmp_path, bundle))["r/llm"]
- assert resolved["classifier_prompt"] == classifier._config.system_prompt
- assert resolved["max_request_chars"] == classifier._config.max_request_chars
- assert resolved["recent_turn_window"] == classifier._config.recent_turn_window
-
-
-def test_prompt_env_vars_match_route_bundle_expansion(
- tmp_path: Path,
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- module = _load_resolver_module()
- monkeypatch.setenv("CUSTOM_CLASSIFIER_PROMPT", "expanded prompt")
-
- out = module.resolve_classifier_prompts(
- _write(tmp_path, _bundle("${CUSTOM_CLASSIFIER_PROMPT}")),
- )
-
- assert out["r/llm"]["classifier_prompt"] == "expanded prompt"
diff --git a/tests/test_run_baseline_script.py b/tests/test_run_baseline_script.py
index 85550f743..86e19e97e 100644
--- a/tests/test_run_baseline_script.py
+++ b/tests/test_run_baseline_script.py
@@ -62,6 +62,50 @@ exit 2
return harbor
+def _write_fake_docker(bin_dir: Path) -> Path:
+ docker = bin_dir / "docker"
+ docker.write_text(
+ "#!/usr/bin/env bash\n"
+ "if [[ -n \"${FAKE_DOCKER_LOG:-}\" ]]; then\n"
+ " printf '%s\\n' \"$*\" >> \"${FAKE_DOCKER_LOG}\"\n"
+ "fi\n"
+ "exit 0\n"
+ )
+ docker.chmod(0o755)
+ return docker
+
+
+def _write_fake_curl(bin_dir: Path) -> Path:
+ curl = bin_dir / "curl"
+ curl.write_text(
+ """
+#!/usr/bin/env bash
+set -euo pipefail
+output=""
+url=""
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ -o) output="$2"; shift 2 ;;
+ *) url="$1"; shift ;;
+ esac
+done
+case "${url}" in
+ */health) body='{"status":"ok"}' ;;
+ */metrics) body='switchyard_total_requests 2' ;;
+ */v1/stats) body='{"total_requests":2,"total_errors":0,"models":{"strong":{"calls":2}}}' ;;
+ *) exit 22 ;;
+esac
+if [[ -n "${output}" ]]; then
+ printf '%s\\n' "${body}" > "${output}"
+else
+ printf '%s\\n' "${body}"
+fi
+""".lstrip()
+ )
+ curl.chmod(0o755)
+ return curl
+
+
def _write_fake_harbor_patch(tmp_path: Path) -> Path:
patch_file = tmp_path / "fake-harbor-agent-patches.diff"
patch_file.write_text(
@@ -106,10 +150,14 @@ def _run_baseline(
*args: str,
env: dict[str, str] | None = None,
include_dataset: bool = True,
+ fake_server: bool = False,
):
fake_bin = tmp_path / "default-bin"
fake_bin.mkdir(exist_ok=True)
_write_fake_harbor(fake_bin)
+ if fake_server:
+ _write_fake_docker(fake_bin)
+ _write_fake_curl(fake_bin)
fake_patch = _write_fake_harbor_patch(tmp_path)
fake_harbor_python = _write_fake_harbor_python(tmp_path)
base_path = env.get("PATH", os.environ["PATH"]) if env else os.environ["PATH"]
@@ -158,21 +206,29 @@ def _option_values(argv: list[str], option: str) -> list[str]:
return [argv[idx + 1] for idx, value in enumerate(argv[:-1]) if value == option]
-def _write_route_profile(path: Path) -> Path:
+def _write_server_config(path: Path) -> Path:
path.write_text(
"""
-routes:
- tb-lite-random-routing:
- type: random_routing
- strong_probability: 0.5
- rng_seed: 444
- fallback_target_on_evict: strong
- strong:
- id: strong
- model: strong-model
- weak:
- id: weak
- model: weak-model
+schema_version = 1
+
+[llm_clients.test]
+format = "openai_chat"
+base_url = "https://example.test/v1"
+
+[targets.strong]
+id = "strong-model"
+llm_client = "test"
+
+[targets.weak]
+id = "weak-model"
+llm_client = "test"
+
+[routes.random]
+id = "tb-lite-random-routing"
+type = "random"
+targets = ["strong", "weak"]
+weights = [1, 1]
+seed = 444
""".lstrip()
)
return path
@@ -222,8 +278,8 @@ def test_direct_mode_defaults_to_openrouter_upstream_without_switchyard(tmp_path
assert "upstream_url: https://openrouter.ai/api/v1" in result.stdout
assert "api_key_env: OPENROUTER_API_KEY" in result.stdout
assert "SERVER_CMD: <direct-upstream>" in result.stdout
- assert "switchyard serve" not in result.stdout
- assert "route_profile:" not in result.stdout
+ assert "switchyard-server" not in result.stdout
+ assert "server_config:" not in result.stdout
def test_direct_mode_uses_generic_upstream_key_when_set(tmp_path: Path) -> None:
@@ -269,7 +325,7 @@ def test_direct_mode_rejects_server_only_options(tmp_path: Path) -> None:
)
assert result.returncode != 0
- assert "--server-extra requires --routing-profiles" in result.stderr
+ assert "--server-extra requires --server-config" in result.stderr
result = _run_baseline(
tmp_path,
@@ -279,14 +335,29 @@ def test_direct_mode_rejects_server_only_options(tmp_path: Path) -> None:
)
assert result.returncode != 0
- assert "--route-model requires --routing-profiles" in result.stderr
+ assert "--route-model requires --server-config" in result.stderr
-def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> None:
+def test_legacy_routing_profiles_are_rejected(tmp_path: Path) -> None:
result = _run_baseline(
tmp_path,
"--routing-profiles",
- str(REPO / "benchmark" / "routing-profiles" / "tb-lite-single-opus-4-7.yaml"),
+ str(tmp_path / "routes.yaml"),
+ "--model",
+ "switchyard",
+ "--dry-run",
+ )
+
+ assert result.returncode != 0
+ assert "--routing-profiles is no longer supported" in result.stderr
+ assert "use --server-config" in result.stderr
+
+
+def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> None:
+ result = _run_baseline(
+ tmp_path,
+ "--server-config",
+ str(REPO / "benchmark" / "server-configs" / "tb-lite-single-opus-4-7.toml"),
"--route-model",
"tb-lite-single-opus-4-7",
"--agent",
@@ -299,6 +370,7 @@ def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> Non
assert _option_value(harbor, "--model") == "tb-lite-single-opus-4-7"
assert "version=2.1.211" in _option_values(harbor, "--ak")
assert "reasoning_effort=high" in _option_values(harbor, "--ak")
+ assert "thinking=adaptive" in _option_values(harbor, "--ak")
assert "reasoning: high" in result.stdout
assert "harbor_server: http://switchyard:4000" in result.stdout
@@ -306,8 +378,8 @@ def test_dry_run_claude_code_opus_defaults_high_reasoning(tmp_path: Path) -> Non
def test_dry_run_codex_gpt_defaults_high_reasoning(tmp_path: Path) -> None:
result = _run_baseline(
tmp_path,
- "--routing-profiles",
- str(REPO / "benchmark" / "routing-profiles" / "tb-lite-single-gpt-5-5.yaml"),
+ "--server-config",
+ str(REPO / "benchmark" / "server-configs" / "tb-lite-single-gpt-5-5.toml"),
"--route-model",
"tb-lite-single-gpt-5-5",
"--agent",
@@ -330,11 +402,11 @@ def test_dry_run_codex_gpt_defaults_high_reasoning(tmp_path: Path) -> None:
def test_dry_run_explicit_empty_reasoning_omits_kwarg(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -348,15 +420,16 @@ def test_dry_run_explicit_empty_reasoning_omits_kwarg(tmp_path: Path) -> None:
assert result.returncode == 0, result.stderr
harbor = _line_argv(result.stdout, "HARBOR_CMD: ")
assert not any(value.startswith("reasoning_effort=") for value in _option_values(harbor, "--ak"))
+ assert not any(value.startswith("thinking=") for value in _option_values(harbor, "--ak"))
assert "reasoning: unset" in result.stdout
def test_dry_run_explicit_reasoning_overrides_default(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -370,14 +443,41 @@ def test_dry_run_explicit_reasoning_overrides_default(tmp_path: Path) -> None:
assert result.returncode == 0, result.stderr
harbor = _line_argv(result.stdout, "HARBOR_CMD: ")
assert "reasoning_effort=medium" in _option_values(harbor, "--ak")
+ assert "thinking=adaptive" in _option_values(harbor, "--ak")
+
+
+def test_dry_run_explicit_claude_thinking_overrides_adaptive_default(tmp_path: Path) -> None:
+ profile = _write_server_config(tmp_path / "routes.toml")
+
+ result = _run_baseline(
+ tmp_path,
+ "--server-config",
+ str(profile),
+ "--route-model",
+ "tb-lite-random-routing",
+ "--agent",
+ "claude-code",
+ "--reasoning-effort",
+ "high",
+ "--harbor-extra",
+ "--ak",
+ "--harbor-extra",
+ "thinking=disabled",
+ "--dry-run",
+ )
+
+ assert result.returncode == 0, result.stderr
+ harbor = _line_argv(result.stdout, "HARBOR_CMD: ")
+ thinking = [value for value in _option_values(harbor, "--ak") if value.startswith("thinking=")]
+ assert thinking == ["thinking=disabled"]
def test_harbor_path_is_required(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -391,14 +491,14 @@ def test_harbor_path_is_required(tmp_path: Path) -> None:
def test_closed_book_dataset_rejects_unpatched_harbor(tmp_path: Path) -> None:
dataset = _write_closed_book_dataset(tmp_path / "dataset")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
fake_python = _write_fake_harbor_python(tmp_path / "unpatched", patched=False)
result = _run_baseline(
tmp_path,
"--harbor-path",
str(dataset),
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -415,12 +515,12 @@ def test_closed_book_dataset_rejects_unpatched_harbor(tmp_path: Path) -> None:
assert "patch -p1 <" in result.stderr
-def test_dry_run_routing_profile_uses_switchyard_serve(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+def test_dry_run_server_config_uses_rust_switchyard_server(tmp_path: Path) -> None:
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--model",
"tb-lite-random-routing",
@@ -430,23 +530,47 @@ def test_dry_run_routing_profile_uses_switchyard_serve(tmp_path: Path) -> None:
assert result.returncode == 0, result.stderr
server = _line_argv(result.stdout, "SERVER_CMD: ")
harbor = _line_argv(result.stdout, "HARBOR_CMD: ")
- # New form: uv run --no-sync switchyard --routing-profiles FILE -- serve ...
- assert server[:4] == ["uv", "run", "--no-sync", "switchyard"]
- assert _option_value(server, "--routing-profiles") == str(profile)
- assert "serve" in server
+ assert server[0] == "switchyard-server"
+ assert _option_value(server, "--config") == str(profile)
assert _option_value(server, "--host") == "0.0.0.0"
assert _option_value(server, "--port") == "4000"
assert _option_value(harbor, "--model") == "openai/tb-lite-random-routing"
- assert "route_profile:" in result.stdout
+ assert "server_config:" in result.stdout
assert "route_model: tb-lite-random-routing" in result.stdout
-def test_dry_run_route_model_alias_uses_switchyard_serve(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+def test_foreground_server_run_collects_rust_stats_endpoint(tmp_path: Path) -> None:
+ profile = _write_server_config(tmp_path / "routes.toml")
+ docker_log = tmp_path / "docker.log"
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
+ str(profile),
+ "--route-model",
+ "tb-lite-random-routing",
+ "--foreground",
+ fake_server=True,
+ env={"FAKE_DOCKER_LOG": str(docker_log)},
+ )
+
+ assert result.returncode == 0, result.stderr
+ run_dir = next((tmp_path / "out").glob("baseline-serve-serve-*"))
+ manifest = json.loads((run_dir / "run_manifest.json").read_text())
+ stats = json.loads((run_dir / "routing_stats_final.json").read_text())
+
+ assert stats["total_requests"] == 2
+ assert manifest["outcomes"]["routing_stats_json_status"] == "present"
+ assert manifest["outcomes"]["server_metrics_prom_status"] == "present"
+ assert "--env NVIDIA_API_KEY" in docker_log.read_text()
+
+
+def test_dry_run_route_model_alias_uses_rust_switchyard_server(tmp_path: Path) -> None:
+ profile = _write_server_config(tmp_path / "routes.toml")
+
+ result = _run_baseline(
+ tmp_path,
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -456,18 +580,18 @@ def test_dry_run_route_model_alias_uses_switchyard_serve(tmp_path: Path) -> None
assert result.returncode == 0, result.stderr
server = _line_argv(result.stdout, "SERVER_CMD: ")
harbor = _line_argv(result.stdout, "HARBOR_CMD: ")
- assert server[:4] == ["uv", "run", "--no-sync", "switchyard"]
- assert "serve" in server
+ assert server[0] == "switchyard-server"
+ assert _option_value(server, "--config") == str(profile)
assert _option_value(harbor, "--model") == "openai/tb-lite-random-routing"
assert "route_model: tb-lite-random-routing" in result.stdout
-def test_dry_run_routing_profile_honors_explicit_harbor_model(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+def test_dry_run_server_config_honors_explicit_harbor_model(tmp_path: Path) -> None:
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -481,33 +605,12 @@ def test_dry_run_routing_profile_honors_explicit_harbor_model(tmp_path: Path) ->
assert _option_value(harbor, "--model") == "openai/custom-route-label"
-def test_checked_in_routing_profiles_load(monkeypatch) -> None:
- monkeypatch.setenv("OPENROUTER_API_KEY", "or-test")
-
- import switchyard.cli.route_bundle as route_bundle
-
- profiles = {
- "gpt-5-5-deepseek-gemini.yaml": "switchyard",
- "gpt-5-5-kimi-gemini.yaml": "switchyard",
- "tb-lite-llm-classifier-opus-deepseek-gemini.yaml": "switchyard",
- "tb-lite-llm-classifier-opus-kimi-gemini.yaml": "switchyard",
- "tb-lite-single-opus-4-7.yaml": "tb-lite-single-opus-4-7",
- "tb-lite-single-gpt-5-5.yaml": "tb-lite-single-gpt-5-5",
- }
- monkeypatch.setattr(route_bundle, "fetch_model_ids", lambda _base_url, _api_key: [])
- for filename, model in profiles.items():
- table = route_bundle.load_route_bundle_table(
- REPO / "benchmark" / "routing-profiles" / filename
- )
- assert model in table.registered_models()
-
-
def test_dry_run_harbor_extra(tmp_path: Path) -> None:
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -529,13 +632,13 @@ def test_dry_run_harbor_path_uses_local_dataset_and_closed_book_artifact(
tmp_path: Path,
) -> None:
dataset = _write_closed_book_dataset(tmp_path / "dataset")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
"--harbor-path",
str(dataset),
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -572,13 +675,13 @@ def test_dry_run_harbor_path_uses_local_dataset_and_closed_book_artifact(
def test_dry_run_open_book_uses_proxy_topology_without_tool_disables(tmp_path: Path) -> None:
dataset = _write_closed_book_dataset(tmp_path / "dataset")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
"--harbor-path",
str(dataset),
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -604,13 +707,13 @@ def test_dry_run_open_book_uses_proxy_topology_without_tool_disables(tmp_path: P
def test_dry_run_claude_closed_book_merges_disallowed_tools(tmp_path: Path) -> None:
dataset = _write_closed_book_dataset(tmp_path / "dataset")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
"--harbor-path",
str(dataset),
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -632,13 +735,13 @@ def test_dry_run_claude_closed_book_merges_disallowed_tools(tmp_path: Path) -> N
def test_dry_run_opencode_closed_book_disables_webfetch(tmp_path: Path) -> None:
dataset = _write_closed_book_dataset(tmp_path / "dataset")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
"--harbor-path",
str(dataset),
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -658,11 +761,11 @@ def test_dry_run_opencode_closed_book_disables_webfetch(tmp_path: Path) -> None:
def test_task_list_file_expands_to_include_task_name(tmp_path: Path) -> None:
task_list = tmp_path / "tasks.txt"
task_list.write_text("# comment\nalpha\n\nbeta # inline\n")
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
@@ -680,11 +783,11 @@ def test_dry_run_uses_harbor_bin_override(tmp_path: Path) -> None:
override_bin = tmp_path / "override-bin"
override_bin.mkdir()
expected_harbor = _write_fake_harbor(override_bin)
- profile = _write_route_profile(tmp_path / "routes.yaml")
+ profile = _write_server_config(tmp_path / "routes.toml")
result = _run_baseline(
tmp_path,
- "--routing-profiles",
+ "--server-config",
str(profile),
"--route-model",
"tb-lite-random-routing",
diff --git a/tests/test_run_manifest.py b/tests/test_run_manifest.py
index e920ac26d..d107f825c 100644
--- a/tests/test_run_manifest.py
+++ b/tests/test_run_manifest.py
@@ -117,14 +117,12 @@ def test_cli_write_applies_extra_fields(tmp_path: Path) -> None:
assert manifest["harbor"]["dataset_fingerprint"].startswith("sha256:")
-def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:
+def test_cli_write_records_server_config_digest(tmp_path: Path) -> None:
module = _load_manifest_module()
out = tmp_path / "run_manifest.json"
run_dir = tmp_path / "run"
- profile = tmp_path / "routes.yaml"
- profile.write_text(
- "routes:\n tb-lite-random-routing:\n type: model\n target: some/model\n"
- )
+ config = tmp_path / "routes.toml"
+ config.write_text('schema_version = 1\n\n[routes.noop]\nid = "noop"\ntype = "noop"\n')
dataset = tmp_path / "dataset"
dataset.mkdir()
@@ -135,12 +133,10 @@ def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:
str(out),
"--server-preset",
"serve",
- "--routing-profiles",
- str(profile),
+ "--server-config",
+ str(config),
"--route-model",
- "tb-lite-random-routing",
- "--classifier-prompts-json",
- '{"tb-lite-random-routing":{"classifier_prompt_sha256":"abc"}}',
+ "noop",
"--harbor-path",
str(dataset),
"--agent",
@@ -166,55 +162,13 @@ def test_cli_write_records_routing_profile_digest(tmp_path: Path) -> None:
assert rc == 0
manifest = json.loads(out.read_text())
- assert manifest["server"]["routing_profiles"] == str(profile)
- assert manifest["server"]["routing_profiles_digest"] == module.path_digest(profile)
- snapshot = run_dir / "routing_profiles" / profile.name
- assert manifest["server"]["routing_profiles_snapshot"] == str(snapshot)
- assert manifest["server"]["routing_profiles_snapshot_digest"] == module.path_digest(snapshot)
- assert snapshot.read_bytes() == profile.read_bytes()
- assert manifest["server"]["route_model"] == "tb-lite-random-routing"
- assert manifest["server"]["classifier_prompts"] == {
- "tb-lite-random-routing": {"classifier_prompt_sha256": "abc"}
- }
-
-
-def test_cli_write_rejects_non_object_classifier_prompts(tmp_path: Path) -> None:
- module = _load_manifest_module()
- out = tmp_path / "run_manifest.json"
- run_dir = tmp_path / "run"
-
- rc = module._cli_main(
- [
- "write",
- "--output",
- str(out),
- "--server-preset",
- "serve",
- "--classifier-prompts-json",
- "[]",
- "--agent",
- "terminus-2",
- "--harbor-model",
- "tb-lite-random-routing",
- "--n-concurrent",
- "1",
- "--max-retries",
- "0",
- "--agent-timeout-multiplier",
- "1.0",
- "--run-dir",
- str(run_dir),
- "--log-path",
- str(run_dir / "run.log"),
- "--harbor-result-json",
- str(run_dir / "harbor_result.json"),
- "--routing-stats-json",
- str(run_dir / "routing_stats_final.json"),
- ]
- )
-
- assert rc == 2
- assert not out.exists()
+ assert manifest["server"]["server_config"] == str(config)
+ assert manifest["server"]["server_config_digest"] == module.path_digest(config)
+ snapshot = run_dir / "server_config" / config.name
+ assert manifest["server"]["server_config_snapshot"] == str(snapshot)
+ assert manifest["server"]["server_config_snapshot_digest"] == module.path_digest(snapshot)
+ assert snapshot.read_bytes() == config.read_bytes()
+ assert manifest["server"]["route_model"] == "noop"
def test_cli_write_records_direct_upstream_mode_without_routing_stats(tmp_path: Path) -> None:
@@ -272,7 +226,7 @@ def test_cli_write_records_direct_upstream_mode_without_routing_stats(tmp_path:
assert manifest["server"]["mode"] == "direct"
assert manifest["server"]["upstream_base_url"] == "https://inference-api.nvidia.com/v1"
assert manifest["server"]["upstream_api_key_env"] == "NVIDIA_API_KEY"
- assert manifest["server"]["routing_profiles"] is None
+ assert manifest["server"]["server_config"] is None
assert manifest["outcomes"]["routing_stats_json_status"] == "not-requested"
rc = module.finalize_manifest(out, harbor_rc=0, harbor_job_dir=run_dir / "jobs" / "job")
@@ -417,24 +371,35 @@ def test_finalize_copies_harbor_result_and_marks_stats(tmp_path: Path) -> None:
stats = run_dir / "routing_stats_final.json"
stats.parent.mkdir(parents=True, exist_ok=True)
stats.write_text('{"total_requests":1}\n')
+ metrics = run_dir / "server_metrics_final.prom"
+ metrics.write_text("switchyard_requests_total 1\n")
module.write_manifest(
out,
outcomes={
"harbor_result_json": str(run_dir / "harbor_result.json"),
"harbor_result_json_status": "predicted",
+ "server_metrics_prom": str(metrics),
+ "server_metrics_prom_status": "predicted",
"routing_stats_json": str(stats),
"routing_stats_json_status": "predicted",
"harbor_rc": None,
},
)
- rc = module.finalize_manifest(out, harbor_rc=0, harbor_job_dir=job_dir, routing_stats=stats)
+ rc = module.finalize_manifest(
+ out,
+ harbor_rc=0,
+ harbor_job_dir=job_dir,
+ server_metrics=metrics,
+ routing_stats=stats,
+ )
assert rc == 0
manifest = json.loads(out.read_text())
assert manifest["outcomes"]["harbor_rc"] == 0
assert manifest["outcomes"]["harbor_result_json_status"] == "present"
+ assert manifest["outcomes"]["server_metrics_prom_status"] == "present"
assert manifest["outcomes"]["routing_stats_json_status"] == "present"
assert (run_dir / "harbor_result.json").is_file()
@@ -469,149 +434,3 @@ def test_finalize_copies_proxy_strip_log_artifact(tmp_path: Path) -> None:
manifest = json.loads(out.read_text())
assert manifest["closed_book"]["proxy_strip_log_status"] == "present"
assert (run_dir / "proxy_strip_log.jsonl").read_text() == '{"removed":["web_search"]}\n'
-
-
-def _record(task: str, ts: str, model: str, tier: str, *, trial: str = "t", session: str = "s",
- **tokens: int) -> str:
- base = {"prompt_tokens": 0, "cached_tokens": 0, "cache_creation_tokens": 0,
- "completion_tokens": 0, "reasoning_tokens": 0, "total_tokens": 0}
- base.update(tokens)
- return json.dumps({"task": task, "ts": ts, "trial_id": trial, "session_id": session,
- "model": model, "tier": tier, **base})
-
-
-def test_summarize_routing_log_groups_by_task_and_model(tmp_path: Path) -> None:
- module = _load_manifest_module()
- log = tmp_path / "routing_requests.jsonl"
- log.write_text(
- "\n".join([
- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",
- prompt_tokens=10, cached_tokens=6, completion_tokens=2,
- reasoning_tokens=1, total_tokens=12),
- _record("task-a", "2026-01-01T00:00:02Z", "kimi", "weak",
- prompt_tokens=4, completion_tokens=1, total_tokens=5),
- "not json",
- json.dumps(["not", "a", "dict"]),
- ]) + "\n"
- )
-
- summary = module.summarize_routing_log(log)
-
- assert summary["total_requests"] == 2
- task_a = summary["tasks"]["task-a"]
- assert task_a["requests"] == 2
- assert task_a["n_retries"] == 0
- assert task_a["retries"]["calls"] == 0
- assert task_a["final"]["calls"] == 2
- assert task_a["final"]["totals"] == {
- "prompt_tokens": 14, "cached_tokens": 6, "cache_creation_tokens": 0,
- "completion_tokens": 3, "reasoning_tokens": 1, "total_tokens": 17,
- }
- assert [(b["model"], b["tier"], b["calls"]) for b in task_a["final"]["models"]] == [
- ("kimi", "weak", 1), ("opus", "strong", 1),
- ]
-
-
-def test_summarize_routing_log_nets_out_retried_attempt(tmp_path: Path) -> None:
- module = _load_manifest_module()
- log = tmp_path / "routing_requests.jsonl"
- # Same trial, two attempts: the earlier session was retried away.
- log.write_text(
- "\n".join([
- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",
- trial="trial-1", session="retry-sess",
- prompt_tokens=100, completion_tokens=10, total_tokens=110),
- _record("task-a", "2026-01-01T00:05:01Z", "opus", "strong",
- trial="trial-1", session="final-sess",
- prompt_tokens=20, completion_tokens=3, total_tokens=23),
- ]) + "\n"
- )
-
- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]
-
- assert task_a["n_retries"] == 1
- assert task_a["final"]["totals"]["total_tokens"] == 23
- assert task_a["retries"]["totals"]["total_tokens"] == 110
- assert task_a["final"]["calls"] == 1
- assert task_a["retries"]["calls"] == 1
-
-
-def test_summarize_routing_log_keeps_all_concurrent_trials(tmp_path: Path) -> None:
- module = _load_manifest_module()
- log = tmp_path / "routing_requests.jsonl"
- # Two trials of the same task (k>1). Neither retried, so both are final even
- # though their calls interleave in time.
- log.write_text(
- "\n".join([
- _record("task-a", "2026-01-01T00:00:01Z", "opus", "strong",
- trial="trial-1", session="s1", total_tokens=10),
- _record("task-a", "2026-01-01T00:00:02Z", "opus", "strong",
- trial="trial-2", session="s2", total_tokens=20),
- ]) + "\n"
- )
-
- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]
-
- assert task_a["n_retries"] == 0
- assert task_a["final"]["calls"] == 2
- assert task_a["final"]["totals"]["total_tokens"] == 30
- assert task_a["retries"]["calls"] == 0
-
-
-def test_summarize_routing_log_untagged_records_all_final(tmp_path: Path) -> None:
- module = _load_manifest_module()
- log = tmp_path / "routing_requests.jsonl"
- # No trial_id -> no netting, everything counts as final.
- log.write_text(json.dumps({
- "task": "task-a", "ts": "2026-01-01T00:00:01Z", "session_id": "s1",
- "model": "opus", "tier": "strong", "total_tokens": 5,
- }) + "\n")
-
- task_a = module.summarize_routing_log(log)["tasks"]["task-a"]
-
- assert task_a["final"]["calls"] == 1
- assert task_a["retries"]["calls"] == 0
- assert task_a["n_retries"] == 0
-
-
-def test_finalize_writes_routing_stats_by_task(tmp_path: Path) -> None:
- module = _load_manifest_module()
- out = tmp_path / "run_manifest.json"
- run_dir = tmp_path / "run"
- run_dir.mkdir()
- log = run_dir / "routing_requests.jsonl"
- log.write_text(json.dumps({
- "task": "task-a", "session_id": "s1", "model": "opus", "tier": "strong",
- "prompt_tokens": 10, "cached_tokens": 3, "cache_creation_tokens": 1,
- "completion_tokens": 2, "total_tokens": 12,
- }) + "\n")
- by_task = run_dir / "routing_stats_by_task.json"
-
- module.write_manifest(out, outcomes={"harbor_rc": None})
- rc = module.finalize_manifest(
- out, harbor_rc=0, routing_log=log, routing_stats_by_task=by_task,
- )
-
- assert rc == 0
- summary = json.loads(by_task.read_text())
- assert summary["tasks"]["task-a"]["final"]["models"][0]["calls"] == 1
- assert summary["tasks"]["task-a"]["final"]["models"][0]["cached_tokens"] == 3
- manifest = json.loads(out.read_text())
- assert manifest["outcomes"]["routing_stats_by_task_json_status"] == "present"
-
-
-def test_finalize_marks_routing_stats_by_task_missing_without_log(tmp_path: Path) -> None:
- module = _load_manifest_module()
- out = tmp_path / "run_manifest.json"
- by_task = tmp_path / "routing_stats_by_task.json"
-
- module.write_manifest(out, outcomes={"harbor_rc": None})
- rc = module.finalize_manifest(
- out, harbor_rc=1,
- routing_log=tmp_path / "absent.jsonl", routing_stats_by_task=by_task,
- )
-
- assert rc == 0
- assert not by_task.exists()
- manifest = json.loads(out.read_text())
- assert manifest["outcomes"]["routing_stats_by_task_json_status"] == "missing"