Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

chore(agents): restore stage router scorer skill (#154)

nachiketb-nvidiad2172958aa13
diff --git a/.agents/skills/switchyard-stage-router-scorer/SKILL.md b/.agents/skills/switchyard-stage-router-scorer/SKILL.md
new file mode 100644
index 000000000..46c3f3c5c
--- /dev/null
+++ b/.agents/skills/switchyard-stage-router-scorer/SKILL.md
@@ -0,0 +1,97 @@
+---
+name: switchyard-stage-router-scorer
+description: Score benchmark run trajectories through the stage-router Rust scorer and picker, then visualise score distributions. Use when you want to replay trajectories through the picker, analyse routing splits, or compare score distributions across configs.
+---
+
+# Skill: switchyard-stage-router-scorer
+
+## Scripts
+
+| Script | Purpose | Input | Output |
+|--------|---------|-------|--------|
+| `benchmark/score_staged_run.py` | Score a live run dir via real picker | run dir path | per-turn JSONL + per-task CSV |
+
+## Quick Reference
+
+```bash
+# Score a run
+uv run python benchmark/score_staged_run.py --run benchmark/tb_runs/<run-name>
+# → /tmp/<run-name>-scores.jsonl (per turn)
+# → /tmp/<run-name>-per-task.csv (per task)
+
+# Custom threshold or window
+uv run python benchmark/score_staged_run.py \
+ --run benchmark/tb_runs/<run-name> \
+ --threshold 0.15 --window 3
+```
+
+## Scoring pipeline (what score_staged_run.py does per turn)
+
+```
+trajectory step (tool_use + tool_result)
+ → append to cumulative Anthropic messages list
+ → ChatRequest.anthropic({"model": ..., "messages": messages}) # Rust binding
+ → dc.process(ctx, request) # DimensionCollector — one per task, accumulates state
+ → get_tool_result_signal(ctx) # read signal from ctx
+ → stage_score_signal(signal) # raw (score, confidence) from the Rust scorer (for analysis)
+ → pick_capable_first(ctx, threshold) # actual cf decision (same ctx, no re-process)
+ → pick_efficient_first(ctx, threshold) # actual ef decision (same ctx, no re-process)
+```
+
+**Key:** `dc.process()` is called **once per turn** on a single ctx. Both pickers read the signal
+already stored in that ctx — no duplicate processing.
+
+**What the picker does beyond raw score:**
+- **escalate** (`should_escalate`): `compacted` OR `severity >= 1.0` → force CAPABLE
+- **de-escalate** (`should_deescalate`): `tests_passed AND recent_write+edit >= 1 AND severity <= 0` → force EFFICIENT
+- `confidence < threshold` → fall_open to default tier (CAPABLE for cf, EFFICIENT for ef)
+- Only when `confidence >= threshold`: route by score direction
+
+## Per-turn JSONL schema
+
+```json
+{
+ "task_name": "terminal-bench/...", "trial_name": "...", "run_id": "...",
+ "reward": 1.0, "turn_depth": 5,
+ "score": -0.83, "confidence": 0.95,
+ "tool_name": "Bash", "is_error": false,
+ "write_count": 2, "edit_count": 1, "read_count": 3,
+ "no_error_streak": 4, "pure_bash_streak": 2, "tests_passed": false,
+ "pick_cf": 1,
+ "pick_ef": 0
+}
+```
+
+`pick_cf` / `pick_ef`: `1` = CAPABLE (Opus), `0` = EFFICIENT (Nemotron)
+
+## Per-task CSV columns
+
+`run_id, task_name, reward, n_turns, mean_score, mean_confidence,
+pct_strong_clear, pct_strong_uncertain, pct_weak_uncertain, pct_weak_clear,
+opus_pct_cf, nemotron_pct_cf, opus_pct_ef, nemotron_pct_ef`
+
+`opus_pct_cf` / `opus_pct_ef` are derived from actual `pick_cf` / `pick_ef` decisions, not score bands.
+
+## Band definitions (for histogram colouring, threshold T)
+
+| Band | Score range | cf default | ef default |
+|------|-------------|------------|------------|
+| strong_clear | ≥ T | Opus | Opus |
+| strong_uncertain | (0, T) | Opus (fall_open) | Nemotron (fall_open) |
+| weak_uncertain | (-T, 0) | Nemotron (fall_open) | Nemotron (fall_open) |
+| weak_clear | ≤ -T | Nemotron | Nemotron |
+
+Overrides can change any band. Use `pick_cf`/`pick_ef` for the true decision.
+
+## Key findings (v0.2.0 baseline tarballs, T=0.20)
+
+- Distribution is **highly bimodal** — most turns land in strong_clear or weak_clear
+- `cf, t=0.20`: ~34% Opus on partial run; ~42% on full baseline
+- `ef, t=0.20`: ~20% Opus on partial run; ~41% on full baseline
+- Live routing split differs from shadow score — Nemotron trajectories are shorter and reshape the distribution
+
+## Anti-patterns
+
+- Don't call `dc.process()` multiple times per turn for different pickers — both pickers read from the same ctx. One `dc.process()` call per turn is correct.
+- Don't infer routing split from score bands alone — overrides and fall_open change the actual decision. Always use `pick_cf` / `pick_ef`.
+- Don't compare cost directly across configs: Nemotron has ~39% cache hit rate vs ~92% for Opus.
diff --git a/AGENTS.md b/AGENTS.md
index f33718f7c..0a3117ad4 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -130,6 +130,7 @@ testing, and review do not require loading a skill.
| `switchyard-coding-agent-launchers` | Claude Code, Codex, or OpenClaw launcher behavior |
| `switchyard-docs` | Published MkDocs pages, strict builds, previews, and docs CI |
| `switchyard-rust-review` | Focused review of Rust, PyO3, async, streaming, and crate boundaries |
+| `switchyard-stage-router-scorer` | Replaying benchmark trajectories through the stage-router scorer and picker |
| `switchyard-testing-ci` | Selecting non-obvious validation or diagnosing CI failures |
Skills should contain stable operational constraints, not mutable architecture inventories. Read the