Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
craft-taskgen
Automated CRAFT benchmark task generation pipeline.
Code Style
This repository is intended to be made public. Follow common Python conventions:
- Prefer explicit named functions over callable parameters for branching behaviour
- Use
from __future__ import annotationsand type hints throughout - Private helpers are
_-prefixed but should still be readable to outside contributors
Commands
uv sync # Install deps (NEVER use pip)
uv run pytest tests/ -v # Run tests (387 tests)
uv run ruff check src/ tests/ # Lint
uv run ruff format src/ tests/ # Format
uv run ruff format --check src/ tests/ # Check format
All code must pass ruff check and ruff format --check before merge.
Operational CLIs
Registered via pyproject.toml [project.scripts]:
craft-taskgen— main pipeline (Tools dimension by default;--dimension searchfor the Search pipeline)craft-taskgen-mine— GitHub PR miner (needsgh auth)craft-taskgen-validate— task directory validatorcraft-taskgen-convert— Harbor task converter (see Adapters)craft-taskgen-preflight— env validator: repos/, disk, Harbor, Docker, authcraft-taskgen-status— one-screen state.json summary for monitoringcraft-taskgen-split— balance candidate files into N shards (multi-machine runs)craft-taskgen-dashboard— interactive HTML dashboard from state.json (--watchregenerates every 10s)craft-taskgen-planning-score— planning-track scorer (seesrc/craft_taskgen/planning/)craft-taskgen-import— PR-reference importer for external benchmarks (seesrc/craft_taskgen/importers/)scripts/run-pipeline.sh— launcher: preflight → nohup → logged pipeline runscripts/run-baselines.sh— baseline launcher (claude-code/codex/opencode); seedocs/runbooks/baseline-reproducibility.md
For end-to-end batch setup (fresh VM, slice repos, mine, launch, monitor, resume) see docs/runbooks/running-a-batch.md.
Architecture
Pipeline: select candidates → evaluate → build task (instruction) → alignment judge → assemble artifacts (task.toml, solve.sh, Dockerfile, postmerge tests) → F2P/P2P classify → oracle check → agent smoke (Harbor; codex + GPT-5.5 by default) → triage (Opus per-test skip/keep + cross-family fairness review, severity-gated Build regen) → auto-skip loop → accept/reject (efficiency + reviewer-concern flags as soft signals for human review).
Per-step model/transport map (from the Apr 2026 refactor):
| Step | Transport | Model | Notes |
|---|---|---|---|
| evaluate | direct API (litellm) | Opus 4.6 | Binary accept/reject on PR candidates |
| build + alignment | direct API | Opus 4.6 + GPT-5.4 | Combined orchestrator — N parallel candidate loops (see below) |
| build_dockerfile | claude -p |
Opus 4.6 | Needs Bash/Write/Edit against the repo |
| docker_classify / oracle | subprocess | — | No LLM |
| smoke | Harbor (subprocess) | codex + GPT-5.5 (default) | Agent trial produces reward + trial dir. Agent/model configurable via smoke_agent/smoke_model/smoke_reasoning_effort (defaults in config.py). codex is cross-agent from the Opus deep-dive judge. Internal label/state fields keep the historical opus_ prefix. |
| deep dive (Opus) | direct API | Opus 4.6 | Per-test skip / keep verdict on failing reference tests. Deterministic auto-skip writes to f2p_skip.txt and re-scores. |
| fairness review | direct API | GPT-5.4 | Cross-family task-level severity verdict (none/minor/major). major + verbatim instruction quote + named failing test triggers one-shot Build regen. Else reviewer_concern_flag (soft signal). |
| task_summary | direct API | Opus 4.6 | One-line narrative from diagnostics |
Judge models are calibrated — don't bump them casually. The smoke agent/model is freely configurable (it's just the candidate being judged). The judge models are not: the triage severity-gating (MAX_TRIAGE_REGENS, the major+quote+test regen trigger) and the deep-dive/fairness prompts were tuned for Opus 4.6 + GPT-5.4. A June 2026 replay over 84 previously-fair v2b tasks (scripts/triage-replay.py) found GPT-5.5 returns 20–25 major verdicts vs GPT-5.4's 2 on identical inputs (codex or Opus-4.6 trajectories alike) — a pure model-strictness shift, ~78% run-to-run stable. Opus-4.6 deep-dive proposes 0 new skips on codex trials (all keep). So: changing a judge model (or its prompt) requires re-running triage-replay.py to recalibrate thresholds/prompts before trusting the verdicts. See docs/analyses/ for the full write-up.
Build + alignment orchestrator detail. Runs N=3 parallel build+align candidate loops per task (build_n_candidates, capped at 4). Each loop: build → alignment (alignment_max_retries=1 by default — single roll, no retention bias; configurable [1,5]) → on leaked/narrow_tests rebuild with cumulative feedback up to max_build_regens_per_candidate=2 times → re-align. Winner picked uniformly at random among passers; task rejects only if all candidates fail. Apr 25 2026 calibration on rerun-accepts-v2 cohort: default config (N=3, α=1, r=2) yielded 79/80 vs prior N=1's 73/80 at lower cost. Triage-induced Build regen is a separate single-shot path (see steps.py::_run_triage_build_regen).
Two-family design, separated concerns at triage: Opus returns per-test skip/keep verdicts, GPT-5.4 returns one task-level fairness-review severity. Reviewer's severity=major with both a verbatim instruction quote AND a named failing test triggers a one-shot Build regen; anything else sets reviewer_concern_flag as a soft signal (task still ships). A separate reward==1.0 + easiness_flag path also triggers Build regen (with prescriptive-instruction feedback asking for a more abstract rewrite); regen budget is shared with the reviewer path via MAX_TRIAGE_REGENS=1, and a second-pass easiness flag shelves the task as NEEDS_FIX. No merge logic at triage. No step has an LLM judging output from its own family.
Key modules in src/craft_taskgen/:
- config.py — PipelineProfile (TOML), PipelineContext, state models (Stage, TaskState, PipelineState).
PipelineState.run_infohas hostname + CLI args. - steps.py — step implementations. Uses
_cfg.CONSTANTfor profile-controlled values. - pipeline.py — entry point. Top-level
asyncio.gather(return_exceptions=True); a crashed task is NEEDS_FIX, the rest of the run continues. - prompts.py — prompt templates + schemas (EVALUATE_SCHEMA, BUILD_SCHEMA, ALIGNMENT_SCHEMA, DEEP_DIVE_SCHEMA, REVIEWER_SCHEMA, SUMMARY_SCHEMA).
- rubrics.py — authoritative rubric text as string constants (H1–H6, V4 audit, anti-leakage, alignment categories). Interpolated into prompts.
- llm_judge.py —
async judge(prompt, schema, model)via litellm; manual-parse +jsonschema.validate; one parse-retry with error feedback; 5× transient retry. Returns usage + latency for cost tracking. - gateway.py —
build_gateway_env(model)for bothclaude -pandllm_judge.judge. Gateway-only (no OAuth fallback). - claude_cli.py —
run_claude_asyncfor the remainingclaude -pcall (build_dockerfile). Retry: 5s base, 30s cap, jittered. - runner.py — Harbor smoke test runner
- docker.py — Docker build/verify helpers
- prefilters.py — pre-LLM candidate rejection (docs-only, CI-only, etc.)
- task_format.py — task directory validator
- preflight.py / status.py / split.py — operational CLIs
- adapters/ / mining/ / search/ — subpackages
- baselines/ — baseline-launcher helpers (output cap, reasoning-effort defaults, run-manifest schema). Driven by
scripts/run-baselines.sh. - planning/ — planning-track scorer + harbor + synth + CLI (
craft-taskgen-planning-score) - importers/ — external-benchmark PR-reference importer + CLI (
craft-taskgen-import)
All tools-pipeline LLM steps are direct-API. The only remaining claude -p call is build_dockerfile. (The empty launch/ and v2/ directories are stale skeletons left over from earlier iterations and can be removed.)
Skills in .claude/skills/: harbor-f2p-p2p-deep-dive (interactive triage helper). Note: craft-bench's CLAUDE.md mentions harbor-trial-deep-dive and task-hardness-checker as living here — both are stale references.
External CLIs:
- harbor-lab — required by the triage step.
_fetch_deep_dive_contextinsteps.pyshells out toharbor-lab errors / edits / tool-sequence / metricsto pre-assemble context for the direct-API deep-dive judge. Resolver:$HARBOR_LAB→ PATH →~/Documents/vscode/harbor-lab/.venv/bin/harbor-lab→/data/projects/harbor-lab/.venv/bin/harbor-lab. Preflight fails if none work.
Referenced docs (operational — load these when changing pipeline behavior):
- docs/runbooks/running-a-batch.md — canonical end-to-end runbook for a deterministic repo batch.
- docs/runbooks/baseline-reproducibility.md — reasoning-effort matrix + version-pin rationale for
scripts/run-baselines.sh. - docs/reference/easiness-heuristics.md — objective rules that flag accepted tasks as possibly "too easy" (efficiency signals from harbor-lab trajectory). Calibrated against the Apr 17 2026 run.
- docs/reference/pipeline-state-machine.md — stage transitions,
state.jsonschema. - docs/runbooks/task-review-workflow.md — manual-review procedure for
reviewer_concern_flag/easiness_flagtasks. - docs/reference/n-parallel-calibration-apr25.md — N-parallel build+alignment calibration that drove
build_n_candidates=3. - scripts/calibrate-alignment.py, scripts/calibrate-deep-dive.py, scripts/analyze-easiness.py — offline calibration tools for the direct-API judges and easiness flags.
- scripts/smoke-probe.py — fast smoke-step iteration harness. Runs only the Harbor agent trial (
runner._run_smoke_async) against an already-built task dir, so a new smoke agent/model (e.g. codex + GPT-5.5) can be validated in minutes instead of a 2-hr full pipeline run.--dry-runprints the resolved harbor argv + env (catalog path, reasoning effort) with no Docker.
Research-artifact docs (don't load by default — these document one-off analyses):
- docs/analyses/ — frozen-in-time analyses of completed runs; date-prefixed (e.g.
may02-v2b-regression-rigorous.md). Seedocs/analyses/README.mdfor the index.
When Changing Pipeline Logic
Any change to pipeline steps, stage ordering, or what each step produces must be reflected in:
README.md— Pipeline Steps section (step names, numbering, descriptions)references/task-building-guide.md— Pipeline steps list, "Verifier debugging" step reference, and any section that describes what a step produces (e.g. "Selecting Reference Tests")src/craft_taskgen/prompts.py— Check whether any prompt template references files, steps, or artifacts that changed (e.g. if a file is now generated by a different step, the prompt that previously created it needs updating)
Key Conventions
from __future__ import annotationsin all files- Profile-controlled constants accessed via
import craft_taskgen.config as _cfgthen_cfg.CONSTANT(NOTfrom config import CONSTANT— that creates copies that profile.apply() can't override) - ruff, line-length 110, target py312
- HTML template strings in dashboard.py are exempt from E501
- Gateway-only LLM:
claude -pis pinned to the NVIDIA gateway..envmust setANTHROPIC_API_KEY+ANTHROPIC_BASE_URL; every profile must setllm_step_modelto a gateway-shaped ID (e.g.aws/anthropic/bedrock-claude-opus-4-6). Never add an OAuth fallback path. Model is passed viaANTHROPIC_MODELenv, not--modelon the CLI. Tests that shell out need the fake gateway env fromtests/conftest.py(autouse) or must assert theRuntimeErrorguardrail explicitly. - Claude binary pinning:
config._resolve_claude_cmd()prefers~/.local/share/claude/versions/{CC_VERSION}over the system binary for consistent behavior across machines. Falls back to systemclaudeif the pinned version isn't installed.
Profiles
PipelineProfile loads from TOML, calls .apply() to update module-level constants. Stored in PipelineState.profile_data for run reproducibility.
profiles/craft-tools-v4.toml— tools pipeline (default forscripts/run-pipeline.sh); tuned for 12-CPU xlarge with concurrency 10,max_promising_per_repo=25, output underharbor-tasks/craft-tools-v4/profiles/craft-search.toml— Search dimension
Search Pipeline
craft-taskgen --dimension search routes to src/craft_taskgen/search/ package.
Steps: extract → synthesize → validate → dedup → harbor → smoke-opus → smoke-codex → smoke-haiku → gold-review → filter → report.
Uses SearchPipelineState (not PipelineState). State file: runs/<ts>/state.json.
Extract/validate delegate to craft-bench via subprocess (needs --craft-bench-dir).
--limit N processes only first N input tasks (useful for testing).
Key search modules:
- search/synthesize.py — 3-model LLM synthesis + cross-judging via litellm
- search/dedup.py — Embedding cosine similarity dedup (threshold 0.65)
- search/harbor.py — Search task → Harbor directory conversion (bundles verifier templates)
- search/gold_review.py — Automated gold review via Harbor + /gold-review skill
Harbor
Harbor pinned at git+...@a56546f (post-v0.13.1 main; includes #1826 native pi log-trim and #1840 wildcard network allowlist). Patches in patches/harbor-agent-patches.diff:
- base.py: raise
_truncate_outputcap 1000→8000 chars (diagnosable setup/exec logs) - claude_code.py: version-aware skip-install (skip only when the installed agent matches the pinned version) + apt tolerance. The attribution-header / telemetry /
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETASopt-outs moved out of the patch into the launchers (scripts/run-baselines.shandplanning/run_plan_impl.sh) via--agent-env - codex.py: preserve full model path for NVIDIA gateway; full
[model_providers.nvidia_gateway]stanza (wire_api=responses); custom model catalog (CODEX_MODEL_CATALOG_JSON) +--disable unified_exec; install hardening (version-aware skip-install, NVM retry, node16/ripgrep fallbacks) - opencode.py: custom
nvidia/vllmprovider (chat-completions) via unquoted-heredoc config write, per-mode sampling env-vars (OPENCODE_BUILD_*/OPENCODE_PLAN_*),permission=allowfor non-interactive containers, Node16-direct install (NVM bypass), version-aware skip-install - pi.py: npm package rename
@mariozechner→@earendil-works; customnvidiaprovider via planted~/.pi/agent/models.json - openhands_sdk.py: uv-bootstrap a parallel Python 3.12 when the container's
python3is older
The host-gateway reachability for --backend vllm is no longer a compose patch — it's patches/compose-overrides/host-gateway.yaml passed via harbor's native --extra-docker-compose (wired into scripts/run-baselines.sh).
Reapply after uv sync --reinstall-package harbor. If patch context mismatches, apply manually (see patches/README.md).
Agent-specific job output dirs (jobs/{agent}/) prevent collisions when running agents in parallel.
Adapters
src/craft_taskgen/adapters/ is the home for Harbor task converters. Each adapter is a subpackage with a converter.py exposing run_convert(...). Register new adapters in adapters/cli.py::_ADAPTERS — they become available as craft-taskgen-convert --adapter <name>.
Current adapters:
- search_native — native Search converter; builds fresh Dockerfile per task (clones repo, installs agent runtimes).
Shared search helpers (_harbor_utils.py in search/) define the canonical task.toml, instruction, verifier, and solve.sh layout so from-T2 and native converters stay in sync.