Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
CRAFT Task Format Specification
A CRAFT task is a directory containing everything needed to evaluate an agent on a single coding challenge.
Required Structure
<task_id>/
instruction.md # What the agent must accomplish (50-200 words, wrapped in <task>...</task>)
task.toml # Task metadata (name, tags, difficulty, resource limits)
environment/
Dockerfile # Builds the evaluation environment (clones repo at base_sha commit)
solution/
solve.sh # Oracle: applies changes.patch to /code
changes.patch # git diff of the PR commit — generated mechanically from git diff
tests/ # Generated by F2P/P2P classification step — do not edit by hand
fail_to_pass.txt # Test IDs that fail pre-merge and pass post-merge (one per line)
pass_to_pass.txt # Test IDs that pass in both states (one per line)
test.sh # Harbor entry point: overlays postmerge tests, runs pytest, calls score.py
score.py # Generated scorer: reads F2P/P2P lists, writes reward.json
postmerge_tests/ # Test files extracted from the PR commit
<repo>/<path>/test_*.py
File Requirements
instruction.md
- 50-200 words
- Outcome-oriented: describes what "done" looks like, not how to get there
- Names the API contract (class names, module paths, config keys that tests import)
- Wrapped in
<task>...</task>XML tags - Mentions the base repo and commit SHA
- Agent workspace is at
/code/
Example wrapper format:
You are working in the {owner}/{repo} repository. The source code is in `/code/`.
Produce a solution for the given task. Write your changes directly to the files in `/code/`. Parallelize across agents.
<task>
{task description here}
The base repo is {owner}/{repo} at commit {base_sha}.
</task>
task.toml
[metadata]section withname,difficulty("easy"/"medium"/"hard"), andtags[agent]section withtimeout_sec[verifier]section withtimeout_sec[environment]section withcpus,memory_mb,storage_mb,allow_internet
environment/Dockerfile
- Clones repo at
base_shacommit (pre-merge state, not the merge commit) - Installs all dependencies including test deps
- Resets git history for clean agent workspace:
rm -rf .git && git init && git config ... - Agent workspace at
/code/
solution/solve.sh
- Applies
changes.patchmechanically:git apply /solution/changes.patch - Generated automatically from
git diff base_sha..merge_sha - Do not edit by hand
solution/changes.patch
- Standard unified diff from
git diff base_sha..merge_sha - Generated by the pipeline at classify step
tests/fail_to_pass.txt
- One fully-qualified pytest test ID per line:
module/path/test_file.py::test_name - Tests that fail on pre-merge code and pass after applying the fix
- Must be non-empty — at least one F2P test required
tests/pass_to_pass.txt
- Same format as fail_to_pass.txt
- Tests that pass in both pre-merge and post-merge states
- May be empty
tests/test.sh
- Harbor entry point, called by the scoring harness
- Overlays
postmerge_tests/into/code/(so newly-added tests exist during scoring) - Runs pytest on the union of F2P and P2P test lists
- Pipes output through
tee /logs/verifier/verify_full_output.txt - Calls
python3 /tests/score.pyto write reward.json
tests/score.py
- Generated at classification time — do not edit by hand
- Reads
fail_to_pass.txtandpass_to_pass.txt - Parses
verify_full_output.txtfor PASSED test IDs - Writes
reward.jsonwithreward,resolved,f2p_score,p2p_score, and counts
tests/postmerge_tests/
- Test files extracted from the PR commit via
git show {commit_sha}:{rel_path} - Stored under the same relative path as in the repo
- Overlaid into
/code/at scoring time so pytest can collect them
task.toml template
version = "1.0"
[metadata]
name = "{repo}_{pr_number}"
difficulty = "medium"
tags = ["{harbor-tasks_repo}", "{owner}/{repo}"]
[agent]
timeout_sec = 3600.0
[verifier]
timeout_sec = 600.0
[environment]
cpus = 2
memory_mb = 8192
storage_mb = 40960
allow_internet = true