Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
4.1 KB
Raw Blame

CRAFT Task Format Specification

A CRAFT task is a directory containing everything needed to evaluate an agent on a single coding challenge.

Required Structure

<task_id>/
  instruction.md          # What the agent must accomplish (50-200 words, wrapped in <task>...</task>)
  task.toml               # Task metadata (name, tags, difficulty, resource limits)
  environment/
    Dockerfile            # Builds the evaluation environment (clones repo at base_sha commit)
  solution/
    solve.sh              # Oracle: applies changes.patch to /code
    changes.patch         # git diff of the PR commit — generated mechanically from git diff
  tests/                  # Generated by F2P/P2P classification step — do not edit by hand
    fail_to_pass.txt      # Test IDs that fail pre-merge and pass post-merge (one per line)
    pass_to_pass.txt      # Test IDs that pass in both states (one per line)
    test.sh               # Harbor entry point: overlays postmerge tests, runs pytest, calls score.py
    score.py              # Generated scorer: reads F2P/P2P lists, writes reward.json
    postmerge_tests/      # Test files extracted from the PR commit
      <repo>/<path>/test_*.py

File Requirements

instruction.md

  • 50-200 words
  • Outcome-oriented: describes what "done" looks like, not how to get there
  • Names the API contract (class names, module paths, config keys that tests import)
  • Wrapped in <task>...</task> XML tags
  • Mentions the base repo and commit SHA
  • Agent workspace is at /code/

Example wrapper format:

You are working in the {owner}/{repo} repository. The source code is in `/code/`.

Produce a solution for the given task. Write your changes directly to the files in `/code/`. Parallelize across agents.

<task>
{task description here}

The base repo is {owner}/{repo} at commit {base_sha}.
</task>

task.toml

  • [metadata] section with name, difficulty ("easy"/"medium"/"hard"), and tags
  • [agent] section with timeout_sec
  • [verifier] section with timeout_sec
  • [environment] section with cpus, memory_mb, storage_mb, allow_internet

environment/Dockerfile

  • Clones repo at base_sha commit (pre-merge state, not the merge commit)
  • Installs all dependencies including test deps
  • Resets git history for clean agent workspace: rm -rf .git && git init && git config ...
  • Agent workspace at /code/

solution/solve.sh

  • Applies changes.patch mechanically: git apply /solution/changes.patch
  • Generated automatically from git diff base_sha..merge_sha
  • Do not edit by hand

solution/changes.patch

  • Standard unified diff from git diff base_sha..merge_sha
  • Generated by the pipeline at classify step

tests/fail_to_pass.txt

  • One fully-qualified pytest test ID per line: module/path/test_file.py::test_name
  • Tests that fail on pre-merge code and pass after applying the fix
  • Must be non-empty — at least one F2P test required

tests/pass_to_pass.txt

  • Same format as fail_to_pass.txt
  • Tests that pass in both pre-merge and post-merge states
  • May be empty

tests/test.sh

  • Harbor entry point, called by the scoring harness
  • Overlays postmerge_tests/ into /code/ (so newly-added tests exist during scoring)
  • Runs pytest on the union of F2P and P2P test lists
  • Pipes output through tee /logs/verifier/verify_full_output.txt
  • Calls python3 /tests/score.py to write reward.json

tests/score.py

  • Generated at classification time — do not edit by hand
  • Reads fail_to_pass.txt and pass_to_pass.txt
  • Parses verify_full_output.txt for PASSED test IDs
  • Writes reward.json with reward, resolved, f2p_score, p2p_score, and counts

tests/postmerge_tests/

  • Test files extracted from the PR commit via git show {commit_sha}:{rel_path}
  • Stored under the same relative path as in the repo
  • Overlaid into /code/ at scoring time so pytest can collect them

task.toml template

version = "1.0"

[metadata]
name = "{repo}_{pr_number}"
difficulty = "medium"
tags = ["{harbor-tasks_repo}", "{owner}/{repo}"]

[agent]
timeout_sec = 3600.0

[verifier]
timeout_sec = 600.0

[environment]
cpus = 2
memory_mb = 8192
storage_mb = 40960
allow_internet = true