Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
craft-taskgen Pipeline State Machine
Map of the Tools (tool-orchestration) pipeline after the direct-API refactor. Source of truth for stage transitions, regen routing, and fast-paths.
Happy-path stage machine
stateDiagram-v2
direction LR
[*] --> CANDIDATE: miner
CANDIDATE --> EVALUATED: select_candidates
EVALUATED --> PROMISING: step_evaluate (accept)
EVALUATED --> REJECTED: step_evaluate (reject)
EVALUATED --> REJECTED: max_promising_per_repo cap
PROMISING --> ALIGNMENT_CHECKED: _run_build_align_candidates (any of N candidates passed)
PROMISING --> REJECTED: _run_build_align_candidates (all N candidates failed)
BUILT --> ALIGNMENT_CHECKED: _run_alignment_only_for_triage (post-triage Build regen path)
ALIGNMENT_CHECKED --> TESTS_DISCOVERED: _run_assemble_task_dir_artifacts_one
TESTS_DISCOVERED --> DOCKERFILE_BUILT: _run_build_dockerfile_one
DOCKERFILE_BUILT --> F2P_P2P_CLASSIFIED: _run_docker_classify_one
F2P_P2P_CLASSIFIED --> ORACLE_CHECKED: _run_oracle_check_one (RESOLVED)
ORACLE_CHECKED --> OPUS_SMOKE_TESTED: Opus harbor smoke
OPUS_SMOKE_TESTED --> OPUS_TRIAGED: _run_triage_one (accept)
OPUS_TRIAGED --> ACCEPTED: _compare_and_accept
NEEDS_FIX: NEEDS_FIX<br/>(human review)
REJECTED: REJECTED<br/>(terminal)
ACCEPTED: ACCEPTED<br/>(terminal)
What changed (vs. the pre-refactor diagram):
- Evaluate emits a binary
accept/rejectverdict — noMAYBEband. - Hardness check deleted. H-rule enforcement moved upstream into the Build prompt; audit moved downstream into the alignment judge.
- Alignment judge (new): GPT-5.4 cross-family audit of
instruction ↔ reference-test alignment. Two retry mechanisms apply,
addressing different failure modes:
- Assessor retry (judge-flakiness mitigation): re-poll the
judge on the same instruction up to
_cfg.ALIGNMENT_MAX_RETRIEStimes; accept on firstok. Default α=1 (Apr 25 2026 update — no retention bias; single roll per evaluation). Earlier α=3 biased toward retention (oneokwould override multiplenos, letting borderline-leaky instructions through). Profile knob; can be raised back to 3 via TOML if false-alarm rate becomes a problem in production. - Build retry (instruction-flakiness mitigation): if assessor
rejected with
leaked/narrow_tests, a Build regen runs with the assessor's leakage-evidence in the prompt, then a fresh assessor evaluation on the new instruction. Bounded at_cfg.MAX_BUILD_REGENS_PER_CANDIDATEper pipeline (default 2; profile knob).
- Assessor retry (judge-flakiness mitigation): re-poll the
judge on the same instruction up to
- N parallel build+alignment candidates per task (Apr 2026 — addresses
a third failure mode: unlucky build): Build is non-deterministic —
the same PR can produce a clean instruction on one roll and a leaky
one on another. Neither assessor retry (which doesn't change the
instruction) nor build retry (which can't always rescue a structurally
bad instruction) handles this. The orchestrator runs
_cfg.BUILD_N_CANDIDATES(default 2) independent build → assessor → build-retry → assessor loops concurrently per task; picks a passing candidate uniformly at random; rejects only when all candidates fail. See_run_build_align_candidatesand_one_candidate_loop. - Haiku smoke deleted. After OPUS_TRIAGED the task goes straight
to
_compare_and_accept, which now just sets ACCEPTED. Rank-inversion and both-zero gates removed (0 fires across ~500 bulk-run tasks).
Failure sinks
flowchart TD
subgraph Evaluate
E1[step_evaluate]
E1 -->|verdict=reject| R0[REJECTED: eval_reject]
E1 -->|direct-API / schema error| NF0[NEEDS_FIX: eval_error]
end
subgraph BuildAlign["Build + Alignment (N parallel candidates)"]
BA1[_run_build_align_candidates]
BA1 -->|all N candidates: build infra error| NF1[NEEDS_FIX: build_exhausted]
BA1 -->|all N candidates: assessor non-ok non-actionable<br/>vague/misaligned| R1[REJECTED: alignment_reject]
BA1 -->|all N candidates: assessor non-ok actionable<br/>leaked/narrow_tests, build-retry budget exhausted| R1b[REJECTED: alignment_reject]
end
subgraph Assemble
A1[_run_assemble_task_dir_artifacts_one]
A1 -->|solve.sh gen fail| NF4a[NEEDS_FIX: solve_gen_failed]
A1 -->|git infra failure| NF4b[NEEDS_FIX: git_infra]
A1 -->|no test files in diff| NF4c[NEEDS_FIX: no_test_files]
A1 -->|all postmerge extracts failed| NF4d[NEEDS_FIX: postmerge_extract_failed]
end
subgraph BuildDockerfile
BD1[_run_build_dockerfile_one]
BD1 -->|Claude error| NF5a[NEEDS_FIX: dockerfile_claude_error]
BD1 -->|Dockerfile not created| NF5b[NEEDS_FIX: dockerfile_not_created]
end
subgraph Classify
C1[_run_docker_classify_one]
C1 -->|OVERLAY_UNCOLLECTED| R2[REJECTED: overlay_uncollected]
C1 -->|OVERLAY_REGRESSION| R3[REJECTED: test_regressed]
C1 -->|ORACLE_ZERO after fix fails| NF7[NEEDS_FIX: classify_oracle_empty]
end
subgraph Oracle
O1[_run_oracle_check_one]
O1 -->|f2p<1 or p2p<1| NF8[NEEDS_FIX: oracle_not_resolved]
end
subgraph Smoke
S1[harbor run smoke]
S1 -->|timeout 30min| NF9[NEEDS_FIX: smoke_timeout]
S1 -->|infra_failure / no_trial| NF10[NEEDS_FIX: smoke_infra]
end
subgraph Triage
T1[_run_triage_one]
T1 -->|thin F2P: remaining ≤1| R4[REJECTED: thin_f2p]
T1 -->|deep-dive direct-API error| NF12[NEEDS_FIX: triage_error]
T1 -->|triage_regen_count ≥ MAX_TRIAGE_REGENS| NF13[NEEDS_FIX: triage_regen_exhausted]
T1 -->|Build regen LLM error| NF14[NEEDS_FIX: triage_regen_error]
end
style NF0 fill:#fef3c7,stroke:#d97706
style NF1 fill:#fef3c7,stroke:#d97706
style NF4a fill:#fef3c7,stroke:#d97706
style NF4b fill:#fef3c7,stroke:#d97706
style NF4c fill:#fef3c7,stroke:#d97706
style NF4d fill:#fef3c7,stroke:#d97706
style NF5a fill:#fef3c7,stroke:#d97706
style NF5b fill:#fef3c7,stroke:#d97706
style NF7 fill:#fef3c7,stroke:#d97706
style NF8 fill:#fef3c7,stroke:#d97706
style NF9 fill:#fef3c7,stroke:#d97706
style NF10 fill:#fef3c7,stroke:#d97706
style NF12 fill:#fef3c7,stroke:#d97706
style NF13 fill:#fef3c7,stroke:#d97706
style NF14 fill:#fef3c7,stroke:#d97706
style R0 fill:#fca5a5,stroke:#b91c1c
style R1 fill:#fca5a5,stroke:#b91c1c
style R1b fill:#fca5a5,stroke:#b91c1c
style R2 fill:#fca5a5,stroke:#b91c1c
style R3 fill:#fca5a5,stroke:#b91c1c
style R4 fill:#fca5a5,stroke:#b91c1c
Triage internals (_run_triage_one)
Parallel dual deep dive (primary Opus + secondary GPT-5.4, same
pre-assembled evidence), deterministic per-failure merge, filter chain,
then classify + deterministic auto-skip + optional Build regen. The
claude-p fix-agent that edited instruction / skip / Dockerfile files
was replaced with a single direct-API Build regen call that rewrites
instruction.md from a <triage_feedback> block; Dockerfile and skip
edits are handled deterministically or not at all at triage time.
flowchart TD
START([entry: stage=OPUS_SMOKE_TESTED])
START --> CTX[_fetch_deep_dive_context<br/>harbor-lab errors/edits/<br/>tool-sequence/metrics + static files]
CTX --> REWARD{reward == 1.0?}
REWARD -->|yes| EASE1[_deterministic_easiness<br/>grep_read / pytest counts]
EASE1 --> EASEGATE{easiness_flag<br/>AND regen budget<br/>available?}
EASEGATE -->|no flag| ACCEPT0([stage = accept_stage<br/>reviewer skipped — no failures])
EASEGATE -->|flag + budget OK| EASEREGEN[Build regen<br/>prescriptive-instruction feedback]
EASEGATE -->|flag + budget spent| NF_EASY[NEEDS_FIX:<br/>structurally too easy]
EASEREGEN --> EASEREROUTE[stage = BUILT<br/>pending_fix_type='instruction'<br/>triage_regen_count++]
EASEREROUTE --> ALIGNMENT2[next: run_task_pipeline<br/>alignment → smoke → triage]
REWARD -->|no| JUDGES[[asyncio.gather]]
JUDGES --> DD[Opus deep dive<br/>llm_judge.judge DEEP_DIVE_SCHEMA<br/>Opus 4.6 - _cfg.LLM_STEP_MODEL<br/>per-test skip/keep verdict]
JUDGES --> REV[Fairness review<br/>llm_judge.judge FAIRNESS_REVIEW_SCHEMA<br/>GPT-5.4 - _cfg.LLM_ALIGNMENT_MODEL<br/>severity + evidence]
DD -->|error| NF[NEEDS_FIX]
REV -->|error| REVFAIL[reviewer result = None<br/>treated as severity=none]
REVFAIL --> SKIPFILTER
DD --> SKIPFILTER
REV --> SKIPFILTER
SKIPFILTER{Skip-filter:<br/>drop verdicts already in<br/>f2p_skip.txt / p2p_skip.txt}
SKIPFILTER --> REWARDFILTER{Reward-filter:<br/>drop verdicts on tests<br/>that actually passed}
REWARDFILTER --> AUTOSKIP
AUTOSKIP{Any `skip` verdicts<br/>on F2P tests?}
AUTOSKIP -->|yes| WRITE[Append skip lines<br/>to f2p_skip.txt]
WRITE --> RESCORE[_rescore_trial<br/>against existing trial]
AUTOSKIP -->|no| THINCHECK
RESCORE -->|reward == 1.0| ACCEPT1([stage = accept_stage<br/>reviewer flag preserved as soft signal])
RESCORE -->|reward < 1.0| THINCHECK
THINCHECK{Thin-F2P check:<br/>f2p_tests − skippable ≤ 1?}
THINCHECK -->|yes| REJ([REJECTED: thin_f2p])
THINCHECK -->|no| REVGATE
REVGATE{severity=major AND<br/>evidence_quote AND<br/>evidence_test?}
REVGATE -->|no| ACCEPT2([stage = accept_stage<br/>keep verdicts = genuine gap])
REVGATE -->|yes| REGENGATE{triage_regen_count<br/>≥ MAX_TRIAGE_REGENS?}
REGENGATE -->|yes| NF2[NEEDS_FIX<br/>regen budget exhausted]
REGENGATE -->|no| REGEN[_run_triage_build_regen<br/>direct API — Build rewrites<br/>instruction.md with reviewer<br/>evidence in feedback block]
REGEN -->|LLM error / empty output| NF3[NEEDS_FIX<br/>regen error]
REGEN -->|success| REROUTE[stage = BUILT<br/>pending_fix_type='instruction'<br/>triage_regen_count++]
REROUTE --> ALIGNMENT[next: run_task_pipeline<br/>alignment → smoke → triage]
style REWARD fill:#bae6fd,stroke:#0284c7
style AUTOSKIP fill:#bae6fd,stroke:#0284c7
style SKIPFILTER fill:#bae6fd,stroke:#0284c7
style REWARDFILTER fill:#bae6fd,stroke:#0284c7
style THINCHECK fill:#bae6fd,stroke:#0284c7
style REVGATE fill:#bae6fd,stroke:#0284c7
style REGENGATE fill:#bae6fd,stroke:#0284c7
style REGEN fill:#fde68a,stroke:#d97706
style ACCEPT0 fill:#bbf7d0,stroke:#166534
style ACCEPT1 fill:#bbf7d0,stroke:#166534
style ACCEPT2 fill:#bbf7d0,stroke:#166534
style REJ fill:#fca5a5,stroke:#b91c1c
style NF fill:#fef3c7,stroke:#d97706
style NF2 fill:#fef3c7,stroke:#d97706
style NF3 fill:#fef3c7,stroke:#d97706
Per-task control loop (run_task_pipeline)
Dispatches each task through stages based on current stage and
pending_fix_type. Two fast-path shortcuts keep the instruction-only
and dockerfile-only fix cycles from re-running work that can't change.
flowchart TD
ENTRY([run_task_pipeline: loop while stage in active_stages])
ENTRY --> CHECKSTAGE{task.stage}
CHECKSTAGE --> PROMISING_B[PROMISING: _run_build_align_candidates<br/>N parallel candidates]
CHECKSTAGE --> BUILT_A[BUILT: _run_alignment_only_for_triage<br/>post-triage assessor-only path]
CHECKSTAGE --> ALIGNED[ALIGNMENT_CHECKED: fast-path check]
CHECKSTAGE --> TESTS_BD[TESTS_DISCOVERED: _run_build_dockerfile_one]
CHECKSTAGE --> DOCKERFILE_C[DOCKERFILE_BUILT: _run_docker_classify_one]
CHECKSTAGE --> CLASSIFIED_O[F2P_P2P_CLASSIFIED: _run_oracle_check_one]
CHECKSTAGE --> ORACLE_S[ORACLE_CHECKED: Opus smoke]
CHECKSTAGE --> OPUS_T[OPUS_SMOKE_TESTED: Opus triage]
CHECKSTAGE --> TRIAGED_C[OPUS_TRIAGED: _compare_and_accept → ACCEPTED]
PROMISING_B --> ALIGN_CHECK{any candidate's<br/>final verdict ok?}
ALIGN_CHECK -->|yes| ALIGNED
ALIGN_CHECK -->|all candidates failed,<br/>build-retry budgets exhausted| REJ_AL[REJECTED: alignment_reject]
ALIGNED --> PENDING_CHECK{pending_fix_type<br/>== 'instruction' AND<br/>task.toml exists?}
PENDING_CHECK -->|yes| FASTORACLE[⚡ fast-path:<br/>stage = ORACLE_CHECKED<br/>skip assemble+dockerfile+<br/>classify+oracle]
PENDING_CHECK -->|no| NORMALASSEMBLE[_run_assemble_task_dir_artifacts_one]
style FASTORACLE fill:#fef08a,stroke:#ca8a04,stroke-width:2px
style PENDING_CHECK fill:#bae6fd,stroke:#0284c7
style ALIGN_CHECK fill:#bae6fd,stroke:#0284c7
style REJ_AL fill:#fca5a5,stroke:#b91c1c
Legend
- Green (double border) — ACCEPTED terminal state (happy path).
- Green (single border) — ACCEPTED via triage fast-path.
- Red — REJECTED terminal state.
- Yellow (rounded) — NEEDS_FIX (awaiting human review).
- Blue (diamond) — decision point (LLM classification or routing guard).
- Pale yellow (stage label) — fast-path shortcut that skips downstream steps.
Model / transport per step
| Step | Transport | Model | Structured output |
|---|---|---|---|
| step_evaluate | direct API (litellm) | Opus 4.6 | EVALUATE_SCHEMA — binary verdict |
| _build_instruction (per candidate) | direct API | Opus 4.6 | BUILD_SCHEMA — instruction_md + task_slug |
| _run_alignment_retry (per candidate) | direct API | GPT-5.4 | ALIGNMENT_SCHEMA — verdict enum + V4 audit |
| run_assemble* | subprocess (git) | — | no LLM |
| _run_build_dockerfile_one | claude -p |
Opus 4.6 | none (fs-writing agent) |
| _run_docker_classify_one | subprocess (docker) | — | no LLM |
| _run_oracle_check_one | subprocess (docker) | — | no LLM |
| _run_smoke_one | Harbor subprocess | Opus 4.6 | reward.json |
| _run_triage_one (Opus DD) | direct API | Opus 4.6 | DEEP_DIVE_SCHEMA — per-test skip / keep verdict. Skipped when reward==1.0 (deterministic easiness runs instead). |
| _run_triage_one (fairness review) | direct API | GPT-5.4 | FAIRNESS_REVIEW_SCHEMA — severity enum + evidence quote + evidence test. Parallel with Opus DD; also skipped on reward==1.0. |
| _run_triage_build_regen | direct API | Opus 4.6 | BUILD_SCHEMA — rewrites instruction.md using reviewer evidence as feedback |
| _generate_summary | direct API | Opus 4.6 | SUMMARY_SCHEMA — one-line narrative |
Two-family design, separated concerns: Opus generates (evaluate / build / Opus DD / Build-regen / summary); GPT-5.4 judges at build-time (alignment) and at triage-time (fairness review). No merge logic at triage — the two judges answer different questions (per-test skip/keep vs. task-level unfairness). No step has an LLM judging output from its own family.
REVIEWER_SCHEMA and skeptical_reviewer_prompt remain in
prompts.py only so scripts/calibrate-deep-dive.py can replay
historical cohorts. They are not on the live triage path and can be
deleted once that calibration is retired.
Key insights
Where the complexity lives
_run_triage_oneis the single most complex function — parallel Opus DD + fairness review, two filter stages (skip-file and reward.json), deterministic auto-skip + re-score, and the severity-gated Build regen branch. Everything else in the pipeline is sequential; this is where the branches multiply.Build retry loop is now internal to each candidate. After the N-parallel refactor, the leaked/narrow_tests → Build regen path no longer toggles
stage = PROMISINGand bounces through the outer state machine. Each candidate's_one_candidate_loopruns the build → assessor → (build retry once) → assessor sequence end-to-end before returning. Only the orchestrator's outer stage transitions are visible torun_task_pipeline.Triage regen loop is bounded by
task.triage_regen_count(capped atMAX_TRIAGE_REGENS). The counter persists across pipeline iterations (not reset per triage entry) so a task can't bounce between triage and Build indefinitely.State persistence via
pending_fix_typeoutlives a single triage invocation — it's onTaskState, serialized to state.json._run_triage_build_regensets it to"instruction";run_task_pipelineconsumes it at ALIGNMENT_CHECKED for the fast-path that skips assemble/dockerfile/classify/oracle.
Fast-paths (in order they fire)
| Name | When | Savings | Notes |
|---|---|---|---|
| Deep-dive skip-filter | Already-skipped tests surface in FAILED lines | Skip downstream filters for drop-outs | |
| Reward-filter | Classification points at a test that passed in-trial | Drops phantom classifications before accept/regen decisions | Parses reward.json f2p_failed/p2p_failed, falls back to pytest FAILED lines |
| Auto-write skip verdicts | Opus DD emits skip on an F2P test |
Deterministic write + fast rescore | |
| Skip-then-rescore to reward=1.0 | Applying skips brings trial reward to 1.0 | Accept without re-smoke; reviewer concern (if any) preserved as soft signal | |
| all-keep accept | All DD verdicts are keep, thick enough F2P, reviewer not major+evidence |
Accept at current score without regen | Task kept as hard-but-fair |
| Instruction-only re-alignment | After Build regen, task.toml still present | Skip assemble+dockerfile+classify+oracle | Only alignment + smoke + triage re-run |
| Alignment regen feedback | Alignment leaked/narrow_tests with evidence |
Build retry with flagged phrases removed |
Easiness / "too easy" detection
Inline signal (deterministic, no LLM): _deterministic_easiness
in steps.py. On reward == 1.0 trials, parses the harbor-lab
tool-sequence for Grep+Read count and pytest invocations. Sets
easiness_flag = True when grep_read_count <= _EASINESS_GREP_READ_MAX (5 today). pytest_runs is recorded for
observability but does not trigger the flag on its own.
Action, not soft flag. On easiness_flag=True:
- First pass (regen budget available): triggers Build regen with
prescriptive-instruction feedback from
easiness_triage_feedback_block— asks Build to strip named files/classes/data-structures/procedural language while preserving outcome + API contracts. Task routes back through alignment → smoke → triage. - Second pass (
triage_regen_count >= MAX_TRIAGE_REGENS): shelves asNEEDS_FIXwithneeds_human_review=True. Regen didn't help → task is structurally too easy.
Budget is shared with the reviewer-regen path. Calibrated against the
Apr 17 2026 cohort — see docs/reference/easiness-heuristics.md.
DD + reviewer are both skipped on reward == 1.0. The per-test
skip/keep has nothing to classify and the fairness reviewer has no
failing-test anchor. We spend only the deterministic easiness check
before either accepting or routing through easiness regen. Saves 2 LLM
calls (~$0.15-0.30 + ~20s) per passing task.
Offline companion: scripts/analyze-easiness.py computes richer
cohort-relative stats (p10 wall time, p10 turns, edit counts) for
post-run batch analysis. Not inline — a human invokes it manually.
Coverage gap acknowledged: the deterministic check is
cause-agnostic — it flags low-exploration trajectories without
distinguishing "instruction leaked the answer" from "task was just
small and well-scoped." Subtle instruction-leakage patterns that
slip past the build-time alignment judge and produce a
recipe-following reward=1.0 solve will show up in the easiness flag
but won't be diagnosed. A targeted leakage-audit LLM call gated on
easiness_flag = True is a plausible follow-up if bulk-run data
shows this failure mode.
Removed in Apr 23 2026 refactor: the short-lived LLM
easiness_signal field in DEEP_DIVE_SCHEMA + the
RUBRIC_EASINESS four-rule rubric. Calibration on the Apr 17 cohort
showed the LLM rubric flagged 21/27 reward=1.0 tasks as too_easy
(78%) — the judges were hallucinating no_exploration triggers on
tasks that had 30-156 Grep+Read calls. The deterministic count caught
the one genuinely-suspicious case (spack-229aca2b, 9 Grep+Read) and
no others.
Legacy signal still on TaskState for state.json compatibility but
no longer populated: easiness_concern.
Reviewer concern flag. reviewer_concern_flag mirrors
easiness_flag semantically — a soft signal from the cross-family
fairness-review step that does not block acceptance.
reviewer_concern_severity (none/minor/major) and the two
evidence fields accompany it. Human batch review filters on these
alongside easiness_flag.
Design invariants worth preserving
- Every stage has exactly one step function that consumes it and advances. No two steps handle the same stage.
f2p_skip.txt/p2p_skip.txtare write-once per triage cycle. Auto-skip is the only writer.pending_fix_typeis cleared on successful accept or full rebuild. Check this when adding new branches — easy to leak stale values.- Triage can only rewrite
instruction.md. All other file changes (skip files, Dockerfile) happen through deterministic paths (auto-skip, upstream fix_docker). Build regen touches exactly one file, which keeps the instruction-only fast-path sound. - No step's LLM judges output from its own family. Opus (Build) → GPT-5.4 (alignment) → (Opus DD ∥ GPT-5.4 fairness review). The two triage judges answer different questions (per-test skip/keep vs. task-level unfairness), so there is no merge step to contort — each acts directly on the signal it is best positioned to produce.
Tech debt: 3 separate retry counters around Build
Three distinct Build-retry mechanisms exist, each addressing a different failure mode and gated by its own counter. They share no state, so any one analysis only sees one slice — empirical tuning of any one cap can't be interpreted in isolation.
(Independent of these per-loop counters, the orchestrator runs
BUILD_N_CANDIDATES (default 2) parallel candidate loops per task.
Each candidate has its own copy of the per-loop budgets below; counters
are not shared across candidates. Parallelism addresses unlucky-build
variance, which retry counters cannot.)
| Mechanism | Counter | Cap | Trigger | Failure mode addressed |
|---|---|---|---|---|
| Build retry (alignment-feedback regen, per candidate) | alignment_regen_count (per CandidateResult) |
_cfg.MAX_BUILD_REGENS_PER_CANDIDATE (profile, default 2, capped at [0,3]) |
assessor verdict leaked/narrow_tests after assessor retries exhausted |
Build produced an instruction the assessor genuinely flagged |
| Triage Build regen | task.triage_regen_count gated by MAX_TRIAGE_REGENS |
profile constant (1) | triage fairness reviewer demands instruction rewrite | post-smoke quality concern |
| Outer fix-attempts gate (docker-fix paths) | task.fix_attempts |
MAX_FIX_ATTEMPTS (2) |
LLM gateway error / docker-fix failure | infra/transient failures |
Note: assessor retries (_ALIGNMENT_MAX_RETRIES=3) are NOT in this
table — they aren't a Build retry, they're a re-poll of the alignment
judge on the same instruction (judge-flakiness mitigation, separate
concern from instruction quality).
Same counter task.fix_attempts is overloaded by mechanisms #2 and #3
with different reset and increment rules: build path increments-then-checks
(MAX=2 means 1 retry); claude_cli docker-fix paths check-then-increment
(MAX=2 means 2 retries).
TODO (cleanup): unify into one named counter per logical retry ("build-retry budget", "triage-regen budget", "docker-fix budget"), drop the shared-counter overload, and re-cut the empirical accept-rate analysis once each axis is measurable independently.