Skip to content
TypeScript
2.1 KB
Raw Blame

Acceptance — 20 verification metrics (PRD §19)

Each maps to a regression test under tests/. Run pnpm test.

BR — behavior (tests/br_behavior.test.ts)

id check
BR1 combo validity & exclude reasons (no silent drop)
BR2 subset determinism (same bench/tasks/seed → same subset; no Math.random)
BR3 runner never throws (returns success:false + artifacts)
BR4 harness per-run application (args/env/status)
BR5 bench setup + recompute-verify (independent solver writes correct answer → verify true; wrong → false)
BR6 multi-level subagent parsing (layout B, depth-2 via meta.toolUseId; token totals summed)
BR7 fail-soft (synthetic failure recorded, not dropped)

CO — consistency (tests/co_consistency.test.ts)

id check
CO1 totalTokens == sum of token fields
CO2 subagentCount ≤ toolCalls; codex/gpt = 0 subagents
CO3 sort base key subagentCount → totalTokens → wallTimeMs
CO4 cost precedence actual > estimated > none
CO5 null ≠ 0 (uncollected stays null)
CO6 codex failure fail-soft (missing binary → success:false, artifacts written)
CO7 golden correctness via independent solvers (Bellman-Ford vs dijkstra; independent ledger sum)

UX — presentation (tests/ux_presentation.test.ts)

id check
UX1 scatter present
UX2 task matrix present
UX3 flow graph present (inline SVG, MAIN LOOP)
UX4 node toggle present (skills/tools/log via native details)
UX5 i18n EN↔KR switch — no Korean leak in EN report; KR headers present
UX6 self-contained (no <script>, no CDN, no external src/link; only allowed http ref is the SVG namespace)

Integration (tests/integration.test.ts): full mock pipeline, recorder layout, determinism, accuracy bounds.

Definition of Done

pnpm typecheck && pnpm lint && pnpm test && pnpm build all exit 0; runtime changes verified by real (mock) execution end-to-end. Real model (cc/codex) call paths are implemented and type-checked but not exercised with billable calls in CI.