TypeScript
Architecture
Build/dependency order (= module layering):
models → config/pricing → combos/subset → runners → transcripts →
metrics → flow → recorder → report → orchestrator → cli/tui
Cross-cutting: i18n, harness, preflight, benches, judge.
Data flow (one compare)
cli.compareAction
→ combos.expand(models,agents,harnesses) → {valid, excluded}
→ preflight.analyze (warnings + real-run gate)
→ orchestrator.execute
→ subset.select(Subset|Balanced) # deterministic (sha256)
→ for each (combo × task × run), concurrency-bounded, fail-soft:
runners.runnerFor(agent, runner, {bench}) # auto dispatch / mock fallback
runner.run(inv) → RawRunArtifact # NEVER throws; always writes output+transcript
transcripts.parserFor(agent).parse(...) # tokens, subagentCount (★), tools, flow events
pricing.resolveCost(...) # actual > estimated > none
judge?.score(...) # opt-in
flow.buildFlow(...) → flow.json
recorder.appendResult(RunResult)
→ metrics.aggregateAll → accuracy + perspectives
→ recorder.writeMeta / writeAccuracy / finalizeSqlite(optional)
→ report.renderReport (report.html) + flow.renderGraphHtml (graph.html)
Key types (models.ts)
Combo, RunResult, TranscriptEvent, ParsedTranscript, FlowNode/Edge/Graph,
RunInvocation, RawRunArtifact, RunOptions. zod schemas guard every external input
(headless JSON, transcript lines, persisted results).
Failure model
- Runners are fail-soft: task failure/timeout/IO error →
success:false+ reason, never an exception. - The orchestrator additionally wraps
run()defensively and records a synthetic fail-soft result. success(ran to completion) andresolved(hidden verification passed) are independent axes.