Skip to content
TypeScript
1.9 KB
Raw Blame

Architecture

Build/dependency order (= module layering):

models → config/pricing → combos/subset → runners → transcripts →
metrics → flow → recorder → report → orchestrator → cli/tui

Cross-cutting: i18n, harness, preflight, benches, judge.

Data flow (one compare)

cli.compareAction
  → combos.expand(models,agents,harnesses) → {valid, excluded}
  → preflight.analyze (warnings + real-run gate)
  → orchestrator.execute
      → subset.select(Subset|Balanced)            # deterministic (sha256)
      → for each (combo × task × run), concurrency-bounded, fail-soft:
          runners.runnerFor(agent, runner, {bench})   # auto dispatch / mock fallback
          runner.run(inv) → RawRunArtifact            # NEVER throws; always writes output+transcript
          transcripts.parserFor(agent).parse(...)      # tokens, subagentCount (★), tools, flow events
          pricing.resolveCost(...)                      # actual > estimated > none
          judge?.score(...)                             # opt-in
          flow.buildFlow(...) → flow.json
          recorder.appendResult(RunResult)
      → metrics.aggregateAll → accuracy + perspectives
      → recorder.writeMeta / writeAccuracy / finalizeSqlite(optional)
      → report.renderReport (report.html)  +  flow.renderGraphHtml (graph.html)

Key types (models.ts)

Combo, RunResult, TranscriptEvent, ParsedTranscript, FlowNode/Edge/Graph, RunInvocation, RawRunArtifact, RunOptions. zod schemas guard every external input (headless JSON, transcript lines, persisted results).

Failure model

  • Runners are fail-soft: task failure/timeout/IO error → success:false + reason, never an exception.
  • The orchestrator additionally wraps run() defensively and records a synthetic fail-soft result.
  • success (ran to completion) and resolved (hidden verification passed) are independent axes.