Skip to content
TypeScript
README
theseus — compare model × agent × harness, and judge the trade-offs

theseus

Compare model × agent × harness on your own tasks — then judge the trade-offs yourself.

CI Pages node license

🌐 English · 한국어 · 日本語 · 中文


What it does

theseus runs the same task subset across model (L0) × agent (L1) × harness (L2) combinations and lays out, side by side, each combination's accuracy · cost · time · tool calls · subagent call flow — so you understand the flow and pick the trade‑off that fits you. It is not a leaderboard and produces no single score by default; it recommends per‑perspective winners and marks the cost/accuracy Pareto frontier.

flowchart LR
  S(["<b>select</b><br/>model × agent × harness"]):::pick
  S --> SUB["<b>subset</b><br/>deterministic · seed"]:::step
  SUB --> J["<b>run jobs</b><br/>per combo × task · fail-soft"]:::step
  J --> RM["<b>results + metrics</b><br/>accuracy · cost · time · flow"]:::step
  RM --> V(["<b>visualize</b><br/>report · graph · TUI · view"]):::out
  V -. "refine the selection — run again" .-> S

  classDef pick fill:#cdd2d8,stroke:#0a0b0d,color:#0a0b0d;
  classDef step fill:#15171b,stroke:#8b929b,color:#eef0f2;
  classDef out fill:#1d2b33,stroke:#cdd2d8,color:#eef0f2;

Every run emits: structured data (results.jsonl + optional theseus.sqlite), a self‑contained report.html (no external JS/CDN), and a subagent flow graph graph.html.

The name — a ship, and a judgment

theseus is the Ship of Theseus: replace plank after plank and the ship still endures. Your workflow is the ship; the model, agent, and harness are planks you swap. theseus shows you which planks actually change the voyage.

And its purpose is judgment — it doesn't crown a winner, it hands you the evidence (accuracy, cost, time, flow) so you weigh the trade‑offs and pass your own verdict. That spirit traces to Díkē (Δίκη), the Greek embodiment of justice and judgment; theseus is the instrument, your call is the judgment.

Live demo gallery

A self‑contained example report + flow graph, regenerated by CI on every push to main.

  • Hosted: https://olympusresearch.github.io/theseus/ (after a maintainer enables Pages — see below)
  • Open it locally (no network needed):
    npm run build:site            # builds ./site (index.html + report.html + graph.html)
    open site/index.html          # macOS · Linux: xdg-open · Windows: start
    
  • From any of your own runs:
    # write a run and pop its report in your browser
    theseus --model opus4.8 --agent cc --harness none,omc --bench subagent --runner mock --view open
    # write a run and serve all runs at http://localhost:7777
    theseus --model opus4.8 --agent cc --harness none,omc --bench subagent --runner mock --view serve --port 7777
    
  • Visualize every trace you've recorded (the whole pipeline's history, no new run):
    theseus view --open            # serve all recorded runs at localhost and open the gallery
    

Enabling the hosted page: repo Settings → Pages → Source: GitHub Actions (the Pages workflow does the rest).

Install

pnpm install        # or npm install
pnpm build          # → dist/cli.js
npm link            # exposes `theseus`  (or run: node dist/cli.js …)

Node ≥ 20, ESM. better-sqlite3, ink, and react are optional — if a native build or the TUI deps are missing, theseus degrades gracefully (JSONL stays the source of truth; the TUI falls back to a table).

Quick start (offline, deterministic)

theseus --model opus4.8,gpt5.5 --agent cc,opencode,codex \
        --harness none,omc,omo,ponytail --bench subagent --tasks 4 --runner mock -L en

The matrix

L0 models

  • claude family — opus4.8 / 4.7 / 4.6, sonnet4.6 / 4.5, haiku4.5, fable5
  • gpt family — gpt5-codex (account default), gpt5.5, gpt5.1-codex
  • open family (OpenRouter) — qwen3-coder, qwen3-coder-free (free budget), glm4.6, kimi-k2, deepseek-v3.2 … passed as openrouter/<vendor>/<model> by the model-agnostic agents below. Add more by dropping any OpenRouter id into src/config.ts; :free models are billed at $0.

L1 agents

agent binary model families harnesses
cc claude claude none, superpowers, omc, ecc, ponytail, mattpocock
codex codex gpt none, superpowers, omx, lazycodex, omo, ponytail
opencode opencode claude + gpt none, omo, ponytail, mattpocock
openclaw openclaw claude + gpt none, ponytail
hermes hermes claude + gpt none, ponytail

opencode/openclaw/hermes are model‑agnostic: they accept any family, so theseus passes the model as provider/id — anthropic/…, openai/…, or openrouter/<vendor>/<model> (e.g. a free Qwen). default is omitted. (cc stays claude-only, codex gpt-only.)

L2 harnesses

harness type agents
none baseline all
superpowers shared cc, codex
omc / ecc orchestration cc
omx / lazycodex orchestration codex
omo (oh‑my‑openagent) orchestration opencode, codex
ponytail minimalism rule‑pack all
mattpocock skills pack cc, codex, opencode

Incompatible combos are excluded with a reason (never silently dropped).

Interactive session

Type theseus with no arguments and an interactive session opens — like claude or codex. Describe your goal in plain text, then configure the run with one unified slash command per feature, and /run:

$ theseus
theseus [1m×1a×1h] ▸ compare cc vs opencode on a flaky bug
theseus [1m×1a×1h] ▸ /agents cc,opencode
theseus [2m×1a×1h] ▸ /models opus4.8,gpt5.5
theseus [2m×2a×1h] ▸ /runner mock
theseus [2m×2a×1h] ▸ /run                      # → decision table + report.html
theseus [2m×2a×1h] ▸ /view                     # → serve every recorded trace at localhost
theseus [2m×2a×1h] ▸ /exit
command what it does
plain text · /goal <text> what you want to achieve — accumulated from plain text, recorded in the run meta, shown on the report
/models /agents /harness /bench the L0×L1×L2 matrix — bare command lists, with args selects
/runs /tasks /seed /runner /judge /score /lang sweep parameters
/run run the comparison with the current selection
/view [open] serve every recorded trace at localhost
/config /clear /help /exit inspect · reset · help · quit

Start it explicitly with theseus session (aliases: chat, repl). The same actions are available as one-shot flags for scripting — see below.

Commands

theseus                                              # interactive session (slash commands)
theseus session | chat | repl                        # same, explicit
theseus [compare] --model … --agent … --harness …   # default command
theseus verify  --model … --agent … --harness …     # per-combo subagent-call capture check
theseus results [--run <id>]                         # show a recorded run
theseus view [--open] [--port 7777]                  # serve & visualize EVERY recorded trace
theseus tui [--out runs] [--open] [--rich]           # interactive ink UI (table fallback)
theseus install <cc|codex|opencode|openclaw|hermes|tb|<harness>|all>
theseus install --check                              # status table
theseus doctor                                       # environment self-diagnosis

Visualization modes — --view

mode what it does
inline (default) write self‑contained report.html + graph.html, print paths
open also pop the report open in your browser
serve start a localhost server (--port, default 7777) with a live index of all runs
term render bar charts (accuracy/tokens/cost/subagents) + an ANSI flow tree in the terminal — no browser

Key flags

--bench (default terminal-bench-v2) · --tasks (20) · --runs (1) · --seed · --runner (auto|mock|cc|codex|opencode|openclaw|hermes|tb) · --concurrency (4) · --timeout (600) · --out (runs) · --goal · --judge (none|heuristic|cc) · --score · --balanced · --dry-run · --yes/-y · --view · --port · --no-logo · --language/-L (kr default; en|ja|zh).

Benches

  • terminal-bench-v2 — external Terminal‑Bench (Docker; delegated). Falls back to mock if tb absent.
  • local — built‑in, file‑verified happy‑path tasks (no Docker).
  • subagent — built‑in hard tasks (subagent-101…106); verifiers recompute the answer from the input fixtures (no hardcoded golden → no drift). Each carries one agent‑agnostic delegation hint.

Verifying subagent flow capture

theseus verify runs each valid combo on a delegation‑heavy task and reports whether subagent calls were captured:

combo                  subagents  maxDepth  captured?
cc+opus4.8+omc         4          1         ✓ captured
opencode+opus4.8+omo   3          0         ✓ captured
codex+gpt5.5+none      0          0         — flat (expected)

Tree‑capable combos (cc + orchestration, opencode + omo) must fan out; single‑loop combos (codex, none, hermes) report flat (expected). Defaults to the offline mock runner; real runners need --yes.

Invariants (PRD §20)

Sort key subagentCount → totalTokens → wallTimeMs · no single score (perspectives + Pareto) · determinism (sha256 subset, no Math.random) · fail‑soft (no silent drops; success ≠ resolved) · null = uncollected · runners not reimplemented (mock fallback + banner) · self‑contained HTML · harness per‑run (no global pollution) · recompute‑in‑verify · i18n.

Development

pnpm typecheck && pnpm lint && pnpm test && pnpm build

See docs/ for architecture, registry, transcript schema, visualization, brand, and acceptance.