theseus
Compare model × agent × harness on your own tasks — then judge the trade-offs yourself.
What it does
theseus runs the same task subset across model (L0) × agent (L1) × harness (L2) combinations and lays out, side by side, each combination's accuracy · cost · time · tool calls · subagent call flow — so you understand the flow and pick the trade‑off that fits you. It is not a leaderboard and produces no single score by default; it recommends per‑perspective winners and marks the cost/accuracy Pareto frontier.
flowchart LR
S(["<b>select</b><br/>model × agent × harness"]):::pick
S --> SUB["<b>subset</b><br/>deterministic · seed"]:::step
SUB --> J["<b>run jobs</b><br/>per combo × task · fail-soft"]:::step
J --> RM["<b>results + metrics</b><br/>accuracy · cost · time · flow"]:::step
RM --> V(["<b>visualize</b><br/>report · graph · TUI · view"]):::out
V -. "refine the selection — run again" .-> S
classDef pick fill:#cdd2d8,stroke:#0a0b0d,color:#0a0b0d;
classDef step fill:#15171b,stroke:#8b929b,color:#eef0f2;
classDef out fill:#1d2b33,stroke:#cdd2d8,color:#eef0f2;
Every run emits: structured data (results.jsonl + optional theseus.sqlite), a self‑contained
report.html (no external JS/CDN), and a subagent flow graph graph.html.
The name — a ship, and a judgment
theseus is the Ship of Theseus: replace plank after plank and the ship still endures. Your workflow is the ship; the model, agent, and harness are planks you swap. theseus shows you which planks actually change the voyage.
And its purpose is judgment — it doesn't crown a winner, it hands you the evidence (accuracy, cost, time, flow) so you weigh the trade‑offs and pass your own verdict. That spirit traces to Díkē (Δίκη), the Greek embodiment of justice and judgment; theseus is the instrument, your call is the judgment.
Live demo gallery
A self‑contained example report + flow graph, regenerated by CI on every push to main.
- Hosted: https://olympusresearch.github.io/theseus/ (after a maintainer enables Pages — see below)
- Open it locally (no network needed):
npm run build:site # builds ./site (index.html + report.html + graph.html) open site/index.html # macOS · Linux: xdg-open · Windows: start - From any of your own runs:
# write a run and pop its report in your browser theseus --model opus4.8 --agent cc --harness none,omc --bench subagent --runner mock --view open # write a run and serve all runs at http://localhost:7777 theseus --model opus4.8 --agent cc --harness none,omc --bench subagent --runner mock --view serve --port 7777 - Visualize every trace you've recorded (the whole pipeline's history, no new run):
theseus view --open # serve all recorded runs at localhost and open the gallery
Enabling the hosted page: repo Settings → Pages → Source: GitHub Actions (the
Pagesworkflow does the rest).
Install
pnpm install # or npm install
pnpm build # → dist/cli.js
npm link # exposes `theseus` (or run: node dist/cli.js …)
Node ≥ 20, ESM. better-sqlite3, ink, and react are optional — if a native build or the TUI deps
are missing, theseus degrades gracefully (JSONL stays the source of truth; the TUI falls back to a table).
Quick start (offline, deterministic)
theseus --model opus4.8,gpt5.5 --agent cc,opencode,codex \
--harness none,omc,omo,ponytail --bench subagent --tasks 4 --runner mock -L en
The matrix
L0 models
- claude family —
opus4.8 / 4.7 / 4.6,sonnet4.6 / 4.5,haiku4.5,fable5 - gpt family —
gpt5-codex(account default),gpt5.5,gpt5.1-codex - open family (OpenRouter) —
qwen3-coder,qwen3-coder-free(free budget),glm4.6,kimi-k2,deepseek-v3.2… passed asopenrouter/<vendor>/<model>by the model-agnostic agents below. Add more by dropping any OpenRouter id intosrc/config.ts;:freemodels are billed at $0.
L1 agents
| agent | binary | model families | harnesses |
|---|---|---|---|
cc |
claude | claude | none, superpowers, omc, ecc, ponytail, mattpocock |
codex |
codex | gpt | none, superpowers, omx, lazycodex, omo, ponytail |
opencode |
opencode | claude + gpt | none, omo, ponytail, mattpocock |
openclaw |
openclaw | claude + gpt | none, ponytail |
hermes |
hermes | claude + gpt | none, ponytail |
opencode/openclaw/hermes are model‑agnostic: they accept any family, so theseus passes the model as
provider/id — anthropic/…, openai/…, or openrouter/<vendor>/<model> (e.g. a free Qwen). default is
omitted. (cc stays claude-only, codex gpt-only.)
L2 harnesses
| harness | type | agents |
|---|---|---|
none |
baseline | all |
superpowers |
shared | cc, codex |
omc / ecc |
orchestration | cc |
omx / lazycodex |
orchestration | codex |
omo (oh‑my‑openagent) |
orchestration | opencode, codex |
ponytail |
minimalism rule‑pack | all |
mattpocock |
skills pack | cc, codex, opencode |
Incompatible combos are excluded with a reason (never silently dropped).
Interactive session
Type theseus with no arguments and an interactive session opens — like claude or codex.
Describe your goal in plain text, then configure the run with one unified slash command per feature,
and /run:
$ theseus
theseus [1m×1a×1h] ▸ compare cc vs opencode on a flaky bug
theseus [1m×1a×1h] ▸ /agents cc,opencode
theseus [2m×1a×1h] ▸ /models opus4.8,gpt5.5
theseus [2m×2a×1h] ▸ /runner mock
theseus [2m×2a×1h] ▸ /run # → decision table + report.html
theseus [2m×2a×1h] ▸ /view # → serve every recorded trace at localhost
theseus [2m×2a×1h] ▸ /exit
| command | what it does |
|---|---|
plain text · /goal <text> |
what you want to achieve — accumulated from plain text, recorded in the run meta, shown on the report |
/models /agents /harness /bench |
the L0×L1×L2 matrix — bare command lists, with args selects |
/runs /tasks /seed /runner /judge /score /lang |
sweep parameters |
/run |
run the comparison with the current selection |
/view [open] |
serve every recorded trace at localhost |
/config /clear /help /exit |
inspect · reset · help · quit |
Start it explicitly with theseus session (aliases: chat, repl). The same actions are available as
one-shot flags for scripting — see below.
Commands
theseus # interactive session (slash commands)
theseus session | chat | repl # same, explicit
theseus [compare] --model … --agent … --harness … # default command
theseus verify --model … --agent … --harness … # per-combo subagent-call capture check
theseus results [--run <id>] # show a recorded run
theseus view [--open] [--port 7777] # serve & visualize EVERY recorded trace
theseus tui [--out runs] [--open] [--rich] # interactive ink UI (table fallback)
theseus install <cc|codex|opencode|openclaw|hermes|tb|<harness>|all>
theseus install --check # status table
theseus doctor # environment self-diagnosis
Visualization modes — --view
| mode | what it does |
|---|---|
inline (default) |
write self‑contained report.html + graph.html, print paths |
open |
also pop the report open in your browser |
serve |
start a localhost server (--port, default 7777) with a live index of all runs |
term |
render bar charts (accuracy/tokens/cost/subagents) + an ANSI flow tree in the terminal — no browser |
Key flags
--bench (default terminal-bench-v2) · --tasks (20) · --runs (1) · --seed · --runner
(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · --concurrency (4) · --timeout (600) · --out (runs)
· --goal · --judge (none|heuristic|cc) · --score · --balanced · --dry-run · --yes/-y · --view
· --port · --no-logo · --language/-L (kr default; en|ja|zh).
Benches
- terminal-bench-v2 — external Terminal‑Bench (Docker; delegated). Falls back to mock if
tbabsent. - local — built‑in, file‑verified happy‑path tasks (no Docker).
- subagent — built‑in hard tasks (
subagent-101…106); verifiers recompute the answer from the input fixtures (no hardcoded golden → no drift). Each carries one agent‑agnostic delegation hint.
Verifying subagent flow capture
theseus verify runs each valid combo on a delegation‑heavy task and reports whether subagent calls were
captured:
combo subagents maxDepth captured?
cc+opus4.8+omc 4 1 ✓ captured
opencode+opus4.8+omo 3 0 ✓ captured
codex+gpt5.5+none 0 0 — flat (expected)
Tree‑capable combos (cc + orchestration, opencode + omo) must fan out; single‑loop combos (codex, none,
hermes) report flat (expected). Defaults to the offline mock runner; real runners need --yes.
Invariants (PRD §20)
Sort key subagentCount → totalTokens → wallTimeMs · no single score (perspectives + Pareto) ·
determinism (sha256 subset, no Math.random) · fail‑soft (no silent drops; success ≠ resolved) ·
null = uncollected · runners not reimplemented (mock fallback + banner) · self‑contained HTML ·
harness per‑run (no global pollution) · recompute‑in‑verify · i18n.
Development
pnpm typecheck && pnpm lint && pnpm test && pnpm build
See docs/ for architecture, registry, transcript schema, visualization, brand, and acceptance.