Skip to content
TypeScript

Stage 4: regression tests for the audit fixes, cleanup, hygiene

Claudeb10745bb5bec
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..1e044b7
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,40 @@
+# Contributing to theseus
+
+Thanks for helping compare the planks. A few ground rules keep the ship afloat.
+
+## Setup
+
+```bash
+npm install # Node ≥ 20; better-sqlite3/ink/react are optional deps
+npm run typecheck && npm run lint && npm test && npm run build
+```
+
+Everything runs offline: the deterministic **mock runner** exercises the full pipeline
+(`theseus --runner mock …`), so no API keys or paid calls are needed for development.
+
+## Invariants (do not break)
+
+These are asserted by tests and documented in the PRD (`docs/`):
+
+- **Sort key** `subagentCount → totalTokens → wallTimeMs` — the canonical, reproducible row order.
+- **No single score by default** — perspective winners + Pareto frontier; `--score` is opt-in.
+- **Determinism** — task subsets via sha256; no `Math.random` in selection paths.
+- **Fail-soft** — no silent drops; `success ≠ resolved`; excluded combos carry a reason.
+- **`null` = not collected** — never zero-fill unknown values.
+- **Self-contained HTML** — `report.html`/`graph.html` must have zero `<script>`, `src=`, `<link>`,
+ or CDN references (tests enforce this; interactivity is CSS-only).
+- **Harnesses apply per-run** — never mutate global state (`~/.claude`, `~/.codex`).
+- **i18n** — user-visible strings go through `src/i18n.ts` (en/kr/ja/zh).
+
+## Workflow
+
+1. Branch from `main`; keep PRs focused.
+2. `npm run typecheck && npm run lint && npm test` must pass (CI runs them on Node 20 and 22).
+3. New behavior needs a test — offline (mock runner), no network.
+4. User-visible strings: add all four languages in `src/i18n.ts`.
+5. Update `NOTES.md` with an AS-IS → TO-BE entry for non-trivial changes.
+
+## Reporting bugs
+
+Open an issue with: the command you ran, expected vs actual, and — when relevant — the run
+directory's `meta.json`/`results.jsonl` (they contain no secrets; scrub task content if private).
diff --git a/NOTES.md b/NOTES.md
index 11ee358..0d44daa 100644
--- a/NOTES.md
+++ b/NOTES.md
@@ -2,6 +2,25 @@
비전문 감시자용 기록. 용어 최소화, AS-IS → TO-BE.
+## 2026-07-01 — 전 저장소 감사 → 4단계 개선
+
+3개 병렬 감사(코어 소스 · 테스트/CI/패키징 · 제품/문서) + PR/CI 라이브 점검으로 결함을 찾아 4단계로 수정.
+
+- **1단계(정합성)**: 리포트 best 하이라이트 복구(`--best` CSS 변수 누락) · REPL 이 /run·/view 실패에도
+ 생존(try/catch) · 미지 벤치 → 대안 나열하는 명확한 에러 · **codex+룰 하네스가 조용히 baseline 으로
+ 돌던 버그**(AGENTS.md work_dir materialize 로 실제 적용) · DefensiveParser 툴콜 이중집계(2×) 수정 ·
+ Pages 워크플로 이중 트리거로 PR 이 unstable 되던 것 수정.
+- **2단계(정직성)**: 목표(goal)가 실제로 기록됨(RunOptions→meta.json→리포트 헤더, `--goal` 플래그) ·
+ **제품 i18n 을 README 와 같은 4개 언어로**(ja/zh 전체 번역) · makeRunId 밀리초 충돌 방지 ·
+ costSource 를 보고값에서 도출 · preflight 게이트를 실제 subset 크기로 클램프.
+- **3단계(정리)**: 결과 표 렌더러 3벌 → 1벌(`viz/table.ts`, 결정지표 우선 컬럼 통일) · 세션에
+ /concurrency·/timeout 추가 · README mermaid 4개 알파인 색으로 · `--view term` 무지개 제거 ·
+ 갤러리 원칙 스트립+파비콘 · dead export 4개 제거 · **npm publish 가 빈 패키지를 배포하던 문제**
+ (`prepublishOnly` + license/repository 메타).
+- **4단계(내구성)**: 감사 결함 회귀 테스트 9개(파서·벤치 에러·preflight·extractScore·runId·costSource·
+ codex 룰 적용) · verify 임시폴더 정리 · CONTRIBUTING/SECURITY · 번역 README 에 주요 플래그·벤치 섹션
+ 백포트. 총 78 테스트.
+
## 2026-06-30 (4) — Olympus Research 브랜드 적용(알파인 모노크롬)
**AS-IS**: 비주얼이 "테세우스의 배"(항해/바다 블루·로프 골드) 테마였다.
diff --git a/README.ja.md b/README.ja.md
index c27f367..36325f2 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -176,6 +176,18 @@ theseus doctor # 実行環境の自己診
| `serve` | localhost サーバ(`--port`, 既定 7777)で全 run のライブインデックス |
| `term` | 棒グラフ(正確性/トークン/コスト/サブエージェント)+ ANSI **フローツリー** を端末に — ブラウザ不要 |
+### 主なフラグ
+`--bench`(既定 `terminal-bench-v2`)· `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`
+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)
+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`
+· `--port` · `--no-logo` · `--language/-L`(既定 kr; en|ja|zh).
+
+## ベンチ
+- **terminal-bench-v2** — 外部 Terminal‑Bench(Docker; 委譲実行)。`tb` が無ければ mock にフォールバック。
+- **local** — 内蔵の happy‑path タスク(ファイル検証、Docker 不要)。
+- **subagent** — 内蔵の高難度タスク(`subagent-101…106`)。検証器は **入力 fixture から答えを再計算**
+ (ハードコードされた golden なし → drift なし)。各タスクにエージェント非依存の委譲ヒントを1文含む。
+
## サブエージェントフロー捕捉チェック
`theseus verify` は委譲を促すタスクで各有効組合せを実行し、サブエージェント呼び出しが実際に捕捉されるかを
diff --git a/README.ko.md b/README.ko.md
index ec30fcb..12aee58 100644
--- a/README.ko.md
+++ b/README.ko.md
@@ -175,6 +175,18 @@ theseus doctor # 실행 환경 자가진
| `serve` | localhost 서버(`--port`, 기본 7777)에 모든 run 라이브 인덱스 |
| `term` | 막대 차트(정확도/토큰/비용/서브에이전트) + ANSI **흐름 트리** 를 터미널에 — 브라우저 불필요 |
+### 주요 플래그
+`--bench`(기본 `terminal-bench-v2`) · `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`
+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)
+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`
+· `--port` · `--no-logo` · `--language/-L`(기본 kr; en|ja|zh).
+
+## 벤치
+- **terminal-bench-v2** — 외부 Terminal‑Bench(Docker; 위임 실행). `tb` 가 없으면 mock 폴백.
+- **local** — 내장 happy‑path 과제(파일 검증, Docker 불필요).
+- **subagent** — 내장 고난도 과제(`subagent-101…106`); 검증기가 **입력 fixture 에서 답을 재계산**
+ (하드코딩 golden 없음 → drift 없음). 각 과제에 에이전트 불문 위임 힌트 1문장 포함.
+
## 서브에이전트 흐름 포착 점검
`theseus verify` 는 위임 유도 과제로 각 유효 조합을 돌려, 서브에이전트 콜이 실제로 잡히는지 표로 보고한다.
diff --git a/README.md b/README.md
index e54f724..3279505 100644
--- a/README.md
+++ b/README.md
@@ -185,8 +185,8 @@ theseus doctor # environment self-diagnosi
### Key flags
`--bench` (default `terminal-bench-v2`) · `--tasks` (20) · `--runs` (1) · `--seed` · `--runner`
(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency` (4) · `--timeout` (600) · `--out` (runs)
-· `--judge` (none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view` · `--port`
-· `--no-logo` · `--language/-L` (kr default; en).
+· `--goal` · `--judge` (none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`
+· `--port` · `--no-logo` · `--language/-L` (kr default; en|ja|zh).
## Benches
- **terminal-bench-v2** — external Terminal‑Bench (Docker; delegated). Falls back to mock if `tb` absent.
diff --git a/README.zh.md b/README.zh.md
index ae5310a..30336ea 100644
--- a/README.zh.md
+++ b/README.zh.md
@@ -173,6 +173,18 @@ theseus doctor # 运行环境自检
| `serve` | 启动 localhost 服务(`--port`,默认 7777),含全部运行的实时索引 |
| `term` | 在终端渲染柱状图(准确率/令牌/成本/子智能体)+ ANSI **流程树** —— 无需浏览器 |
+### 主要标志
+`--bench`(默认 `terminal-bench-v2`)· `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`
+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)
+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`
+· `--port` · `--no-logo` · `--language/-L`(默认 kr;en|ja|zh)。
+
+## 基准(Bench)
+- **terminal-bench-v2** — 外部 Terminal‑Bench(Docker;委托执行)。缺少 `tb` 时回退到 mock。
+- **local** — 内置 happy‑path 任务(文件校验,无需 Docker)。
+- **subagent** — 内置高难度任务(`subagent-101…106`);校验器**从输入 fixture 重新计算答案**
+ (无硬编码 golden → 无漂移)。每个任务含一句与代理无关的委托提示。
+
## 子智能体流程捕获检查
`theseus verify` 用强制委派的任务运行每个有效组合,并用表格报告子智能体调用是否被真正捕获。
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 0000000..22faacc
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,22 @@
+# Security Policy
+
+## Supported versions
+
+`theseus` is pre-1.0; only the latest release (and `main`) receive fixes.
+
+## Reporting a vulnerability
+
+Please **do not** open a public issue for security reports. Use GitHub's
+[private vulnerability reporting](https://github.com/OlympusResearch/theseus/security/advisories/new)
+on this repository. You can expect an acknowledgement within a week.
+
+## Scope notes
+
+- theseus shells out to locally installed agent CLIs (`claude`, `codex`, `opencode`, `openclaw`,
+ `hermes`) and runs them against **your** tasks in per-run work directories. It never sends your
+ code anywhere itself — network egress happens only inside the agent CLIs you invoke.
+- Generated `report.html`/`graph.html` are self-contained by design (no `<script>`, no external
+ fetches) and all embedded strings are HTML-escaped. A report that violates this is a bug —
+ please report it.
+- `theseus view` / `--view serve` bind a local static server that refuses path traversal outside
+ the runs directory. It binds localhost usage patterns; do not expose it to untrusted networks.
diff --git a/src/verify.ts b/src/verify.ts
index b139ebe..002805d 100644
--- a/src/verify.ts
+++ b/src/verify.ts
@@ -5,7 +5,7 @@
* 기대: 트리형 조합(cc+오케스트레이션, opencode+omo)은 fan-out 필요 / 단일 루프(codex·none·hermes)는 flat 정상.
* mock(오프라인·결정적, 기본) 또는 real(--yes 게이트). 절대 throw 금지.
*/
-import { mkdtempSync } from "node:fs";
+import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expand, parseComboId } from "./combos";
@@ -82,6 +82,9 @@ export async function verifyCombos(opts: VerifyOptions): Promise<VerifyRow[]> {
);
} catch {
// fail-soft: 누적 최댓값 유지
+ } finally {
+ // 시도당 임시 폴더 정리(attempts × combos 만큼 누적되는 것 방지)
+ rmSync(workDir, { recursive: true, force: true });
}
}
const expectFanout = expectsFanout(agent, harness);
diff --git a/tests/audit_fixes.test.ts b/tests/audit_fixes.test.ts
new file mode 100644
index 0000000..764f005
--- /dev/null
+++ b/tests/audit_fixes.test.ts
@@ -0,0 +1,173 @@
+/**
+ * 감사(audit)에서 확인된 결함들의 회귀 테스트.
+ * 파서 이중집계 · 미지 벤치 에러 · preflight 클램프/경고 · extractScore ·
+ * makeRunId 충돌 · costSource 일관성 · codex 룰 하네스 materialize.
+ */
+import { existsSync, mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { taskIds } from "../src/benches/index";
+import { expand } from "../src/combos";
+import { applyHarnessToWorkdir, ensureHarnessRulesStub } from "../src/harness";
+import { extractScore } from "../src/judge";
+import { aggregateCombo } from "../src/metrics";
+import type { RunResult } from "../src/models";
+import { execute } from "../src/orchestrator";
+import { analyze } from "../src/preflight";
+import { makeRunId } from "../src/recorder";
+import { DefensiveParser } from "../src/transcripts/defensive";
+
+function tmp(): string {
+ return mkdtempSync(join(tmpdir(), "theseus-audit-"));
+}
+
+describe("DefensiveParser — no double counting of wrapper+item events", () => {
+ it("counts one tool call per logical event; nested subagent marker still registers", () => {
+ const f = join(tmp(), "t.jsonl");
+ writeFileSync(
+ f,
+ [
+ JSON.stringify({ type: "item.completed", item: { type: "command_execution" } }),
+ JSON.stringify({ type: "item.completed", item: { type: "tool_use", name: "read_file" } }),
+ JSON.stringify({ type: "item.completed", item: { type: "subagent", name: "task" } }),
+ JSON.stringify({ type: "message", usage: { input_tokens: 100, output_tokens: 50 } }),
+ ].join("\n"),
+ );
+ const p = new DefensiveParser("opencode").parse(f);
+ expect(p.toolCalls).toBe(3); // 이전 버그: 6 (래퍼+item 각각 집계)
+ expect(p.subagentCount).toBe(1);
+ expect(p.totalTokens).toBe(150);
+ });
+});
+
+describe("orchestrator — unknown bench raises one clear error", () => {
+ it("names the bench and lists the available ones", async () => {
+ await expect(
+ execute({
+ models: ["opus4.8"],
+ agents: ["cc"],
+ harnesses: ["none"],
+ bench: "definitely-not-a-bench",
+ tasks: 2,
+ runs: 1,
+ seed: 0,
+ runner: "mock",
+ concurrency: 1,
+ timeoutS: 5,
+ outDir: tmp(),
+ judge: "none",
+ score: false,
+ balanced: false,
+ language: "en",
+ }),
+ ).rejects.toThrow(/unknown bench 'definitely-not-a-bench'.*subagent/);
+ });
+});
+
+describe("preflight — scale clamps to catalog, 0 tasks warns", () => {
+ const combos = () => expand(["opus4.8"], ["cc"], ["none"]);
+
+ it("clamps requested tasks to the local-bench catalog size", () => {
+ const { valid, excluded } = combos();
+ const pf = analyze(["opus4.8"], ["cc"], ["none"], valid, excluded, {
+ bench: "subagent",
+ runner: "mock",
+ tasks: 9999,
+ runs: 1,
+ });
+ expect(pf.scale).toBe(valid.length * taskIds("subagent").length);
+ });
+
+ it("warns when 0 tasks are selected", () => {
+ const { valid, excluded } = combos();
+ const pf = analyze(["opus4.8"], ["cc"], ["none"], valid, excluded, {
+ bench: "subagent",
+ runner: "mock",
+ tasks: 0,
+ runs: 3,
+ });
+ expect(pf.scale).toBe(0);
+ expect(pf.warnings.join(" ")).toMatch(/0 tasks/);
+ });
+});
+
+describe("judge.extractScore", () => {
+ it("extracts integers, decimals, and 10; clamps and rejects garbage", () => {
+ expect(extractScore("7")).toBe(7);
+ expect(extractScore("Score: 8.5 / 10")).toBe(8.5);
+ expect(extractScore("10")).toBe(10);
+ expect(extractScore("no digits here")).toBeNull();
+ expect(extractScore("")).toBeNull();
+ });
+});
+
+describe("recorder.makeRunId — same-millisecond collision", () => {
+ it("suffixes duplicate ids instead of overwriting", () => {
+ const d = new Date(1750000000000);
+ const a = makeRunId(d);
+ const b = makeRunId(d);
+ const c = makeRunId(d);
+ expect(a).not.toBe(b);
+ expect(b).not.toBe(c);
+ expect(b).toBe(`${a}-2`);
+ expect(c).toBe(`${a}-3`);
+ });
+});
+
+describe("metrics.aggregateCost — costSource derived from reported values", () => {
+ const run = (over: Partial<RunResult>): RunResult =>
+ ({
+ comboId: "cc+opus4.8+none",
+ taskId: "t",
+ runIdx: 0,
+ success: true,
+ resolved: true,
+ subagentCount: 0,
+ totalTokens: 10,
+ inputTokens: 5,
+ outputTokens: 5,
+ cacheReadTokens: 0,
+ cacheCreationTokens: 0,
+ costUsd: null,
+ estimatedCostUsd: null,
+ costSource: "none",
+ wallTimeMs: 10,
+ toolCalls: 0,
+ skillFires: 0,
+ mcpCalls: 0,
+ numTurns: 1,
+ qualityScore: null,
+ error: null,
+ transcriptPath: "",
+ resultPath: "",
+ ...over,
+ }) as RunResult;
+
+ it("a run *claiming* actual with a null cost no longer yields costSource=actual", () => {
+ const m = aggregateCombo("cc+opus4.8+none", [
+ run({ costSource: "actual", costUsd: null }),
+ run({ costSource: "none" }),
+ ]);
+ expect(m.costUsd).toBeNull();
+ expect(m.costSource).toBe("none");
+ });
+
+ it("actual values present → actual; only estimates → estimated", () => {
+ expect(aggregateCombo("c", [run({ costUsd: 0.5, costSource: "actual" })]).costSource).toBe(
+ "actual",
+ );
+ expect(
+ aggregateCombo("c", [run({ estimatedCostUsd: 0.2, costSource: "estimated" })]).costSource,
+ ).toBe("estimated");
+ });
+});
+
+describe("codex rules harnesses — materialized into the workdir (no silent baseline)", () => {
+ it("superpowers rules cache applies to a workdir via AGENTS.md", () => {
+ ensureHarnessRulesStub("superpowers", "# superpowers rules (test stub)\n");
+ const work = tmp();
+ expect(applyHarnessToWorkdir("superpowers", work)).toBe(true);
+ expect(existsSync(join(work, "AGENTS.md"))).toBe(true);
+ });
+});