TypeScript
Stage 4: regression tests for the audit fixes, cleanup, hygiene
Claudeb10745bb5bec
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdnew file mode 100644index 0000000..1e044b7--- /dev/null+++ b/CONTRIBUTING.md@@ -0,0 +1,40 @@+# Contributing to theseus++Thanks for helping compare the planks. A few ground rules keep the ship afloat.++## Setup++```bash+npm install # Node ≥ 20; better-sqlite3/ink/react are optional deps+npm run typecheck && npm run lint && npm test && npm run build+```++Everything runs offline: the deterministic **mock runner** exercises the full pipeline+(`theseus --runner mock …`), so no API keys or paid calls are needed for development.++## Invariants (do not break)++These are asserted by tests and documented in the PRD (`docs/`):++- **Sort key** `subagentCount → totalTokens → wallTimeMs` — the canonical, reproducible row order.+- **No single score by default** — perspective winners + Pareto frontier; `--score` is opt-in.+- **Determinism** — task subsets via sha256; no `Math.random` in selection paths.+- **Fail-soft** — no silent drops; `success ≠ resolved`; excluded combos carry a reason.+- **`null` = not collected** — never zero-fill unknown values.+- **Self-contained HTML** — `report.html`/`graph.html` must have zero `<script>`, `src=`, `<link>`,+ or CDN references (tests enforce this; interactivity is CSS-only).+- **Harnesses apply per-run** — never mutate global state (`~/.claude`, `~/.codex`).+- **i18n** — user-visible strings go through `src/i18n.ts` (en/kr/ja/zh).++## Workflow++1. Branch from `main`; keep PRs focused.+2. `npm run typecheck && npm run lint && npm test` must pass (CI runs them on Node 20 and 22).+3. New behavior needs a test — offline (mock runner), no network.+4. User-visible strings: add all four languages in `src/i18n.ts`.+5. Update `NOTES.md` with an AS-IS → TO-BE entry for non-trivial changes.++## Reporting bugs++Open an issue with: the command you ran, expected vs actual, and — when relevant — the run+directory's `meta.json`/`results.jsonl` (they contain no secrets; scrub task content if private).diff --git a/NOTES.md b/NOTES.mdindex 11ee358..0d44daa 100644--- a/NOTES.md+++ b/NOTES.md@@ -2,6 +2,25 @@비전문 감시자용 기록. 용어 최소화, AS-IS → TO-BE.+## 2026-07-01 — 전 저장소 감사 → 4단계 개선++3개 병렬 감사(코어 소스 · 테스트/CI/패키징 · 제품/문서) + PR/CI 라이브 점검으로 결함을 찾아 4단계로 수정.++- **1단계(정합성)**: 리포트 best 하이라이트 복구(`--best` CSS 변수 누락) · REPL 이 /run·/view 실패에도+ 생존(try/catch) · 미지 벤치 → 대안 나열하는 명확한 에러 · **codex+룰 하네스가 조용히 baseline 으로+ 돌던 버그**(AGENTS.md work_dir materialize 로 실제 적용) · DefensiveParser 툴콜 이중집계(2×) 수정 ·+ Pages 워크플로 이중 트리거로 PR 이 unstable 되던 것 수정.+- **2단계(정직성)**: 목표(goal)가 실제로 기록됨(RunOptions→meta.json→리포트 헤더, `--goal` 플래그) ·+ **제품 i18n 을 README 와 같은 4개 언어로**(ja/zh 전체 번역) · makeRunId 밀리초 충돌 방지 ·+ costSource 를 보고값에서 도출 · preflight 게이트를 실제 subset 크기로 클램프.+- **3단계(정리)**: 결과 표 렌더러 3벌 → 1벌(`viz/table.ts`, 결정지표 우선 컬럼 통일) · 세션에+ /concurrency·/timeout 추가 · README mermaid 4개 알파인 색으로 · `--view term` 무지개 제거 ·+ 갤러리 원칙 스트립+파비콘 · dead export 4개 제거 · **npm publish 가 빈 패키지를 배포하던 문제**+ (`prepublishOnly` + license/repository 메타).+- **4단계(내구성)**: 감사 결함 회귀 테스트 9개(파서·벤치 에러·preflight·extractScore·runId·costSource·+ codex 룰 적용) · verify 임시폴더 정리 · CONTRIBUTING/SECURITY · 번역 README 에 주요 플래그·벤치 섹션+ 백포트. 총 78 테스트.+## 2026-06-30 (4) — Olympus Research 브랜드 적용(알파인 모노크롬)**AS-IS**: 비주얼이 "테세우스의 배"(항해/바다 블루·로프 골드) 테마였다.diff --git a/README.ja.md b/README.ja.mdindex c27f367..36325f2 100644--- a/README.ja.md+++ b/README.ja.md@@ -176,6 +176,18 @@ theseus doctor # 実行環境の自己診| `serve` | localhost サーバ(`--port`, 既定 7777)で全 run のライブインデックス || `term` | 棒グラフ(正確性/トークン/コスト/サブエージェント)+ ANSI **フローツリー** を端末に — ブラウザ不要 |+### 主なフラグ+`--bench`(既定 `terminal-bench-v2`)· `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`+· `--port` · `--no-logo` · `--language/-L`(既定 kr; en|ja|zh).++## ベンチ+- **terminal-bench-v2** — 外部 Terminal‑Bench(Docker; 委譲実行)。`tb` が無ければ mock にフォールバック。+- **local** — 内蔵の happy‑path タスク(ファイル検証、Docker 不要)。+- **subagent** — 内蔵の高難度タスク(`subagent-101…106`)。検証器は **入力 fixture から答えを再計算**+ (ハードコードされた golden なし → drift なし)。各タスクにエージェント非依存の委譲ヒントを1文含む。+## サブエージェントフロー捕捉チェック`theseus verify` は委譲を促すタスクで各有効組合せを実行し、サブエージェント呼び出しが実際に捕捉されるかをdiff --git a/README.ko.md b/README.ko.mdindex ec30fcb..12aee58 100644--- a/README.ko.md+++ b/README.ko.md@@ -175,6 +175,18 @@ theseus doctor # 실행 환경 자가진| `serve` | localhost 서버(`--port`, 기본 7777)에 모든 run 라이브 인덱스 || `term` | 막대 차트(정확도/토큰/비용/서브에이전트) + ANSI **흐름 트리** 를 터미널에 — 브라우저 불필요 |+### 주요 플래그+`--bench`(기본 `terminal-bench-v2`) · `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`+· `--port` · `--no-logo` · `--language/-L`(기본 kr; en|ja|zh).++## 벤치+- **terminal-bench-v2** — 외부 Terminal‑Bench(Docker; 위임 실행). `tb` 가 없으면 mock 폴백.+- **local** — 내장 happy‑path 과제(파일 검증, Docker 불필요).+- **subagent** — 내장 고난도 과제(`subagent-101…106`); 검증기가 **입력 fixture 에서 답을 재계산**+ (하드코딩 golden 없음 → drift 없음). 각 과제에 에이전트 불문 위임 힌트 1문장 포함.+## 서브에이전트 흐름 포착 점검`theseus verify` 는 위임 유도 과제로 각 유효 조합을 돌려, 서브에이전트 콜이 실제로 잡히는지 표로 보고한다.diff --git a/README.md b/README.mdindex e54f724..3279505 100644--- a/README.md+++ b/README.md@@ -185,8 +185,8 @@ theseus doctor # environment self-diagnosi### Key flags`--bench` (default `terminal-bench-v2`) · `--tasks` (20) · `--runs` (1) · `--seed` · `--runner`(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency` (4) · `--timeout` (600) · `--out` (runs)-· `--judge` (none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view` · `--port`-· `--no-logo` · `--language/-L` (kr default; en).+· `--goal` · `--judge` (none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`+· `--port` · `--no-logo` · `--language/-L` (kr default; en|ja|zh).## Benches- **terminal-bench-v2** — external Terminal‑Bench (Docker; delegated). Falls back to mock if `tb` absent.diff --git a/README.zh.md b/README.zh.mdindex ae5310a..30336ea 100644--- a/README.zh.md+++ b/README.zh.md@@ -173,6 +173,18 @@ theseus doctor # 运行环境自检| `serve` | 启动 localhost 服务(`--port`,默认 7777),含全部运行的实时索引 || `term` | 在终端渲染柱状图(准确率/令牌/成本/子智能体)+ ANSI **流程树** —— 无需浏览器 |+### 主要标志+`--bench`(默认 `terminal-bench-v2`)· `--tasks`(20) · `--runs`(1) · `--seed` · `--runner`+(auto|mock|cc|codex|opencode|openclaw|hermes|tb) · `--concurrency`(4) · `--timeout`(600) · `--out`(runs)+· `--goal` · `--judge`(none|heuristic|cc) · `--score` · `--balanced` · `--dry-run` · `--yes/-y` · `--view`+· `--port` · `--no-logo` · `--language/-L`(默认 kr;en|ja|zh)。++## 基准(Bench)+- **terminal-bench-v2** — 外部 Terminal‑Bench(Docker;委托执行)。缺少 `tb` 时回退到 mock。+- **local** — 内置 happy‑path 任务(文件校验,无需 Docker)。+- **subagent** — 内置高难度任务(`subagent-101…106`);校验器**从输入 fixture 重新计算答案**+ (无硬编码 golden → 无漂移)。每个任务含一句与代理无关的委托提示。+## 子智能体流程捕获检查`theseus verify` 用强制委派的任务运行每个有效组合,并用表格报告子智能体调用是否被真正捕获。diff --git a/SECURITY.md b/SECURITY.mdnew file mode 100644index 0000000..22faacc--- /dev/null+++ b/SECURITY.md@@ -0,0 +1,22 @@+# Security Policy++## Supported versions++`theseus` is pre-1.0; only the latest release (and `main`) receive fixes.++## Reporting a vulnerability++Please **do not** open a public issue for security reports. Use GitHub's+[private vulnerability reporting](https://github.com/OlympusResearch/theseus/security/advisories/new)+on this repository. You can expect an acknowledgement within a week.++## Scope notes++- theseus shells out to locally installed agent CLIs (`claude`, `codex`, `opencode`, `openclaw`,+ `hermes`) and runs them against **your** tasks in per-run work directories. It never sends your+ code anywhere itself — network egress happens only inside the agent CLIs you invoke.+- Generated `report.html`/`graph.html` are self-contained by design (no `<script>`, no external+ fetches) and all embedded strings are HTML-escaped. A report that violates this is a bug —+ please report it.+- `theseus view` / `--view serve` bind a local static server that refuses path traversal outside+ the runs directory. It binds localhost usage patterns; do not expose it to untrusted networks.diff --git a/src/verify.ts b/src/verify.tsindex b139ebe..002805d 100644--- a/src/verify.ts+++ b/src/verify.ts@@ -5,7 +5,7 @@* 기대: 트리형 조합(cc+오케스트레이션, opencode+omo)은 fan-out 필요 / 단일 루프(codex·none·hermes)는 flat 정상.* mock(오프라인·결정적, 기본) 또는 real(--yes 게이트). 절대 throw 금지.*/-import { mkdtempSync } from "node:fs";+import { mkdtempSync, rmSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";import { expand, parseComboId } from "./combos";@@ -82,6 +82,9 @@ export async function verifyCombos(opts: VerifyOptions): Promise<VerifyRow[]> {);} catch {// fail-soft: 누적 최댓값 유지+ } finally {+ // 시도당 임시 폴더 정리(attempts × combos 만큼 누적되는 것 방지)+ rmSync(workDir, { recursive: true, force: true });}}const expectFanout = expectsFanout(agent, harness);diff --git a/tests/audit_fixes.test.ts b/tests/audit_fixes.test.tsnew file mode 100644index 0000000..764f005--- /dev/null+++ b/tests/audit_fixes.test.ts@@ -0,0 +1,173 @@+/**+ * 감사(audit)에서 확인된 결함들의 회귀 테스트.+ * 파서 이중집계 · 미지 벤치 에러 · preflight 클램프/경고 · extractScore ·+ * makeRunId 충돌 · costSource 일관성 · codex 룰 하네스 materialize.+ */+import { existsSync, mkdtempSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { describe, expect, it } from "vitest";+import { taskIds } from "../src/benches/index";+import { expand } from "../src/combos";+import { applyHarnessToWorkdir, ensureHarnessRulesStub } from "../src/harness";+import { extractScore } from "../src/judge";+import { aggregateCombo } from "../src/metrics";+import type { RunResult } from "../src/models";+import { execute } from "../src/orchestrator";+import { analyze } from "../src/preflight";+import { makeRunId } from "../src/recorder";+import { DefensiveParser } from "../src/transcripts/defensive";++function tmp(): string {+ return mkdtempSync(join(tmpdir(), "theseus-audit-"));+}++describe("DefensiveParser — no double counting of wrapper+item events", () => {+ it("counts one tool call per logical event; nested subagent marker still registers", () => {+ const f = join(tmp(), "t.jsonl");+ writeFileSync(+ f,+ [+ JSON.stringify({ type: "item.completed", item: { type: "command_execution" } }),+ JSON.stringify({ type: "item.completed", item: { type: "tool_use", name: "read_file" } }),+ JSON.stringify({ type: "item.completed", item: { type: "subagent", name: "task" } }),+ JSON.stringify({ type: "message", usage: { input_tokens: 100, output_tokens: 50 } }),+ ].join("\n"),+ );+ const p = new DefensiveParser("opencode").parse(f);+ expect(p.toolCalls).toBe(3); // 이전 버그: 6 (래퍼+item 각각 집계)+ expect(p.subagentCount).toBe(1);+ expect(p.totalTokens).toBe(150);+ });+});++describe("orchestrator — unknown bench raises one clear error", () => {+ it("names the bench and lists the available ones", async () => {+ await expect(+ execute({+ models: ["opus4.8"],+ agents: ["cc"],+ harnesses: ["none"],+ bench: "definitely-not-a-bench",+ tasks: 2,+ runs: 1,+ seed: 0,+ runner: "mock",+ concurrency: 1,+ timeoutS: 5,+ outDir: tmp(),+ judge: "none",+ score: false,+ balanced: false,+ language: "en",+ }),+ ).rejects.toThrow(/unknown bench 'definitely-not-a-bench'.*subagent/);+ });+});++describe("preflight — scale clamps to catalog, 0 tasks warns", () => {+ const combos = () => expand(["opus4.8"], ["cc"], ["none"]);++ it("clamps requested tasks to the local-bench catalog size", () => {+ const { valid, excluded } = combos();+ const pf = analyze(["opus4.8"], ["cc"], ["none"], valid, excluded, {+ bench: "subagent",+ runner: "mock",+ tasks: 9999,+ runs: 1,+ });+ expect(pf.scale).toBe(valid.length * taskIds("subagent").length);+ });++ it("warns when 0 tasks are selected", () => {+ const { valid, excluded } = combos();+ const pf = analyze(["opus4.8"], ["cc"], ["none"], valid, excluded, {+ bench: "subagent",+ runner: "mock",+ tasks: 0,+ runs: 3,+ });+ expect(pf.scale).toBe(0);+ expect(pf.warnings.join(" ")).toMatch(/0 tasks/);+ });+});++describe("judge.extractScore", () => {+ it("extracts integers, decimals, and 10; clamps and rejects garbage", () => {+ expect(extractScore("7")).toBe(7);+ expect(extractScore("Score: 8.5 / 10")).toBe(8.5);+ expect(extractScore("10")).toBe(10);+ expect(extractScore("no digits here")).toBeNull();+ expect(extractScore("")).toBeNull();+ });+});++describe("recorder.makeRunId — same-millisecond collision", () => {+ it("suffixes duplicate ids instead of overwriting", () => {+ const d = new Date(1750000000000);+ const a = makeRunId(d);+ const b = makeRunId(d);+ const c = makeRunId(d);+ expect(a).not.toBe(b);+ expect(b).not.toBe(c);+ expect(b).toBe(`${a}-2`);+ expect(c).toBe(`${a}-3`);+ });+});++describe("metrics.aggregateCost — costSource derived from reported values", () => {+ const run = (over: Partial<RunResult>): RunResult =>+ ({+ comboId: "cc+opus4.8+none",+ taskId: "t",+ runIdx: 0,+ success: true,+ resolved: true,+ subagentCount: 0,+ totalTokens: 10,+ inputTokens: 5,+ outputTokens: 5,+ cacheReadTokens: 0,+ cacheCreationTokens: 0,+ costUsd: null,+ estimatedCostUsd: null,+ costSource: "none",+ wallTimeMs: 10,+ toolCalls: 0,+ skillFires: 0,+ mcpCalls: 0,+ numTurns: 1,+ qualityScore: null,+ error: null,+ transcriptPath: "",+ resultPath: "",+ ...over,+ }) as RunResult;++ it("a run *claiming* actual with a null cost no longer yields costSource=actual", () => {+ const m = aggregateCombo("cc+opus4.8+none", [+ run({ costSource: "actual", costUsd: null }),+ run({ costSource: "none" }),+ ]);+ expect(m.costUsd).toBeNull();+ expect(m.costSource).toBe("none");+ });++ it("actual values present → actual; only estimates → estimated", () => {+ expect(aggregateCombo("c", [run({ costUsd: 0.5, costSource: "actual" })]).costSource).toBe(+ "actual",+ );+ expect(+ aggregateCombo("c", [run({ estimatedCostUsd: 0.2, costSource: "estimated" })]).costSource,+ ).toBe("estimated");+ });+});++describe("codex rules harnesses — materialized into the workdir (no silent baseline)", () => {+ it("superpowers rules cache applies to a workdir via AGENTS.md", () => {+ ensureHarnessRulesStub("superpowers", "# superpowers rules (test stub)\n");+ const work = tmp();+ expect(applyHarnessToWorkdir("superpowers", work)).toBe(true);+ expect(existsSync(join(work, "AGENTS.md"))).toBe(true);+ });+});