Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

feat(team): write a sessions-directory record for each team run

Jack Rudenko0558546e6eec
diff --git a/ai-docs/architecture/team-lifecycle.md b/ai-docs/architecture/team-lifecycle.md
index 76ae92c0..93cd89ce 100644
--- a/ai-docs/architecture/team-lifecycle.md
+++ b/ai-docs/architecture/team-lifecycle.md
@@ -148,6 +148,50 @@ ceiling ever mattered rather than working around it.
`runModels` remains as the blocking form, for `run-and-judge` — a pipeline
cannot judge answers that do not exist yet.
+### The run's record in the sessions directory, ended exactly once
+
+Returning early has a cost: the run's completion reaches the caller only through
+a channel frame, and Claude Code drops channel frames unless the session named
+claudish in `--channels`. The run's own `status.json` sits wherever the caller
+pointed `path`, which no outside observer knows. Review panels finished unseen
+for days this way. So `team(mode:"run")` also writes a record into the sessions
+directory the magus `claudish` plugin monitor already watches:
+
+- `<sessionsDir>/team-<8 hex>/spawn.json` — `kind: "team"`, `teamPath`, `slots`,
+ `hostPid`, `mcpPid` (and `launcherPid`, `parentClaudeSessionId` when they
+ apply), written by `SessionManager.recordTeamRun` BEFORE `startModels`; a throw
+ fails the call with nothing started. The `run` result carries the id as
+ `monitor_record`.
+- `<sessionsDir>/team-<8 hex>/meta.json` — `{kind:"team", status, startedAt,
+ completedAt, elapsedSeconds, slots, ok, failed, cancelled}` plus
+ `reason: "start-failed"` when the run never started; written atomically by
+ `finishTeamRun`, which is first-call-wins and never throws.
+
+**The end is wired INTO `startModels`, not after it.** `TeamRunOptions.onSettled`
+is called from `done`'s `finally`, after the settled `status.txt` render, inside
+its own `try`. Read inside `startModels`, it exists before the first child does,
+so a run that settles at once and a slow one take the same path. A settle hung
+on `handle.done.then(…)` in the handler would depend on code that runs after
+`startModels` returns, and would never be installed when `startModels` throws.
+
+**A throw inside the spawn loop kills what it already spawned.** The ticker, the
+registry entry and `done` are all built after the loop, so a synchronous
+`spawn()` error or a failed `status.json` write for slot N used to leave slots
+1..N-1 running and billing, unreachable, with the SIGINT handler still installed.
+The loop's `catch` sends SIGTERM to every spawned slot's process group
+(`signalProcessTree`, as the SIGINT handler does), removes the SIGINT handler and
+rethrows; the handler's `catch` writes `failed`, `reason: start-failed` with
+counts from `status.json` (`summarise(readTeamStatus(path))`), then rethrows the
+ORIGINAL error. `spawnChild` is the seam a test uses to make one spawn throw.
+
+`summarise` counts `COMPLETED` as ok, a `cancelled` reason as cancelled, and every
+other state — `PENDING` and `RUNNING` included — as failed; the verdict is
+`completed` when any slot is ok (the channel frame's rule), `cancelled` when all
+were cancelled. Only `run` writes a record: `run-and-judge` returns its verdict in
+the call, and `judge`/`status`/`cancel` start no work. `loadDiskRecord` returns
+null for a `kind: "team"` directory, so session tools answer a `team-*` id as
+unknown and `list_sessions` (in-memory only) never lists one.
+
### Blinding is unaffected
`run` returns display model → slot id. That does not weaken blind judging. The
diff --git a/docs/usage/mcp-server.md b/docs/usage/mcp-server.md
index ebeea464..9b6464a1 100644
--- a/docs/usage/mcp-server.md
+++ b/docs/usage/mcp-server.md
@@ -136,7 +136,7 @@ Run AI models on a task with anonymized outputs and optional blind judging.
**Modes:**
| Mode | What it does |
|------|-------------|
-| `run` | Run models on the task, write anonymized outputs to session directory |
+| `run` | Start the models on the task and return at once; outputs are written anonymized to the session directory. The result's `monitor_record` names the run's record under `~/.claudish/sessions/` (see "Session records on disk") |
| `judge` | Blind-vote on existing outputs in the session directory |
| `run-and-judge` | Full pipeline: run models, then judge the outputs |
| `status` | Check progress of a running or completed session |
@@ -313,6 +313,13 @@ contract.
| `waits.jsonl` | one line each time the session starts and stops waiting for `send_input` | `{"wait":"open","since":…,"turns":…}` then `{"wait":"closed","since":…,"at":…,"to":…}`. Interactive sessions only; append-only, so a wait that opened and closed between two reads still shows. Stops at 1 MB |
| `meta.json` | once, when the session ends | the final `SessionInfo`: status, exit code, turns, tokens, cost, and `parentClaudeSessionId` exactly when `spawn.json` has it |
+A `team(mode:"run")` gets a record in the same directory, `team-<8 hex>/`, named by the
+`monitor_record` key of the `run` result. Its `spawn.json` has `kind: "team"`, `teamPath` and
+`slots` in place of the session fields; its `meta.json`, written atomically once the run
+settles (or fails to start), is `{"kind":"team","status":…,"startedAt":…,"completedAt":…,"elapsedSeconds":…,"slots":…,"ok":…,"failed":…,"cancelled":…}`,
+plus `"reason":"start-failed"` when no run started. Session tools such as `get_session`
+answer a `team-*` id as unknown; use `team(mode:"status")` for the run itself.
+
---
## Error Reporting
diff --git a/packages/cli/src/channel/session-manager.ts b/packages/cli/src/channel/session-manager.ts
index bbc60eee..e7704865 100644
--- a/packages/cli/src/channel/session-manager.ts
+++ b/packages/cli/src/channel/session-manager.ts
@@ -47,7 +47,12 @@ import { redactSecrets } from "../redact.js";
import { transcriptPathFor } from "../session/session-discovery.js";
import { resolveClaudishSpawn } from "../spawn-claudish.js";
import { decodeChunk } from "../stdio-decode.js";
-import { STDOUT_TAIL_LIMIT, classifyRunOutput, meaningfulStderr } from "../team-orchestrator.js";
+import {
+ STDOUT_TAIL_LIMIT,
+ type TeamRunOutcome,
+ classifyRunOutput,
+ meaningfulStderr,
+} from "../team-orchestrator.js";
import { readTokenStatsAt } from "../team-stats.js";
import { hostPidFrom } from "./parent-proof.js";
import { ScrollbackBuffer } from "./scrollback-buffer.js";
@@ -771,6 +776,10 @@ export class SessionManager {
private sigintHandler: (() => void) | null = null;
private readonly _hostPid: number;
private readonly _launcherPid: number | undefined;
+ /** Team run record id → its `startedAt`, until `finishTeamRun` writes its end. */
+ private readonly teamRunStarts = new Map<string, string>();
+ /** Team run records already ended. First `finishTeamRun` call wins. */
+ private readonly settledTeamRuns = new Set<string>();
constructor(options?: SessionManagerOptions) {
// Computed once: the Claude Code process that launched this MCP server
@@ -802,6 +811,94 @@ export class SessionManager {
return this._launcherPid;
}
+ /**
+ * Start the record of one `team(mode:"run")`: create
+ * `<sessionsDir>/team-<8 hex>/` and write its `spawn.json` (`kind: "team"`).
+ * Returns the record id.
+ *
+ * Called BEFORE `startModels`, so a run never starts unrecorded, and it
+ * throws — failing the tool call before any child exists — when the record
+ * cannot be written. The directory holds only `spawn.json` and, once the run
+ * ends, `meta.json` (`finishTeamRun`). It is invisible to `list_sessions`
+ * (in-memory only), and `loadDiskRecord` refuses its id, so no session tool
+ * mistakes it for a session.
+ */
+ recordTeamRun(opts: { teamPath: string; slots: number; parentClaudeSessionId?: string }): string {
+ mkdirSync(this.sessionsDir, { recursive: true });
+ let id = "";
+ let dir = "";
+ // Non-recursive mkdir fails on an existing directory, so a collision of
+ // two random 8-hex ids can never adopt someone else's record.
+ for (let attempt = 0; ; attempt++) {
+ id = `team-${randomUUID().replace(/-/g, "").slice(0, 8)}`;
+ dir = join(this.sessionsDir, id);
+ try {
+ mkdirSync(dir);
+ break;
+ } catch (err) {
+ if ((err as NodeJS.ErrnoException)?.code !== "EEXIST" || attempt >= 4) throw err;
+ }
+ }
+ const startedAt = new Date().toISOString();
+ this.writeSpawnRecord(dir, {
+ kind: "team",
+ sessionId: id,
+ parentClaudeSessionId: opts.parentClaudeSessionId,
+ startedAt,
+ teamPath: resolve(opts.teamPath),
+ slots: opts.slots,
+ });
+ this.teamRunStarts.set(id, startedAt);
+ return id;
+ }
+
+ /**
+ * End a team run's record: write `<record dir>/meta.json` atomically, at most
+ * once.
+ *
+ * First call wins; a later call writes nothing and reports one stderr line.
+ * It NEVER throws: a write failure is reported on stderr, so a caller's
+ * `catch` always rethrows its ORIGINAL error rather than a disk error that
+ * masked it. A lost write leaves a record with no end, which an observer
+ * reports from the writer's liveness — never a false verdict.
+ */
+ finishTeamRun(record: string, outcome: TeamRunOutcome): void {
+ if (this.settledTeamRuns.has(record)) {
+ process.stderr.write(`[claudish] team run record ${record} already ended; ignoring\n`);
+ return;
+ }
+ this.settledTeamRuns.add(record);
+ try {
+ const dir = this.diskSessionDir(record);
+ if (dir === null) throw new Error("not a record id");
+ const completedAt = new Date().toISOString();
+ const startedAt =
+ this.teamRunStarts.get(record) ??
+ metaString(readJsonObject(join(dir, "spawn.json"), META_READ_LIMIT)?.startedAt) ??
+ completedAt;
+ const elapsedMs = Date.parse(completedAt) - Date.parse(startedAt);
+ writeJsonAtomic(join(dir, "meta.json"), {
+ kind: "team",
+ status: outcome.status,
+ startedAt,
+ completedAt,
+ elapsedSeconds: Number.isFinite(elapsedMs) ? Math.max(0, Math.round(elapsedMs / 1000)) : 0,
+ slots: outcome.slots,
+ ok: outcome.ok,
+ failed: outcome.failed,
+ cancelled: outcome.cancelled,
+ ...(outcome.reason ? { reason: outcome.reason } : {}),
+ });
+ } catch (err) {
+ process.stderr.write(
+ `[claudish] could not write the end of team run record ${record}: ` +
+ `${err instanceof Error ? err.message : String(err)}\n`
+ );
+ } finally {
+ this.teamRunStarts.delete(record);
+ }
+ }
+
/**
* Write `<dir>/spawn.json`, the start-time record, atomically (tmp + rename),
* one key per line.
diff --git a/packages/cli/src/mcp-server.ts b/packages/cli/src/mcp-server.ts
index acab036c..b09f998c 100644
--- a/packages/cli/src/mcp-server.ts
+++ b/packages/cli/src/mcp-server.ts
@@ -55,14 +55,17 @@ import { route } from "./providers/routing-rules.js";
import { createProxyServer } from "./proxy-server.js";
import { sanitizeForReport } from "./redact.js";
import {
+ type TeamRunOutcome,
cancelTeamRun,
getStatus,
judgeResponses,
readTeamInputFile,
+ readTeamStatus,
runModels,
setupSession,
shutdownAllTeamRuns,
startModels,
+ summarise,
teamSlotActivity,
teamSlotIdleSeconds,
teamSlotLiveBytes,
@@ -1414,12 +1417,53 @@ function defineTools(
case "run": {
if (!models?.length) throw new Error("'models' is required for 'run' mode");
setupSession(resolved, models, input);
+
+ // The run's record in the sessions directory, for observers outside
+ // this process (the magus claudish plugin monitor): the run's own
+ // status.json lives wherever the caller pointed `path`, and its only
+ // completion push is a channel frame Claude Code drops without
+ // `--channels`. Written BEFORE any child exists; a throw here fails
+ // the call with nothing started.
+ const parentClaudeSessionId = await proveParentForCall({
+ toolUseId: ctx.toolUseId,
+ hostPid: sessionManager.hostPid,
+ });
+ const monitorRecord = sessionManager.recordTeamRun({
+ teamPath: resolved,
+ slots: models.length,
+ parentClaudeSessionId,
+ });
+ const settleRecord = (outcome: TeamRunOutcome): void =>
+ sessionManager.finishTeamRun(monitorRecord, outcome);
+
// Returns once the children EXIST, not once they finish. A team slot
// is a full Claude Code session and can legitimately work for a long
// time; holding the tool call open for that made the run's duration
// the client's problem, and the deadline that existed to bound it
// killed working slots. Poll `mode: "status"` instead.
- const handle = await startModels(resolved, runOpts);
+ //
+ // The record's end is passed IN as `onSettled`, so it exists before
+ // the first child and fires however fast the run settles. If
+ // `startModels` throws, it has already killed whatever it spawned;
+ // the record ends `failed`, `reason: start-failed`, and the ORIGINAL
+ // error reaches the caller (`finishTeamRun` never throws).
+ let handle: Awaited<ReturnType<typeof startModels>>;
+ try {
+ handle = await startModels(resolved, {
+ ...runOpts,
+ onSettled: (status) => settleRecord(summarise(status)),
+ });
+ } catch (err) {
+ const counts = summarise(readTeamStatus(resolved));
+ settleRecord({
+ ...counts,
+ slots: Math.max(counts.slots, models.length),
+ failed: models.length - counts.ok - counts.cancelled,
+ status: "failed",
+ reason: "start-failed",
+ });
+ throw err;
+ }
return {
content: [
{
@@ -1429,6 +1473,7 @@ function defineTools(
started: true,
team_session_id: handle.teamSessionId,
session_path: handle.sessionPath,
+ monitor_record: monitorRecord,
slots: handle.slots,
next: {
status: `team(mode:"status", path:"${handle.sessionPath}")`,
diff --git a/packages/cli/src/team-mcp-shape-contract.e2e.test.ts b/packages/cli/src/team-mcp-shape-contract.e2e.test.ts
index 09ccc689..e94dad4d 100644
--- a/packages/cli/src/team-mcp-shape-contract.e2e.test.ts
+++ b/packages/cli/src/team-mcp-shape-contract.e2e.test.ts
@@ -137,6 +137,9 @@ describe("MCP team shape contract", () => {
CLAUDISH_CONFIG: configPath,
CLAUDISH_DISABLE_OP: "1",
CLAUDISH_MCP_TOOLS: "all",
+ // A team run writes its record under the sessions directory; keep
+ // it inside this test's temp root, never the real ~/.claudish.
+ CLAUDISH_SESSIONS_DIR: join(tempRoot, "sessions"),
},
});
diff --git a/packages/cli/src/team-orchestrator.ts b/packages/cli/src/team-orchestrator.ts
index e59adee4..84025e79 100644
--- a/packages/cli/src/team-orchestrator.ts
+++ b/packages/cli/src/team-orchestrator.ts
@@ -200,6 +200,23 @@ export interface TeamRunOptions {
heartbeatSeconds?: number;
/** Spawn-plan factory seam for hermetic call-site tests. */
spawnPlanner?: (models: (string | undefined)[]) => Promise<SpawnPlan>;
+ /**
+ * Child-process spawn seam for hermetic tests (default `node:child_process`
+ * `spawn`). Lets a test make one slot's spawn throw synchronously.
+ */
+ spawnChild?: typeof spawn;
+ /**
+ * Called exactly once, when the run SETTLES: after the last slot finished and
+ * the settled `status.txt` render, from inside `done`'s `finally`. Receives
+ * the final status.
+ *
+ * Read inside `startModels`, so the hook exists before the first child is
+ * spawned and does not depend on anything the caller does after
+ * `startModels` returns — a run that settles at once takes the same path as a
+ * slow one. A throwing consumer cannot fail the run. NOT called when
+ * `startModels` itself throws: no run started, and the caller settles that.
+ */
+ onSettled?: (status: TeamStatus) => void;
}
/**
@@ -1054,436 +1071,461 @@ export async function startModels(
const completionPromises: Promise<void>[] = [];
- for (const [anonId, entry] of Object.entries(manifest.models)) {
- const outputPath = join(sessionPath, `response-${anonId}.md`);
- const errorLogPath = join(sessionPath, "errors", `${anonId}.log`);
- const upstreamErrorLogPath = join(sessionPath, "errors", `${anonId}-upstream.jsonl`);
-
- // Spawn with the parent-resolved explicit spec when there is one, so the
- // child skips routing entirely and finds its key in the inherited env.
- // ABSENT from the map is not an error — it means "spawn it bare", which is
- // exactly the pre-pinning behaviour. The manifest keeps `entry.model` (the
- // user's string) as the run's identity; only argv changes.
- const spawnModel = spawnPlan.pinned.get(entry.model) ?? entry.model;
-
- // CRITICAL FIX: do NOT use -p flag (-p means --profile in claudish)
- // --stdin triggers non-interactive single-shot mode
- //
- // ORDER IS LOAD-BEARING when recovery is on. claudish consumes --verbose as
- // its OWN log-verbosity flag (it sets quiet=false) and separately forwards a
- // copy to the child `claude`, which hard-errors on
- // `--print --output-format stream-json` without it. Putting --verbose BEFORE
- // --quiet gets the forward while letting --quiet win claudish's own
- // verbosity — reversed, every child would narrate itself onto stderr.
- // `--output-format stream-json` is an unknown flag to claudish and passes
- // through to `claude` with its value.
- const args = [
- "--model",
- spawnModel,
- "-y",
- "--stdin",
- ...(captureMode === "stream-json"
- ? ["--verbose", "--quiet", "--output-format", "stream-json"]
- : ["--quiet"]),
- ...(opts.claudeFlags ?? []),
- ];
-
- updateModelStatus(anonId, {
- state: "RUNNING",
- startedAt: new Date().toISOString(),
- });
-
- // See session-manager: resolved so a harness can spawn the tree under test
- // instead of the installed binary. Unset in production → plain "claudish".
- const teamSpawnTarget = resolveClaudishSpawn();
- const proc = spawn(teamSpawnTarget.command, [...teamSpawnTarget.prefixArgs, ...args], {
- stdio: ["pipe", "pipe", "pipe"],
- shell: false,
- // Each child leads its OWN process group, so we can signal the whole
- // subtree with `process.kill(-pid)`.
- //
- // `claudish` is a tree, not a process: the bin is a Node launcher that
- // runs the real CLI under Bun, which spawns `claude`, which spawns its
- // own tools and MCP servers. Signalling the direct child alone reaches
- // only the launcher — measured 2026-08-15, both SIGTERM and SIGKILL to
- // the pid left the Bun process alive and still holding the write end of
- // the pipe feeding `response-<id>.md`, which is how a run declared
- // TIMEOUT with 0 B gained a complete 40,699 B answer six minutes later.
- // Only the group kill was clean.
- //
- // Trade-off: a detached child no longer receives the terminal's Ctrl+C
- // (that goes to the foreground group, which the child has just left), so
- // the SIGINT handler below MUST kill the groups explicitly. It also means
- // an orchestrator that dies without running its handlers leaves children
- // behind — the same exposure as before this change, not a new one.
- detached: KILL_PROCESS_GROUP,
- env: {
- ...process.env,
- // Point this child's token tracker at a path WE choose, so its
- // tokens/cost can be attributed back to this model. Without this the
- // child writes to tokens-<its-own-port>.json and nothing links the two.
- [ENV.CLAUDISH_TOKEN_FILE]: tokenFileFor(sessionPath, anonId),
- // Un-no-op `captureUpstreamError` (handlers/composed-handler.ts), which
- // is opt-in on this env var and was therefore a guaranteed no-op for
- // every team child. Its own comment says what that costs: `log()` only
- // persists under `--debug`, so the upstream body that separates a
- // retryable rate limit from a hard quota wall is gone the moment it has
- // been classified — and a run that already failed cannot be re-run with
- // a flag. The channel has set this all along; team never did.
- //
- // Per SLOT, unconditionally: the records carry no slot id, so one shared
- // path would interleave every model in the run into an unattributable
- // file.
- [UPSTREAM_ERROR_LOG_ENV]: upstreamErrorLogPath,
- },
- });
-
- /**
- * When this child last wrote on either pipe.
- *
- * A SEPARATE listener from the capture below, deliberately. Capture asks
- * "is this an answer?" and answers no for a `tool_progress` heartbeat or a
- * thinking frame; liveness asks "is anything alive down there?" and those
- * same frames answer yes. Conflating the two questions is precisely how the
- * old reaper concluded that a compiling child was dead.
- */
- let lastOutputAt = Date.now();
- const stampLiveness = (): void => {
- lastOutputAt = Date.now();
- };
-
- /**
- * One decoder per pipe, never shared.
- *
- * A `data` chunk ends at the pipe's read boundary, which lands mid-codepoint
- * often enough to matter: `chunk.toString()` replaces the dangling bytes
- * with U+FFFD, so any CJK character or emoji straddling a boundary was
- * permanently mangled in `response-<id>.md` and mis-sized in `outputSize`.
- * The channel has decoded this way all along; team did not.
- */
- const stdoutDecoder = newStdioDecoder();
- const stderrDecoder = newStdioDecoder();
- proc.stdout?.on("data", stampLiveness);
- proc.stderr?.on("data", stampLiveness);
-
- // Count bytes flowing through stdout for accurate outputSize tracking
- let byteCount = 0;
- // Bounded tail of stdout. Claude Code writes "[API Error: ...]" to stdout
- // and still exits 0, so the failure signal is often here rather than on
- // stderr. Bounded so a 30 KB answer doesn't get buffered twice.
- let stdoutTail = "";
-
- const outputStream = createWriteStream(outputPath);
-
- /** See ModelRuntime.flushPartial. Reassigned below when recovery is on. */
- let flushPartial: () => void = () => {};
-
- /**
- * The stream-json supervisor for this slot, or null under `"print"`.
- *
- * `team` used to drive `createAssistantTextCapture()` directly and hand-roll
- * everything around it. The channel wraps that SAME capture in
- * `StreamJsonReducer` and adds what team was missing: a state machine that
- * knows the difference between thinking and running a tool, and `sawResult`
- * — the child's own terminal `result` frame, which is a real completion
- * oracle where exit 0 is not (`claude -p` exits 0 on API errors too).
- *
- * Two implementations of one job existed because this file predates the
- * reducer by four months. There is now one parser; this is the caller that
- * moved onto it.
- */
- let reducer: StreamJsonReducer | null = null;
-
- if (captureMode === "print") {
- // Legacy path: whatever `claude -p` printed, byte for byte.
- proc.stdout?.on("data", (chunk: Buffer) => {
- byteCount += chunk.length;
- stdoutTail = (stdoutTail + decodeChunk(stdoutDecoder, chunk)).slice(-STDOUT_TAIL_LIMIT);
- });
- // Stream stdout to disk via pipe — no memory buffering
- proc.stdout?.pipe(outputStream);
- } else {
- // Recovery path. The child's stdout is a stream-json event log, so the
- // ANSWER has to be extracted from it rather than piped through.
+ // The spawn loop has its own catch because NOTHING after it exists yet when
+ // it throws: the ticker, the registry entry and `done` are all built below.
+ // A synchronous `spawn()` error or a failed `status.json` write for slot N
+ // would otherwise leave slots 1..N-1 running and billing with no `done`, no
+ // registry entry to cancel them through, and the SIGINT handler installed.
+ try {
+ for (const [anonId, entry] of Object.entries(manifest.models)) {
+ const outputPath = join(sessionPath, `response-${anonId}.md`);
+ const errorLogPath = join(sessionPath, "errors", `${anonId}.log`);
+ const upstreamErrorLogPath = join(sessionPath, "errors", `${anonId}-upstream.jsonl`);
+
+ // Spawn with the parent-resolved explicit spec when there is one, so the
+ // child skips routing entirely and finds its key in the inherited env.
+ // ABSENT from the map is not an error — it means "spawn it bare", which is
+ // exactly the pre-pinning behaviour. The manifest keeps `entry.model` (the
+ // user's string) as the run's identity; only argv changes.
+ const spawnModel = spawnPlan.pinned.get(entry.model) ?? entry.model;
+
+ // CRITICAL FIX: do NOT use -p flag (-p means --profile in claudish)
+ // --stdin triggers non-interactive single-shot mode
//
- // byteCount and stdoutTail are deliberately fed the RECOVERED prose, not
- // the raw JSON. Every downstream consumer — the empty check, the
- // minOutputBytes threshold, the [API Error:] match, the reported
- // outputSize — is asking about the answer, and raw JSON bytes would
- // inflate all of them (an empty answer wrapped in events is still
- // kilobytes). Feeding recovered prose keeps `classifyRunOutput`
- // completely unaware that the wire format changed.
- const slotReducer = new StreamJsonReducer({
- sessionId: anonId,
- // 0 disables the reducer's stall watchdog. That watchdog ANNOUNCES
- // silence; team publishes the number through `teamSlotIdleSeconds` and
- // leaves the verdict to the caller, so a second opinion on the same
- // question would only be noise. It also means no timer is armed here.
- stallSeconds: 0,
- // Preserve anything not positively recognised. `response-<id>.md` is the
- // only place a reader sees what this child printed, and discarding a
- // real answer is the failure this file has already been burned by —
- // see ai-docs/architecture/team-capture.md.
- keepUnrecognizedJson: true,
- // State changes are read on demand via `getActivity`, not pushed. Team
- // already has its own status file and progress ticker; routing reducer
- // transitions into a second notification path would duplicate it.
- callback: () => {},
- // stdin is written and ended right after spawn (below), so a `result`
- // is always the end of the slot's work: `finishing` until it exits,
- // never `waiting_for_input`.
- onResult: () => "stdin-closed",
+ // ORDER IS LOAD-BEARING when recovery is on. claudish consumes --verbose as
+ // its OWN log-verbosity flag (it sets quiet=false) and separately forwards a
+ // copy to the child `claude`, which hard-errors on
+ // `--print --output-format stream-json` without it. Putting --verbose BEFORE
+ // --quiet gets the forward while letting --quiet win claudish's own
+ // verbosity — reversed, every child would narrate itself onto stderr.
+ // `--output-format stream-json` is an unknown flag to claudish and passes
+ // through to `claude` with its value.
+ const args = [
+ "--model",
+ spawnModel,
+ "-y",
+ "--stdin",
+ ...(captureMode === "stream-json"
+ ? ["--verbose", "--quiet", "--output-format", "stream-json"]
+ : ["--quiet"]),
+ ...(opts.claudeFlags ?? []),
+ ];
+
+ updateModelStatus(anonId, {
+ state: "RUNNING",
+ startedAt: new Date().toISOString(),
});
- reducer = slotReducer;
-
- const absorb = (text: string): void => {
- if (text.length === 0) return;
- byteCount += Buffer.byteLength(text);
- stdoutTail = (stdoutTail + text).slice(-STDOUT_TAIL_LIMIT);
- outputStream.write(text);
- };
- // `feed` returns exactly what `capture.write` returned — the recovered
- // prose for this chunk — so every downstream consumer of `byteCount` and
- // `stdoutTail` is unaffected by the swap.
- proc.stdout?.on("data", (chunk: Buffer) =>
- absorb(slotReducer.feed(decodeChunk(stdoutDecoder, chunk)))
+ // See session-manager: resolved so a harness can spawn the tree under test
+ // instead of the installed binary. Unset in production → plain "claudish".
+ const teamSpawnTarget = resolveClaudishSpawn();
+ const proc = (opts.spawnChild ?? spawn)(
+ teamSpawnTarget.command,
+ [...teamSpawnTarget.prefixArgs, ...args],
+ {
+ stdio: ["pipe", "pipe", "pipe"],
+ shell: false,
+ // Each child leads its OWN process group, so we can signal the whole
+ // subtree with `process.kill(-pid)`.
+ //
+ // `claudish` is a tree, not a process: the bin is a Node launcher that
+ // runs the real CLI under Bun, which spawns `claude`, which spawns its
+ // own tools and MCP servers. Signalling the direct child alone reaches
+ // only the launcher — measured 2026-08-15, both SIGTERM and SIGKILL to
+ // the pid left the Bun process alive and still holding the write end of
+ // the pipe feeding `response-<id>.md`, which is how a run declared
+ // TIMEOUT with 0 B gained a complete 40,699 B answer six minutes later.
+ // Only the group kill was clean.
+ //
+ // Trade-off: a detached child no longer receives the terminal's Ctrl+C
+ // (that goes to the foreground group, which the child has just left), so
+ // the SIGINT handler below MUST kill the groups explicitly. It also means
+ // an orchestrator that dies without running its handlers leaves children
+ // behind — the same exposure as before this change, not a new one.
+ detached: KILL_PROCESS_GROUP,
+ env: {
+ ...process.env,
+ // Point this child's token tracker at a path WE choose, so its
+ // tokens/cost can be attributed back to this model. Without this the
+ // child writes to tokens-<its-own-port>.json and nothing links the two.
+ [ENV.CLAUDISH_TOKEN_FILE]: tokenFileFor(sessionPath, anonId),
+ // Un-no-op `captureUpstreamError` (handlers/composed-handler.ts), which
+ // is opt-in on this env var and was therefore a guaranteed no-op for
+ // every team child. Its own comment says what that costs: `log()` only
+ // persists under `--debug`, so the upstream body that separates a
+ // retryable rate limit from a hard quota wall is gone the moment it has
+ // been classified — and a run that already failed cannot be re-run with
+ // a flag. The channel has set this all along; team never did.
+ //
+ // Per SLOT, unconditionally: the records carry no slot id, so one shared
+ // path would interleave every model in the run into an unattributable
+ // file.
+ [UPSTREAM_ERROR_LOG_ENV]: upstreamErrorLogPath,
+ },
+ }
);
+ // Tracked the moment it exists, so a throw later in this iteration still
+ // reaches it through the loop's catch below.
+ processes.set(anonId, proc);
- // `end()` is idempotent, so a caller draining early does not disturb the
- // normal finalisation below.
- flushPartial = () => absorb(slotReducer.end());
-
- // The write stream is ours to close now that nothing pipes into it, and
- // `finish()` hangs off its "close". Both events are wired because "end"
- // does not fire on a destroyed stream (a killed child), and a run that
- // never resolves is worse than one that resolves empty.
- let captureFinalized = false;
- const finalizeCapture = (): void => {
- if (captureFinalized) return;
- captureFinalized = true;
- absorb(slotReducer.end());
- // NOT disposed here: `finish()` decides the slot's outcome later, off
- // outputStream "close", and a disposed reducer ignores `settle()`. It
- // settles and disposes the reducer there, once.
- outputStream.end();
+ /**
+ * When this child last wrote on either pipe.
+ *
+ * A SEPARATE listener from the capture below, deliberately. Capture asks
+ * "is this an answer?" and answers no for a `tool_progress` heartbeat or a
+ * thinking frame; liveness asks "is anything alive down there?" and those
+ * same frames answer yes. Conflating the two questions is precisely how the
+ * old reaper concluded that a compiling child was dead.
+ */
+ let lastOutputAt = Date.now();
+ const stampLiveness = (): void => {
+ lastOutputAt = Date.now();
};
- proc.stdout?.on("end", finalizeCapture);
- proc.stdout?.on("close", finalizeCapture);
- }
- // Collect stderr for error logging
- let stderr = "";
- proc.stderr?.on("data", (chunk: Buffer) => {
- stderr += decodeChunk(stderrDecoder, chunk);
- });
+ /**
+ * One decoder per pipe, never shared.
+ *
+ * A `data` chunk ends at the pipe's read boundary, which lands mid-codepoint
+ * often enough to matter: `chunk.toString()` replaces the dangling bytes
+ * with U+FFFD, so any CJK character or emoji straddling a boundary was
+ * permanently mangled in `response-<id>.md` and mis-sized in `outputSize`.
+ * The channel has decoded this way all along; team did not.
+ */
+ const stdoutDecoder = newStdioDecoder();
+ const stderrDecoder = newStdioDecoder();
+ proc.stdout?.on("data", stampLiveness);
+ proc.stderr?.on("data", stampLiveness);
- const command = `claudish ${args.join(" ")}`;
- runtimes.set(anonId, {
- command,
- errorLogPath,
- getStderr: () => stderr,
- getStdoutTail: () => stdoutTail,
- getByteCount: () => byteCount,
- // The raw-pipe stamp, not `reducer.idleMs`. The reducer's clock advances
- // per complete LINE, so a child writing one long line slowly would look
- // quiet; this one sees every byte, on both pipes. Broadest definition of
- // "still alive", which is the question being asked.
- getIdleMs: () => Math.max(0, Date.now() - lastOutputAt),
- getActivity: () => reducer?.state ?? null,
- flushPartial: () => flushPartial(),
- });
+ // Count bytes flowing through stdout for accurate outputSize tracking
+ let byteCount = 0;
+ // Bounded tail of stdout. Claude Code writes "[API Error: ...]" to stdout
+ // and still exits 0, so the failure signal is often here rather than on
+ // stderr. Bounded so a 30 KB answer doesn't get buffered twice.
+ let stdoutTail = "";
- // Pipe input to stdin. A slot cancelled (or a child that exits) before it
- // drains stdin closes the pipe under a pending write, and Node surfaces
- // that as an `error` event on the stream — unhandled, it is an uncaught
- // exception in the orchestrator, not the child. EPIPE here means only
- // "the reader went away", which the exit/close handlers already report;
- // anything else is still worth a log line.
- proc.stdin?.on("error", (err: NodeJS.ErrnoException) => {
- if (err?.code === "EPIPE") return;
- stderr += `[claudish] stdin error for slot ${anonId}: ${err?.message ?? String(err)}\n`;
- });
- proc.stdin?.write(inputContent);
- proc.stdin?.end();
+ const outputStream = createWriteStream(outputPath);
- const completionPromise = new Promise<void>((resolve) => {
- let exitCode: number | null = null;
- let resolved = false;
+ /** See ModelRuntime.flushPartial. Reassigned below when recovery is on. */
+ let flushPartial: () => void = () => {};
/**
- * Settle the reducer to the outcome just recorded, then release it.
+ * The stream-json supervisor for this slot, or null under `"print"`.
+ *
+ * `team` used to drive `createAssistantTextCapture()` directly and hand-roll
+ * everything around it. The channel wraps that SAME capture in
+ * `StreamJsonReducer` and adds what team was missing: a state machine that
+ * knows the difference between thinking and running a tool, and `sawResult`
+ * — the child's own terminal `result` frame, which is a real completion
+ * oracle where exit 0 is not (`claude -p` exits 0 on API errors too).
*
- * Without the settle the reducer stays wherever the stream left it — a
- * `result` frame parks it in `finishing` — so an exited slot read as
- * still exiting rather than done. Runs only once `state` is final, which is why
- * the dispose lives here and not in `finalizeCapture`. No-op under
- * `"print"`, which has no reducer.
+ * Two implementations of one job existed because this file predates the
+ * reducer by four months. There is now one parser; this is the caller that
+ * moved onto it.
*/
- const settleReducer = (): void => {
- if (!reducer) return;
- const state = statusCache.models[anonId]?.state;
- const failed = state === "FAILED" || state === "EMPTY";
- reducer.settle(
- failed && cancelledSlots.has(anonId)
- ? "cancelled"
- : state === "COMPLETED"
- ? "completed"
- : state === "TIMEOUT"
- ? "timeout"
- : "failed"
- );
- reducer.dispose();
- };
+ let reducer: StreamJsonReducer | null = null;
- const finish = () => {
- if (resolved) return;
- // The timeout handler may have fired between proc "exit" and
- // outputStream "close". Don't clobber TIMEOUT — but do RECONCILE.
+ if (captureMode === "print") {
+ // Legacy path: whatever `claude -p` printed, byte for byte.
+ proc.stdout?.on("data", (chunk: Buffer) => {
+ byteCount += chunk.length;
+ stdoutTail = (stdoutTail + decodeChunk(stdoutDecoder, chunk)).slice(-STDOUT_TAIL_LIMIT);
+ });
+ // Stream stdout to disk via pipe — no memory buffering
+ proc.stdout?.pipe(outputStream);
+ } else {
+ // Recovery path. The child's stdout is a stream-json event log, so the
+ // ANSWER has to be extracted from it rather than piped through.
//
- // Reaching here means the stdout pipe has closed, so `response-<id>.md`
- // is final and `byteCount` is its true size. A child killed at the
- // deadline can still have flushed a complete answer in the window
- // between the signal and the pipe closing, and the old code threw that
- // away: it recorded whatever byte count existed at KILL time (0 B for a
- // `--quiet` child, which emits only at the end) and never looked again.
- // The judging phase then scored a real answer as an empty submission.
- if (statusCache.models[anonId].state === "TIMEOUT") {
- resolved = true;
- reconcileTimedOutOutput(anonId, byteCount, stdoutTail, stderr);
- // After the reconcile, which can upgrade TIMEOUT to COMPLETED.
- settleReducer();
- resolve();
- return;
- }
- resolved = true;
-
- const outputSize = byteCount;
-
- // A non-zero exit is an outright failure. A zero exit still has to earn
- // it: `claude -p` exits 0 on API errors and on background-task
- // termination, so exit code alone would file both as success.
- const crashed = exitCode !== 0;
- const fullOutput = readFullOutputIfNeeded({
- crashed,
- requirePattern,
- outputSize,
- outputPath,
+ // byteCount and stdoutTail are deliberately fed the RECOVERED prose, not
+ // the raw JSON. Every downstream consumer — the empty check, the
+ // minOutputBytes threshold, the [API Error:] match, the reported
+ // outputSize — is asking about the answer, and raw JSON bytes would
+ // inflate all of them (an empty answer wrapped in events is still
+ // kilobytes). Feeding recovered prose keeps `classifyRunOutput`
+ // completely unaware that the wire format changed.
+ const slotReducer = new StreamJsonReducer({
+ sessionId: anonId,
+ // 0 disables the reducer's stall watchdog. That watchdog ANNOUNCES
+ // silence; team publishes the number through `teamSlotIdleSeconds` and
+ // leaves the verdict to the caller, so a second opinion on the same
+ // question would only be noise. It also means no timer is armed here.
+ stallSeconds: 0,
+ // Preserve anything not positively recognised. `response-<id>.md` is the
+ // only place a reader sees what this child printed, and discarding a
+ // real answer is the failure this file has already been burned by —
+ // see ai-docs/architecture/team-capture.md.
+ keepUnrecognizedJson: true,
+ // State changes are read on demand via `getActivity`, not pushed. Team
+ // already has its own status file and progress ticker; routing reducer
+ // transitions into a second notification path would duplicate it.
+ callback: () => {},
+ // stdin is written and ended right after spawn (below), so a `result`
+ // is always the end of the slot's work: `finishing` until it exits,
+ // never `waiting_for_input`.
+ onResult: () => "stdin-closed",
});
- const degraded = crashed
- ? null
- : classifyRunOutput({
- outputSize,
- stdoutTail,
- stderr,
- minOutputBytes,
- requirePattern,
- fullOutput,
- captureMode,
- });
+ reducer = slotReducer;
+
+ const absorb = (text: string): void => {
+ if (text.length === 0) return;
+ byteCount += Buffer.byteLength(text);
+ stdoutTail = (stdoutTail + text).slice(-STDOUT_TAIL_LIMIT);
+ outputStream.write(text);
+ };
+
+ // `feed` returns exactly what `capture.write` returned — the recovered
+ // prose for this chunk — so every downstream consumer of `byteCount` and
+ // `stdoutTail` is unaffected by the swap.
+ proc.stdout?.on("data", (chunk: Buffer) =>
+ absorb(slotReducer.feed(decodeChunk(stdoutDecoder, chunk)))
+ );
- const failed = crashed || degraded !== null;
- const state: ModelState = crashed ? "FAILED" : degraded ? "EMPTY" : "COMPLETED";
-
- if (failed) {
- // A slot the caller stopped exited non-zero, but it did not crash and
- // saying so would be a lie in the permanent record. `cancelled` is the
- // one failure reason that is not a defect: the caller looked at the
- // evidence and decided. Nothing else in claudish can produce it.
- const wasCancelled = cancelledSlots.has(anonId);
- const reason: FailureReason = wasCancelled
- ? "cancelled"
- : crashed
- ? "nonzero_exit"
- : degraded!.reason;
- const detail = wasCancelled
- ? 'Stopped on the caller\'s instruction via team(mode:"cancel"). ' +
- "Whatever the child had written up to that point is in its response file."
- : crashed
- ? `Child exited with code ${exitCode}.`
- : degraded!.detail;
-
- persistErrorLog(errorLogPath, `${state}: ${detail}`, stderr, stdoutTail);
-
- updateModelStatus(anonId, {
- state,
- exitCode: exitCode ?? 1,
- completedAt: new Date().toISOString(),
- outputSize,
- error: {
- model: anonId,
- command,
- reason,
- detail,
- // Redacted: these land in status.json on disk and are read back
- // by anything inspecting the run.
- stderrSnippet: stderr ? redactSecrets(stderr).slice(-2000) : undefined,
- stdoutSnippet: stdoutTail ? snippetHeadAndTail(redactSecrets(stdoutTail)) : undefined,
- errorLogPath,
- // Only when the child actually wrote one. Naming a file that does
- // not exist sends a reader after evidence that was never captured.
- upstreamErrorLogPath: existsSync(upstreamErrorLogPath)
- ? upstreamErrorLogPath
- : undefined,
- workDir: sessionPath,
- },
- });
- } else {
- updateModelStatus(anonId, {
- state,
- exitCode: exitCode ?? 0,
- completedAt: new Date().toISOString(),
- outputSize,
- error: undefined,
- });
- }
+ // `end()` is idempotent, so a caller draining early does not disturb the
+ // normal finalisation below.
+ flushPartial = () => absorb(slotReducer.end());
+
+ // The write stream is ours to close now that nothing pipes into it, and
+ // `finish()` hangs off its "close". Both events are wired because "end"
+ // does not fire on a destroyed stream (a killed child), and a run that
+ // never resolves is worse than one that resolves empty.
+ let captureFinalized = false;
+ const finalizeCapture = (): void => {
+ if (captureFinalized) return;
+ captureFinalized = true;
+ absorb(slotReducer.end());
+ // NOT disposed here: `finish()` decides the slot's outcome later, off
+ // outputStream "close", and a disposed reducer ignores `settle()`. It
+ // settles and disposes the reducer there, once.
+ outputStream.end();
+ };
+ proc.stdout?.on("end", finalizeCapture);
+ proc.stdout?.on("close", finalizeCapture);
+ }
+
+ // Collect stderr for error logging
+ let stderr = "";
+ proc.stderr?.on("data", (chunk: Buffer) => {
+ stderr += decodeChunk(stderrDecoder, chunk);
+ });
- // Before the caller's callback, so a throwing consumer cannot skip the
- // dispose and leak the reducer's tool-batch timer.
- settleReducer();
- opts.onStatusChange?.(anonId, statusCache.models[anonId]);
- resolve();
- };
+ const command = `claudish ${args.join(" ")}`;
+ runtimes.set(anonId, {
+ command,
+ errorLogPath,
+ getStderr: () => stderr,
+ getStdoutTail: () => stdoutTail,
+ getByteCount: () => byteCount,
+ // The raw-pipe stamp, not `reducer.idleMs`. The reducer's clock advances
+ // per complete LINE, so a child writing one long line slowly would look
+ // quiet; this one sees every byte, on both pipes. Broadest definition of
+ // "still alive", which is the question being asked.
+ getIdleMs: () => Math.max(0, Date.now() - lastOutputAt),
+ getActivity: () => reducer?.state ?? null,
+ flushPartial: () => flushPartial(),
+ });
- // "close" always fires after the stream ends or errors — single resolution point
- outputStream.on("close", finish);
+ // Pipe input to stdin. A slot cancelled (or a child that exits) before it
+ // drains stdin closes the pipe under a pending write, and Node surfaces
+ // that as an `error` event on the stream — unhandled, it is an uncaught
+ // exception in the orchestrator, not the child. EPIPE here means only
+ // "the reader went away", which the exit/close handlers already report;
+ // anything else is still worth a log line.
+ proc.stdin?.on("error", (err: NodeJS.ErrnoException) => {
+ if (err?.code === "EPIPE") return;
+ stderr += `[claudish] stdin error for slot ${anonId}: ${err?.message ?? String(err)}\n`;
+ });
+ proc.stdin?.write(inputContent);
+ proc.stdin?.end();
+
+ const completionPromise = new Promise<void>((resolve) => {
+ let exitCode: number | null = null;
+ let resolved = false;
+
+ /**
+ * Settle the reducer to the outcome just recorded, then release it.
+ *
+ * Without the settle the reducer stays wherever the stream left it — a
+ * `result` frame parks it in `finishing` — so an exited slot read as
+ * still exiting rather than done. Runs only once `state` is final, which is why
+ * the dispose lives here and not in `finalizeCapture`. No-op under
+ * `"print"`, which has no reducer.
+ */
+ const settleReducer = (): void => {
+ if (!reducer) return;
+ const state = statusCache.models[anonId]?.state;
+ const failed = state === "FAILED" || state === "EMPTY";
+ reducer.settle(
+ failed && cancelledSlots.has(anonId)
+ ? "cancelled"
+ : state === "COMPLETED"
+ ? "completed"
+ : state === "TIMEOUT"
+ ? "timeout"
+ : "failed"
+ );
+ reducer.dispose();
+ };
+
+ const finish = () => {
+ if (resolved) return;
+ // The timeout handler may have fired between proc "exit" and
+ // outputStream "close". Don't clobber TIMEOUT — but do RECONCILE.
+ //
+ // Reaching here means the stdout pipe has closed, so `response-<id>.md`
+ // is final and `byteCount` is its true size. A child killed at the
+ // deadline can still have flushed a complete answer in the window
+ // between the signal and the pipe closing, and the old code threw that
+ // away: it recorded whatever byte count existed at KILL time (0 B for a
+ // `--quiet` child, which emits only at the end) and never looked again.
+ // The judging phase then scored a real answer as an empty submission.
+ if (statusCache.models[anonId].state === "TIMEOUT") {
+ resolved = true;
+ reconcileTimedOutOutput(anonId, byteCount, stdoutTail, stderr);
+ // After the reconcile, which can upgrade TIMEOUT to COMPLETED.
+ settleReducer();
+ resolve();
+ return;
+ }
+ resolved = true;
- proc.on("exit", (code) => {
- const timedOut = statusCache.models[anonId]?.state === "TIMEOUT";
+ const outputSize = byteCount;
- // On TIMEOUT this handler must NOT settle the promise. "exit" fires
- // BEFORE the stdout pipe closes, and an answer flushed during shutdown
- // is still in flight at this moment — resolving here marked the run
- // finished with the byte count from KILL time and made the later
- // "close" a no-op, which is how a complete answer was reported as 0 B
- // and judged as an empty submission. Let "close" drive finish(), which
- // re-measures. `runModels` bounds the wait, so a pipe that never closes
- // degrades to a stale read rather than a hang.
- //
- // Still guard the error log: persistErrorLog has just written the
- // TIMEOUT diagnostics there and a raw stderr dump would erase them.
- // Only write a log when there is something worth reading. Every healthy
- // child emits Claude Code's `unrecognized_model` line — normal for a
- // proxied model — and writing it produced an `errors/NN.log` for runs
- // that had no error at all, which is exactly the noise that hides a real
- // one. `finish()` still writes the full log on any genuine failure.
- if (!timedOut && meaningfulStderr(stderr)) {
- // Redacted like every other persistence point — provider stderr can
- // echo key material and this file is read by agents.
- writeFileSync(errorLogPath, redactSecrets(stderr), "utf-8");
- }
+ // A non-zero exit is an outright failure. A zero exit still has to earn
+ // it: `claude -p` exits 0 on API errors and on background-task
+ // termination, so exit code alone would file both as success.
+ const crashed = exitCode !== 0;
+ const fullOutput = readFullOutputIfNeeded({
+ crashed,
+ requirePattern,
+ outputSize,
+ outputPath,
+ });
+ const degraded = crashed
+ ? null
+ : classifyRunOutput({
+ outputSize,
+ stdoutTail,
+ stderr,
+ minOutputBytes,
+ requirePattern,
+ fullOutput,
+ captureMode,
+ });
+
+ const failed = crashed || degraded !== null;
+ const state: ModelState = crashed ? "FAILED" : degraded ? "EMPTY" : "COMPLETED";
+
+ if (failed) {
+ // A slot the caller stopped exited non-zero, but it did not crash and
+ // saying so would be a lie in the permanent record. `cancelled` is the
+ // one failure reason that is not a defect: the caller looked at the
+ // evidence and decided. Nothing else in claudish can produce it.
+ const wasCancelled = cancelledSlots.has(anonId);
+ const reason: FailureReason = wasCancelled
+ ? "cancelled"
+ : crashed
+ ? "nonzero_exit"
+ : degraded!.reason;
+ const detail = wasCancelled
+ ? 'Stopped on the caller\'s instruction via team(mode:"cancel"). ' +
+ "Whatever the child had written up to that point is in its response file."
+ : crashed
+ ? `Child exited with code ${exitCode}.`
+ : degraded!.detail;
+
+ persistErrorLog(errorLogPath, `${state}: ${detail}`, stderr, stdoutTail);
+
+ updateModelStatus(anonId, {
+ state,
+ exitCode: exitCode ?? 1,
+ completedAt: new Date().toISOString(),
+ outputSize,
+ error: {
+ model: anonId,
+ command,
+ reason,
+ detail,
+ // Redacted: these land in status.json on disk and are read back
+ // by anything inspecting the run.
+ stderrSnippet: stderr ? redactSecrets(stderr).slice(-2000) : undefined,
+ stdoutSnippet: stdoutTail
+ ? snippetHeadAndTail(redactSecrets(stdoutTail))
+ : undefined,
+ errorLogPath,
+ // Only when the child actually wrote one. Naming a file that does
+ // not exist sends a reader after evidence that was never captured.
+ upstreamErrorLogPath: existsSync(upstreamErrorLogPath)
+ ? upstreamErrorLogPath
+ : undefined,
+ workDir: sessionPath,
+ },
+ });
+ } else {
+ updateModelStatus(anonId, {
+ state,
+ exitCode: exitCode ?? 0,
+ completedAt: new Date().toISOString(),
+ outputSize,
+ error: undefined,
+ });
+ }
- exitCode = code;
- // If the stream already closed before exit fired, finish immediately
- if (outputStream.destroyed) {
- finish();
- }
- // Otherwise wait for outputStream "close" to call finish()
+ // Before the caller's callback, so a throwing consumer cannot skip the
+ // dispose and leak the reducer's tool-batch timer.
+ settleReducer();
+ opts.onStatusChange?.(anonId, statusCache.models[anonId]);
+ resolve();
+ };
+
+ // "close" always fires after the stream ends or errors — single resolution point
+ outputStream.on("close", finish);
+
+ proc.on("exit", (code) => {
+ const timedOut = statusCache.models[anonId]?.state === "TIMEOUT";
+
+ // On TIMEOUT this handler must NOT settle the promise. "exit" fires
+ // BEFORE the stdout pipe closes, and an answer flushed during shutdown
+ // is still in flight at this moment — resolving here marked the run
+ // finished with the byte count from KILL time and made the later
+ // "close" a no-op, which is how a complete answer was reported as 0 B
+ // and judged as an empty submission. Let "close" drive finish(), which
+ // re-measures. `runModels` bounds the wait, so a pipe that never closes
+ // degrades to a stale read rather than a hang.
+ //
+ // Still guard the error log: persistErrorLog has just written the
+ // TIMEOUT diagnostics there and a raw stderr dump would erase them.
+ // Only write a log when there is something worth reading. Every healthy
+ // child emits Claude Code's `unrecognized_model` line — normal for a
+ // proxied model — and writing it produced an `errors/NN.log` for runs
+ // that had no error at all, which is exactly the noise that hides a real
+ // one. `finish()` still writes the full log on any genuine failure.
+ if (!timedOut && meaningfulStderr(stderr)) {
+ // Redacted like every other persistence point — provider stderr can
+ // echo key material and this file is read by agents.
+ writeFileSync(errorLogPath, redactSecrets(stderr), "utf-8");
+ }
+
+ exitCode = code;
+ // If the stream already closed before exit fired, finish immediately
+ if (outputStream.destroyed) {
+ finish();
+ }
+ // Otherwise wait for outputStream "close" to call finish()
+ });
});
- });
- processes.set(anonId, proc);
- completionPromises.push(completionPromise);
+ completionPromises.push(completionPromise);
+ }
+ } catch (err) {
+ // Nothing reported as failed to start may keep running or billing: signal
+ // every slot already spawned, by process group, exactly as the SIGINT
+ // handler does. `onSettled` is NOT called — no run started; the caller
+ // records the failure.
+ for (const [, proc] of processes) {
+ signalProcessTree(proc, "SIGTERM");
+ }
+ process.off("SIGINT", sigintHandler);
+ throw err;
}
// ── Live progress ─────────────────────────────────────────────────────────
@@ -1581,6 +1623,13 @@ export async function startModels(
// A settled run answers from status.json, not from memory. Dropping the
// entry is also what stops this map growing for the life of the server.
liveTeamRuns.delete(teamSessionId);
+ // The caller's settle hook, exactly once, after the settled render. Its
+ // own try, the `onProgress` convention: a consumer can never fail the run.
+ try {
+ opts.onSettled?.(statusCache);
+ } catch {
+ // A settle consumer must never be able to fail the run.
+ }
}
return statusCache;
})();
@@ -1678,6 +1727,77 @@ export function getStatus(sessionPath: string): TeamStatus {
return JSON.parse(readFileSync(join(sessionPath, "status.json"), "utf-8"));
}
+/**
+ * `<sessionPath>/status.json`, or a status with an EMPTY `models` map when it
+ * cannot be read. Never throws: it feeds the start-failure record, which must
+ * be written whatever state the run's directory is in.
+ */
+export function readTeamStatus(sessionPath: string): TeamStatus {
+ try {
+ const parsed = getStatus(sessionPath);
+ if (
+ parsed &&
+ typeof parsed === "object" &&
+ parsed.models &&
+ typeof parsed.models === "object"
+ ) {
+ return parsed;
+ }
+ } catch {
+ // Unreadable or partial — `status.json` writes are not atomic.
+ }
+ return { startedAt: new Date().toISOString(), models: {} };
+}
+
+/**
+ * How one `team(mode:"run")` ended, as its session-directory record states it
+ * (`<sessionsDir>/team-<8 hex>/meta.json`, written by `finishTeamRun`).
+ */
+export interface TeamRunOutcome {
+ status: "completed" | "failed" | "cancelled";
+ slots: number;
+ ok: number;
+ failed: number;
+ cancelled: number;
+ /** Present only when the run never started: `startModels` threw. */
+ reason?: "start-failed";
+}
+
+/**
+ * Counts and verdict for ANY `TeamStatus`, settled or not, so a run that failed
+ * partway still yields counts.
+ *
+ * - `COMPLETED` → ok.
+ * - `FAILED` / `EMPTY` / `TIMEOUT` whose `error.reason` is `cancelled` →
+ * cancelled; with any other reason → failed.
+ * - `PENDING` / `RUNNING` → failed. A settled run has none; in a run that
+ * failed to start they are the slots never started and the slots just killed.
+ *
+ * `slots` is the number of entries. `status` is `completed` when at least one
+ * slot is ok (the channel frame's rule), `cancelled` when every slot was
+ * cancelled, otherwise `failed`.
+ */
+export function summarise(status: TeamStatus): TeamRunOutcome {
+ const models = Object.values(status?.models ?? {});
+ let ok = 0;
+ let failed = 0;
+ let cancelled = 0;
+ for (const model of models) {
+ if (model?.state === "COMPLETED") ok++;
+ else if (
+ model?.error?.reason === "cancelled" &&
+ model.state !== "PENDING" &&
+ model.state !== "RUNNING"
+ ) {
+ cancelled++;
+ } else failed++;
+ }
+ const slots = models.length;
+ const verdict: TeamRunOutcome["status"] =
+ ok >= 1 ? "completed" : slots > 0 && cancelled === slots ? "cancelled" : "failed";
+ return { status: verdict, slots, ok, failed, cancelled };
+}
+
// ─── Internal Helpers ─────────────────────────────────────────────────────────
export function fisherYatesShuffle<T>(arr: T[]): T[] {