Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

refactor: stop reasons converge to four values (completed / aborted / retryable / fatal) (#458)

Yaowei Zheng573259f0965b
diff --git a/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.md b/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.md
new file mode 100644
index 00000000..8b9892f4
--- /dev/null
+++ b/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.md
@@ -0,0 +1,25 @@
+# Stop reasons converge to four values
+
+- **Date:** 2026-08-24
+- **Type:** refactor
+- **Scope:** `core`, `server`, `web`, `cli`
+- **Breaking:** yes — `StopReason` drops `failed` / `timeout` / `malformed` / `auth` in favor of `retryable` / `fatal` and becomes the only stop-reason vocabulary (`ToolStopReason` / `CompactionStatus` / `McpConnectStatus` are removed); `LLMOutcome.permanent` is removed; the Web composer's auth-dead gate is retired
+
+[中文版](2026-08-24-stop-reason-converges-to-four-values.zh.md)
+
+There is now exactly one stop-reason vocabulary: `completed` / `aborted` / `retryable` / `fatal`, used by every message and event that carries a stop reason — LLM fragments, tool outputs, compaction ends, MCP connect ends. A stop reason answers exactly one question — should this be retried — and the concrete failure rides on `error_message` (`LLMOutcome.errorMessage` → `request_end.error_message`) instead of on the reason value.
+
+**`retryable`** covers everything worth another attempt: transport drops, idle timeouts, 408/429/5xx, malformed or truncated responses, and every unclassifiable error — the fatal detector is a deterministic allowlist, so a gateway phrasing a transient fault its own way keeps its retries. The engine reconnects on the existing backoff ladder, unchanged.
+
+**`fatal`** covers what no retry can fix: a provider 4xx rejection (408/429 excluded), a credentials failure, fast mode on a model without a fast tier, and input that fails to assemble into a request. The engine stops the run and surfaces the message at once. Previously a deterministic 400 was classified `failed` and burned the whole five-step ladder before the user saw the actionable error — the direct motivation, uncovered while diagnosing the Vertex mixed-content 400.
+
+## Details
+
+- `LLMOutcome.permanent` is gone: `fatal` is the classification, not a flag on a classification. The engine's fatal branch replaces the former `auth` and permanent-failed branches.
+- The dedicated `auth` status is gone, and with it the Web composer's auth-dead input gate (the time-gated lock cleared by `credentials_updated`): a fatal's reason shows in the failure banner, and after fixing the key the user simply sends again. `credentials_updated` remains as an informational broadcast; the server-side runtime-cache invalidation it announced is unchanged.
+- Tool outputs rarely say `retryable`: a tool error or timeout is definitive for that call — nothing in the harness retries a tool — so failures converge to `fatal` and are fed back to the model as content for it to adjust. The separate `ToolStopReason` type is gone.
+- `compaction_end.status` distinguishes `retryable` — abandoned this time on exhausted retries, and the standing trigger makes it up at the next opportunity — from `fatal` — the attempt died on a failure no retry can fix, so the model configuration needs fixing first. The Web and CLI compaction banners word the two differently. The separate `CompactionStatus` type is gone.
+- `mcp_connect_end` (overall and per-server) joins the same vocabulary: a failed connect is `fatal` — nothing retries it within the run. The separate `McpConnectStatus` type is gone.
+- The server's error records reclassify: `llm_fatal` (unexpected), `llm_failed` (retries exhausted, unexpected), `llm_retried` (absorbed by the ladder, expected); tool failures file as `tool_fatal:<name>`. The old `llm_auth` / `llm_timeout` / `llm_malformed` / `llm_failed_retried` / `tool_failed:` / `tool_timeout:` codes are no longer produced. An exhausted retry run is now recorded as unexpected — previously an exhausted timeout still filed as expected.
+- The model connectivity probe passes any `retryable` ending that streamed real content (previously only the `malformed` shape did): content arriving is what a connectivity test measures.
+- Legacy Traces stay readable: replay only ever compares against `completed`, and the Web, CLI and trace analysis keep rendering the retired spellings (`failed` / `timeout` / `malformed` / `auth` on LLM fragments, `failed` / `timeout` on tool outputs, `failed` on compaction and MCP ends).
diff --git a/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.zh.md b/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.zh.md
new file mode 100644
index 00000000..bfb9de19
--- /dev/null
+++ b/changelog/unreleased/2026-08-24-stop-reason-converges-to-four-values.zh.md
@@ -0,0 +1,25 @@
+# stop_reason 收敛为四值
+
+- **Date:** 2026-08-24
+- **Type:** refactor
+- **Scope:** `core`, `server`, `web`, `cli`
+- **Breaking:** yes — `StopReason` 移除 `failed` / `timeout` / `malformed` / `auth`,改为 `retryable` / `fatal`,并成为唯一的停止原因词表(`ToolStopReason` / `CompactionStatus` / `McpConnectStatus` 删除);`LLMOutcome.permanent` 删除;Web 输入区的鉴权置灰门控退役
+
+[English](2026-08-24-stop-reason-converges-to-four-values.md)
+
+全站现在只有一套停止原因词表:`completed` / `aborted` / `retryable` / `fatal`——凡携带停止原因的消息与事件(LLM 分段、工具输出、compaction_end、mcp_connect_end)一律用它。停止原因只回答一个问题——该不该重试;具体错误一律由 `error_message` 承载(`LLMOutcome.errorMessage` → `request_end.error_message`),不再编码在原因值里。
+
+**`retryable`** 覆盖一切值得再试的失败:传输断开、空闲超时、408/429/5xx、响应不完整或解析异常,以及所有无法归类的错误——fatal 判定是确定性白名单,网关用自己的措辞表达瞬态故障时必须保住重试。引擎沿既有退避阶梯 reconnect,行为不变。
+
+**`fatal`** 覆盖重试无法修复的失败:Provider 的 4xx 拒绝(408/429 除外)、凭据错误、无 fast tier 的模型开启 fast mode、输入无法组装成请求。引擎直接停止并立即呈现错误信息。此前确定性的 400 被归为 `failed`、烧完整条五级阶梯用户才看到可行动的报错——这正是排查 Vertex 混合内容 400 时暴露的直接动因。
+
+## 细节
+
+- `LLMOutcome.permanent` 删除:`fatal` 本身就是分类,不是分类上的旗标。引擎的 fatal 分支取代了原先的 `auth` 分支与 permanent-failed 分支。
+- 独立的 `auth` 状态删除,随之退役的还有 Web 输入区的鉴权置灰门控(由 `credentials_updated` 解锁的时间门控):fatal 的原因经失败横幅呈现,改好 key 后直接再次发送即可。`credentials_updated` 保留为提示性广播;其宣告的服务端运行时缓存失效逻辑不变。
+- 工具输出一般不出现 `retryable`:工具错误或超时对该次调用是确定性的——harness 不重试工具——失败收敛为 `fatal`,作为内容回灌给模型自行调整。独立类型 `ToolStopReason` 删除。
+- `compaction_end.status` 区分 `retryable`(重试耗尽后本次放弃、保留原上下文,常设触发条件在下次机会自动补上)与 `fatal`(尝试死于不可重试失败,需先修复模型配置)。Web 与 CLI 的压缩提示对两者措辞相应区分。独立类型 `CompactionStatus` 删除。
+- `mcp_connect_end`(整体与逐 Server)并入同一词表:连接失败为 `fatal`——运行内无人重试。独立类型 `McpConnectStatus` 删除。
+- 服务端异常记录重划错误码:llm 侧 `llm_fatal`(unexpected)、`llm_failed`(重试耗尽,unexpected)、`llm_retried`(被阶梯吸收,expected);工具失败记为 `tool_fatal:<name>`。旧的 `llm_auth` / `llm_timeout` / `llm_malformed` / `llm_failed_retried` / `tool_failed:` / `tool_timeout:` 不再产生。重试耗尽现在记为 unexpected——此前耗尽的 timeout 仍按 expected 归档。
+- 模型连通性探测放宽:任何流出过真实内容的 `retryable` 结束都判通过(此前只有 `malformed` 形态享此待遇)——内容到达本身就是连通性测试要测的东西。
+- 旧 Trace 保持可读:重放判定只与 `completed` 比较,Web / CLI / 轨迹分析对废弃拼写保留渲染(LLM 分段的 `failed` / `timeout` / `malformed` / `auth`、工具输出的 `failed` / `timeout`、压缩与 MCP 收尾的 `failed`)。
diff --git a/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.md b/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.md
new file mode 100644
index 00000000..7f7efba5
--- /dev/null
+++ b/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.md
@@ -0,0 +1,14 @@
+# Abort marks user interruptions only
+
+- **Date:** 2026-08-25
+- **Type:** refactor
+- **Scope:** `core`, `server`, `web`, `cli`
+- **Breaking:** yes — the engine no longer emits an abort event for LLM or compaction failures (their terminal record is the `request_end` / `compaction_end` already on the stream), and error information converges on one `error_code` + `error_message` pair across payloads
+
+[中文版](2026-08-25-abort-marks-user-interruptions-only.zh.md)
+
+The abort event now means exactly one thing: the user interrupted the run. An LLM fatal, an exhausted retry ladder, and a failed mid-task compaction no longer emit one — each failure's terminal record is the event already on the stream: a `request_end` carrying its status and the error pair (and no `retry_in_ms` — a non-completed end without a planned retry **is** the run ending), or the `compaction_end` with its status and detail.
+
+Error information itself converges on one shape, omnimessage's `ErrorInfo`: a machine-readable `error_code` render layers localize from, plus the verbatim `error_message`. It rides on the abort payload (`user_abort` / `backoff_interrupted` / `compaction_interrupted` — the three user-interruption causes), on `request_end` and `compaction_end` (the classified LLM failure: `timeout` / `network` / `malformed` / `auth` / `rejected` / `unsupported` / `invalid_input`, originating as `LLMOutcome.errorCode` — the status answers "retry?", the code says what kind of error), and on `mcp_connect_end` and its per-server results (`connect_failed`; the old `error` field becomes `error_message`). Nothing parses prose any more: `parseAbortReason` is deleted, and a legacy Trace's `reason` text (or `error` field) is simply shown as-is.
+
+The frontends render failures from those records: the Web shows an error banner off a fatal `request_end`, settles the retry hint as given-up (final error attached) the moment a retryable end announces no retry, and words the reconnect cause from the code (`连接超时` vs `响应不完整` vs `网络或服务暂时不可用` — detail the four-value status alone could no longer express); the CLI prints the same lines. The server's error watcher resolves pending LLM failures at the next `request_begin` or at run close, composing the record prose from the event fields, so error-record text is unchanged without the abort. And how a run ended is now stated by the run generator's own **return value** (`RunCutoff | null`: `abort` / `llm_failure` / `compaction_failure` with the error pair, or `null` for ran-to-completion) — `ContextEngine.run`, `Session.run` and `SubagentHandle.run` propagate it, so the goal loop (deciding whether to re-fire a round) and the subagent round report (completed vs failed) read the fact the engine already knows instead of re-deriving it from the stream. A dead credential still stops a goal instead of re-firing rounds against it, and a failed subagent still reports `failed` to its parent.
diff --git a/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.zh.md b/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.zh.md
new file mode 100644
index 00000000..b831e2ec
--- /dev/null
+++ b/changelog/unreleased/2026-08-25-abort-marks-user-interruptions-only.zh.md
@@ -0,0 +1,14 @@
+# abort 只标记用户中断
+
+- **Date:** 2026-08-25
+- **Type:** refactor
+- **Scope:** `core`, `server`, `web`, `cli`
+- **Breaking:** yes — 引擎不再为 LLM 与压缩失败产出 abort 事件(其终局记录是流上已有的 `request_end` / `compaction_end`),错误信息收敛为跨载荷统一的 `error_code` + `error_message` 对
+
+[English](2026-08-25-abort-marks-user-interruptions-only.md)
+
+abort 事件从此只有一个含义:用户中断了本次运行。LLM fatal、重试阶梯耗尽、任务中途压缩失败都不再产出 abort——每种失败的终局记录就是流上已有的事件:携带 status 与错误对的 `request_end`(且无 `retry_in_ms`——没有计划重试的非 completed 收尾**就是**运行的终点),或带状态与明细的 `compaction_end`。
+
+错误信息本身收敛为一个形状,即 omnimessage 的 `ErrorInfo`:机器可读的 `error_code`(渲染层据此本地化)加原样展示的 `error_message`。它携带于 abort 载荷(`user_abort` / `backoff_interrupted` / `compaction_interrupted` 三种用户中断原因)、`request_end` 与 `compaction_end`(LLM 失败的分类原因:`timeout` / `network` / `malformed` / `auth` / `rejected` / `unsupported` / `invalid_input`,起源于 `LLMOutcome.errorCode`——status 回答"要不要重试",code 回答"是哪类错")、以及 `mcp_connect_end` 与其逐 Server results(`connect_failed`;原 `error` 字段更名为 `error_message`)。任何地方都不再解析散文:`parseAbortReason` 删除,旧 Trace 的 `reason` 文本(与 `error` 字段)按原样展示。
+
+前端从这些记录渲染失败:Web 对 fatal 的 `request_end` 出错误横幅,对宣布不再重试的 retryable 收尾当场把重试提示落为"已放弃"并附最终错误,重连提示的原因措辞由 code 驱动(「连接超时」/「响应不完整」/「网络或服务暂时不可用」——四值 status 本身已表达不了的细分);CLI 打印同样的行。服务端异常观察器把挂起的 LLM 失败在下一个 `request_begin` 或运行收束时落定,记录文本由事件字段拼成,没有 abort 事件也保持同样的记录文本。运行如何收束改由 run 生成器自己的**返回值**陈述(`RunCutoff | null`:`abort` / `llm_failure` / `compaction_failure` 携带错误对,`null` 即跑完)——`ContextEngine.run`、`Session.run`、`SubagentHandle.run` 逐层透传,goal 循环(判定是否续发轮次)与子会话轮次上报(完成或失败)直接读引擎本就知道的事实,不再从流内反推。坏掉的凭据依旧会终止 goal 而不是反复发起新轮,失败的子会话照常向父会话上报 `failed`。
diff --git a/changelog/unreleased/2026-08-25-agenthub-0-4-7.md b/changelog/unreleased/2026-08-25-agenthub-0-4-7.md
new file mode 100644
index 00000000..cf4f8340
--- /dev/null
+++ b/changelog/unreleased/2026-08-25-agenthub-0-4-7.md
@@ -0,0 +1,9 @@
+# AgentHub 0.4.7: Gemini function responses ride in their own contents
+
+- **Date:** 2026-08-25
+- **Type:** fix
+- **Scope:** `core`
+
+[中文版](2026-08-25-agenthub-0-4-7.zh.md)
+
+`@prismshadow/agenthub` moves to 0.4.7. The Gemini client there splits a message that mixes tool results with other content into consecutive same-role contents, fixing the Vertex AI 400 `Requests ending with a model turn are not supported.` that hit every request whose trailing user message carried a tool result together with text — an interrupted turn's carry-over resent with the next prompt, or tool outputs folded into a mid-task compaction request. A session that ran into it kept failing identically on every retry and resend until the context was discarded; with the split, those same requests go through, and the pnpm release-age exemption rolls forward to the new version.
diff --git a/changelog/unreleased/2026-08-25-agenthub-0-4-7.zh.md b/changelog/unreleased/2026-08-25-agenthub-0-4-7.zh.md
new file mode 100644
index 00000000..93d1b40c
--- /dev/null
+++ b/changelog/unreleased/2026-08-25-agenthub-0-4-7.zh.md
@@ -0,0 +1,9 @@
+# AgentHub 0.4.7:Gemini 的 function response 独立成 content
+
+- **Date:** 2026-08-25
+- **Type:** fix
+- **Scope:** `core`
+
+[English](2026-08-25-agenthub-0-4-7.md)
+
+`@prismshadow/agenthub` 升至 0.4.7。其中的 Gemini client 把工具结果与其他内容混排的消息拆分为连续的同 role content,修复了 Vertex AI 的 400 `Requests ending with a model turn are not supported.`——凡结尾 user 消息同时携带工具结果与文本的请求(被中断轮次的 carry-over 随下一条 prompt 重发、或工具输出折叠进 mid-task 压缩请求)都会命中。撞上它的会话此前每次重试与补发都一模一样地失败,直到丢弃上下文为止;拆分后同样的请求可以通过。pnpm 的发布时龄豁免随之滚动到新版本。
diff --git a/packages/cli/src/i18n.ts b/packages/cli/src/i18n.ts
index 6153f9ac..fbb798ea 100644
--- a/packages/cli/src/i18n.ts
+++ b/packages/cli/src/i18n.ts
@@ -193,14 +193,22 @@ export interface Messages {
elapsed: string;
elapsedDelta: string;
}): string;
- /** Abort event label (may include a reason). */
- abortLabel(reason?: string): string;
+ /** Abort line (user interruptions only): the cause localizes from `errorCode`; `errorMessage` (raw, untranslatable) rides verbatim; a legacy Trace without a code renders its English `reason` prose as-is. */
+ abortLabel(abort?: { errorCode?: string; errorMessage?: string; reason?: string }): string;
+ /** Run-ending LLM failure (request_end status fatal — no abort event follows); the provider's error text rides verbatim. */
+ llmFatalLabel(errorMessage?: string): string;
+ /** The retry ladder gave up (request_end `retryable` with no planned retry); `attempt` is the final attempt's ordinal, `errorMessage` the last failure's detail. */
+ reconnectGaveUpLabel(attempt: number, errorMessage?: string): string;
/**
- * request_end ended with a status the engine reconnects on (`failed` / `timeout` /
- * `malformed` — only `auth` is terminal): the engine retries carrying already-produced
- * content; attempt is the retry count.
+ * request_end ended with a status the engine reconnects on: the engine retries carrying
+ * already-produced content; attempt is the retry count. The cause wording localizes from
+ * `errorCode` (legacy Traces: the retired status spellings say the same thing).
*/
- reconnectLabel(status: "failed" | "timeout" | "malformed", attempt: number): string;
+ reconnectLabel(
+ status: "retryable" | "failed" | "timeout" | "malformed",
+ attempt: number,
+ errorCode?: string,
+ ): string;
/** compaction start event: indicates compaction in progress (mode is summarize/discard, reason is context/turns/manual). */
compactionStart(mode: string, reason: string): string;
/**
@@ -517,15 +525,35 @@ const en: Messages = {
approvePrompt: () => "? Approve this tool call? [Y/n] ",
taskStats: (s) =>
`[stats] context ${s.context} (${s.contextDelta}) · tokens ${s.tokens} (${s.tokensDelta}) · ${s.elapsed} (${s.elapsedDelta})`,
- abortLabel: (reason) => `[abort]${reason ? `: ${reason}` : ""}`,
- reconnectLabel: (status, attempt) =>
- `[retry] ${
- status === "timeout"
+ abortLabel: (abort) => {
+ const cause =
+ abort?.errorCode === "user_abort"
+ ? "aborted by user"
+ : abort?.errorCode === "backoff_interrupted"
+ ? "aborted during reconnect backoff"
+ : abort?.errorCode === "compaction_interrupted"
+ ? "aborted during compaction"
+ : (abort?.errorCode ?? abort?.reason ?? "");
+ const text = cause ? `${cause}${abort?.errorMessage ? `: ${abort.errorMessage}` : ""}` : "";
+ return `[abort]${text ? `: ${text}` : ""}`;
+ },
+ llmFatalLabel: (errorMessage) =>
+ `[error] llm request error${errorMessage ? `: ${errorMessage}` : ""}`,
+ reconnectGaveUpLabel: (attempt, errorMessage) =>
+ `[retry] giving up after attempt ${attempt}${errorMessage ? `: ${errorMessage}` : ""}`,
+ reconnectLabel: (status, attempt, errorCode) =>
+ // The live protocol carries the classified cause on error_code; the legacy status
+ // spellings say the same thing for pre-convergence Traces.
+ `[retry] ${((kind) =>
+ kind === "timeout"
? "connection timed out"
- : status === "malformed"
+ : kind === "malformed"
? "response incomplete or unparseable"
- : "the model provider returned an error"
- }; sending retry #${attempt}…`,
+ : kind === "network"
+ ? "network or service temporarily unavailable"
+ : kind === "failed"
+ ? "the model provider returned an error"
+ : "the request failed")(errorCode ?? status)}; sending retry #${attempt}…`,
compactionStart: (mode, reason) =>
mode === "discard"
? `[compaction] discarding context (${reason})…`
@@ -542,8 +570,17 @@ const en: Messages = {
? mode === "discard"
? "[compaction] done; old context discarded"
: "[compaction] done; continuing with the summarized context"
- : `[compaction] ${status}${errorMessage !== undefined ? ` (${errorMessage})` : ""}; keeping the current context`) +
- (tokens ? ` · tokens ${tokens.total} (${tokens.delta})` : ""),
+ : status === "aborted"
+ ? "[compaction] aborted; keeping the current context"
+ : `[compaction] failed${errorMessage !== undefined ? ` (${errorMessage})` : ""}; keeping the current context${
+ // retryable = abandoned this time, retried at the next trigger; fatal = a config
+ // or credential change has to come first. Legacy Traces spell both "failed".
+ status === "retryable"
+ ? "; retries at the next trigger"
+ : status === "fatal"
+ ? "; fix the model configuration to retry"
+ : ""
+ }`) + (tokens ? ` · tokens ${tokens.total} (${tokens.delta})` : ""),
compactNothing: () => "[compaction] nothing to compact yet",
clearDone: () => "[clear] started a fresh session (the previous one is kept and resumable)",
goalRound: (round) => `[goal] round ${round}`,
@@ -797,15 +834,32 @@ const zh: Messages = {
approvePrompt: () => "? 批准此工具调用?[Y/n] ",
taskStats: (s) =>
`[统计信息] 上下文 ${s.context} (${s.contextDelta}) · tokens ${s.tokens} (${s.tokensDelta}) · 用时 ${s.elapsed} (${s.elapsedDelta})`,
- abortLabel: (reason) => `[已中断]${reason ? `:${reason}` : ""}`,
- reconnectLabel: (status, attempt) =>
- `[重试] ${
- status === "timeout"
+ abortLabel: (abort) => {
+ const cause =
+ abort?.errorCode === "user_abort"
+ ? "用户中断"
+ : abort?.errorCode === "backoff_interrupted"
+ ? "重试等待中被中断"
+ : abort?.errorCode === "compaction_interrupted"
+ ? "压缩过程中被中断"
+ : (abort?.errorCode ?? abort?.reason ?? "");
+ const text = cause ? `${cause}${abort?.errorMessage ? `:${abort.errorMessage}` : ""}` : "";
+ return `[已中断]${text ? `:${text}` : ""}`;
+ },
+ llmFatalLabel: (errorMessage) => `[错误] 模型请求错误${errorMessage ? `:${errorMessage}` : ""}`,
+ reconnectGaveUpLabel: (attempt, errorMessage) =>
+ `[重试] 第 ${attempt} 次尝试后放弃${errorMessage ? `:${errorMessage}` : ""}`,
+ reconnectLabel: (status, attempt, errorCode) =>
+ `[重试] ${((kind) =>
+ kind === "timeout"
? "连接超时或网络中断"
- : status === "malformed"
+ : kind === "malformed"
? "响应不完整或无法解析"
- : "模型服务返回错误"
- },正在发起第 ${attempt} 次重试……`,
+ : kind === "network"
+ ? "网络或服务暂时不可用"
+ : kind === "failed"
+ ? "模型服务返回错误"
+ : "请求失败")(errorCode ?? status)},正在发起第 ${attempt} 次重试……`,
compactionStart: (mode, reason) =>
mode === "discard"
? `[压缩] 正在丢弃旧上下文(${reason})……`
@@ -822,8 +876,16 @@ const zh: Messages = {
? mode === "discard"
? "[压缩] 完成,旧上下文已丢弃"
: "[压缩] 完成,已切换到摘要后的新上下文"
- : `[压缩] ${status === "aborted" ? "已中断" : `失败${errorMessage !== undefined ? `(${errorMessage})` : ""}`},保留当前上下文`) +
- (tokens ? ` · tokens ${tokens.total} (${tokens.delta})` : ""),
+ : status === "aborted"
+ ? "[压缩] 已中断,保留当前上下文"
+ : `[压缩] 失败${errorMessage !== undefined ? `(${errorMessage})` : ""},保留当前上下文${
+ // retryable = 本次放弃、下次触发自动重试;fatal = 需先修复模型配置或凭据。旧 Trace 两者都拼作 "failed"。
+ status === "retryable"
+ ? ",下次触发时重试"
+ : status === "fatal"
+ ? ",需修复模型配置后重试"
+ : ""
+ }`) + (tokens ? ` · tokens ${tokens.total} (${tokens.delta})` : ""),
compactNothing: () => "[压缩] 当前上下文为空,无需压缩",
clearDone: () => "[清空] 已开启全新 Session(原会话仍保留,可恢复)",
goalRound: (round) => `[目标] 第 ${round} 轮`,
diff --git a/packages/cli/src/render.ts b/packages/cli/src/render.ts
index d79c16de..5be3b327 100644
--- a/packages/cli/src/render.ts
+++ b/packages/cli/src/render.ts
@@ -186,7 +186,14 @@ function humanizeDuration(ms: number): string {
}
export function formatAbort(p: AbortPayload, t: Messages, c: Palette = STDOUT_PALETTE): string {
- return dim(t.abortLabel(p.reason ?? undefined), c);
+ return dim(
+ t.abortLabel({
+ ...(p.error_code !== undefined ? { errorCode: p.error_code } : {}),
+ ...(p.error_message !== undefined ? { errorMessage: p.error_message } : {}),
+ ...(p.reason != null ? { reason: p.reason } : {}),
+ }),
+ c,
+ );
}
/** Options shared by history and streaming rendering. */
@@ -457,8 +464,10 @@ export class StreamRenderer {
*/
private taskFirstTsMs: number | null = null;
private taskLastReqEndMs: number | null = null;
- /** Retryable terminal state (failed/timeout/malformed) of the previous request: the next request_begin is a retry, at which point a notice is printed. */
- private pendingRetry: "failed" | "timeout" | "malformed" | null = null;
+ /** Retryable terminal state of the previous request (legacy Traces carry the finer pre-convergence spellings): the next request_begin is a retry, at which point a notice is printed. */
+ private pendingRetry: "retryable" | "failed" | "timeout" | "malformed" | null = null;
+ /** The pending failure's classified cause (request_end.error_code), for the retry line's wording. */
+ private pendingRetryCode: string | undefined;
/** The pending failure's attempt ordinal (request_end.attempt — the core's authoritative count, printed on the retry line). */
private pendingRetryAttempt: number | undefined;
@@ -780,9 +789,11 @@ export class StreamRenderer {
this.out.write(`${dim(this.t.approvalDecision(p.decision), this.c)}\n`);
this.lastLineKey = null;
} else if (payload.type === "abort") {
- // Run ended (user interrupt / retries exhausted): clear any pending retry state so the next run doesn't mistakenly print a retry line.
+ // Run ended (a user interruption — failures end on their own terminal records):
+ // clear any pending retry state so the next run doesn't mistakenly print a retry line.
this.pendingRetry = null;
this.pendingRetryAttempt = undefined;
+ this.pendingRetryCode = undefined;
this.finishLine();
this.out.write(`${formatAbort(payload as AbortPayload, this.t, this.c)}\n`);
this.lastLineKey = null;
@@ -797,9 +808,12 @@ export class StreamRenderer {
const attempt = this.pendingRetryAttempt ?? 1;
this.pendingRetryAttempt = undefined;
this.finishLine();
- this.out.write(`${dim(this.t.reconnectLabel(this.pendingRetry, attempt), this.c)}\n`);
+ this.out.write(
+ `${dim(this.t.reconnectLabel(this.pendingRetry, attempt, this.pendingRetryCode), this.c)}\n`,
+ );
this.lastLineKey = null;
this.pendingRetry = null;
+ this.pendingRetryCode = undefined;
}
} else if (payload.type === "request_end") {
const p = payload as RequestEndPayload;
@@ -817,15 +831,44 @@ export class StreamRenderer {
this.hasUsage = true;
}
}
- // Every status the engine reconnects on, `failed` included — only `auth` is terminal.
- // Leaving `failed` out would print nothing for a retry that is really happening, and
- // reset the counter mid-ladder so a mixed run renumbers back to retry #1.
- if (p.status === "failed" || p.status === "timeout" || p.status === "malformed") {
- this.pendingRetry = p.status;
+ const endStatus = p.status as string;
+ if (endStatus === "fatal") {
+ // A fatal end stops the run; no abort event follows — this request_end is the
+ // terminal record, printed here.
+ this.pendingRetry = null;
+ this.pendingRetryAttempt = undefined;
+ this.pendingRetryCode = undefined;
+ this.finishLine();
+ this.out.write(`${dim(this.t.llmFatalLabel(p.error_message), this.c)}\n`);
+ this.lastLineKey = null;
+ } else if (endStatus === "retryable" && p.retry_in_ms === undefined) {
+ // A live-protocol retryable with no planned wait is the ladder giving up — the
+ // run ends on it. The legacy spellings never take this branch: their era stamped
+ // no retry_in_ms mid-ladder, and their exhaustion printed from the abort event.
+ this.pendingRetry = null;
+ this.pendingRetryAttempt = undefined;
+ this.pendingRetryCode = undefined;
+ this.finishLine();
+ this.out.write(
+ `${dim(this.t.reconnectGaveUpLabel(p.attempt ?? 1, p.error_message), this.c)}\n`,
+ );
+ this.lastLineKey = null;
+ } else if (
+ endStatus === "retryable" ||
+ endStatus === "failed" ||
+ endStatus === "timeout" ||
+ endStatus === "malformed"
+ ) {
+ // Every status the engine reconnects on gets the retry notice (printed when the
+ // retry actually starts); the legacy spellings keep pre-convergence Traces
+ // printing the same way.
+ this.pendingRetry = endStatus;
this.pendingRetryAttempt = p.attempt;
+ this.pendingRetryCode = p.error_code;
} else {
this.pendingRetry = null;
this.pendingRetryAttempt = undefined;
+ this.pendingRetryCode = undefined;
}
} else if (payload.type === "compaction_begin") {
// Paired compaction events: begin signals compaction is in progress.
@@ -875,8 +918,12 @@ export class StreamRenderer {
// One line, reasons included: the failure reason (spawn error, timeout, HTTP
// status) otherwise never reaches the terminal — only the server-side stderr warning.
const failures = p.results
- .filter((r) => r.status === "failed")
- .map((r) => ({ server: r.server, error: r.error ?? "unknown error" }));
+ .filter((r) => r.status === "fatal" || (r.status as string) === "failed")
+ .map((r) => ({
+ server: r.server,
+ // Live results carry the unified pair; legacy Traces spell the detail `error`.
+ error: r.error_message ?? (r as { error?: string }).error ?? "unknown error",
+ }));
this.out.write(
`${dim(this.t.mcpConnectStop(durationMs, failures, p.status === "aborted"), this.c)}\n`,
);
diff --git a/packages/cli/test/render.test.ts b/packages/cli/test/render.test.ts
index 61e8924f..bfae8af5 100644
--- a/packages/cli/test/render.test.ts
+++ b/packages/cli/test/render.test.ts
@@ -85,9 +85,21 @@ describe("pure formatters", () => {
it("renderHistory includes abort events from resumed sessions", () => {
const { stream, text } = collector();
- renderHistory([assistantText("partial", "aborted"), abortEvent("aborted by user")], stream, t);
+ renderHistory([assistantText("partial", "aborted"), abortEvent()], stream, t);
expect(stripAnsi(text())).toBe("partial [aborted]\n[abort]: aborted by user\n");
});
+
+ it("a legacy abort without an error_code renders its reason prose verbatim", () => {
+ const { stream, text } = collector();
+ const legacy = abortEvent();
+ delete (legacy.payload as { error_code?: string }).error_code;
+ (legacy.payload as { reason?: string }).reason =
+ "llm request error: 401 Missing Authentication header";
+ renderHistory([legacy], stream, t);
+ expect(stripAnsi(text())).toBe(
+ "[abort]: llm request error: 401 Missing Authentication header\n",
+ );
+ });
});
describe("StreamRenderer", () => {
@@ -302,47 +314,61 @@ describe("StreamRenderer", () => {
const { stream, text } = collector();
const r = new StreamRenderer(stream, t);
r.handle(requestBegin());
- r.handle(requestEnd("malformed", { attempt: 1 }));
+ r.handle(requestEnd("retryable", { attempt: 1, retryInMs: 2000 }));
expect(stripAnsi(text())).toBe(""); // the failure itself prints nothing; only the retry's start does
r.handle(requestBegin()); // retry #1 begins
expect(stripAnsi(text())).toContain("retry #1");
- r.handle(requestEnd("timeout", { attempt: 2 }));
+ r.handle(requestEnd("retryable", { attempt: 2, retryInMs: 4000 }));
r.handle(requestBegin()); // retry #2 begins
expect(stripAnsi(text())).toContain("retry #2");
- // Retry #2 fails again and retries are exhausted: no next request_begin, only abort — no retry #3 appears.
- r.handle(requestEnd("malformed", { attempt: 3 }));
- r.handle(abortEvent("malformed response failed after 2 retries"));
+ // Retry #2 fails again and retries are exhausted: the terminal request_end (no
+ // retry planned) prints the give-up line itself — no abort event follows.
+ r.handle(requestEnd("retryable", { attempt: 3, errorMessage: "socket hang up" }));
+ expect(stripAnsi(text())).toContain("giving up after attempt 3: socket hang up");
expect(stripAnsi(text())).not.toContain("retry #3");
// The first request of the next run is not a retry, so it prints nothing; the next run's
// failures are stamped from 1 again.
r.handle(requestBegin());
- r.handle(requestEnd("timeout", { attempt: 1 }));
+ r.handle(requestEnd("retryable", { attempt: 1, retryInMs: 2000 }));
r.handle(requestBegin());
const lines = stripAnsi(text());
expect(lines.match(/retry #1/g)).toHaveLength(2);
expect(lines).not.toContain("retry #3");
});
- it("prints the retry line for a failed request too, straight from the stamped ordinal", () => {
- // The engine reconnects on `failed` as well, so the CLI has to say so — otherwise the
- // session goes quiet for the whole ladder. The number is the event's own attempt, so a
- // mixed timeout → failed → timeout run keeps counting without any client-side state.
+ it("a fatal request_end prints the error line; an interim-build abort duplicate is dropped", () => {
+ const { stream, text } = collector();
+ const r = new StreamRenderer(stream, t);
+ r.handle(requestBegin());
+ r.handle(requestEnd("fatal", { errorCode: "auth", errorMessage: "401 invalid x-api-key" }));
+ expect(stripAnsi(text())).toBe("[error] llm request error: 401 invalid x-api-key\n");
+ // A user abort afterwards still prints.
+ r.handle(abortEvent());
+ expect(stripAnsi(text())).toContain("[abort]: aborted by user");
+ });
+
+ it("prints the retry line for legacy-trace spellings too, straight from the stamped ordinal", () => {
+ // Traces written before the stop-reason convergence spell retryable ends as
+ // failed/timeout/malformed; the CLI keeps printing their retries. The number is the
+ // event's own attempt, so a mixed run keeps counting without any client-side state.
const { stream, text } = collector();
const r = new StreamRenderer(stream, t);
r.handle(requestBegin());
- r.handle(requestEnd("failed", { errorMessage: "Upstream HTTP/2 stream failed", attempt: 1 }));
+ const legacyEnd = (status: string, retry?: { attempt?: number; errorMessage?: string }) =>
+ requestEnd(status as Parameters<typeof requestEnd>[0], retry);
+ r.handle(legacyEnd("failed", { errorMessage: "Upstream HTTP/2 stream failed", attempt: 1 }));
r.handle(requestBegin()); // retry #1 begins
let lines = stripAnsi(text());
expect(lines).toContain("the model provider returned an error");
expect(lines).toContain("retry #1");
- r.handle(requestEnd("timeout", { attempt: 2 }));
+ r.handle(legacyEnd("timeout", { attempt: 2 }));
r.handle(requestBegin()); // retry #2 begins — the count does not restart
lines = stripAnsi(text());
expect(lines).toContain("connection timed out");
expect(lines).toContain("retry #2");
- // `auth` is terminal: the engine never retries it, so the next request_begin (a new run)
- // must not be announced as a retry.
- r.handle(requestEnd("auth", { errorMessage: "401 invalid x-api-key", attempt: 3 }));
+ // `fatal` is terminal: the engine never retries it, so the next request_begin (a new
+ // run) must not be announced as a retry.
+ r.handle(requestEnd("fatal", { errorMessage: "401 invalid x-api-key", attempt: 3 }));
r.handle(requestBegin());
expect(stripAnsi(text())).not.toContain("retry #3");
});
@@ -486,14 +512,14 @@ describe("StreamRenderer", () => {
r.handle(compactionEnd({ reason: "context", mode: "summarize", status: "completed" }));
r.handle(compactionBegin({ reason: "manual", mode: "discard", context: 10, turns: 1 }));
r.handle(compactionEnd({ reason: "manual", mode: "discard", status: "completed" }));
- r.handle(compactionEnd({ reason: "context", mode: "summarize", status: "failed" }));
+ r.handle(compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }));
expect(stripAnsi(text())).toBe(
[
"[compaction] summarizing context (context)…",
"[compaction] done; continuing with the summarized context",
"[compaction] discarding context (manual)…",
"[compaction] done; old context discarded",
- "[compaction] failed; keeping the current context",
+ "[compaction] failed; keeping the current context; retries at the next trigger",
"",
].join("\n"),
);
diff --git a/packages/cli/test/task-loop.test.ts b/packages/cli/test/task-loop.test.ts
index 6e22bc1d..850b1f6c 100644
--- a/packages/cli/test/task-loop.test.ts
+++ b/packages/cli/test/task-loop.test.ts
@@ -33,7 +33,7 @@ function silentRenderer(): StreamRenderer {
describe("runTask abort reporting", () => {
it("reports aborted=true when the task ends with a main-session abort event", async () => {
- const result = await runTask(fakeSession([abortEvent("llm request error: 401")]), [], {
+ const result = await runTask(fakeSession([abortEvent()]), [], {
renderer: silentRenderer(),
t,
});
@@ -42,7 +42,7 @@ describe("runTask abort reporting", () => {
it("reports aborted=false on normal completion; child-session aborts do not count", async () => {
const result = await runTask(
- fakeSession([withOrigin(abortEvent("child aborted"), "sess_child"), assistantText("done")]),
+ fakeSession([withOrigin(abortEvent(), "sess_child"), assistantText("done")]),
[],
{ renderer: silentRenderer(), t },
);
diff --git a/packages/core/package.json b/packages/core/package.json
index 7e7e3c64..4d5c95a0 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -49,7 +49,7 @@
},
"dependencies": {
"@modelcontextprotocol/client": "^2.0.0",
- "@prismshadow/agenthub": "^0.4.6",
+ "@prismshadow/agenthub": "^0.4.7",
"@prismshadow/penguin-skills": "workspace:*",
"arktype": "^2.2.3",
"smol-toml": "^1.3.0",
diff --git a/packages/core/src/agent.ts b/packages/core/src/agent.ts
index 21007cc5..2d1d776e 100644
--- a/packages/core/src/agent.ts
+++ b/packages/core/src/agent.ts
@@ -505,7 +505,7 @@ export class Agent {
// must not block the resume itself.
if (resumed.danglingCompaction) {
try {
- await trace.write(compactionEnd({ ...resumed.danglingCompaction, status: "failed" }));
+ await trace.write(compactionEnd({ ...resumed.danglingCompaction, status: "retryable" }));
} catch (err) {
const detail = err instanceof Error ? err.message : String(err);
process.stderr.write(`[trace] interrupted-compaction closure failed: ${detail}\n`);
@@ -770,12 +770,17 @@ export class Agent {
// dispatch, none for a human's message from a host panel — so the child's Trace
// records who actually spoke.
for (const input of messages) yield withOrigin(input, hop);
- for await (const msg of childSession.run(messages, {
+ // Manual iteration so the child run's return value — whether the round was cut
+ // off — propagates to the handle's own return for the parent's round report.
+ const it = childSession.run(messages, {
...(signal ? { signal } : {}),
...(childApprove ? { approve: childApprove } : {}),
...(turnThinkingLevel !== undefined ? { thinkingLevel: turnThinkingLevel } : {}),
- })) {
- yield withOrigin(msg, hop);
+ });
+ for (;;) {
+ const res = await it.next();
+ if (res.done) return res.value;
+ yield withOrigin(res.value, hop);
}
},
// Mid-run steering rides the child Session's own steering queue — the same
diff --git a/packages/core/src/engine/context-engine.ts b/packages/core/src/engine/context-engine.ts
index 99f328b9..977d18fb 100644
--- a/packages/core/src/engine/context-engine.ts
+++ b/packages/core/src/engine/context-engine.ts
@@ -60,6 +60,7 @@ import {
userSteeringText,
} from "../omnimessage/markers/index.js";
import type {
+ ErrorCode,
ApprovalDecision,
CompactionMode,
CompactionReason,
@@ -74,6 +75,7 @@ import type {
ToolCallPayload,
} from "../omnimessage/index.js";
import type {
+ RunCutoff,
ApproveFn,
EnvironmentInterface,
LLMInterface,
@@ -102,14 +104,17 @@ export interface CompactionSettings {
prompt: string;
}
-/** Result of one compaction run: status is a terminal state (completed / failed / aborted); carries the summary message when summarize succeeds. */
+/** Result of one compaction run: a StopReason terminal state (completed / aborted / retryable — abandoned, made up at the next trigger / fatal — needs a config change first); carries the summary message when summarize succeeds. */
interface CompactionResult {
status: StopReason;
summary?: OmniMessage;
+ /** The failure's classified cause and detail (mirrors the compaction_end's error pair) — present on `retryable` / `fatal` ends. */
+ errorCode?: ErrorCode;
+ errorMessage?: string;
/**
* Whether at least one summarize attempt was **committed** by AgentHub (only a `completed`
- * attempt commits — timeout/malformed end an incomplete stream and failed/auth/aborted throw
- * or cut off before a clean end). The carry rule at every caller is a two-case binary on
+ * attempt commits — a `retryable` attempt ends an incomplete stream and fatal/aborted
+ * throw or cut off before a clean end). The carry rule at every caller is a two-case binary on
* this flag (issue #85): committed → the input the caller folded in (mid-Task tool outputs,
* or the carry-over a manual `compact()` folds in) now lives in the old LLM object's history
* and must never be resent — strict providers reject the duplicates as stale tool_results;
@@ -193,7 +198,7 @@ export interface ContextEngineDeps {
/**
* Maximum retries for a failing compaction request — one budget for every failure kind:
* the transport statuses (see RETRY_STATUSES) and a committed response that isn't a usable
- * summary (empty, or tool calls) all draw from it; only `auth` stops without retrying.
+ * summary (empty, or tool calls) all draw from it; only `fatal` stops without retrying.
* Defaults to the shared `maxReconnects`: a compaction request is an ordinary LLM request
* and deserves the same patience (issue #170 — the earlier tighter budget made a
* struggling provider fail compaction fast, and a session whose every turn re-triggers
@@ -406,13 +411,11 @@ function downgradeCarriedGoalInput(msg: OmniMessage): OmniMessage {
}
/**
- * LLM outcomes that reconnect in-run — everything but `auth`, which cannot be retried into
- * working, and a `failed` carrying `permanent: true` (a deterministic client-side rejection,
- * see LLMOutcome.permanent), which the turn loop aborts on before consulting this set.
- * `failed` is otherwise included on purpose: the classifier producing it is an allowlist of
- * known transport codes and message vocabulary, so a gateway phrasing a transient fault its
- * own way lands here; retrying a genuinely permanent error costs the ladder and ends the same
- * way, while aborting a transient one destroys the turn.
+ * LLM outcomes that reconnect in-run: exactly `retryable`. The classification lives in the
+ * LLM interface (see LLMOutcome) — transport drops, timeouts, 408/429/5xx, malformed
+ * responses and everything unclassifiable come back `retryable`, while definitive provider
+ * rejections and credential failures come back `fatal` and the turn loop aborts on them
+ * before consulting this set.
*
* Both loops retry the same set on the same backoff ladder, and by default with the same
* budget: compaction runs on `compactionMaxReconnects`, which follows `maxReconnects` unless
@@ -420,7 +423,7 @@ function downgradeCarriedGoalInput(msg: OmniMessage): OmniMessage {
* turn loop's patience; compaction additionally routes a committed-but-unusable summary
* through the same budget, see summarizeContext).
*/
-const RETRY_STATUSES: readonly StopReason[] = ["failed", "timeout", "malformed"];
+const RETRY_STATUSES: readonly StopReason[] = ["retryable"];
/**
* Delay before reconnect attempt N (1-based): exponential growth from `base` with a hard
@@ -534,12 +537,17 @@ export class ContextEngine {
* per-tool approval callback.
* Docs: /docs/agent-loop § "The loop at a glance".
*/
- async *run(newMessages: OmniMessage[], opts?: RunOptions): AsyncGenerator<OmniMessage> {
+ async *run(
+ newMessages: OmniMessage[],
+ opts?: RunOptions,
+ ): AsyncGenerator<OmniMessage, RunCutoff | null> {
// Steering window: only while this generator is being driven. The finally also covers
// abort/failure exits — anything still queued is discarded (see steeringQueue).
this.taskRunning = true;
try {
- yield* this.runToCompletion(newMessages, opts);
+ // The return value says how the run ended (yield* propagates it): null = ran to
+ // completion; a RunCutoff = ended early, with the terminal record's error pair.
+ return yield* this.runToCompletion(newMessages, opts);
} finally {
this.taskRunning = false;
this.steeringQueue = [];
@@ -647,7 +655,7 @@ export class ContextEngine {
private async *runToCompletion(
newMessages: OmniMessage[],
opts?: RunOptions,
- ): AsyncGenerator<OmniMessage> {
+ ): AsyncGenerator<OmniMessage, RunCutoff | null> {
const signal = opts?.signal;
// Default approval policy: deny (conservative). CLI/Web will inject a real callback (interactive or permission-mode based).
const approve: ApproveFn = opts?.approve ?? (async () => "deny");
@@ -689,8 +697,8 @@ export class ContextEngine {
// multimodal input isn't lost. The message is already written to Trace, so it won't be
// rewritten on the next send.
this.pendingCarryOver = input;
- yield* this.emitAbort("aborted by user");
- return;
+ yield* this.emitAbort("user_abort");
+ return { kind: "abort", errorCode: "user_abort" };
}
let turnCount = 0;
@@ -715,15 +723,15 @@ export class ContextEngine {
// (400, see issue #33).
this.pendingCarryOver = nextInput;
yield* this.emitMaxTurns();
- return;
+ return null;
}
turnCount += 1;
// This turn's input. The safety invariant behind resending it: **no retryable attempt
// is ever committed to AgentHub's history**. AgentHub appends a turn to `_history` only
// after its stream has been consumed to the end and validated, so every abnormal exit —
- // `timeout`, `malformed` and `failed` alike, whether the stream was cut, the payload
- // failed to parse, or the request was rejected outright — leaves history untouched.
+ // whether the stream was cut, the payload failed to parse, or the request was rejected
+ // outright — leaves history untouched.
// Nothing can therefore be duplicated or left as an unanswered tool_use by a reconnect;
// that is what makes it safe to resend this turn's input unchanged, appending a
// `[turn_retried]` block carrying what the failed attempt already produced — the model
@@ -748,70 +756,59 @@ export class ContextEngine {
// during tool execution): stop and hand control back to the user.
if (signal?.aborted || turn.outcome.status === "aborted") {
this.pendingCarryOver = this.buildCarryOver(attemptInput, turn);
- yield* this.emitAbort("aborted by user");
- return;
+ yield* this.emitAbort("user_abort");
+ return { kind: "abort", errorCode: "user_abort" };
}
- // `auth` is the one status that stops the run: a rejected credential cannot be retried
- // into working, and hosts read it off the turn's request_end to gate input. Everything
- // else — `failed` included — goes to the reconnect loop below.
- if (turn.outcome.status === "auth") {
+ // `fatal` stops the run: a definitive provider rejection, a dead credential, or a
+ // deterministic client-side rejection — the identical request can never succeed, so
+ // the ladder would only delay the actionable message the outcome already carries.
+ // No abort event: abort marks a user interruption, and the already-written
+ // request_end (status `fatal`, `error_message`, no `retry_in_ms`) is the terminal
+ // record frontends and observability read.
+ if (turn.outcome.status === "fatal") {
this.pendingCarryOver = this.buildCarryOver(attemptInput, turn);
- yield* this.emitAbort(`llm request error: ${turn.outcome.errorMessage ?? "unknown"}`);
- return;
- }
- // A permanent `failed` is the other non-retried terminal: a deterministic
- // client-side rejection thrown before any network I/O (fast_mode on a model
- // without a fast tier — see LLMOutcome.permanent), where the identical request can
- // never succeed and the fix is a config change. The general retry-everything
- // rationale below doesn't apply: this isn't a gateway phrasing a transient fault
- // its own way, it's AgentHub's own typed error, so the ladder would only delay the
- // actionable message.
- if (turn.outcome.status === "failed" && turn.outcome.permanent) {
- this.pendingCarryOver = this.buildCarryOver(attemptInput, turn);
- yield* this.emitAbort(`llm request error: ${turn.outcome.errorMessage ?? "unknown"}`);
- return;
+ return {
+ kind: "llm_failure",
+ ...(turn.outcome.errorCode !== undefined ? { errorCode: turn.outcome.errorCode } : {}),
+ ...(turn.outcome.errorMessage !== undefined
+ ? { errorMessage: turn.outcome.errorMessage }
+ : {}),
+ };
}
// Completed normally.
if (turn.outcome.status === "completed") break;
- // failed / timeout / malformed remain: reconnect automatically within the same run.
- //
- // `failed` retries too, even though the classifier judged it non-transient. That
- // judgement is a hint, not a verdict: it is an allowlist of known network codes,
- // statuses and message vocabulary, so every gateway that phrases a transient failure
- // its own way falls through it — `Upstream HTTP/2 stream failed
- // (upstream_http2_stream_error)` is plainly a transport fault and matched nothing.
- // Retrying a genuinely permanent error costs the ladder and then ends the same way;
- // aborting a transient one destroys the turn. So the *classification* stays honest
- // (`failed` is still recorded as a real failure, not relabelled a timeout) while the
- // *policy* retries it. Only `auth` is terminal, above.
+ // `retryable` remains: reconnect automatically within the same run. The class is
+ // deliberately wide — the LLM interface sends every unclassifiable error here,
+ // because its fatal detector is an allowlist and a gateway phrasing a transient
+ // fault its own way (`Upstream HTTP/2 stream failed (upstream_http2_stream_error)`)
+ // must keep its retries; the concrete failure rides on errorMessage.
//
// When retries are exhausted or the backoff is interrupted, the retry input is held
// as-is as carry-over (the original input is already written to Trace, so it isn't
// rewritten). The frontend surfaces the retry process and count via
- // request_end(failed|timeout|malformed) followed by the next request_begin.
+ // request_end(retryable) followed by the next request_begin.
failedTurns.push(turn);
attemptInput = this.withRetriedTurns(nextInput, failedTurns);
+ // Exhausted: give up without an abort event — abort marks a user interruption,
+ // and the last failure's request_end (status `retryable` with no `retry_in_ms`,
+ // since no retry is planned) is the terminal record frontends and observability
+ // read; `attempt` and `error_message` ride on it.
if (reconnects >= this.maxReconnects) {
this.pendingCarryOver = attemptInput;
- // This reason is user-visible (the Web App's error panel, the CLI's abort line) and
- // is what observability persists as the error message, so it has to read as a
- // sentence: what gave out, then how many attempts it took, then — for the one class
- // that carries provider words worth showing — the detail, last.
- const reason =
- turn.outcome.status === "malformed"
- ? `malformed response failed after ${this.maxReconnects} retries`
- : turn.outcome.status === "failed"
- ? `llm request failed after ${this.maxReconnects} retries: ${turn.outcome.errorMessage ?? "unknown"}`
- : `reconnect failed after ${this.maxReconnects} retries`;
- yield* this.emitAbort(reason);
- return;
+ return {
+ kind: "llm_failure",
+ ...(turn.outcome.errorCode !== undefined ? { errorCode: turn.outcome.errorCode } : {}),
+ ...(turn.outcome.errorMessage !== undefined
+ ? { errorMessage: turn.outcome.errorMessage }
+ : {}),
+ };
}
reconnects += 1;
if (!(await this.backoff(reconnects, signal))) {
this.pendingCarryOver = attemptInput;
- yield* this.emitAbort("aborted during reconnect backoff");
- return;
+ yield* this.emitAbort("backoff_interrupted");
+ return { kind: "abort", errorCode: "backoff_interrupted" };
}
}
@@ -828,7 +825,7 @@ export class ContextEngine {
// The queues are only peeked here — delivery happens at the input assembly below.
if (!midTask && this.steeringQueue.length === 0 && this.pendingNoticeCount() === 0) {
yield* this.discardContext(compactionReason);
- return;
+ return null;
}
} else {
const result = yield* this.summarizeContext(
@@ -845,7 +842,7 @@ export class ContextEngine {
// Task boundary: the summary is merged with the next user Prompt as the new
// context's first input.
this.pendingSummary = result.summary!;
- return;
+ return null;
}
// Mid-Task: the summary itself becomes the new LLM object's first input (this
// turn's tool results were already folded into the compaction request and absorbed
@@ -879,15 +876,22 @@ export class ContextEngine {
...this.buildCarryOver(attemptInput, turn),
];
}
- yield* this.emitAbort(
- result.status === "aborted" ? "aborted during compaction" : "compaction failed",
- );
- return;
+ // Only the user interruption is an abort; a failed compaction's terminal
+ // record is the compaction_end (status + error_message) already written.
+ if (result.status === "aborted") {
+ yield* this.emitAbort("compaction_interrupted");
+ return { kind: "abort", errorCode: "compaction_interrupted" };
+ }
+ return {
+ kind: "compaction_failure",
+ ...(result.errorCode !== undefined ? { errorCode: result.errorCode } : {}),
+ ...(result.errorMessage !== undefined ? { errorMessage: result.errorMessage } : {}),
+ };
}
// Task boundary: the final reply has already streamed out. A user abort hands
// control straight back; a failure falls through — queued steering may still
// continue the loop, and the assembly below otherwise ends the run.
- if (result.status === "aborted") return;
+ if (result.status === "aborted") return null;
}
}
@@ -906,14 +910,14 @@ export class ContextEngine {
];
// No tool_call this turn and nothing injected -> the Task ends (the final reply has
// already been streamed out). A compaction stash, if any, rides the next run.
- if (!midTask && injected.length === 0) return;
+ if (!midTask && injected.length === 0) return null;
// Anything a failed boundary compaction stashed (synthesized repair outputs from
// rejected attempts) rides the very next request, ahead of the turn outputs so
// tool_results stay contiguous and first.
const stashed = this.pendingCarryOver;
this.pendingCarryOver = [];
nextInput = [...stashed, ...turn.toolOutputs, ...injected];
- if (nextInput.length === 0) return;
+ if (nextInput.length === 0) return null;
}
}
@@ -973,7 +977,7 @@ export class ContextEngine {
}
// The carry seam is a clean binary on whether the compaction committed anything to
// AgentHub (PR #87 review):
- // - nothing committed (every attempt timeout/malformed/failed/auth/aborted): the fold
+ // - nothing committed (every attempt retryable/fatal/aborted): the fold
// never reached the model context — restore the prior carry-over **verbatim**. Zero
// committed attempts also means zero synthesized repairs, so there is no stash to
// interleave with (pinned by tests);
@@ -1009,9 +1013,6 @@ export class ContextEngine {
cap: number,
retries: readonly StopReason[],
): number | undefined {
- // A permanent failure is aborted on, never retried (see the turn loop): announcing a
- // countdown for it would be a lie.
- if (outcome.permanent) return undefined;
if (!retries.includes(outcome.status)) return undefined;
if (reconnectsSoFar >= cap) return undefined;
return reconnectDelayMs(
@@ -1137,6 +1138,7 @@ export class ContextEngine {
RETRY_STATUSES,
);
const stopEvt = requestEnd(outcome.status, {
+ ...(outcome.errorCode !== undefined ? { errorCode: outcome.errorCode } : {}),
...(outcome.errorMessage !== undefined ? { errorMessage: outcome.errorMessage } : {}),
// The authoritative attempt ordinal (1-based, within this retry run); a clean
// first-try completion stays unstamped so the common case adds no noise.
@@ -1171,9 +1173,9 @@ export class ContextEngine {
// added to this turn's ledger, and gets no paired output backfilled: such a tool_call
// was never committed to history by AgentHub, so there's nothing to pair. This turn
// must then end with a non-completed outcome (only interruption closure produces such
- // a tool_call): a retryable outcome (failed/timeout/malformed) is cleaned up by
+ // a tool_call): a retryable outcome is cleaned up by
// reconnect resending the flatten carry-over, while the run-ending ones
- // (aborted/auth) exit directly.
+ // (aborted/fatal) exit directly.
if (isCompleteModelMessage(msg) && msg.payload.type === "tool_call") {
const tc = msg as OmniMessage<ToolCallPayload>;
if (tc.payload.stop_reason !== "completed") continue;
@@ -1321,7 +1323,7 @@ export class ContextEngine {
const failed = toolCallOutput({
output: `[tool error] ${message}`,
toolCallId: toolCall.payload.tool_call_id,
- stopReason: "failed",
+ stopReason: "fatal",
});
queue.push(failed);
await this.write(failed);
@@ -1336,13 +1338,13 @@ export class ContextEngine {
const partials = [
partialText("start"),
partialText("delta", text),
- partialText("stop", "", "failed"),
+ partialText("stop", "", "fatal"),
];
for (const partial of partials) {
yield partial;
await this.write(partial);
}
- const note = assistantText(text, "failed");
+ const note = assistantText(text, "fatal");
yield note;
await this.write(note);
}
@@ -1411,9 +1413,9 @@ export class ContextEngine {
* text and no tool calls in the response. Everything short of that is one kind of failure,
* handled exactly like an ordinary LLM request's (issue #170): an unusable committed
* response (empty summary, or tool calls — answered with synthesized failed outputs and
- * retried behind a corrective note, see the loop body) and the transport failures
- * (failed/timeout/malformed) all reconnect under the one `compactionMaxReconnects` budget
- * (defaulting to the turn loop's budget and ladder — see RETRY_STATUSES); only `auth` stops
+ * retried behind a corrective note, see the loop body) and the retryable failures all
+ * reconnect under the one `compactionMaxReconnects` budget
+ * (defaulting to the turn loop's budget and ladder — see RETRY_STATUSES); only `fatal` stops
* without retrying. Once the budget is exhausted the compaction fails; on failure/abort, the original
* context and Trace index are kept — it does not fall back to discard. The first **committed**
* attempt absorbs `pendingToolOutputs` into the old context's history (issue #85): later
@@ -1446,9 +1448,9 @@ export class ContextEngine {
await this.write(prompt);
// Whether the folded input was absorbed into the old object's history — true once any
- // attempt was committed by AgentHub (only `completed` commits: timeout and
- // malformed end an incomplete stream, and failed/auth/aborted throw or cut off before a
- // clean end — none of those reach the stateful commit). Returned as
+ // attempt was committed by AgentHub (only `completed` commits: a retryable failure ends
+ // an incomplete stream, and fatal/aborted throw or cut off before a clean end — none of
+ // those reach the stateful commit). Returned as
// `committed`: the callers' two-case carry rule branches on it.
let committed = false;
// Synthesized outputs answering the latest unusable attempt's tool calls, not yet carried
@@ -1456,13 +1458,14 @@ export class ContextEngine {
// the compaction be abandoned first (see stashRepairs).
let pendingRepairs: OmniMessage[] = [];
// One retry budget for every failure: an unusable committed response (empty summary /
- // tool calls) counts exactly like a transport failure (issue #170) — same counter, same
- // exponential ladder — and only `auth` stops without retrying.
+ // tool calls) counts exactly like a retryable failure (issue #170) — same counter, same
+ // exponential ladder — and only `fatal` stops without retrying.
let reconnects = 0;
// The compaction_end event's share of the RetryDetail block (also what the server's
// error record carries): the final attempt ordinal, and the last failure's detail.
let attempts = 0;
let lastError: string | undefined;
+ let lastErrorCode: ErrorCode | undefined;
for (;;) {
if (signal?.aborted) {
this.stashRepairs(pendingRepairs);
@@ -1518,6 +1521,8 @@ export class ContextEngine {
// that committed turn is plain assistant text/thinking, and re-sending the compaction
// Prompt on top of it is structurally sound.
unusable = true;
+ // An unusable summary is a malformed response for this request's purpose.
+ lastErrorCode = "malformed";
lastError =
attempt.toolCalls.length > 0
? "the response called tools instead of writing a summary"
@@ -1526,7 +1531,7 @@ export class ContextEngine {
toolCallOutput({
output: "[tool error] the compaction request expects a summary, not tool calls",
toolCallId: tc.payload.tool_call_id,
- stopReason: "failed",
+ stopReason: "fatal",
}),
);
for (const repair of pendingRepairs) await this.write(repair);
@@ -1534,35 +1539,50 @@ export class ContextEngine {
this.stashRepairs(pendingRepairs);
yield* this.emitCompactionEnd(reason, "summarize", "aborted", { attempt: attempts });
return { status: "aborted", committed };
- } else if (attempt.status === "auth") {
- // `auth` is the one status that never retries: credentials don't heal on a ladder.
- // It folds into `failed` here: the compaction event pair keeps its
- // completed/failed/aborted set, the original context is kept, and the host learns
- // about the credential problem from the request's own terminal status (a turn-loop
- // request will surface it; the compaction request_end is Trace-only).
+ } else if (attempt.status === "fatal") {
+ // `fatal` never retries: a definitive rejection or a dead credential doesn't heal
+ // on a ladder. The compaction ends `fatal` too — the next trigger will hit the
+ // same wall until a config or credential change; the original context is kept and
+ // the error detail rides the compaction_end for the host to surface.
this.stashRepairs(pendingRepairs);
- yield* this.emitCompactionEnd(reason, "summarize", "failed", {
+ yield* this.emitCompactionEnd(reason, "summarize", "fatal", {
attempt: attempts,
+ ...(attempt.errorCode !== undefined ? { errorCode: attempt.errorCode } : {}),
...(attempt.errorMessage !== undefined ? { errorMessage: attempt.errorMessage } : {}),
});
- return { status: "failed", committed };
+ return {
+ status: "fatal",
+ committed,
+ ...(attempt.errorCode !== undefined ? { errorCode: attempt.errorCode } : {}),
+ ...(attempt.errorMessage !== undefined ? { errorMessage: attempt.errorMessage } : {}),
+ };
} else {
- // Transport failure: keep its detail as the last error of record.
+ // Retryable failure: keep its cause and detail as the last error of record.
lastError = attempt.errorMessage;
+ lastErrorCode = attempt.errorCode;
}
- // One failure path for everything else — unusable summaries and the transport statuses
- // (failed / timeout / malformed, never committed by AgentHub) — treated like an
+ // One failure path for everything else — unusable summaries and retryable failures
+ // (never committed by AgentHub) — treated like an
// ordinary LLM request's failures: the same budget (defaulting to the shared
// maxReconnects, issue #170) and the same exponential ladder. An unusable attempt's
// request_end carries status completed, for which no retry_in_ms is announced — the
// backoff wait still happens.
if (reconnects >= this.compactionMaxReconnects) {
+ // Retries exhausted on retryable failures (transport faults and unusable
+ // summaries alike): the compaction ends `retryable` — abandoned this time, and
+ // the standing trigger makes it up at the next opportunity.
this.stashRepairs(pendingRepairs);
- yield* this.emitCompactionEnd(reason, "summarize", "failed", {
+ yield* this.emitCompactionEnd(reason, "summarize", "retryable", {
attempt: attempts,
+ ...(lastErrorCode !== undefined ? { errorCode: lastErrorCode } : {}),
...(lastError !== undefined ? { errorMessage: lastError } : {}),
});
- return { status: "failed", committed };
+ return {
+ status: "retryable",
+ committed,
+ ...(lastErrorCode !== undefined ? { errorCode: lastErrorCode } : {}),
+ ...(lastError !== undefined ? { errorMessage: lastError } : {}),
+ };
}
reconnects += 1;
const ok = await this.backoff(reconnects, signal);
@@ -1579,7 +1599,7 @@ export class ContextEngine {
// breaks the copy-my-own-mistake loop: the model's freshest example is its committed
// bad output, and rewriting history to hide it would invalidate the provider's
// prompt cache (issue #84) — correcting forward is the one cache-safe option
- // (issue #170). Transport failures skip this: nothing was committed, so their input
+ // (issue #170). Retryable failures skip this: nothing was committed, so their input
// is resent unchanged (any pending repairs included).
const guidance = userText(SUMMARY_RETRY_GUIDANCE);
await this.write(guidance);
@@ -1633,7 +1653,8 @@ export class ContextEngine {
text: string;
toolCalls: OmniMessage<ToolCallPayload>[];
usage: OmniMessage | null;
- /** Error detail (LLMOutcome.errorMessage) on non-completed statuses — becomes compaction_end.error_message when this failure ends the compaction. */
+ /** Classified cause and error detail (LLMOutcome) on non-completed statuses — become compaction_end.error_code / error_message when this failure ends the compaction. */
+ errorCode?: ErrorCode;
errorMessage?: string;
}
> {
@@ -1687,6 +1708,7 @@ export class ContextEngine {
text,
toolCalls,
usage,
+ ...(res.value.errorCode !== undefined ? { errorCode: res.value.errorCode } : {}),
...(res.value.errorMessage !== undefined ? { errorMessage: res.value.errorMessage } : {}),
};
}
@@ -1764,16 +1786,18 @@ export class ContextEngine {
reason: CompactionReason,
mode: CompactionMode,
status: StopReason,
- detail?: { attempt?: number; errorMessage?: string },
+ detail?: { attempt?: number; errorCode?: ErrorCode; errorMessage?: string },
): AsyncGenerator<OmniMessage> {
const msg = compactionEnd({ reason, mode, status, ...detail });
yield msg;
await this.write(msg);
}
- /** Interruption: emits an abort event. Cleanup/resending is managed centrally by `run` via carry-over; the LLM history is never touched again. */
- private async *emitAbort(reason: string): AsyncGenerator<OmniMessage> {
- const msg = abortEvent(reason);
+ /** User interruption: emits an abort event carrying the machine-readable cause. Cleanup/resending is managed centrally by `run` via carry-over; the LLM history is never touched again. */
+ private async *emitAbort(
+ errorCode: "user_abort" | "backoff_interrupted" | "compaction_interrupted",
+ ): AsyncGenerator<OmniMessage> {
+ const msg = abortEvent(errorCode);
yield msg;
await this.write(msg);
}
@@ -1781,7 +1805,7 @@ export class ContextEngine {
/**
* Builds the interruption resend content (carry-over, interruption cleanup)
* based on the LLM's terminal state. Used only for the **exit** cleanup of the statuses that
- * end the run: `aborted` and `auth` (a retried `failed` does not reach here, and neither
+ * end the run: `aborted` and `fatal` (a retried `retryable` does not reach here, and neither
* does reconnect retry: retry input is assembled by withRetriedTurns, appending
* `[turn_retried]` with the failed attempt's output, distinct from the user-interruption
* `[turn_aborted]`):
diff --git a/packages/core/src/environment/environment.ts b/packages/core/src/environment/environment.ts
index 49d8418e..2d870f1f 100644
--- a/packages/core/src/environment/environment.ts
+++ b/packages/core/src/environment/environment.ts
@@ -726,10 +726,12 @@ export class Environment implements EnvironmentInterface {
stopReason = "aborted";
notes.push(TOOL_ABORTED_NOTE);
} else if (timedOut) {
- stopReason = "failed";
+ // A tool timeout or error is definitive for this call — nothing in the harness
+ // retries a tool, so the result is fed back to the model as fatal.
+ stopReason = "fatal";
notes.push(`[tool timeout: exceeded ${timeoutMs}ms]`);
} else if (thrown != null) {
- stopReason = "failed";
+ stopReason = "fatal";
notes.push(`[tool error] ${thrown instanceof Error ? thrown.message : String(thrown)}`);
} else {
stopReason = selfReported ?? "completed";
@@ -783,6 +785,6 @@ export class Environment implements EnvironmentInterface {
/** Upfront failure (unknown tool/argument parse failure): delta(explanation) -> stop -> full failed output (start already emitted by the caller). */
function* emitFailure(toolCallId: string, message: string): Generator<OmniMessage> {
yield partialToolCallOutput({ eventType: "delta", output: message, toolCallId });
- yield partialToolCallOutput({ eventType: "stop", toolCallId, stopReason: "failed" });
- yield toolCallOutput({ output: message, toolCallId, stopReason: "failed" });
+ yield partialToolCallOutput({ eventType: "stop", toolCallId, stopReason: "fatal" });
+ yield toolCallOutput({ output: message, toolCallId, stopReason: "fatal" });
}
diff --git a/packages/core/src/environment/mcp/provider.ts b/packages/core/src/environment/mcp/provider.ts
index df89e396..7548bdcd 100644
--- a/packages/core/src/environment/mcp/provider.ts
+++ b/packages/core/src/environment/mcp/provider.ts
@@ -21,7 +21,7 @@
* cancels the in-flight request. Result content maps as: text blocks → output text; image
* blocks → `images` data URLs; audio/binary-resource blocks → placeholder lines;
* `structuredContent` is serialized only when no text block was present; `isError` →
- * `stopReason: "failed"`. Permission for the frontend's read-only mode comes from the
+ * `stopReason: "fatal"`. Permission for the frontend's read-only mode comes from the
* spec's `readOnlyHint` annotation (`true` → `"r"`, anything else → `"rw"` — hints are
* untrusted, so the default is the restrictive direction), unless the entry sets an
* explicit `permission`, which then applies to every tool of that server.
@@ -289,9 +289,11 @@ export class McpToolProvider {
result: {
server: server.name,
transport: server.transport.kind,
- status: aborted ? "aborted" : "failed",
+ status: aborted ? "aborted" : "fatal",
duration_ms: durationMs(),
- ...(aborted ? {} : { error: describeError(err) }),
+ ...(aborted
+ ? {}
+ : { error_code: "connect_failed" as const, error_message: describeError(err) }),
},
};
}
@@ -442,7 +444,7 @@ export class McpToolProvider {
});
}
return {
- stopReason: failed ? "failed" : "completed",
+ stopReason: failed ? "fatal" : "completed",
...(rendered.images.length > 0 ? { images: rendered.images } : {}),
};
},
diff --git a/packages/core/src/environment/tools/command/session.ts b/packages/core/src/environment/tools/command/session.ts
index aaf9ee5a..8b693759 100644
--- a/packages/core/src/environment/tools/command/session.ts
+++ b/packages/core/src/environment/tools/command/session.ts
@@ -383,8 +383,8 @@ export function isStopSignal(signal: NodeJS.Signals | null | undefined): boolean
/** Converts exit info into a tool result (the terminal marker is appended via `note`, outside the truncation, so it isn't lost with long output). */
export function resultForExit(exit: ProcessExit | null): ToolResult {
if (!exit) return { stopReason: "completed" };
- if (exit.signal) return { stopReason: "failed", note: `[terminated by signal ${exit.signal}]` };
+ if (exit.signal) return { stopReason: "fatal", note: `[terminated by signal ${exit.signal}]` };
if (exit.code !== 0)
- return { stopReason: "failed", note: `[exit code: ${exit.code ?? "unknown"}]` };
+ return { stopReason: "fatal", note: `[exit code: ${exit.code ?? "unknown"}]` };
return { stopReason: "completed" };
}
diff --git a/packages/core/src/environment/tools/describe-image.ts b/packages/core/src/environment/tools/describe-image.ts
index 6efd08a4..ab3f5ccc 100644
--- a/packages/core/src/environment/tools/describe-image.ts
+++ b/packages/core/src/environment/tools/describe-image.ts
@@ -52,7 +52,7 @@ export function createDescribeImageTool(
const source = args["source"];
if (typeof source !== "string" || source.length === 0) {
yield delta('Missing required argument "source" for describe_image.');
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (describer.modelId === null || describer.createLLM === undefined) {
yield delta(
@@ -60,14 +60,14 @@ export function createDescribeImageTool(
"images; ask the user to pick a vision model in the model settings (vision_model) " +
"to enable image reading.",
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const res = await loadImage(source, ctx.workspaceDir, signal);
if (!res.ok) {
if (res.reason === "aborted") return { stopReason: "aborted" };
yield delta(res.message);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const prompt =
@@ -113,7 +113,7 @@ export function createDescribeImageTool(
yield delta(
`${streamedAny ? "\n" : ""}Vision model (${describer.modelId}) request ${outcome?.status ?? "failed"}${detail}`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (!streamedAny) yield delta("[vision model returned no text]");
},
diff --git a/packages/core/src/environment/tools/edit-file.ts b/packages/core/src/environment/tools/edit-file.ts
index f490e7b3..f6c1d683 100644
--- a/packages/core/src/environment/tools/edit-file.ts
+++ b/packages/core/src/environment/tools/edit-file.ts
@@ -75,29 +75,29 @@ export function createEditFileTool(definition: ToolDefinitionConfig): BuiltinToo
const filePath = args["file_path"];
if (typeof filePath !== "string" || filePath.length === 0) {
yield delta(`Missing required argument "file_path" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const oldString = args["old_string"];
if (typeof oldString !== "string") {
yield delta(`Missing required argument "old_string" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (oldString.length === 0) {
yield delta(
"old_string must not be empty — edit_file replaces existing text. To create a file or rewrite it wholesale, use write_file.",
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const newString = args["new_string"];
if (typeof newString !== "string") {
yield delta(`Missing required argument "new_string" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (oldString === newString) {
yield delta(
"old_string and new_string are identical — nothing to change. Make new_string the desired replacement text.",
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const replaceAll = args["replace_all"] === true;
@@ -108,7 +108,7 @@ export function createEditFileTool(definition: ToolDefinitionConfig): BuiltinToo
const st = await stat(resolved);
if (st.isDirectory()) {
yield delta(`Cannot edit "${filePath}": it is a directory.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
fileMode = st.mode & 0o777;
content = await readFile(resolved, { encoding: "utf8", ...(signal ? { signal } : {}) });
@@ -126,7 +126,7 @@ export function createEditFileTool(definition: ToolDefinitionConfig): BuiltinToo
const message = err instanceof Error ? err.message : String(err);
yield delta(`Failed to read "${filePath}": ${message}`);
}
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (signal?.aborted) return { stopReason: "aborted" };
@@ -140,13 +140,13 @@ export function createEditFileTool(definition: ToolDefinitionConfig): BuiltinToo
yield delta(
`old_string not found in "${filePath}". Make sure it matches the file content exactly, including whitespace and indentation.${crlfHint}`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (occurrences > 1 && !replaceAll) {
yield delta(
`old_string occurs ${occurrences} times in "${filePath}". Add surrounding context to make it unique, or set replace_all to true to replace every occurrence.`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const replaceStart = content.indexOf(oldString);
@@ -165,7 +165,7 @@ export function createEditFileTool(definition: ToolDefinitionConfig): BuiltinToo
if (signal?.aborted) return { stopReason: "aborted" };
const message = err instanceof Error ? err.message : String(err);
yield delta(`Failed to write "${filePath}": ${message}`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const replaced = replaceAll ? occurrences : 1;
diff --git a/packages/core/src/environment/tools/exec-command.ts b/packages/core/src/environment/tools/exec-command.ts
index cefdea29..a3682501 100644
--- a/packages/core/src/environment/tools/exec-command.ts
+++ b/packages/core/src/environment/tools/exec-command.ts
@@ -58,13 +58,13 @@ export function createExecCommandTool(
if (!manager) {
yield delta(`[${definition.name} unavailable: no command session manager configured]`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const cmd = args["cmd"];
if (typeof cmd !== "string" || cmd.length === 0) {
yield delta(`Missing required argument "cmd" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// workdir defaults to workspaceDir; relative paths are resolved against workspaceDir.
const rawWorkdir = args["workdir"];
@@ -88,7 +88,7 @@ export function createExecCommandTool(
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
yield delta(`[spawn error: ${message}]`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// run_in_background: no collect window at all — register immediately, arm the completion
@@ -128,7 +128,7 @@ export function createExecCommandTool(
// Already exited: report exit status; process group cleanup (reaping any leftover
// background children) is handled uniformly in finally.
if (session.error) {
- return { stopReason: "failed", note: `[spawn error: ${session.error.message}]` };
+ return { stopReason: "fatal", note: `[spawn error: ${session.error.message}]` };
}
return resultForExit(session.exit);
} finally {
diff --git a/packages/core/src/environment/tools/input-command.ts b/packages/core/src/environment/tools/input-command.ts
index 331b3764..07b30c0a 100644
--- a/packages/core/src/environment/tools/input-command.ts
+++ b/packages/core/src/environment/tools/input-command.ts
@@ -57,20 +57,20 @@ export function createInputCommandTool(
if (!manager) {
yield delta("[input_command unavailable: no command session manager configured]");
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const processId = args["process_id"];
if (typeof processId !== "string" || processId.length === 0) {
yield delta('Missing required argument "process_id" for input_command.');
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const session = manager.get(processId);
if (!session) {
yield delta(
`[input_command error: unknown process_id ${processId} (the session may have exited and been cleared)]`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// Termination path (kill: true): disarm the report, drain, kill the group, drop the row.
@@ -120,7 +120,7 @@ export function createInputCommandTool(
yield delta(
'[input_command error: chars mixes U+0003 (Ctrl-C) with other content; send "\\u0003" alone to interrupt]',
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
} else session.write(chars);
}
@@ -136,7 +136,7 @@ export function createInputCommandTool(
// Already exited: clean up the registry and report the exit status.
manager.remove(processId);
if (session.error) {
- return { stopReason: "failed", note: `[spawn error: ${session.error.message}]` };
+ return { stopReason: "fatal", note: `[spawn error: ${session.error.message}]` };
}
return resultForExit(session.exit);
},
diff --git a/packages/core/src/environment/tools/input-subagent.ts b/packages/core/src/environment/tools/input-subagent.ts
index 805ca5dd..762f8fdd 100644
--- a/packages/core/src/environment/tools/input-subagent.ts
+++ b/packages/core/src/environment/tools/input-subagent.ts
@@ -57,13 +57,13 @@ export function createInputSubagentTool(
if (!manager) {
yield delta("[input_subagent unavailable: no subagent session manager configured]");
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const subagentId = args["subagent_id"];
if (typeof subagentId !== "string" || subagentId.length === 0) {
yield delta('Missing required argument "subagent_id" for input_subagent.');
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
let session = manager.get(subagentId);
if (!session) {
@@ -96,7 +96,7 @@ export function createInputSubagentTool(
`[input_subagent error: unknown subagent_id ${subagentId} ` +
`(not from this conversation, or its session could not be resumed)]`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const prompt = typeof args["prompt"] === "string" ? (args["prompt"] as string) : "";
@@ -137,7 +137,7 @@ export function createInputSubagentTool(
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
yield delta(`[input_subagent error: ${message}]`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
} else {
// Running, but the handle predates steering (an older embedder's SubagentRunner).
@@ -145,7 +145,7 @@ export function createInputSubagentTool(
`[input_subagent error: subagent ${subagentId} is still running; ` +
`poll with an empty prompt to collect progress first]`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
}
diff --git a/packages/core/src/environment/tools/read-file.ts b/packages/core/src/environment/tools/read-file.ts
index e3132d5d..766ceda6 100644
--- a/packages/core/src/environment/tools/read-file.ts
+++ b/packages/core/src/environment/tools/read-file.ts
@@ -256,12 +256,12 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
const filePath = args["file_path"];
if (typeof filePath !== "string" || filePath.length === 0) {
yield delta(`Missing required argument "file_path" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const offsetArg = coerceCount(args["offset"], 1);
if (!offsetArg.ok) {
yield delta(`Invalid "offset": expected a number (got ${JSON.stringify(args["offset"])}).`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const offset = Math.max(1, offsetArg.value);
const limitArg = coerceCount(args["limit"], DEFAULT_READ_FILE_LIMIT);
@@ -269,7 +269,7 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
yield delta(
`Invalid "limit": expected a positive number (got ${JSON.stringify(args["limit"])}).`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const limit = limitArg.value;
@@ -284,7 +284,7 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
yield delta(
`Refusing to read "${filePath}": ${secretHit} holds the user's secrets and must never enter the conversation.`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
let size: number;
@@ -294,7 +294,7 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
yield delta(
`Cannot read "${filePath}": it is a directory. Pass the path of a file inside it.`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
size = st.size;
} catch (err) {
@@ -311,7 +311,7 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
const message = err instanceof Error ? err.message : String(err);
yield delta(`Failed to read "${filePath}": ${message}`);
}
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (signal?.aborted) return { stopReason: "aborted" };
if (size === 0) {
@@ -326,14 +326,14 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
if (signal?.aborted) return { stopReason: "aborted" };
const message = err instanceof Error ? err.message : String(err);
yield delta(`Failed to read "${filePath}": ${message}`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (scan.aborted || signal?.aborted) return { stopReason: "aborted" };
if (scan.binary) {
yield delta(
`"${filePath}" looks like a binary file (contains NUL bytes). Use shell commands to inspect it, or read_image if it is an image.`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (scan.capBeforeWindow) {
const mb = Math.round(READ_FILE_SCAN_CAP_BYTES / (1024 * 1024));
@@ -342,13 +342,13 @@ export function createReadFileTool(definition: ToolDefinitionConfig): BuiltinToo
`(scanned ${scan.total} line${scan.total === 1 ? "" : "s"}). The offset is too deep or the file has extremely long ` +
`lines — use shell commands (e.g. sed -n '${offset},${offset + limit - 1}p') for this file.`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (scan.totalKnown && offset > scan.total) {
yield delta(
`Offset ${offset} is past the end of "${filePath}" (${scan.total} line${scan.total === 1 ? "" : "s"} total).`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// Render the window, self-budgeted below the tool's output cap so the trailing notes
diff --git a/packages/core/src/environment/tools/read-image.ts b/packages/core/src/environment/tools/read-image.ts
index e4e5fda7..ce06b8b9 100644
--- a/packages/core/src/environment/tools/read-image.ts
+++ b/packages/core/src/environment/tools/read-image.ts
@@ -228,14 +228,14 @@ export function createReadImageTool(definition: ToolDefinitionConfig): BuiltinTo
const source = args["source"];
if (typeof source !== "string" || source.length === 0) {
yield delta('Missing required argument "source" for read_image.');
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const res = await loadImage(source, ctx.workspaceDir, signal);
if (!res.ok) {
if (res.reason === "aborted") return { stopReason: "aborted" };
yield delta(res.message);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// Success: yield a brief one-line description as a text delta (both in the streaming and
diff --git a/packages/core/src/environment/tools/run-subagent.ts b/packages/core/src/environment/tools/run-subagent.ts
index 562dd466..952c46fa 100644
--- a/packages/core/src/environment/tools/run-subagent.ts
+++ b/packages/core/src/environment/tools/run-subagent.ts
@@ -76,16 +76,16 @@ export function createSubagentTool(
// than throwing (consistent with other tools).
if (!runner) {
yield* fail("[run_subagent unavailable: no subagent runner configured]");
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
if (!manager || manager.isDisposed) {
yield* fail("[run_subagent unavailable: no subagent session manager available]");
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const prompt = typeof args.prompt === "string" ? args.prompt : "";
if (prompt.trim().length === 0) {
yield* fail("[run_subagent error: missing required string argument `prompt`]");
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const agentId = typeof args.agent_id === "string" ? args.agent_id : undefined;
const modelId = typeof args.model_id === "string" ? args.model_id : undefined;
@@ -96,7 +96,7 @@ export function createSubagentTool(
yield* fail(
"[run_subagent error: `model_id` and `provider` must be given together (a model reference is the pair), or both omitted to inherit the parent session's model]",
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// Thinking-level override: an explicit value must be one of the selectable tiers — a typo
// must fail loudly rather than silently running the child at an inherited level the caller
@@ -110,7 +110,7 @@ export function createSubagentTool(
`[run_subagent error: invalid \`thinking_level\` ${JSON.stringify(rawThinkingLevel)}; ` +
`use one of ${SUBAGENT_THINKING_LEVELS.join(" / ")}, or omit it to inherit the parent session's level]`,
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const thinkingLevel = rawThinkingLevel as ThinkingLevelName | undefined;
const background = args.run_in_background === true;
@@ -128,7 +128,7 @@ export function createSubagentTool(
yield* fail(
"[run_subagent error: too many background subagents; poll or finish existing ones first]",
);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// Spawn the child Session (precheck errors such as exceeding the depth limit or a
@@ -152,7 +152,7 @@ export function createSubagentTool(
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
yield* fail(`[run_subagent error: ${message}]`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// run_in_background: no collect window — register immediately, arm the completion
diff --git a/packages/core/src/environment/tools/subagent/session.ts b/packages/core/src/environment/tools/subagent/session.ts
index 736b5c3f..78d5dd0d 100644
--- a/packages/core/src/environment/tools/subagent/session.ts
+++ b/packages/core/src/environment/tools/subagent/session.ts
@@ -35,7 +35,12 @@
*/
import type { OmniMessage } from "../../../omnimessage/index.js";
import type { ApprovalDecision, ToolCallPayload } from "../../../omnimessage/index.js";
-import type { ApproveFn, SubagentHandle, ThinkingLevelName } from "../../../interfaces/index.js";
+import type {
+ ApproveFn,
+ RunCutoff,
+ SubagentHandle,
+ ThinkingLevelName,
+} from "../../../interfaces/index.js";
import type { ToolResult } from "../types.js";
import { CappedTextBuffer, WakeSignal } from "../background/index.js";
@@ -350,30 +355,36 @@ export class ManagedSubagentSession {
thinkingLevel?: ThinkingLevelName,
): Promise<void> {
let wroteAny = false;
- let childAbort: string | null = null;
+ // Whether the round was cut off early (a user abort, a terminal LLM failure, a
+ // mid-task compaction failure): read from the run generator's return value — the
+ // child engine states it directly; a cut-off round is reported failed, not completed.
+ let cut: RunCutoff | null = null;
try {
// Either scope ends the round: the session-lifetime kill, or this run's own abort.
- for await (const msg of this.handle.run({
+ // Manual iteration (not for-await) so the handle's return value — whether the round
+ // was cut off — is read; a child session failure doesn't throw.
+ const it = this.handle.run({
messages,
signal: AbortSignal.any([this.abortCtrl.signal, runCtrl.signal]),
approve: this.childApprove,
...(thinkingLevel !== undefined ? { thinkingLevel } : {}),
- })) {
+ });
+ for (;;) {
+ const res = await it.next();
+ if (res.done) {
+ // Older embedders' handles may return nothing: that counts as ran-to-completion.
+ cut = res.value ?? null;
+ break;
+ }
+ const msg = res.value;
this.bufferMessage(msg);
if ((msg.origin?.length ?? 0) === 1) {
const p = msg.payload as {
type?: string;
event_type?: string;
text?: string;
- reason?: string;
};
- // A direct child layer's abort event: the child session was interrupted/failed (LLM
- // request error, user interruption, etc). A child session failure doesn't throw, it
- // only emits an event, based on which this round is reported as failed rather than
- // marked completed.
- if (p.type === "abort") {
- childAbort = p.reason ?? "aborted";
- } else if (
+ if (
p.type === "partial_text" &&
p.event_type === "delta" &&
typeof p.text === "string" &&
@@ -396,8 +407,14 @@ export class ManagedSubagentSession {
}
this.wakeSignal.notify();
}
- if (childAbort !== null) {
- this.exitInfo = { status: "failed", note: `[subagent aborted: ${childAbort}]` };
+ if (cut !== null) {
+ // The note reaches the parent model: the cause code plus the raw detail.
+ const detail = cut.errorMessage;
+ const cause = cut.errorCode ?? (cut.kind === "abort" ? "aborted" : cut.kind);
+ this.exitInfo = {
+ status: "failed",
+ note: `[subagent ${cut.kind === "abort" ? "aborted" : "failed"}: ${cause}${detail ? ` — ${detail}` : ""}]`,
+ };
} else if (!wroteAny) {
this.exitInfo = {
status: "completed",
@@ -497,8 +514,11 @@ export class ManagedSubagentSession {
}
}
-/** Converts a run's terminal state into a tool result (note is appended outside the truncation, so it isn't lost with long output). */
+/** Converts a run's terminal state into a tool result (note is appended outside the truncation, so it isn't lost with long output). A failed child run is fatal for this call — nothing retries a tool. */
export function resultForSubagentExit(exit: SubagentExit | null): ToolResult {
if (!exit) return { stopReason: "completed" };
- return { stopReason: exit.status, ...(exit.note !== undefined ? { note: exit.note } : {}) };
+ return {
+ stopReason: exit.status === "failed" ? "fatal" : exit.status,
+ ...(exit.note !== undefined ? { note: exit.note } : {}),
+ };
}
diff --git a/packages/core/src/environment/tools/write-file.ts b/packages/core/src/environment/tools/write-file.ts
index 63eeb2bd..fb147ce3 100644
--- a/packages/core/src/environment/tools/write-file.ts
+++ b/packages/core/src/environment/tools/write-file.ts
@@ -75,14 +75,14 @@ export function createWriteFileTool(definition: ToolDefinitionConfig): BuiltinTo
const filePath = args["file_path"];
if (typeof filePath !== "string" || filePath.length === 0) {
yield delta(`Missing required argument "file_path" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
// An empty string is valid content (creates an empty file); only a missing/non-string
// value is an argument error.
const content = args["content"];
if (typeof content !== "string") {
yield delta(`Missing required argument "content" for ${definition.name}.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const resolved = path.resolve(ctx.workspaceDir, filePath);
@@ -95,7 +95,7 @@ export function createWriteFileTool(definition: ToolDefinitionConfig): BuiltinTo
const st = await stat(resolved);
if (st.isDirectory()) {
yield delta(`Cannot write "${filePath}": it is a directory.`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
existed = true;
fileMode = st.mode & 0o777; // Preserved across the atomic temp-file + rename write
@@ -124,7 +124,7 @@ export function createWriteFileTool(definition: ToolDefinitionConfig): BuiltinTo
if (signal?.aborted) return { stopReason: "aborted" };
const message = err instanceof Error ? err.message : String(err);
yield delta(`Failed to write "${filePath}": ${message}`);
- return { stopReason: "failed" };
+ return { stopReason: "fatal" };
}
const lines = countLines(content);
diff --git a/packages/core/src/goal/goal-loop.ts b/packages/core/src/goal/goal-loop.ts
index 6a594412..b8558ee2 100644
--- a/packages/core/src/goal/goal-loop.ts
+++ b/packages/core/src/goal/goal-loop.ts
@@ -14,9 +14,10 @@
* normalize to `blocked` — see goal-file.ts),
* - the loop's own token accounting against the budget (internal counters; the budget
* line in each round's block is composed from them, never read from anywhere),
- * - a round the engine cut off rather than finished — a main-session abort (LLM failure,
- * user interrupt) or a final assistant notice with `stop_reason: "failed"` (the engine's
- * max_turns cutoff emits exactly that, and no abort event): the model never got to write
+ * - a round the engine cut off rather than finished — a user abort, a terminal LLM
+ * failure, or a mid-task compaction failure, read from the run generator's own return
+ * value (`RunCutoff`; failures emit no abort event) — or a final assistant notice with
+ * `stop_reason: "fatal"` (the engine's max_turns cutoff): the model never got to write
* the file, so re-firing would loop the same cutoff forever, and
* - a hard round cap (`maxRounds`, default 100) as a runaway backstop independent of the
* budget — without it an unbudgeted goal whose model simply never writes the file would
@@ -30,7 +31,8 @@
* the stream — including origin-marked ones from subagent sessions, which are part of the
* goal's cost — contributes `request.total - request.cache_read`.
*/
-import { goalFinished, isEventMessage, isModelMessage, userText } from "../omnimessage/index.js";
+import { goalFinished, isModelMessage, userText } from "../omnimessage/index.js";
+import type { RunCutoff } from "../interfaces/shared.js";
import type { GoalOutcomeStatus, OmniMessage } from "../omnimessage/index.js";
import { stripLeadingMarkerBlocks } from "../omnimessage/markers/index.js";
import { readGoalStatus, writeGoalFile, UNLIMITED_BUDGET } from "./goal-file.js";
@@ -39,7 +41,8 @@ import { goalTokenDelta } from "./goal-stream.js";
/** The slice of Session the loop drives: one Task per round (structural, so tests can substitute a fake). */
export interface GoalRoundRunner {
- run(newMessages: OmniMessage[]): AsyncGenerator<OmniMessage>;
+ /** One round = one Task run; the return value says whether the round was cut off early (`Session.run`'s contract). Fakes returning nothing count as "ran to completion". */
+ run(newMessages: OmniMessage[]): AsyncGenerator<OmniMessage, RunCutoff | null | void>;
}
export interface GoalLoopOptions {
@@ -66,16 +69,12 @@ export interface GoalLoopOptions {
/** Default `maxRounds`: the runaway backstop for goals with no (or a huge) budget. */
export const GOAL_MAX_ROUNDS = 100;
-/** Whether this message is the **main** session's abort event (subagent aborts don't end the goal). */
-function isMainAbort(msg: OmniMessage): boolean {
- return isEventMessage(msg) && msg.payload.type === "abort" && (msg.origin?.length ?? 0) === 0;
-}
-
/**
* The main session's assistant text, or null. Used to track how a round ended: the engine's
* max_turns cutoff finishes the stream with an assistant notice carrying
- * `stop_reason: "failed"` (and no abort event) — the only failure mode that neither
- * `isMainAbort` nor the goal file can see.
+ * `stop_reason: "fatal"` (and no abort event) — the only failure mode that neither
+ * `isMainAbort` nor the goal file can see. Traces written before the stop-reason
+ * convergence spell the same notice "failed", so both spellings count.
*/
function mainAssistantStopReason(msg: OmniMessage): string | null {
if (msg.origin && msg.origin.length > 0) return null;
@@ -117,13 +116,22 @@ export async function* runGoalLoop(
}),
);
yield input;
- for await (const msg of session.run([input])) {
+ // Manual iteration (not for-await) to read the run's return value: whether the round
+ // was cut off early — a user abort, a terminal LLM failure, or a mid-task compaction
+ // failure. The engine states it directly; a cut-off round must not re-fire.
+ const it = session.run([input]);
+ for (;;) {
+ const res = await it.next();
+ if (res.done) {
+ if (res.value) aborted = true;
+ break;
+ }
+ const msg = res.value;
used += goalTokenDelta(msg);
- if (isMainAbort(msg)) aborted = true;
// The LAST assistant text decides: a mid-round failed notice followed by normal text
// means the round recovered; the max_turns cutoff is always the final message.
const stop = mainAssistantStopReason(msg);
- if (stop !== null) roundFailed = stop === "failed";
+ if (stop !== null) roundFailed = stop === "fatal" || stop === "failed";
yield msg;
}
}
diff --git a/packages/core/src/interfaces/environment.ts b/packages/core/src/interfaces/environment.ts
index e22b0110..08549913 100644
--- a/packages/core/src/interfaces/environment.ts
+++ b/packages/core/src/interfaces/environment.ts
@@ -14,7 +14,7 @@
* Docs: packages/docs/content/interfaces.{zh,en}.md (site path /docs/interfaces).
*/
import type { OmniMessage, ToolCallPayload, ToolDefinition } from "../omnimessage/types.js";
-import type { ApproveFn, ThinkingLevelName } from "./shared.js";
+import type { ApproveFn, RunCutoff, ThinkingLevelName } from "./shared.js";
import type { LLMInterface } from "./llm.js";
// Concrete classes, used only for EnvironmentServices type annotations (type-only import; no runtime dependency, no circular reference).
import type { CommandSessionManager } from "../environment/tools/command/session-manager.js";
@@ -106,7 +106,9 @@ export interface SubagentHandle {
* Runs one turn of a task on the child Session. Emitted child-session messages **all already
* carry the origin marker** (the child Session id); the first message of the first run is the
* child Session's `session_meta`, and tool_calls received by the forwarded approval callback
- * carry origin as well.
+ * carry origin as well. The generator's return value says whether the round was cut off
+ * early (`RunCutoff`) or ran to completion (`null`) — the parent reports a cut-off round
+ * as failed instead of handing the model a half answer as a result.
*/
run(input: {
/**
@@ -121,7 +123,7 @@ export interface SubagentHandle {
approve?: ApproveFn;
/** Per-turn thinking level for THIS round only (a host follow-up's picker); omitted keeps the child Session's own level. */
thinkingLevel?: ThinkingLevelName;
- }): AsyncGenerator<OmniMessage>;
+ }): AsyncGenerator<OmniMessage, RunCutoff | null>;
/**
* Queues a steering message for the child Session's running Task (the same mechanism as a
* user steering the main session: delivered as a `[user_steering]` user message at the
diff --git a/packages/core/src/interfaces/llm.ts b/packages/core/src/interfaces/llm.ts
index f455e35d..3829c452 100644
--- a/packages/core/src/interfaces/llm.ts
+++ b/packages/core/src/interfaces/llm.ts
@@ -8,7 +8,7 @@
*
* Docs: packages/docs/content/interfaces.{zh,en}.md (site path /docs/interfaces).
*/
-import type { OmniMessage, StopReason, ToolDefinition } from "../omnimessage/types.js";
+import type { ErrorCode, OmniMessage, StopReason, ToolDefinition } from "../omnimessage/types.js";
import type { ThinkingLevelName } from "./shared.js";
// Concrete class, used only as a type annotation (type-only import; no runtime dependency, no circular reference).
import type { ToolCallIdAllocator } from "../llm/tool-call-ids.js";
@@ -50,7 +50,7 @@ export interface GenerativeModelConfig {
* request opts into the provider's faster serving tier at premium pricing (AgentHub
* UniConfig `fast_mode`). Off by default. Models without a fast tier reject the parameter
* before any network I/O (AgentHub `UnsupportedParameterError`); `streamGenerate` reports
- * that as a permanent `failed` outcome so the engine surfaces it instead of retrying.
+ * that as a `fatal` outcome so the engine surfaces it instead of retrying.
*/
fastMode?: boolean;
/** Construction-time default thinking level; a per-request `GenerativeModelParameters.thinkingLevel` overrides it for that request. */
@@ -79,45 +79,41 @@ export interface GenerativeModelParameters {
/**
* The terminal state of an LLM request, returned as the **return value** of the `streamGenerate`
- * async generator (not a yielded message). The status values share the same six-value protocol
+ * async generator (not a yielded message). The status values share the same four-value protocol
* as OmniMessage `stop_reason`:
* - `completed`: finished normally (already produced `token_usage`);
- * - `timeout`: LLM timed out or lost connection, needs reconnect — retried by `context_engine`
- * within the same run;
- * - `malformed`: AgentHub response failed JSON parsing, needs reconnect — also retried by
- * `context_engine`;
* - `aborted`: user-initiated interruption — stop and hand back to the user;
- * - `failed`: an error the retry classifier did not judge transient (params, etc.) — still
- * retried by `context_engine` within the same run (`errorMessage` provides the display text).
- * The classification stays honest — this is reported as `failed`, not relabelled a
- * timeout — while the *policy* retries it, because that classifier is an allowlist and a
- * gateway phrasing a transient fault its own way lands here;
- * - `auth`: the provider rejected the credentials (see `isAuthenticationError`) — the one
- * status that stops the run outright, since no retry can turn a rejected credential into
- * a working one; hosts also key on it to disable input until the model's API key is
- * updated (only the model reference is fixed at Session creation; credentials come from
- * the current Project config, so a key update lets the Session continue).
+ * - `retryable`: a failure worth retrying — transport drops, idle timeouts, 408/429/5xx,
+ * malformed or truncated responses, and anything unclassifiable. `context_engine`
+ * reconnects on its backoff ladder; `errorMessage` names the concrete failure.
+ * Unclassifiable errors land here on purpose: the fatal detector is an allowlist, and a
+ * gateway phrasing a transient fault its own way must keep its retries;
+ * - `fatal`: a failure no retry can fix — a provider 4xx rejection (invalid request or
+ * quota; 408/429 stay retryable), a credentials failure (see `isAuthenticationError`),
+ * or a deterministic client-side rejection thrown before any network I/O (fast mode on
+ * a model without a fast tier). The engine stops the run and surfaces `errorMessage`;
+ * the fix is a config or credential change, then a new request (only the model
+ * reference is fixed at Session creation — credentials come from the current Project
+ * config, so a key update lets the Session continue).
* Docs: /docs/interfaces § "LLMOutcome semantics".
*/
export interface LLMOutcome {
status: StopReason;
/**
- * Error detail (`describeError` text): present on `failed` / `auth`, and on `timeout` /
- * `malformed` when a concrete transport/provider error was caught (a plain idle timeout
- * has none). Carried onto the `request_end` event as `error_message` — one name across
- * the internal outcome and the wire — so observability (the Cost center's errors panel)
- * can show the real reason behind a retried request.
+ * Classified cause (the omnimessage ErrorCode vocabulary): present on every
+ * non-completed failure outcome. Carried onto the `request_end` event as `error_code`,
+ * next to `error_message` — the status answers "retry?", the code says what kind of
+ * error it was, machine-readably.
*/
- errorMessage?: string;
+ errorCode?: ErrorCode;
/**
- * Marks a `failed` outcome as deterministic: a client-side rejection thrown before any
- * network I/O (currently AgentHub's `UnsupportedParameterError` for `fast_mode` on a model
- * without a fast tier), which the identical request can never retry into working. The
- * engine skips the reconnect ladder for it and aborts the run with `errorMessage` — the
- * same terminal handling as `auth`, but the fix is a config change (turn off fast mode for
- * the model), not a credential update, so it stays a `failed` and hosts don't gate input.
+ * Error detail (`describeError` text): present on `fatal`, and on `retryable` when a
+ * concrete transport/provider error was caught (a plain idle timeout has none). Carried
+ * onto the `request_end` event as `error_message` — one name across the internal outcome
+ * and the wire — so observability (the Cost center's errors panel) can show the real
+ * reason behind a retried request.
*/
- permanent?: boolean;
+ errorMessage?: string;
}
/**
diff --git a/packages/core/src/interfaces/shared.ts b/packages/core/src/interfaces/shared.ts
index 32aa0883..d99fe5bf 100644
--- a/packages/core/src/interfaces/shared.ts
+++ b/packages/core/src/interfaces/shared.ts
@@ -9,7 +9,12 @@
*
* Docs: packages/docs/content/interfaces.{zh,en}.md (site path /docs/interfaces).
*/
-import type { ApprovalDecision, OmniMessage, ToolCallPayload } from "../omnimessage/types.js";
+import type {
+ ApprovalDecision,
+ ErrorCode,
+ OmniMessage,
+ ToolCallPayload,
+} from "../omnimessage/types.js";
// ToolDefinition is defined in omnimessage/types.ts (the tool_list_ready event carries the full tool schema); re-exported here to keep the original import path.
export type { ToolDefinition } from "../omnimessage/types.js";
@@ -17,6 +22,24 @@ export type { ToolDefinition } from "../omnimessage/types.js";
/** Thinking level of one request. Shared: the LLM takes it per request and as a construction default, and a spawned child Session inherits or overrides it. */
export type ThinkingLevelName = "none" | "low" | "medium" | "high" | "xhigh" | "max";
+/**
+ * How a run was cut off early, returned as the **return value** of the run generators
+ * (`ContextEngine.run` / `Session.run` / `SubagentHandle.run`): `null` means the run ran
+ * to completion. The engine knows which exit it took, so consumers that must not treat a
+ * cut-off run as finished (the goal loop deciding whether to re-fire a round, a subagent
+ * round reporting completion) read this instead of re-deriving it from the stream —
+ * failures emit no abort event, only their terminal request_end / compaction_end records.
+ * `abort` is a user interruption; `llm_failure` a terminal LLM request failure;
+ * `compaction_failure` a compaction given up mid-task (at a Task boundary the same
+ * failure is advisory and the run returns null). The error pair mirrors the terminal
+ * record's `error_code` / `error_message`.
+ */
+export interface RunCutoff {
+ kind: "abort" | "llm_failure" | "compaction_failure";
+ errorCode?: ErrorCode;
+ errorMessage?: string;
+}
+
/**
* Per-tool approval callback: the Human boundary gives allow/deny for each complete `tool_call`.
* `context_engine` calls it once per tool call within a turn. Subagents forward the parent's
diff --git a/packages/core/src/llm/generative-model.ts b/packages/core/src/llm/generative-model.ts
index 08d0cff5..d0a2cd4c 100644
--- a/packages/core/src/llm/generative-model.ts
+++ b/packages/core/src/llm/generative-model.ts
@@ -13,18 +13,15 @@
* 3. Interruption/error handling: `finishInterrupted` first closes any open
* streaming segments and backfills the complete message, then the output ends — never
* leaking a malformed structure. This interface **never retries internally** — it only
- * labels, and `context_engine` owns the retry policy: every LLM error retries on the
- * engine's ladder EXCEPT `auth`. The label picks the taxonomy, not the policy:
- * transport-shaped errors (network/transport drops, timeouts, 429/5xx, see
- * `isRetryableError`) end with `timeout`; AgentHub JSON parse errors end with
- * `malformed`; everything else — provider 4xx rejections included — ends with
- * `failed`, and the engine reconnects on all three the same. One `failed` shape is
- * additionally marked `permanent`: AgentHub's fast-mode UnsupportedParameterError
- * (see `isFastModeUnsupportedError`), a deterministic client-side rejection the
- * engine aborts on instead of retrying. User interruption ends
- * with `aborted`; credentials failures end with their own terminal status `auth` (see
- * `isAuthenticationError`) — the one status the engine refuses to retry, and the one
- * hosts key on to gate input.
+ * classifies, and `context_engine` owns the retry policy: `retryable` rides the
+ * engine's reconnect ladder, `fatal` stops the run. The split is an allowlist of
+ * certainty: only failures a retry provably cannot fix are `fatal` — a provider 4xx
+ * rejection (408/429 excluded, see `isFatalProviderRejection`), a credentials failure
+ * (`isAuthenticationError`), AgentHub's fast-mode UnsupportedParameterError
+ * (`isFastModeUnsupportedError`), and input that fails to assemble into a request at
+ * all. Everything else — network/transport drops, timeouts, 429/5xx, AgentHub parse
+ * errors and truncated streams, and every unclassifiable error — ends `retryable`,
+ * with the concrete failure on `errorMessage`. User interruption ends with `aborted`.
*
* `context_engine` only consumes OmniMessage; all Uni* protocol details are encapsulated here.
* Docs: /docs/interfaces § "The built-in implementation: GenerativeModel".
@@ -520,7 +517,7 @@ export class EventTranslator {
* as `start → delta → stop → complete message`. Closes any unclosed thinking/text segments and
* backfills their complete messages, then backfills partial(stop) + complete tool_call for
* tool_calls that only have deltas and were never emitted eagerly. All backfilled messages are
- * uniformly tagged with the interruption `stopReason` (`aborted` / `timeout` / `failed`), to
+ * uniformly tagged with the interruption `stopReason` (`aborted` / `retryable` / `fatal`), to
* distinguish them from normal completion (`completed`).
*
* Differs from `finish`: doesn't read `finish_reason`, and doesn't produce `token_usage` (an
@@ -666,8 +663,11 @@ export class EventTranslator {
}
/**
- * Converts an AgentHub finish_reason into an OmniMessage stop_reason (the six-value protocol).
- * "stop", "tool_call", and null are treated as completed; length or unknown reasons map to failed.
+ * Converts an AgentHub finish_reason into an OmniMessage stop_reason. "stop",
+ * "tool_call", and null are treated as completed; length or unknown reasons map to
+ * `fatal` — the request itself committed, so nothing retries, but the segment ended
+ * abnormally in a way only a human can fix (raise the output cap), which is what the
+ * abnormal terminal label tells the render layers.
*/
private omniStopReason(): StopReason {
if (
@@ -677,7 +677,7 @@ export class EventTranslator {
) {
return "completed";
}
- return "failed";
+ return "fatal";
}
}
@@ -755,7 +755,7 @@ export function describeError(error: unknown): string {
* (a completed response carrying thinking only, which cannot be replayed).
*
* In every case the turn was **not committed** to AgentHub history: this is not an
- * auth/parameter failure but an incomplete LLM Request, and should end with `malformed` and
+ * auth/parameter failure but an incomplete LLM Request, and should end `retryable` and
* be handed to the engine to reconnect and retry. Judged by exception type with a `name`
* fallback (covers cross-realm or deserialization-reconstructed errors), probing down the
* `cause` chain for wrapped errors.
@@ -789,7 +789,7 @@ export function isMalformedJsonParseError(error: unknown): boolean {
* thrown), AgentHub's `_validateLastEvent` reports the missing or incomplete final event as a
* plain `Error` ("Streaming response yielded no events" / "Last event must carry
* usage_metadata|finish_reason"). This is not an auth/parameter failure but an incomplete LLM
- * Request, and should end with `malformed` and be handed to the engine to reconnect and retry.
+ * Request, and should end `retryable` and be handed to the engine to reconnect and retry.
* AgentHub doesn't provide an error type for this, so we match by message prefix
* (verified against @prismshadow/agenthub 0.4.x), probing down the `cause` chain.
*/
@@ -809,28 +809,6 @@ export function isIncompleteStreamError(error: unknown): boolean {
return false;
}
-/**
- * Retryable network/transport error codes: the classic Node socket codes plus undici's
- * transport failures (`UND_ERR_*`). Node's `fetch` surfaces a dropped connection as
- * `TypeError: terminated` whose `cause` carries the real code (e.g. "other side closed"
- * with `code: "UND_ERR_SOCKET"`) — a transport disconnect, not a provider verdict, so it
- * reconnects like any network drop.
- */
-const RETRYABLE_NETWORK_CODES: ReadonlySet<string> = new Set([
- "ECONNRESET",
- "ECONNREFUSED",
- "ETIMEDOUT",
- "EPIPE",
- "EAI_AGAIN",
- "ENOTFOUND",
- "ECONNABORTED",
- // undici (Node fetch) transport failures
- "UND_ERR_SOCKET",
- "UND_ERR_CONNECT_TIMEOUT",
- "UND_ERR_HEADERS_TIMEOUT",
- "UND_ERR_BODY_TIMEOUT",
-]);
-
/** Credentials/authentication error codes and types (OpenAI-compatible bodies / SDK errors). */
const AUTH_CODES: ReadonlySet<string> = new Set([
"invalid_api_key",
@@ -901,7 +879,7 @@ export function isAuthenticationError(error: unknown): boolean {
* parameter by throwing **before any network I/O**, so with the same frozen config the
* failure is deterministic — retrying the identical request can never succeed. Deliberately
* scoped to `parameter === "fast_mode"`: other UnsupportedParameterError sources keep the
- * default `failed` classification (and the engine's ladder) unchanged. Judged by exception
+ * default `retryable` classification (and the engine's ladder) unchanged. Judged by exception
* type with a `name` fallback (covers cross-realm or deserialization-reconstructed errors,
* same approach as isMalformedJsonParseError), probing down the `cause` chain.
*/
@@ -923,78 +901,23 @@ export const FAST_MODE_UNSUPPORTED_GUIDANCE =
"Fast mode is enabled for this model; turn it off in the model settings to use this model.";
/**
- * Determines whether an error is transport-shaped — the `timeout` vs `failed` LABEL for
- * an errored request, not the retry gate: the engine retries every LLM error except
- * `auth`, whatever this returns. What the label buys is honest taxonomy for
- * observability and hosts — `timeout` reads as "connectivity/provider hiccup", `failed`
- * as "the provider rejected the request" — while both ride the same reconnect ladder.
- *
- * `timeout`-shaped: network/transport errors (including undici's `UND_ERR_*` on the
- * `cause` chain), timeouts, connection reset, HTTP 429 / 5xx.
- * `failed`-shaped: everything else, HTTP 4xx provider rejections included (a quota or
- * subscription rejection lands here too — it retries all the same and its real message
- * rides on the outcome).
- * Authentication errors are checked FIRST and are never transport-shaped (see
- * `isAuthenticationError`); JSON parse errors are classified separately as `malformed`
- * by `isMalformedJsonParseError`.
- *
- * Since AgentHub doesn't guarantee the shape of error objects, this uses a lenient check:
- * first the status code, then error codes / message keywords. When undeterminable, label
- * it `failed` — still retried, just reported for what it is.
+ * Determines whether an error is a definitive provider rejection of the request itself —
+ * the `fatal` detector for errored requests. Only an explicit HTTP client-error status
+ * counts: 4xx minus 408 (request timeout) and 429 (rate limit), which are transient by
+ * definition. Deliberately an allowlist of certainty, probed down the `cause` chain
+ * (SDKs wrap the real response error): everything this misses stays `retryable`, because
+ * retrying a genuinely fatal error costs one ladder and ends with the same message, while
+ * refusing to retry a transient one destroys the turn. Authentication is checked
+ * separately (`isAuthenticationError`) and first — a 401 is fatal through that path with
+ * its more specific message handling.
*/
-export function isRetryableError(error: unknown): boolean {
- if (error == null) return false;
-
- // 0. Authentication is definitive and terminal: retrying a dead credential can never
- // succeed, and no vocabulary heuristic below may reclassify it — a definitive
- // credential signal always wins.
- if (isAuthenticationError(error)) return false;
-
- const err = error as {
- status?: number;
- statusCode?: number;
- name?: string;
- message?: string;
- };
-
- // 1. HTTP status code takes priority.
- const status = err.status ?? err.statusCode;
- if (typeof status === "number") {
- if (status === 429 || status === 408) return true; // Rate limited / request timeout (transient)
- if (status >= 500 && status <= 599) return true; // Server error
- if (status >= 400 && status <= 499) return false; // Provider rejection: labeled failed (still retried by the engine)
- }
-
- // 2. Network/transport error codes, probing the `cause` chain (Node fetch wraps the real
- // transport failure as `TypeError: terminated` with the code on `cause`, see describeError).
- if (
- anyInCauseChain(error, (level) => {
- const code = (level as { code?: unknown }).code;
- return typeof code === "string" && RETRYABLE_NETWORK_CODES.has(code);
- })
- ) {
- return true;
- }
-
- // 3. Error name / message keywords (timeout, network, rate limit, undici disconnects).
- if (err.name === "AbortError") return false; // User interruption, not retryable
- const text = `${err.name ?? ""} ${err.message ?? ""}`.toLowerCase();
- if (
- /timeout|timed out|network|socket hang up|econnreset|connection reset|other side closed|fetch failed|too many requests|rate limit|temporarily unavailable|503|502|504/.test(
- text,
- )
- ) {
- return true;
- }
- // `terminated` alone is ambiguous — providers use the word in verdict copy too ("Request
- // terminated by content filter"), which must NOT retry — so it only counts alongside
- // transport vocabulary. The real socket case is already caught by the cause-chain
- // `UND_ERR_*` codes in step 2; this is a last-resort fallback for cause-less wrappers.
- if (/terminated/.test(text) && /socket|connection|closed/.test(text)) {
- return true;
- }
-
- return false;
+export function isFatalProviderRejection(error: unknown): boolean {
+ return anyInCauseChain(error, (level) => {
+ const err = level as { status?: unknown; statusCode?: unknown };
+ const status = typeof err.status === "number" ? err.status : err.statusCode;
+ if (typeof status !== "number") return false;
+ return status >= 400 && status <= 499 && status !== 408 && status !== 429;
+ });
}
// ---------------------------------------------------------------------------
@@ -1153,25 +1076,23 @@ export class GenerativeModel implements LLMInterface {
* and the terminal state is then returned as `LLMOutcome`:
* - **Normal completion**: `finish()` closes out and produces `token_usage` (usage is only
* produced in this case) → `completed`;
- * - **Idle timeout / network drop** (transport-shaped errors like network/429/5xx):
- * `finishInterrupted("timeout")` closes out, produces no usage → `timeout` (carrying
- * the error detail as `message` when a concrete error was caught), reconnected by
- * `context_engine` within the same run;
- * - **AgentHub JSON parse error**: `finishInterrupted("malformed")` closes out, produces no
- * usage → `malformed`, likewise reconnected by `context_engine` within the same run;
* - **User interruption**: `finishInterrupted("aborted")` closes out, produces no usage →
* `aborted`;
- * - **Credentials failure**: `finishInterrupted("auth")` closes out, produces no usage →
- * `auth` (carrying `message`) — the one status the engine stops the run on, and the one
- * hosts key on to gate input until the model's API key is updated;
- * - **Every other error** (parameters etc., and input that never assembled into a
- * request): `finishInterrupted("failed")` closes out, produces no usage → `failed`
- * (carrying `message`), which `context_engine` reconnects on as well — the
- * classification stays honest, the retry decision is the engine's. Exception: a
- * fast-mode rejection (`isFastModeUnsupportedError`, thrown deterministically before
- * any network I/O when this config enables `fast_mode` on a model without a fast
- * tier) is a `failed` marked `permanent: true`, which the engine aborts on with the
- * actionable message instead of retrying.
+ * - **Fatal failure** — a definitive provider 4xx rejection (`isFatalProviderRejection`),
+ * a credentials failure (`isAuthenticationError`), a fast-mode rejection
+ * (`isFastModeUnsupportedError`, thrown deterministically before any network I/O when
+ * this config enables `fast_mode` on a model without a fast tier), or input that never
+ * assembled into a request: `finishInterrupted("fatal")` closes out, produces no usage
+ * → `fatal` (carrying `errorMessage`), which the engine stops the run on instead of
+ * retrying — the identical request can never succeed, so the ladder would only delay
+ * the actionable message;
+ * - **Every other failure** — idle timeout, network/transport drops, 408/429/5xx,
+ * AgentHub parse errors and truncated streams, and anything unclassifiable:
+ * `finishInterrupted("retryable")` closes out, produces no usage → `retryable`
+ * (carrying `errorMessage` when a concrete error was caught), reconnected by
+ * `context_engine` within the same run. Unclassifiable errors stay retryable on
+ * purpose: the fatal detector is an allowlist, and a gateway phrasing a transient
+ * fault its own way must keep its retries.
*
* Timeout detection: the idle timer resets on every event received; once idle exceeds
* `requestTimeoutMs`, the underlying stream is aborted and handled as needing reconnection
@@ -1186,13 +1107,14 @@ export class GenerativeModel implements LLMInterface {
if (userSignal?.aborted) return { status: "aborted" };
// Input merging is placed inside a guarded block: build failures such as empty input /
- // mixed roles / argument JSON also collapse to a failed outcome, never throwing to
- // context_engine.
+ // mixed roles / argument JSON collapse to a fatal outcome — the same input can never
+ // assemble on a retry, so burning the reconnect ladder would only delay the message —
+ // and never throw to context_engine.
let uniMessage: UniMessage;
try {
uniMessage = mergeOmniToUniMessage(params.newMessages);
} catch (err) {
- return { status: "failed", errorMessage: describeError(err) };
+ return { status: "fatal", errorCode: "invalid_input", errorMessage: describeError(err) };
}
const translator = new EventTranslator(this.toolCallIds);
@@ -1239,8 +1161,8 @@ export class GenerativeModel implements LLMInterface {
ac.signal.addEventListener("abort", () => resolve(STOPPED), { once: true });
});
- // Terminal-state classification: timeout (timed out/network drop) / malformed (response
- // parse error) / aborted (user) / failed (other). null means it ended normally.
+ // Terminal-state classification: retryable (transport/parse failures) / fatal (definitive
+ // rejections) / aborted (user). null means it ended normally.
let outcome: LLMOutcome | null = null;
try {
const it = this.openStream(
@@ -1278,7 +1200,9 @@ export class GenerativeModel implements LLMInterface {
// close (best effort — a stream that ignored the signal may ignore this too, so the
// rejection is swallowed and the promise is not awaited) and classify by trigger.
void Promise.resolve(it.return?.(undefined)).catch(() => undefined);
- outcome = userSignal?.aborted ? { status: "aborted" } : { status: "timeout" };
+ outcome = userSignal?.aborted
+ ? { status: "aborted" }
+ : { status: "retryable", errorCode: "timeout" };
break;
}
if (res.done) break;
@@ -1294,43 +1218,48 @@ export class GenerativeModel implements LLMInterface {
if (userSignal?.aborted) {
outcome = { status: "aborted" };
} else if (timedOut) {
- outcome = { status: "timeout" }; // Idle timeout -> needs reconnection
+ outcome = { status: "retryable", errorCode: "timeout" }; // Idle timeout -> needs reconnection
} else if (isMalformedJsonParseError(error) || isIncompleteStreamError(error)) {
// A response JSON parse error, or a cleanly truncated stream (AgentHub's final-event
- // validation failed): both are an incomplete LLM Request, handed to the engine as
- // malformed to reconnect and retry — must not be classified as failed.
+ // validation failed): both are an incomplete LLM Request — the turn was never
+ // committed, so the engine reconnects and retries. Checked before the fatal
+ // detectors: these carry no HTTP status, but the explicit branch keeps the message
+ // and the intent readable.
outcome = {
- status: "malformed",
+ status: "retryable",
+ errorCode: "malformed",
errorMessage: describeError(error),
};
} else if (isAuthenticationError(error)) {
- // Credentials failure: its own terminal status so hosts can tell "update this
- // model's API key, then this Session can continue" (only the model reference is
- // fixed at Session creation; the credential is read from the current Project
- // config on load) apart from a one-off failure. Checked before the retryable
- // branch as a belt — isRetryableError itself already refuses auth signals.
- outcome = { status: "auth", errorMessage: describeError(error) };
+ // Credentials failure: fatal — no retry can turn a rejected credential into a
+ // working one. The errorMessage tells the user to update this model's API key
+ // (only the model reference is fixed at Session creation; the credential is read
+ // from the current Project config on load), after which the Session continues.
+ outcome = { status: "fatal", errorCode: "auth", errorMessage: describeError(error) };
} else if (this.uniConfig.fast_mode === true && isFastModeUnsupportedError(error)) {
// Fast mode rejected by a model without a fast tier: AgentHub throws its
// UnsupportedParameterError before any network I/O, so with this object's frozen
- // config the failure is deterministic — marked permanent so the engine surfaces
- // the message immediately instead of burning the reconnect ladder on a request
- // that can never succeed. The guard on our own config keeps the special case
- // honest: without fast_mode on the wire this error cannot be ours to explain.
+ // config the failure is deterministic — fatal, so the engine surfaces the message
+ // immediately instead of burning the reconnect ladder on a request that can never
+ // succeed. The guard on our own config keeps the special case honest: without
+ // fast_mode on the wire this error cannot be ours to explain.
outcome = {
- status: "failed",
- permanent: true,
+ status: "fatal",
+ errorCode: "unsupported",
errorMessage: `${describeError(error)} ${FAST_MODE_UNSUPPORTED_GUIDANCE}`,
};
- } else if (isRetryableError(error)) {
- // Transport-shaped failure (network drop, 429/5xx) -> labeled timeout. The detail
- // rides on the outcome so observability (request_end -> the Cost center's errors
- // panel) shows the real reason behind a retried request, not just "timeout".
- outcome = { status: "timeout", errorMessage: describeError(error) };
+ } else if (isFatalProviderRejection(error)) {
+ // A definitive provider 4xx rejection (408/429 excluded): the identical request
+ // fails identically on every retry, so stop now with the provider's own message.
+ outcome = { status: "fatal", errorCode: "rejected", errorMessage: describeError(error) };
} else if ((error as { name?: string })?.name === "AbortError") {
outcome = { status: "aborted" }; // Fallback: an unexpected abort (neither timeout nor user)
} else {
- outcome = { status: "failed", errorMessage: describeError(error) };
+ // Everything else — network/transport drops, 429/5xx, and anything unclassifiable —
+ // retries on the engine's ladder. The detail rides on the outcome so observability
+ // (request_end -> the Cost center's errors panel) shows the real reason behind a
+ // retried request.
+ outcome = { status: "retryable", errorCode: "network", errorMessage: describeError(error) };
}
} finally {
clearTimer();
@@ -1349,12 +1278,12 @@ export class GenerativeModel implements LLMInterface {
// tool_use (400), and the engine has no fix-up path left that touches LLM history.
if (!outcome && !translator.sawFinishReason()) {
if (userSignal?.aborted) outcome = { status: "aborted" };
- else if (timedOut) outcome = { status: "timeout" };
+ else if (timedOut) outcome = { status: "retryable", errorCode: "timeout" };
}
if (outcome) {
// Interrupted/errored: close any opened streaming segments and backfill the complete message, producing no token_usage.
- const reason: StopReason = outcome.status === "completed" ? "failed" : outcome.status;
+ const reason: StopReason = outcome.status === "completed" ? "retryable" : outcome.status;
for (const msg of translator.finishInterrupted(reason)) yield msg;
return outcome;
}
diff --git a/packages/core/src/llm/index.ts b/packages/core/src/llm/index.ts
index 63316da0..3d5de546 100644
--- a/packages/core/src/llm/index.ts
+++ b/packages/core/src/llm/index.ts
@@ -14,7 +14,7 @@ export {
usageToTokenCounts,
isMalformedJsonParseError,
isIncompleteStreamError,
- isRetryableError,
+ isFatalProviderRejection,
isAuthenticationError,
isFastModeUnsupportedError,
FAST_MODE_UNSUPPORTED_GUIDANCE,
diff --git a/packages/core/src/omnimessage/builders.ts b/packages/core/src/omnimessage/builders.ts
index 5d994dd9..8729ae21 100644
--- a/packages/core/src/omnimessage/builders.ts
+++ b/packages/core/src/omnimessage/builders.ts
@@ -5,6 +5,7 @@
* Docs: /docs/omni-message § "Builders and guards".
*/
import type {
+ ErrorCode,
AbortPayload,
ApprovalDecision,
ApprovalDecisionPayload,
@@ -34,7 +35,6 @@ import type {
Role,
SessionMetaMessage,
SessionMetaPayload,
- McpConnectStatus,
StopReason,
StreamEventType,
SubagentPayload,
@@ -274,8 +274,10 @@ export function approvalDecision(
return event({ type: "approval_decision", decision, tool_call_id: toolCallId });
}
-export function abortEvent(reason: string | null = null): OmniMessage<AbortPayload> {
- return event({ type: "abort", reason });
+export function abortEvent(
+ errorCode: "user_abort" | "backoff_interrupted" | "compaction_interrupted" = "user_abort",
+): OmniMessage<AbortPayload> {
+ return event({ type: "abort", error_code: errorCode });
}
/** request begin event: marks the start of one LLM Request. */
@@ -292,11 +294,17 @@ export function requestBegin(): OmniMessage<RequestBeginPayload> {
*/
export function requestEnd(
status: StopReason,
- retry: { errorMessage?: string; attempt?: number; retryInMs?: number } = {},
+ retry: {
+ errorCode?: ErrorCode;
+ errorMessage?: string;
+ attempt?: number;
+ retryInMs?: number;
+ } = {},
): OmniMessage<RequestEndPayload> {
return event({
type: "request_end",
status,
+ ...(retry.errorCode !== undefined ? { error_code: retry.errorCode } : {}),
...(retry.errorMessage !== undefined ? { error_message: retry.errorMessage } : {}),
...(retry.attempt !== undefined ? { attempt: retry.attempt } : {}),
...(retry.retryInMs !== undefined ? { retry_in_ms: retry.retryInMs } : {}),
@@ -325,6 +333,7 @@ export function compactionEnd(args: {
mode: CompactionMode;
status: StopReason;
attempt?: number;
+ errorCode?: ErrorCode;
errorMessage?: string;
}): OmniMessage<CompactionEndPayload> {
return event({
@@ -333,6 +342,7 @@ export function compactionEnd(args: {
mode: args.mode,
status: args.status,
...(args.attempt !== undefined ? { attempt: args.attempt } : {}),
+ ...(args.errorCode !== undefined ? { error_code: args.errorCode } : {}),
...(args.errorMessage !== undefined ? { error_message: args.errorMessage } : {}),
});
}
@@ -372,12 +382,20 @@ export function mcpConnectBegin(servers: string[]): OmniMessage<McpConnectBeginP
return event({ type: "mcp_connect_begin", servers });
}
-/** mcp_connect end event: overall status (compaction_end-style) + per-server outcomes; total wall time = end timestamp − begin timestamp. */
+/** mcp_connect end event: overall status (shared StopReason vocabulary) + per-server outcomes; total wall time = end timestamp − begin timestamp. */
export function mcpConnectEnd(args: {
- status: McpConnectStatus;
+ status: StopReason;
results: McpServerConnectResult[];
+ errorCode?: ErrorCode;
+ errorMessage?: string;
}): OmniMessage<McpConnectEndPayload> {
- return event({ type: "mcp_connect_end", status: args.status, results: args.results });
+ return event({
+ type: "mcp_connect_end",
+ status: args.status,
+ results: args.results,
+ ...(args.errorCode !== undefined ? { error_code: args.errorCode } : {}),
+ ...(args.errorMessage !== undefined ? { error_message: args.errorMessage } : {}),
+ });
}
/** Adds two sets of Token counts together, used to maintain cumulative Session usage. */
diff --git a/packages/core/src/omnimessage/types.ts b/packages/core/src/omnimessage/types.ts
index 3a9cdb48..09bc7cb6 100644
--- a/packages/core/src/omnimessage/types.ts
+++ b/packages/core/src/omnimessage/types.ts
@@ -27,19 +27,30 @@ export type Role = "user" | "assistant";
* allowed:
* - `completed`: finished normally, including completed text, thinking, tool requests, or
* tool output;
- * - `failed`: a non-retryable error or tool execution failure;
+ * - `completed`: the request (or the segment it closed) ended normally;
* - `aborted`: user-initiated interruption or cancellation;
- * - `timeout`: LLM request timed out;
- * - `malformed`: the LLM response was malformed (e.g. AgentHub JSON parsing exception).
- * Only LLM timeout / malformed trigger a context_engine reconnect;
- * - `auth`: the LLM rejected the credentials (see `isAuthenticationError`) — a
- * `failed`-shaped stop that no in-run retry can fix; hosts disable input until the
- * model's API key is updated (only the model reference is fixed at Session creation —
- * credentials come from the current Project config), after which the Session
- * continues.
+ * - `retryable`: a failure worth retrying — transport drops, timeouts, 408/429/5xx,
+ * malformed or truncated responses, and anything unclassifiable. The engine reconnects
+ * on its backoff ladder; the specific error rides on `error_message`
+ * (`LLMOutcome.errorMessage` / `request_end.error_message`), never on the reason value;
+ * - `fatal`: a failure no retry can fix — a provider 4xx rejection (invalid request,
+ * quota, and the credential failures of `isAuthenticationError`) or a deterministic
+ * client-side rejection (fast mode on a model without a fast tier). The engine stops
+ * the run and surfaces the error; the fix is a config/credential change, then a new
+ * request.
+ *
+ * The one vocabulary every stop_reason-carrying record shares — model messages, tool
+ * results, request and compaction terminals alike. Tool executions rarely have a
+ * retryable failure: a tool error or timeout is definitive for that call and is fed back
+ * to the model as a `fatal` result (nothing in the harness retries it), with `retryable`
+ * available for genuinely transient tool situations.
+ *
+ * Legacy Traces may still carry the retired values `failed` / `timeout` / `malformed` /
+ * `auth` on their records; replay logic only ever compares against `completed`, and render
+ * layers keep read-only handling for the old spellings.
* Docs: /docs/omni-message § "stop_reason".
*/
-export type StopReason = "completed" | "failed" | "aborted" | "timeout" | "malformed" | "auth";
+export type StopReason = "completed" | "aborted" | "retryable" | "fatal";
/** The event phase of a streaming fragment. `stop` marks the end of a fragment and usually carries no incremental content. */
export type StreamEventType = "start" | "delta" | "stop";
@@ -278,8 +289,45 @@ export interface ApprovalDecisionPayload {
tool_call_id: string;
}
-export interface AbortPayload {
+/**
+ * The one error vocabulary of the omnimessage layer. Every payload that reports a
+ * failure carries the same pair: `error_code` — machine-readable and stable, what
+ * render layers localize from — and `error_message` — the raw human text (provider
+ * output, exception prose), shown verbatim because it is not translatable.
+ */
+export type ErrorCode =
+ // Abort causes (a user interruption, the only thing an abort event marks).
+ | "user_abort"
+ | "backoff_interrupted"
+ | "compaction_interrupted"
+ // LLM request failures (request_end / compaction_end; the status carries the retry
+ // decision, the code carries the classified cause).
+ | "timeout" // idle timeout / connection went silent
+ | "network" // transport drops, provider 429/5xx, and anything unclassifiable
+ | "malformed" // response failed parsing/validation, or the stream was truncated
+ | "auth" // the provider rejected the credentials
+ | "rejected" // a definitive provider 4xx rejection (params, quota; 408/429 excluded)
+ | "unsupported" // a deterministic client-side rejection (fast mode without a fast tier)
+ | "invalid_input" // the input failed to assemble into a request
+ // MCP connect failures.
+ | "connect_failed";
+
+/** The shared error-reporting block (see ErrorCode). */
+export interface ErrorInfo {
+ /** Machine-readable cause; render layers localize from it. */
+ error_code?: ErrorCode;
+ /** Raw human text of the failure, shown verbatim. */
+ error_message?: string;
+}
+
+export interface AbortPayload extends ErrorInfo {
type: "abort";
+ /**
+ * Legacy field: Traces written before the unified error pair carry the cause as English
+ * prose here (including retired failure spellings from when non-user causes emitted
+ * aborts). The engine no longer writes it; renderers show it verbatim when no
+ * `error_code` is present.
+ */
reason?: string | null;
}
@@ -309,17 +357,13 @@ export interface RequestBeginPayload {
* `origin`) rather than accreting as scattered ad-hoc parameters. Every field is optional
* and additive: old Traces replay unchanged.
*/
-export interface RetryDetail {
- /**
- * Error detail, one name across the stack (`LLMOutcome.errorMessage` internally): present
- * only on non-completed statuses — the real reason behind a retried/failed Request (e.g.
- * `403 … (insufficient_user_quota)`), for observability — the server's error records /
- * Cost center read it here because a retried request never produces an abort event. Not
- * plain `message`: in this protocol "message" means an OmniMessage / model output, and
- * this field is neither. (Traces written before the rename carry it as `message`; nothing
- * reads that field semantically after the fact, so no dual-read is kept.)
- */
- error_message?: string;
+export interface RetryDetail extends ErrorInfo {
+ // error_code / error_message (ErrorInfo): present only on non-completed statuses — the
+ // classified cause and the real reason behind a retried/failed Request (e.g. `403 …
+ // (insufficient_user_quota)`); the server's error records / Cost center read them here
+ // because a failed request never produces an abort event. Not plain `message`: in this
+ // protocol "message" means an OmniMessage. (Traces written before the rename carry the
+ // text as `message`; nothing reads that field semantically after the fact.)
/**
* 1-based ordinal of this Request within its retry run — the authoritative retry count
* the CLI/Web display verbatim. Stamped on every non-completed request_end and on a
@@ -329,18 +373,19 @@ export interface RetryDetail {
attempt?: number;
/**
* Planned in-run retry wait (ms) — present ONLY when the engine will retry this failure
- * within the same run (status `timeout`/`malformed` with attempts remaining under the
+ * within the same run (status `retryable` with attempts remaining under the
* applicable cap). Computed by the same formula as the actual backoff sleep
* (`reconnectDelayMs`), so the announced wait and the real one cannot drift; the Web App
- * renders it as a live countdown to the next attempt. Absent on final failures (an abort
- * follows instead) and on completed requests.
+ * renders it as a live countdown to the next attempt. Absent on completed requests and on
+ * final failures — a non-completed request_end without `retry_in_ms` is the run's
+ * terminal record (no abort event follows: abort marks a user interruption).
*/
retry_in_ms?: number;
}
export interface RequestEndPayload extends RetryDetail {
type: "request_end";
- /** Terminal state of this Request (reuses the six StopReason values, sharing its source with this turn's complete message's stop_reason / LLMOutcome; `auth` is the credentials-failure signal hosts key on). */
+ /** Terminal state of this Request (the four StopReason values, sharing its source with this turn's complete message's stop_reason / LLMOutcome; `fatal` carries the no-retry failures — credentials included — with the detail on `error_message`). */
status: StopReason;
}
@@ -355,11 +400,12 @@ export type CompactionMode = "summarize" | "discard";
* the stream carries only each attempt's `token_usage` and the summary being written as
* ordinary `partial_text` fragments (or the complete text when nothing streamed) — the
* compaction request's other raw messages stay Trace-only. Both `reason` and
- * `mode` are carried on both events, for stateless frontend rendering; `status` reuses the
- * six-value `StopReason` protocol (compaction converges to a terminal state, taking
- * `completed` / `failed` / `aborted` in practice — `timeout` / `malformed` are handled internally
- * by the compaction request's existing retry mechanism, collapsing to `failed` once retries are
- * exhausted).
+ * `mode` are carried on both events, for stateless frontend rendering; `status` uses the
+ * shared StopReason vocabulary: `retryable` means "abandoned this time on exhausted
+ * retries — the standing trigger makes it up at the next opportunity", while `fatal`
+ * means the compaction request died on a failure no retry can fix (credentials, a
+ * provider rejection) and needs a config change first. Legacy Traces spell the abandoned
+ * case `failed`.
*/
export interface CompactionBeginPayload {
type: "compaction_begin";
@@ -374,7 +420,7 @@ export interface CompactionBeginPayload {
/**
* Inherits the shared RetryDetail block: `attempt` is the final attempt's 1-based ordinal
* (failed attempts and retries included — stamped by summarize mode), `error_message` is
- * the last failure's detail (present only when `status` is `failed`), and `retry_in_ms` is
+ * the last failure's detail (present on `retryable` / `fatal` ends), and `retry_in_ms` is
* never stamped here (compaction retries are announced on the compaction request's own
* request_end, which is Trace-only).
*/
@@ -434,29 +480,29 @@ export interface ToolListReadyPayload {
tools: ToolDefinition[];
}
-/** Terminal status of the MCP connect phase (and of each server inside it) — the same style as `compaction_end.status`. */
-export type McpConnectStatus = "completed" | "failed" | "aborted";
-
/** One MCP server's outcome inside `mcp_connect_end.results`; `duration_ms` covers that server's connect + tool discovery (individual servers have no messages of their own to derive it from). */
export interface McpServerConnectResult {
server: string;
transport: "stdio" | "http" | "sse";
- status: McpConnectStatus;
+ /** `completed` / `fatal` (connect or discovery failed; not retried within the run) / `aborted`. Legacy Traces spell the failure `failed`. */
+ status: StopReason;
duration_ms: number;
/** Number of tools discovered (present on completed). */
tools?: number;
- /** Failure detail (present on failed). */
- error?: string;
+ /** Failure cause and detail (present on fatal); legacy Traces carry the detail as `error`. */
+ error_code?: ErrorCode;
+ error_message?: string;
}
/**
* MCP connect boundary events: one pair around the first run's connect + discovery phase,
* emitted only when MCP Servers are configured. The begin lists the servers being
* contacted (frontends show a connecting status off it); the end carries the overall
- * `status` (compaction_end-style) plus the per-server results — the phase's total wall
+ * `status` (the shared `StopReason` vocabulary) plus the per-server results — the phase's total wall
* time is the pair's timestamp difference (messages carry timestamps, so the payload
* records no duplicate duration). Failures are per-server and non-fatal: an unreachable
- * server is skipped — its tools are absent — and the run continues. `status: "aborted"`
+ * server is skipped — its tools are absent — and the run continues (a failed phase is
+ * `fatal`: nothing retries it within the run). `status: "aborted"`
* means the user interrupted mid-connect: the attempt is cancelled and the next run
* reconnects from scratch. Streamed live; written to the Trace right after the run's
* input so the phase belongs to the new turn (an attempt aborted before the engine
@@ -469,10 +515,10 @@ export interface McpConnectBeginPayload {
servers: string[];
}
-export interface McpConnectEndPayload {
+export interface McpConnectEndPayload extends ErrorInfo {
type: "mcp_connect_end";
- /** Overall terminal status: completed (every server connected) / failed (some server failed) / aborted (user interrupted). */
- status: McpConnectStatus;
+ /** Overall terminal status: completed (every server connected) / fatal (some server failed) / aborted (user interrupted). Legacy Traces spell the failure `failed`. */
+ status: StopReason;
/** Per-server outcomes (empty on an aborted attempt — nothing settled is claimed). */
results: McpServerConnectResult[];
}
diff --git a/packages/core/src/session.ts b/packages/core/src/session.ts
index 539635b4..ee9f25a1 100644
--- a/packages/core/src/session.ts
+++ b/packages/core/src/session.ts
@@ -38,6 +38,7 @@ import { runGoalLoop } from "./goal/goal-loop.js";
import { goalFinishedOf } from "./goal/goal-stream.js";
import type {
ApproveFn,
+ RunCutoff,
BackgroundCommandInfo,
BackgroundSubagentInfo,
SubagentMessageOptions,
@@ -379,7 +380,10 @@ export class Session {
* the final message is exactly one `goal_finished` event carrying the outcome.
* Docs: /docs/goal-mode.
*/
- async *run(newMessages: OmniMessage[], opts?: SessionRunOptions): AsyncGenerator<OmniMessage> {
+ async *run(
+ newMessages: OmniMessage[],
+ opts?: SessionRunOptions,
+ ): AsyncGenerator<OmniMessage, RunCutoff | null> {
// The sandbox command policy is applied here, at the Human boundary itself: the
// injected callback is wrapped so a vetoed command is refused before the host is ever
// asked, which is what makes the policy outrank every approval mode — no host, and no
@@ -391,18 +395,20 @@ export class Session {
}
if (opts?.goal) {
// Rounds run with the caller's per-call options minus `goal` (each round is a plain Task).
+ // Per-round cutoffs are consumed by the goal loop itself; goal mode always closes
+ // with its own goal_finished terminal event.
const { goal, ...roundOpts } = opts;
yield* this.runGoal(newMessages, goal, roundOpts);
- return;
+ return null;
}
- yield* this.runTask(newMessages, opts);
+ return yield* this.runTask(newMessages, opts);
}
/** The single-Task path (a goal round runs one of these per round). */
private async *runTask(
newMessages: OmniMessage[],
opts?: RunOptions,
- ): AsyncGenerator<OmniMessage> {
+ ): AsyncGenerator<OmniMessage, RunCutoff | null> {
// Folded before Trace and title material, so the path lines are what gets recorded.
if (!this.modelHasVision) newMessages = await this.foldImages(newMessages);
// A previously aborted bootstrap dropped these before the engine existed: they lead
@@ -434,7 +440,7 @@ export class Session {
this.abortedBootstrapRecords = [];
// Carry the merged list, so repeated aborts keep accumulating exactly once each.
this.carryOverInput = newMessages;
- return;
+ return { kind: "abort", errorCode: "user_abort" };
}
// Self-captures title material (the title is derived from the first-turn
// conversation text): while material isn't frozen yet, collect this call's user text and
@@ -451,7 +457,17 @@ export class Session {
);
}
}
- for await (const msg of this.engine!.run(newMessages, opts)) {
+ // Manual iteration (not for-await) so the engine's return value — how the run ended —
+ // propagates to this generator's own return.
+ const it = this.engine!.run(newMessages, opts);
+ let cutoff: RunCutoff | null = null;
+ for (;;) {
+ const res = await it.next();
+ if (res.done) {
+ cutoff = res.value;
+ break;
+ }
+ const msg = res.value;
if (capture) {
this.titleAssistantText = appendTitleText(
this.titleAssistantText,
@@ -463,6 +479,7 @@ export class Session {
yield msg;
}
if (capture && this.titleUserText.trim()) this.titleMaterialFrozen = true;
+ return cutoff;
}
/**
@@ -514,16 +531,23 @@ export class Session {
yield end;
this.abortedBootstrapRecords.push(end);
}
- const aborted = abortEvent("user");
+ const aborted = abortEvent("user_abort");
yield aborted;
this.abortedBootstrapRecords.push(aborted);
return false;
}
const { tools, llm, mcp } = outcome.value;
if (this.mcpServers.length > 0) {
+ const failed = mcp.filter((r) => r.status !== "completed").map((r) => r.server);
const end = mcpConnectEnd({
- status: mcp.some((r) => r.status !== "completed") ? "failed" : "completed",
+ status: failed.length > 0 ? "fatal" : "completed",
results: mcp,
+ ...(failed.length > 0
+ ? {
+ errorCode: "connect_failed" as const,
+ errorMessage: `unavailable: ${failed.join(", ")}`,
+ }
+ : {}),
});
yield end;
records.push(end);
diff --git a/packages/core/test/agent.test.ts b/packages/core/test/agent.test.ts
index 0df5d8dd..9abed3e1 100644
--- a/packages/core/test/agent.test.ts
+++ b/packages/core/test/agent.test.ts
@@ -14,6 +14,7 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import type { OmniMessage } from "../src/omnimessage/index.js";
import {
addModel,
createAgent,
@@ -390,19 +391,19 @@ describe("run_subagent spawning follows the PARENT session (never the Project de
const gen = handle.run({ messages: [userText("noop")] });
const first = await gen.next();
expect(first.done).toBe(false);
- const msg = first.value!;
+ const msg = first.value as OmniMessage;
expect(msg.type).toBe("session_meta");
// Child messages are stamped with the child Session id as the origin hop.
expect(msg.origin?.[0]).toBe(handle.sessionId);
for (;;) {
const next = await gen.next();
expect(next.done).toBe(false);
- if ((next.value!.payload as { type?: string }).type === "tool_list_ready") {
+ if (((next.value as OmniMessage).payload as { type?: string }).type === "tool_list_ready") {
llm = capturedLLMConfigs.list.at(-1)!;
break;
}
}
- await gen.return(undefined);
+ await gen.return(null);
return {
...(msg.payload as {
provider: string;
@@ -614,7 +615,7 @@ describe("run_subagent spawning follows the PARENT session (never the Project de
role: "user",
text: "count the TODO items",
});
- await gen.return(undefined);
+ await gen.return(null);
} finally {
handle.dispose();
parent.dispose();
diff --git a/packages/core/test/background-execution.test.ts b/packages/core/test/background-execution.test.ts
index f090d9ed..7fe93fbf 100644
--- a/packages/core/test/background-execution.test.ts
+++ b/packages/core/test/background-execution.test.ts
@@ -17,7 +17,7 @@ import { SubagentSessionManager } from "../src/environment/tools/subagent/index.
import { DEFAULT_EMPTY_POLL_YIELD_MS } from "../src/environment/tools/command/index.js";
import { ContextEngine } from "../src/engine/context-engine.js";
import {
- abortEvent,
+ requestEnd,
assistantText,
buildBackgroundTaskDoneMessage,
emptyTokenCounts,
@@ -278,18 +278,18 @@ describe("exec_command run_in_background", () => {
expect(events).toHaveLength(0);
// And the session is gone from the registry.
const again = await runTool(env, "input_command", { process_id: id, kill: true });
- expect(again.stopReason).toBe("failed");
+ expect(again.stopReason).toBe("fatal");
});
it("input_command kill fails on an unknown process_id; the legacy kill_command name is an unknown tool", async () => {
const { env } = await makeEnv();
const res = await runTool(env, "input_command", { process_id: "proc-deadbeef", kill: true });
- expect(res.stopReason).toBe("failed");
+ expect(res.stopReason).toBe("fatal");
expect(res.output).toContain("unknown process_id");
// A stale stored config may still carry the removed tool; the registry no longer
// assembles it, so a model call collapses to the standard unknown-tool failure.
const legacy = await runTool(env, "kill_command", { process_id: "proc-deadbeef" });
- expect(legacy.stopReason).toBe("failed");
+ expect(legacy.stopReason).toBe("fatal");
expect(legacy.output).toContain("Unknown tool: kill_command");
});
@@ -565,8 +565,14 @@ describe("run_subagent run_in_background", () => {
const manager = new SubagentSessionManager();
cleanups.push(() => manager.dispose());
const runner = runnerOf(async function* () {
- // A child-session failure surfaces as an origin-tagged abort event, never a throw.
- yield withHop(abortEvent("llm request failed after 5 retries"));
+ // A child-session failure surfaces as its origin-tagged terminal record plus the
+ // run's return value (Session.run's contract), never a throw.
+ yield withHop(requestEnd("retryable", { attempt: 6, errorMessage: "socket hang up" }));
+ return {
+ kind: "llm_failure" as const,
+ errorCode: "network" as const,
+ errorMessage: "socket hang up",
+ };
});
const events: BackgroundTaskDoneEvent[] = [];
const tapped: OmniMessage[] = [];
@@ -581,9 +587,9 @@ describe("run_subagent run_in_background", () => {
extractSubagentId(res.note);
await waitFor(() => events.length === 1);
expect(events[0]).toMatchObject({ kind: "subagent", status: "failed" });
- expect(events[0]!.detail).toContain("aborted");
- // The failure event reached the live tap too (origin-tagged), not a poll buffer.
- expect(tapped.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(true);
+ expect(events[0]!.detail).toContain("socket hang up");
+ // The failure record reached the live tap too (origin-tagged), not a poll buffer.
+ expect(tapped.some((m) => (m.payload as { type?: string }).type === "request_end")).toBe(true);
});
it("a background child's approvals resolve through the launching call's standing sink", async () => {
@@ -623,7 +629,7 @@ describe("run_subagent run_in_background", () => {
cleanups.push(() => manager.dispose());
const input = createInputSubagentTool(SUB_INPUT_DEF, { subagentSessions: manager });
const res = await drive(input, { subagent_id: "subagent-deadbeef" });
- expect(res.stopReason).toBe("failed");
+ expect(res.stopReason).toBe("fatal");
expect(res.output).toContain("unknown subagent_id");
});
});
diff --git a/packages/core/test/compaction.test.ts b/packages/core/test/compaction.test.ts
index d8148561..ee2a24f5 100644
--- a/packages/core/test/compaction.test.ts
+++ b/packages/core/test/compaction.test.ts
@@ -84,7 +84,7 @@ class ScriptedLLM implements LLMInterface {
this.calls.push(params.newMessages);
const next = this.responses.shift();
if (!next) {
- return { status: "failed", errorMessage: `${this.label}: no scripted response` };
+ return { status: "retryable", errorMessage: `${this.label}: no scripted response` };
}
for (const msg of next.messages) yield msg;
return next.outcome ?? { status: "completed" };
@@ -140,6 +140,18 @@ async function collect(gen: AsyncGenerator<OmniMessage>): Promise<OmniMessage[]>
return all;
}
+/** Like collect, but also captures the run generator's return value (RunCutoff | null). */
+async function collectWithReturn<R>(
+ gen: AsyncGenerator<OmniMessage, R>,
+): Promise<{ all: OmniMessage[]; cutoff: R }> {
+ const all: OmniMessage[] = [];
+ for (;;) {
+ const res = await gen.next();
+ if (res.done) return { all, cutoff: res.value };
+ all.push(res.value);
+ }
+}
+
type CompactionEventPayload = CompactionBeginPayload | CompactionEndPayload;
const compactionEvents = (msgs: OmniMessage[]): CompactionEventPayload[] =>
@@ -318,8 +330,8 @@ describe("context compaction", () => {
{
messages: [toolCall({ name: "t", arguments: "{}", toolCallId: "c1" }), usage(150, 150)],
},
- // Compaction request fails on the one status no ladder can fix (a rejected credential).
- { messages: [], outcome: { status: "auth", errorMessage: "auth error" } },
+ // Compaction request fails on the status no ladder can fix (a rejected credential).
+ { messages: [], outcome: { status: "fatal", errorMessage: "auth error" } },
// Original context is kept. The mid-task failure ends the run by the interruption
// flow, so the next run resends this round's tool output with the user's message and
// the task finishes on the old instance (context usage keeps growing).
@@ -346,14 +358,20 @@ describe("context compaction", () => {
});
const oldPath = trace.currentPath();
- const out = await collect(engine.run([userText("go")], { approve: allowAll }));
- // The mid-task failure ends this run: stop=failed, original context kept, no LLM swap.
+ const { all: out, cutoff } = await collectWithReturn(
+ engine.run([userText("go")], { approve: allowAll }),
+ );
+ // The mid-task failure ends this run: stop=failed, original context kept, no LLM swap —
+ // and the run generator returns the cutoff (the compaction_end's error pair mirrored).
+ expect(cutoff).toMatchObject({ kind: "compaction_failure" });
expect(
compactionEvents(out).map(
(e) => `${e.type}:${(e as Partial<CompactionEndPayload>).status ?? ""}`,
),
- ).toEqual(["compaction_begin:", "compaction_end:failed"]);
- expect(payloadTypes(out)).toContain("abort");
+ ).toEqual(["compaction_begin:", "compaction_end:fatal"]);
+ // No abort event — abort marks a user interruption; the compaction_end is the
+ // failure's terminal record and the run just ends on it.
+ expect(payloadTypes(out)).not.toContain("abort");
expect(created).toBe(0);
// The next run resends the held tool output with the new message, and the compaction is
@@ -415,12 +433,12 @@ describe("context compaction", () => {
expect((newTrace[2]!.payload as { text?: string }).text).toBe("task 2");
});
- it("reconnect exhaustion on the compaction request converges to failed", async () => {
+ it("reconnect exhaustion on the compaction request converges to retryable", async () => {
const llm1 = new ScriptedLLM(
[
{ messages: [assistantText("answer"), usage(150, 150)] },
- { messages: [], outcome: { status: "timeout" } },
- { messages: [], outcome: { status: "timeout" } },
+ { messages: [], outcome: { status: "retryable" } },
+ { messages: [], outcome: { status: "retryable" } },
],
"llm1",
);
@@ -438,7 +456,7 @@ describe("context compaction", () => {
// The end event reports the attempts spent (issue #170's cost-center row message).
expect(events[1]).toMatchObject({
type: "compaction_end",
- status: "failed",
+ status: "retryable",
attempt: 2,
});
// The retry resends the original input (tool results + prompt; here there are no tool results, just the prompt).
@@ -452,7 +470,7 @@ describe("context compaction", () => {
// this script would have failed before reaching the 5th, succeeding attempt.
const failing = (n: number): ScriptedResponse => ({
messages: [],
- outcome: { status: "failed", errorMessage: `blip ${n}` },
+ outcome: { status: "retryable", errorMessage: `blip ${n}` },
});
const llm1 = new ScriptedLLM(
[
@@ -484,15 +502,15 @@ describe("context compaction", () => {
expect(llm1.calls).toHaveLength(6);
});
- it("a failed compaction request takes the ladder and can recover on a later attempt", async () => {
- // The compaction loop retries the same statuses the turn loop does: `failed` is where a
- // transient fault lands whenever the classifier doesn't recognize the gateway's wording,
- // and giving up here keeps the full context, so the next request re-triggers compaction
- // against the same wall with less headroom.
+ it("a retryable compaction request takes the ladder and can recover on a later attempt", async () => {
+ // The compaction loop retries the same status the turn loop does: `retryable` is where
+ // a transient fault lands whenever the fatal detector doesn't recognize the gateway's
+ // wording, and giving up here keeps the full context, so the next request re-triggers
+ // compaction against the same wall with less headroom.
const llm1 = new ScriptedLLM(
[
{ messages: [assistantText("answer"), usage(150, 150)] },
- { messages: [], outcome: { status: "failed", errorMessage: "502 upstream" } },
+ { messages: [], outcome: { status: "retryable", errorMessage: "502 upstream" } },
{ messages: [assistantText("[summary]recovered[/summary]")] },
],
"llm1",
@@ -526,9 +544,9 @@ describe("context compaction", () => {
const llm1 = new ScriptedLLM(
[
{ messages: [assistantText("answer"), usage(150, 150)] },
- { messages: [], outcome: { status: "timeout" } },
- { messages: [], outcome: { status: "timeout" } },
- { messages: [], outcome: { status: "timeout" } },
+ { messages: [], outcome: { status: "retryable" } },
+ { messages: [], outcome: { status: "retryable" } },
+ { messages: [], outcome: { status: "retryable" } },
// Never reached: the compaction cap stops at 3 compaction attempts in total.
{ messages: [assistantText("[summary]late[/summary]")] },
],
@@ -552,7 +570,7 @@ describe("context compaction", () => {
const out = await collect(engine.run([userText("go")], { approve: allowAll }));
const events = compactionEvents(out);
- expect(events[1]).toMatchObject({ type: "compaction_end", status: "failed" });
+ expect(events[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
// 1 turn request + 3 compaction attempts (initial + compactionMaxReconnects retries).
expect(llm1.calls).toHaveLength(4);
// The Trace-written compaction request_ends announce the planned backoff under the
@@ -571,7 +589,7 @@ describe("context compaction", () => {
expect(attempts).toEqual([undefined, 1, 2, 3]);
});
- it("an empty compaction response (thinking only, no text) is retried like any failure: 5 attempts, then failed with the context kept", async () => {
+ it("an empty compaction response (thinking only, no text) is retried like any failure: 5 attempts, then retryable with the context kept", async () => {
// Issue #83/#170: the compaction request completes but yields no text. Committing the
// empty summary would discard the whole context and lose the task state — the response
// counts as one more failed attempt on the unified reconnect budget, and once the budget
@@ -618,7 +636,7 @@ describe("context compaction", () => {
const events = compactionEvents(out1);
expect(
events.map((e) => `${e.type}:${(e as Partial<CompactionEndPayload>).status ?? ""}`),
- ).toEqual(["compaction_begin:", "compaction_end:failed"]);
+ ).toEqual(["compaction_begin:", "compaction_end:retryable"]);
expect(events[1]).toMatchObject({
attempt: 5,
error_message: "the response contained no usable summary",
@@ -703,7 +721,7 @@ describe("context compaction", () => {
const out = await collect(engine.run([userText("go")], { approve: allowAll }));
const events = compactionEvents(out);
- expect(events[1]).toMatchObject({ type: "compaction_end", status: "failed" });
+ expect(events[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
// The rejected tool calls are never approved or executed, and nothing of the compaction
// dialogue (repairs included) reaches the output stream.
expect(payloadTypes(out)).not.toContain("approval_decision");
@@ -731,7 +749,7 @@ describe("context compaction", () => {
expect(repair.output).toBe(
"[tool error] the compaction request expects a summary, not tool calls",
);
- expect(repair.stop_reason).toBe("failed");
+ expect(repair.stop_reason).toBe("fatal");
expect(textOf(retry[1]!)).toBe(SUMMARY_RETRY_GUIDANCE);
expect(textOf(retry[2]!)).toBe("COMPACT NOW");
}
@@ -878,7 +896,7 @@ describe("context compaction", () => {
expect(repair.output).toBe(
"[tool error] the compaction request expects a summary, not tool calls",
);
- expect(repair.stop_reason).toBe("failed");
+ expect(repair.stop_reason).toBe("fatal");
expect(textOf(retry[1]!)).toBe(SUMMARY_RETRY_GUIDANCE);
expect(textOf(retry[2]!)).toBe("COMPACT NOW");
@@ -909,7 +927,7 @@ describe("context compaction", () => {
{ messages: [assistantText("answer"), usage(150, 150)] },
{
messages: [
- toolCall({ name: "t", arguments: "", toolCallId: "cz", stopReason: "timeout" }),
+ toolCall({ name: "t", arguments: "", toolCallId: "cz", stopReason: "retryable" }),
assistantText("[summary]still fine[/summary]"),
],
},
@@ -993,11 +1011,13 @@ describe("context compaction", () => {
const out1 = await collect(engine.run([userText("task one")], { approve: allowAll }));
- expect(compactionEvents(out1)[1]).toMatchObject({ type: "compaction_end", status: "failed" });
- // The abandonment ends the run through the interruption flow: an abort follows the
- // failed compaction_end, and no further request is issued in this run.
- const abort = out1.find((m) => (m.payload as { type?: string }).type === "abort");
- expect(abort?.payload).toMatchObject({ reason: "compaction failed" });
+ expect(compactionEvents(out1)[1]).toMatchObject({
+ type: "compaction_end",
+ status: "retryable",
+ });
+ // The abandonment ends the run with the compaction_end as its terminal record (no
+ // abort event — abort marks a user interruption); no further request is issued.
+ expect(payloadTypes(out1)).not.toContain("abort");
expect(llm1.calls).toHaveLength(6);
// Attempt 1 folds the turn's tool output in; attempt 2 carries the repair + note + Prompt
// but NOT the absorbed output; attempts 3-5 are note + Prompt.
@@ -1052,8 +1072,8 @@ describe("context compaction", () => {
});
const out = await collect(engine.run([userText("go")], { approve: allowAll }));
- expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "failed" });
- expect(payloadTypes(out)).toContain("abort");
+ expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
+ expect(payloadTypes(out)).not.toContain("abort");
expect(llm1.calls).toHaveLength(6);
// Attempt 1 folds the outputs; attempts 2-5 are note + Prompt (absorbed by the first commit).
expect(payloadTypes(llm1.calls[1]!)).toEqual(["tool_call_output", "text"]);
@@ -1082,8 +1102,8 @@ describe("context compaction", () => {
{
messages: [toolCall({ name: "t", arguments: "{}", toolCallId: "ct" }), usage(150, 150)],
},
- { messages: [], outcome: { status: "timeout" } },
- { messages: [], outcome: { status: "timeout" } },
+ { messages: [], outcome: { status: "retryable" } },
+ { messages: [], outcome: { status: "retryable" } },
// Next run: the carried outputs lead, then the new prompt (under the threshold).
{ messages: [assistantText("done on old context"), usage(60, 400)] },
],
@@ -1099,8 +1119,8 @@ describe("context compaction", () => {
});
const out = await collect(engine.run([userText("go")], { approve: allowAll }));
- expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "failed" });
- expect(payloadTypes(out)).toContain("abort");
+ expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
+ expect(payloadTypes(out)).not.toContain("abort");
expect(llm1.calls).toHaveLength(3);
await collect(engine.run([userText("carry on")], { approve: allowAll }));
@@ -1432,8 +1452,8 @@ describe("context compaction", () => {
[
{ messages: [assistantText("hi"), usage(10, 10)] },
// Manual compaction attempts: transport failures only — nothing committed.
- { messages: [], outcome: { status: "timeout" } },
- { messages: [], outcome: { status: "timeout" } },
+ { messages: [], outcome: { status: "retryable" } },
+ { messages: [], outcome: { status: "retryable" } },
// Run 3: the restored carry-over leads the input, exactly as before the compact().
{ messages: [assistantText("resumed"), usage(20, 40)] },
],
@@ -1456,7 +1476,7 @@ describe("context compaction", () => {
expect(llm1.calls).toHaveLength(1); // the aborted run never reached the LLM
const out = await collect(engine.compact());
- expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "failed" });
+ expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
// The compaction folded the carry-over into its (uncommitted) attempts...
expect(llm1.calls).toHaveLength(3);
expect(llm1.calls[1]!.map(textOf)).toEqual(["pending question", "COMPACT NOW"]);
@@ -1504,7 +1524,7 @@ describe("context compaction", () => {
);
const out = await collect(engine.compact());
- expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "failed" });
+ expect(compactionEvents(out)[1]).toMatchObject({ type: "compaction_end", status: "retryable" });
expect(llm1.calls).toHaveLength(6);
// Attempt 1 folded the carry-over in (and committed it); later attempts resend the note +
// Prompt — the absorbed carry-over never again.
@@ -2025,7 +2045,7 @@ describe("mid-task compaction runs the tools first (issue #288)", () => {
yield toolCallOutput({
output: "[tool error] boom",
toolCallId: tc.payload.tool_call_id,
- stopReason: "failed",
+ stopReason: "fatal",
});
},
toolPermission() {
@@ -2052,7 +2072,7 @@ describe("mid-task compaction runs the tools first (issue #288)", () => {
expect(payloadTypes(compactionInput)).toEqual(["tool_call_output", "text"]);
const out = compactionInput[0]!.payload as { output: string; stop_reason?: string };
expect(out.output).toBe("[tool error] boom");
- expect(out.stop_reason).toBe("failed");
+ expect(out.stop_reason).toBe("fatal");
});
it("a denied tool still pairs: the denial output rides the compaction request", async () => {
diff --git a/packages/core/test/describe-image.test.ts b/packages/core/test/describe-image.test.ts
index f822492c..709c499c 100644
--- a/packages/core/test/describe-image.test.ts
+++ b/packages/core/test/describe-image.test.ts
@@ -123,7 +123,7 @@ describe("describe_image (the text-only-model variant of read_image)", () => {
}
text += (res.value.payload as { output?: string }).output ?? "";
}
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("No vision model");
});
@@ -170,20 +170,20 @@ describe("describe_image (the text-only-model variant of read_image)", () => {
it("no vision model configured: failed plus an explanation of how to configure one", async () => {
await writeFile(path.join(tmp, "a.png"), PNG_1X1);
const { result, text } = await run({ source: "a.png" }, tmp, { modelId: null });
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("No vision model");
expect(text).toContain("vision_model");
});
it("vision model request failure: failed with the status and message", async () => {
await writeFile(path.join(tmp, "a.png"), PNG_1X1);
- const { llm } = fakeLLM("", { status: "failed", errorMessage: "401 unauthorized" });
+ const { llm } = fakeLLM("", { status: "fatal", errorMessage: "401 unauthorized" });
const { result, text } = await run({ source: "a.png" }, tmp, {
modelId: "vis-1",
createLLM: () => llm,
});
- expect(result?.stopReason).toBe("failed");
- expect(text).toContain("failed");
+ expect(result?.stopReason).toBe("fatal");
+ expect(text).toContain("fatal");
expect(text).toContain("401 unauthorized");
});
@@ -194,7 +194,7 @@ describe("describe_image (the text-only-model variant of read_image)", () => {
modelId: "vis-1",
createLLM: () => llm,
});
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(calls).toHaveLength(0);
});
diff --git a/packages/core/test/engine.test.ts b/packages/core/test/engine.test.ts
index 9e875887..118cd988 100644
--- a/packages/core/test/engine.test.ts
+++ b/packages/core/test/engine.test.ts
@@ -28,6 +28,7 @@ import {
} from "../src/omnimessage/index.js";
import { BUILTIN_TOOL_FACTORIES } from "../src/environment/tools/registry.js";
import type {
+ RunCutoff,
GenerativeModelParameters,
LLMInterface,
LLMOutcome,
@@ -118,6 +119,22 @@ const allowAll: ApproveFn = async () => "allow";
const denyAll: ApproveFn = async () => "deny";
/** Collects all output from a run. */
+/** Like collectRun, but also captures the run generator's return value (the RunCutoff | null contract). */
+async function collectRunWithReturn(
+ engine: ContextEngine,
+ prompt: OmniMessage[],
+ approve: ApproveFn,
+ signal?: AbortSignal,
+): Promise<{ all: OmniMessage[]; cutoff: RunCutoff | null }> {
+ const all: OmniMessage[] = [];
+ const it = engine.run(prompt, { approve, ...(signal ? { signal } : {}) });
+ for (;;) {
+ const res = await it.next();
+ if (res.done) return { all, cutoff: res.value };
+ all.push(res.value);
+ }
+}
+
async function collectRun(
engine: ContextEngine,
prompt: OmniMessage[],
@@ -568,7 +585,7 @@ describe("ContextEngine ReAct loop (mock LLM, approve callback)", () => {
expect(partials).toMatchObject([
{ event_type: "start", text: "" },
{ event_type: "delta", text: maxTurnText },
- { event_type: "stop", text: "", stop_reason: "failed" },
+ { event_type: "stop", text: "", stop_reason: "fatal" },
]);
// No more leading newline.
expect(maxTurnText.startsWith("\n")).toBe(false);
@@ -725,8 +742,8 @@ describe("ContextEngine ReAct loop (mock LLM, approve callback)", () => {
if (++calls === 1) {
yield partialText("start", "");
yield partialText("delta", "half a thought");
- // `auth`: the one LLM status that still exits straight to the flatten path.
- return { status: "auth", errorMessage: "boom" };
+ // `fatal`: exits straight to the flatten path (no retry).
+ return { status: "fatal", errorMessage: "boom" };
}
yield assistantText("ok");
yield tokenUsage(emptyTokenCounts(), {
@@ -971,8 +988,8 @@ describe("ContextEngine ReAct loop (mock LLM, approve callback)", () => {
calls += 1;
inputs.push(params.newMessages);
if (calls === 1) {
- yield assistantText("half", "timeout");
- return { status: "timeout" };
+ yield assistantText("half", "retryable");
+ return { status: "retryable" };
}
yield assistantText("recovered");
yield tokenUsage(emptyTokenCounts(), {
@@ -1022,8 +1039,8 @@ describe("ContextEngine ReAct loop (mock LLM, approve callback)", () => {
levels.push(params.thinkingLevel);
if (calls === 1) {
// First attempt drops: the reconnect retry must carry the same per-turn level.
- yield assistantText("half", "timeout");
- return { status: "timeout" };
+ yield assistantText("half", "retryable");
+ return { status: "retryable" };
}
if (calls === 2) {
// Retry completes with usage above the compaction threshold → a Task-boundary
@@ -1266,7 +1283,7 @@ describe("ContextEngine tool execution resilience", () => {
yield toolCallOutput({
output: "[tool error] boom",
toolCallId: id,
- stopReason: "failed",
+ stopReason: "fatal",
});
},
};
@@ -1308,11 +1325,11 @@ describe("ContextEngine tool execution resilience", () => {
for await (const msg of engine.run([userText("go")], { approve: allowAll })) {
all.push(msg);
}
- // A failed failure output was produced, and the Task normally advanced to the second turn.
+ // A fatal failure output was produced, and the Task normally advanced to the second turn.
const failed = all.find(
(m) =>
(m.payload as { type?: string }).type === "tool_call_output" &&
- (m.payload as { stop_reason?: string }).stop_reason === "failed",
+ (m.payload as { stop_reason?: string }).stop_reason === "fatal",
);
expect(failed).toBeDefined();
expect(
@@ -1368,7 +1385,7 @@ describe("ContextEngine tool execution resilience", () => {
);
expect(outputs).toHaveLength(1);
expect((outputs[0]!.payload as { output: string }).output).toContain("custom env exploded");
- expect((outputs[0]!.payload as { stop_reason?: string }).stop_reason).toBe("failed");
+ expect((outputs[0]!.payload as { stop_reason?: string }).stop_reason).toBe("fatal");
yield assistantText("survived");
yield usage();
return { status: "completed" };
@@ -1623,9 +1640,9 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
// returning timeout.
yield partialText("start");
yield partialText("delta", "thinking...");
- yield partialText("stop", "", "timeout");
- yield assistantText("thinking...", "timeout");
- return { status: "timeout" };
+ yield partialText("stop", "", "retryable");
+ yield assistantText("thinking...", "retryable");
+ return { status: "retryable" };
}
// Retry succeeds: completes normally.
yield assistantText("done");
@@ -1685,9 +1702,9 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
name: "exec_command",
arguments: '{"cmd": "ec',
toolCallId: "tc-broken",
- stopReason: "malformed",
+ stopReason: "retryable",
});
- return { status: "malformed", errorMessage: "incomplete stream" };
+ return { status: "retryable", errorMessage: "incomplete stream" };
}
yield assistantText("done");
yield tokenUsage(emptyTokenCounts(), {
@@ -1740,10 +1757,10 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
if (calls === 1) {
yield partialText("start");
yield partialText("delta", "partial json response");
- yield partialText("stop", "", "malformed");
- yield assistantText("partial json response", "malformed");
+ yield partialText("stop", "", "retryable");
+ yield assistantText("partial json response", "retryable");
return {
- status: "malformed",
+ status: "retryable",
message: "Unexpected token < in JSON at position 0",
};
}
@@ -1787,15 +1804,15 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(all.map((m) => (m.payload as { type?: string }).type)).not.toContain("abort");
});
- it("emits abort and carries the original input over when reconnect retries are exhausted", async () => {
+ it("ends the run without an abort event and carries the original input over when reconnect retries are exhausted", async () => {
let calls = 0;
const inputs: OmniMessage[][] = [];
const llm: LLMInterface = {
async *streamGenerate(params) {
calls += 1;
inputs.push(params.newMessages);
- yield assistantText("partial...", "timeout");
- return { status: "timeout" }; // Always needs a reconnect.
+ yield assistantText("partial...", "retryable");
+ return { status: "retryable" }; // Always needs a reconnect.
},
};
const environment = new Environment({
@@ -1811,9 +1828,18 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
const all = await collectRun(engine, [userText("go")], allowAll);
expect(calls).toBe(2); // Initial attempt + maxReconnects(1) retries.
- const abort = all.find((m) => (m.payload as { type?: string }).type === "abort");
- expect(abort).toBeDefined();
- expect((abort!.payload as { reason?: string }).reason).toContain("reconnect failed");
+ // No abort event — abort marks a user interruption; the last failure's request_end
+ // (retryable, no retry_in_ms since no retry is planned) is the terminal record.
+ expect(all.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(false);
+ const ends = all.filter((m) => (m.payload as { type?: string }).type === "request_end");
+ const last = ends[ends.length - 1]!.payload as {
+ status?: string;
+ attempt?: number;
+ retry_in_ms?: number;
+ };
+ expect(last.status).toBe("retryable");
+ expect(last.attempt).toBe(2);
+ expect(last.retry_in_ms).toBeUndefined();
// carry-over = original input + [turn_retried] (accumulating partial products from both
// failed attempts): the next run resends it merged with the new input, without producing
@@ -1828,19 +1854,21 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(nextRunTexts.join("\n")).not.toContain("[turn_aborted]");
});
- it("a failed outcome retries like a timeout, then converges to a graceful abort (run does not throw)", async () => {
+ it("a retryable outcome with a message converges to a terminal request_end carrying it (run does not throw)", async () => {
let calls = 0;
const inputs: OmniMessage[][] = [];
const llm: LLMInterface = {
// The LLM must never throw an exception at the engine: an error resolves by returning
- // a failed outcome after closing the structure. `failed` is still the honest
- // classification for a parameter error — it is simply retried anyway, because the
- // classifier cannot reliably tell a permanent 4xx from a gateway's transient one.
+ // a retryable outcome after closing the structure; the concrete failure rides on
+ // errorMessage and must survive into the exhaustion abort verbatim.
// eslint-disable-next-line require-yield
async *streamGenerate(params) {
calls += 1;
inputs.push(params.newMessages);
- return { status: "failed", errorMessage: "400 unknown parameter: max_output_tokens" };
+ return {
+ status: "retryable",
+ errorMessage: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
+ };
},
};
const environment = new Environment({
@@ -1854,17 +1882,33 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
reconnectBackoffMs: 0,
});
- // Must not throw; should gracefully converge to an abort once the ladder is spent.
- const all = await collectRun(engine, [userText("go")], allowAll);
- expect(calls).toBe(3); // initial attempt + maxReconnects(2): `failed` takes the ladder now.
- const abort = all.find((m) => (m.payload as { type?: string }).type === "abort");
- expect(abort).toBeDefined();
- // Asserted whole, not by fragments: this string is shown verbatim in the error panel and
- // the CLI and is persisted as the error message, so its grammar is part of the contract.
- const reason = (abort!.payload as { reason?: string }).reason ?? "";
- expect(reason).toBe(
- "llm request failed after 2 retries: 400 unknown parameter: max_output_tokens",
- );
+ // Must not throw; should gracefully converge to a terminal request_end once the
+ // ladder is spent — no abort event (abort marks a user interruption).
+ const { all, cutoff } = await collectRunWithReturn(engine, [userText("go")], allowAll);
+ expect(calls).toBe(3); // initial attempt + maxReconnects(2).
+ expect(cutoff).toEqual({
+ kind: "llm_failure",
+ errorMessage: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
+ });
+ expect(all.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(false);
+ const ends = all.filter((m) => (m.payload as { type?: string }).type === "request_end");
+ expect(ends).toHaveLength(3);
+ // The last failure is the terminal record: no retry planned, the detail and the
+ // authoritative attempt ordinal ride on it for frontends and observability.
+ const last = ends[2]!.payload as {
+ status?: string;
+ attempt?: number;
+ retry_in_ms?: number;
+ error_message?: string;
+ };
+ expect(last).toMatchObject({
+ status: "retryable",
+ attempt: 3,
+ error_message: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
+ });
+ expect(last.retry_in_ms).toBeUndefined();
+ // The mid-ladder failures DID announce their planned retry.
+ expect((ends[0]!.payload as { retry_in_ms?: number }).retry_in_ms).not.toBeUndefined();
// The spent turn's input is stashed as carry-over; the next run (attempt index 3, after
// this run's three) resends it merged with the new input.
@@ -1874,18 +1918,18 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(text).toContain("next");
});
- it("a failed request that succeeds on retry never reaches the user as an error", async () => {
- // The point of retrying `failed`: the classifier is an allowlist, so a transient gateway
- // fault phrased its own way ("Upstream HTTP/2 stream failed") lands here. It used to kill
- // the turn; now the turn simply completes.
+ it("a retryable request that succeeds on retry never reaches the user as an error", async () => {
+ // The point of the wide retryable class: the fatal detector is an allowlist, so a
+ // transient gateway fault phrased its own way ("Upstream HTTP/2 stream failed") stays
+ // retryable, and the turn simply completes.
let calls = 0;
const llm: LLMInterface = {
async *streamGenerate() {
calls += 1;
if (calls === 1) {
return {
- status: "failed" as const,
- message: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
+ status: "retryable" as const,
+ errorMessage: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
};
}
yield assistantText("recovered");
@@ -1903,11 +1947,11 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
const all = await collectRun(engine, [userText("go")], allowAll);
expect(calls).toBe(2);
expect(all.find((m) => (m.payload as { type?: string }).type === "abort")).toBeUndefined();
- // The failure is still classified `failed` on the wire — the retry is a policy decision,
- // not a relabelling, so observability still sees a real failure rather than a "timeout".
+ // The failure keeps its honest `retryable` status on the wire, so observability still
+ // sees a real failure behind the recovered turn.
const ends = all.filter((m) => (m.payload as { type?: string }).type === "request_end");
expect(ends.map((m) => (m.payload as { status?: string }).status)).toEqual([
- "failed",
+ "retryable",
"completed",
]);
// ...and it announces its retry wait like any other retryable failure, so the frontend
@@ -1915,17 +1959,17 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect((ends[0]!.payload as { retry_in_ms?: number }).retry_in_ms).toBeGreaterThanOrEqual(0);
});
- it("auth is the only LLM status that stops the run: no retry, request_end carries status auth", async () => {
+ it("fatal stops the run: no retry, request_end carries status fatal", async () => {
let calls = 0;
const llm: LLMInterface = {
- // GenerativeModel classifies a 401/invalid_api_key as status "auth" (see
- // llm.test.ts); the engine must stop directly — the auth branch is the second belt
- // keeping a dead credential out of the retry loop (the classifier is the first).
- // Every other failure, `failed` included, takes the ladder instead.
+ // GenerativeModel classifies a 401/invalid_api_key as fatal (see llm.test.ts); the
+ // engine must stop directly — the fatal branch is the second belt keeping a dead
+ // credential out of the retry loop (the classifier is the first). Every retryable
+ // failure takes the ladder instead.
// eslint-disable-next-line require-yield
async *streamGenerate() {
calls += 1;
- return { status: "auth", errorMessage: "401 invalid x-api-key" };
+ return { status: "fatal", errorMessage: "401 invalid x-api-key" };
},
};
const environment = new Environment({
@@ -1934,34 +1978,35 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
});
const engine = new ContextEngine({ llm, environment, reconnectBackoffMs: 1 });
- const all = await collectRun(engine, [userText("go")], allowAll);
+ const { all, cutoff } = await collectRunWithReturn(engine, [userText("go")], allowAll);
expect(calls).toBe(1); // A rejected credential cannot be retried into working.
+ // The run generator's return value states the cutoff directly (consumers like the
+ // goal loop and the subagent round report read it instead of the stream).
+ expect(cutoff).toEqual({ kind: "llm_failure", errorMessage: "401 invalid x-api-key" });
// The request's own terminal status is the host signal (streams to the web).
const end = all.find((m) => (m.payload as { type?: string }).type === "request_end");
- expect((end!.payload as { status?: string }).status).toBe("auth");
+ expect((end!.payload as { status?: string }).status).toBe("fatal");
expect((end!.payload as { error_message?: string }).error_message).toBe(
"401 invalid x-api-key",
);
- // No planned retry is announced for a terminal failure.
+ // No planned retry is announced for a terminal failure, and no abort event follows —
+ // abort marks a user interruption; this request_end is the run's terminal record.
expect((end!.payload as { retry_in_ms?: number }).retry_in_ms).toBeUndefined();
- const abort = all.find((m) => (m.payload as { type?: string }).type === "abort");
- expect(abort).toBeDefined();
- expect((abort!.payload as { reason?: string }).reason).toContain("llm request error");
+ expect(all.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(false);
});
- it("a permanent failed (fast-mode rejection) stops the run without burning the ladder", async () => {
+ it("a fast-mode rejection (fatal) stops the run without burning the ladder", async () => {
let calls = 0;
const llm: LLMInterface = {
- // GenerativeModel marks AgentHub's fast_mode UnsupportedParameterError as a permanent
- // failed (see llm.test.ts): thrown before any network I/O, deterministic for the
- // session's frozen config, so retrying the identical request can never succeed. The
- // engine must abort with the actionable message instead of costing the ladder.
+ // GenerativeModel classifies AgentHub's fast_mode UnsupportedParameterError as fatal
+ // (see llm.test.ts): thrown before any network I/O, deterministic for the session's
+ // frozen config, so retrying the identical request can never succeed. The engine
+ // must abort with the actionable message instead of costing the ladder.
// eslint-disable-next-line require-yield
async *streamGenerate() {
calls += 1;
return {
- status: "failed" as const,
- permanent: true,
+ status: "fatal" as const,
errorMessage:
"Kimi does not support fast mode. Fast mode is enabled for this model; " +
"turn it off in the model settings to use this model.",
@@ -1976,29 +2021,27 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
const all = await collectRun(engine, [userText("go")], allowAll);
expect(calls).toBe(1); // Deterministic client-side rejection: one attempt, no reconnects.
- // The classification stays an honest `failed` on the wire (not auth: hosts must not
- // gate input for a credential fix — the fix is the model's fast-mode setting).
const end = all.find((m) => (m.payload as { type?: string }).type === "request_end");
- expect((end!.payload as { status?: string }).status).toBe("failed");
+ expect((end!.payload as { status?: string }).status).toBe("fatal");
// No planned retry is announced: the frontend must not render a countdown for a
- // retry that never comes.
+ // retry that never comes. No abort event either — the request_end is the terminal
+ // record, its error_message carrying the actionable guidance.
expect((end!.payload as { retry_in_ms?: number }).retry_in_ms).toBeUndefined();
- const abort = all.find((m) => (m.payload as { type?: string }).type === "abort");
- expect(abort).toBeDefined();
- const reason = (abort!.payload as { reason?: string }).reason ?? "";
- expect(reason).toContain("llm request error");
- expect(reason).toContain("turn it off in the model settings");
+ expect((end!.payload as { error_message?: string }).error_message).toContain(
+ "turn it off in the model settings",
+ );
+ expect(all.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(false);
});
- it("a plain failed outcome (no permanent flag) still takes the ladder", async () => {
- // The permanent flag is opt-in and narrowly scoped: an ordinary failed without it keeps
- // the retry-everything policy (the classifier is an allowlist; a gateway phrasing a
+ it("a retryable outcome takes the ladder", async () => {
+ // The fatal class is opt-in and narrowly scoped: an ordinary retryable failure keeps
+ // the retry policy (the fatal detector is an allowlist; a gateway phrasing a
// transient fault its own way must still reconnect).
let calls = 0;
const llm: LLMInterface = {
async *streamGenerate() {
calls += 1;
- if (calls === 1) return { status: "failed" as const, errorMessage: "flaky gateway" };
+ if (calls === 1) return { status: "retryable" as const, errorMessage: "flaky gateway" };
yield assistantText("recovered");
return { status: "completed" as const };
},
@@ -2016,15 +2059,14 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(all.find((m) => (m.payload as { type?: string }).type === "abort")).toBeUndefined();
});
- it("a bare 403 (classified failed) retries within the default cap and succeeds", async () => {
+ it("a retryable failure retries within the default cap and succeeds", async () => {
let calls = 0;
const llm: LLMInterface = {
async *streamGenerate() {
calls += 1;
- // Two provider 403 rejections (GenerativeModel labels them failed — there is no
- // quota/message heuristic anymore), then success: every non-auth failure rides
- // the ladder, and attempt 3 is within the default cap of 5.
- if (calls <= 2) return { status: "failed", errorMessage: "403 forbidden" };
+ // Two transient 503s (GenerativeModel classifies 5xx retryable), then success:
+ // attempt 3 is within the default cap of 5.
+ if (calls <= 2) return { status: "retryable", errorMessage: "503 service unavailable" };
yield assistantText("recovered");
yield tokenUsage(emptyTokenCounts(), {
cache_read: 0,
@@ -2052,13 +2094,13 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(all.map((m) => (m.payload as { type?: string }).type)).not.toContain("abort");
});
- it("default reconnect cap is 5: the exhaustion abort message says so", async () => {
+ it("default reconnect cap is 5: the terminal request_end carries the final attempt ordinal", async () => {
let calls = 0;
const llm: LLMInterface = {
// eslint-disable-next-line require-yield
async *streamGenerate() {
calls += 1;
- return { status: "timeout" }; // Always needs a reconnect.
+ return { status: "retryable" }; // Always needs a reconnect.
},
};
const environment = new Environment({
@@ -2071,8 +2113,13 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
const all = await collectRun(engine, [userText("go")], allowAll);
expect(calls).toBe(6); // Initial attempt + 5 retries.
- const abort = all.find((m) => (m.payload as { type?: string }).type === "abort");
- expect((abort!.payload as { reason?: string }).reason).toBe("reconnect failed after 5 retries");
+ expect(all.some((m) => (m.payload as { type?: string }).type === "abort")).toBe(false);
+ const ends = all.filter((m) => (m.payload as { type?: string }).type === "request_end");
+ expect(ends).toHaveLength(6);
+ const last = ends[5]!.payload as { status?: string; attempt?: number; retry_in_ms?: number };
+ expect(last.status).toBe("retryable");
+ expect(last.attempt).toBe(6);
+ expect(last.retry_in_ms).toBeUndefined();
});
it("skipReconnectWait wakes the backoff early ('retry now'): attempt numbering unchanged; no-op without a wait", async () => {
@@ -2080,7 +2127,7 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
const llm: LLMInterface = {
async *streamGenerate() {
calls += 1;
- if (calls === 1) return { status: "timeout" };
+ if (calls === 1) return { status: "retryable" };
yield assistantText("ok");
yield tokenUsage(emptyTokenCounts(), {
cache_read: 0,
@@ -2119,7 +2166,7 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
expect(calls).toBe(2);
const ends = all.filter((m) => (m.payload as { type?: string }).type === "request_end");
expect(ends.map((m) => (m.payload as { status?: string }).status)).toEqual([
- "timeout",
+ "retryable",
"completed",
]);
expect(all.map((m) => (m.payload as { type?: string }).type)).not.toContain("abort");
@@ -2147,11 +2194,11 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
async *streamGenerate() {
calls += 1;
if (calls === 1) {
- // A retryable provider rejection: the detail must reach observability via the
+ // A retryable failure with detail: the detail must reach observability via the
// event — a retried request never produces an abort to carry it.
return {
- status: "timeout",
- errorMessage: "403 quota exceeded (insufficient_user_quota)",
+ status: "retryable",
+ errorMessage: "429 rate limited (please slow down)",
};
}
yield assistantText("ok");
@@ -2175,8 +2222,8 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
payload: { status?: string; error_message?: string; attempt?: number; retry_in_ms?: number };
}[];
expect(ends).toHaveLength(2);
- expect(ends[0]!.payload.status).toBe("timeout");
- expect(ends[0]!.payload.error_message).toBe("403 quota exceeded (insufficient_user_quota)");
+ expect(ends[0]!.payload.status).toBe("retryable");
+ expect(ends[0]!.payload.error_message).toBe("429 rate limited (please slow down)");
// The engine will retry: the planned backoff is announced (base 0 here -> 0ms), and the
// authoritative attempt ordinal is stamped (this was the run's 1st request).
expect(ends[0]!.payload.retry_in_ms).toBe(0);
@@ -2195,7 +2242,7 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
// eslint-disable-next-line require-yield
async *streamGenerate() {
calls += 1;
- return { status: "timeout" }; // Always needs a reconnect.
+ return { status: "retryable" }; // Always needs a reconnect.
},
};
const environment = new Environment({
@@ -2230,14 +2277,14 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
inputs.push(params.newMessages);
if (calls === 1) {
// Real tool_call -> the engine dispatches it for execution (appends to a file, a
- // side effect), followed by a timeout/network drop (timeout).
+ // side effect), followed by a timeout/network drop (retryable).
yield toolCall({
name: "exec_command",
arguments: JSON.stringify({ cmd: "printf x >> count.txt" }),
toolCallId: "t1",
stopReason: "completed",
});
- return { status: "timeout" };
+ return { status: "retryable" };
}
yield assistantText("second");
yield tokenUsage(emptyTokenCounts(), {
@@ -2292,13 +2339,13 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
inputs.push(params.newMessages);
if (calls === 1) {
// Before the terminal error, partial thinking and text were already produced (the
- // LLM finishes them as complete messages). `auth` is the trigger because it is the
- // one LLM status that still exits straight to the flatten path — `failed` now takes
- // the reconnect ladder, whose carry-over is [turn_retried] instead (covered by the
- // exhausted-retries test below).
- yield thinkingMessage("half-thought", "failed");
- yield assistantText("half-text", "failed");
- return { status: "auth", errorMessage: "boom" };
+ // LLM finishes them as complete messages). `fatal` is the trigger because it
+ // exits straight to the flatten path — `retryable` takes the reconnect ladder,
+ // whose carry-over is [turn_retried] instead (covered by the exhausted-retries
+ // test below).
+ yield thinkingMessage("half-thought", "fatal");
+ yield assistantText("half-text", "fatal");
+ return { status: "fatal", errorMessage: "boom" };
}
yield assistantText("ok");
yield tokenUsage(emptyTokenCounts(), {
@@ -2345,13 +2392,13 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
toolCallId: "t1",
stopReason: "completed",
});
- return { status: "timeout" };
+ return { status: "retryable" };
}
if (calls === 2) {
// Attempt 2 (retry, original input resent): produces partial thinking then times out
// again -> retries exhausted.
- yield thinkingMessage("retry-thought", "timeout");
- return { status: "timeout" };
+ yield thinkingMessage("retry-thought", "retryable");
+ return { status: "retryable" };
}
yield assistantText("ok");
yield tokenUsage(emptyTokenCounts(), {
@@ -2419,8 +2466,8 @@ describe("ContextEngine LLM timeout / network interruption (PRN-012)", () => {
calls += 1;
inputs.push(params.newMessages);
if (calls === 1) {
- yield thinkingMessage("half-1", "timeout");
- return { status: "timeout" };
+ yield thinkingMessage("half-1", "retryable");
+ return { status: "retryable" };
}
if (calls === 2) {
// Interrupted by the user while the retry is in progress.
diff --git a/packages/core/test/environment.test.ts b/packages/core/test/environment.test.ts
index 6e61a989..e916eea0 100644
--- a/packages/core/test/environment.test.ts
+++ b/packages/core/test/environment.test.ts
@@ -814,7 +814,7 @@ describe("Environment.executeTool — relaxed tool contract", () => {
output: "result",
toolCallId: ctx.toolCallId,
});
- return { stopReason: "failed" as const };
+ return { stopReason: "fatal" as const };
},
});
try {
@@ -839,7 +839,7 @@ describe("Environment.executeTool — relaxed tool contract", () => {
};
expect(complete.type).toBe("tool_call_output");
expect(complete.output).toBe("partial result");
- expect(complete.stop_reason).toBe("failed"); // Finish reason reported via the return value
+ expect(complete.stop_reason).toBe("fatal"); // Finish reason reported via the return value
expect((out[out.length - 2]!.payload as { event_type?: string }).event_type).toBe("stop");
} finally {
delete BUILTIN_TOOL_FACTORIES[NAME];
@@ -913,7 +913,7 @@ describe("Environment.executeTool — relaxed tool contract", () => {
output: "broken",
toolCallId: ctx.toolCallId,
});
- return { stopReason: "failed" as const, images: ["data:image/png;base64,AAAA"] };
+ return { stopReason: "fatal" as const, images: ["data:image/png;base64,AAAA"] };
},
});
try {
@@ -932,7 +932,7 @@ describe("Environment.executeTool — relaxed tool contract", () => {
const complete = out[out.length - 1]!.payload as { stop_reason?: string; images?: string[] };
// Only a normal completion carries images: a failed finish drops them (neither the
// stream nor the complete message carries them), keeping the finish handling simple.
- expect(complete.stop_reason).toBe("failed");
+ expect(complete.stop_reason).toBe("fatal");
expect(complete.images).toBeUndefined();
for (const m of out) {
const p = m.payload as { type?: string };
@@ -1030,7 +1030,7 @@ describe("Environment.executeTool — timeoutMs (PRN-013)", () => {
stop_reason?: string;
};
expect(last.type).toBe("tool_call_output");
- expect(last.stop_reason).toBe("failed");
+ expect(last.stop_reason).toBe("fatal");
// Kept-prior-output is asserted only where the shell can win the race: a Git-Bash login
// shell on Windows needs several hundred ms to start, so nothing is printed before a
// sub-250ms timeout there — the timeout mechanics above are still fully exercised.
@@ -1117,7 +1117,7 @@ describe("Environment.executeTool — robustness", () => {
expect(payload.type).toBe("tool_call_output");
expect(payload.output).toContain("Unknown tool: not_a_real_tool"); // Complete content matches
expect(payload.tool_call_id).toBe("call_unknown");
- expect(payload.stop_reason).toBe("failed");
+ expect(payload.stop_reason).toBe("fatal");
});
/** Runs one tool call and returns the payload of the last complete tool_call_output. */
@@ -1144,14 +1144,14 @@ describe("Environment.executeTool — robustness", () => {
const payload = await runTool(args, `call_badjson_${i}`);
expect(payload.type).toBe("tool_call_output");
expect(payload.output, args).toContain("not valid JSON");
- expect(payload.stop_reason).toBe("failed");
+ expect(payload.stop_reason).toBe("fatal");
}
});
it("tells the model the arguments were empty", async () => {
const payload = await runTool("", "call_emptyargs");
expect(payload.output).toContain("arguments field is empty");
- expect(payload.stop_reason).toBe("failed");
+ expect(payload.stop_reason).toBe("fatal");
});
it("never returns an empty tool output", async () => {
@@ -1207,7 +1207,7 @@ describe("Environment.executeTool — robustness", () => {
stop_reason?: string;
};
expect(payload.output).toContain("[exit code: 3]");
- expect(payload.stop_reason).toBe("failed");
+ expect(payload.stop_reason).toBe("fatal");
// The exit-code marker is also produced via streaming (renderable by the frontend); the
// streamed deltas concatenated == the complete content (short output here, no truncation).
const streamed = messages
@@ -1319,9 +1319,9 @@ describe("Environment structure invariant on tool throw (PRN-012)", () => {
const noteDelta = out[2]!.payload as { output?: string };
expect(noteDelta.output).toContain("kaboom"); // The error marker is produced via a streamed delta
const stop = out[3]!.payload as { stop_reason?: string };
- expect(stop.stop_reason).toBe("failed");
+ expect(stop.stop_reason).toBe("fatal");
const complete = out[4]!.payload as { stop_reason?: string; output?: string };
- expect(complete.stop_reason).toBe("failed");
+ expect(complete.stop_reason).toBe("fatal");
expect(complete.output).toContain("kaboom");
expect(complete.output).toContain("working"); // Keeps the partial content already streamed.
// Concatenating the streamed deltas == the complete content (relevant for frontend rendering).
diff --git a/packages/core/test/exec-session.test.ts b/packages/core/test/exec-session.test.ts
index 47f009e8..bd74b16f 100644
--- a/packages/core/test/exec-session.test.ts
+++ b/packages/core/test/exec-session.test.ts
@@ -193,7 +193,7 @@ describe("exec_command — long-running command sessions", () => {
const elapsed = Date.now() - startedAt;
expect(elapsed).toBeLessThan(3000); // Did not wait for the full sleep 30
expect(res.output).not.toContain("still running");
- expect(res.stopReason).toBe("failed"); // Interrupted by signal -> non-zero exit
+ expect(res.stopReason).toBe("fatal"); // Interrupted by signal -> non-zero exit
});
it("input_command rejects chars mixing U+0003 with other content", async () => {
@@ -209,7 +209,7 @@ describe("exec_command — long-running command sessions", () => {
yield_time_ms: 2000,
});
expect(res.output).toContain('send "\\u0003" alone');
- expect(res.stopReason).toBe("failed");
+ expect(res.stopReason).toBe("fatal");
// The session was not mistakenly killed: still running.
const poll = await runTool(env, "input_command", { process_id: pid, yield_time_ms: 300 });
@@ -219,7 +219,7 @@ describe("exec_command — long-running command sessions", () => {
it("input_command reports an unknown process_id without throwing", async () => {
const res = await runTool(env, "input_command", { process_id: "proc-deadbeef" });
expect(res.output).toContain("unknown process_id proc-deadbeef");
- expect(res.stopReason).toBe("failed");
+ expect(res.stopReason).toBe("fatal");
});
it("input_command ignores writes to a closed stdin pipe without crashing", async () => {
@@ -264,7 +264,7 @@ describe("exec_command — long-running command sessions", () => {
});
expect(res.output).toContain("command session manager disposed");
expect(res.output).not.toContain("should-not-run");
- expect(res.stopReason).toBe("failed");
+ expect(res.stopReason).toBe("fatal");
});
it("delivers output arriving while the consumer is suspended without waiting out the window", async () => {
diff --git a/packages/core/test/file-tools.test.ts b/packages/core/test/file-tools.test.ts
index 8e2ee1a4..bc02d621 100644
--- a/packages/core/test/file-tools.test.ts
+++ b/packages/core/test/file-tools.test.ts
@@ -126,7 +126,7 @@ describe("read_file", () => {
it("fails with a path hint when the file does not exist", async () => {
const { result, text } = await run(tool(), { file_path: "missing.txt" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("File not found");
expect(text).toContain("missing.txt");
expect(text).toContain("workspace");
@@ -142,7 +142,7 @@ describe("read_file", () => {
await writeFile(path.join(tmp, "agent_state", "AGENTS.md"), "persona\n");
await writeFile(path.join(tmp, "notes.txt"), "n\n");
const { result, text } = await run(tool(), { file_path: path.join(tmp, "AGENTS.md") }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain(
`The directory "${modelVisiblePath(tmp)}" exists but has no entry "AGENTS.md".`,
);
@@ -153,7 +153,7 @@ describe("read_file", () => {
it("reports the first missing segment of a deeper path", async () => {
const missing = path.join(tmp, "nope", "deep", "file.txt");
const { result, text } = await run(tool(), { file_path: missing }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain(
`The directory "${modelVisiblePath(tmp)}" exists but has no entry "nope".`,
);
@@ -163,7 +163,7 @@ describe("read_file", () => {
await writeFile(path.join(tmp, "a.txt"), "x\n");
const inner = path.join(tmp, "a.txt", "inner.txt");
const { result, text } = await run(tool(), { file_path: inner }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("File not found");
expect(text).toContain(
`Note: "${modelVisiblePath(path.join(tmp, "a.txt"))}" exists but is a file, not a directory.`,
@@ -180,14 +180,14 @@ describe("read_file", () => {
it("fails when the path is a directory", async () => {
await mkdir(path.join(tmp, "subdir"));
const { result, text } = await run(tool(), { file_path: "subdir" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("directory");
});
it("fails on binary content (NUL bytes) and advises other tools", async () => {
await writeFile(path.join(tmp, "bin.dat"), Buffer.from([0x89, 0x50, 0x00, 0x0a, 0x42]));
const { result, text } = await run(tool(), { file_path: "bin.dat" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("binary");
expect(text).toContain("read_image");
});
@@ -195,14 +195,14 @@ describe("read_file", () => {
it("fails when offset is past the end of the file", async () => {
await writeFile(path.join(tmp, "two.txt"), "a\nb\n");
const { result, text } = await run(tool(), { file_path: "two.txt", offset: 5 }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("past the end");
expect(text).toContain("2 lines");
});
it("fails when file_path is missing", async () => {
const { result, text } = await run(tool(), {}, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain('"file_path"');
});
});
@@ -259,7 +259,7 @@ describe("edit_file", () => {
{ file_path: "f.txt", old_string: "goodbye", new_string: "farewell" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain('old_string not found in "f.txt"');
expect(await readFile(path.join(tmp, "f.txt"), "utf8")).toBe("hello\n"); // Untouched
});
@@ -271,7 +271,7 @@ describe("edit_file", () => {
{ file_path: "dup.txt", old_string: "x", new_string: "y" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("3 times");
expect(text).toContain("replace_all");
expect(await readFile(path.join(tmp, "dup.txt"), "utf8")).toBe("x\nx\nx\n");
@@ -284,7 +284,7 @@ describe("edit_file", () => {
{ file_path: "same.txt", old_string: "abc", new_string: "abc" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("identical");
});
@@ -294,7 +294,7 @@ describe("edit_file", () => {
{ file_path: "nope.txt", old_string: "a", new_string: "b" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("File not found");
expect(text).toContain("write_file");
expect(text).toContain(
@@ -309,19 +309,19 @@ describe("edit_file", () => {
{ file_path: "d", old_string: "a", new_string: "b" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("directory");
});
it("fails when required arguments are missing", async () => {
const missingPath = await run(tool(), { old_string: "a", new_string: "b" }, tmp);
- expect(missingPath.result?.stopReason).toBe("failed");
+ expect(missingPath.result?.stopReason).toBe("fatal");
expect(missingPath.text).toContain('"file_path"');
const missingOld = await run(tool(), { file_path: "f", new_string: "b" }, tmp);
- expect(missingOld.result?.stopReason).toBe("failed");
+ expect(missingOld.result?.stopReason).toBe("fatal");
expect(missingOld.text).toContain('"old_string"');
const missingNew = await run(tool(), { file_path: "f", old_string: "a" }, tmp);
- expect(missingNew.result?.stopReason).toBe("failed");
+ expect(missingNew.result?.stopReason).toBe("fatal");
expect(missingNew.text).toContain('"new_string"');
});
});
@@ -376,7 +376,7 @@ describe("write_file", () => {
it("fails when the path is a directory", async () => {
await mkdir(path.join(tmp, "adir"));
const { result, text } = await run(tool(), { file_path: "adir", content: "x" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("directory");
});
@@ -387,13 +387,13 @@ describe("write_file", () => {
{ file_path: "plain.txt/child.txt", content: "y" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("Failed to write");
});
it("fails when content is missing (but not when it is empty)", async () => {
const { result, text } = await run(tool(), { file_path: "nocontent.txt" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain('"content"');
});
});
@@ -417,10 +417,10 @@ describe("read_file — argument coercion, CRLF, secret guard", () => {
it("fails on non-numeric offset/limit instead of silently defaulting", async () => {
await writeFile(path.join(tmp, "a.txt"), "x\n");
const badOffset = await run(tool(), { file_path: "a.txt", offset: "abc" }, tmp);
- expect(badOffset.result?.stopReason).toBe("failed");
+ expect(badOffset.result?.stopReason).toBe("fatal");
expect(badOffset.text).toContain('Invalid "offset"');
const badLimit = await run(tool(), { file_path: "a.txt", limit: 0 }, tmp);
- expect(badLimit.result?.stopReason).toBe("failed");
+ expect(badLimit.result?.stopReason).toBe("fatal");
expect(badLimit.text).toContain('Invalid "limit"');
});
@@ -438,10 +438,10 @@ describe("read_file — argument coercion, CRLF, secret guard", () => {
await mkdir(path.join(tmp, "sub"));
await writeFile(path.join(tmp, "sub", ".project_config.toml"), "key=1\n");
const vault = await run(tool(), { file_path: ".vault.toml" }, tmp);
- expect(vault.result?.stopReason).toBe("failed");
+ expect(vault.result?.stopReason).toBe("fatal");
expect(vault.text).toContain("Refusing to read");
const cfg = await run(tool(), { file_path: "sub/.project_config.toml" }, tmp);
- expect(cfg.result?.stopReason).toBe("failed");
+ expect(cfg.result?.stopReason).toBe("fatal");
expect(cfg.text).toContain("Refusing to read");
});
@@ -450,12 +450,12 @@ describe("read_file — argument coercion, CRLF, secret guard", () => {
// Case variant: a case-insensitive filesystem (macOS/Windows) opens ".VAULT.TOML" as
// the store itself; on a case-sensitive one refusing the name is a harmless false positive.
const upper = await run(tool(), { file_path: ".VAULT.TOML" }, tmp);
- expect(upper.result?.stopReason).toBe("failed");
+ expect(upper.result?.stopReason).toBe("fatal");
expect(upper.text).toContain("Refusing to read");
// Symlink: the link's own basename says nothing about what it dereferences to.
await symlink(path.join(tmp, ".vault.toml"), path.join(tmp, "notes.txt"));
const linked = await run(tool(), { file_path: "notes.txt" }, tmp);
- expect(linked.result?.stopReason).toBe("failed");
+ expect(linked.result?.stopReason).toBe("fatal");
expect(linked.text).toContain("Refusing to read");
expect(linked.text).not.toContain("SECRET=1");
// A symlink NAMED like a store but pointing elsewhere is refused on the lexical name —
@@ -463,7 +463,7 @@ describe("read_file — argument coercion, CRLF, secret guard", () => {
await writeFile(path.join(tmp, "plain.txt"), "hello\n");
await symlink(path.join(tmp, "plain.txt"), path.join(tmp, ".project_config.toml"));
const named = await run(tool(), { file_path: ".project_config.toml" }, tmp);
- expect(named.result?.stopReason).toBe("failed");
+ expect(named.result?.stopReason).toBe("fatal");
// An ordinary symlink to an ordinary file still reads.
await symlink(path.join(tmp, "plain.txt"), path.join(tmp, "alias.txt"));
const ok = await run(tool(), { file_path: "alias.txt" }, tmp);
@@ -496,7 +496,7 @@ describe("read_file — bounded scan and output budget", () => {
await writeFile(path.join(tmp, "huge-line.txt"), "x".repeat(READ_FILE_SCAN_CAP_BYTES + 1024));
const tool = () => createReadFileTool(def(READ_FILE_NAME, "r"));
const { result, text } = await run(tool(), { file_path: "huge-line.txt" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("Stopped after scanning 8 MB");
expect(text).toContain("sed -n");
});
@@ -560,7 +560,7 @@ describe("edit_file — review follow-ups", () => {
{ file_path: "f.txt", old_string: "", new_string: "x" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("old_string must not be empty");
expect(text).toContain("write_file");
});
@@ -572,7 +572,7 @@ describe("edit_file — review follow-ups", () => {
{ file_path: "crlf.txt", old_string: "alpha\nbeta", new_string: "gamma" },
tmp,
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("old_string not found");
expect(text).toContain("CRLF");
});
@@ -808,7 +808,7 @@ describe("edit_file / write_file — symlinked targets", () => {
await symlink("b.txt", path.join(tmp, "a.txt"));
await symlink("a.txt", path.join(tmp, "b.txt"));
const { result, text } = await run(write(), { file_path: "a.txt", content: "x\n" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("Too many levels of symbolic links");
});
diff --git a/packages/core/test/goal.test.ts b/packages/core/test/goal.test.ts
index 7e2f3f18..83ebf10c 100644
--- a/packages/core/test/goal.test.ts
+++ b/packages/core/test/goal.test.ts
@@ -9,6 +9,7 @@ import {
abortEvent,
assistantText,
buildSkillsMessage,
+ compactionEnd,
downgradeGoalInput,
emptyTokenCounts,
goalFilePath,
@@ -17,13 +18,16 @@ import {
isGoalRoundInput,
modelVisiblePath,
parseGoalMessage,
+ requestEnd,
sessionScratchpadDir,
stripConversationMarkers,
tokenUsage,
+ toolCallOutput,
userText,
withOrigin,
} from "../src/index.js";
import type {
+ RunCutoff,
EnvironmentInterface,
GoalOutcome,
LLMInterface,
@@ -75,7 +79,7 @@ function roundArgs(objective: string, over: Partial<GoalPromptArgs> = {}): GoalP
* optional side effect (standing in for the model editing GOAL.yaml with shell tools).
*/
function fakeSession(
- rounds: Array<{ messages?: OmniMessage[]; then?: () => Promise<void> }>,
+ rounds: Array<{ messages?: OmniMessage[]; then?: () => Promise<void>; cutoff?: RunCutoff }>,
): GoalRoundRunner & { prompts: string[] } {
let i = 0;
const prompts: string[] = [];
@@ -88,6 +92,8 @@ function fakeSession(
prompts.push(p.text ?? "");
for (const msg of round.messages ?? []) yield msg;
await round.then?.();
+ // Session.run's contract: the return value says whether the round was cut off.
+ return round.cutoff ?? null;
},
};
}
@@ -265,10 +271,10 @@ describe("runGoalLoop", () => {
});
it("treats a round the engine cut off (failed final assistant text) as terminal", async () => {
- // The max_turns cutoff: a final assistant notice with stop_reason "failed", no abort
+ // The max_turns cutoff: a final assistant notice with stop_reason "fatal", no abort
// event, and the model never reached the goal file — re-firing would loop forever.
const session = fakeSession([
- { messages: [assistantText("[reached max turns (100); stopping]", "failed")] },
+ { messages: [assistantText("[reached max turns (100); stopping]", "fatal")] },
]);
const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
expect(outcome).toEqual({ outcome: "aborted", rounds: 1, tokensUsed: 0 });
@@ -279,7 +285,7 @@ describe("runGoalLoop", () => {
it("a mid-round failed notice followed by normal text does not end the goal", async () => {
const session = fakeSession([
{
- messages: [assistantText("tool hiccup", "failed"), assistantText("recovered, done")],
+ messages: [assistantText("tool hiccup", "fatal"), assistantText("recovered, done")],
then: () => setStatus("complete"),
},
]);
@@ -371,16 +377,83 @@ describe("runGoalLoop", () => {
it("stops without re-firing when the main session aborts, leaving the goal active", async () => {
const session = fakeSession([
- { messages: [tokenUsage(usage(80), usage(80)), abortEvent("interrupted")] },
+ {
+ messages: [tokenUsage(usage(80), usage(80)), abortEvent()],
+ cutoff: { kind: "abort", errorCode: "user_abort" },
+ },
+ ]);
+ const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
+ expect(outcome).toEqual({ outcome: "aborted", rounds: 1, tokensUsed: 80 });
+ expect(await readGoalStatus(file)).toBe("active");
+ });
+
+ it("stops without re-firing when the run returns an LLM-failure cutoff (no abort event)", async () => {
+ // A fatal end (and equally a retryable one with no retry planned) cuts the round;
+ // re-firing would loop the same failure forever. The run generator's return value
+ // carries the fact — the loop never re-derives it from the stream.
+ const session = fakeSession([
+ {
+ messages: [tokenUsage(usage(80), usage(80)), requestEnd("fatal", { errorMessage: "401" })],
+ cutoff: { kind: "llm_failure", errorCode: "auth", errorMessage: "401" },
+ },
]);
const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
expect(outcome).toEqual({ outcome: "aborted", rounds: 1, tokensUsed: 80 });
expect(await readGoalStatus(file)).toBe("active");
});
+ it("a retryable request_end that announces its retry does NOT stop the loop", async () => {
+ // Mid-ladder failures carry retry_in_ms; only the give-up (no retry planned) is terminal.
+ const session = fakeSession([
+ {
+ messages: [
+ requestEnd("retryable", { attempt: 1, retryInMs: 2000 }),
+ tokenUsage(usage(90), usage(90)),
+ ],
+ then: () => setStatus("complete"),
+ },
+ ]);
+ const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
+ expect(outcome).toEqual({ outcome: "complete", rounds: 1, tokensUsed: 90 });
+ });
+
+ it("stops without re-firing when a mid-task compaction is given up (no abort event)", async () => {
+ // Tool outputs owed the model a follow-up turn; the compaction failure ended the run
+ // before that turn — the round is cut, not finished.
+ const session = fakeSession([
+ {
+ messages: [
+ tokenUsage(usage(70), usage(70)),
+ toolCallOutput({ output: "ok", toolCallId: "t1" }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
+ ],
+ cutoff: { kind: "compaction_failure" },
+ },
+ ]);
+ const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
+ expect(outcome).toEqual({ outcome: "aborted", rounds: 1, tokensUsed: 70 });
+ expect(await readGoalStatus(file)).toBe("active");
+ });
+
+ it("a Task-boundary compaction failure is advisory: the loop keeps going", async () => {
+ // No tool outputs pending a follow-up turn when the compaction failed — the model
+ // finished its round; the standing trigger retries the compaction next opportunity.
+ const session = fakeSession([
+ {
+ messages: [
+ tokenUsage(usage(60), usage(60)),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
+ ],
+ then: () => setStatus("complete"),
+ },
+ ]);
+ const { outcome } = await drain(runGoalLoop(session, { text: "o", goalFilePath: file }));
+ expect(outcome).toEqual({ outcome: "complete", rounds: 1, tokensUsed: 60 });
+ });
+
it("counts uncached input + output, including subagent (origin-marked) usage", async () => {
const childUsage = withOrigin(tokenUsage(usage(500, 200), usage(500, 200)), "child-session");
- const childAbort = withOrigin(abortEvent("child failed"), "child-session");
+ const childAbort = withOrigin(abortEvent(), "child-session");
const session = fakeSession([
{
// Main request: total 1000 with 400 cached → 600; child: total 500 with 200 cached → 300.
diff --git a/packages/core/test/llm.test.ts b/packages/core/test/llm.test.ts
index e06d90c7..d1b0bfe5 100644
--- a/packages/core/test/llm.test.ts
+++ b/packages/core/test/llm.test.ts
@@ -29,7 +29,7 @@ import {
isIncompleteStreamError,
isFastModeUnsupportedError,
isMalformedJsonParseError,
- isRetryableError,
+ isFatalProviderRejection,
FAST_MODE_UNSUPPORTED_GUIDANCE,
mapThinkingLevel,
mergeOmniToUniMessage,
@@ -964,7 +964,7 @@ describe("EventTranslator.finishInterrupted (PRN-012 structural closure)", () =>
]) {
for (const m of tr.pushEvent(e)) out.push(m);
}
- for (const m of tr.finishInterrupted("timeout")) out.push(m);
+ for (const m of tr.finishInterrupted("retryable")) out.push(m);
const types = out.map((m) => (m.payload as { type: string }).type);
expect(types).toEqual([
@@ -978,10 +978,10 @@ describe("EventTranslator.finishInterrupted (PRN-012 structural closure)", () =>
const stop = out[3]!.payload as { event_type: string; stop_reason: string };
expect(stop.event_type).toBe("stop");
- expect(stop.stop_reason).toBe("timeout");
+ expect(stop.stop_reason).toBe("retryable");
const complete = out[4]!.payload as TextPayload;
expect(complete.text).toBe("Partial");
- expect(complete.stop_reason).toBe("timeout");
+ expect(complete.stop_reason).toBe("retryable");
});
it("closes an open thinking segment with the interruption reason on both partial stop and complete message", () => {
@@ -1077,7 +1077,7 @@ describe("EventTranslator.finishInterrupted (PRN-012 structural closure)", () =>
for (const m of tr.pushEvent(e)) out.push(m);
}
const before = out.length;
- for (const m of tr.finishInterrupted("timeout")) out.push(m);
+ for (const m of tr.finishInterrupted("retryable")) out.push(m);
expect(out.length).toBe(before); // Already produced immediately, not duplicated.
const complete = out.find((m) => (m.payload as { type: string }).type === "tool_call")!
.payload as ToolCallPayload;
@@ -1163,100 +1163,75 @@ describe("config helpers", () => {
});
});
-describe("isRetryableError (the timeout-vs-failed label; the engine retries every non-auth error)", () => {
- it("labels 429, 408 and 5xx transport-shaped (timeout)", () => {
- expect(isRetryableError({ status: 429 })).toBe(true);
- expect(isRetryableError({ status: 408 })).toBe(true); // Request Timeout (transient)
- expect(isRetryableError({ status: 500 })).toBe(true);
- expect(isRetryableError({ statusCode: 503 })).toBe(true);
- });
-
- it("labels provider 4xx rejections failed — including quota-coded and bare 403s", () => {
- // The label is taxonomy, not policy: `failed` is in the engine's RETRY_STATUSES, so
- // every one of these retries on the ladder and fails after exhaustion. There is no
- // quota detector anymore — a 402/403 with a quota code or subscription message
- // classifies exactly like a bare one.
- expect(isRetryableError({ status: 400 })).toBe(false);
- expect(isRetryableError({ status: 403 })).toBe(false);
- expect(isRetryableError({ status: 404 })).toBe(false);
- expect(isRetryableError({ status: 403, code: "insufficient_user_quota" })).toBe(false);
- expect(isRetryableError({ status: 402, code: "insufficient_quota" })).toBe(false);
- expect(isRetryableError({ status: 403, message: "no active subscription" })).toBe(false);
- // 401 is auth (terminal), not merely failed-shaped.
- expect(isRetryableError({ status: 401 })).toBe(false);
- });
-
- it("labels network error codes transport-shaped", () => {
- expect(isRetryableError({ code: "ECONNRESET" })).toBe(true);
- expect(isRetryableError({ code: "ETIMEDOUT" })).toBe(true);
- expect(isRetryableError(new Error("socket hang up"))).toBe(true);
- expect(isRetryableError(new Error("request timeout"))).toBe(true);
- });
-
- it("labels undici transport failures transport-shaped, probing the cause chain", () => {
- // Node fetch wraps a dropped connection as TypeError("terminated") with the real code on
- // `cause` — exactly the field observed: "terminated: other side closed (UND_ERR_SOCKET)".
+describe("isFatalProviderRejection (the fatal allowlist; everything it misses stays retryable)", () => {
+ it("matches definitive 4xx rejections — including quota-coded and bare 403s", () => {
+ expect(isFatalProviderRejection({ status: 400 })).toBe(true);
+ expect(isFatalProviderRejection({ status: 403 })).toBe(true);
+ expect(isFatalProviderRejection({ status: 404 })).toBe(true);
+ expect(isFatalProviderRejection({ statusCode: 422 })).toBe(true);
+ expect(isFatalProviderRejection({ status: 403, code: "insufficient_user_quota" })).toBe(true);
+ expect(isFatalProviderRejection({ status: 402, code: "insufficient_quota" })).toBe(true);
+ expect(isFatalProviderRejection({ status: 403, message: "no active subscription" })).toBe(true);
+ // 401 matches here too, but the auth detector runs first and gives it the more
+ // specific credentials handling.
+ expect(isFatalProviderRejection({ status: 401 })).toBe(true);
+ });
+
+ it("leaves the transient statuses out: 408, 429 and 5xx keep their retries", () => {
+ expect(isFatalProviderRejection({ status: 408 })).toBe(false);
+ expect(isFatalProviderRejection({ status: 429 })).toBe(false);
+ expect(isFatalProviderRejection({ status: 500 })).toBe(false);
+ expect(isFatalProviderRejection({ statusCode: 503 })).toBe(false);
+ });
+
+ it("finds the status down the cause chain (SDKs wrap the response error)", () => {
expect(
- isRetryableError(
- new TypeError("terminated", {
- cause: Object.assign(new Error("other side closed"), { code: "UND_ERR_SOCKET" }),
+ isFatalProviderRejection(
+ new Error("request failed", {
+ cause: Object.assign(new Error("bad request"), { status: 400 }),
}),
),
).toBe(true);
- expect(isRetryableError({ code: "UND_ERR_SOCKET" })).toBe(true);
- expect(isRetryableError({ code: "UND_ERR_CONNECT_TIMEOUT" })).toBe(true);
- expect(isRetryableError({ code: "UND_ERR_HEADERS_TIMEOUT" })).toBe(true);
- expect(isRetryableError({ code: "UND_ERR_BODY_TIMEOUT" })).toBe(true);
- // A cause-less "fetch failed" still counts via the message keywords; "terminated" only
- // counts alongside transport vocabulary — the real socket case always carries the
- // UND_ERR_* code on its cause, and bare "terminated" appears in unrelated provider
- // copy too (see the content-filter counter-case below).
- expect(isRetryableError(new TypeError("fetch failed"))).toBe(true);
- expect(isRetryableError(new TypeError("terminated: other side closed"))).toBe(true);
- expect(isRetryableError(new TypeError("terminated"))).toBe(false);
- // Provider verdict copy that merely contains the word is failed-shaped, not transport.
- expect(isRetryableError(new Error("Request terminated by content filter"))).toBe(false);
- // Classic Node codes wrapped one level down are found too.
expect(
- isRetryableError(
+ isFatalProviderRejection(
new Error("request failed", {
- cause: Object.assign(new Error("read ECONNRESET"), { code: "ECONNRESET" }),
+ cause: Object.assign(new Error("throttled"), { status: 429 }),
}),
),
- ).toBe(true);
+ ).toBe(false);
});
- it("auth always wins: a definitive credential signal is terminal, never merely a label", () => {
+ it("never matches status-less errors: transport drops and local failures stay retryable", () => {
+ expect(isFatalProviderRejection({ code: "ECONNRESET" })).toBe(false);
+ expect(isFatalProviderRejection(new Error("socket hang up"))).toBe(false);
+ expect(
+ isFatalProviderRejection(
+ new TypeError("terminated", {
+ cause: Object.assign(new Error("other side closed"), { code: "UND_ERR_SOCKET" }),
+ }),
+ ),
+ ).toBe(false);
+ const abort = new Error("aborted");
+ abort.name = "AbortError";
+ expect(isFatalProviderRejection(abort)).toBe(false);
+ expect(isFatalProviderRejection(new Error("unexpected token in JSON"))).toBe(false);
+ expect(isFatalProviderRejection(null)).toBe(false);
+ expect(isFatalProviderRejection(undefined)).toBe(false);
+ });
+
+ it("the auth detector keeps its own contract: a definitive credential signal, wherever it rides", () => {
// A 403 whose BODY carries a definitive auth code but whose MESSAGE mentions a
// subscription (SDKs routinely put the body's message on err.message): the explicit
- // credential signal decides, keeping the dead credential off the retry ladder.
+ // credential signal decides, and the classifier checks auth before the generic 4xx.
const err = Object.assign(new Error("subscription key invalid"), {
status: 403,
error: { code: "invalid_api_key" },
});
expect(isAuthenticationError(err)).toBe(true);
- expect(isRetryableError(err)).toBe(false);
- // Same with the auth signal on the cause chain.
- expect(
- isRetryableError(
- new Error("request failed: quota subscription issue", {
- cause: { status: 403, error: { type: "authentication_error" } },
- }),
- ),
- ).toBe(false);
- // A bare 403 carries no credential signal: NOT auth (it rides the retry ladder as
- // failed; see the engine tests).
+ // A bare 403 carries no credential signal: NOT auth (it is fatal through the generic
+ // 4xx rejection instead; see the classification tests).
expect(isAuthenticationError({ status: 403, message: "forbidden" })).toBe(false);
});
-
- it("labels abort and unknown local errors failed-shaped", () => {
- const abort = new Error("aborted");
- abort.name = "AbortError";
- expect(isRetryableError(abort)).toBe(false);
- expect(isRetryableError(new Error("unexpected token in JSON"))).toBe(false);
- expect(isRetryableError(null)).toBe(false);
- expect(isRetryableError(undefined)).toBe(false);
- });
});
describe("isAuthenticationError", () => {
@@ -1316,9 +1291,9 @@ describe("isFastModeUnsupportedError (fast_mode rejected by a model without a fa
});
it("stays scoped to fast_mode: other unsupported parameters and unrelated errors do not match", () => {
- // Another UnsupportedParameterError source (e.g. temperature) keeps the default failed
- // classification and the engine's retry ladder — this detector must not widen the
- // permanent special case.
+ // Another UnsupportedParameterError source (e.g. temperature) keeps the default
+ // retryable classification and the engine's retry ladder — this detector must not
+ // widen the fatal special case.
expect(
isFastModeUnsupportedError(
new UnsupportedParameterError({
@@ -1703,10 +1678,12 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
return { messages, outcome: res.value as LLMOutcome };
}
- it("returns failed on a build failure such as empty input (never throws)", async () => {
+ it("returns fatal on a build failure such as empty input (never throws)", async () => {
const model = new SeamModel((sig) => hang(sig));
const { messages, outcome } = await drain(model.streamGenerate({ newMessages: [] }));
- expect(outcome.status).toBe("failed"); // A mergeOmniToUniMessage failure converges to failed, never throws
+ // A mergeOmniToUniMessage failure converges to fatal (the same input can never
+ // assemble on a retry), never throws.
+ expect(outcome.status).toBe("fatal");
expect(messages).toHaveLength(0);
});
@@ -1747,18 +1724,18 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
expect(res.value).toMatchObject({ status: "aborted" });
});
- it("classifies an idle timeout as timeout, with no token_usage", async () => {
+ it("classifies an idle timeout as retryable, with no token_usage", async () => {
const model = new SeamModel((sig) => hang(sig), 30); // 30ms idle timeout
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("timeout");
+ expect(outcome.status).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
- it("classifies an idle timeout as timeout even when the stream ends gracefully on abort", async () => {
+ it("classifies an idle timeout as retryable even when the stream ends gracefully on abort", async () => {
// The underlying implementation does not throw on abort, and ends gracefully with done:
- // this must still be classified as timeout, not mistakenly as completed.
+ // this must still be classified as retryable, not mistakenly as completed.
async function* gracefulHang(signal: AbortSignal): AsyncGenerator<UniEvent> {
await new Promise<void>((resolve) => {
if (signal.aborted) {
@@ -1772,32 +1749,32 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("timeout");
+ expect(outcome.status).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
- it("classifies a network drop as timeout, closing the open text segment, no token_usage", async () => {
+ it("classifies a network drop as retryable, closing the open text segment, no token_usage", async () => {
const model = new SeamModel(() => dropAfterText());
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("timeout");
+ expect(outcome.status).toBe("retryable");
const complete = messages.find((m) => typeOf(m) === "text");
expect((complete!.payload as TextPayload).text).toBe("hi");
- expect((complete!.payload as TextPayload).stop_reason).toBe("timeout");
+ expect((complete!.payload as TextPayload).stop_reason).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
- it("classifies an AgentHub JSON parse exception as malformed and closes partial output", async () => {
+ it("classifies an AgentHub JSON parse exception as retryable and closes partial output", async () => {
const model = new SeamModel(() => malformedJsonAfterText());
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("malformed");
+ expect(outcome.status).toBe("retryable");
expect(outcome.errorMessage).toContain("Unexpected token");
const complete = messages.find((m) => typeOf(m) === "text");
expect((complete!.payload as TextPayload).text).toBe("hi");
- expect((complete!.payload as TextPayload).stop_reason).toBe("malformed");
+ expect((complete!.payload as TextPayload).stop_reason).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
@@ -1813,7 +1790,7 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("malformed");
+ expect(outcome.status).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
async function* noEvents(): AsyncGenerator<UniEvent> {
@@ -1823,7 +1800,7 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { outcome: outcome2 } = await drain(
model2.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome2.status).toBe("malformed");
+ expect(outcome2.status).toBe("retryable");
});
it("a user abort ends the run even when upstream ignores the signal and never settles", async () => {
@@ -1844,23 +1821,22 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
drain(model.streamGenerate({ newMessages: [userText("go")] })),
2000,
);
- expect(outcome.status).toBe("timeout");
+ expect(outcome.status).toBe("retryable");
});
- it("classifies a credentials failure as its own terminal status auth; params stay failed", async () => {
+ it("classifies a credentials failure as fatal; a parameter 400 is fatal too", async () => {
const model = new SeamModel(() => authError());
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- // 401 = credentials failure: its own stop reason so hosts can gate input until the
- // model's API key is updated (same engine behavior as failed).
- expect(outcome.status).toBe("auth");
+ // 401 = credentials failure: fatal — the engine stops the run and the errorMessage
+ // tells the user to update the model's API key.
+ expect(outcome.status).toBe("fatal");
expect(outcome.errorMessage).toContain("invalid api key");
expect(messages.map(typeOf)).not.toContain("token_usage");
- // A parameter error (400) labels failed — honestly reported, and still retried by the
- // engine like every non-auth failure (see engine.test.ts: it burns the ladder, then
- // surfaces verbatim).
+ // A parameter error (400) is a definitive provider rejection: fatal — the identical
+ // request fails identically on every retry, so the engine surfaces it immediately.
async function* paramError(): AsyncGenerator<UniEvent> {
throw Object.assign(new Error("unknown parameter: max_output_tokens"), { status: 400 });
}
@@ -1868,14 +1844,14 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { outcome: outcome2 } = await drain(
model2.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome2.status).toBe("failed");
+ expect(outcome2.status).toBe("fatal");
});
- it("marks a fast-mode rejection as permanent failed with actionable guidance (guarded on its own config)", async () => {
+ it("marks a fast-mode rejection as fatal with actionable guidance (guarded on its own config)", async () => {
// AgentHub throws UnsupportedParameterError before any network I/O when fast_mode is
// enabled on a model without a fast tier: deterministic for this object's frozen config,
- // so it is the one failed shape the engine aborts on instead of retrying
- // (LLMOutcome.permanent; the engine side is covered in engine.test.ts).
+ // so it is fatal — the engine aborts on it instead of retrying (the engine side is
+ // covered in engine.test.ts).
async function* fastModeRejected(): AsyncGenerator<UniEvent> {
throw new UnsupportedParameterError({
client: "KimiK3Client",
@@ -1905,27 +1881,26 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("failed");
- expect(outcome.permanent).toBe(true);
+ expect(outcome.status).toBe("fatal");
// The surfaced text keeps AgentHub's own words and appends where the switch lives.
expect(outcome.errorMessage).toContain("Kimi does not support fast mode.");
expect(outcome.errorMessage).toContain(FAST_MODE_UNSUPPORTED_GUIDANCE);
expect(messages.map(typeOf)).not.toContain("token_usage");
// Guard: the same error on a config that never sent fast_mode cannot be ours to
- // explain — it stays a plain failed (no permanent) and rides the engine's ladder.
+ // explain — it carries no HTTP status, so it stays retryable and rides the engine's
+ // ladder.
const model2 = new FastSeamModel(false);
const { outcome: outcome2 } = await drain(
model2.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome2.status).toBe("failed");
- expect(outcome2.permanent).toBeUndefined();
+ expect(outcome2.status).toBe("retryable");
});
- it("an explicit auth signal wins whatever the message says: status auth, never retried", async () => {
+ it("an explicit auth signal wins whatever the message says: fatal, never retried", async () => {
// 403 + body code invalid_api_key + a message mentioning the subscription: the
- // explicit credential signal decides — auth is the one terminal status, and no
- // message vocabulary may pull a dead credential back onto the retry ladder.
+ // explicit credential signal decides — fatal, and no message vocabulary may pull a
+ // dead credential back onto the retry ladder.
async function* dressedAuthError(): AsyncGenerator<UniEvent> {
throw Object.assign(new Error("subscription key invalid"), {
status: 403,
@@ -1936,17 +1911,15 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("auth");
+ expect(outcome.status).toBe("fatal");
expect(outcome.errorMessage).toContain("subscription key invalid");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
- it("labels a bare 403 as failed — a retryable status, no quota/message heuristics involved", async () => {
- // Every LLM error retries except auth: a 403 without a credential signal lands in
- // `failed` (which the engine reconnects on) instead of being probed for quota or
- // subscription vocabulary. The former quota detector is gone; a quota-coded provider
- // rejection classifies exactly the same way, and its real message still rides on the
- // outcome for observability.
+ it("labels a bare 403 as fatal — a definitive provider rejection, no message heuristics involved", async () => {
+ // A 4xx (minus 408/429) is a definitive rejection: fatal, whatever the message says.
+ // A quota-coded provider rejection classifies exactly the same way, and its real
+ // message still rides on the outcome for observability.
async function* bare403(): AsyncGenerator<UniEvent> {
throw Object.assign(new Error("forbidden"), { status: 403 });
}
@@ -1954,7 +1927,7 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("failed");
+ expect(outcome.status).toBe("fatal");
expect(outcome.errorMessage).toContain("forbidden");
expect(messages.map(typeOf)).not.toContain("token_usage");
@@ -1968,11 +1941,11 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { outcome: outcome2 } = await drain(
model2.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome2.status).toBe("failed");
+ expect(outcome2.status).toBe("fatal");
expect(outcome2.errorMessage).toContain("insufficient_user_quota");
});
- it("classifies an undici transport drop (TypeError terminated, cause UND_ERR_SOCKET) as timeout", async () => {
+ it("classifies an undici transport drop (TypeError terminated, cause UND_ERR_SOCKET) as retryable", async () => {
async function* socketDrop(): AsyncGenerator<UniEvent> {
yield ev({ content_items: [{ type: "text", text: "hi" }] });
throw new TypeError("terminated", {
@@ -1983,13 +1956,13 @@ describe("GenerativeModel.streamGenerate outcome classification (PRN-013)", () =
const { messages, outcome } = await drain(
model.streamGenerate({ newMessages: [userText("go")] }),
);
- expect(outcome.status).toBe("timeout");
+ expect(outcome.status).toBe("retryable");
const complete = messages.find((m) => typeOf(m) === "text");
- expect((complete!.payload as TextPayload).stop_reason).toBe("timeout");
+ expect((complete!.payload as TextPayload).stop_reason).toBe("retryable");
expect(messages.map(typeOf)).not.toContain("token_usage");
});
- it("classifies a user abort (mid idle) as aborted, not timeout", async () => {
+ it("classifies a user abort (mid idle) as aborted, not retryable", async () => {
const controller = new AbortController();
const model = new SeamModel((sig) => hang(sig)); // Default 10s timeout, won't fire first
const p = drain(
diff --git a/packages/core/test/mcp-tools.test.ts b/packages/core/test/mcp-tools.test.ts
index 697a2560..1386a8dd 100644
--- a/packages/core/test/mcp-tools.test.ts
+++ b/packages/core/test/mcp-tools.test.ts
@@ -162,7 +162,7 @@ describe("MCP over stdio through Environment", () => {
it("maps a tool-level isError result to failed with the server's message", async () => {
const final = finalPayload(await runTool(env, "mcp__fx__fail", {}));
- expect(final.stop_reason).toBe("failed");
+ expect(final.stop_reason).toBe("fatal");
expect(final.output).toBe("boom");
});
@@ -193,7 +193,7 @@ describe("MCP over stdio through Environment", () => {
it("collapses an unknown MCP name into the unknown-tool reply", async () => {
const final = finalPayload(await runTool(env, "mcp__fx__nope", {}));
- expect(final.stop_reason).toBe("failed");
+ expect(final.stop_reason).toBe("fatal");
expect(final.output).toContain("Unknown tool: mcp__fx__nope");
});
});
@@ -233,7 +233,7 @@ describe("MCP over stdio — per-server budgets and interruption", () => {
const env = makeEnv({ timeoutMs: 300 });
try {
const final = finalPayload(await runTool(env, "mcp__fx__slow", { ms: 60_000 }));
- expect(final.stop_reason).toBe("failed");
+ expect(final.stop_reason).toBe("fatal");
expect(final.output).toContain("[tool timeout: exceeded 300ms]");
} finally {
env.dispose();
@@ -602,8 +602,13 @@ describe("Session first-run bootstrap events", () => {
status: "completed",
tools: 6,
});
- expect(results[1]).toMatchObject({ server: "broken", transport: "stdio", status: "failed" });
- expect(results[1]!.error).toBeTruthy();
+ expect(results[1]).toMatchObject({
+ server: "broken",
+ transport: "stdio",
+ status: "fatal",
+ error_code: "connect_failed",
+ });
+ expect(results[1]!.error_message).toBeTruthy();
expect(results[1]!.duration_ms).toBeGreaterThanOrEqual(0);
} finally {
await provider.close();
diff --git a/packages/core/test/omnimessage.test.ts b/packages/core/test/omnimessage.test.ts
index 440151cd..891cc0b8 100644
--- a/packages/core/test/omnimessage.test.ts
+++ b/packages/core/test/omnimessage.test.ts
@@ -38,9 +38,13 @@ describe("builders", () => {
decision: "allow",
tool_call_id: "call_1",
});
- expect(abortEvent("stop").payload).toMatchObject({
+ expect(abortEvent().payload).toMatchObject({
type: "abort",
- reason: "stop",
+ error_code: "user_abort",
+ });
+ expect(abortEvent("backoff_interrupted").payload).toMatchObject({
+ type: "abort",
+ error_code: "backoff_interrupted",
});
});
diff --git a/packages/core/test/read-image.test.ts b/packages/core/test/read-image.test.ts
index 3e6cb43e..a05960f5 100644
--- a/packages/core/test/read-image.test.ts
+++ b/packages/core/test/read-image.test.ts
@@ -78,7 +78,7 @@ describe("read_image — local files", () => {
it("finishes as failed with an explanation when the file does not exist", async () => {
const { result, text } = await run({ source: "missing.png" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(result?.images).toBeUndefined();
expect(text).toContain("missing.png");
});
@@ -88,21 +88,21 @@ describe("read_image — local files", () => {
PNG_1X1.copy(big); // Header carries the PNG magic number, ensuring the failure is due to size, not type
await writeFile(path.join(tmp, "big.png"), big);
const { result, text } = await run({ source: "big.png" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("too large");
});
it("finishes as failed for an unsupported image type", async () => {
await writeFile(path.join(tmp, "img.bmp"), Buffer.from("BM not really an image"));
const { result, text } = await run({ source: "img.bmp" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("Unsupported image type");
});
it("finishes as failed with a clear note when source points to a directory (no raw EISDIR passthrough)", async () => {
await mkdir(path.join(tmp, "subdir"));
const { result, text } = await run({ source: "subdir" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("not a file");
expect(text).not.toContain("EISDIR");
});
@@ -110,13 +110,13 @@ describe("read_image — local files", () => {
it("finishes as failed for an empty file (the extension fallback must not let empty base64 through)", async () => {
await writeFile(path.join(tmp, "empty.png"), Buffer.alloc(0));
const { result, text } = await run({ source: "empty.png" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("empty");
});
it("finishes as failed when the source argument is missing", async () => {
const { result, text } = await run({}, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain('"source"');
});
});
@@ -141,7 +141,7 @@ describe("read_image — http(s) URL", () => {
vi.fn(async () => new Response("nope", { status: 404 })),
);
const { result, text } = await run({ source: "https://example.com/missing.png" }, tmp);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(text).toContain("404");
});
});
diff --git a/packages/core/test/replay.test.ts b/packages/core/test/replay.test.ts
index 035b7ef3..7e2bb657 100644
--- a/packages/core/test/replay.test.ts
+++ b/packages/core/test/replay.test.ts
@@ -97,7 +97,7 @@ describe("resumeTrace", () => {
meta(),
userText("A"),
requestBegin(),
- requestEnd("timeout"), // failed with zero output
+ requestEnd("retryable"), // failed with zero output
requestBegin(),
assistantText("ok"),
requestEnd("completed"),
@@ -308,8 +308,8 @@ describe("resumeTrace", () => {
compactionBegin({ reason: "context", mode: "summarize", context: 10, turns: 1 }),
userText("please summarize"),
requestBegin(),
- requestEnd("failed"),
- compactionEnd({ reason: "context", mode: "summarize", status: "failed" }),
+ requestEnd("retryable"),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
userText("continue"),
requestBegin(),
assistantText("sure"),
@@ -598,7 +598,7 @@ describe("resumeTrace regressions (PR #39 review)", () => {
requestBegin(),
toolCall({ name: "exec_command", arguments: "{}", toolCallId: "tc1" }),
toolCallOutput({ output: "ran-during-timeout", toolCallId: "tc1" }),
- requestEnd("timeout"), // this round is dropped: tc1 never entered AgentHub history
+ requestEnd("retryable"), // this round is dropped: tc1 never entered AgentHub history
requestBegin(),
assistantText("recovered"),
requestEnd("completed"),
@@ -707,13 +707,13 @@ describe("resumeTrace regressions (PR #39 review)", () => {
toolCallOutput({
output: "[tool error] the compaction request expects a summary, not tool calls",
toolCallId: "c1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
requestBegin(),
thinkingMessage("still nothing"),
requestEnd("completed"),
tokenUsage(usage(170), usage(480)),
- compactionEnd({ reason: "context", mode: "summarize", status: "failed" }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
]);
expect(result.contextClosed).toBe(false);
expect(result.pendingSummary).toBeUndefined();
@@ -757,7 +757,7 @@ describe("resumeTrace regressions (PR #39 review)", () => {
toolCallOutput({
output: "[tool error] the compaction request expects a summary, not tool calls",
toolCallId: "c1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
compactionEnd({ reason: "context", mode: "summarize", status: "aborted" }),
]);
diff --git a/packages/core/test/resume.test.ts b/packages/core/test/resume.test.ts
index d286e6a3..863b188a 100644
--- a/packages/core/test/resume.test.ts
+++ b/packages/core/test/resume.test.ts
@@ -167,7 +167,7 @@ describe("agent.resumeSession", () => {
requestBegin(),
assistantText("partial answer", "aborted"),
requestEnd("aborted"),
- abortEvent("aborted by user"),
+ abortEvent(),
]);
const session = await agent.resumeSession({ sessionId: SID });
@@ -203,7 +203,7 @@ describe("agent.resumeSession", () => {
type: "compaction_end",
reason: "context",
mode: "summarize",
- status: "failed",
+ status: "retryable",
});
// Nothing was reconstructed from the interrupted compaction: the original context is
// resumed as it stood before it, with no `[context_summary]` injected.
@@ -441,7 +441,7 @@ describe("agent.resumeSession", () => {
userText("q1"),
requestBegin(),
assistantText("half-writ"),
- abortEvent("user"),
+ abortEvent(),
]);
const session = await agent.resumeSession({ sessionId: SID });
diff --git a/packages/core/test/session-title.test.ts b/packages/core/test/session-title.test.ts
index 1f024683..5a2390c7 100644
--- a/packages/core/test/session-title.test.ts
+++ b/packages/core/test/session-title.test.ts
@@ -96,7 +96,7 @@ describe("session-title", () => {
assistantText("partial"),
tokenUsage(emptyTokenCounts(), { cache_read: 0, cache_write: 0, output: 1, total: 1 }),
],
- { status: "failed", errorMessage: "401" },
+ { status: "fatal", errorMessage: "401" },
),
{ userText: "u", assistantText: "a" },
);
diff --git a/packages/core/test/subagent.test.ts b/packages/core/test/subagent.test.ts
index da765d6d..e4a36483 100644
--- a/packages/core/test/subagent.test.ts
+++ b/packages/core/test/subagent.test.ts
@@ -22,6 +22,7 @@ import { collectWindow } from "../src/environment/tools/subagent/collect.js";
import type { MessageOrigin, OmniMessage } from "../src/omnimessage/index.js";
import type {
ApproveFn,
+ RunCutoff,
EnvironmentServices,
SubagentHandle,
SubagentRunner,
@@ -232,7 +233,7 @@ describe("run_subagent tool (foreground)", () => {
const { services } = makeServices(runner);
const tool = createSubagentTool(DEF, services);
const { out, result } = await collectWithReturn(tool.execute(args, CTX));
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("must be given together");
expect(spawned).toHaveLength(0);
}
@@ -273,7 +274,7 @@ describe("run_subagent tool (foreground)", () => {
const { out, result } = await collectWithReturn(
tool.execute({ prompt: "x", thinking_level }, CTX),
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("invalid `thinking_level`");
expect(ownDeltas(out)).toContain("low / medium / high / xhigh / max");
expect(spawned).toHaveLength(0);
@@ -320,7 +321,7 @@ describe("run_subagent tool (foreground)", () => {
const { services } = makeServices();
const tool = createSubagentTool(DEF, services);
const { out, result } = await collectWithReturn(tool.execute({ prompt: "x" }, CTX));
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("no subagent runner");
});
@@ -334,7 +335,7 @@ describe("run_subagent tool (foreground)", () => {
const { services } = makeServices(runner);
const tool = createSubagentTool(DEF, services);
const { out, result } = await collectWithReturn(tool.execute({}, CTX));
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("prompt");
});
@@ -355,13 +356,14 @@ describe("run_subagent tool (foreground)", () => {
it("reports a failed delegation when the child session aborts", async () => {
const runner = runnerOf(async function* () {
yield withOrigin(partialText("delta", "partial"), HOP);
- yield withOrigin(abortEvent("llm error"), HOP);
+ yield withOrigin(abortEvent(), HOP);
+ return { kind: "abort" as const, errorCode: "user_abort" as const };
});
const { services } = makeServices(runner);
const tool = createSubagentTool(DEF, services);
const { result } = await collectWithReturn(tool.execute({ prompt: "x" }, CTX));
- expect(result?.stopReason).toBe("failed");
- expect(result?.note).toContain("subagent aborted: llm error");
+ expect(result?.stopReason).toBe("fatal");
+ expect(result?.note).toContain("subagent aborted: user_abort");
});
it("surfaces child approval requests through the parent approve callback", async () => {
@@ -507,7 +509,7 @@ describe("run_subagent backgrounding + input_subagent", () => {
const { out, result } = await collectWithReturn(
writeTool.execute({ subagent_id: id, prompt: "more", yield_time_ms: 250 }, CTX),
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("still running");
child.release();
});
@@ -518,7 +520,7 @@ describe("run_subagent backgrounding + input_subagent", () => {
const { out, result } = await collectWithReturn(
writeTool.execute({ subagent_id: "subagent-deadbeef" }, CTX),
);
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("unknown subagent_id subagent-deadbeef");
});
@@ -576,7 +578,7 @@ describe("run_subagent backgrounding + input_subagent", () => {
}
const tool = createSubagentTool(DEF, services);
const { out, result } = await collectWithReturn(tool.execute({ prompt: "x" }, CTX));
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(ownDeltas(out)).toContain("too many background subagents");
});
@@ -675,7 +677,7 @@ describe("subagent steering and per-run abort", () => {
async spawn() {
const handle: SubagentHandle = {
sessionId: HOP,
- async *run({ messages, signal }): AsyncGenerator<OmniMessage> {
+ async *run({ messages, signal }): AsyncGenerator<OmniMessage, RunCutoff | null> {
const prompt = promptOf(messages);
inputs.push(messages);
prompts.push(prompt);
@@ -683,15 +685,16 @@ describe("subagent steering and per-run abort", () => {
yield withOrigin(partialText("delta", `start:${prompt} `), HOP);
await Promise.race([gate, aborted(signal)]);
if (signal?.aborted) {
- yield withOrigin(abortEvent("run aborted"), HOP);
- return;
+ yield withOrigin(abortEvent(), HOP);
+ return { kind: "abort", errorCode: "user_abort" };
}
yield withOrigin(partialText("delta", `end:${prompt}`), HOP);
yield withOrigin(assistantText(`end:${prompt}`), HOP);
- return;
+ return null;
}
yield withOrigin(partialText("delta", `ran:${prompt}`), HOP);
yield withOrigin(assistantText(`ran:${prompt}`), HOP);
+ return null;
},
steer(messages) {
steers.push(messages);
@@ -745,7 +748,7 @@ describe("subagent steering and per-run abort", () => {
);
expect(ownDeltas(out)).toContain(`aborting subagent ${id}'s current run`);
// The aborted round settles as failed with the child's abort reason; the session is kept.
- expect(result?.stopReason).toBe("failed");
+ expect(result?.stopReason).toBe("fatal");
expect(result?.note).toContain("subagent aborted");
expect(result?.note).toContain(`subagent idle with subagent_id ${id}`);
expect(manager.get(id)).toBeDefined();
diff --git a/packages/server/src/http/routes/agent-config.ts b/packages/server/src/http/routes/agent-config.ts
index 35fc84b1..d13bd6a2 100644
--- a/packages/server/src/http/routes/agent-config.ts
+++ b/packages/server/src/http/routes/agent-config.ts
@@ -112,7 +112,9 @@ export function agentConfigRoutes(deps: AppDeps): Hono<AppEnv> {
if (outcome === undefined || outcome.status !== "completed") {
return c.json({
ok: false,
- error: outcome?.error ?? (warnings.length > 0 ? warnings.join("; ") : "connect failed"),
+ error:
+ outcome?.error_message ??
+ (warnings.length > 0 ? warnings.join("; ") : "connect failed"),
latencyMs,
} satisfies McpServerTestResponse);
}
diff --git a/packages/server/src/runtime/stream-error-watcher.ts b/packages/server/src/runtime/stream-error-watcher.ts
index 7bbdb2ce..337b8c36 100644
--- a/packages/server/src/runtime/stream-error-watcher.ts
+++ b/packages/server/src/runtime/stream-error-watcher.ts
@@ -10,28 +10,21 @@
* (its state wraps up accordingly, see close).
*
* LLM (source = `llm`): reads the status of `request_end` —
- * - `auth` → unexpected (`llm_auth`): the credential was rejected, the engine never retries
- * it, and only a human holding the key can fix it. Its own code rather than the failed
- * bucket, because dedup is `(source, code, Project)` over a short window: sharing a code
- * with a class that now fires on every recovered blip would let a real credential failure
- * be swallowed as a duplicate of something already handled.
- * - `failed` → depends on whether the retry ladder carried it. The engine reconnects on
- * `failed` too, so the same status covers both "a gateway hiccup nobody needed to know
- * about" and "the run died on it":
+ * - `fatal` → unexpected (`llm_fatal`): a definitive rejection (provider 4xx, credentials,
+ * fast mode on a model without a fast tier) the engine never retries; the run stopped on
+ * it and only a human changing config or credentials can fix it.
+ * - `retryable` → depends on whether the retry ladder carried it:
* - another attempt followed (a `request_begin` resolved the pending record) → expected
- * (`llm_failed_retried`), same footing as timeout/malformed: recorded for the record,
- * not raised at an operator;
+ * (`llm_retried`): recorded for the record, not raised at an operator;
* - nothing followed but the end of the run (an `abort`, or close) → unexpected
* (`llm_failed`): the retries did not recover it, the user lost the turn, needs a human.
- * - `timeout` / `malformed` → expected (the engine already reconnects and retries, part
- * of normal operation);
* - `aborted` / `completed` are not recorded (the former is a user-initiated interrupt,
* not an error).
*
* The message uses the real reason: `request_end` carries the error detail on
* non-completed statuses (`error_message`, from LLMOutcome), and the `abort` event's reason is
- * core's failure-reason prose (e.g. `llm request error: 401 …` / `malformed response
- * failed after N retries`). A `request_end` failure is first held pending (status + its
+ * core's failure-reason prose (e.g. `llm request error: 401 …` / `llm request failed
+ * after N retries: …`). A `request_end` failure is first held pending (status + its
* own detail), not persisted immediately, and is resolved at the next request boundary:
* - Immediately followed by `abort` → use its reason as the message (the real reason);
* - Immediately followed by `request_begin` (the engine is retrying — no abort will ever
@@ -45,9 +38,10 @@
* Pending state is bucketed by origin: subagent messages interleave with the parent
* session's (even more so with parallel subagents), and mixing them up would misattribute.
*
- * Environment (source = `environment`): reads `tool_call_output`'s stop_reason ∈
- * {failed, timeout} → expected (the error is fed back to the model, and the Agent
- * adjusts on its own; `aborted` is denial/interruption, not recorded). Exactly one shape is
+ * Environment (source = `environment`): reads `tool_call_output`'s stop_reason —
+ * `fatal` (and the legacy failed/timeout spellings) → expected (the error is fed back to
+ * the model, and the Agent adjusts on its own; `aborted` is denial/interruption, not
+ * recorded). Exactly one shape is
* dropped: a command tool's ordinary non-zero exit, which is how a shell command reports
* information rather than a fault — see isOrdinaryCommandExit; the same tools' environment
* faults (killed by a signal, spawn failure, tool timeout) still record.
@@ -89,7 +83,7 @@ export const TOOL_NAMES_MAX = 1000;
export const ORIGIN_CTX_MAX = 200;
/** Recorded LLM failure states (`aborted` / `completed` are not errors and aren't included here). */
-type LlmFailure = "failed" | "timeout" | "malformed" | "auth";
+type LlmFailure = "retryable" | "fatal";
/** Error code, classification, and fallback message (the last used when no abort reason is available). */
interface FailureSpec {
@@ -101,54 +95,45 @@ interface FailureSpec {
/** LLM failure state → its spec, for a failure the retry ladder did NOT carry (see the file header). */
const LLM_FAILURES: Record<LlmFailure, FailureSpec> = {
// Reached here only when no further attempt followed: the ladder ran out (or the run ended
- // on it), so the user lost the turn and someone should look. A `failed` that was retried
- // takes LLM_FAILED_RETRIED instead.
- failed: {
+ // on it), so the user lost the turn and someone should look. A `retryable` that was retried
+ // takes LLM_RETRIED instead.
+ retryable: {
code: "llm_failed",
kind: "unexpected",
text: "LLM request failed and the retries did not recover it.",
},
// Its own code, not the failed bucket: dedup is `(source, code, Project)` over a short
- // window, and `llm_failed_retried` can now fire on any recovered blip — sharing a bucket
- // would let a genuine credential failure be dropped as a duplicate of one.
- auth: {
- code: "llm_auth",
+ // window, and `llm_retried` can fire on any recovered blip — sharing a bucket would let
+ // a genuine run-ending failure be dropped as a duplicate of one. The engine never
+ // retries a fatal, so this pending record always resolves through its abort.
+ fatal: {
+ code: "llm_fatal",
kind: "unexpected",
- text: "LLM request rejected: the provider did not accept the credentials.",
- },
- timeout: {
- code: "llm_timeout",
- kind: "expected",
- text: "LLM request timed out (the engine reconnects and retries).",
- },
- malformed: {
- code: "llm_malformed",
- kind: "expected",
- text: "LLM response could not be parsed (the engine reconnects and retries).",
+ text: "LLM request failed with a non-retryable error.",
},
};
/**
- * A `failed` the engine went on to retry: expected, exactly like timeout/malformed — the
- * engine's defined handling path absorbed it and the user never lost anything, so it belongs
- * in the record but not in an operator's queue. Its own code so the exhausted case keeps
- * `llm_failed` to itself and neither can dedup the other away.
+ * A `retryable` the engine went on to retry: expected — the engine's defined handling path
+ * absorbed it and the user never lost anything, so it belongs in the record but not in an
+ * operator's queue. Its own code so the exhausted case keeps `llm_failed` to itself and
+ * neither can dedup the other away.
*/
-const LLM_FAILED_RETRIED: FailureSpec = {
- code: "llm_failed_retried",
+const LLM_RETRIED: FailureSpec = {
+ code: "llm_retried",
kind: "expected",
text: "LLM request failed (the engine reconnects and retries).",
};
-/** Recorded tool failure states (`aborted` = denial/interruption, not an error). */
-type ToolFailure = "failed" | "timeout";
+/** Recorded tool failure states (`aborted` = denial/interruption, not an error): `fatal` is the live spelling, failed/timeout the legacy Trace ones. */
+type ToolFailure = "fatal" | "failed" | "timeout";
function isLlmFailure(s: unknown): s is LlmFailure {
- return s === "failed" || s === "timeout" || s === "malformed" || s === "auth";
+ return s === "retryable" || s === "fatal";
}
function isToolFailure(s: unknown): s is ToolFailure {
- return s === "failed" || s === "timeout";
+ return s === "fatal" || s === "failed" || s === "timeout";
}
/**
@@ -219,8 +204,11 @@ function toolFailureText(output: string): string {
}
export class StreamErrorWatcher {
- /** LLM failures awaiting a real reason: origin → failure state + the request_end's own detail (see file header; each session has at most one in-flight Request). */
- private readonly pending = new Map<string, { status: LlmFailure; message?: string }>();
+ /** LLM failures awaiting resolution: origin → failure state + the request_end's own detail, attempt ordinal, and whether a retry was planned (see file header; each session has at most one in-flight Request). */
+ private readonly pending = new Map<
+ string,
+ { status: LlmFailure; message?: string; attempt?: number; willRetry?: boolean }
+ >();
/** Names of in-flight tool calls: `origin \0 tool_call_id` → tool name (dequeued once output arrives). */
private readonly toolNames = new Map<string, string>();
/** Subagent identity: origin → that subagent's `{agentId, sessionId}` (see the file header's attribution section). */
@@ -297,6 +285,8 @@ export class StreamErrorWatcher {
status?: StopReason;
reason?: string | null;
error_message?: string;
+ attempt?: number;
+ retry_in_ms?: number;
};
const key = originKey(msg);
if (p.type === "compaction_end") {
@@ -308,8 +298,12 @@ export class StreamErrorWatcher {
if (isLlmFailure(p.status)) {
this.pending.set(key, {
status: p.status,
+ ...(typeof p.attempt === "number" ? { attempt: p.attempt } : {}),
+ // retry_in_ms present = the engine planned another attempt: a failure resolved
+ // non-retried despite it was cut short (an interrupt), not exhausted.
+ ...(p.retry_in_ms !== undefined ? { willRetry: true } : {}),
// The event's own error detail (request_end.error_message): the message of record
- // when no abort follows (the retry path).
+ // (an LLM failure produces no abort event; legacy streams still resolve through one).
...(typeof p.error_message === "string" && p.error_message.trim()
? { message: p.error_message }
: {}),
@@ -324,8 +318,9 @@ export class StreamErrorWatcher {
this.flush(key, null, true);
return;
}
- // Interrupted/failed exit: reason is core's only failure-reason text. No attempt follows
- // an abort, so a `failed` resolved here is one the retries did not recover.
+ // Abort marks a user interruption (only legacy streams carried failure reasons here).
+ // No attempt follows an abort, so a failure resolved here is one the retries did not
+ // recover — the interruption cut the ladder short.
if (p.type === "abort") {
this.flush(key, typeof p.reason === "string" ? p.reason : null);
}
@@ -347,13 +342,15 @@ export class StreamErrorWatcher {
const entry = this.pending.get(key);
if (entry === undefined) return;
this.pending.delete(key);
- const spec =
- retried && entry.status === "failed" ? LLM_FAILED_RETRIED : LLM_FAILURES[entry.status];
+ const spec = retried && entry.status === "retryable" ? LLM_RETRIED : LLM_FAILURES[entry.status];
const trimmed = reason?.trim();
- // Message priority: the abort reason (core's failure prose) → the staged request_end's
- // own detail (the retry path: no abort ever arrives) → the generic status text. A
- // user-interrupt message isn't a failure reason (see file header).
- const message = trimmed && !isUserAbortReason(trimmed) ? trimmed : (entry.message ?? spec.text);
+ // Message priority: a legacy abort reason (old cores' failure prose) → a sentence
+ // composed from the staged request_end's own detail and attempt — the terminal
+ // request_end is the record now, so the prose the abort used to carry is rebuilt here
+ // — → the generic status text. A user-interrupt message isn't a failure reason (see
+ // file header).
+ const message =
+ trimmed && !isUserAbortReason(trimmed) ? trimmed : this.composeMessage(entry, retried, spec);
this.errors.record({
source: "llm",
err: message,
@@ -363,6 +360,22 @@ export class StreamErrorWatcher {
});
}
+ /** The record prose for a failure resolved without an abort reason (see flush). */
+ private composeMessage(
+ entry: { status: LlmFailure; message?: string; attempt?: number; willRetry?: boolean },
+ retried: boolean,
+ spec: FailureSpec,
+ ): string {
+ if (entry.message === undefined) return spec.text;
+ // A retried blip — or one whose planned retry was cut short by an interrupt — keeps
+ // the raw detail; the terminal records read as sentences, the same shape the abort
+ // reason used to carry.
+ if (retried || entry.willRetry) return entry.message;
+ if (entry.status === "fatal") return `llm request error: ${entry.message}`;
+ const retries = Math.max(0, (entry.attempt ?? 1) - 1);
+ return `llm request failed after ${retries} ${retries === 1 ? "retry" : "retries"}: ${entry.message}`;
+ }
+
/**
* A failed compaction becomes its own error record (source = `compaction`) — a compaction
* is an ordinary LLM request whose failures (unusable summary and transport alike) core
@@ -384,7 +397,10 @@ export class StreamErrorWatcher {
attempt?: number;
error_message?: string;
};
- if (p.status !== "failed") return;
+ // Abandoned (retryable — made up at the next trigger) and fatal ends both record;
+ // legacy Traces spell the abandoned case "failed".
+ const status = p.status as string | undefined;
+ if (status !== "retryable" && status !== "fatal" && status !== "failed") return;
const attempts =
typeof p.attempt === "number" && p.attempt > 0
? ` after ${p.attempt} attempt${p.attempt === 1 ? "" : "s"}`
diff --git a/packages/server/src/services/message-window.ts b/packages/server/src/services/message-window.ts
index d7d6d6ea..a0385414 100644
--- a/packages/server/src/services/message-window.ts
+++ b/packages/server/src/services/message-window.ts
@@ -355,8 +355,14 @@ export async function scanMessages(
state.task.pendingCompactionUsage = false;
}
const status = p.status;
- // A retryable end renders a reconnect-hint item.
- if (status === "failed" || status === "timeout" || status === "malformed") {
+ // A retryable end renders a reconnect-hint item. The legacy spellings
+ // (failed/timeout/malformed) keep pre-convergence Traces rendering the same way.
+ if (
+ status === "retryable" ||
+ status === "failed" ||
+ status === "timeout" ||
+ status === "malformed"
+ ) {
breakRuns(state);
}
continue;
diff --git a/packages/server/src/services/project-config-service.ts b/packages/server/src/services/project-config-service.ts
index 2d1d81b6..f21a5964 100644
--- a/packages/server/src/services/project-config-service.ts
+++ b/packages/server/src/services/project-config-service.ts
@@ -1040,16 +1040,17 @@ export function isProbeContent(msg: OmniMessage): boolean {
* Probe verdict from the terminal LLM outcome. `completed` always passes. A `malformed`
* ending after genuine streamed content also passes: the typical case is a reasoning-heavy
* model that spends the probe's tiny max_tokens entirely on thinking (finish_reason=length ->
- * AgentHub's EmptyResponseError) — the endpoint, credential, and model id all demonstrably
- * work, which is what a connectivity test measures. Everything else (auth/parameter failures,
- * timeouts, malformed with nothing received) fails with the outcome's message.
+ * AgentHub's EmptyResponseError, a `retryable` outcome that still streamed content) — the
+ * endpoint, credential, and model id all demonstrably work, which is what a connectivity
+ * test measures. Everything else (fatal rejections, and retryable failures with nothing
+ * received) fails with the outcome's message.
*/
export function probeVerdict(
outcome: LLMOutcome,
sawContent: boolean,
): { ok: true } | { ok: false; message: string } {
if (outcome.status === "completed") return { ok: true };
- if (outcome.status === "malformed" && sawContent) return { ok: true };
+ if (outcome.status === "retryable" && sawContent) return { ok: true };
const detail =
"errorMessage" in outcome && outcome.errorMessage ? outcome.errorMessage : outcome.status;
return { ok: false, message: String(detail).slice(0, 300) };
diff --git a/packages/server/src/services/trace-service.ts b/packages/server/src/services/trace-service.ts
index d621138b..fec20dbb 100644
--- a/packages/server/src/services/trace-service.ts
+++ b/packages/server/src/services/trace-service.ts
@@ -1066,13 +1066,14 @@ export class TraceService {
// continue the turn, otherwise a single blip would split that turn's
// Tokens/duration/TPS across two Tasks and inflate the Task count.
//
- // This list must track the engine's, and the engine's differs by loop:
- // the turn loop retries `failed` too, while compaction deliberately fails
- // fast and stops on it (core's TURN_RETRY_STATUSES vs
- // COMPACTION_RETRY_STATUSES). Hence the compactionActive guard — a failed
- // compaction request really is the end of that request, and counting it
- // as a reconnect would invent an attempt that never happened.
+ // This list must track the engine's: `retryable` is the one reconnecting
+ // status today (both loops share RETRY_STATUSES). The legacy spellings keep
+ // pre-convergence Traces analyzable — in that era timeout/malformed always
+ // reconnected, while `failed` reconnected in the turn loop but ended a
+ // compaction attempt (the then fail-fast compaction policy), hence the
+ // compactionActive guard on it.
const retryable =
+ status === "retryable" ||
status === "timeout" ||
status === "malformed" ||
(status === "failed" && !compactionActive);
diff --git a/packages/server/test/context-breakdown.test.ts b/packages/server/test/context-breakdown.test.ts
index a2cebd4f..bb8d46ac 100644
--- a/packages/server/test/context-breakdown.test.ts
+++ b/packages/server/test/context-breakdown.test.ts
@@ -153,7 +153,7 @@ describe("buildContextBreakdown", () => {
{ cache_read: 0, cache_write: 0, output: 0, total: 0 },
),
requestEnd("completed"),
- compactionEnd({ reason: "context", mode: "summarize", status: "failed" }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
];
const plain = buildContextBreakdown(base);
const compacted = buildContextBreakdown(withCompaction);
diff --git a/packages/server/test/errors.test.ts b/packages/server/test/errors.test.ts
index fc59db98..7745e259 100644
--- a/packages/server/test/errors.test.ts
+++ b/packages/server/test/errors.test.ts
@@ -24,7 +24,7 @@ import {
toolCallOutput,
withOrigin,
} from "@prismshadow/penguin-core";
-import type { OmniMessage } from "@prismshadow/penguin-core";
+import type { OmniMessage, StopReason } from "@prismshadow/penguin-core";
import type { ProjectCreateResponse, UsageErrorsPage, UsageResponse } from "../src/api/types.js";
import { openDatabase } from "../src/db/database.js";
import { ErrorsRepo } from "../src/db/repos/errors.js";
@@ -339,6 +339,14 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
const rows = () =>
db.prepare("SELECT * FROM error_records ORDER BY id").all() as Array<Record<string, unknown>>;
+ /** A legacy stream's abort: prose reason only (cores from before the unified error pair). */
+ function legacyAbort(reason: string): OmniMessage {
+ const msg = abortEvent();
+ delete (msg.payload as { error_code?: string }).error_code;
+ (msg.payload as { reason?: string }).reason = reason;
+ return msg;
+ }
+
/** Feeds a sequence of messages and finalizes (close: persists any still-pending failure), returning the persisted rows. */
function feed(msgs: OmniMessage[]): Array<Record<string, unknown>> {
const w = watcher();
@@ -373,8 +381,8 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
// lost the turn and it belongs in front of an operator.
const got = feed([
requestBegin(),
- requestEnd("failed"),
- abortEvent("llm request failed after 5 retries: 400 unknown parameter"),
+ requestEnd("retryable"),
+ legacyAbort("llm request failed after 5 retries: 400 unknown parameter"),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({
@@ -389,50 +397,85 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
});
});
- it("LLM timeout / malformed → expected (engine retries); message from the abort reason", () => {
+ it("a retried retryable → expected; an exhausted one → unexpected; messages from abort/fallback", () => {
const got = feed([
requestBegin(),
- requestEnd("timeout"), // First attempt times out → the engine retries (revealed by the next request_begin: no reason text yet)
+ requestEnd("retryable"), // First attempt fails → the engine retries (revealed by the next request_begin: no reason text yet)
requestBegin(),
- requestEnd("malformed"),
- abortEvent("malformed response failed after 2 retries"),
+ requestEnd("retryable"),
+ legacyAbort("llm request failed after 2 retries"),
]);
expect(got).toHaveLength(2);
- expect(got[0]).toMatchObject({ source: "llm", kind: "expected", code: "llm_timeout" });
- expect(got[0]!.message).toContain("timed out"); // No abort arrived: falls back to the status text
+ expect(got[0]).toMatchObject({ source: "llm", kind: "expected", code: "llm_retried" });
+ // No abort arrived for the carried failure: falls back to the generic status text.
+ expect(got[0]!.message).toContain("reconnects and retries");
expect(got[1]).toMatchObject({
source: "llm",
- kind: "expected",
- code: "llm_malformed",
- message: "malformed response failed after 2 retries",
+ kind: "unexpected",
+ code: "llm_failed",
+ message: "llm request failed after 2 retries",
});
});
- it("request_end(auth) gets its own llm_auth code, out of the failed dedup bucket", () => {
- // Credentials rejection needs a code of its own now that `failed` fires on every blip
- // the ladder absorbs: dedup is (source, code, Project) over a short window, so sharing
- // a bucket would let a real credential failure be dropped as a duplicate.
+ it("request_end(fatal) gets its own llm_fatal code, out of the retried dedup bucket", () => {
+ // A run-ending rejection needs a code of its own: dedup is (source, code, Project)
+ // over a short window, and `llm_retried` fires on every blip the ladder absorbs —
+ // sharing a bucket would let a real credential failure be dropped as a duplicate.
const got = feed([
requestBegin(),
- requestEnd("auth", { errorMessage: "401 invalid x-api-key (invalid_api_key)" }),
- abortEvent("llm request error: 401 invalid x-api-key (invalid_api_key)"),
+ requestEnd("fatal", { errorMessage: "401 invalid x-api-key (invalid_api_key)" }),
+ legacyAbort("llm request error: 401 invalid x-api-key (invalid_api_key)"),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({
source: "llm",
kind: "unexpected",
- code: "llm_auth",
+ code: "llm_fatal",
message: "llm request error: 401 invalid x-api-key (invalid_api_key)",
});
});
- it("a failed the ladder carried → expected under its own code, not an operator incident", () => {
- // The inversion this guards against: the engine retries `failed`, so the same status now
- // covers "a gateway hiccup the user never saw" and "the run died on it". A following
- // request_begin proves another attempt happened — that one is expected, like a timeout.
+ it("a fatal with no abort (the live protocol): close persists it with the same prose", () => {
+ // Failures no longer emit an abort event; the pending record resolves at close, and
+ // the message is composed from the request_end's own detail.
+ const got = feed([
+ requestBegin(),
+ requestEnd("fatal", { errorMessage: "401 invalid x-api-key (invalid_api_key)" }),
+ ]);
+ expect(got).toHaveLength(1);
+ expect(got[0]).toMatchObject({
+ source: "llm",
+ kind: "unexpected",
+ code: "llm_fatal",
+ message: "llm request error: 401 invalid x-api-key (invalid_api_key)",
+ });
+ });
+
+ it("an exhausted ladder with no abort: the terminal request_end's attempt shapes the record", () => {
+ const got = feed([
+ requestBegin(),
+ requestEnd("retryable", { errorMessage: "socket hang up", attempt: 1, retryInMs: 2000 }),
+ requestBegin(),
+ // The final failure plans no retry (no retry_in_ms) — the run ends on it.
+ requestEnd("retryable", { errorMessage: "socket hang up", attempt: 2 }),
+ ]);
+ expect(got).toHaveLength(2);
+ expect(got[0]).toMatchObject({ code: "llm_retried", kind: "expected" });
+ expect(got[1]).toMatchObject({
+ source: "llm",
+ kind: "unexpected",
+ code: "llm_failed",
+ message: "llm request failed after 1 retry: socket hang up",
+ });
+ });
+
+ it("a retryable the ladder carried → expected under its own code, not an operator incident", () => {
+ // The same status covers "a gateway hiccup the user never saw" and "the run died on
+ // it". A following request_begin proves another attempt happened — that one is
+ // expected.
const got = feed([
requestBegin(),
- requestEnd("failed", {
+ requestEnd("retryable", {
errorMessage: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
}),
requestBegin(),
@@ -442,39 +485,38 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
expect(got[0]).toMatchObject({
source: "llm",
kind: "expected",
- code: "llm_failed_retried",
+ code: "llm_retried",
// No abort ever arrives on the retry path: the staged request_end's own detail is the
// message of record.
message: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
});
});
- it("a recovered failed does not dedup away a credential failure that lands right after", () => {
- // The two share a 2s dedup window and used to share the `llm_failed` code, so the auth
- // record was dropped outright — the one failure that always needs a human, silenced by
- // the one that never does.
+ it("a recovered retryable does not dedup away a fatal that lands right after", () => {
+ // The two share a 2s dedup window; separate codes keep the one failure that always
+ // needs a human from being silenced by the one that never does.
const got = feed([
requestBegin(),
- requestEnd("failed", { errorMessage: "Upstream HTTP/2 stream failed" }),
+ requestEnd("retryable", { errorMessage: "Upstream HTTP/2 stream failed" }),
requestBegin(), // The retry: resolves the failure above as recovered.
- requestEnd("auth", { errorMessage: "401 invalid x-api-key" }),
- abortEvent("llm request error: 401 invalid x-api-key"),
+ requestEnd("fatal", { errorMessage: "401 invalid x-api-key" }),
+ legacyAbort("llm request error: 401 invalid x-api-key"),
]);
expect(got.map((r) => [r.code, r.kind])).toEqual([
- ["llm_failed_retried", "expected"],
- ["llm_auth", "unexpected"],
+ ["llm_retried", "expected"],
+ ["llm_fatal", "unexpected"],
]);
});
- it("a retried failure keeps its real detail: request_end(timeout).message lands in the record", () => {
+ it("a retried failure keeps its real detail: request_end(retryable).message lands in the record", () => {
// The retry path: the engine reconnects (request_begin) and eventually succeeds, so no
// abort ever arrives for the staged failure — the request_end's own failure detail
- // (LLMOutcome.message, e.g. a quota code) is the message of record, not the generic
- // "timed out" status text. This is what the Cost center shows for a retried quota 403.
+ // (LLMOutcome.errorMessage, e.g. a rate-limit code) is the message of record, not the
+ // generic status text. This is what the Cost center shows for a retried 429.
const got = feed([
requestBegin(),
- requestEnd("timeout", {
- errorMessage: "403 no active subscription (insufficient_user_quota)",
+ requestEnd("retryable", {
+ errorMessage: "429 rate limited (slow down)",
}),
requestBegin(),
requestEnd("completed"),
@@ -483,16 +525,16 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
expect(got[0]).toMatchObject({
source: "llm",
kind: "expected",
- code: "llm_timeout",
- message: "403 no active subscription (insufficient_user_quota)",
+ code: "llm_retried",
+ message: "429 rate limited (slow down)",
});
});
- it("an abort reason still outranks the staged request_end detail (failed exit path)", () => {
+ it("an abort reason still outranks the staged request_end detail (fatal exit path)", () => {
const got = feed([
requestBegin(),
- requestEnd("failed", { errorMessage: "401 invalid x-api-key (invalid_api_key)" }),
- abortEvent("llm request error: 401 invalid x-api-key (invalid_api_key)"),
+ requestEnd("fatal", { errorMessage: "401 invalid x-api-key (invalid_api_key)" }),
+ legacyAbort("llm request error: 401 invalid x-api-key (invalid_api_key)"),
]);
expect(got).toHaveLength(1);
// The abort's prose (with core's "llm request error" framing) wins over the raw detail.
@@ -504,11 +546,12 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
// request_end detail IS the failure's reason — prefer it over the generic status text.
const got = feed([
requestBegin(),
- requestEnd("timeout", { errorMessage: "403 quota exceeded (insufficient_user_quota)" }),
- abortEvent("aborted during reconnect backoff"),
+ // A backoff interrupt cuts a PLANNED retry short — the failure announced one.
+ requestEnd("retryable", { errorMessage: "429 rate limited (slow down)", retryInMs: 4000 }),
+ abortEvent("backoff_interrupted"),
]);
expect(got).toHaveLength(1);
- expect(got[0]!.message).toBe("403 quota exceeded (insufficient_user_quota)");
+ expect(got[0]!.message).toBe("429 rate limited (slow down)");
});
it("aborted (user clicked Stop) is not an error: not recorded; neither is completed", () => {
@@ -518,27 +561,25 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
requestEnd("completed"),
requestBegin(),
requestEnd("aborted"),
- abortEvent("aborted by user"),
+ abortEvent(),
]),
).toHaveLength(0);
});
- it("interrupt during retry backoff: timeout still recorded, interrupt text distrusted", () => {
- const got = feed([
- requestBegin(),
- requestEnd("timeout"),
- abortEvent("aborted during reconnect backoff"),
- ]);
+ it("interrupt during retry backoff: the failure still recorded, interrupt text distrusted", () => {
+ const got = feed([requestBegin(), requestEnd("retryable"), abortEvent("backoff_interrupted")]);
expect(got).toHaveLength(1);
- expect(got[0]).toMatchObject({ code: "llm_timeout", kind: "expected" });
- expect(got[0]!.message).toContain("timed out");
+ // No retry followed (the interrupt ended the run): the conservative branch records it
+ // as the unrecovered class.
+ expect(got[0]).toMatchObject({ code: "llm_failed", kind: "unexpected" });
+ expect(got[0]!.message).toContain("did not recover");
expect(got[0]!.message).not.toContain("aborted");
});
it("failure pends for its reason; unresolved at run end → close persists (status text)", () => {
const w = watcher();
w.observe(requestBegin());
- w.observe(requestEnd("failed"));
+ w.observe(requestEnd("retryable"));
expect(rows()).toHaveLength(0); // Pending: waiting for the abort that immediately follows to supply the real reason
w.close();
const got = rows();
@@ -549,18 +590,20 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
});
it("parent/child LLM failures pend separately by origin; abort reasons never cross over", () => {
+ // The child fails fatal and the parent retryable-unrecovered: distinct codes, so the
+ // short-window dedup (source, code, Project) suppresses neither.
const got = feed([
requestBegin(), // parent session initiates
withOrigin(requestBegin(), "session-child"),
- withOrigin(requestEnd("timeout"), "session-child"),
- withOrigin(abortEvent("reconnect failed after 2 retries"), "session-child"),
- requestEnd("failed"), // the parent session's failure only wraps up now
- abortEvent("llm request error: 500 upstream"),
+ withOrigin(requestEnd("fatal"), "session-child"),
+ withOrigin(legacyAbort("llm request error: 401 invalid api key"), "session-child"),
+ requestEnd("retryable"), // the parent session's failure only wraps up now
+ legacyAbort("llm request error: 500 upstream"),
]);
expect(got).toHaveLength(2);
expect(got[0]).toMatchObject({
- code: "llm_timeout",
- message: "reconnect failed after 2 retries",
+ code: "llm_fatal",
+ message: "llm request error: 401 invalid api key",
});
expect(got[1]).toMatchObject({
code: "llm_failed",
@@ -584,22 +627,22 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
toolCallOutput({
output: "grep: no match\n[exit code: 1]",
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
call("input_command", "tc-2"),
toolCallOutput({
output: "make: *** [build] Error 2\n[exit code: 2]",
toolCallId: "tc-2",
- stopReason: "failed",
+ stopReason: "fatal",
}),
call("write_file", "tc-3"),
toolCallOutput({
output: "EACCES\n[exit code: 1]",
toolCallId: "tc-3",
- stopReason: "failed",
+ stopReason: "fatal",
}),
]);
- expect(got.map((r) => r.code)).toEqual(["tool_failed:write_file"]);
+ expect(got.map((r) => r.code)).toEqual(["tool_fatal:write_file"]);
});
it("a command tool killed by a signal, or that never spawned, is still recorded", () => {
@@ -613,19 +656,16 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
toolCallOutput({
output: "cc1plus: out of memory\n[terminated by signal SIGKILL]",
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
call("input_command", "tc-2"),
toolCallOutput({
output: "[spawn error: ENOENT: no such file or directory, posix_spawn '/bin/nope']",
toolCallId: "tc-2",
- stopReason: "failed",
+ stopReason: "fatal",
}),
]);
- expect(got.map((r) => r.code)).toEqual([
- "tool_failed:exec_command",
- "tool_failed:input_command",
- ]);
+ expect(got.map((r) => r.code)).toEqual(["tool_fatal:exec_command", "tool_fatal:input_command"]);
});
it("a command tool's timeout and a missing session manager are recorded (neither is an exit)", () => {
@@ -638,57 +678,60 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
toolCallOutput({
output: "still building…\n[tool timeout: exceeded 60000ms]",
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
call("input_command", "tc-2"),
toolCallOutput({
output: "[input_command unavailable: no command session manager configured]",
toolCallId: "tc-2",
- stopReason: "failed",
+ stopReason: "fatal",
}),
]);
- expect(got.map((r) => r.code)).toEqual([
- "tool_failed:exec_command",
- "tool_failed:input_command",
- ]);
+ expect(got.map((r) => r.code)).toEqual(["tool_fatal:exec_command", "tool_fatal:input_command"]);
});
- it("an exit marker with no cached tool name is dropped, not filed under tool_failed:unknown", () => {
+ it("an exit marker with no cached tool name is dropped, not filed under tool_fatal:unknown", () => {
// Only the command tools ever write that marker, so a cache miss (the tool_call evicted, or
// never seen) is still that noise — recording it nameless would defeat the exclusion.
const got = feed([
- toolCallOutput({ output: "[exit code: 1]", toolCallId: "tc-1", stopReason: "failed" }),
- toolCallOutput({ output: "boom", toolCallId: "tc-2", stopReason: "failed" }),
+ toolCallOutput({ output: "[exit code: 1]", toolCallId: "tc-1", stopReason: "fatal" }),
+ toolCallOutput({ output: "boom", toolCallId: "tc-2", stopReason: "fatal" }),
]);
- expect(got.map((r) => [r.code, r.message])).toEqual([["tool_failed:unknown", "boom"]]);
+ expect(got.map((r) => [r.code, r.message])).toEqual([["tool_fatal:unknown", "boom"]]);
});
- it("tool failed / timeout → environment + expected, code carries the tool name", () => {
+ it("a tool fatal → environment + expected, code carries the tool name; legacy failed/timeout spellings still record", () => {
+ const legacyOutput = (args: { output: string; toolCallId: string; stopReason: string }) =>
+ toolCallOutput(args as Parameters<typeof toolCallOutput>[0]);
const got = feed([
call("write_file", "tc-1"),
toolCallOutput({
output: "EACCES: permission denied\n[tool error] write failed",
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
+ // Traces written before the stop-reason unification spell tool failures failed/timeout.
call("read_file", "tc-2"),
- toolCallOutput({
+ legacyOutput({
output: "[tool timeout: exceeded 30000ms]",
toolCallId: "tc-2",
stopReason: "timeout",
}),
+ call("edit_file", "tc-3"),
+ legacyOutput({ output: "[tool error] boom", toolCallId: "tc-3", stopReason: "failed" }),
]);
- expect(got).toHaveLength(2);
+ expect(got).toHaveLength(3);
expect(got[0]).toMatchObject({
source: "environment",
kind: "expected", // error fed back to the model; the Agent adjusts on its own — no human needed
- code: "tool_failed:write_file",
+ code: "tool_fatal:write_file",
project_id: "p1",
agent_id: "a1",
session_id: "s1",
});
expect(got[0]!.message).toContain("[tool error] write failed"); // the actual error text
expect(got[1]).toMatchObject({ code: "tool_timeout:read_file", kind: "expected" });
+ expect(got[2]).toMatchObject({ code: "tool_failed:edit_file", kind: "expected" });
});
it("tool aborted (denial / user interrupt) and completed are not recorded", () => {
@@ -711,11 +754,11 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
call("write_file", "tc-1"),
call("read_file", "tc-2"),
call("write_file", "tc-3"),
- toolCallOutput({ output: "boom-2", toolCallId: "tc-2", stopReason: "failed" }),
+ toolCallOutput({ output: "boom-2", toolCallId: "tc-2", stopReason: "fatal" }),
toolCallOutput({ output: "ok", toolCallId: "tc-3", stopReason: "completed" }),
- toolCallOutput({ output: "boom-1", toolCallId: "tc-1", stopReason: "failed" }),
+ toolCallOutput({ output: "boom-1", toolCallId: "tc-1", stopReason: "fatal" }),
]);
- expect(got.map((r) => r.code)).toEqual(["tool_failed:read_file", "tool_failed:write_file"]);
+ expect(got.map((r) => r.code)).toEqual(["tool_fatal:read_file", "tool_fatal:write_file"]);
expect(got.map((r) => r.message)).toEqual(["boom-2", "boom-1"]);
});
@@ -724,18 +767,18 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
call("read_file", "tc-1"), // parent session
withOrigin(call("write_file", "tc-1"), "session-child"), // sub-session happens to share the same id
withOrigin(
- toolCallOutput({ output: "child boom", toolCallId: "tc-1", stopReason: "failed" }),
+ toolCallOutput({ output: "child boom", toolCallId: "tc-1", stopReason: "fatal" }),
"session-child",
),
- toolCallOutput({ output: "parent boom", toolCallId: "tc-1", stopReason: "failed" }),
+ toolCallOutput({ output: "parent boom", toolCallId: "tc-1", stopReason: "fatal" }),
]);
expect(got).toHaveLength(2);
expect(got[0]).toMatchObject({
- code: "tool_failed:write_file", // the sub-session's tool name, not overwritten by the parent's tc-1
+ code: "tool_fatal:write_file", // the sub-session's tool name, not overwritten by the parent's tc-1
message: "child boom",
session_id: "s1", // this test didn't feed the sub-session's session_meta → attribution falls back to the parent ctx (see the "attribution" test cases below)
});
- expect(got[1]).toMatchObject({ code: "tool_failed:read_file", message: "parent boom" });
+ expect(got[1]).toMatchObject({ code: "tool_fatal:read_file", message: "parent boom" });
});
it("overlong tool output: message takes the tail (the reason is at the end)", () => {
@@ -744,7 +787,7 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
toolCallOutput({
output: `${"x".repeat(2000)}\n[tool error] boom`,
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
}),
]);
const message = got[0]!.message as string;
@@ -758,7 +801,7 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
feed([
assistantText("normal output"),
call("write_file", "tc-1"),
- partialToolCallOutput({ eventType: "stop", toolCallId: "tc-1", stopReason: "failed" }),
+ partialToolCallOutput({ eventType: "stop", toolCallId: "tc-1", stopReason: "fatal" }),
]),
).toHaveLength(0);
});
@@ -769,8 +812,8 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
const got = feed([
childMeta("session-child", "/data/agents/agent-child/agent_state"),
withOrigin(requestBegin(), "session-child"),
- withOrigin(requestEnd("failed"), "session-child"),
- withOrigin(abortEvent("llm request error: 401 invalid api key"), "session-child"),
+ withOrigin(requestEnd("retryable"), "session-child"),
+ withOrigin(legacyAbort("llm request error: 401 invalid api key"), "session-child"),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({
@@ -788,14 +831,14 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
childMeta("session-child", "/data/agents/agent-child/agent_state"),
withOrigin(call("write_file", "tc-1"), "session-child"),
withOrigin(
- toolCallOutput({ output: "child boom", toolCallId: "tc-1", stopReason: "failed" }),
+ toolCallOutput({ output: "child boom", toolCallId: "tc-1", stopReason: "fatal" }),
"session-child",
),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({
source: "environment",
- code: "tool_failed:write_file",
+ code: "tool_fatal:write_file",
message: "child boom",
agent_id: "agent-child",
session_id: "session-child",
@@ -808,31 +851,31 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
requestBegin(), // parent session initiates
childMeta("session-child", "/data/agents/agent-child/agent_state"),
withOrigin(requestBegin(), "session-child"),
- withOrigin(requestEnd("timeout"), "session-child"),
- withOrigin(abortEvent("reconnect failed after 2 retries"), "session-child"),
+ withOrigin(requestEnd("fatal"), "session-child"),
+ withOrigin(legacyAbort("llm request error: 401 dead key"), "session-child"),
withOrigin(call("write_file", "tc-9"), "session-child"),
withOrigin(
- toolCallOutput({ output: "child tool boom", toolCallId: "tc-9", stopReason: "failed" }),
+ toolCallOutput({ output: "child tool boom", toolCallId: "tc-9", stopReason: "fatal" }),
"session-child",
),
call("read_file", "tc-9"), // parent session happens to share the same id
- toolCallOutput({ output: "parent tool boom", toolCallId: "tc-9", stopReason: "failed" }),
- requestEnd("failed"), // the parent session's LLM failure only wraps up now
- abortEvent("llm request error: 500 upstream"),
+ toolCallOutput({ output: "parent tool boom", toolCallId: "tc-9", stopReason: "fatal" }),
+ requestEnd("retryable"), // the parent session's LLM failure only wraps up now
+ legacyAbort("llm request error: 500 upstream"),
]);
// The sub-session's LLM / tool failures attribute to it, the parent's to the parent — the four entries never mix (each has a distinct code, so short-window dedup doesn't suppress any of them).
expect(got.map((r) => [r.code, r.agent_id, r.session_id])).toEqual([
- ["llm_timeout", "agent-child", "session-child"],
- ["tool_failed:write_file", "agent-child", "session-child"],
- ["tool_failed:read_file", "a1", "s1"],
+ ["llm_fatal", "agent-child", "session-child"],
+ ["tool_fatal:write_file", "agent-child", "session-child"],
+ ["tool_fatal:read_file", "a1", "s1"],
["llm_failed", "a1", "s1"],
]);
});
it("failure before session_meta arrives: falls back to the parent ctx, no crash", () => {
const got = feed([
- withOrigin(requestEnd("failed"), "session-child"), // the sub-session's meta hasn't arrived yet
- withOrigin(abortEvent("llm request error: 500"), "session-child"),
+ withOrigin(requestEnd("retryable"), "session-child"), // the sub-session's meta hasn't arrived yet
+ withOrigin(legacyAbort("llm request error: 500"), "session-child"),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({ code: "llm_failed", agent_id: "a1", session_id: "s1" });
@@ -841,8 +884,8 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
it("malformed agent_state path (empty): not registered, falls back to the parent ctx", () => {
const got = feed([
childMeta("session-child", ""), // path.basename(path.dirname("")) === "." → caught by the defensive check
- withOrigin(requestEnd("failed"), "session-child"),
- withOrigin(abortEvent("llm request error: 500"), "session-child"),
+ withOrigin(requestEnd("retryable"), "session-child"),
+ withOrigin(legacyAbort("llm request error: 500"), "session-child"),
]);
expect(got).toHaveLength(1);
expect(got[0]).toMatchObject({ code: "llm_failed", agent_id: "a1", session_id: "s1" });
@@ -850,15 +893,15 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
// —— Compaction ——
- it("a failed compaction records one error row; the message carries attempts and the error", () => {
+ it("an abandoned compaction records one error row; the message carries attempts and the error", () => {
// Issue #170: a compaction is an ordinary LLM request whose failures core retries under
- // the standard budget — a failed end means the retries ran out, and the cost center's
- // row shows how many attempts were spent and what the last failure was.
+ // the standard budget — a retryable end means the retries ran out, and the cost
+ // center's row shows how many attempts were spent and what the last failure was.
const got = feed([
compactionEnd({
reason: "context",
mode: "summarize",
- status: "failed",
+ status: "retryable",
attempt: 5,
errorMessage: "the response contained no usable summary",
}),
@@ -876,15 +919,21 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
});
});
- it("compaction completed / aborted are not errors; an old-core failed still records", () => {
+ it("compaction completed / aborted are not errors; fatal and an old-core failed still record", () => {
+ const legacyEnd = (status: string) =>
+ compactionEnd({ reason: "turns", mode: "summarize", status: status as StopReason });
const got = feed([
compactionEnd({ reason: "context", mode: "summarize", status: "completed", attempt: 1 }),
compactionEnd({ reason: "manual", mode: "summarize", status: "aborted", attempt: 2 }),
- // An old core's compaction_end has no attempt/error fields at all.
- compactionEnd({ reason: "turns", mode: "summarize", status: "failed" }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "fatal", attempt: 1 }),
]);
expect(got).toHaveLength(1);
- expect(got[0]).toMatchObject({
+ expect(got[0]).toMatchObject({ code: "compaction_failed" });
+ // An old core's compaction_end (retired `failed` spelling, no attempt/error fields)
+ // still records — its own feed, since the short-window dedup shares the code above.
+ const legacy = feed([legacyEnd("failed")]);
+ expect(legacy).toHaveLength(2); // rows are cumulative: the fatal row above plus this one
+ expect(legacy[1]).toMatchObject({
code: "compaction_failed",
message: "summarize compaction failed; trigger turns, original context kept.",
});
@@ -894,7 +943,7 @@ describe("stream-error-watcher (LLM / Environment errors)", () => {
const got = feed([
childMeta("session-child", "/data/agents/agent-child/agent_state"),
withOrigin(
- compactionEnd({ reason: "context", mode: "summarize", status: "failed", attempt: 6 }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable", attempt: 6 }),
"session-child",
),
]);
diff --git a/packages/server/test/messages-page.test.ts b/packages/server/test/messages-page.test.ts
index fe38adef..820ce899 100644
--- a/packages/server/test/messages-page.test.ts
+++ b/packages/server/test/messages-page.test.ts
@@ -223,7 +223,7 @@ describe("messages windowed reads", () => {
// ...process died; the resume closed the span before writing anything else:
at(
"2026-07-20T10:01:30.000Z",
- compactionEnd({ reason: "context", mode: "summarize", status: "failed" }),
+ compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }),
),
...turn(2, 2, 2000),
]);
@@ -259,7 +259,7 @@ describe("messages windowed reads", () => {
"2026-07-20T10:01:02.000Z",
compactionEnd({ reason: "context", mode: "summarize", status: "aborted" }),
),
- at("2026-07-20T10:01:03.000Z", abortEvent("aborted during compaction")),
+ at("2026-07-20T10:01:03.000Z", abortEvent("compaction_interrupted")),
...turn(2, 2, 2000),
]);
diff --git a/packages/server/test/model-probe.test.ts b/packages/server/test/model-probe.test.ts
index ae13b554..1d270b7c 100644
--- a/packages/server/test/model-probe.test.ts
+++ b/packages/server/test/model-probe.test.ts
@@ -21,14 +21,14 @@ describe("isProbeContent", () => {
it("counts thinking and text content, partial or complete", () => {
expect(isProbeContent(partialThinking("delta", "let me think"))).toBe(true);
expect(isProbeContent(partialText("delta", "pong"))).toBe(true);
- expect(isProbeContent(thinkingMessage("burned the whole budget", "malformed"))).toBe(true);
+ expect(isProbeContent(thinkingMessage("burned the whole budget", "retryable"))).toBe(true);
expect(isProbeContent(assistantText("pong"))).toBe(true);
});
it("ignores empty segments, events, and tool calls", () => {
expect(isProbeContent(partialThinking("start"))).toBe(false);
- expect(isProbeContent(partialThinking("stop", "", "malformed"))).toBe(false);
- expect(isProbeContent(thinkingMessage("", "malformed", { id: "rs_1" }))).toBe(false);
+ expect(isProbeContent(partialThinking("stop", "", "retryable"))).toBe(false);
+ expect(isProbeContent(thinkingMessage("", "retryable", { id: "rs_1" }))).toBe(false);
expect(isProbeContent(tokenUsage(emptyTokenCounts(), emptyTokenCounts()))).toBe(false);
expect(isProbeContent(toolCall({ name: "t", arguments: "{}", toolCallId: "tc1" }))).toBe(false);
});
@@ -39,22 +39,22 @@ describe("probeVerdict", () => {
expect(probeVerdict({ status: "completed" }, false)).toEqual({ ok: true });
});
- it("passes a thinking-only malformed ending when content was streamed (reasoning model hit the tiny cap)", () => {
+ it("passes a retryable ending when content was streamed (e.g. a reasoning model burned the tiny cap)", () => {
const outcome = {
- status: "malformed" as const,
- message: 'OpenaiClient returned no content other than thinking (finish_reason="length").',
+ status: "retryable" as const,
+ errorMessage:
+ 'OpenaiClient returned no content other than thinking (finish_reason="length").',
};
expect(probeVerdict(outcome, true)).toEqual({ ok: true });
});
- it("fails a malformed ending with nothing received (broken response, not a working endpoint)", () => {
- const verdict = probeVerdict({ status: "malformed", errorMessage: "unexpected EOF" }, false);
+ it("fails a retryable ending with nothing received (broken response, not a working endpoint)", () => {
+ const verdict = probeVerdict({ status: "retryable", errorMessage: "unexpected EOF" }, false);
expect(verdict).toEqual({ ok: false, message: "unexpected EOF" });
});
- it("fails timeouts and errors even when content was streamed (flaky is not ok)", () => {
- expect(probeVerdict({ status: "timeout" }, true)).toEqual({ ok: false, message: "timeout" });
- expect(probeVerdict({ status: "failed", errorMessage: "401 unauthorized" }, true)).toEqual({
+ it("fails fatal rejections and aborts even when content was streamed", () => {
+ expect(probeVerdict({ status: "fatal", errorMessage: "401 unauthorized" }, true)).toEqual({
ok: false,
message: "401 unauthorized",
});
@@ -62,7 +62,7 @@ describe("probeVerdict", () => {
});
it("truncates long failure messages to 300 characters", () => {
- const verdict = probeVerdict({ status: "failed", errorMessage: "x".repeat(500) }, false);
+ const verdict = probeVerdict({ status: "fatal", errorMessage: "x".repeat(500) }, false);
expect(verdict.ok).toBe(false);
if (!verdict.ok) expect(verdict.message).toHaveLength(300);
});
diff --git a/packages/server/test/session-manager.test.ts b/packages/server/test/session-manager.test.ts
index 6683cf26..b0631c2b 100644
--- a/packages/server/test/session-manager.test.ts
+++ b/packages/server/test/session-manager.test.ts
@@ -551,10 +551,15 @@ describe("session-manager", () => {
yield toolCallOutput({
output: "ls: /nope\n[tool error] exit code 2",
toolCallId: "tc-1",
- stopReason: "failed",
+ stopReason: "fatal",
+ });
+ // The ladder's terminal failure: no retry planned (no retry_in_ms), the run ends
+ // on this record — no abort event follows.
+ yield requestEnd("retryable", {
+ attempt: 6,
+ errorCode: "network",
+ errorMessage: "500 upstream",
});
- yield requestEnd("failed");
- yield abortEvent("llm request error: 500 upstream");
},
async *compact(): AsyncGenerator<OmniMessage> {},
};
@@ -563,12 +568,12 @@ describe("session-manager", () => {
await waitFor(() => manager.statusOf("session-1") === "idle" && captured.length >= 2);
expect(captured.map((a) => [a.source, a.code, a.kind])).toEqual([
- ["environment", "tool_failed:write_file", "expected"], // error fed back to the model; the Agent adjusts on its own
- ["llm", "llm_failed", "unexpected"], // the abort follows it: the retries did not recover it, so a human is needed
+ ["environment", "tool_fatal:write_file", "expected"], // error fed back to the model; the Agent adjusts on its own
+ ["llm", "llm_failed", "unexpected"], // nothing followed it: the retries did not recover it, so a human is needed
]);
expect(captured[0]!.ctx).toEqual({ projectId: "p1", agentId: "a1", sessionId: "session-1" });
expect(String(captured[0]!.err)).toContain("[tool error] exit code 2");
- expect(String(captured[1]!.err)).toBe("llm request error: 500 upstream"); // the abort's real reason
+ expect(String(captured[1]!.err)).toBe("llm request failed after 5 retries: 500 upstream");
});
it("mutual exclusion: startTask again while running → 409 task_in_progress; compact → 409", async () => {
diff --git a/packages/server/test/trace-service.test.ts b/packages/server/test/trace-service.test.ts
index 962900f5..35973dba 100644
--- a/packages/server/test/trace-service.test.ts
+++ b/packages/server/test/trace-service.test.ts
@@ -25,7 +25,16 @@ import {
userText,
withOrigin,
} from "@prismshadow/penguin-core";
-import type { OmniMessage, SessionMetaPayload, TokenCounts } from "@prismshadow/penguin-core";
+import type {
+ OmniMessage,
+ SessionMetaPayload,
+ TokenCounts,
+ StopReason,
+} from "@prismshadow/penguin-core";
+
+// Traces written before the stop-reason convergence carry the retired spellings; analysis
+// must keep reading them, so these fixtures write them through a cast.
+const legacyEnd = (status: string) => requestEnd(status as StopReason);
import type { TraceService } from "../src/services/trace-service.js";
import type { SessionRow } from "../src/db/repos/sessions.js";
import { SessionSources } from "../src/runtime/session-sources.js";
@@ -146,7 +155,7 @@ describe("trace-service", () => {
sessionMeta(metaPayload()),
at("2026-07-05T10:00:00.000Z", userText("hi")),
at("2026-07-05T10:00:01.000Z", requestBegin()),
- at("2026-07-05T10:00:03.000Z", requestEnd("timeout")), // reconnect +1
+ at("2026-07-05T10:00:03.000Z", requestEnd("retryable")), // reconnect +1
at("2026-07-05T10:00:03.500Z", requestBegin()),
at(
"2026-07-05T10:00:04.000Z",
@@ -169,7 +178,7 @@ describe("trace-service", () => {
const analysis = await service.analyze(P, A, S, 1);
expect(analysis.requests).toHaveLength(3);
- expect(analysis.requests[0]!.status).toBe("timeout");
+ expect(analysis.requests[0]!.status).toBe("retryable");
expect(analysis.requests[0]!.durationMs).toBe(2000);
expect(analysis.requests[1]!.status).toBe("completed");
expect(analysis.requests[1]!.durationMs).toBe(1500);
@@ -377,7 +386,7 @@ describe("trace-service", () => {
compactionBegin({ reason: "context", mode: "summarize", context: 1000, turns: 1 }),
),
at("2026-07-05T10:00:04.000Z", requestBegin()), // the compaction request
- at("2026-07-05T10:00:05.000Z", requestEnd("timeout")), // retries exhausted, ends in timeout
+ at("2026-07-05T10:00:05.000Z", requestEnd("retryable")), // retries exhausted
at(
"2026-07-05T10:00:06.000Z",
compactionEnd({ reason: "context", mode: "summarize", status: "aborted" }),
@@ -423,7 +432,7 @@ describe("trace-service", () => {
// Second turn: the Prompt precedes the Request; this turn's request fails outright, with no model output or tool call at all.
at("2026-07-05T10:01:00.000Z", userText("question two")),
at("2026-07-05T10:01:01.000Z", requestBegin()),
- at("2026-07-05T10:01:04.000Z", requestEnd("failed")),
+ at("2026-07-05T10:01:04.000Z", requestEnd("retryable")),
]);
const a = await service.analyze(P, A, S, 1);
@@ -446,16 +455,15 @@ describe("trace-service", () => {
expect(a.elapsedMs).toBe(4_100);
});
- it("a failed request the engine retries stays inside the same turn (it is a reconnect, not a turn end)", async () => {
- // The engine reconnects on `failed` as well as timeout/malformed, so the resent Request
- // belongs to the same user turn. If segmentation still keyed on timeout/malformed only,
- // one gateway blip would split a single turn's Tokens, duration and TPS across two Tasks
- // and inflate the Task count — the same smearing the timeout rule exists to prevent.
+ it("a legacy-trace `failed` request the engine retried stays inside the same turn (reconnect, not a turn end)", async () => {
+ // Traces from the pre-convergence era spell a retried request `failed`; segmentation
+ // must keep reading it as a reconnect, or one gateway blip would split a single turn's
+ // Tokens, duration and TPS across two Tasks and inflate the Task count.
await writeTraceFile(root, P, A, "2026-07-05", S, 1, [
at("2026-07-05T10:00:00.000Z", sessionMeta(metaPayload())),
at("2026-07-05T10:00:00.000Z", userText("question one")),
at("2026-07-05T10:00:01.000Z", requestBegin()),
- at("2026-07-05T10:00:02.000Z", requestEnd("failed")), // a gateway error → the engine reconnects
+ at("2026-07-05T10:00:02.000Z", legacyEnd("failed")), // a gateway error → that era's engine reconnected
at("2026-07-05T10:00:03.000Z", requestBegin()),
at("2026-07-05T10:00:05.000Z", requestEnd("completed")),
at("2026-07-05T10:00:05.100Z", tokenUsage(counts(1000), buckets(0, 900, 100))),
@@ -467,10 +475,10 @@ describe("trace-service", () => {
expect(a.tasks[0]!.tokens.output).toBe(100);
});
- it("a failed compaction request is NOT a reconnect: compaction fails fast on it", async () => {
- // Segmentation has to track the engine loop by loop: the turn loop retries `failed`, the
- // compaction loop deliberately does not (it keeps the old context and tries again on the
- // next trigger). Counting this as a reconnect would invent an attempt that never happened.
+ it("a legacy-trace `failed` compaction request is NOT a reconnect: that era failed fast on it", async () => {
+ // Legacy segmentation nuance: the pre-convergence turn loop retried `failed`, its
+ // compaction loop deliberately did not. Counting the give-up as a reconnect would
+ // invent an attempt that never happened in that Trace.
await writeTraceFile(root, P, A, "2026-07-05", S, 1, [
at("2026-07-05T10:00:00.000Z", sessionMeta(metaPayload())),
at("2026-07-05T10:00:00.000Z", userText("question one")),
@@ -482,18 +490,19 @@ describe("trace-service", () => {
compactionBegin({ reason: "context", mode: "summarize", context: 1000, turns: 1 }),
),
at("2026-07-05T10:00:04.000Z", requestBegin()),
- at("2026-07-05T10:00:05.000Z", requestEnd("failed")), // compaction gives up here
+ at("2026-07-05T10:00:05.000Z", legacyEnd("failed")), // compaction gives up here
at(
"2026-07-05T10:00:06.000Z",
- compactionEnd({ reason: "context", mode: "summarize", status: "failed" }),
+ // Legacy Trace spelling for an abandoned compaction (see legacyEnd).
+ compactionEnd({ reason: "context", mode: "summarize", status: "failed" as StopReason }),
),
]);
const a = await service.analyze(P, A, S, 1);
expect(a.reconnectCount).toBe(0);
});
- it("after the previous turn ends in timeout (retries exhausted), a new user message starts a new turn", async () => {
- // "timeout → continuation" holds only for **automatic retries within the
+ it("after the previous turn's retries are exhausted, a new user message starts a new turn", async () => {
+ // "retryable → continuation" holds only for **automatic retries within the
// same run**. Once retries are exhausted and the engine gives up, a message
// the user sends afterward starts a new turn — if continuation were still
// stuck at true, this turn would get folded into the failed one, mixing
@@ -503,7 +512,7 @@ describe("trace-service", () => {
at("2026-07-05T10:00:00.000Z", sessionMeta(metaPayload())),
at("2026-07-05T10:00:00.000Z", userText("question one")),
at("2026-07-05T10:00:01.000Z", requestBegin()),
- at("2026-07-05T10:00:02.000Z", requestEnd("timeout")), // retries exhausted → gives up
+ at("2026-07-05T10:00:02.000Z", requestEnd("retryable")), // retries exhausted → gives up
at("2026-07-05T10:00:03.000Z", abortEvent()),
// A new user send
at("2026-07-05T10:01:00.000Z", userText("question two")),
@@ -1117,8 +1126,8 @@ describe("trace-service", () => {
),
at(T("04.500"), requestBegin()),
at(T("05.000"), assistantText("bad summary")),
- at(T("05.500"), requestEnd("failed")),
- at(T("06.000"), compactionEnd({ reason: "context", mode: "summarize", status: "failed" })),
+ at(T("05.500"), requestEnd("retryable")),
+ at(T("06.000"), compactionEnd({ reason: "context", mode: "summarize", status: "retryable" })),
at(T("07.000"), requestBegin()),
at(T("08.000"), assistantText("answer")),
at(T("08.500"), requestEnd("completed")),
@@ -1140,10 +1149,10 @@ describe("trace-service", () => {
name: "exec_command",
arguments: "{}",
toolCallId: "t1",
- stopReason: "timeout",
+ stopReason: "retryable",
}),
),
- at(T("02.500"), requestEnd("timeout")),
+ at(T("02.500"), requestEnd("retryable")),
at(T("03.000"), requestBegin()),
at(T("04.000"), toolCall({ name: "read_file", arguments: "{}", toolCallId: "t2" })),
at(T("04.500"), toolCallOutput({ output: "ok", toolCallId: "t2" })),
@@ -1153,6 +1162,6 @@ describe("trace-service", () => {
// A phantom call gets no lane: only t2 goes into toolSpans.
expect(a.toolSpans.map((sp) => sp.toolCallId)).toEqual(["t2"]);
// But the duration list still records t1 (flagged with the interrupted status).
- expect(a.toolCalls.find((c) => c.toolCallId === "t1")?.stopReason).toBe("timeout");
+ expect(a.toolCalls.find((c) => c.toolCallId === "t1")?.stopReason).toBe("retryable");
});
});
diff --git a/packages/server/test/vision-detect.test.ts b/packages/server/test/vision-detect.test.ts
index ffcfa03b..03eed9bc 100644
--- a/packages/server/test/vision-detect.test.ts
+++ b/packages/server/test/vision-detect.test.ts
@@ -17,7 +17,7 @@ import {
isImageRejection,
} from "../src/services/vision-detect.js";
-const failed = (errorMessage: string): LLMOutcome => ({ status: "failed", errorMessage });
+const failed = (errorMessage: string): LLMOutcome => ({ status: "fatal", errorMessage });
describe("VISION_PROBE_IMAGE", () => {
it("is a tiny inline PNG data URL, so the probe sends the least it can", () => {
diff --git a/packages/web/src/features/chat/chat-input.tsx b/packages/web/src/features/chat/chat-input.tsx
index cb7b7b72..e5dc3beb 100644
--- a/packages/web/src/features/chat/chat-input.tsx
+++ b/packages/web/src/features/chat/chat-input.tsx
@@ -798,10 +798,6 @@ export function ChatInput({
onHandoffTargetChange,
initialPendingModelRef,
onPendingModelChange,
- modelAuthDead = false,
- onOpenModels,
- onRetryModelAuth,
- onNewSession,
controlRef,
variant = "session",
}: {
@@ -984,18 +980,6 @@ export function ChatInput({
initialPendingModelRef?: ModelRefDto;
/** Callback when the staged model switch changes (picked/removed; the clear after a successful send does not call back, same as onTextChange). */
onPendingModelChange?: (ref: ModelRefDto | null) => void;
- /**
- * Session state: the Session's model credentials failed authentication (abort with
- * status "auth") and the Project's credentials have not been updated since (the parent
- * computes the time gate — see isModelAuthDead). Recoverable: only the model reference
- * is fixed at creation, credentials come from the current Project config — so the
- * composer disables and grays itself with a notice pointing at the Models page (primary),
- * a Retry affordance, and a New Session escape. Updating the key auto-unlocks (live via
- * the credentials_updated event, and across reloads via the time gate).
- */
- modelAuthDead?: boolean;
- /** Navigates to the Models page (where the credential is actually fixed); renders the notice's primary button when supplied. */
- onOpenModels?: () => void;
/**
* Which surface this composer serves. `"session"` (default) is the main conversation with
* every affordance. `"subagent"` drives one subagent child from the panel: the same body,
@@ -1006,14 +990,6 @@ export function ChatInput({
* model or agent, so there is no locked-model hint to click for.
*/
variant?: "session" | "subagent";
- /**
- * Clears the auth-dead state and re-enables the composer for another attempt (the state
- * re-arms on the next auth failure). The escape hatch for credential changes the
- * timestamps miss (e.g. edited outside the UI).
- */
- onRetryModelAuth?: () => void;
- /** Navigates to a fresh draft (`/chat/new`); renders the notice's "New Session" button when supplied. */
- onNewSession?: () => void;
/**
* Handle for the one thing a parent reaches in and does: the draft screen's example cards
* fill this composer (see ComposerControl). A ref rather than a `fill` prop because it is a
@@ -1124,7 +1100,6 @@ export function ChatInput({
!running &&
!compacting &&
!busy &&
- !modelAuthDead &&
(goalOn
? text.trim().length > 0 &&
attachments.length === 0 &&
@@ -1228,7 +1203,6 @@ export function ChatInput({
const midRun = midRunAction({
sending: busy,
goalOn,
- modelAuthDead,
canSteerChannel: onSteer !== undefined,
canQueueChannel: onQueueFollowUp !== undefined,
followUpMode,
@@ -2238,62 +2212,13 @@ export function ChatInput({
)
)}
- {/* Auth-dead session (model credentials failed): slim notice above the composer, same
- width — the abort line in the stream keeps the raw reason, this adds the "what now"
- guidance. Recoverable: primary CTA opens the Models page (fixing the key there
- auto-unlocks the session), Retry clears the state for another attempt (re-arms on
- the next auth failure), New Session stays as the escape. The composer below is
- disabled and hazed. */}
- {modelAuthDead && (
- <div className="anim-fade mb-1.5 flex flex-wrap items-center justify-between gap-x-3 gap-y-1.5 rounded-md border border-gray-200 bg-gray-50 px-3 py-2 text-xs text-gray-600 dark:border-gray-700 dark:bg-gray-800/60 dark:text-gray-300">
- <span className="min-w-0 flex-1 basis-56">{S.chat.modelAuthDead}</span>
- <span className="flex shrink-0 flex-wrap items-center gap-1.5">
- {onOpenModels && (
- <button
- type="button"
- onClick={onOpenModels}
- className="shrink-0 rounded-md bg-gray-900 px-2.5 py-1 font-medium text-white transition-colors duration-150 hover:bg-gray-700 dark:bg-gray-100 dark:text-gray-900 dark:hover:bg-gray-300"
- >
- {S.chat.modelAuthDeadOpenModels}
- </button>
- )}
- {onRetryModelAuth && (
- <button
- type="button"
- onClick={onRetryModelAuth}
- className="shrink-0 rounded-md border border-gray-300 bg-white px-2.5 py-1 font-medium text-gray-700 transition-colors duration-150 hover:bg-gray-100 dark:border-gray-600 dark:bg-gray-900 dark:text-gray-200 dark:hover:bg-gray-700"
- >
- {S.chat.modelAuthDeadRetry}
- </button>
- )}
- {onNewSession && (
- <button
- type="button"
- onClick={onNewSession}
- className="shrink-0 rounded-md border border-gray-300 bg-white px-2.5 py-1 font-medium text-gray-700 transition-colors duration-150 hover:bg-gray-100 dark:border-gray-600 dark:bg-gray-900 dark:text-gray-200 dark:hover:bg-gray-700"
- >
- {S.chat.modelAuthDeadCta}
- </button>
- )}
- </span>
- </div>
- )}
-
{/* Unified input card: the multi-line text body occupies the top area, with all controls
collected onto a single bottom row that never shares a line with the text.
@container: the bottom toolbar row collapses based on the **card's actual width**
(help text/button text visibility uses @md/@lg container breakpoints) — because the
card's width changes with the viewport and the Files panel squeezing it, viewport
breakpoints wouldn't judge it accurately. */}
- <div
- // Auth-dead haze deliberately keeps pointer events and text selection: the
- // textarea's `disabled` already blocks editing, and the stuck draft must stay
- // selectable/copyable — a long message that failed to send is exactly what the
- // user wants to copy back out.
- className={`@container rounded-lg border border-gray-300 bg-white px-2.5 pb-2 pt-2 transition-[border-color,box-shadow] duration-200 focus-within:border-gray-500 focus-within:ring-2 focus-within:ring-gray-400/30 dark:border-gray-700 dark:bg-gray-900 dark:focus-within:border-gray-400${
- modelAuthDead ? " opacity-50 grayscale" : ""
- }`}
- >
+ <div className="@container rounded-lg border border-gray-300 bg-white px-2.5 pb-2 pt-2 transition-[border-color,box-shadow] duration-200 focus-within:border-gray-500 focus-within:ring-2 focus-within:ring-gray-400/30 dark:border-gray-700 dark:bg-gray-900 dark:focus-within:border-gray-400">
{/* Chip row above the text body: the staged switch target (an /agent handoff or a
/model fork — never both) followed by the selected skills, all sharing the same
chip look. Remove buttons recolor the x on hover (no background wash). */}
@@ -2504,21 +2429,18 @@ export function ChatInput({
onSelect={(e) => setCaret(e.currentTarget.selectionStart ?? 0)}
onKeyDown={onKeyDown}
onPaste={onPaste}
- disabled={modelAuthDead}
placeholder={
- modelAuthDead
- ? S.chat.modelAuthDeadPlaceholder
- : running && followUpMode
+ running && followUpMode
+ ? narrow
+ ? S.chat.followUpPlaceholderShort
+ : S.chat.followUpPlaceholder
+ : running && onSteer
? narrow
- ? S.chat.followUpPlaceholderShort
- : S.chat.followUpPlaceholder
- : running && onSteer
- ? narrow
- ? S.chat.steerPlaceholderShort
- : S.chat.steerPlaceholder
- : narrow
- ? S.chat.inputPlaceholderShort
- : S.chat.inputPlaceholder
+ ? S.chat.steerPlaceholderShort
+ : S.chat.steerPlaceholder
+ : narrow
+ ? S.chat.inputPlaceholderShort
+ : S.chat.inputPlaceholder
}
className="block max-h-44 min-h-[60px] w-full resize-none bg-transparent px-1 py-0.5 text-base leading-6 placeholder:text-gray-400 focus:outline-none disabled:cursor-not-allowed disabled:opacity-60 dark:placeholder:text-gray-500"
/>
diff --git a/packages/web/src/features/chat/chat-page.tsx b/packages/web/src/features/chat/chat-page.tsx
index a0fdf601..a149d40a 100644
--- a/packages/web/src/features/chat/chat-page.tsx
+++ b/packages/web/src/features/chat/chat-page.tsx
@@ -48,7 +48,6 @@ import {
approvalKey,
createStreamModel,
finalizeHistory,
- isModelAuthDead,
pushMessage,
} from "../../lib/omni/stream-model";
import type { StreamModel } from "../../lib/omni/stream-model";
@@ -1347,9 +1346,6 @@ export function ChatPage() {
}, [user, projectId, navigate]);
// Auth-dead notice primary CTA: the Models page is where the credential is actually fixed.
- const openModels = useCallback(() => {
- navigate("/models");
- }, [navigate]);
const onFork = useCallback(
async (target: ForkTarget): Promise<void> => {
@@ -1576,17 +1572,6 @@ export function ChatPage() {
const emptyChat =
selected !== null && !stream.loading && !stream.error && stream.model.items.length === 0;
- // Auth-dead gate (recoverable): an auth failure is on record AND the Project's credentials
- // have not been updated since — only the model reference is fixed at creation, credentials
- // come from the current Project config, so a key update (Models page) unlocks the session
- // (live via the credentials_updated event; across reloads via this time comparison against
- // the models response's updatedAt).
- const credsUpdatedMs = models?.updatedAt !== undefined ? Date.parse(models.updatedAt) : NaN;
- const modelAuthDead = isModelAuthDead(
- stream.lastAuthFailureMs,
- Number.isFinite(credsUpdatedMs) ? credsUpdatedMs : null,
- );
-
// Input area in session state: Agent / Workspace / Model are already locked by the Session
// (the model selector isn't rendered; models feeds the locked model's read-only display and
// the /model switch picker) — approval mode and the per-turn thinking level stay editable;
@@ -1594,10 +1579,6 @@ export function ChatPage() {
const input = selected && (
<ChatInput
status={stream.taskState}
- modelAuthDead={modelAuthDead}
- onOpenModels={openModels}
- onRetryModelAuth={stream.dismissModelAuthDead}
- onNewSession={newChat}
onSend={onSend}
onSteer={onSteer}
// Count of steering messages already visible in the stream: the input area keeps its
diff --git a/packages/web/src/features/chat/composer-send.ts b/packages/web/src/features/chat/composer-send.ts
index c80b3927..24c123b3 100644
--- a/packages/web/src/features/chat/composer-send.ts
+++ b/packages/web/src/features/chat/composer-send.ts
@@ -28,8 +28,6 @@ export interface MidRunComposerState {
sending: boolean;
/** The goal chip is engaged: the body is an objective, which no mid-run channel carries. */
goalOn: boolean;
- /** The model API rejected this Session's credentials — nothing can be sent at all. */
- modelAuthDead: boolean;
/** The host wired a steer channel (`onSteer`); the draft page does not. */
canSteerChannel: boolean;
/** The host wired a follow-up queue (`onQueueFollowUp`); the draft page does not. */
@@ -66,9 +64,9 @@ export interface MidRunComposerState {
*/
export function midRunAction(s: MidRunComposerState): MidRunAction {
if (s.sending) return "disabled";
- // A goal draft and a dead key close both channels; a blocked `/model` fork closes the queue
+ // A goal draft closes both channels; a blocked `/model` fork closes the queue
// (see stagedSendRoute) and cannot reach steering anyway, since a staged chip rules it out.
- const open = !s.goalOn && !s.modelAuthDead;
+ const open = !s.goalOn;
const canSteer =
open &&
s.canSteerChannel &&
diff --git a/packages/web/src/features/chat/mcp-connect-banner.tsx b/packages/web/src/features/chat/mcp-connect-banner.tsx
index bddfc6aa..ce5f22fa 100644
--- a/packages/web/src/features/chat/mcp-connect-banner.tsx
+++ b/packages/web/src/features/chat/mcp-connect-banner.tsx
@@ -34,7 +34,7 @@ function serverOf(toolName: string, servers: string[]): string | null {
/** One server's group: a collapsible row; the body is its tool list, or the failure detail. */
function ServerGroup({ outcome, tools }: { outcome: McpServerOutcome; tools: McpToolSummary[] }) {
const [open, setOpen] = useState(false);
- const failed = outcome.status === "failed";
+ const failed = outcome.status === "fatal" || (outcome.status as string) === "failed";
// A failed server expands into its error; a connected one into its tools (when any).
const expandable = failed ? outcome.error !== undefined : tools.length > 0;
const meta = failed ? S.chat.mcpServerFailed : S.chat.mcpToolsCount(outcome.tools ?? 0);
diff --git a/packages/web/src/features/chat/message-item.tsx b/packages/web/src/features/chat/message-item.tsx
index 1cdc90f3..8b6a25b2 100644
--- a/packages/web/src/features/chat/message-item.tsx
+++ b/packages/web/src/features/chat/message-item.tsx
@@ -128,7 +128,16 @@ function ReconnectLine({ item, ctx }: { item: ReconnectItem; ctx: StreamRenderCo
<p
className={`anim-msg my-1 flex flex-wrap items-center gap-x-2 gap-y-1 font-mono text-xs ${toneInk.attention}`}
>
- <span>{S.chat.reconnect(item.status, state, item.attempt, seconds)}</span>
+ <span>
+ {S.chat.reconnect(
+ item.status,
+ state,
+ item.attempt,
+ seconds,
+ item.errorMessage,
+ item.errorCode,
+ )}
+ </span>
{showControls && (
<span className="flex shrink-0 items-center gap-1.5">
{ctx.onRetryNow && (
@@ -379,7 +388,13 @@ export function MessageItem({ item, ctx }: { item: ChatItem; ctx: StreamRenderCo
case "abort":
return (
<p className="anim-msg my-1 font-mono text-xs text-gray-500 dark:text-gray-400">
- {S.chat.aborted(item.reason)}
+ {S.chat.aborted(item)}
+ </p>
+ );
+ case "llm_error":
+ return (
+ <p className="anim-msg my-1 font-mono text-xs text-gray-500 dark:text-gray-400">
+ {S.chat.llmError(item.errorMessage)}
</p>
);
case "reconnect":
diff --git a/packages/web/src/features/chat/use-session-stream.ts b/packages/web/src/features/chat/use-session-stream.ts
index 9fa40d8c..9b7dcbf8 100644
--- a/packages/web/src/features/chat/use-session-stream.ts
+++ b/packages/web/src/features/chat/use-session-stream.ts
@@ -73,21 +73,6 @@ export interface SessionStreamState {
* runtimes and on old servers — the topology then falls back to its text heuristics.
*/
subagents: SubagentRuntimeInfo[];
- /**
- * Timestamp (ms) of the last main-session auth failure (request_end with status "auth"),
- * or null. Derived from the model, so it survives history replay and resets when
- * switching sessions; cleared by a later completed request, a `credentials_updated`
- * server event, or dismissModelAuthDead. The composer gates on it TOGETHER with the
- * models response's `updatedAt` (see isModelAuthDead): an abort older than the last
- * credential update no longer disables the composer.
- */
- lastAuthFailureMs: number | null;
- /**
- * Clears the auth-dead state (the user clicked Retry / dismissed the notice — e.g. the
- * credential changed outside the UI in a way the timestamps miss); the state re-arms if
- * the next request aborts on auth again.
- */
- dismissModelAuthDead: () => void;
/** approvalKey(origin, toolCallId) → pending approval. */
pendingApprovals: ReadonlyMap<string, PendingApproval>;
/** Recorded when this client clicks an approval decision (marks it as "manual"). */
@@ -313,14 +298,6 @@ export function useSessionStream(
void controllerRef.current?.loadOlder();
}, []);
- const dismissModelAuthDead = useCallback(() => {
- const m = controllerRef.current?.model;
- if (m && m.lastAuthFailureMs !== null) {
- m.lastAuthFailureMs = null;
- setVersion((v) => v + 1);
- }
- }, []);
-
// pendingTick participates in the render dependencies, ensuring pending-table changes trigger a re-render.
void pendingTick;
@@ -338,8 +315,6 @@ export function useSessionStream(
pendingSteering,
pendingFollowUps,
subagents,
- lastAuthFailureMs: (controllerRef.current?.model ?? placeholderRef.current).lastAuthFailureMs,
- dismissModelAuthDead,
pendingApprovals: controllerRef.current?.pendingApprovals ?? EMPTY_PENDING,
markLocalDecision,
resolveApproval,
diff --git a/packages/web/src/features/traces/trace-event-row.tsx b/packages/web/src/features/traces/trace-event-row.tsx
index 3737a399..1180066b 100644
--- a/packages/web/src/features/traces/trace-event-row.tsx
+++ b/packages/web/src/features/traces/trace-event-row.tsx
@@ -118,8 +118,13 @@ export function summarizeEvent(msg: OmniMessage): string {
}
case "tool_list_ready":
return Array.isArray(p["tools"]) ? S.traces.toolDefs(p["tools"].length) : "";
- case "abort":
- return p["reason"] != null ? String(p["reason"]) : "";
+ case "abort": {
+ // Live events carry the unified pair; legacy Traces spell the cause as `reason` prose.
+ const code = p["error_code"] != null ? String(p["error_code"]) : "";
+ const detail = p["error_message"] != null ? String(p["error_message"]) : "";
+ const legacy = p["reason"] != null ? String(p["reason"]) : "";
+ return [code || legacy, detail].filter(Boolean).join(" · ");
+ }
case "goal_finished":
return `${String(p["outcome"])} · rounds=${String(p["rounds"])} · tokens=${String(p["tokens_used"])}`;
case "subagent":
diff --git a/packages/web/src/lib/omni/stream-controller.ts b/packages/web/src/lib/omni/stream-controller.ts
index 171b994f..4ceae07f 100644
--- a/packages/web/src/lib/omni/stream-controller.ts
+++ b/packages/web/src/lib/omni/stream-controller.ts
@@ -329,16 +329,11 @@ export function createStreamController(deps: StreamControllerDeps): StreamContro
void rebuild();
return;
}
- case "credentials_updated": {
- // The Project's model credentials changed (Models page save): the server has
- // already invalidated cached runtimes, so any auth-dead state is stale — clear it
- // and let the user simply continue (it re-arms if the new key fails too). Buffered
- // during history load like other model-affecting events, so it replays AFTER the
- // history that set the flag.
- model.lastAuthFailureMs = null;
- deps.onModelChange();
+ case "credentials_updated":
+ // The Project's model credentials changed (Models page save): the server already
+ // invalidated cached runtimes, so the next task simply runs with the new key —
+ // nothing to update client-side.
return;
- }
case "hello":
return;
}
diff --git a/packages/web/src/lib/omni/stream-model.ts b/packages/web/src/lib/omni/stream-model.ts
index f053f2bd..6c747285 100644
--- a/packages/web/src/lib/omni/stream-model.ts
+++ b/packages/web/src/lib/omni/stream-model.ts
@@ -23,7 +23,7 @@
* - Events: approval_decision annotates the corresponding tool card
* (labeled "manual" if clicked on this end, "automatic" otherwise);
* abort → an interruption marker item; request_end ending in any status
- * the engine reconnects on (failed/timeout/malformed) → a retry-hint item (the engine discards that
+ * the engine reconnects on (retryable) → a retry-hint item (the engine discards that
* attempt and resends the original input; the next request_begin marks the hint as resent, and an
* arriving abort marks it as retries exhausted); other request_begin/end
* events aren't rendered (Request duration is covered by Trace
@@ -61,7 +61,6 @@ import type {
CompactionReason,
CompleteModelPayload,
EventPayload,
- McpConnectStatus,
OmniMessage,
PartialModelPayload,
SessionMetaPayload,
@@ -227,32 +226,52 @@ export interface SubagentItem {
export interface AbortItem {
kind: "abort";
id: number;
+ /** Machine-readable cause (the omnimessage ErrorCode vocabulary); the banner localizes from it. */
+ errorCode?: string;
+ /** Raw failure detail, shown verbatim (absent on live user interrupts). */
+ errorMessage?: string;
+ /** Legacy Traces only: the cause as English prose, rendered verbatim when no errorCode is present. */
reason?: string;
}
/**
- * The statuses the engine reconnects on — every LLM failure except `auth`, which is terminal
- * (see core's TURN_RETRY_STATUSES). A retry the user cannot see is a stalled session with no
- * explanation and no way out, so all three render the same countdown and the same controls.
+ * A run-ending LLM failure, from a `request_end` with status `fatal` (an LLM failure
+ * produces no abort event — the request_end is its terminal record).
*/
-export type ReconnectStatus = "failed" | "timeout" | "malformed";
+export interface LlmErrorItem {
+ kind: "llm_error";
+ id: number;
+ /** Machine-readable cause (request_end.error_code). */
+ errorCode?: string;
+ /** The provider's own error text (request_end.error_message), shown verbatim. */
+ errorMessage?: string;
+}
+
+/** `retryable` is the live protocol; failed/timeout/malformed are legacy Trace spellings kept for replay. */
+export type ReconnectStatus = "retryable" | "failed" | "timeout" | "malformed";
-function isReconnectStatus(status: StopReason | undefined): status is ReconnectStatus {
- return status === "failed" || status === "timeout" || status === "malformed";
+function isReconnectStatus(status: string | undefined): status is ReconnectStatus {
+ return (
+ status === "retryable" || status === "failed" || status === "timeout" || status === "malformed"
+ );
}
-/** An LLM Request ending in failed/timeout/malformed → the engine retries carrying the content already produced. */
+/** An LLM Request ending retryable → the engine retries carrying the content already produced. */
export interface ReconnectItem {
kind: "reconnect";
id: number;
- /** Trigger reason: timeout (timed out / disconnected), malformed (an incomplete or unparseable response), or failed (the provider returned an error). */
+ /** Trigger status (legacy Traces carry the finer pre-convergence spellings). */
status: ReconnectStatus;
/** Which retry attempt this is — request_end.attempt, the core's authoritative 1-based ordinal (1 for Traces written before the field existed). */
attempt: number;
/** The retry request has been sent (set true by the next request_begin). */
retrying: boolean;
- /** Retries exhausted (set true when an abort event arrives; the subsequent interruption marker item gives the reason). */
+ /** Retries exhausted: set at creation when the request_end announced no retry (`retryable` with no retry_in_ms — the run ends there), or by an abort event in legacy Traces. */
gaveUp?: boolean;
+ /** Machine-readable cause (request_end.error_code); the cause wording localizes from it. */
+ errorCode?: string;
+ /** The final failure's detail (request_end.error_message), rendered in the gave-up state. */
+ errorMessage?: string;
/**
* The engine's planned wait before the next attempt (request_end.retry_in_ms; absent
* when the event carried none — old Traces, or a final failure). Waits can reach the
@@ -302,7 +321,8 @@ export interface McpToolSummary {
/** One server's connect outcome (from the end payload), backing the row's per-server groups. */
export interface McpServerOutcome {
server: string;
- status: McpConnectStatus;
+ /** `completed` / `fatal`; legacy Traces spell the failure `failed`. */
+ status: StopReason;
/** That server's own connect + discovery time (the payload's duration_ms). */
durationMs: number;
/** Discovered tool count (present on a completed connect). */
@@ -371,6 +391,7 @@ export type ChatItem =
| ToolCallItem
| SubagentItem
| AbortItem
+ | LlmErrorItem
| ReconnectItem
| CompactionItem
| McpConnectItem
@@ -467,20 +488,6 @@ export interface StreamModel {
* dispatches it via `void executeOne`, which doesn't block the streaming loop — execution happens between two Requests).
*/
openApprovalWaitMs: number;
- /**
- * Timestamp (ms) of the most recent auth failure: a main-session `request_end` with
- * status "auth" arrived on THIS model (a subagent's request events route to the nested
- * model and never mark the parent). Only the model REFERENCE is fixed at Session
- * creation — credentials are read from the current Project config on load — so this is
- * recoverable: the composer disables itself and points at the Models page. Cleared by a
- * later completed main-session request_end (live and history replay take the same path,
- * so a Trace with an auth failure followed by a completed request does not resurrect the
- * state on reload), by a `credentials_updated` server event, or by an explicit user
- * retry/dismiss; a timestamp (not a boolean) so the composer gate can compare it against
- * the Project's credentials-updated time after a reload (see isModelAuthDead). Re-arms
- * on the next auth failure.
- */
- lastAuthFailureMs: number | null;
/** Task segmentation state. */
taskOpen: boolean;
/**
@@ -566,7 +573,6 @@ function newModel(nested: boolean, localDecisions: Set<string>): StreamModel {
lastTsMs: 0,
openRequestBeginMs: null,
openApprovalWaitMs: 0,
- lastAuthFailureMs: null,
taskOpen: false,
taskStartLocalMs: 0,
taskFirstTsMs: 0,
@@ -583,23 +589,6 @@ export function createStreamModel(localDecisions: Set<string> = new Set()): Stre
return newModel(false, localDecisions);
}
-/**
- * Composer auth-dead gate: an auth failure is on record AND the Project's credentials have
- * not been updated since. `credentialsUpdatedMs` is the models response's `updatedAt`
- * (null = unknown / not loaded — nothing proves the credential changed, so the notice
- * stays). The time comparison is what keeps a reload honest: after a key fix the Trace
- * still ends with the auth-failed request, but that failure predates the credential update, so the
- * composer stays alive; a wrong replacement key re-arms naturally because its own auth
- * abort is newer than the update.
- */
-export function isModelAuthDead(
- lastAuthFailureMs: number | null,
- credentialsUpdatedMs: number | null,
-): boolean {
- if (lastAuthFailureMs === null) return false;
- return credentialsUpdatedMs === null || lastAuthFailureMs > credentialsUpdatedMs;
-}
-
function nextId(model: StreamModel): number {
return model.nextItemId++;
}
@@ -1414,10 +1403,10 @@ function closeExecutingToolCards(model: StreamModel): void {
}
/**
- * A tool_call that closed with a non-completed status (produced by a
- * timeout/malformed interrupt closure) was never dispatched for execution
- * and will never get a tool_call_output: settle the card by its closing
- * reason as soon as it arrives, so the execution timer doesn't keep spinning forever.
+ * A tool_call that closed with a non-completed status (produced by an interrupt closure)
+ * was never dispatched for execution and will never get a tool_call_output: settle the
+ * card by its closing reason as soon as it arrives, so the execution timer doesn't keep
+ * spinning forever.
*/
function settleUndispatchedCall(card: ToolCallItem): void {
if (!card.callStopReason || card.callStopReason === "completed" || card.outputComplete) return;
@@ -1539,6 +1528,8 @@ function handleEvent(model: StreamModel, p: EventPayload, tsMs?: number, nowMs?:
const waiting = findLastWaitingReconnect(model);
if (waiting) waiting.gaveUp = true;
const item: AbortItem = { kind: "abort", id: nextId(model) };
+ if (typeof p.error_code === "string") item.errorCode = p.error_code;
+ if (typeof p.error_message === "string") item.errorMessage = p.error_message;
if (p.reason != null) item.reason = p.reason;
model.items.push(item);
return;
@@ -1570,14 +1561,20 @@ function handleEvent(model: StreamModel, p: EventPayload, tsMs?: number, nowMs?:
item.toolCount = p.results.reduce((sum, r) => sum + (r.tools ?? 0), 0);
// Per-server outcomes back the row's server groups (error details live there, not
// on the header line).
- item.results = p.results.map((r) => ({
- server: r.server,
- status: r.status,
- durationMs: r.duration_ms,
- ...(r.tools !== undefined ? { tools: r.tools } : {}),
- ...(r.error !== undefined ? { error: r.error.slice(0, 500) } : {}),
- }));
- const failedResults = p.results.filter((r) => r.status === "failed");
+ item.results = p.results.map((r) => {
+ // Live results carry the unified pair; legacy Traces spell the detail `error`.
+ const detail = r.error_message ?? (r as { error?: string }).error;
+ return {
+ server: r.server,
+ status: r.status,
+ durationMs: r.duration_ms,
+ ...(r.tools !== undefined ? { tools: r.tools } : {}),
+ ...(detail !== undefined ? { error: detail.slice(0, 500) } : {}),
+ };
+ });
+ const failedResults = p.results.filter(
+ (r) => r.status === "fatal" || (r.status as string) === "failed",
+ );
if (failedResults.length > 0) item.failed = failedResults.map((r) => r.server);
return;
}
@@ -1661,25 +1658,13 @@ function handleEvent(model: StreamModel, p: EventPayload, tsMs?: number, nowMs?:
return;
}
case "request_end": {
- // failed/timeout/malformed: the engine retries carrying the content already
+ // A retryable end: the engine retries carrying the content already
// produced, rendering a retry hint (with the attempt number); the terminal
// statuses aren't rendered (Request duration is covered by Trace performance
// analysis) and reset the consecutive-failure count. request events
// within a compaction range (only visible during history rebuild)
// are neither rendered nor counted — the compaction process only exposes the compaction event pair to the Human.
if (model.stats.compactionActive) return;
- // Credentials lifecycle rides on the request's own terminal status:
- // - "auth" records WHEN the failure happened (the event's envelope time), so the
- // composer gate can compare it against the Project's credentials-updated time
- // (see lastAuthFailureMs / isModelAuthDead). Origin routing already sends a
- // subagent's request events to the nested model, so reaching here with "auth"
- // always means the failure belongs to THIS session.
- // - a completed request proves the credentials work (again): the auth-dead state
- // must not outlive a success. Live and history replay share this path, so a Trace
- // with an auth failure followed by a completed request does not resurrect the dead
- // composer on reload.
- if (p.status === "auth") model.lastAuthFailureMs = tsMs ?? Date.now();
- if (p.status === "completed") model.lastAuthFailureMs = null;
// Pairs with request_begin to compute this Request's wall-clock
// duration, deducts the human approval wait, and adds the result to
// this Task's LLM time (for output TPS) — this duration includes tool
@@ -1701,10 +1686,19 @@ function handleEvent(model: StreamModel, p: EventPayload, tsMs?: number, nowMs?:
// round, so the pending compaction usage never reaches this step and is discarded at finalization (not counted into this round).
if (tsMs !== undefined) model.taskLastReqEndMs = tsMs;
commitPendingCompaction(model.stats);
- // Every status the engine reconnects on gets an item, `failed` included: it is retried
- // exactly like the other two, so leaving it out would stall the session for the whole
- // ladder with nothing on screen and no give-up control. It would also reset the counter
- // mid-ladder, renumbering a mixed timeout → failed → timeout run back to retry #1.
+ // A fatal end stops the run — no abort event follows; this request_end is the
+ // terminal record, rendered as an error banner.
+ if (p.status === "fatal") {
+ const item: LlmErrorItem = { kind: "llm_error", id: nextId(model) };
+ if (typeof p.error_code === "string") item.errorCode = p.error_code;
+ if (typeof p.error_message === "string" && p.error_message) {
+ item.errorMessage = p.error_message;
+ }
+ model.items.push(item);
+ return;
+ }
+ // Every status the engine reconnects on gets an item — leaving one out would stall
+ // the session for the whole ladder with nothing on screen and no give-up control.
if (isReconnectStatus(p.status)) {
const item: ReconnectItem = {
kind: "reconnect",
@@ -1715,11 +1709,21 @@ function handleEvent(model: StreamModel, p: EventPayload, tsMs?: number, nowMs?:
attempt: p.attempt ?? 1,
retrying: false,
};
+ if (typeof p.error_code === "string") item.errorCode = p.error_code;
+ if (typeof p.error_message === "string" && p.error_message) {
+ item.errorMessage = p.error_message;
+ }
// The engine announced its planned backoff: keep it with the CLIENT arrival time
// as the countdown anchor (skew-free — see ReconnectItem.arrivedAtMs).
if (typeof p.retry_in_ms === "number" && p.retry_in_ms > 0) {
item.plannedDelayMs = p.retry_in_ms;
item.arrivedAtMs = nowMs ?? Date.now();
+ } else if (p.status === "retryable") {
+ // A live-protocol retryable without a planned wait is the ladder giving up —
+ // the run ends on it (an abort follows only in legacy Traces). The legacy
+ // spellings never self-settle here: their era stamped no retry_in_ms mid-ladder,
+ // and their exhaustion is marked by the abort event instead.
+ item.gaveUp = true;
}
model.items.push(item);
}
diff --git a/packages/web/src/lib/strings-en.ts b/packages/web/src/lib/strings-en.ts
index 50dd5e52..a84d85c7 100644
--- a/packages/web/src/lib/strings-en.ts
+++ b/packages/web/src/lib/strings-en.ts
@@ -1439,34 +1439,52 @@ Scenarios:
thinking: "Thinking",
subagent: "Subagent",
subagentRunning: "Running",
- aborted: (reason?: string) => `[Aborted]${reason ? `: ${reason}` : ""}`,
- /** Auth-dead notice (request_end status "auth"): action-only copy — updating the key on the Models page auto-unlocks this Session. */
- modelAuthDead:
- "Model API authentication failed: update this model's API key on the Models page, or start a new Session.",
- modelAuthDeadOpenModels: "Open Models page",
- modelAuthDeadRetry: "Retry",
- modelAuthDeadCta: "New Session",
- modelAuthDeadPlaceholder: "Model authentication failed — update the API key first",
+ /**
+ * Abort banner (user interruptions only). The cause localizes from `errorCode`;
+ * `errorMessage` (raw, untranslatable) rides verbatim. A legacy Trace without a code
+ * renders its English `reason` prose as-is.
+ */
+ aborted: (item?: { errorCode?: string; errorMessage?: string; reason?: string }) => {
+ const cause =
+ item?.errorCode === "user_abort"
+ ? "aborted by user"
+ : item?.errorCode === "backoff_interrupted"
+ ? "aborted during reconnect backoff"
+ : item?.errorCode === "compaction_interrupted"
+ ? "aborted during compaction"
+ : (item?.errorCode ?? item?.reason ?? "");
+ const text = cause ? `${cause}${item?.errorMessage ? `: ${item.errorMessage}` : ""}` : "";
+ return `[Aborted]${text ? `: ${text}` : ""}`;
+ },
/**
* Reconnect hint line; `secondsLeft` (waiting state only) switches to the live-countdown
- * wording. `failed` is in the union because the engine retries it like the other two —
- * its cause names the provider rather than the transport, since that is where it came from.
+ * wording. `retryable` is the live status; the finer spellings only appear when
+ * replaying Traces written before the stop-reason convergence.
*/
reconnect: (
- status: "failed" | "timeout" | "malformed",
+ status: "retryable" | "failed" | "timeout" | "malformed",
state: "waiting" | "retried" | "gaveUp",
attempt: number,
secondsLeft?: number,
+ errorMessage?: string,
+ errorCode?: string,
) => {
+ // The live protocol carries the classified cause on error_code; the legacy status
+ // spellings (failed/timeout/malformed) say the same thing for pre-convergence Traces.
+ const kind = errorCode ?? status;
const cause =
- status === "timeout"
+ kind === "timeout"
? "Connection timed out"
- : status === "malformed"
+ : kind === "malformed"
? "Response incomplete or unparseable"
- : "The model provider returned an error";
+ : kind === "network"
+ ? "Network or service temporarily unavailable"
+ : kind === "failed"
+ ? "The model provider returned an error"
+ : "The request failed";
const action =
state === "gaveUp"
- ? "no further retries"
+ ? `giving up after attempt ${attempt}${errorMessage ? `: ${errorMessage}` : ""}`
: state === "retried"
? `retry #${attempt} sent`
: secondsLeft !== undefined
@@ -1474,6 +1492,9 @@ Scenarios:
: `starting retry #${attempt}…`;
return `[Retry] ${cause}; ${action}`;
},
+ /** Run-ending LLM failure banner (request_end status fatal); the provider's error text rides verbatim. */
+ llmError: (errorMessage?: string) =>
+ `[Error]: llm request error${errorMessage ? `: ${errorMessage}` : ""}`,
/** "Retry now" on the reconnect countdown (skips the remaining backoff wait). */
reconnectRetryNow: "Retry now",
/** "Give up" on the reconnect countdown (the ordinary session abort). */
@@ -1610,9 +1631,16 @@ Scenarios:
compactionTitle: (mode: string): string => (mode === "discard" ? "Clear" : "Compaction"),
compactionFailed: (status: string, errorMessage?: string): string => {
if (status === "aborted") return "aborted, keeping current context";
- return errorMessage !== undefined
- ? `failed (${errorMessage}), keeping current context`
- : "failed, keeping current context";
+ const detail = errorMessage !== undefined ? ` (${errorMessage})` : "";
+ // retryable = abandoned this time, the standing trigger retries it; fatal = a config
+ // or credential change has to come first. Legacy Traces spell both "failed".
+ if (status === "retryable") {
+ return `failed${detail}, keeping current context; retries at the next trigger`;
+ }
+ if (status === "fatal") {
+ return `failed${detail}, keeping current context; fix the model configuration to retry`;
+ }
+ return `failed${detail}, keeping current context`;
},
unknownTool: "(unknown tool)",
workRunning: "Running",
diff --git a/packages/web/src/lib/strings.ts b/packages/web/src/lib/strings.ts
index e15de1e3..f9e02af0 100644
--- a/packages/web/src/lib/strings.ts
+++ b/packages/web/src/lib/strings.ts
@@ -1397,33 +1397,52 @@ Benchmark:
thinking: "思考",
subagent: "子会话",
subagentRunning: "运行中",
- aborted: (reason?: string) => `[已中断]${reason ? `:${reason}` : ""}`,
- /** Auth-dead notice (request_end status "auth"): action-only copy — updating the key on the Models page auto-unlocks this Session. */
- modelAuthDead: "模型 API 认证失败:请在模型配置页更新该模型的 API key,或新建会话。",
- modelAuthDeadOpenModels: "打开模型配置",
- modelAuthDeadRetry: "重试",
- modelAuthDeadCta: "新建会话",
- modelAuthDeadPlaceholder: "模型认证失败,请先更新 API key",
+ /**
+ * Abort banner (user interruptions only). The cause localizes from `errorCode`;
+ * `errorMessage` (raw, untranslatable) rides verbatim. A legacy Trace without a code
+ * renders its English `reason` prose as-is.
+ */
+ aborted: (item?: { errorCode?: string; errorMessage?: string; reason?: string }) => {
+ const cause =
+ item?.errorCode === "user_abort"
+ ? "用户中断"
+ : item?.errorCode === "backoff_interrupted"
+ ? "重试等待中被中断"
+ : item?.errorCode === "compaction_interrupted"
+ ? "压缩过程中被中断"
+ : (item?.errorCode ?? item?.reason ?? "");
+ const text = cause ? `${cause}${item?.errorMessage ? `:${item.errorMessage}` : ""}` : "";
+ return `[已中断]${text ? `:${text}` : ""}`;
+ },
/**
* Reconnect hint line; `secondsLeft` (waiting state only) switches to the live-countdown
- * wording. `failed` is in the union because the engine retries it like the other two —
- * its cause names the provider rather than the transport, since that is where it came from.
+ * wording. `retryable` is the live status; the finer spellings only appear when
+ * replaying Traces written before the stop-reason convergence.
*/
reconnect: (
- status: "failed" | "timeout" | "malformed",
+ status: "retryable" | "failed" | "timeout" | "malformed",
state: "waiting" | "retried" | "gaveUp",
attempt: number,
secondsLeft?: number,
+ errorMessage?: string,
+ errorCode?: string,
) => {
+ // The live protocol carries the classified cause on error_code; the legacy status
+ // spellings (failed/timeout/malformed) say the same thing for pre-convergence Traces.
+ const kind = errorCode ?? status;
const cause =
- status === "timeout"
+ kind === "timeout"
? "连接超时或网络中断"
- : status === "malformed"
+ : kind === "malformed"
? "响应不完整或无法解析"
- : "模型服务返回错误";
+ : kind === "network"
+ ? "网络或服务暂时不可用"
+ : kind === "failed"
+ ? "模型服务返回错误"
+ : "请求失败";
const action =
state === "gaveUp"
- ? "已停止重试"
+ ? `第 ${attempt} 次尝试后放弃${errorMessage ? `:${errorMessage}` : ""}`
: state === "retried"
? `已发起第 ${attempt} 次重试`
: secondsLeft !== undefined
@@ -1431,6 +1450,9 @@ Benchmark:
: `正在发起第 ${attempt} 次重试…`;
return `[重试] ${cause},${action}`;
},
+ /** Run-ending LLM failure banner (request_end status fatal); the provider's error text rides verbatim. */
+ llmError: (errorMessage?: string) =>
+ `[错误]:模型请求错误${errorMessage ? `:${errorMessage}` : ""}`,
/** "Retry now" on the reconnect countdown (skips the remaining backoff wait). */
reconnectRetryNow: "立即重试",
/** "Give up" on the reconnect countdown (the ordinary session abort). */
@@ -1571,9 +1593,11 @@ Benchmark:
compactionTitle: (mode: string): string => (mode === "discard" ? "清空" : "压缩"),
compactionFailed: (status: string, errorMessage?: string): string => {
if (status === "aborted") return "已中断,保留当前上下文";
- return errorMessage !== undefined
- ? `失败(${errorMessage}),保留当前上下文`
- : "失败,保留当前上下文";
+ const detail = errorMessage !== undefined ? `(${errorMessage})` : "";
+ // retryable = 本次放弃、下次触发自动重试;fatal = 需先修复模型配置或凭据。旧 Trace 两者都拼作 "failed"。
+ if (status === "retryable") return `失败${detail},保留当前上下文,下次触发时重试`;
+ if (status === "fatal") return `失败${detail},保留当前上下文,需修复模型配置后重试`;
+ return `失败${detail},保留当前上下文`;
},
unknownTool: "(未知工具)",
workRunning: "运行中",
diff --git a/packages/web/test/composer-send.test.ts b/packages/web/test/composer-send.test.ts
index 4d574c81..5ac40d14 100644
--- a/packages/web/test/composer-send.test.ts
+++ b/packages/web/test/composer-send.test.ts
@@ -15,7 +15,6 @@ import type { MidRunComposerState } from "../src/features/chat/composer-send";
const BASE: MidRunComposerState = {
sending: false,
goalOn: false,
- modelAuthDead: false,
canSteerChannel: true,
canQueueChannel: true,
followUpMode: false,
@@ -90,12 +89,6 @@ describe("midRunAction — Stop is the fallthrough, not a case", () => {
expect(act({ goalOn: true, hasImages: true })).toBe("stop");
});
- it("a rejected model key leaves Stop, for every draft there is", () => {
- expect(act({ modelAuthDead: true })).toBe("stop");
- expect(act({ modelAuthDead: true, followUpMode: true })).toBe("stop");
- expect(act({ modelAuthDead: true, ...EMPTY, hasContent: true })).toBe("stop");
- });
-
it("a host with no channels at all leaves Stop rather than a dead Send", () => {
expect(act({ canSteerChannel: false, canQueueChannel: false })).toBe("stop");
});
diff --git a/packages/web/test/stream-model.test.ts b/packages/web/test/stream-model.test.ts
index 386e2fea..679014c2 100644
--- a/packages/web/test/stream-model.test.ts
+++ b/packages/web/test/stream-model.test.ts
@@ -42,7 +42,6 @@ import {
finalizeHistory,
findToolCard,
isDuplicate,
- isModelAuthDead,
notifyTaskIdle,
pushMessage,
pushMessages,
@@ -370,93 +369,36 @@ describe("approvals and events", () => {
expect((items(m)[0] as ToolCallItem).decision).toBe("allow");
});
- it("a main-session request_end(auth) records the failure timestamp; other aborts/events do not", () => {
+ it("a fatal request_end renders the error banner; an interim-build abort duplicate is dropped", () => {
const m = createStreamModel();
- pushMessage(m, abortEvent("aborted by user"));
- expect(m.lastAuthFailureMs).toBeNull();
- const end = requestEnd("auth", { errorMessage: "401 invalid x-api-key" });
- pushMessage(m, end);
- // The recorded time is the event's envelope timestamp (so a reload can compare it
- // against the Project's credentials-updated time).
- expect(m.lastAuthFailureMs).toBe(Date.parse(end.timestamp));
- // The abort that follows still renders its line (the notice is additional).
- pushMessage(m, abortEvent("llm request error: 401 invalid x-api-key"));
- expect(items(m)[1]).toMatchObject({
- kind: "abort",
- reason: "llm request error: 401 invalid x-api-key",
+ pushMessage(
+ m,
+ requestEnd("fatal", { errorCode: "auth", errorMessage: "401 invalid x-api-key" }),
+ );
+ expect(items(m)).toHaveLength(1);
+ expect(items(m)[0]).toMatchObject({
+ kind: "llm_error",
+ errorCode: "auth",
+ errorMessage: "401 invalid x-api-key",
});
- // Unrelated later messages don't clear it: only a COMPLETED request does (below) —
- // request_begin alone proves nothing about the credential.
- pushMessage(m, userText("hello?"));
- pushMessage(m, requestBegin());
- expect(m.lastAuthFailureMs).not.toBeNull();
- });
-
- it("a later completed request clears the auth-dead state; a new auth failure re-arms it", () => {
- const m = createStreamModel();
- pushMessage(m, requestEnd("auth", { errorMessage: "401" }));
- expect(m.lastAuthFailureMs).not.toBeNull();
- // The key was fixed and a request succeeded: the state must not outlive the success.
- pushMessage(m, userText("again"));
- pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("completed"));
- expect(m.lastAuthFailureMs).toBeNull();
- // A fresh auth failure re-arms (e.g. the replacement key is wrong too).
- pushMessage(m, requestEnd("auth", { errorMessage: "401" }));
- expect(m.lastAuthFailureMs).not.toBeNull();
- });
-
- it("history replay: order decides — auth failure then completed request stays alive; the reverse stays dead", () => {
- // Replay of a Trace where the auth failure was followed by a successful request (the
- // user fixed the key and continued): reload must NOT resurrect the dead composer.
- const recovered = createStreamModel();
- pushMessages(recovered, [
- userText("go"),
- requestBegin(),
- requestEnd("auth", { errorMessage: "401 invalid x-api-key" }),
- abortEvent("llm request error: 401 invalid x-api-key"),
- userText("after fix"),
- requestBegin(),
- requestEnd("completed"),
- ]);
- finalizeHistory(recovered);
- expect(recovered.lastAuthFailureMs).toBeNull();
-
- // Replay where the auth failure is the LAST word: the dead state is rebuilt.
- const dead = createStreamModel();
- pushMessages(dead, [
- userText("go"),
- requestBegin(),
- requestEnd("completed"),
- userText("later"),
- requestBegin(),
- requestEnd("auth", { errorMessage: "401 invalid x-api-key" }),
- abortEvent("llm request error: 401 invalid x-api-key"),
- ]);
- finalizeHistory(dead);
- expect(dead.lastAuthFailureMs).not.toBeNull();
});
- it("isModelAuthDead gates on the credentials-updated timestamp", () => {
- expect(isModelAuthDead(null, null)).toBe(false); // no auth failure on record
- expect(isModelAuthDead(1000, null)).toBe(true); // no update info -> nothing proves a fix
- expect(isModelAuthDead(1000, 2000)).toBe(false); // key updated after the failure -> alive
- expect(isModelAuthDead(3000, 2000)).toBe(true); // a fresh failure after the update re-arms
- });
-
- it("a subagent-origin auth failure does NOT kill the parent session's input", () => {
- const m = createStreamModel();
- pushMessage(m, withOrigin(requestEnd("auth", { errorMessage: "401" }), "child1"));
- // The failure belongs to the child session: its nested model carries the state, the
- // parent composer stays usable (the subagent simply surfaces as failed).
- expect(m.lastAuthFailureMs).toBeNull();
- expect(m.subagents.get("child1")!.lastAuthFailureMs).not.toBeNull();
- });
-
- it("abort events produce an abort marker item", () => {
+ it("abort events produce an abort marker item carrying the unified error pair", () => {
const m = createStreamModel();
- pushMessage(m, abortEvent("user abort"));
- expect(items(m)[0]).toMatchObject({ kind: "abort", reason: "user abort" });
+ pushMessage(m, abortEvent());
+ expect(items(m)[0]).toMatchObject({ kind: "abort", errorCode: "user_abort" });
+ pushMessage(m, abortEvent("backoff_interrupted"));
+ expect(items(m)[1]).toMatchObject({ kind: "abort", errorCode: "backoff_interrupted" });
+ // A legacy Trace's abort carries only the English prose — kept verbatim on the item.
+ const legacy = abortEvent();
+ delete (legacy.payload as { error_code?: string }).error_code;
+ (legacy.payload as { reason?: string }).reason = "llm request error: 401 nope";
+ pushMessage(m, legacy);
+ expect(items(m)[2]).toMatchObject({
+ kind: "abort",
+ reason: "llm request error: 401 nope",
+ });
+ expect((items(m)[2] as { errorCode?: string }).errorCode).toBeUndefined();
});
it("compaction begin/end produce a banner item; tokens accounting for the completion row", () => {
@@ -491,13 +433,13 @@ describe("approvals and events", () => {
compactionEnd({
reason: "context",
mode: "summarize",
- status: "failed",
+ status: "retryable",
attempt: 5,
errorMessage: "the response contained no usable summary",
}),
);
expect(banner.running).toBe(false);
- expect(banner.status).toBe("failed");
+ expect(banner.status).toBe("retryable");
expect(banner.errorMessage).toBe("the response contained no usable summary");
});
@@ -586,8 +528,8 @@ describe("approvals and events", () => {
pushMessage(m, userText("COMPACT NOW"));
pushMessage(m, requestBegin());
pushMessage(m, assistantText("[summary]half-writ")); // the draft the crash interrupted
- // ...process died here; the resume closes the span as failed before writing anything else:
- pushMessage(m, compactionEnd({ reason: "context", mode: "summarize", status: "failed" }));
+ // ...process died here; the resume closes the span as retryable before writing anything else:
+ pushMessage(m, compactionEnd({ reason: "context", mode: "summarize", status: "retryable" }));
// The conversation continues and must render.
pushMessage(m, userText("q2 after the crash"));
pushMessage(m, requestBegin());
@@ -605,7 +547,7 @@ describe("approvals and events", () => {
// The interrupted compaction reads as failed, and its half-written draft is discarded
// rather than shown as if it were the adopted summary.
const banner = items(m).find((i) => i.kind === "compaction") as CompactionItem;
- expect(banner).toMatchObject({ running: false, status: "failed" });
+ expect(banner).toMatchObject({ running: false, status: "retryable" });
expect(banner.summaryText).toBeUndefined();
});
@@ -647,15 +589,16 @@ describe("approvals and events", () => {
m,
at(
mcpConnectEnd({
- status: "failed",
+ status: "fatal",
results: [
{ server: "fx", transport: "stdio", status: "completed", duration_ms: 180, tools: 2 },
{
server: "bad",
transport: "stdio",
- status: "failed",
+ status: "fatal",
duration_ms: 60,
- error: "spawn nope ENOENT",
+ error_code: "connect_failed",
+ error_message: "spawn nope ENOENT",
},
],
}),
@@ -669,10 +612,30 @@ describe("approvals and events", () => {
// Per-server outcomes back the expanded server groups (failure reasons live there).
expect(row.results).toEqual([
{ server: "fx", status: "completed", durationMs: 180, tools: 2 },
- { server: "bad", status: "failed", durationMs: 60, error: "spawn nope ENOENT" },
+ { server: "bad", status: "fatal", durationMs: 60, error: "spawn nope ENOENT" },
]);
});
+ it('mcp connect row: a legacy Trace\'s "failed" spelling still lists the server as failed', () => {
+ const m = createStreamModel();
+ pushMessage(m, mcpConnectBegin(["old"]));
+ const legacy = mcpConnectEnd({
+ status: "fatal",
+ results: [{ server: "old", transport: "stdio", status: "fatal", duration_ms: 10 }],
+ });
+ // Traces written before the one-vocabulary convergence spell the failure "failed"
+ // and carry the detail as `error`.
+ (legacy.payload as { status: string }).status = "failed";
+ const legacyResult = (legacy.payload as { results: { status: string; error?: string }[] })
+ .results[0]!;
+ legacyResult.status = "failed";
+ legacyResult.error = "gone";
+ pushMessage(m, legacy);
+ const row = items(m)[0] as McpConnectItem;
+ expect(row.running).toBe(false);
+ expect(row.failed).toEqual(["old"]);
+ });
+
it("tool_list_ready attaches the MCP share of the toolset to the connect row (built-ins filtered out)", () => {
const m = createStreamModel();
pushMessage(m, mcpConnectBegin(["fx"]));
@@ -708,15 +671,16 @@ describe("approvals and events", () => {
expect(items(m)).toHaveLength(0);
});
- it("request_end final state timeout/malformed produces a retry notice item (with the stamped attempt); request_begin marks it as resent", () => {
+ it("a retryable request_end produces a retry notice item (with the stamped attempt); request_begin marks it as resent", () => {
const m = createStreamModel();
pushMessage(m, requestBegin());
- // The core stamps the authoritative ordinal on every retryable request_end.
- pushMessage(m, requestEnd("malformed", { attempt: 1 }));
+ // The core stamps the authoritative ordinal — and the planned backoff — on every
+ // mid-ladder retryable request_end.
+ pushMessage(m, requestEnd("retryable", { attempt: 1, retryInMs: 2000 }));
const retry = items(m)[0] as ReconnectItem;
expect(retry).toMatchObject({
kind: "reconnect",
- status: "malformed",
+ status: "retryable",
attempt: 1,
retrying: false,
});
@@ -724,31 +688,51 @@ describe("approvals and events", () => {
pushMessage(m, requestBegin());
expect(retry.retrying).toBe(true);
// Retry fails again: a second notice, carrying the next ordinal.
- pushMessage(m, requestEnd("timeout", { attempt: 2 }));
+ pushMessage(m, requestEnd("retryable", { attempt: 2, retryInMs: 4000 }));
const retry2 = items(m)[1] as ReconnectItem;
- expect(retry2).toMatchObject({ kind: "reconnect", status: "timeout", attempt: 2 });
+ expect(retry2).toMatchObject({ kind: "reconnect", status: "retryable", attempt: 2 });
// Retry succeeds: no new entry; the next round's failure carries attempt 1 again.
pushMessage(m, requestBegin());
pushMessage(m, requestEnd("completed"));
expect(items(m)).toHaveLength(2);
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout", { attempt: 1 }));
+ pushMessage(m, requestEnd("retryable", { attempt: 1 }));
expect((items(m)[2] as ReconnectItem).attempt).toBe(1);
// A Trace written before the field existed lacks it: rendered as attempt 1.
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout"));
+ pushMessage(m, requestEnd("retryable"));
expect((items(m)[3] as ReconnectItem).attempt).toBe(1);
});
- it("request_end(failed) renders a retry notice too, with its countdown inputs and give-up target", () => {
- // The engine reconnects on `failed` exactly like timeout/malformed. Without an item there
- // is no countdown and findLastWaitingReconnect returns null, so "Retry now" / "Give up"
- // never render either — the session just stalls for up to 7.75s with nothing on screen.
+ it("a terminal retryable (no retry planned) settles as gave-up at creation, detail attached", () => {
+ // The ladder ran out: the run ends on this request_end — no abort event follows.
+ const m = createStreamModel();
+ pushMessage(m, requestBegin());
+ pushMessage(m, requestEnd("retryable", { attempt: 6, errorMessage: "socket hang up" }));
+ const item = items(m)[0] as ReconnectItem;
+ expect(item).toMatchObject({
+ kind: "reconnect",
+ status: "retryable",
+ attempt: 6,
+ gaveUp: true,
+ errorMessage: "socket hang up",
+ });
+ // The legacy spellings never self-settle (their era stamped no retry_in_ms mid-ladder).
+ pushMessage(m, requestBegin());
+ pushMessage(m, requestEnd("timeout" as never, { attempt: 1 }));
+ const legacy = items(m)[1] as ReconnectItem;
+ expect(legacy.gaveUp).toBeUndefined();
+ });
+
+ it("the retry notice carries its countdown inputs and give-up target", () => {
+ // Without an item there is no countdown and findLastWaitingReconnect returns null, so
+ // "Retry now" / "Give up" never render either — the session would just stall with
+ // nothing on screen.
const m = createStreamModel();
pushMessage(m, requestBegin());
pushMessage(
m,
- requestEnd("failed", {
+ requestEnd("retryable", {
errorMessage: "Upstream HTTP/2 stream failed (upstream_http2_stream_error)",
retryInMs: 4000,
}),
@@ -757,57 +741,66 @@ describe("approvals and events", () => {
const retry = items(m)[0] as ReconnectItem;
expect(retry).toMatchObject({
kind: "reconnect",
- status: "failed",
+ status: "retryable",
attempt: 1,
retrying: false,
plannedDelayMs: 4000, // the countdown
arrivedAtMs: 111_000, // its client-clock anchor
});
// Waiting, so it is the item the retry-now / give-up controls attach to; the retry then
- // flips it out of the waiting state exactly like the other two statuses.
+ // flips it out of the waiting state.
pushMessage(m, requestBegin());
expect(retry.retrying).toBe(true);
});
- it("a mixed ladder keeps counting: the stamped ordinal carries across failure kinds", () => {
- // The engine numbers timeout → failed → timeout as one run (all three draw on the same
- // budget), and each notice shows the event's own ordinal.
+ it("a legacy-trace mixed ladder keeps counting: the finer pre-convergence spellings still render", () => {
+ // Traces written before the stop-reason convergence spell retryable ends as
+ // timeout/failed/malformed; replay renders the same ladder with each event's ordinal.
+ const legacyEnd = (status: string, retry?: { attempt?: number; errorMessage?: string }) =>
+ requestEnd(status as Parameters<typeof requestEnd>[0], retry);
const m = createStreamModel();
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout", { attempt: 1 }));
+ pushMessage(m, legacyEnd("timeout", { attempt: 1 }));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("failed", { errorMessage: "502 bad gateway", attempt: 2 }));
+ pushMessage(m, legacyEnd("failed", { errorMessage: "502 bad gateway", attempt: 2 }));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout", { attempt: 3 }));
+ pushMessage(m, legacyEnd("malformed", { attempt: 3 }));
expect((items(m) as ReconnectItem[]).map((i) => [i.status, i.attempt])).toEqual([
["timeout", 1],
["failed", 2],
- ["timeout", 3],
+ ["malformed", 3],
]);
// After a normal finish the next run's first failure is stamped #1 again.
pushMessage(m, requestBegin());
pushMessage(m, requestEnd("completed"));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("failed", { attempt: 1 }));
+ pushMessage(m, legacyEnd("failed", { attempt: 1 }));
expect((items(m)[3] as ReconnectItem).attempt).toBe(1);
});
- it("request_end(auth) stays out of the ladder: terminal, so no retry notice", () => {
- // The one status the engine does not retry — an item would promise a countdown and a
- // "Retry now" that will never happen, on top of the composer already being gated.
+ it("request_end(fatal) stays out of the ladder: terminal, so no retry notice", () => {
+ // The status the engine does not retry — an item would promise a countdown and a
+ // "Retry now" that will never happen.
const m = createStreamModel();
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout", { attempt: 1 }));
+ pushMessage(m, requestEnd("retryable", { attempt: 1 }));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("auth", { errorMessage: "401 invalid x-api-key", attempt: 2 }));
+ pushMessage(m, requestEnd("fatal", { errorMessage: "401 invalid x-api-key", attempt: 2 }));
expect((items(m) as ReconnectItem[]).filter((i) => i.kind === "reconnect")).toHaveLength(1);
});
it("retries exhausted: an arriving abort marks the waiting retry notice gaveUp and resets the consecutive-failure count", () => {
const m = createStreamModel();
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout"));
- pushMessage(m, abortEvent("reconnect failed after 5 retries"));
+ // Legacy flow: a pre-convergence Trace (spelling "timeout", no retry_in_ms) whose
+ // exhaustion was marked by the abort event.
+ const legacyEnd = requestEnd("retryable");
+ (legacyEnd.payload as { status: string }).status = "timeout";
+ pushMessage(m, legacyEnd);
+ const legacyAbort = abortEvent();
+ delete (legacyAbort.payload as { error_code?: string }).error_code;
+ (legacyAbort.payload as { reason?: string }).reason = "llm request failed after 5 retries";
+ pushMessage(m, legacyAbort);
const retry = items(m)[0] as ReconnectItem;
expect(retry).toMatchObject({
kind: "reconnect",
@@ -819,7 +812,7 @@ describe("approvals and events", () => {
// A new failure in the next run starts back at 1; a gaveUp notice isn't revived by request_begin.
pushMessage(m, requestBegin());
expect(retry.retrying).toBe(false);
- pushMessage(m, requestEnd("malformed"));
+ pushMessage(m, requestEnd("retryable", { retryInMs: 2000 }));
expect((items(m)[2] as ReconnectItem).attempt).toBe(1);
});
@@ -831,8 +824,8 @@ describe("approvals and events", () => {
// bend the ticker.
pushMessage(
m,
- requestEnd("timeout", {
- errorMessage: "403 quota (insufficient_user_quota)",
+ requestEnd("retryable", {
+ errorMessage: "429 rate limited (slow down)",
retryInMs: 4000,
}),
111_000,
@@ -840,7 +833,7 @@ describe("approvals and events", () => {
const item = items(m)[0] as ReconnectItem;
expect(item).toMatchObject({
kind: "reconnect",
- status: "timeout",
+ status: "retryable",
attempt: 1,
retrying: false,
plannedDelayMs: 4000,
@@ -849,7 +842,7 @@ describe("approvals and events", () => {
// An event without the field (old Traces / final failures) leaves the fields unset —
// the view keeps the plain waiting text.
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout"));
+ pushMessage(m, requestEnd("retryable"));
const plain = items(m)[1] as ReconnectItem;
expect(plain.plannedDelayMs).toBeUndefined();
expect(plain.arrivedAtMs).toBeUndefined();
@@ -862,7 +855,7 @@ describe("approvals and events", () => {
pushMessages(retried, [
userText("go"),
requestBegin(),
- requestEnd("timeout", { errorMessage: "quota", retryInMs: 30_000 }),
+ requestEnd("retryable", { errorMessage: "quota", retryInMs: 30_000 }),
requestBegin(), // the engine's retry — replayed immediately after
requestEnd("completed"),
]);
@@ -874,8 +867,8 @@ describe("approvals and events", () => {
pushMessages(aborted, [
userText("go"),
requestBegin(),
- requestEnd("timeout", { errorMessage: "quota", retryInMs: 30_000 }),
- abortEvent("aborted during reconnect backoff"), // the user gave up mid-wait
+ requestEnd("retryable", { errorMessage: "quota", retryInMs: 30_000 }),
+ abortEvent("backoff_interrupted"), // the user gave up mid-wait
]);
finalizeHistory(aborted);
const gaveUp = items(aborted).find((i) => i.kind === "reconnect") as ReconnectItem;
@@ -886,7 +879,7 @@ describe("approvals and events", () => {
const m = createStreamModel();
pushMessage(m, userText("go"));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout")); // dangling at the tail: no abort, no retry begin
+ pushMessage(m, requestEnd("retryable")); // dangling at the tail: no abort, no retry begin
const dangling = items(m).find((i) => i.kind === "reconnect") as ReconnectItem;
expect(dangling.retrying).toBe(false);
// New Task: the dangling item is marked gaveUp (the new request isn't its retry), count resets.
@@ -894,12 +887,12 @@ describe("approvals and events", () => {
pushMessage(m, requestBegin());
expect(dangling.gaveUp).toBe(true);
expect(dangling.retrying).toBe(false);
- pushMessage(m, requestEnd("timeout"));
+ pushMessage(m, requestEnd("retryable"));
const fresh = items(m).filter((i) => i.kind === "reconnect")[1] as ReconnectItem;
expect(fresh.attempt).toBe(1);
});
- it("a tool_call closed non-completed (malformed closure) settles the card on arrival instead of showing as running", () => {
+ it("a tool_call closed non-completed (interrupt closure) settles the card on arrival instead of showing as running", () => {
const m = createStreamModel();
pushMessage(
m,
@@ -907,22 +900,22 @@ describe("approvals and events", () => {
name: "exec_command",
arguments: '{"cmd": "ec',
toolCallId: "tc-broken",
- stopReason: "malformed",
+ stopReason: "retryable",
}),
);
const card = findToolCard(m, undefined, "tc-broken")!;
expect(card.callComplete).toBe(true);
// This call was never dispatched for execution and will never have output: close it
- // immediately with the close reason, so execution timing doesn't spin idle.
+ // immediately with the call's own closure reason, so execution timing doesn't spin idle.
expect(card.outputComplete).toBe(true);
- expect(card.outputStopReason).toBe("malformed");
+ expect(card.outputStopReason).toBe("retryable");
});
it("request events inside a compaction span produce no retry notice (history rebuild exposes only the event pair for compaction)", () => {
const m = createStreamModel();
pushMessage(m, compactionBegin({ reason: "context", mode: "summarize", context: 1, turns: 1 }));
pushMessage(m, requestBegin());
- pushMessage(m, requestEnd("timeout"));
+ pushMessage(m, requestEnd("retryable"));
pushMessage(m, compactionEnd({ reason: "context", mode: "summarize", status: "completed" }));
expect(items(m).filter((i) => i.kind === "reconnect")).toHaveLength(0);
});
@@ -1480,7 +1473,7 @@ describe("compaction attribution by position: in-round counts toward the round,
at(requestBegin(), "2026-07-05T00:00:01.000Z"),
at(tokenUsage(out(100), out(100)), "2026-07-05T00:00:02.000Z"),
at(requestEnd("aborted"), "2026-07-05T00:00:03.000Z"),
- at(abortEvent("user"), "2026-07-05T00:00:03.000Z"),
+ at(abortEvent(), "2026-07-05T00:00:03.000Z"),
// The user doesn't send the next message until 60 seconds later
at(userText("ask again"), "2026-07-05T00:01:03.000Z"),
at(requestBegin(), "2026-07-05T00:01:04.000Z"),
@@ -2100,7 +2093,7 @@ describe("thinking/tool durations (collapsed-row display data)", () => {
const m = createStreamModel();
pushMessage(m, at(userText("Q"), T0));
pushMessage(m, at(toolCall({ name: "exec_command", arguments: "{}", toolCallId: "ta" }), T1));
- pushMessage(m, at(abortEvent("user"), T2));
+ pushMessage(m, at(abortEvent(), T2));
const card = items(m).find((i) => i.kind === "tool_call") as ToolCallItem;
expect(card.outputComplete).toBe(true);
expect(card.outputStreaming).toBe(false);
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 61294638..e48ad9fd 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -95,8 +95,8 @@ importers:
specifier: ^2.0.0
version: 2.0.0
'@prismshadow/agenthub':
- specifier: ^0.4.6
+ specifier: ^0.4.7
'@prismshadow/penguin-skills':
specifier: workspace:*
version: link:../skills
@@ -954,6 +954,9 @@ packages:
'@prismshadow/[email protected]':
resolution: {integrity: sha512-0umTpATCfaPpxI9wRwC19oLJswbBKUOvMOCAQMiVcUXALQ6EoDC7sPnciDuFh2F17MLJZUajsVZWGjaBN4V/tQ==}
+ '@prismshadow/[email protected]':
+ resolution: {integrity: sha512-TmsWkFSu/O0Esk9jHyb4BRU0WUD1IZADsmpyPc8jATnxrUA0yVUG7gXttlf8W2cWjmWm58T4h7/R5I421H+dPw==}
+
'@prismshadow/penguin-cli@file:packages/cli':
resolution: {directory: packages/cli, type: directory}
engines: {node: '>=24'}
@@ -4505,7 +4508,7 @@ snapshots:
dependencies:
playwright: 1.61.1
dependencies:
'@anthropic-ai/bedrock-sdk': 0.26.4([email protected])
'@anthropic-ai/sdk': 0.91.1([email protected])
@@ -4520,7 +4523,7 @@ snapshots:
- ws
- zod
dependencies:
'@anthropic-ai/bedrock-sdk': 0.26.4([email protected])
'@anthropic-ai/sdk': 0.91.1([email protected])
@@ -4535,17 +4538,13 @@ snapshots:
- ws
- zod
- '@prismshadow/penguin-cli@file:packages/cli([email protected])':
dependencies:
- '@modelcontextprotocol/client': 2.0.0
- '@prismshadow/agenthub': 0.4.6([email protected])([email protected])
- '@prismshadow/penguin-core': file:packages/core([email protected])
- '@prismshadow/penguin-server': file:packages/server([email protected])
- '@prismshadow/penguin-skills': file:packages/skills
- commander: 13.1.0
- dotenv: 17.4.2
- smol-toml: 1.6.1
- yaml: 2.9.0
+ '@anthropic-ai/bedrock-sdk': 0.26.4([email protected])
+ '@anthropic-ai/sdk': 0.91.1([email protected])
+ '@google/genai': 1.52.0([email protected])
+ express: 4.22.2([email protected])
transitivePeerDependencies:
- '@modelcontextprotocol/sdk'
- bufferutil
@@ -4554,12 +4553,15 @@ snapshots:
- ws
- zod
- '@prismshadow/penguin-core@file:packages/core([email protected])':
+ '@prismshadow/penguin-cli@file:packages/cli([email protected])':
dependencies:
'@modelcontextprotocol/client': 2.0.0
- '@prismshadow/agenthub': 0.4.6([email protected])([email protected])
+ '@prismshadow/agenthub': 0.4.6([email protected])([email protected])([email protected])
+ '@prismshadow/penguin-core': file:packages/core([email protected])([email protected])
+ '@prismshadow/penguin-server': file:packages/server([email protected])
'@prismshadow/penguin-skills': file:packages/skills
- arktype: 2.2.3
+ commander: 13.1.0
+ dotenv: 17.4.2
smol-toml: 1.6.1
yaml: 2.9.0
transitivePeerDependencies:
@@ -4573,7 +4575,7 @@ snapshots:
'@prismshadow/penguin-core@file:packages/core([email protected])([email protected])':
dependencies:
'@modelcontextprotocol/client': 2.0.0
- '@prismshadow/agenthub': 0.4.6([email protected])([email protected])
+ '@prismshadow/agenthub': 0.4.7([email protected])([email protected])
'@prismshadow/penguin-skills': file:packages/skills
arktype: 2.2.3
smol-toml: 1.6.1
@@ -4589,7 +4591,7 @@ snapshots:
'@prismshadow/penguin-core@file:packages/core([email protected])([email protected])([email protected])':
dependencies:
'@modelcontextprotocol/client': 2.0.0
- '@prismshadow/agenthub': 0.4.6([email protected])([email protected])([email protected])
+ '@prismshadow/agenthub': 0.4.7([email protected])([email protected])([email protected])
'@prismshadow/penguin-skills': file:packages/skills
arktype: 2.2.3
smol-toml: 1.6.1
diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml
index b280b404..a29602b7 100644
--- a/pnpm-workspace.yaml
+++ b/pnpm-workspace.yaml
@@ -31,8 +31,8 @@ allowBuilds:
# targets NSIS only, so the script is deliberately not run.
electron-winstaller: false
# pnpm 11 withholds freshly published versions by default (supply-chain gate), which
-# would resolve agenthub back to 0.4.5 and leave DeepSeek on Chat Completions without the
-# vision model. This package is first-party, so the exemption carries no third-party risk.
-# Remove the entry at the next agenthub bump — by then 0.4.6 is long past the age gate.
+# would resolve agenthub back to 0.4.6 and lose the Gemini function-response content
+# split. This package is first-party, so the exemption carries no third-party risk.
+# Remove the entry at the next agenthub bump — by then 0.4.7 is long past the age gate.
minimumReleaseAgeExclude:
- - "@prismshadow/[email protected]"
+ - "@prismshadow/[email protected]"