A local-first multi-agent app development platform, built on PenguinHarness
fix(server): a failed re-hold sweep is recorded, not left as an unhandled rejection
lmlihenga71ae33faaa0
diff --git a/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.md b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.mdnew file mode 100644index 00000000..bed20f68--- /dev/null+++ b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.md@@ -0,0 +1,18 @@+# A failed re-hold sweep is recorded instead of escaping the process++- **Date:** 2026-11-10+- **Type:** fix+- **Scope:** `server`+- **Breaking:** no++[中文版](2026-11-10-machines-keep-held-tick-error.zh.md)++The timer that re-holds dropped machine connections once a minute called `void autoConnect()`, and nothing took the rejection. A timer callback has no caller to throw to: any failure of that sweep — an ssh config that cannot be read, a database closed underneath it — came out as an unhandled rejection once a minute for as long as the App lived, visible only in the process log.++The tick now goes through a small private step that awaits the sweep and reports a failure to the error centre, under the same source and code the boot path already uses for the same fault (`process` / `machines_reconnect_failed`): one machine that will not come back is one class in the panel, and one ignore rule there silences both the boot attempt and the retries.++A `stop()` that arrives while the sweep is still reaching out is not reported: the closed database is the retirement itself, not a fault, so a retired generation says nothing about its own shutdown.++## Verification++`test/machines-lifecycle.test.ts` gains two cases: a failing tick is recorded and nothing escapes the process (a `unhandledRejection` listener observes the escape and sees none), and a `stop()` landing mid-sweep records nothing.diff --git a/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.md b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.mdnew file mode 100644index 00000000..0eae32b5--- /dev/null+++ b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.md@@ -0,0 +1,24 @@+# 「保持连接」定时器里那次重试的失败被记下来,不再冒成 unhandled rejection++- **日期:** 2026-11-10+- **类型:** 修复+- **范围:** `server`+- **破坏性变更:** 否++[English](2026-11-10-machines-keep-held-tick-error.md)++那个每分钟把掉线的机器连接重新保持住的定时器,回调写的是 `void autoConnect()`,拒绝没有人接。定时器回调+没有可以抛给的调用方:这一趟扫描的任何失败 —— 读不了的 ssh 配置、在它脚下被关掉的数据库 —— 都会冒成一条+unhandled rejection,每分钟一条,直到进程结束,而它只出现在进程日志里。++现在这一跳走一个小的私有步骤:把扫描 await 掉,失败就报到错误中心,用的是启动那一趟给同一类故障已经在用的+来源与错误码(`process` / `machines_reconnect_failed`)—— 一台回不来的机器在面板上是**一类**,那里的一条+忽略规则就能把启动那次与之后的重试一起静音。++`stop()` 如果正好落在这趟扫描还在走网络的时候,不记:被关掉的数据库就是「退役」本身,不是故障,退役的那一代+对自己这次关停什么也不说。++## 验证++`test/machines-lifecycle.test.ts` 新增两条:失败的定时器扫描会被记下来、且没有任何东西逃出进程(用一个+`unhandledRejection` 监听器观察逃逸,观察到的是零条);以及扫描途中落下的 `stop()` 一条都不记。diff --git a/packages/server/src/machines/service.ts b/packages/server/src/machines/service.tsindex e6dc7814..d60cd805 100644--- a/packages/server/src/machines/service.ts+++ b/packages/server/src/machines/service.ts@@ -75,6 +75,7 @@ import type { ClassCtx } from "@adelie/core/kernel";import { machinesRoutes } from "../http/routes/machines.js";import { machinesProxy } from "./proxy.js";import { HttpError } from "../http/errors.js";+import type { Errors } from "../mechanisms/observability.js";import type { Access } from "../mechanisms/projects.js";import { Hono } from "hono";import { MachinesRepo } from "../db/repos/machines.js";@@ -132,6 +133,12 @@ export interface MachinesEffects {loadConfig: (projectId: string) => Promise<ProjectConfig>;/** Injected so a test can pin the recorded timestamp instead of asserting around the clock. */now: () => Date;+ /**+ * Where a failure with no caller to throw to is reported — the standing-intent timer below+ * is the one such path here. Silent by default: a service built outside the platform (a test)+ * has no error table, and the rejection it would have recorded is the test's own to assert.+ */+ recordError: (err: unknown) => void;}/** The id of the entry standing for the machine this server runs on. */@@ -230,6 +237,7 @@ export class MachinesService {upgrade: upgradeRemote,loadConfig: (projectId) => loadProjectConfig(dataRoot, projectId),now: () => new Date(),+ recordError: () => {},...effects,};}@@ -1107,11 +1115,36 @@ export class MachinesService {// The check above and this one both look at #stopped: either await is long enough for a// hot push's stop() to land, and a timer installed after it would never be cleared.if (!this.#stopped && this.#keepHeld === null) {- this.#keepHeld = setInterval(() => void this.autoConnect(), KEEP_HELD_MS);+ this.#keepHeld = setInterval(() => void this.#keepHeldTick(), KEEP_HELD_MS);this.#keepHeld.unref?.();}}+ /**+ * One attempt of the standing intent, from the timer.+ *+ * The boot path in platform.ts is fire-and-forget too, but it has somewhere to put a+ * rejection (its own `.catch`); a timer callback has nobody to throw to, so a rejection here+ * would escape to the process as an unhandled rejection — which is what used to happen once a+ * minute whenever a sweep failed. It is recorded instead, under the same source and code as+ * that boot path: one machine failing to come back is one class in the error centre, and one+ * rule there silences both occasions.+ *+ * The guard first: a stop() landing between the tick and this call retires the generation,+ * and reading a closed database to tell its error table about it helps nobody.+ */+ async #keepHeldTick(): Promise<void> {+ if (this.#stopped) return;+ try {+ await this.autoConnect();+ } catch (err) {+ // A stop() that arrived while this sweep was awaiting closed the database underneath it:+ // that rejection IS the retirement, and reporting it would be telling the error table of a+ // database this generation no longer owns about its own shutdown.+ if (!this.#stopped) this.#effects.recordError(err);+ }+ }+/*** Closes every connection THIS generation opened — the platform's dispose effect on a hot* push. The record of which were held stays, so the successor's start() brings each back.@@ -1537,6 +1570,7 @@ export class MachinesModule {@Use() private readonly db!: Db;@Use() private readonly hmr!: Hmr;@Use() private readonly access!: Access;+ @Use() private readonly errors!: Errors;@Provide() machines!: Machines;@Bind("MachinesModule.routes") routes!: Hono<AppEnv>;@Bind("MachinesModule.server-proxy") serverProxyRoutes!: Hono<AppEnv>;@@ -1545,8 +1579,18 @@ export class MachinesModule {// after — every stored reference to this machine, here and on the machines it reaches,// points at it. A test that supplies its own service mints none.const repo = new MachinesRepo(this.db as unknown as DatabaseSync);- const machines = new MachinesService(this.paths.root, repo.ownId(), repo, {}, () =>- this.hmr.assetsDir(),+ const machines = new MachinesService(+ this.paths.root,+ repo.ownId(),+ repo,+ {+ // The standing-intent timer's failures have no caller to throw to; they go to the error+ // centre under the same source and code the boot path uses for the same failure+ // (platform.ts), so one machine that will not come back is one silenceable class.+ recordError: (err) =>+ this.errors.record({ source: "process", err, code: "machines_reconnect_failed" }),+ },+ () => this.hmr.assetsDir(),);this.machines = machines;this.routes = machinesRoutes({ machines, access: this.access });diff --git a/packages/server/test/machines-lifecycle.test.ts b/packages/server/test/machines-lifecycle.test.tsindex 97764775..18105d11 100644--- a/packages/server/test/machines-lifecycle.test.ts+++ b/packages/server/test/machines-lifecycle.test.ts@@ -13,6 +13,10 @@* - A stop() landing while start() is still reaching out installs no timer at all.* - A retired generation's start() reaches nothing, so a late caller cannot revive it.* - An App's cleanup retires its machines generation — a start() afterwards touches nothing.+ * - A sweep from the timer that fails is RECORDED: a timer callback has no caller to throw to,+ * so before this it escaped to the process as an unhandled rejection, once a minute.+ * - A stop() landing while a sweep is still reaching out records nothing: the closed database+ * underneath it is the retirement, not a fault.*/import { afterEach, describe, expect, it, vi } from "vitest";import fs from "node:fs";@@ -45,9 +49,11 @@ describe("machines generation lifetime", () => {/*** A service over a fresh root and store, with only the reaching-out faked: one alias in the* ssh config, no build to hand over, and a counter for every read of that config — which is- * what a re-hold sweep starts with.+ * what a re-hold sweep starts with. `overrides` replaces the fake itself: a sweep that fails+ * is the subject of two cases below, and one of them needs the service's own handle while the+ * fake is still being built.*/- const harness = async () => {+ const harness = async (overrides: Partial<MachinesEffects> = {}) => {const root = await makeTempRoot();const store = openDatabase(":memory:");open.push({ store, root });@@ -59,6 +65,7 @@ describe("machines generation lifetime", () => {},resolvePlan: () => null,now: () => new Date("2026-08-24T12:00:00.000Z"),+ ...overrides,};const service = new MachinesService(root,@@ -117,4 +124,61 @@ describe("machines generation lifetime", () => {await built.service.start();expect(built.reads()).toBe(afterCleanup);});++ it("a sweep from the timer that fails is recorded, not left as an unhandled rejection", async () => {+ vi.useFakeTimers();+ // A rejection nobody handles is reported a turn later; the listener is how this case can+ // see one that escapes, since a registered listener keeps Node from failing the run itself.+ const escapes: unknown[] = [];+ const onEscape = (reason: unknown) => escapes.push(reason);+ process.on("unhandledRejection", onEscape);+ try {+ const recorded: unknown[] = [];+ let reads = 0;+ const built = await harness({+ // The boot's own sweep reads the config and gets an answer; every read after it is the+ // timer's, and that one fails — an ssh config gone unreadable, a closed store, anything.+ listAliases: () => {+ reads += 1;+ if (reads > 1) throw new Error("ssh config unreadable");+ return ["nas"];+ },+ recordError: (err) => recorded.push(err),+ });+ await built.service.start();+ await vi.advanceTimersByTimeAsync(KEEP_HELD_MS);+ await new Promise((resolve) => process.nextTick(resolve));++ expect(recorded.map((err) => (err as Error).message)).toEqual(["ssh config unreadable"]);+ expect(escapes).toEqual([]);+ } finally {+ process.off("unhandledRejection", onEscape);+ }+ });++ it("a stop() landing while a sweep is reaching out records nothing", async () => {+ vi.useFakeTimers();+ const recorded: unknown[] = [];+ let service: MachinesService | null = null;+ let reads = 0;+ const built = await harness({+ listAliases: () => {+ reads += 1;+ if (reads > 1) {+ // The hot push arriving mid-sweep: what the sweep was reading closes underneath it.+ // That failure is the retirement itself, so no one needs to hear about it.+ service?.stop();+ throw new Error("database is not open");+ }+ return ["nas"];+ },+ recordError: (err) => recorded.push(err),+ });+ service = built.service;+ await built.service.start();+ await vi.advanceTimersByTimeAsync(KEEP_HELD_MS);++ expect(reads).toBeGreaterThan(1);+ expect(recorded).toEqual([]);+ });});