Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

fix(server): a failed re-hold sweep is recorded, not left as an unhandled rejection

lmlihenga71ae33faaa0
diff --git a/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.md b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.md
new file mode 100644
index 00000000..bed20f68
--- /dev/null
+++ b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.md
@@ -0,0 +1,18 @@
+# A failed re-hold sweep is recorded instead of escaping the process
+
+- **Date:** 2026-11-10
+- **Type:** fix
+- **Scope:** `server`
+- **Breaking:** no
+
+[中文版](2026-11-10-machines-keep-held-tick-error.zh.md)
+
+The timer that re-holds dropped machine connections once a minute called `void autoConnect()`, and nothing took the rejection. A timer callback has no caller to throw to: any failure of that sweep — an ssh config that cannot be read, a database closed underneath it — came out as an unhandled rejection once a minute for as long as the App lived, visible only in the process log.
+
+The tick now goes through a small private step that awaits the sweep and reports a failure to the error centre, under the same source and code the boot path already uses for the same fault (`process` / `machines_reconnect_failed`): one machine that will not come back is one class in the panel, and one ignore rule there silences both the boot attempt and the retries.
+
+A `stop()` that arrives while the sweep is still reaching out is not reported: the closed database is the retirement itself, not a fault, so a retired generation says nothing about its own shutdown.
+
+## Verification
+
+`test/machines-lifecycle.test.ts` gains two cases: a failing tick is recorded and nothing escapes the process (a `unhandledRejection` listener observes the escape and sees none), and a `stop()` landing mid-sweep records nothing.
diff --git a/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.md b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.md
new file mode 100644
index 00000000..0eae32b5
--- /dev/null
+++ b/changelog/unreleased/2026-11-10-machines-keep-held-tick-error.zh.md
@@ -0,0 +1,24 @@
+# 「保持连接」定时器里那次重试的失败被记下来,不再冒成 unhandled rejection
+
+- **日期:** 2026-11-10
+- **类型:** 修复
+- **范围:** `server`
+- **破坏性变更:** 否
+
+[English](2026-11-10-machines-keep-held-tick-error.md)
+
+那个每分钟把掉线的机器连接重新保持住的定时器,回调写的是 `void autoConnect()`,拒绝没有人接。定时器回调
+没有可以抛给的调用方:这一趟扫描的任何失败 —— 读不了的 ssh 配置、在它脚下被关掉的数据库 —— 都会冒成一条
+unhandled rejection,每分钟一条,直到进程结束,而它只出现在进程日志里。
+
+现在这一跳走一个小的私有步骤:把扫描 await 掉,失败就报到错误中心,用的是启动那一趟给同一类故障已经在用的
+来源与错误码(`process` / `machines_reconnect_failed`)—— 一台回不来的机器在面板上是**一类**,那里的一条
+忽略规则就能把启动那次与之后的重试一起静音。
+
+`stop()` 如果正好落在这趟扫描还在走网络的时候,不记:被关掉的数据库就是「退役」本身,不是故障,退役的那一代
+对自己这次关停什么也不说。
+
+## 验证
+
+`test/machines-lifecycle.test.ts` 新增两条:失败的定时器扫描会被记下来、且没有任何东西逃出进程(用一个
+`unhandledRejection` 监听器观察逃逸,观察到的是零条);以及扫描途中落下的 `stop()` 一条都不记。
diff --git a/packages/server/src/machines/service.ts b/packages/server/src/machines/service.ts
index e6dc7814..d60cd805 100644
--- a/packages/server/src/machines/service.ts
+++ b/packages/server/src/machines/service.ts
@@ -75,6 +75,7 @@ import type { ClassCtx } from "@adelie/core/kernel";
import { machinesRoutes } from "../http/routes/machines.js";
import { machinesProxy } from "./proxy.js";
import { HttpError } from "../http/errors.js";
+import type { Errors } from "../mechanisms/observability.js";
import type { Access } from "../mechanisms/projects.js";
import { Hono } from "hono";
import { MachinesRepo } from "../db/repos/machines.js";
@@ -132,6 +133,12 @@ export interface MachinesEffects {
loadConfig: (projectId: string) => Promise<ProjectConfig>;
/** Injected so a test can pin the recorded timestamp instead of asserting around the clock. */
now: () => Date;
+ /**
+ * Where a failure with no caller to throw to is reported — the standing-intent timer below
+ * is the one such path here. Silent by default: a service built outside the platform (a test)
+ * has no error table, and the rejection it would have recorded is the test's own to assert.
+ */
+ recordError: (err: unknown) => void;
}
/** The id of the entry standing for the machine this server runs on. */
@@ -230,6 +237,7 @@ export class MachinesService {
upgrade: upgradeRemote,
loadConfig: (projectId) => loadProjectConfig(dataRoot, projectId),
now: () => new Date(),
+ recordError: () => {},
...effects,
};
}
@@ -1107,11 +1115,36 @@ export class MachinesService {
// The check above and this one both look at #stopped: either await is long enough for a
// hot push's stop() to land, and a timer installed after it would never be cleared.
if (!this.#stopped && this.#keepHeld === null) {
- this.#keepHeld = setInterval(() => void this.autoConnect(), KEEP_HELD_MS);
+ this.#keepHeld = setInterval(() => void this.#keepHeldTick(), KEEP_HELD_MS);
this.#keepHeld.unref?.();
}
}
+ /**
+ * One attempt of the standing intent, from the timer.
+ *
+ * The boot path in platform.ts is fire-and-forget too, but it has somewhere to put a
+ * rejection (its own `.catch`); a timer callback has nobody to throw to, so a rejection here
+ * would escape to the process as an unhandled rejection — which is what used to happen once a
+ * minute whenever a sweep failed. It is recorded instead, under the same source and code as
+ * that boot path: one machine failing to come back is one class in the error centre, and one
+ * rule there silences both occasions.
+ *
+ * The guard first: a stop() landing between the tick and this call retires the generation,
+ * and reading a closed database to tell its error table about it helps nobody.
+ */
+ async #keepHeldTick(): Promise<void> {
+ if (this.#stopped) return;
+ try {
+ await this.autoConnect();
+ } catch (err) {
+ // A stop() that arrived while this sweep was awaiting closed the database underneath it:
+ // that rejection IS the retirement, and reporting it would be telling the error table of a
+ // database this generation no longer owns about its own shutdown.
+ if (!this.#stopped) this.#effects.recordError(err);
+ }
+ }
+
/**
* Closes every connection THIS generation opened — the platform's dispose effect on a hot
* push. The record of which were held stays, so the successor's start() brings each back.
@@ -1537,6 +1570,7 @@ export class MachinesModule {
@Use() private readonly db!: Db;
@Use() private readonly hmr!: Hmr;
@Use() private readonly access!: Access;
+ @Use() private readonly errors!: Errors;
@Provide() machines!: Machines;
@Bind("MachinesModule.routes") routes!: Hono<AppEnv>;
@Bind("MachinesModule.server-proxy") serverProxyRoutes!: Hono<AppEnv>;
@@ -1545,8 +1579,18 @@ export class MachinesModule {
// after — every stored reference to this machine, here and on the machines it reaches,
// points at it. A test that supplies its own service mints none.
const repo = new MachinesRepo(this.db as unknown as DatabaseSync);
- const machines = new MachinesService(this.paths.root, repo.ownId(), repo, {}, () =>
- this.hmr.assetsDir(),
+ const machines = new MachinesService(
+ this.paths.root,
+ repo.ownId(),
+ repo,
+ {
+ // The standing-intent timer's failures have no caller to throw to; they go to the error
+ // centre under the same source and code the boot path uses for the same failure
+ // (platform.ts), so one machine that will not come back is one silenceable class.
+ recordError: (err) =>
+ this.errors.record({ source: "process", err, code: "machines_reconnect_failed" }),
+ },
+ () => this.hmr.assetsDir(),
);
this.machines = machines;
this.routes = machinesRoutes({ machines, access: this.access });
diff --git a/packages/server/test/machines-lifecycle.test.ts b/packages/server/test/machines-lifecycle.test.ts
index 97764775..18105d11 100644
--- a/packages/server/test/machines-lifecycle.test.ts
+++ b/packages/server/test/machines-lifecycle.test.ts
@@ -13,6 +13,10 @@
* - A stop() landing while start() is still reaching out installs no timer at all.
* - A retired generation's start() reaches nothing, so a late caller cannot revive it.
* - An App's cleanup retires its machines generation — a start() afterwards touches nothing.
+ * - A sweep from the timer that fails is RECORDED: a timer callback has no caller to throw to,
+ * so before this it escaped to the process as an unhandled rejection, once a minute.
+ * - A stop() landing while a sweep is still reaching out records nothing: the closed database
+ * underneath it is the retirement, not a fault.
*/
import { afterEach, describe, expect, it, vi } from "vitest";
import fs from "node:fs";
@@ -45,9 +49,11 @@ describe("machines generation lifetime", () => {
/**
* A service over a fresh root and store, with only the reaching-out faked: one alias in the
* ssh config, no build to hand over, and a counter for every read of that config — which is
- * what a re-hold sweep starts with.
+ * what a re-hold sweep starts with. `overrides` replaces the fake itself: a sweep that fails
+ * is the subject of two cases below, and one of them needs the service's own handle while the
+ * fake is still being built.
*/
- const harness = async () => {
+ const harness = async (overrides: Partial<MachinesEffects> = {}) => {
const root = await makeTempRoot();
const store = openDatabase(":memory:");
open.push({ store, root });
@@ -59,6 +65,7 @@ describe("machines generation lifetime", () => {
},
resolvePlan: () => null,
now: () => new Date("2026-08-24T12:00:00.000Z"),
+ ...overrides,
};
const service = new MachinesService(
root,
@@ -117,4 +124,61 @@ describe("machines generation lifetime", () => {
await built.service.start();
expect(built.reads()).toBe(afterCleanup);
});
+
+ it("a sweep from the timer that fails is recorded, not left as an unhandled rejection", async () => {
+ vi.useFakeTimers();
+ // A rejection nobody handles is reported a turn later; the listener is how this case can
+ // see one that escapes, since a registered listener keeps Node from failing the run itself.
+ const escapes: unknown[] = [];
+ const onEscape = (reason: unknown) => escapes.push(reason);
+ process.on("unhandledRejection", onEscape);
+ try {
+ const recorded: unknown[] = [];
+ let reads = 0;
+ const built = await harness({
+ // The boot's own sweep reads the config and gets an answer; every read after it is the
+ // timer's, and that one fails — an ssh config gone unreadable, a closed store, anything.
+ listAliases: () => {
+ reads += 1;
+ if (reads > 1) throw new Error("ssh config unreadable");
+ return ["nas"];
+ },
+ recordError: (err) => recorded.push(err),
+ });
+ await built.service.start();
+ await vi.advanceTimersByTimeAsync(KEEP_HELD_MS);
+ await new Promise((resolve) => process.nextTick(resolve));
+
+ expect(recorded.map((err) => (err as Error).message)).toEqual(["ssh config unreadable"]);
+ expect(escapes).toEqual([]);
+ } finally {
+ process.off("unhandledRejection", onEscape);
+ }
+ });
+
+ it("a stop() landing while a sweep is reaching out records nothing", async () => {
+ vi.useFakeTimers();
+ const recorded: unknown[] = [];
+ let service: MachinesService | null = null;
+ let reads = 0;
+ const built = await harness({
+ listAliases: () => {
+ reads += 1;
+ if (reads > 1) {
+ // The hot push arriving mid-sweep: what the sweep was reading closes underneath it.
+ // That failure is the retirement itself, so no one needs to hear about it.
+ service?.stop();
+ throw new Error("database is not open");
+ }
+ return ["nas"];
+ },
+ recordError: (err) => recorded.push(err),
+ });
+ service = built.service;
+ await built.service.start();
+ await vi.advanceTimersByTimeAsync(KEEP_HELD_MS);
+
+ expect(reads).toBeGreaterThan(1);
+ expect(recorded).toEqual([]);
+ });
});