Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(plugins): terminate a sandbox worker whose event loop is blocked

Yudhi Armyndharis307a90c1ca44
diff --git a/docs/19-plugin-architecture.md b/docs/19-plugin-architecture.md
index 23e88d020..1c302c31d 100644
--- a/docs/19-plugin-architecture.md
+++ b/docs/19-plugin-architecture.md
@@ -777,8 +777,12 @@ tier automatically. Key properties:
- Each worker has a heap cap (`maxOldGenerationSizeMb`, default **256 MB**) — an OOM terminates the
worker, not the host.
- A sandboxed hook handler has a **5 s** time budget (`SANDBOX_HOOK_TIMEOUT_MS`); on timeout the host
- resolves `{ continue: true }` (fail-open) so a slow/wedged handler never stalls the hook chain. The
- same fail-open value drains in-flight hooks if the worker crashes.
+ resolves `{ continue: true }` (fail-open), so each sandboxed subscriber delays the hook chain by at most
+ 5 s per event. The same fail-open value drains in-flight hooks if the worker crashes. The timeout is
+ logged at most once per event per minute and shows as the last hook error in plugin health.
+- After a hook, webhook or search dispatch times out, the host pings the worker. A worker whose event
+ loop is blocked (a synchronous loop) cannot answer within 5 s, so it is terminated and the plugin is
+ set to `ERROR`, as on a crash; a slow async handler answers and is left running.
- Lifecycle methods (`load`/`onLoad`/`onEnable`/`onDisable`) and `healthCheck` are bounded by a **30 s**
/ **5 s** timeout respectively, so a wedged plugin can't hang an ADMIN enable/disable or the health endpoint.
- The worker gets a **minimal allowlisted env** (`NODE_ENV`, `NODE_EXTRA_CA_CERTS`, `TZ`) — host secrets
diff --git a/docs/30-plugin-sandboxing.md b/docs/30-plugin-sandboxing.md
index 882244c0d..44e043a31 100644
--- a/docs/30-plugin-sandboxing.md
+++ b/docs/30-plugin-sandboxing.md
@@ -45,13 +45,18 @@ from disk is user-installed and sandboxed.
host terminates, authenticates and enqueues the delivery first.
- **Hook safety.** Hook handlers run in the worker and are dispatched with a **time budget**
(`SANDBOX_HOOK_TIMEOUT_MS`, a hardcoded 5s). A slow or wedged handler is skipped (`continue: true`)
- so it can never stall the host's hook chain.
+ once the budget runs out, so each sandboxed subscriber can delay the hook chain by at most 5 s per
+ event, never indefinitely.
- **Resource & runaway containment.** Each worker has a heap cap (`maxOldGenerationSizeMb`, a hardcoded
256 MB). An OOM terminates the worker, not the host. A wedged **lifecycle** call (load/unload) times out
and tears the worker down, and a crash rejects its in-flight calls — the host survives either way. A
- runaway **hook** handler (e.g. an infinite synchronous loop) is skipped on the hook timeout so it can't
- stall the host's hook chain, but the worker keeps running it (pegging a core) until the plugin is
- reloaded or hits the heap cap — it is contained to its own thread, not instantly force-killed.
+ runaway **hook** handler (e.g. an infinite synchronous loop) is skipped on the hook timeout. After any
+ hook, webhook or search dispatch times out, the host sends the worker a liveness ping that the worker
+ answers before running plugin code. A slow async handler answers at once and keeps running. A worker
+ still working through a backlog of dispatches reads the ping late, but every result it returns counts
+ as an answer. A worker whose event loop is blocked answers nothing for 5 s, so the host terminates it
+ and sets the plugin to `ERROR` (no automatic restart, the same as a crash); the operator re-enables it
+ once fixed.
- **Memory-kind boundary.** The heap cap bounds the V8 heap only. `Buffer`/`ArrayBuffer` allocations
are native memory outside `maxOldGenerationSizeMb`, and worker threads share the host's address
space, so a plugin that accumulates Buffers grows host RSS until the container's memory limit
diff --git a/src/core/plugins/plugin-loader-sandbox-routing.spec.ts b/src/core/plugins/plugin-loader-sandbox-routing.spec.ts
index c2a92f1b8..d864b02c7 100644
--- a/src/core/plugins/plugin-loader-sandbox-routing.spec.ts
+++ b/src/core/plugins/plugin-loader-sandbox-routing.spec.ts
@@ -21,6 +21,7 @@ class TestableLoader extends PluginLoaderService {
capturedOnHookSubscribe?: (event: string, priority?: number) => void;
capturedOnLog?: (level: PluginLogLevel, message: string, meta?: Record<string, unknown>) => void;
capturedOnWorkerExit?: (code: number, intentional: boolean) => void;
+ capturedOnUnresponsive?: () => void;
protected createSandboxHost(
_capDispatcher?: (verb: string, args: unknown[]) => Promise<unknown>,
onHookSubscribe?: (event: string, priority?: number) => void,
@@ -29,10 +30,12 @@ class TestableLoader extends PluginLoaderService {
_runWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,
_onSearchProviderRegister?: () => void,
onWorkerExit?: (code: number, intentional: boolean) => void,
+ onUnresponsive?: () => void,
): PluginWorkerHost {
this.capturedOnHookSubscribe = onHookSubscribe;
this.capturedOnLog = onLog;
this.capturedOnWorkerExit = onWorkerExit;
+ this.capturedOnUnresponsive = onUnresponsive;
const host: FakeHost = {
load: jest.fn().mockResolvedValue(undefined),
runLifecycle: jest.fn().mockResolvedValue(undefined),
@@ -335,6 +338,92 @@ describe('PluginLoaderService — sandbox hook error surfacing', () => {
});
});
+describe('PluginLoaderService - blocked sandbox worker', () => {
+ const loggerOf = (loader: TestableLoader): { warn: jest.Mock } =>
+ (loader as unknown as { logger: { warn: jest.Mock } }).logger;
+ const hostsOf = (loader: TestableLoader): Map<string, unknown> =>
+ (loader as unknown as { sandboxHosts: Map<string, unknown> }).sandboxHosts;
+ const storageOf = (loader: TestableLoader): { setPluginStatus: jest.Mock } =>
+ (loader as unknown as { pluginStorage: { setPluginStatus: jest.Mock } }).pluginStorage;
+
+ it('terminates an unresponsive worker and leaves the plugin in ERROR with its hooks removed', async () => {
+ const loader = makeLoader();
+ seed(loader, { builtIn: false, instance: null });
+ const hookManager = (loader as unknown as { hookManager: HookManager }).hookManager;
+ const unregisterSpy = jest.spyOn(hookManager, 'unregisterPlugin');
+ const warnSpy = jest.spyOn(loggerOf(loader), 'warn').mockImplementation(() => undefined);
+ await loader.enablePlugin('p1');
+ const host = loader.hosts[0];
+
+ loader.capturedOnUnresponsive!();
+
+ const plugin = pluginOf(loader);
+ expect(plugin.status).toBe(PluginStatus.ERROR);
+ expect(plugin.error).toMatch(/unresponsive/);
+ expect(storageOf(loader).setPluginStatus).toHaveBeenCalledWith('p1', PluginStatus.ERROR);
+ expect(unregisterSpy).toHaveBeenCalledWith('p1');
+ expect(hostsOf(loader).has('p1')).toBe(false);
+ expect(host.terminate).toHaveBeenCalled();
+ expect(warnSpy).toHaveBeenCalledWith(
+ expect.stringContaining('p1'),
+ expect.objectContaining({ action: 'sandbox_worker_unresponsive', pluginId: 'p1' }),
+ );
+
+ // The deliberate terminate() then exits the worker; that must not undo or repeat the above.
+ loader.capturedOnWorkerExit!(1, true);
+ expect(pluginOf(loader).status).toBe(PluginStatus.ERROR);
+ });
+
+ it('ignores a late report from a worker generation that has already been replaced', async () => {
+ const loader = makeLoader();
+ seed(loader, { builtIn: false, instance: null });
+ await loader.enablePlugin('p1');
+ const staleReport = loader.capturedOnUnresponsive!;
+ await loader.disablePlugin('p1');
+ await loader.enablePlugin('p1');
+ const current = loader.hosts[1];
+
+ staleReport();
+
+ expect(pluginOf(loader).status).toBe(PluginStatus.ENABLED);
+ expect(hostsOf(loader).get('p1')).toBe(current);
+ expect(current.terminate).not.toHaveBeenCalled();
+ });
+
+ it('rate-limits the hook timeout warn per event and surfaces the timeout in plugin health', async () => {
+ const loader = makeLoader();
+ seed(loader, { builtIn: false, instance: null });
+ const hookManager = (loader as unknown as { hookManager: HookManager }).hookManager;
+ const registerSpy = jest.spyOn(hookManager, 'register');
+ await loader.enablePlugin('p1');
+ loader.capturedOnHookSubscribe!('message:received');
+ const handler = registerSpy.mock.calls.find(c => c[1] === 'message:received')![2];
+ loader.hosts[0].dispatchHook.mockImplementation((options: { onTimeout?: () => void }) => {
+ options.onTimeout?.();
+ return Promise.resolve({ continue: true });
+ });
+ const warnSpy = jest.spyOn(loggerOf(loader), 'warn').mockImplementation(() => undefined);
+ const hookCtx: HookContext = {
+ event: 'message:received',
+ data: {},
+ sessionId: 's1',
+ timestamp: new Date(0),
+ source: 'Engine',
+ };
+
+ await handler(hookCtx);
+ await handler(hookCtx);
+
+ const timeoutWarns = warnSpy.mock.calls.filter(
+ c => (c[1] as { action?: string } | undefined)?.action === 'sandbox_hook_timeout',
+ );
+ expect(timeoutWarns).toHaveLength(1);
+ const health = await loader.checkPluginHealth('p1');
+ expect(health.message).toContain("last hook error in 'message:received'");
+ expect(health.message).toContain('timed out');
+ });
+});
+
describe('PluginLoaderService — sandbox log relay bounds', () => {
const loggerOf = (loader: TestableLoader): { warn: jest.Mock; log: jest.Mock } =>
(loader as unknown as { logger: { warn: jest.Mock; log: jest.Mock } }).logger;
diff --git a/src/core/plugins/plugin-loader.service.ts b/src/core/plugins/plugin-loader.service.ts
index 482b4c34f..3e47f67a0 100644
--- a/src/core/plugins/plugin-loader.service.ts
+++ b/src/core/plugins/plugin-loader.service.ts
@@ -54,8 +54,16 @@ const SANDBOX_MAX_INFLIGHT_CAPS = 32;
const SANDBOX_CAP_TIMEOUT_MS = 30000;
/**
- * Host process.env keys an untrusted plugin worker is allowed to see. Everything else — secrets like
- * API_MASTER_KEY, API_KEY_PEPPER, the DATABASE_/REDIS_ vars, DOCKER_HOST — is left out of the worker's
+ * How long a worker has to answer the liveness ping the host sends after a dispatch times out. A worker
+ * stuck in a synchronous loop is terminated about 5 s after its first dispatch timeout (about 10 s after
+ * a stalled hook or ingress dispatch, 15 s after a stalled search); a slow async handler answers at once
+ * and is never affected.
+ */
+const SANDBOX_LIVENESS_TIMEOUT_MS = 5000;
+
+/**
+ * Host process.env keys an untrusted plugin worker is allowed to see. Everything else (secrets like
+ * API_MASTER_KEY, API_KEY_PEPPER, the DATABASE_/REDIS_ vars, DOCKER_HOST) is left out of the worker's
* process.env object. That keeps secrets from being handed over ambiently; it does not put them out of
* reach, since a thread in the same OS process can read /proc/self/environ and the data directory. The
* worker is a thread, so it needs no PATH to start and require() resolves via module paths, not env.
@@ -174,6 +182,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap
runWithHookGuard,
onSearchProviderRegister,
onWorkerExit,
+ onUnresponsive,
) =>
this.createSandboxHost(
capDispatcher,
@@ -183,6 +192,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap
runWithHookGuard,
onSearchProviderRegister,
onWorkerExit,
+ onUnresponsive,
),
// Exported from this module (specs import it here); passed so the bridge never imports back.
resolvePluginMainPath,
@@ -372,6 +382,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap
runWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,
onSearchProviderRegister?: () => void,
onWorkerExit?: (code: number, intentional: boolean) => void,
+ onUnresponsive?: () => void,
): PluginWorkerHost {
const workerEntry = path.join(__dirname, 'sandbox', 'worker-bootstrap.js');
return new PluginWorkerHost(
@@ -390,6 +401,8 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap
onSearchProviderRegister,
onWorkerExit,
this.configService.get<number>('plugins.capTimeoutMs') ?? SANDBOX_CAP_TIMEOUT_MS,
+ SANDBOX_LIVENESS_TIMEOUT_MS,
+ onUnresponsive,
);
}
diff --git a/src/core/plugins/plugin-sandbox-bridge.ts b/src/core/plugins/plugin-sandbox-bridge.ts
index 395569909..a3fafd592 100644
--- a/src/core/plugins/plugin-sandbox-bridge.ts
+++ b/src/core/plugins/plugin-sandbox-bridge.ts
@@ -66,6 +66,7 @@ type CreateSandboxHostFn = (
runWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,
onSearchProviderRegister?: () => void,
onWorkerExit?: (code: number, intentional: boolean) => void,
+ onUnresponsive?: () => void,
) => PluginWorkerHost;
/**
@@ -122,20 +123,23 @@ export class PluginSandboxBridge {
event: string,
error: string,
rateLimit: Map<string, { lastAt: number; suppressed: number }>,
+ action: 'sandbox_hook_error' | 'sandbox_hook_timeout' = 'sandbox_hook_error',
): void {
this.lastSandboxHookError.set(pluginId, { event, error, at: new Date() });
const now = Date.now();
- const state = rateLimit.get(event);
+ // Errors and timeouts are windowed apart, so a burst of one never hides the other.
+ const key = `${action}:${event}`;
+ const state = rateLimit.get(key);
if (state && now - state.lastAt < SANDBOX_HOOK_ERROR_LOG_INTERVAL_MS) {
state.suppressed++;
return;
}
const suppressed = state?.suppressed ?? 0;
- rateLimit.set(event, { lastAt: now, suppressed: 0 });
+ rateLimit.set(key, { lastAt: now, suppressed: 0 });
this.logger.warn(`Sandboxed plugin ${pluginId} hook '${event}' handler failed: ${error}`, {
pluginId,
event,
- action: 'sandbox_hook_error',
+ action,
...(suppressed > 0 ? { suppressed } : {}),
});
}
@@ -340,7 +344,21 @@ export class PluginSandboxBridge {
const onWorkerExit = this.buildWorkerExitHandler(pluginId, logRelayState);
- const host = this.createHost(
+ // A worker whose event loop stays blocked after a dispatch timeout (a synchronous loop in plugin
+ // code) would otherwise keep running and queue every later dispatch for the full budget each.
+ // Same policy as a crash: terminate it and leave the plugin in ERROR for the operator to re-enable.
+ // The generation check keeps a late report from marking a replacement worker.
+ const onUnresponsive = (): void => {
+ if (this.sandboxHosts.get(pluginId) !== host) return;
+ this.markWorkerFailed(pluginId, 'worker unresponsive: event loop blocked past the liveness probe');
+ this.logger.warn(`Sandboxed plugin ${pluginId} worker is unresponsive; terminating it`, {
+ pluginId,
+ action: 'sandbox_worker_unresponsive',
+ });
+ void host.terminate().catch(() => undefined);
+ };
+
+ const host: PluginWorkerHost = this.createHost(
(verb, args) => dispatchCapabilityVerb(context, verb, args),
onHookSubscribe,
onWebhookSubscribe,
@@ -350,6 +368,7 @@ export class PluginSandboxBridge {
(events, run) => this.hookManager.runInFlight(events as HookEvent[], run),
onSearchProviderRegister,
onWorkerExit,
+ onUnresponsive,
);
this.sandboxHosts.set(pluginId, host);
try {
@@ -508,12 +527,16 @@ export class PluginSandboxBridge {
// against re-firing any event of it (not only this one) once it returns to the host.
inFlight: this.hookManager.currentInFlight(),
timeoutMs: SANDBOX_HOOK_TIMEOUT_MS,
+ // Rate-limited like a handler error: a plugin that times out on every message must be
+ // visible (log + health) without writing one line per event.
onTimeout: () =>
- this.logger.warn(`Sandboxed plugin ${pluginId} hook '${event}' timed out`, {
+ this.recordSandboxHookError(
pluginId,
event,
- action: 'sandbox_hook_timeout',
- }),
+ `timed out after ${SANDBOX_HOOK_TIMEOUT_MS}ms`,
+ hookErrorLogState,
+ 'sandbox_hook_timeout',
+ ),
})
.then(result => {
// The worker reports (not throws) a hook-handler failure: surface it host-side instead
@@ -590,11 +613,23 @@ export class PluginSandboxBridge {
};
}
+ // Shared ERROR transition for a crashed or unresponsive worker: mark the plugin ERROR, drop its hook
+ // shims and the dead host. The exit handler releases the search provider.
+ private markWorkerFailed(pluginId: string, reason: string): void {
+ const plugin = this.plugins.get(pluginId);
+ if (plugin) {
+ plugin.status = PluginStatus.ERROR;
+ plugin.error = reason;
+ this.pluginStorage.setPluginStatus(pluginId, PluginStatus.ERROR);
+ }
+ this.hookManager.unregisterPlugin(pluginId);
+ this.sandboxHosts.delete(pluginId);
+ }
+
// A worker that crashes AFTER a successful enable is otherwise invisible to the loader (handleExit only
// drains in-flight calls). Drop the plugin's search-provider entry so the registry falls back to
// builtin-fts instead of routing every /search to a dead worker (auto mode would otherwise pin the dead
- // provider ACTIVE). Mirrors the enable-failure cleanup. Broader crash-lifecycle cleanup (status, hooks)
- // is a pre-existing gap for all bridges and out of scope here.
+ // provider ACTIVE). Mirrors the enable-failure cleanup.
private buildWorkerExitHandler(
pluginId: string,
logRelayState: SandboxLogRelayState,
@@ -619,14 +654,7 @@ export class PluginSandboxBridge {
// unregister the hook shims (so they don't keep dispatching into the dead worker) + mark the
// plugin ERROR so the dashboard reflects reality. The dispatchHook/dispatchWebhook dead-checks
// fail-fast; this cleanup is the root-cause fix (it also makes the shim's !liveHost guard fire).
- const crashed = this.plugins.get(pluginId);
- if (crashed) {
- crashed.status = PluginStatus.ERROR;
- crashed.error = `worker exited unexpectedly (code ${code})`;
- this.pluginStorage.setPluginStatus(pluginId, PluginStatus.ERROR);
- }
- this.hookManager.unregisterPlugin(pluginId);
- this.sandboxHosts.delete(pluginId);
+ this.markWorkerFailed(pluginId, `worker exited unexpectedly (code ${code})`);
this.logger.warn(`Sandboxed plugin ${pluginId} worker exited unexpectedly (code ${code})`, {
pluginId,
code,
diff --git a/src/core/plugins/sandbox/plugin-worker-host.spec.ts b/src/core/plugins/sandbox/plugin-worker-host.spec.ts
index 3f5c1b6d5..23d72c5fd 100644
--- a/src/core/plugins/sandbox/plugin-worker-host.spec.ts
+++ b/src/core/plugins/sandbox/plugin-worker-host.spec.ts
@@ -863,4 +863,159 @@ describe('PluginWorkerHost', () => {
}
});
});
+
+ describe('liveness probe after a dispatch timeout', () => {
+ const PROBE_MS = 1000;
+ const pings = (ch: FakeChannel) =>
+ ch.sent.filter((m): m is Extract<HostToWorkerMessage, { kind: 'ping' }> => m.kind === 'ping');
+ const probingHost = (ch: FakeChannel, onUnresponsive: () => void) =>
+ new PluginWorkerHost(
+ ch,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ PROBE_MS,
+ onUnresponsive,
+ );
+ const hook = (host: PluginWorkerHost) =>
+ host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 500 });
+ const webhook = (host: PluginWorkerHost) =>
+ host.dispatchWebhook({
+ instanceId: 'i',
+ route: 'r',
+ method: 'POST',
+ headers: {},
+ query: {},
+ body: '',
+ rawBody: '',
+ verified: true,
+ deliveryId: 'd',
+ timeoutMs: 500,
+ });
+
+ beforeEach(() => jest.useFakeTimers());
+ afterEach(() => jest.useRealTimers());
+
+ it('posts one ping on a hook timeout, and none while that probe is still in flight', async () => {
+ const ch = new FakeChannel();
+ const host = probingHost(ch, jest.fn());
+ const first = hook(host);
+ const second = hook(host);
+ jest.advanceTimersByTime(500);
+ await Promise.all([first, second]);
+ expect(pings(ch)).toHaveLength(1);
+ });
+
+ it('a pong clears the probe: a slow async handler is not reported', async () => {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ ch.reply({ kind: 'pong', id: pings(ch)[0].id });
+ jest.advanceTimersByTime(PROBE_MS * 3);
+ expect(onUnresponsive).not.toHaveBeenCalled();
+
+ // The next timeout probes again rather than being suppressed by the answered one.
+ const next = hook(host);
+ jest.advanceTimersByTime(500);
+ await next;
+ expect(pings(ch)).toHaveLength(2);
+ });
+
+ it('reports a worker that never answers the probe, exactly once', async () => {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ jest.advanceTimersByTime(PROBE_MS);
+ expect(onUnresponsive).toHaveBeenCalledTimes(1);
+
+ const again = hook(host);
+ jest.advanceTimersByTime(500 + PROBE_MS * 2);
+ await again;
+ expect(onUnresponsive).toHaveBeenCalledTimes(1);
+ });
+
+ it('a worker still answering a backlog is not reported while its pong waits behind it', async () => {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ const hookId = (ch.sent.find(m => m.kind === 'hook') as { id: number }).id;
+
+ // Late results keep arriving, each inside the window: the event loop is turning.
+ for (let i = 0; i < 3; i++) {
+ jest.advanceTimersByTime(PROBE_MS - 100);
+ ch.reply({ kind: 'hook-result', id: hookId, continue: true });
+ }
+ expect(onUnresponsive).not.toHaveBeenCalled();
+
+ // A full window with no answer at all is still reported.
+ jest.advanceTimersByTime(PROBE_MS);
+ expect(onUnresponsive).toHaveBeenCalledTimes(1);
+ });
+
+ it('log lines do not count as progress: a synchronous loop can emit them', async () => {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ jest.advanceTimersByTime(PROBE_MS - 100);
+ ch.reply({ kind: 'log', level: 'log', message: 'still spinning' });
+ jest.advanceTimersByTime(100);
+ expect(onUnresponsive).toHaveBeenCalledTimes(1);
+ });
+
+ it('webhook and search timeouts probe too', async () => {
+ for (const dispatch of [
+ webhook,
+ (host: PluginWorkerHost) => host.dispatchSearch({ query: { q: 'x' }, timeoutMs: 500 }),
+ ]) {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = dispatch(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ expect(pings(ch)).toHaveLength(1);
+ jest.advanceTimersByTime(PROBE_MS);
+ expect(onUnresponsive).toHaveBeenCalledTimes(1);
+ }
+ });
+
+ it('does not probe when no liveness budget is configured', async () => {
+ const ch = new FakeChannel();
+ const host = new PluginWorkerHost(ch);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ expect(pings(ch)).toHaveLength(0);
+ });
+
+ it('a worker exit cancels a probe in flight', async () => {
+ const ch = new FakeChannel();
+ const onUnresponsive = jest.fn();
+ const host = probingHost(ch, onUnresponsive);
+ const pending = hook(host);
+ jest.advanceTimersByTime(500);
+ await pending;
+ ch.crash(1);
+ jest.advanceTimersByTime(PROBE_MS * 2);
+ expect(onUnresponsive).not.toHaveBeenCalled();
+ });
+ });
});
diff --git a/src/core/plugins/sandbox/plugin-worker-host.ts b/src/core/plugins/sandbox/plugin-worker-host.ts
index 0e1708ff8..ec2d8bc6c 100644
--- a/src/core/plugins/sandbox/plugin-worker-host.ts
+++ b/src/core/plugins/sandbox/plugin-worker-host.ts
@@ -21,6 +21,15 @@ const SEND_CAP_VERBS: ReadonlySet<string> = new Set(['messages.sendText', 'messa
/** Send-verb budget as a multiple of capTimeoutMs (default 30s → 120s: the 30s media download + upload headroom). */
const SEND_CAP_TIMEOUT_FACTOR = 4;
+/** Worker replies that finish a host request; one arriving during a liveness probe restarts its window. */
+const PROGRESS_KINDS: ReadonlySet<WorkerToHostMessage['kind']> = new Set([
+ 'lifecycle-result',
+ 'hook-result',
+ 'webhook-result',
+ 'health-result',
+ 'search-result',
+]);
+
/**
* Host-side driver for a single untrusted plugin running in a worker. Owns the request/response
* correlation over a {@link PluginWorkerChannel}: it posts `load`/`lifecycle` messages and resolves
@@ -84,6 +93,9 @@ export class PluginWorkerHost {
// True once terminate() is called, so onExit can tell a deliberate kill (disable/enable-failure) from an
// unexpected worker crash — only the latter is logged as a warning.
private terminated = false;
+ // The liveness probe in flight, if any, and whether onUnresponsive has already fired (it fires once).
+ private probe?: { id: number; timer: ReturnType<typeof setTimeout> };
+ private unresponsiveReported = false;
constructor(
private readonly channel: PluginWorkerChannel,
@@ -119,6 +131,13 @@ export class PluginWorkerHost {
// an error cap-result and the slot frees; the underlying host-side work is NOT cancelled (see
// withCapTimeout). Absent/undefined => no per-call timeout (legacy behavior).
private readonly capTimeoutMs?: number,
+ // After a hook/webhook/search dispatch times out, the host pings the worker and waits this long for
+ // the pong the bootstrap sends before any plugin code runs. A full window with neither the pong nor
+ // any other dispatch result means the worker's event loop is blocked (a slow async handler still
+ // answers), and onUnresponsive is called once so the caller can terminate it. Either absent => no
+ // probing (legacy behavior).
+ private readonly livenessTimeoutMs?: number,
+ private readonly onUnresponsive?: () => void,
) {
this.channel.onMessage(message => this.handleMessage(message));
this.channel.onExit(code => this.handleExit(code));
@@ -172,6 +191,7 @@ export class PluginWorkerHost {
const timer = setTimeout(() => {
this.hookPending.delete(id);
options.onTimeout?.();
+ this.probeLiveness();
settle({ continue: true });
}, options.timeoutMs);
this.hookPending.set(id, { resolve: settle, timer });
@@ -217,6 +237,7 @@ export class PluginWorkerHost {
const timer = setTimeout(() => {
this.webhookPending.delete(id);
options.onTimeout?.();
+ this.probeLiveness();
resolve({ ok: false, status: 504 }); // fail-open: provider already ack'd in async mode
}, options.timeoutMs);
this.webhookPending.set(id, { resolve, timer });
@@ -252,6 +273,7 @@ export class PluginWorkerHost {
return new Promise(resolve => {
const timer = setTimeout(() => {
this.searchPending.delete(id);
+ this.probeLiveness();
resolve({ ok: false, error: 'search timed out' });
}, options.timeoutMs);
this.searchPending.set(id, { resolve, timer });
@@ -331,6 +353,34 @@ export class PluginWorkerHost {
});
}
+ /**
+ * Ask the worker whether its event loop is still turning. A dispatch timeout alone cannot tell a
+ * slow async handler (harmless: its side effect still lands) from a synchronous loop that will never
+ * yield and keeps queueing every later dispatch; the bootstrap answers a ping before plugin code, so
+ * only the latter stays silent past livenessTimeoutMs.
+ */
+ private probeLiveness(): void {
+ if (this.livenessTimeoutMs === undefined || !this.onUnresponsive) return;
+ if (this.dead || this.terminated || this.probe || this.unresponsiveReported) return;
+ const id = this.nextId++;
+ this.armProbe(id);
+ this.channel.postMessage({ kind: 'ping', id });
+ }
+
+ // (Re)start the probe window. The port is FIFO, so a worker working through a burst of dispatches
+ // reads the ping only after that backlog; every result it sends meanwhile proves its loop is turning
+ // and restarts the window (handleMessage). Only a full window with no answer at all is reported.
+ private armProbe(id: number): void {
+ if (this.probe) clearTimeout(this.probe.timer);
+ const timer = setTimeout(() => {
+ this.probe = undefined;
+ if (this.dead || this.terminated || this.unresponsiveReported) return;
+ this.unresponsiveReported = true;
+ this.onUnresponsive?.();
+ }, this.livenessTimeoutMs);
+ this.probe = { id, timer };
+ }
+
/** Tear the worker down. */
terminate(): Promise<void> {
this.terminated = true;
@@ -338,6 +388,9 @@ export class PluginWorkerHost {
}
private handleMessage(message: WorkerToHostMessage): void {
+ // Only answers to host requests count as progress: a synchronous loop can still post log, cap or
+ // subscribe messages, but it cannot finish a dispatch.
+ if (this.probe && PROGRESS_KINDS.has(message.kind)) this.armProbe(this.probe.id);
switch (message.kind) {
case 'ready':
this.ready = true;
@@ -408,6 +461,12 @@ export class PluginWorkerHost {
waiter.resolve({ healthy: message.healthy, message: message.message });
break;
}
+ case 'pong':
+ if (this.probe?.id === message.id) {
+ clearTimeout(this.probe.timer);
+ this.probe = undefined;
+ }
+ break;
case 'search-provider-register':
this.onSearchProviderRegister?.();
break;
@@ -518,6 +577,10 @@ export class PluginWorkerHost {
private handleExit(code: number): void {
this.dead = true;
+ if (this.probe) {
+ clearTimeout(this.probe.timer);
+ this.probe = undefined;
+ }
const error = new Error(`plugin worker exited unexpectedly (code ${code})`);
this.drain(this.readyWaiters, w => {
if (w.timer) clearTimeout(w.timer);
diff --git a/src/core/plugins/sandbox/plugin-worker.integration.spec.ts b/src/core/plugins/sandbox/plugin-worker.integration.spec.ts
index cb75fbaad..8eff9e4e1 100644
--- a/src/core/plugins/sandbox/plugin-worker.integration.spec.ts
+++ b/src/core/plugins/sandbox/plugin-worker.integration.spec.ts
@@ -9,6 +9,7 @@ const FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/echo-plugin.cjs');
const CAP_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/cap-echo-plugin.cjs');
const HOOK_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-plugin.cjs');
const HOOK_HANG_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-hang-plugin.cjs');
+const BUSY_HOOK_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/busy-hook-plugin.cjs');
const RUNAWAY_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/runaway-plugin.cjs');
const CTX_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/ctx-aware-plugin.cjs');
const HOOK_CONFIG_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-config-plugin.cjs');
@@ -145,6 +146,76 @@ describe('plugin worker — real worker_threads round-trip (B1)', () => {
await expect(host.terminate()).resolves.toBeUndefined();
});
+ const probingHost = (onUnresponsive: () => void): PluginWorkerHost =>
+ new PluginWorkerHost(
+ makeChannel(),
+ undefined,
+ () => undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ undefined,
+ 300,
+ onUnresponsive,
+ );
+
+ it('a slow async hook handler answers the liveness probe, so it is not reported', async () => {
+ const onUnresponsive = jest.fn();
+ const host = probingHost(onUnresponsive);
+ try {
+ await host.load(HOOK_HANG_FIXTURE);
+ await host.runLifecycle('onEnable');
+ await flushAsync();
+
+ await host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 200 });
+ await new Promise(resolve => setTimeout(resolve, 900));
+ expect(onUnresponsive).not.toHaveBeenCalled();
+ } finally {
+ await host.terminate();
+ }
+ });
+
+ it('a worker working through a burst of short synchronous handlers is not reported', async () => {
+ const onUnresponsive = jest.fn();
+ const host = probingHost(onUnresponsive);
+ try {
+ await host.load(BUSY_HOOK_FIXTURE);
+ await host.runLifecycle('onEnable');
+ await flushAsync();
+
+ // ~1 s of queued work against a 300 ms hook budget and a 300 ms probe: most dispatches time out
+ // and the ping waits behind the backlog, but results keep arriving the whole time.
+ const burst = Array.from({ length: 100 }, () =>
+ host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 300 }),
+ );
+ await Promise.all(burst);
+ await expect(host.healthCheck(5000)).resolves.toMatchObject({ healthy: true });
+ expect(onUnresponsive).not.toHaveBeenCalled();
+ } finally {
+ await host.terminate();
+ }
+ });
+
+ it('reports a worker whose event loop is blocked, and terminate() reclaims it', async () => {
+ let reported!: () => void;
+ const unresponsive = new Promise<void>(resolve => (reported = resolve));
+ const host = probingHost(() => reported());
+ try {
+ await host.load(RUNAWAY_FIXTURE);
+ const wedged = host.runLifecycle('onEnable');
+ wedged.catch(() => undefined); // terminate() rejects this pending call
+
+ // The dispatch queues behind the spinning onEnable and times out; the ping behind it is never read.
+ await host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 200 });
+ await unresponsive;
+ } finally {
+ await expect(host.terminate()).resolves.toBeUndefined();
+ }
+ });
+
it('a throwing worker hook handler is reported back on the hook-result (not silently swallowed)', async () => {
const host = new PluginWorkerHost(makeChannel(), undefined, () => undefined);
await host.load(HOOK_ERROR_FIXTURE);
diff --git a/src/core/plugins/sandbox/protocol.ts b/src/core/plugins/sandbox/protocol.ts
index 90a573292..1284d315e 100644
--- a/src/core/plugins/sandbox/protocol.ts
+++ b/src/core/plugins/sandbox/protocol.ts
@@ -74,7 +74,10 @@ export type HostToWorkerMessage =
// runs the plugin's search handler and replies with `search-result` (same `id` correlation as
// hook/health-check/webhook). No per-session config: search is a global query; session scoping travels
// inside SearchQuery.sessionIds (scoped by SearchService from the caller's API-key).
- | { kind: 'search'; id: number; query: SearchQuery };
+ | { kind: 'search'; id: number; query: SearchQuery }
+ // Liveness probe, sent after a dispatch times out. The worker bootstrap answers it before any plugin
+ // code runs, so a slow async handler still answers and only a blocked event loop stays silent.
+ | { kind: 'ping'; id: number };
export type WorkerToHostMessage =
| { kind: 'ready' }
@@ -112,6 +115,7 @@ export type WorkerToHostMessage =
// carries the handler's error (handler threw / no handler / etc.).
| { kind: 'search-result'; id: number; ok: true; results: SearchResults }
| { kind: 'search-result'; id: number; ok: false; error: string }
+ | { kind: 'pong'; id: number }
| { kind: 'error'; error: string };
/**
diff --git a/src/core/plugins/sandbox/worker-bootstrap.ts b/src/core/plugins/sandbox/worker-bootstrap.ts
index 5ee7e3c4f..a9e3a8300 100644
--- a/src/core/plugins/sandbox/worker-bootstrap.ts
+++ b/src/core/plugins/sandbox/worker-bootstrap.ts
@@ -61,6 +61,13 @@ let context: Record<string, unknown> | null = null;
let baseConfig: Record<string, unknown> = {};
port.on('message', (message: HostToWorkerMessage) => {
+ // Answered here, before any plugin code, so a pending async handler never delays it. Dispatches
+ // queued ahead of it still do; the host counts their results as progress while it waits, so only a
+ // worker whose event loop is blocked (a synchronous loop in plugin code) goes silent.
+ if (message.kind === 'ping') {
+ send({ kind: 'pong', id: message.id });
+ return;
+ }
if (message.kind === 'cap-result') {
capClient.handleResult(message);
return;
diff --git a/test/fixtures/sandbox/busy-hook-plugin.cjs b/test/fixtures/sandbox/busy-hook-plugin.cjs
new file mode 100644
index 000000000..5d16daf89
--- /dev/null
+++ b/test/fixtures/sandbox/busy-hook-plugin.cjs
@@ -0,0 +1,14 @@
+// Sandbox fixture: a hook handler that does ~10 ms of synchronous work per event and then continues.
+// A burst of dispatches queues behind each other in the worker, so late ones time out while the worker
+// is still answering; the liveness probe must not mistake that backlog for a blocked event loop.
+module.exports = class BusyHookPlugin {
+ async onEnable(ctx) {
+ ctx.registerHook('message:received', () => {
+ const until = Date.now() + 10;
+ while (Date.now() < until) {
+ /* busy */
+ }
+ return { continue: true };
+ });
+ }
+};