Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(plugins): terminate a sandbox worker whose event loop is blocked
Yudhi Armyndharis307a90c1ca44
diff --git a/docs/19-plugin-architecture.md b/docs/19-plugin-architecture.mdindex 23e88d020..1c302c31d 100644--- a/docs/19-plugin-architecture.md+++ b/docs/19-plugin-architecture.md@@ -777,8 +777,12 @@ tier automatically. Key properties:- Each worker has a heap cap (`maxOldGenerationSizeMb`, default **256 MB**) — an OOM terminates theworker, not the host.- A sandboxed hook handler has a **5 s** time budget (`SANDBOX_HOOK_TIMEOUT_MS`); on timeout the host- resolves `{ continue: true }` (fail-open) so a slow/wedged handler never stalls the hook chain. The- same fail-open value drains in-flight hooks if the worker crashes.+ resolves `{ continue: true }` (fail-open), so each sandboxed subscriber delays the hook chain by at most+ 5 s per event. The same fail-open value drains in-flight hooks if the worker crashes. The timeout is+ logged at most once per event per minute and shows as the last hook error in plugin health.+- After a hook, webhook or search dispatch times out, the host pings the worker. A worker whose event+ loop is blocked (a synchronous loop) cannot answer within 5 s, so it is terminated and the plugin is+ set to `ERROR`, as on a crash; a slow async handler answers and is left running.- Lifecycle methods (`load`/`onLoad`/`onEnable`/`onDisable`) and `healthCheck` are bounded by a **30 s**/ **5 s** timeout respectively, so a wedged plugin can't hang an ADMIN enable/disable or the health endpoint.- The worker gets a **minimal allowlisted env** (`NODE_ENV`, `NODE_EXTRA_CA_CERTS`, `TZ`) — host secretsdiff --git a/docs/30-plugin-sandboxing.md b/docs/30-plugin-sandboxing.mdindex 882244c0d..44e043a31 100644--- a/docs/30-plugin-sandboxing.md+++ b/docs/30-plugin-sandboxing.md@@ -45,13 +45,18 @@ from disk is user-installed and sandboxed.host terminates, authenticates and enqueues the delivery first.- **Hook safety.** Hook handlers run in the worker and are dispatched with a **time budget**(`SANDBOX_HOOK_TIMEOUT_MS`, a hardcoded 5s). A slow or wedged handler is skipped (`continue: true`)- so it can never stall the host's hook chain.+ once the budget runs out, so each sandboxed subscriber can delay the hook chain by at most 5 s per+ event, never indefinitely.- **Resource & runaway containment.** Each worker has a heap cap (`maxOldGenerationSizeMb`, a hardcoded256 MB). An OOM terminates the worker, not the host. A wedged **lifecycle** call (load/unload) times outand tears the worker down, and a crash rejects its in-flight calls — the host survives either way. A- runaway **hook** handler (e.g. an infinite synchronous loop) is skipped on the hook timeout so it can't- stall the host's hook chain, but the worker keeps running it (pegging a core) until the plugin is- reloaded or hits the heap cap — it is contained to its own thread, not instantly force-killed.+ runaway **hook** handler (e.g. an infinite synchronous loop) is skipped on the hook timeout. After any+ hook, webhook or search dispatch times out, the host sends the worker a liveness ping that the worker+ answers before running plugin code. A slow async handler answers at once and keeps running. A worker+ still working through a backlog of dispatches reads the ping late, but every result it returns counts+ as an answer. A worker whose event loop is blocked answers nothing for 5 s, so the host terminates it+ and sets the plugin to `ERROR` (no automatic restart, the same as a crash); the operator re-enables it+ once fixed.- **Memory-kind boundary.** The heap cap bounds the V8 heap only. `Buffer`/`ArrayBuffer` allocationsare native memory outside `maxOldGenerationSizeMb`, and worker threads share the host's addressspace, so a plugin that accumulates Buffers grows host RSS until the container's memory limitdiff --git a/src/core/plugins/plugin-loader-sandbox-routing.spec.ts b/src/core/plugins/plugin-loader-sandbox-routing.spec.tsindex c2a92f1b8..d864b02c7 100644--- a/src/core/plugins/plugin-loader-sandbox-routing.spec.ts+++ b/src/core/plugins/plugin-loader-sandbox-routing.spec.ts@@ -21,6 +21,7 @@ class TestableLoader extends PluginLoaderService {capturedOnHookSubscribe?: (event: string, priority?: number) => void;capturedOnLog?: (level: PluginLogLevel, message: string, meta?: Record<string, unknown>) => void;capturedOnWorkerExit?: (code: number, intentional: boolean) => void;+ capturedOnUnresponsive?: () => void;protected createSandboxHost(_capDispatcher?: (verb: string, args: unknown[]) => Promise<unknown>,onHookSubscribe?: (event: string, priority?: number) => void,@@ -29,10 +30,12 @@ class TestableLoader extends PluginLoaderService {_runWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,_onSearchProviderRegister?: () => void,onWorkerExit?: (code: number, intentional: boolean) => void,+ onUnresponsive?: () => void,): PluginWorkerHost {this.capturedOnHookSubscribe = onHookSubscribe;this.capturedOnLog = onLog;this.capturedOnWorkerExit = onWorkerExit;+ this.capturedOnUnresponsive = onUnresponsive;const host: FakeHost = {load: jest.fn().mockResolvedValue(undefined),runLifecycle: jest.fn().mockResolvedValue(undefined),@@ -335,6 +338,92 @@ describe('PluginLoaderService — sandbox hook error surfacing', () => {});});+describe('PluginLoaderService - blocked sandbox worker', () => {+ const loggerOf = (loader: TestableLoader): { warn: jest.Mock } =>+ (loader as unknown as { logger: { warn: jest.Mock } }).logger;+ const hostsOf = (loader: TestableLoader): Map<string, unknown> =>+ (loader as unknown as { sandboxHosts: Map<string, unknown> }).sandboxHosts;+ const storageOf = (loader: TestableLoader): { setPluginStatus: jest.Mock } =>+ (loader as unknown as { pluginStorage: { setPluginStatus: jest.Mock } }).pluginStorage;++ it('terminates an unresponsive worker and leaves the plugin in ERROR with its hooks removed', async () => {+ const loader = makeLoader();+ seed(loader, { builtIn: false, instance: null });+ const hookManager = (loader as unknown as { hookManager: HookManager }).hookManager;+ const unregisterSpy = jest.spyOn(hookManager, 'unregisterPlugin');+ const warnSpy = jest.spyOn(loggerOf(loader), 'warn').mockImplementation(() => undefined);+ await loader.enablePlugin('p1');+ const host = loader.hosts[0];++ loader.capturedOnUnresponsive!();++ const plugin = pluginOf(loader);+ expect(plugin.status).toBe(PluginStatus.ERROR);+ expect(plugin.error).toMatch(/unresponsive/);+ expect(storageOf(loader).setPluginStatus).toHaveBeenCalledWith('p1', PluginStatus.ERROR);+ expect(unregisterSpy).toHaveBeenCalledWith('p1');+ expect(hostsOf(loader).has('p1')).toBe(false);+ expect(host.terminate).toHaveBeenCalled();+ expect(warnSpy).toHaveBeenCalledWith(+ expect.stringContaining('p1'),+ expect.objectContaining({ action: 'sandbox_worker_unresponsive', pluginId: 'p1' }),+ );++ // The deliberate terminate() then exits the worker; that must not undo or repeat the above.+ loader.capturedOnWorkerExit!(1, true);+ expect(pluginOf(loader).status).toBe(PluginStatus.ERROR);+ });++ it('ignores a late report from a worker generation that has already been replaced', async () => {+ const loader = makeLoader();+ seed(loader, { builtIn: false, instance: null });+ await loader.enablePlugin('p1');+ const staleReport = loader.capturedOnUnresponsive!;+ await loader.disablePlugin('p1');+ await loader.enablePlugin('p1');+ const current = loader.hosts[1];++ staleReport();++ expect(pluginOf(loader).status).toBe(PluginStatus.ENABLED);+ expect(hostsOf(loader).get('p1')).toBe(current);+ expect(current.terminate).not.toHaveBeenCalled();+ });++ it('rate-limits the hook timeout warn per event and surfaces the timeout in plugin health', async () => {+ const loader = makeLoader();+ seed(loader, { builtIn: false, instance: null });+ const hookManager = (loader as unknown as { hookManager: HookManager }).hookManager;+ const registerSpy = jest.spyOn(hookManager, 'register');+ await loader.enablePlugin('p1');+ loader.capturedOnHookSubscribe!('message:received');+ const handler = registerSpy.mock.calls.find(c => c[1] === 'message:received')![2];+ loader.hosts[0].dispatchHook.mockImplementation((options: { onTimeout?: () => void }) => {+ options.onTimeout?.();+ return Promise.resolve({ continue: true });+ });+ const warnSpy = jest.spyOn(loggerOf(loader), 'warn').mockImplementation(() => undefined);+ const hookCtx: HookContext = {+ event: 'message:received',+ data: {},+ sessionId: 's1',+ timestamp: new Date(0),+ source: 'Engine',+ };++ await handler(hookCtx);+ await handler(hookCtx);++ const timeoutWarns = warnSpy.mock.calls.filter(+ c => (c[1] as { action?: string } | undefined)?.action === 'sandbox_hook_timeout',+ );+ expect(timeoutWarns).toHaveLength(1);+ const health = await loader.checkPluginHealth('p1');+ expect(health.message).toContain("last hook error in 'message:received'");+ expect(health.message).toContain('timed out');+ });+});+describe('PluginLoaderService — sandbox log relay bounds', () => {const loggerOf = (loader: TestableLoader): { warn: jest.Mock; log: jest.Mock } =>(loader as unknown as { logger: { warn: jest.Mock; log: jest.Mock } }).logger;diff --git a/src/core/plugins/plugin-loader.service.ts b/src/core/plugins/plugin-loader.service.tsindex 482b4c34f..3e47f67a0 100644--- a/src/core/plugins/plugin-loader.service.ts+++ b/src/core/plugins/plugin-loader.service.ts@@ -54,8 +54,16 @@ const SANDBOX_MAX_INFLIGHT_CAPS = 32;const SANDBOX_CAP_TIMEOUT_MS = 30000;/**- * Host process.env keys an untrusted plugin worker is allowed to see. Everything else — secrets like- * API_MASTER_KEY, API_KEY_PEPPER, the DATABASE_/REDIS_ vars, DOCKER_HOST — is left out of the worker's+ * How long a worker has to answer the liveness ping the host sends after a dispatch times out. A worker+ * stuck in a synchronous loop is terminated about 5 s after its first dispatch timeout (about 10 s after+ * a stalled hook or ingress dispatch, 15 s after a stalled search); a slow async handler answers at once+ * and is never affected.+ */+const SANDBOX_LIVENESS_TIMEOUT_MS = 5000;++/**+ * Host process.env keys an untrusted plugin worker is allowed to see. Everything else (secrets like+ * API_MASTER_KEY, API_KEY_PEPPER, the DATABASE_/REDIS_ vars, DOCKER_HOST) is left out of the worker's* process.env object. That keeps secrets from being handed over ambiently; it does not put them out of* reach, since a thread in the same OS process can read /proc/self/environ and the data directory. The* worker is a thread, so it needs no PATH to start and require() resolves via module paths, not env.@@ -174,6 +182,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstraprunWithHookGuard,onSearchProviderRegister,onWorkerExit,+ onUnresponsive,) =>this.createSandboxHost(capDispatcher,@@ -183,6 +192,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstraprunWithHookGuard,onSearchProviderRegister,onWorkerExit,+ onUnresponsive,),// Exported from this module (specs import it here); passed so the bridge never imports back.resolvePluginMainPath,@@ -372,6 +382,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstraprunWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,onSearchProviderRegister?: () => void,onWorkerExit?: (code: number, intentional: boolean) => void,+ onUnresponsive?: () => void,): PluginWorkerHost {const workerEntry = path.join(__dirname, 'sandbox', 'worker-bootstrap.js');return new PluginWorkerHost(@@ -390,6 +401,8 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstraponSearchProviderRegister,onWorkerExit,this.configService.get<number>('plugins.capTimeoutMs') ?? SANDBOX_CAP_TIMEOUT_MS,+ SANDBOX_LIVENESS_TIMEOUT_MS,+ onUnresponsive,);}diff --git a/src/core/plugins/plugin-sandbox-bridge.ts b/src/core/plugins/plugin-sandbox-bridge.tsindex 395569909..a3fafd592 100644--- a/src/core/plugins/plugin-sandbox-bridge.ts+++ b/src/core/plugins/plugin-sandbox-bridge.ts@@ -66,6 +66,7 @@ type CreateSandboxHostFn = (runWithHookGuard?: (inFlightEvents: string[], run: () => Promise<unknown>) => Promise<unknown>,onSearchProviderRegister?: () => void,onWorkerExit?: (code: number, intentional: boolean) => void,+ onUnresponsive?: () => void,) => PluginWorkerHost;/**@@ -122,20 +123,23 @@ export class PluginSandboxBridge {event: string,error: string,rateLimit: Map<string, { lastAt: number; suppressed: number }>,+ action: 'sandbox_hook_error' | 'sandbox_hook_timeout' = 'sandbox_hook_error',): void {this.lastSandboxHookError.set(pluginId, { event, error, at: new Date() });const now = Date.now();- const state = rateLimit.get(event);+ // Errors and timeouts are windowed apart, so a burst of one never hides the other.+ const key = `${action}:${event}`;+ const state = rateLimit.get(key);if (state && now - state.lastAt < SANDBOX_HOOK_ERROR_LOG_INTERVAL_MS) {state.suppressed++;return;}const suppressed = state?.suppressed ?? 0;- rateLimit.set(event, { lastAt: now, suppressed: 0 });+ rateLimit.set(key, { lastAt: now, suppressed: 0 });this.logger.warn(`Sandboxed plugin ${pluginId} hook '${event}' handler failed: ${error}`, {pluginId,event,- action: 'sandbox_hook_error',+ action,...(suppressed > 0 ? { suppressed } : {}),});}@@ -340,7 +344,21 @@ export class PluginSandboxBridge {const onWorkerExit = this.buildWorkerExitHandler(pluginId, logRelayState);- const host = this.createHost(+ // A worker whose event loop stays blocked after a dispatch timeout (a synchronous loop in plugin+ // code) would otherwise keep running and queue every later dispatch for the full budget each.+ // Same policy as a crash: terminate it and leave the plugin in ERROR for the operator to re-enable.+ // The generation check keeps a late report from marking a replacement worker.+ const onUnresponsive = (): void => {+ if (this.sandboxHosts.get(pluginId) !== host) return;+ this.markWorkerFailed(pluginId, 'worker unresponsive: event loop blocked past the liveness probe');+ this.logger.warn(`Sandboxed plugin ${pluginId} worker is unresponsive; terminating it`, {+ pluginId,+ action: 'sandbox_worker_unresponsive',+ });+ void host.terminate().catch(() => undefined);+ };++ const host: PluginWorkerHost = this.createHost((verb, args) => dispatchCapabilityVerb(context, verb, args),onHookSubscribe,onWebhookSubscribe,@@ -350,6 +368,7 @@ export class PluginSandboxBridge {(events, run) => this.hookManager.runInFlight(events as HookEvent[], run),onSearchProviderRegister,onWorkerExit,+ onUnresponsive,);this.sandboxHosts.set(pluginId, host);try {@@ -508,12 +527,16 @@ export class PluginSandboxBridge {// against re-firing any event of it (not only this one) once it returns to the host.inFlight: this.hookManager.currentInFlight(),timeoutMs: SANDBOX_HOOK_TIMEOUT_MS,+ // Rate-limited like a handler error: a plugin that times out on every message must be+ // visible (log + health) without writing one line per event.onTimeout: () =>- this.logger.warn(`Sandboxed plugin ${pluginId} hook '${event}' timed out`, {+ this.recordSandboxHookError(pluginId,event,- action: 'sandbox_hook_timeout',- }),+ `timed out after ${SANDBOX_HOOK_TIMEOUT_MS}ms`,+ hookErrorLogState,+ 'sandbox_hook_timeout',+ ),}).then(result => {// The worker reports (not throws) a hook-handler failure: surface it host-side instead@@ -590,11 +613,23 @@ export class PluginSandboxBridge {};}+ // Shared ERROR transition for a crashed or unresponsive worker: mark the plugin ERROR, drop its hook+ // shims and the dead host. The exit handler releases the search provider.+ private markWorkerFailed(pluginId: string, reason: string): void {+ const plugin = this.plugins.get(pluginId);+ if (plugin) {+ plugin.status = PluginStatus.ERROR;+ plugin.error = reason;+ this.pluginStorage.setPluginStatus(pluginId, PluginStatus.ERROR);+ }+ this.hookManager.unregisterPlugin(pluginId);+ this.sandboxHosts.delete(pluginId);+ }+// A worker that crashes AFTER a successful enable is otherwise invisible to the loader (handleExit only// drains in-flight calls). Drop the plugin's search-provider entry so the registry falls back to// builtin-fts instead of routing every /search to a dead worker (auto mode would otherwise pin the dead- // provider ACTIVE). Mirrors the enable-failure cleanup. Broader crash-lifecycle cleanup (status, hooks)- // is a pre-existing gap for all bridges and out of scope here.+ // provider ACTIVE). Mirrors the enable-failure cleanup.private buildWorkerExitHandler(pluginId: string,logRelayState: SandboxLogRelayState,@@ -619,14 +654,7 @@ export class PluginSandboxBridge {// unregister the hook shims (so they don't keep dispatching into the dead worker) + mark the// plugin ERROR so the dashboard reflects reality. The dispatchHook/dispatchWebhook dead-checks// fail-fast; this cleanup is the root-cause fix (it also makes the shim's !liveHost guard fire).- const crashed = this.plugins.get(pluginId);- if (crashed) {- crashed.status = PluginStatus.ERROR;- crashed.error = `worker exited unexpectedly (code ${code})`;- this.pluginStorage.setPluginStatus(pluginId, PluginStatus.ERROR);- }- this.hookManager.unregisterPlugin(pluginId);- this.sandboxHosts.delete(pluginId);+ this.markWorkerFailed(pluginId, `worker exited unexpectedly (code ${code})`);this.logger.warn(`Sandboxed plugin ${pluginId} worker exited unexpectedly (code ${code})`, {pluginId,code,diff --git a/src/core/plugins/sandbox/plugin-worker-host.spec.ts b/src/core/plugins/sandbox/plugin-worker-host.spec.tsindex 3f5c1b6d5..23d72c5fd 100644--- a/src/core/plugins/sandbox/plugin-worker-host.spec.ts+++ b/src/core/plugins/sandbox/plugin-worker-host.spec.ts@@ -863,4 +863,159 @@ describe('PluginWorkerHost', () => {}});});++ describe('liveness probe after a dispatch timeout', () => {+ const PROBE_MS = 1000;+ const pings = (ch: FakeChannel) =>+ ch.sent.filter((m): m is Extract<HostToWorkerMessage, { kind: 'ping' }> => m.kind === 'ping');+ const probingHost = (ch: FakeChannel, onUnresponsive: () => void) =>+ new PluginWorkerHost(+ ch,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ PROBE_MS,+ onUnresponsive,+ );+ const hook = (host: PluginWorkerHost) =>+ host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 500 });+ const webhook = (host: PluginWorkerHost) =>+ host.dispatchWebhook({+ instanceId: 'i',+ route: 'r',+ method: 'POST',+ headers: {},+ query: {},+ body: '',+ rawBody: '',+ verified: true,+ deliveryId: 'd',+ timeoutMs: 500,+ });++ beforeEach(() => jest.useFakeTimers());+ afterEach(() => jest.useRealTimers());++ it('posts one ping on a hook timeout, and none while that probe is still in flight', async () => {+ const ch = new FakeChannel();+ const host = probingHost(ch, jest.fn());+ const first = hook(host);+ const second = hook(host);+ jest.advanceTimersByTime(500);+ await Promise.all([first, second]);+ expect(pings(ch)).toHaveLength(1);+ });++ it('a pong clears the probe: a slow async handler is not reported', async () => {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ ch.reply({ kind: 'pong', id: pings(ch)[0].id });+ jest.advanceTimersByTime(PROBE_MS * 3);+ expect(onUnresponsive).not.toHaveBeenCalled();++ // The next timeout probes again rather than being suppressed by the answered one.+ const next = hook(host);+ jest.advanceTimersByTime(500);+ await next;+ expect(pings(ch)).toHaveLength(2);+ });++ it('reports a worker that never answers the probe, exactly once', async () => {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ jest.advanceTimersByTime(PROBE_MS);+ expect(onUnresponsive).toHaveBeenCalledTimes(1);++ const again = hook(host);+ jest.advanceTimersByTime(500 + PROBE_MS * 2);+ await again;+ expect(onUnresponsive).toHaveBeenCalledTimes(1);+ });++ it('a worker still answering a backlog is not reported while its pong waits behind it', async () => {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ const hookId = (ch.sent.find(m => m.kind === 'hook') as { id: number }).id;++ // Late results keep arriving, each inside the window: the event loop is turning.+ for (let i = 0; i < 3; i++) {+ jest.advanceTimersByTime(PROBE_MS - 100);+ ch.reply({ kind: 'hook-result', id: hookId, continue: true });+ }+ expect(onUnresponsive).not.toHaveBeenCalled();++ // A full window with no answer at all is still reported.+ jest.advanceTimersByTime(PROBE_MS);+ expect(onUnresponsive).toHaveBeenCalledTimes(1);+ });++ it('log lines do not count as progress: a synchronous loop can emit them', async () => {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ jest.advanceTimersByTime(PROBE_MS - 100);+ ch.reply({ kind: 'log', level: 'log', message: 'still spinning' });+ jest.advanceTimersByTime(100);+ expect(onUnresponsive).toHaveBeenCalledTimes(1);+ });++ it('webhook and search timeouts probe too', async () => {+ for (const dispatch of [+ webhook,+ (host: PluginWorkerHost) => host.dispatchSearch({ query: { q: 'x' }, timeoutMs: 500 }),+ ]) {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = dispatch(host);+ jest.advanceTimersByTime(500);+ await pending;+ expect(pings(ch)).toHaveLength(1);+ jest.advanceTimersByTime(PROBE_MS);+ expect(onUnresponsive).toHaveBeenCalledTimes(1);+ }+ });++ it('does not probe when no liveness budget is configured', async () => {+ const ch = new FakeChannel();+ const host = new PluginWorkerHost(ch);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ expect(pings(ch)).toHaveLength(0);+ });++ it('a worker exit cancels a probe in flight', async () => {+ const ch = new FakeChannel();+ const onUnresponsive = jest.fn();+ const host = probingHost(ch, onUnresponsive);+ const pending = hook(host);+ jest.advanceTimersByTime(500);+ await pending;+ ch.crash(1);+ jest.advanceTimersByTime(PROBE_MS * 2);+ expect(onUnresponsive).not.toHaveBeenCalled();+ });+ });});diff --git a/src/core/plugins/sandbox/plugin-worker-host.ts b/src/core/plugins/sandbox/plugin-worker-host.tsindex 0e1708ff8..ec2d8bc6c 100644--- a/src/core/plugins/sandbox/plugin-worker-host.ts+++ b/src/core/plugins/sandbox/plugin-worker-host.ts@@ -21,6 +21,15 @@ const SEND_CAP_VERBS: ReadonlySet<string> = new Set(['messages.sendText', 'messa/** Send-verb budget as a multiple of capTimeoutMs (default 30s → 120s: the 30s media download + upload headroom). */const SEND_CAP_TIMEOUT_FACTOR = 4;+/** Worker replies that finish a host request; one arriving during a liveness probe restarts its window. */+const PROGRESS_KINDS: ReadonlySet<WorkerToHostMessage['kind']> = new Set([+ 'lifecycle-result',+ 'hook-result',+ 'webhook-result',+ 'health-result',+ 'search-result',+]);+/*** Host-side driver for a single untrusted plugin running in a worker. Owns the request/response* correlation over a {@link PluginWorkerChannel}: it posts `load`/`lifecycle` messages and resolves@@ -84,6 +93,9 @@ export class PluginWorkerHost {// True once terminate() is called, so onExit can tell a deliberate kill (disable/enable-failure) from an// unexpected worker crash — only the latter is logged as a warning.private terminated = false;+ // The liveness probe in flight, if any, and whether onUnresponsive has already fired (it fires once).+ private probe?: { id: number; timer: ReturnType<typeof setTimeout> };+ private unresponsiveReported = false;constructor(private readonly channel: PluginWorkerChannel,@@ -119,6 +131,13 @@ export class PluginWorkerHost {// an error cap-result and the slot frees; the underlying host-side work is NOT cancelled (see// withCapTimeout). Absent/undefined => no per-call timeout (legacy behavior).private readonly capTimeoutMs?: number,+ // After a hook/webhook/search dispatch times out, the host pings the worker and waits this long for+ // the pong the bootstrap sends before any plugin code runs. A full window with neither the pong nor+ // any other dispatch result means the worker's event loop is blocked (a slow async handler still+ // answers), and onUnresponsive is called once so the caller can terminate it. Either absent => no+ // probing (legacy behavior).+ private readonly livenessTimeoutMs?: number,+ private readonly onUnresponsive?: () => void,) {this.channel.onMessage(message => this.handleMessage(message));this.channel.onExit(code => this.handleExit(code));@@ -172,6 +191,7 @@ export class PluginWorkerHost {const timer = setTimeout(() => {this.hookPending.delete(id);options.onTimeout?.();+ this.probeLiveness();settle({ continue: true });}, options.timeoutMs);this.hookPending.set(id, { resolve: settle, timer });@@ -217,6 +237,7 @@ export class PluginWorkerHost {const timer = setTimeout(() => {this.webhookPending.delete(id);options.onTimeout?.();+ this.probeLiveness();resolve({ ok: false, status: 504 }); // fail-open: provider already ack'd in async mode}, options.timeoutMs);this.webhookPending.set(id, { resolve, timer });@@ -252,6 +273,7 @@ export class PluginWorkerHost {return new Promise(resolve => {const timer = setTimeout(() => {this.searchPending.delete(id);+ this.probeLiveness();resolve({ ok: false, error: 'search timed out' });}, options.timeoutMs);this.searchPending.set(id, { resolve, timer });@@ -331,6 +353,34 @@ export class PluginWorkerHost {});}+ /**+ * Ask the worker whether its event loop is still turning. A dispatch timeout alone cannot tell a+ * slow async handler (harmless: its side effect still lands) from a synchronous loop that will never+ * yield and keeps queueing every later dispatch; the bootstrap answers a ping before plugin code, so+ * only the latter stays silent past livenessTimeoutMs.+ */+ private probeLiveness(): void {+ if (this.livenessTimeoutMs === undefined || !this.onUnresponsive) return;+ if (this.dead || this.terminated || this.probe || this.unresponsiveReported) return;+ const id = this.nextId++;+ this.armProbe(id);+ this.channel.postMessage({ kind: 'ping', id });+ }++ // (Re)start the probe window. The port is FIFO, so a worker working through a burst of dispatches+ // reads the ping only after that backlog; every result it sends meanwhile proves its loop is turning+ // and restarts the window (handleMessage). Only a full window with no answer at all is reported.+ private armProbe(id: number): void {+ if (this.probe) clearTimeout(this.probe.timer);+ const timer = setTimeout(() => {+ this.probe = undefined;+ if (this.dead || this.terminated || this.unresponsiveReported) return;+ this.unresponsiveReported = true;+ this.onUnresponsive?.();+ }, this.livenessTimeoutMs);+ this.probe = { id, timer };+ }+/** Tear the worker down. */terminate(): Promise<void> {this.terminated = true;@@ -338,6 +388,9 @@ export class PluginWorkerHost {}private handleMessage(message: WorkerToHostMessage): void {+ // Only answers to host requests count as progress: a synchronous loop can still post log, cap or+ // subscribe messages, but it cannot finish a dispatch.+ if (this.probe && PROGRESS_KINDS.has(message.kind)) this.armProbe(this.probe.id);switch (message.kind) {case 'ready':this.ready = true;@@ -408,6 +461,12 @@ export class PluginWorkerHost {waiter.resolve({ healthy: message.healthy, message: message.message });break;}+ case 'pong':+ if (this.probe?.id === message.id) {+ clearTimeout(this.probe.timer);+ this.probe = undefined;+ }+ break;case 'search-provider-register':this.onSearchProviderRegister?.();break;@@ -518,6 +577,10 @@ export class PluginWorkerHost {private handleExit(code: number): void {this.dead = true;+ if (this.probe) {+ clearTimeout(this.probe.timer);+ this.probe = undefined;+ }const error = new Error(`plugin worker exited unexpectedly (code ${code})`);this.drain(this.readyWaiters, w => {if (w.timer) clearTimeout(w.timer);diff --git a/src/core/plugins/sandbox/plugin-worker.integration.spec.ts b/src/core/plugins/sandbox/plugin-worker.integration.spec.tsindex cb75fbaad..8eff9e4e1 100644--- a/src/core/plugins/sandbox/plugin-worker.integration.spec.ts+++ b/src/core/plugins/sandbox/plugin-worker.integration.spec.ts@@ -9,6 +9,7 @@ const FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/echo-plugin.cjs');const CAP_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/cap-echo-plugin.cjs');const HOOK_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-plugin.cjs');const HOOK_HANG_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-hang-plugin.cjs');+const BUSY_HOOK_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/busy-hook-plugin.cjs');const RUNAWAY_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/runaway-plugin.cjs');const CTX_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/ctx-aware-plugin.cjs');const HOOK_CONFIG_FIXTURE = path.resolve(ROOT, 'test/fixtures/sandbox/hook-config-plugin.cjs');@@ -145,6 +146,76 @@ describe('plugin worker — real worker_threads round-trip (B1)', () => {await expect(host.terminate()).resolves.toBeUndefined();});+ const probingHost = (onUnresponsive: () => void): PluginWorkerHost =>+ new PluginWorkerHost(+ makeChannel(),+ undefined,+ () => undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ undefined,+ 300,+ onUnresponsive,+ );++ it('a slow async hook handler answers the liveness probe, so it is not reported', async () => {+ const onUnresponsive = jest.fn();+ const host = probingHost(onUnresponsive);+ try {+ await host.load(HOOK_HANG_FIXTURE);+ await host.runLifecycle('onEnable');+ await flushAsync();++ await host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 200 });+ await new Promise(resolve => setTimeout(resolve, 900));+ expect(onUnresponsive).not.toHaveBeenCalled();+ } finally {+ await host.terminate();+ }+ });++ it('a worker working through a burst of short synchronous handlers is not reported', async () => {+ const onUnresponsive = jest.fn();+ const host = probingHost(onUnresponsive);+ try {+ await host.load(BUSY_HOOK_FIXTURE);+ await host.runLifecycle('onEnable');+ await flushAsync();++ // ~1 s of queued work against a 300 ms hook budget and a 300 ms probe: most dispatches time out+ // and the ping waits behind the backlog, but results keep arriving the whole time.+ const burst = Array.from({ length: 100 }, () =>+ host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 300 }),+ );+ await Promise.all(burst);+ await expect(host.healthCheck(5000)).resolves.toMatchObject({ healthy: true });+ expect(onUnresponsive).not.toHaveBeenCalled();+ } finally {+ await host.terminate();+ }+ });++ it('reports a worker whose event loop is blocked, and terminate() reclaims it', async () => {+ let reported!: () => void;+ const unresponsive = new Promise<void>(resolve => (reported = resolve));+ const host = probingHost(() => reported());+ try {+ await host.load(RUNAWAY_FIXTURE);+ const wedged = host.runLifecycle('onEnable');+ wedged.catch(() => undefined); // terminate() rejects this pending call++ // The dispatch queues behind the spinning onEnable and times out; the ping behind it is never read.+ await host.dispatchHook({ event: 'message:received', data: {}, source: 'Engine', timeoutMs: 200 });+ await unresponsive;+ } finally {+ await expect(host.terminate()).resolves.toBeUndefined();+ }+ });+it('a throwing worker hook handler is reported back on the hook-result (not silently swallowed)', async () => {const host = new PluginWorkerHost(makeChannel(), undefined, () => undefined);await host.load(HOOK_ERROR_FIXTURE);diff --git a/src/core/plugins/sandbox/protocol.ts b/src/core/plugins/sandbox/protocol.tsindex 90a573292..1284d315e 100644--- a/src/core/plugins/sandbox/protocol.ts+++ b/src/core/plugins/sandbox/protocol.ts@@ -74,7 +74,10 @@ export type HostToWorkerMessage =// runs the plugin's search handler and replies with `search-result` (same `id` correlation as// hook/health-check/webhook). No per-session config: search is a global query; session scoping travels// inside SearchQuery.sessionIds (scoped by SearchService from the caller's API-key).- | { kind: 'search'; id: number; query: SearchQuery };+ | { kind: 'search'; id: number; query: SearchQuery }+ // Liveness probe, sent after a dispatch times out. The worker bootstrap answers it before any plugin+ // code runs, so a slow async handler still answers and only a blocked event loop stays silent.+ | { kind: 'ping'; id: number };export type WorkerToHostMessage =| { kind: 'ready' }@@ -112,6 +115,7 @@ export type WorkerToHostMessage =// carries the handler's error (handler threw / no handler / etc.).| { kind: 'search-result'; id: number; ok: true; results: SearchResults }| { kind: 'search-result'; id: number; ok: false; error: string }+ | { kind: 'pong'; id: number }| { kind: 'error'; error: string };/**diff --git a/src/core/plugins/sandbox/worker-bootstrap.ts b/src/core/plugins/sandbox/worker-bootstrap.tsindex 5ee7e3c4f..a9e3a8300 100644--- a/src/core/plugins/sandbox/worker-bootstrap.ts+++ b/src/core/plugins/sandbox/worker-bootstrap.ts@@ -61,6 +61,13 @@ let context: Record<string, unknown> | null = null;let baseConfig: Record<string, unknown> = {};port.on('message', (message: HostToWorkerMessage) => {+ // Answered here, before any plugin code, so a pending async handler never delays it. Dispatches+ // queued ahead of it still do; the host counts their results as progress while it waits, so only a+ // worker whose event loop is blocked (a synchronous loop in plugin code) goes silent.+ if (message.kind === 'ping') {+ send({ kind: 'pong', id: message.id });+ return;+ }if (message.kind === 'cap-result') {capClient.handleResult(message);return;diff --git a/test/fixtures/sandbox/busy-hook-plugin.cjs b/test/fixtures/sandbox/busy-hook-plugin.cjsnew file mode 100644index 000000000..5d16daf89--- /dev/null+++ b/test/fixtures/sandbox/busy-hook-plugin.cjs@@ -0,0 +1,14 @@+// Sandbox fixture: a hook handler that does ~10 ms of synchronous work per event and then continues.+// A burst of dispatches queues behind each other in the worker, so late ones time out while the worker+// is still answering; the liveness probe must not mistake that backlog for a blocked event loop.+module.exports = class BusyHookPlugin {+ async onEnable(ctx) {+ ctx.registerHook('message:received', () => {+ const until = Date.now() + 10;+ while (Date.now() < until) {+ /* busy */+ }+ return { continue: true };+ });+ }+};