The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
fix(server): allow proved idle waits and verify historical cleanup (#16633)
Devin Foley4415665d67b1
diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.mdindex 82f3652fbc..e1d6dd1711 100644--- a/doc/idle-sleep-safety.md+++ b/doc/idle-sleep-safety.md@@ -38,6 +38,64 @@ report, so it counts concurrent writes without blocking on its own scan.## Scheduled routine wake protocol+Historical failed ephemeral leases and terminal custom-image setup sessions can+become idle only after the host observes deletion of their exact provider+resource. Cloud checks bounded batches on already-awake instances, outside the+readiness path. The optional `environmentInspectLease` plugin RPC is read-only:+it must never reconnect, start, stop, or destroy an allocation. Older workers+without this method remain conservative. Daytona accepts its SDK's typed+not-found response; authentication and transport failures are not deletion+proof. Setup checks require the recorded provider-config fingerprint; lease+checks use the recorded plugin and scoped config, never a replacement binding.+For legacy secret references without a version manifest, only an unrotated+initial version with a matching resolved-value hash establishes credential+identity. Rotated/mutable credentials and ambient Daytona keys defer proof;+not-found under another account cannot establish historical cleanup. Inspection+also rejects an ambient API endpoint override when no endpoint was recorded.+The manager permits at most two outstanding inspection RPCs across workers.+Caller timeouts retain their slots until the actual worker reply or exit;+timeouts cannot accumulate additional provider reads in later sweeps.++Host-owned verification metadata binds the company, environment, resource,+provider handle, optional run, and exact row version. Concurrent resume,+replacement, cleanup, or configuration changes invalidate verification.+Provider-supplied metadata cannot supply this marker. Inspection preserves+failure outcomes and history and does not create a run-continuation receipt.+Normal setup cancellation and timeout preserve the host's recorded config+fingerprint and RPC company scope for later read-only verification.+Retained allocations, unfinished runs/accounting/retries, export intents and+live setup credentials continue to block sleep.++Pending human-only connection setup can wait asleep when no active enabled+connection of the requested purpose exists. Available access reblocks sleep so+the existing reconciliation path can run; answering a setup card continues to+use the normal admission and wake path.+The delivery worker uses the same passive-wait proof to release its local+admission hold. On an already-awake instance it keeps its existing five-second+maintenance cadence, without creating a wake alarm. This catches newly available+access even when its write does not notify the connection queue. Pending real+deliveries and their downstream wake requests still block sleep.+An owned drain pauses passive-only maintenance before it admits tracked work,+so a slow pre-sleep backup can finish with an unchanged generation. New producer+notifications still preempt the maintenance timer and invalidate that backup.++An unconfigured Slack draft can also wait for browser setup when its matching+channel connection is disabled and still a draft. It must have no provider+identity, prior activation/activity, or setup error. Partially configured,+enabled, foreign, and other-provider endpoints remain blockers; accepted chat+deliveries retain their independent checks.+An optional Slack slash-command name is configuration only and does not create+background work. Other setup fields still retain their blockers.++Cloud also reconciles missed routine issue-status hooks in bounded batches on+awake instances. Exact company/routine/issue/run ownership and row locks are+required. The normal routine service records completion or transient failure+from the current issue status; a resumed issue can use its existing continuation+rules. This repairs stale handoff bookkeeping without exempting unprocessed+routine runs or consuming deferred comment requests.+Lock contention and retryable transaction failures defer only that candidate;+other repairs continue, and later sweeps retry the deferred row.+Verified signed Cloud control callers may opt into`GET /api/instance/task-drain?idleSleepSafety=1&scheduledWake=1&ownerId=<owner>`.Tenant tokens, browser instance administrators, local implicit actors, and agents@@ -235,7 +293,28 @@ including token refresh after a quota response times out. Credential-recoveryjournals, partial writes and probes block sleep across restart. Empty companydirectories left after successful recovery do not; invalid paths and readfailures remain unknown. Credentials and connection records are retained.-MCP, chat and other connections still block until they have a wake contract.+Disabled draft connections and saved outbound HTTP MCP connections have no+independent listener. They can permit sleep while accepted calls and credential+work still pass the other fences. MCP connections with a retained GitHub+webhook binding, local stdio processes, channels, and other active transports+remain blockers until their inbound work has a supported wake contract.++With the issue-scheduling capability, pending questions, confirmations,+checkbox confirmations and task suggestions can be passive waits. Their+producer must have finished without retry or accounting debt. The response+audience and continuation policy stay unchanged. Unknown pending siblings,+foreign-company records and unfinished producers block sleep. A resolved card+does not make an otherwise runnable issue passive. The schedule generation+includes response identity and audience, including when only a routine owns+the next alarm.+Merge confirmations remain blockers because GitHub polling can accept them+without a human response and does not yet have a durable wake contract.++Known board escalation actions for stranded work, missing disposition and a+lost review path can also be passive. They require a blocked source in the same+company, board ownership, no automatic wake, no monitor or timeout, and no live+execution binding. The action remains visible and unresolved. Other recovery+causes and autonomous owners remain blockers.A failed ephemeral lease is history only with successful cleanup, a releasetimestamp, and an exact provider termination receipt bound to its company, run,@@ -243,6 +322,10 @@ lease and resource. Its run must be finished without retry or accounting debt.Pending cleanup claims and export/stop-and-retain intents still block. Failurelabels or cleanup success without that proof do not authorize sleep. Leasehistory is retained; this check performs no provider teardown or data updates.+A released local ephemeral bookkeeping lease needs no remote termination+receipt when its exact run has finished and both its provider and environment+are local. It must have no provider resource reference or pending cleanup,+export or retention intent. Unknown drivers and remote resources still block.Finished login-session history permits sleep only after its promotion claim iscleared and any retained provider reference has a matching released, expired,@@ -265,10 +348,19 @@ Pending deliveries, missingcompletion evidence, and unfinished handoffs still block. Failed deliverieswithout a pending target are history; other work checks still apply.-Every enabled plugin blocks sleep. A version label does not prove that arbitrary-worker code has no background activity. No plugin approval or configuration-bypass is supported. Scheduled and externally triggered work needs a durable-host wake mechanism before this restriction can be relaxed.+Processed connector-event receipts with a processing timestamp are idempotency+history. Delivered connection-intent outboxes also permit sleep once their+exact accepted wakes finish. A system-retired connection card parked behind a+pending response keeps its undelivered outbox intact; the normal response path+resumes delivery. Pending event application, unfinished wakes and all other+execution checks still block. None of these exceptions fabricates completion+or removes a durable receipt.++Enabled plugins require the current worker's verified idle drain. A version+label does not prove that arbitrary worker code has no background activity.+No plugin approval or configuration bypass is supported. Scheduled and+externally triggered work needs a durable host wake mechanism before it can+permit sleep.This is a single-process admission protocol. A restart clears the owner. Thehost must serialize sleep with deploy/wake operations and bind its final checkdiff --git a/packages/plugins/paperclip-plugin-fake-sandbox/src/plugin.ts b/packages/plugins/paperclip-plugin-fake-sandbox/src/plugin.tsindex 30e7001d5a..4b7d7ef273 100644--- a/packages/plugins/paperclip-plugin-fake-sandbox/src/plugin.ts+++ b/packages/plugins/paperclip-plugin-fake-sandbox/src/plugin.ts@@ -13,6 +13,8 @@ import type {PluginEnvironmentDeleteTemplateParams,PluginEnvironmentDeleteTemplateResult,PluginEnvironmentDestroyLeaseParams,+ PluginEnvironmentInspectLeaseParams,+ PluginEnvironmentLeaseObservation,PluginEnvironmentExecuteParams,PluginEnvironmentExecuteResult,PluginEnvironmentGetInteractiveSetupParams,@@ -344,6 +346,10 @@ const plugin = definePlugin({}},+ async onEnvironmentInspectLease(params: PluginEnvironmentInspectLeaseParams): Promise<PluginEnvironmentLeaseObservation> {+ return { providerLeaseId: params.providerLeaseId, state: leases.has(params.providerLeaseId) ? "active" : "destroyed" };+ },+async onEnvironmentDestroyLease(params: PluginEnvironmentDestroyLeaseParams,): Promise<void> {diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.tsindex 3bd6596044..1803bcd1db 100644--- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts+++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts@@ -40,6 +40,46 @@ import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvirimport manifest from "./manifest.js";import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js";+describe("Daytona read-only resource inspection", () => {+ const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",+ providerLeaseId: "sandbox-123", config: { apiKey: "fixture-not-a-key" } };+ afterEach(() => { vi.clearAllMocks(); vi.unstubAllEnvs(); __resetDaytonaSandboxHandleCacheForTest(); });+ it.each(["started", "stopped"])("observes a %s sandbox without lifecycle calls", async state => {+ const sandbox = createMockSandbox({ state }); mockGet.mockResolvedValue(sandbox);+ expect(await plugin.definition.onEnvironmentInspectLease!(params)).toEqual({ providerLeaseId: "sandbox-123", state: state === "stopped" ? "stopped" : "active" });+ expect(mockGet).toHaveBeenLastCalledWith("sandbox-123");+ for (const method of [sandbox.start, sandbox.stop, sandbox.delete, sandbox.archive, sandbox.refreshData]) expect(method).not.toHaveBeenCalled();+ expect(mockCreate).not.toHaveBeenCalled();+ });+ it("accepts only typed provider not-found as deletion proof", async () => {+ __resetDaytonaSandboxHandleCacheForTest();+ mockGet.mockRejectedValue(new MockDaytonaNotFoundError("not found"));+ expect(await plugin.definition.onEnvironmentInspectLease!(params)).toEqual({ providerLeaseId: "sandbox-123", state: "destroyed" });+ expect(await plugin.definition.onIdleDrain!(new AbortController().signal)).toBe("none");+ });+ it("does not use an ambient account credential to certify historical deletion", async () => {+ expect(await plugin.definition.onEnvironmentInspectLease!({ ...params, config: {} })).toEqual({ providerLeaseId: "sandbox-123", state: "unknown" });+ expect(mockGet).not.toHaveBeenCalled();+ });+ it.each(["DAYTONA_API_URL", "DAYTONA_SERVER_URL"])("does not substitute ambient %s for the recorded API endpoint", async variable => {+ vi.stubEnv(variable, "https://replacement.example.test/api");+ expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("unknown");+ expect(mockGet).not.toHaveBeenCalled();+ mockGet.mockRejectedValue(new MockDaytonaNotFoundError("gone"));+ expect((await plugin.definition.onEnvironmentInspectLease!({ ...params, config: { ...params.config, apiUrl: "https://recorded.example.test/api" } })).state).toBe("destroyed");+ });+ it.each(["401 unauthorized", "404 proxy not found", "network unavailable"])("does not turn %s into proof", async message => {+ mockGet.mockRejectedValue(new Error(message));+ await expect(plugin.definition.onEnvironmentInspectLease!(params)).rejects.toThrow(message);+ });+ it("rejects another resource and re-reads provider state on every observation", async () => {+ mockGet.mockResolvedValueOnce(createMockSandbox({ id: "another-sandbox" })).mockRejectedValueOnce(new MockDaytonaNotFoundError("gone"));+ expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("unknown");+ expect((await plugin.definition.onEnvironmentInspectLease!(params)).state).toBe("destroyed");+ expect(mockGet).toHaveBeenCalledTimes(2);+ });+});+describe("Daytona idle eligibility", () => {it("permits a fresh unused provider but remains conservative after provider access", async () => {__resetDaytonaSandboxHandleCacheForTest();diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.tsindex 5be933bebb..96b9c3f0d0 100644--- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts+++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts@@ -24,6 +24,8 @@ import type {PluginEnvironmentDeleteTemplateParams,PluginEnvironmentDeleteTemplateResult,PluginEnvironmentDestroyLeaseParams,+ PluginEnvironmentInspectLeaseParams,+ PluginEnvironmentLeaseObservation,PluginEnvironmentExecuteParams,PluginEnvironmentExecuteResult,PluginEnvironmentRunnerIngressEndpointParams,@@ -319,8 +321,8 @@ function resolveApiKey(config: DaytonaDriverConfig): string {return envApiKey;}-function createDaytonaClient(config: DaytonaDriverConfig): Daytona {- providerUsed = true;+function createDaytonaClient(config: DaytonaDriverConfig, readOnly = false): Daytona {+ if (!readOnly) providerUsed = true;const clientConfig: DaytonaConfig = {apiKey: resolveApiKey(config),};@@ -2886,6 +2888,30 @@ const plugin = definePlugin({}},+ async onEnvironmentInspectLease(+ params: PluginEnvironmentInspectLeaseParams,+ ): Promise<PluginEnvironmentLeaseObservation> {+ // An environment-variable fallback may have changed accounts since this+ // historical allocation. Inspection requires explicitly scoped credentials.+ if (typeof params.config.apiKey !== "string" || !params.config.apiKey.trim()) {+ return { providerLeaseId: params.providerLeaseId, state: "unknown" };+ }+ const config = parseDriverConfig(params.config);+ if (!config.apiUrl && (process.env.DAYTONA_API_URL?.trim() || process.env.DAYTONA_SERVER_URL?.trim())) {+ return { providerLeaseId: params.providerLeaseId, state: "unknown" };+ }+ // Read directly from the provider. The setup-status path and cached lease+ // reconnect path can start a stopped sandbox, so neither is an observation.+ try {+ const sandbox = await createDaytonaClient(config, true).get(params.providerLeaseId);+ if (sandbox.id !== params.providerLeaseId) return { providerLeaseId: params.providerLeaseId, state: "unknown" };+ return { providerLeaseId: sandbox.id, state: sandbox.state === "stopped" ? "stopped" : "active" };+ } catch (error) {+ if (error instanceof DaytonaNotFoundError) return { providerLeaseId: params.providerLeaseId, state: "destroyed" };+ throw error;+ }+ },+async onEnvironmentDestroyLease(params: PluginEnvironmentDestroyLeaseParams,): Promise<PluginEnvironmentTerminationReceipt | void> {diff --git a/packages/plugins/sdk/src/define-plugin.ts b/packages/plugins/sdk/src/define-plugin.tsindex f206f8c839..3d7d81a31b 100644--- a/packages/plugins/sdk/src/define-plugin.ts+++ b/packages/plugins/sdk/src/define-plugin.ts@@ -55,6 +55,8 @@ import type { PluginContext, ResourceLifecycleEvent, ResourceLifecycleResult } fimport type {PluginEnvironmentAcquireLeaseParams,PluginEnvironmentDestroyLeaseParams,+ PluginEnvironmentInspectLeaseParams,+ PluginEnvironmentLeaseObservation,PluginEnvironmentExecuteParams,PluginEnvironmentExecuteResult,PluginEnvironmentRunnerIngressEndpointParams,@@ -431,6 +433,11 @@ export interface PluginDefinition {params: PluginEnvironmentDestroyLeaseParams,): Promise<PluginEnvironmentTerminationReceipt | void>;+ /** Read provider state only. This must not start or reconnect a sandbox. */+ onEnvironmentInspectLease?(+ params: PluginEnvironmentInspectLeaseParams,+ ): Promise<PluginEnvironmentLeaseObservation>;+/** Called to materialize the run workspace inside the provider lease. */onEnvironmentRealizeWorkspace?(params: PluginEnvironmentRealizeWorkspaceParams,diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.tsindex 5290d59568..8698d6ba4b 100644--- a/packages/plugins/sdk/src/index.ts+++ b/packages/plugins/sdk/src/index.ts@@ -185,6 +185,8 @@ export type {PluginEnvironmentReleaseLeaseParams,PluginEnvironmentTerminationReceipt,PluginEnvironmentDestroyLeaseParams,+ PluginEnvironmentInspectLeaseParams,+ PluginEnvironmentLeaseObservation,PluginEnvironmentRealizeWorkspaceParams,PluginEnvironmentRealizeWorkspaceResult,PluginEnvironmentExecuteParams,diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.tsindex 6e0d1f15e8..c6db765ce6 100644--- a/packages/plugins/sdk/src/protocol.ts+++ b/packages/plugins/sdk/src/protocol.ts@@ -686,6 +686,18 @@ export interface PluginEnvironmentTerminationReceipt {export interface PluginEnvironmentDestroyLeaseParams extends PluginEnvironmentReleaseLeaseParams {}+/** Observe this exact allocation without starting, stopping or deleting it.+ * Missing resources must be distinguished from permission/transport failures. */+export interface PluginEnvironmentInspectLeaseParams extends PluginEnvironmentDriverBaseParams {+ providerLeaseId: string;+ leaseMetadata?: Record<string, unknown>;+}++export interface PluginEnvironmentLeaseObservation {+ providerLeaseId: string;+ state: "active" | "stopped" | "destroyed" | "unknown";+}+export interface PluginEnvironmentRealizeWorkspaceParams extends PluginEnvironmentDriverBaseParams {lease: PluginEnvironmentLease;workspace: {@@ -1400,6 +1412,10 @@ export interface HostToWorkerMethods {params: PluginEnvironmentDestroyLeaseParams,result: PluginEnvironmentTerminationReceipt | void,];+ environmentInspectLease: [+ params: PluginEnvironmentInspectLeaseParams,+ result: PluginEnvironmentLeaseObservation,+ ];environmentRealizeWorkspace: [params: PluginEnvironmentRealizeWorkspaceParams,result: PluginEnvironmentRealizeWorkspaceResult,@@ -1506,6 +1522,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[]"environmentReleaseLease","environmentStopLease","environmentDestroyLease",+ "environmentInspectLease","environmentRealizeWorkspace","environmentExecute","environmentRunnerIngressEndpoint",diff --git a/packages/plugins/sdk/src/testing.ts b/packages/plugins/sdk/src/testing.tsindex 479bd998d8..cbfdca2114 100644--- a/packages/plugins/sdk/src/testing.ts+++ b/packages/plugins/sdk/src/testing.ts@@ -52,6 +52,8 @@ import type {PluginEnvironmentProbeParams,PluginEnvironmentProbeResult,PluginEnvironmentLease,+ PluginEnvironmentInspectLeaseParams,+ PluginEnvironmentLeaseObservation,PluginEnvironmentAcquireLeaseParams,PluginEnvironmentResumeLeaseParams,PluginEnvironmentReleaseLeaseParams,@@ -163,6 +165,7 @@ export interface EnvironmentEventRecord {| "resumeLease"| "releaseLease"| "stopLease"+ | "inspectLease"| "destroyLease"| "realizeWorkspace"| "execute"@@ -190,6 +193,7 @@ export interface EnvironmentTestHarnessOptions extends TestHarnessOptions {onResumeLease?: (params: PluginEnvironmentResumeLeaseParams) => Promise<PluginEnvironmentLease>;onReleaseLease?: (params: PluginEnvironmentReleaseLeaseParams) => Promise<PluginEnvironmentTerminationReceipt | void>;onStopLease?: (params: PluginEnvironmentReleaseLeaseParams) => Promise<PluginEnvironmentTerminationReceipt>;+ onInspectLease?: (params: PluginEnvironmentInspectLeaseParams) => Promise<PluginEnvironmentLeaseObservation>;onDestroyLease?: (params: PluginEnvironmentDestroyLeaseParams) => Promise<PluginEnvironmentTerminationReceipt | void>;onRealizeWorkspace?: (params: PluginEnvironmentRealizeWorkspaceParams) => Promise<PluginEnvironmentRealizeWorkspaceResult>;onExecute?: (params: PluginEnvironmentExecuteParams) => Promise<PluginEnvironmentExecuteResult>;@@ -217,6 +221,8 @@ export interface EnvironmentTestHarness extends TestHarness {releaseLease(params: PluginEnvironmentReleaseLeaseParams): Promise<PluginEnvironmentTerminationReceipt | void>;/** Stop and preserve an allocation independently of its release policy. */stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise<PluginEnvironmentTerminationReceipt>;+ /** Observe an allocation without starting or modifying it. */+ inspectLease(params: PluginEnvironmentInspectLeaseParams): Promise<PluginEnvironmentLeaseObservation | undefined>;/** Invoke the environment driver's destroyLease hook. */destroyLease(params: PluginEnvironmentDestroyLeaseParams): Promise<PluginEnvironmentTerminationReceipt | void>;/** Invoke the environment driver's realizeWorkspace hook. */@@ -2775,6 +2781,9 @@ export function createEnvironmentTestHarness(options: EnvironmentTestHarnessOptiasync stopLease(params) {return callHook("stopLease", driver.onStopLease, params, "onStopLease");},+ async inspectLease(params) {+ return callHook("inspectLease", driver.onInspectLease, params, "onInspectLease");+ },async destroyLease(params) {return callHook("destroyLease", driver.onDestroyLease, params, "onDestroyLease");},diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.tsindex 4812054450..3de92ac521 100644--- a/packages/plugins/sdk/src/worker-rpc-host.ts+++ b/packages/plugins/sdk/src/worker-rpc-host.ts@@ -93,6 +93,7 @@ import type {RefreshExternalObjectsParams,PluginEnvironmentAcquireLeaseParams,PluginEnvironmentDestroyLeaseParams,+ PluginEnvironmentInspectLeaseParams,PluginEnvironmentExecuteParams,PluginEnvironmentRunnerIngressEndpointParams,PluginEnvironmentSyncInParams,@@ -1699,6 +1700,10 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHostcase "environmentDestroyLease":return handleEnvironmentDestroyLease(params as PluginEnvironmentDestroyLeaseParams);+ case "environmentInspectLease":+ if (!plugin.definition.onEnvironmentInspectLease) throw methodNotImplemented("environmentInspectLease");+ return plugin.definition.onEnvironmentInspectLease(params as PluginEnvironmentInspectLeaseParams);+case "environmentRealizeWorkspace":return handleEnvironmentRealizeWorkspace(params as PluginEnvironmentRealizeWorkspaceParams);@@ -1802,6 +1807,7 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHostif (plugin.definition.onEnvironmentReleaseLease) supportedMethods.push("environmentReleaseLease");if (plugin.definition.onEnvironmentStopLease) supportedMethods.push("environmentStopLease");if (plugin.definition.onEnvironmentDestroyLease) supportedMethods.push("environmentDestroyLease");+ if (plugin.definition.onEnvironmentInspectLease) supportedMethods.push("environmentInspectLease");if (plugin.definition.onEnvironmentRealizeWorkspace) supportedMethods.push("environmentRealizeWorkspace");if (plugin.definition.onEnvironmentExecute) supportedMethods.push("environmentExecute");if (plugin.definition.onEnvironmentRunnerIngressEndpoint) {diff --git a/packages/plugins/sdk/tests/worker-rpc-host.test.ts b/packages/plugins/sdk/tests/worker-rpc-host.test.tsindex 0c759222c3..61f523d84a 100644--- a/packages/plugins/sdk/tests/worker-rpc-host.test.ts+++ b/packages/plugins/sdk/tests/worker-rpc-host.test.ts@@ -418,6 +418,24 @@ describe("worker configChanged cross-tenant guard", () => {} finally { worker.stop(); }});+ it.each([false, true])("advertises read-only inspection only when implemented: %s", async supported => {+ let inspections = 0, mutations = 0;+ const worker = makeWorker(definePlugin({ async setup() {},+ async onEnvironmentReleaseLease() { mutations++; },+ async onEnvironmentDestroyLease() { mutations++; },+ ...(supported ? { async onEnvironmentInspectLease() { inspections++; return { providerLeaseId: "allocation", state: "destroyed" as const }; } } : {}),+ }));+ try {+ const initialized = await worker.initialize() as { supportedMethods: string[] };+ expect(initialized.supportedMethods.includes("environmentInspectLease")).toBe(supported);+ const result = worker.callWorker("environmentInspectLease", { driverKey: "fixture", companyId: "company", environmentId: "environment", providerLeaseId: "allocation", config: {} });+ if (supported) await expect(result).resolves.toEqual({ providerLeaseId: "allocation", state: "destroyed" });+ else await expect(result).rejects.toThrow();+ expect(inspections).toBe(supported ? 1 : 0);+ expect(mutations).toBe(0);+ } finally { worker.stop(); }+ });+it("fails closed when a second, distinct company's config would overwrite a single-tenant worker", async () => {const applied: Array<{ companyId: string | null; token: unknown }> = [];const plugin = definePlugin({diff --git a/server/src/__tests__/delivery-work-coordinator.test.ts b/server/src/__tests__/delivery-work-coordinator.test.tsindex d3f7590a79..9531d21deb 100644--- a/server/src/__tests__/delivery-work-coordinator.test.ts+++ b/server/src/__tests__/delivery-work-coordinator.test.ts@@ -2,10 +2,12 @@ import { afterEach, describe, expect, it, vi } from "vitest";import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";import { notifyDeliveryWork, DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";-import { idleWorkSnapshot } from "../services/task-admission.js";+import { beginIdleTrackedWork, idleWorkSnapshot, isIdleTaskDrainActive, readTaskDrain, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";+import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";+import type { Db } from "@paperclipai/db";const coordinators: ReturnType<typeof createDeliveryWorkCoordinator>[] = [];-afterEach(async () => { for (const c of coordinators.splice(0)) await c.stop(); vi.useRealTimers(); });+afterEach(async () => { stopTaskDrain(); for (const c of coordinators.splice(0)) await c.stop(); vi.useRealTimers(); });function setup() {vi.useFakeTimers();let outcome = "committed";@@ -89,6 +91,43 @@ describe("delivery work coordinator", () => {expect(t.run).toHaveBeenCalledTimes(3);expect(idleWorkSnapshot().active).toBe(0);});+ it.each([false, true])("pauses passive maintenance through a slow owned backup while preserving new work (%s)", async newWork => {+ const s = setup(), t = task();+ s.canRun.mockImplementation(() => !isIdleTaskDrainActive());+ s.canReconcile.mockReturnValue(true);+ await s.coordinator.register(DELIVERY_QUEUES.connection, {+ ...t, nextRunAt: () => Date.now() + 5000, pausePassiveMaintenance: true,+ }).ready;+ const drain = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });+ const getStatus = () => {+ const current = readTaskDrain(new Date());+ return { ...current, draining: current !== null, startedAt: current?.startedAt ?? null,+ expiresAt: current?.expiresAt ?? null, activeRuns: 0, pendingWakes: 0 };+ };+ const db = { transaction: async (fn: (tx: unknown) => Promise<unknown>) =>+ fn({ execute: async () => [{ blocked: false }] }) } as unknown as Db;+ let entered!: () => void;+ const started = new Promise<void>(resolve => { entered = resolve; });+ const checkpoint = async () => {+ const finish = beginIdleTrackedWork(); entered();+ try { await new Promise(resolve => setTimeout(resolve, 6000)); return true; }+ finally { finish(); }+ };+ expect(idleWorkSnapshot().active).toBe(0);+ const scan = readIdleSleepSafety(db, getStatus, Date.now, drain.ownerId, async () => "none", undefined, checkpoint);+ await Promise.race([started, scan.then(result => { throw new Error(`Backup did not start: ${JSON.stringify(result)}`); })]);+ expect(idleWorkSnapshot().active).toBe(1);+ if (newWork) await s.enqueue(DELIVERY_QUEUES.connection);+ await vi.advanceTimersByTimeAsync(6000);+ expect(await scan).toEqual({ version: 1, backgroundWork: newWork ? "unknown" : "none" });+ expect(t.run).toHaveBeenCalledTimes(1);+ expect(t.hasPending).toHaveBeenCalledTimes(newWork ? 2 : 1);+ expect(idleWorkSnapshot().active).toBe(0);+ stopTaskDrain();+ await vi.advanceTimersByTimeAsync(5000);+ expect(t.run).toHaveBeenCalledTimes(2);+ expect(s.onError).not.toHaveBeenCalled();+ });it("fences unresolved writes until the server settles, then scans and releases the idle hold", async () => {const s = setup(), t = task();await s.coordinator.register(DELIVERY_QUEUES.feedback, t).ready;diff --git a/server/src/__tests__/environment-custom-images-service.test.ts b/server/src/__tests__/environment-custom-images-service.test.tsindex fd9ba73957..6422cec2f9 100644--- a/server/src/__tests__/environment-custom-images-service.test.ts+++ b/server/src/__tests__/environment-custom-images-service.test.ts@@ -77,6 +77,7 @@ const ALL_CUSTOM_IMAGE_WORKER_METHODS = [function createWorkerManager(options?: { workerMethods?: readonly string[] }) {const supportedMethods = options?.workerMethods ?? ALL_CUSTOM_IMAGE_WORKER_METHODS;const call = vi.fn(async (_pluginId: string, method: string, params: Record<string, unknown>) => {+ if (method === "environmentInspectLease") return { providerLeaseId: params.providerLeaseId, state: "destroyed" };if (method === "environmentStartInteractiveSetup") {return {providerLeaseId: `lease-${params.sessionId}`,@@ -199,6 +200,28 @@ describeEmbeddedPostgres("environmentCustomImageService", () => {return { companyId, environmentId };}+ it.each(["supported", "old_worker", "changed_config"])("inspects historical setup read-only with exact config and capability gates: %s", async mode => {+ const fixture = await seed();+ const worker = createWorkerManager({ workerMethods: [...ALL_CUSTOM_IMAGE_WORKER_METHODS,+ ...(mode === "old_worker" ? [] : ["environmentInspectLease"])] });+ const service = environmentCustomImageService(db, { pluginWorkerManager: worker });+ const created = await service.startSetupSession({ environmentId: fixture.environmentId,+ actor: { userId: "fixture-user" }, now: new Date("2099-01-01T00:00:00Z") });+ await service.cancelSetupSession({ sessionId: created.session.id });+ const [session] = await db.select().from(environmentCustomImageSetupSessions).where(eq(environmentCustomImageSetupSessions.id, created.session.id));+ expect(session!.metadata).toMatchObject({ setupRpcCompanyId: created.session.metadata!.setupRpcCompanyId,+ sourceEnvironmentConfigFingerprint: created.session.metadata!.sourceEnvironmentConfigFingerprint });+ if (mode === "changed_config") await db.update(environments).set({ config: { provider: "fake-plugin", image: "replacement:base" } }).where(eq(environments.id, fixture.environmentId));+ worker.call.mockClear();+ const observed = await service.inspectSetupResource(session!);+ if (mode === "supported") {+ expect(observed).toEqual({ providerLeaseId: session!.providerLeaseId, state: "destroyed" });+ expect(worker.call).toHaveBeenCalledExactlyOnceWith(expect.any(String), "environmentInspectLease", expect.objectContaining({+ companyId: session!.metadata!.setupRpcCompanyId, providerLeaseId: session!.providerLeaseId,+ }), 15_000);+ } else { expect(observed).toBeUndefined(); expect(worker.call).not.toHaveBeenCalled(); }+ });+it("starts, refreshes, finishes, refreshes again, and rolls back setup sessions", async () => {const { environmentId } = await seed();const workerManager = createWorkerManager();@@ -385,6 +408,8 @@ describeEmbeddedPostgres("environmentCustomImageService", () => {reason: "user_cancelled",});expect(cancelled.status).toBe("cancelled");+ expect(cancelled.metadata).toMatchObject({ setupRpcCompanyId: started.session.metadata!.setupRpcCompanyId,+ sourceEnvironmentConfigFingerprint: started.session.metadata!.sourceEnvironmentConfigFingerprint });expect(await service.getActiveTemplate({ environmentId })).toBeNull();const expired = await service.startSetupSession({@@ -399,6 +424,8 @@ describeEmbeddedPostgres("environmentCustomImageService", () => {expect(cleanup).toMatchObject({ scanned: 1, timedOut: 1, failed: 0 });const timedOut = await service.getSessionById(expired.session.id);expect(timedOut?.status).toBe("timed_out");+ expect(timedOut?.metadata).toMatchObject({ setupRpcCompanyId: expired.session.metadata!.setupRpcCompanyId,+ sourceEnvironmentConfigFingerprint: expired.session.metadata!.sourceEnvironmentConfigFingerprint });});it("rejects interactive setup when the provider declares it but the worker omits a setup method", async () => {diff --git a/server/src/__tests__/fixtures/plugin-worker-delayed.cjs b/server/src/__tests__/fixtures/plugin-worker-delayed.cjsindex 48a68cfcc5..baae987cb7 100644--- a/server/src/__tests__/fixtures/plugin-worker-delayed.cjs+++ b/server/src/__tests__/fixtures/plugin-worker-delayed.cjs@@ -8,6 +8,9 @@ const rl = readline.createInterface({input: process.stdin,crlfDelay: Infinity,});+const heldInspections = [];+let inspectionCount = 0;+let activeInspections = 0;rl.on("line", (line) => {if (!line.trim()) return;@@ -20,7 +23,7 @@ rl.on("line", (line) => {id: message.id,result: {ok: true,- supportedMethods: ["environmentExecute", "configChanged"],+ supportedMethods: ["environmentExecute", "configChanged", "environmentInspectLease", "health"],},});return;@@ -45,6 +48,9 @@ rl.on("line", (line) => {}if (method === "configChanged") {+ if (message.params.config.releaseInspections) {+ for (const complete of heldInspections.splice(0)) complete();+ }setTimeout(() => send({jsonrpc: "2.0", id: message.id,...(message.params.config.fail@@ -54,6 +60,20 @@ rl.on("line", (line) => {return;}+ if (method === "environmentInspectLease") {+ inspectionCount++;+ activeInspections++;+ const complete = () => { activeInspections--; send({ jsonrpc: "2.0", id: message.id,+ result: { providerLeaseId: message.params.providerLeaseId, state: "destroyed" } }); };+ if (message.params.leaseMetadata?.holdInspection) heldInspections.push(complete);+ else complete();+ return;+ }+ if (method === "health") {+ send({ jsonrpc: "2.0", id: message.id, result: { status: "ok", details: { inspectionCount, activeInspections } } });+ return;+ }+if (method === "shutdown") {send({jsonrpc: "2.0",diff --git a/server/src/__tests__/historical-cleanup-verification.test.ts b/server/src/__tests__/historical-cleanup-verification.test.tsnew file mode 100644index 0000000000..37bc2dd669--- /dev/null+++ b/server/src/__tests__/historical-cleanup-verification.test.ts@@ -0,0 +1,126 @@+import { randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";+import { agents, companies, createDb, environmentCustomImageSetupSessions, environmentLeases, environments, heartbeatRuns } from "@paperclipai/db";+import { historicalCleanupVerificationService } from "../services/historical-cleanup-verification.js";+import { withoutCleanupVerification } from "../services/cleanup-verification-proof.js";+import { idleWorkSnapshot } from "../services/task-admission.js";+import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";++const now = Date.UTC(2020, 0, 1), ownerId = randomUUID();+const owned = (): IdleSleepDrainStatus => ({ ownerId, draining: true, startedAt: new Date(now - 1000),+ expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 });+const support = await getEmbeddedPostgresTestSupport();+if (!support.supported) console.warn(`Skipping historical cleanup database tests: ${support.reason}`);+(support.supported ? describe : describe.skip)("historical cleanup verification", () => {+ let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("historical-cleanup-"); db = createDb(database.connectionString); }, 60_000);+ afterEach(async () => { await db.execute(sql`SET client_min_messages = warning; TRUNCATE companies, plugins, environments CASCADE`); });+ afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });+ const read = () => readIdleSleepSafety(db, owned, () => now, ownerId, async () => "none", undefined, undefined, true, true);+ const state = async () => (await read()).backgroundWork;+ async function seed(adHoc = false) {+ const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();+ await db.insert(companies).values({ id: companyId, name: "Cleanup fixture", issuePrefix: `T${companyId.slice(0, 6).toUpperCase()}` });+ await db.insert(agents).values({ id: agentId, companyId, name: "Idle fixture", role: "engineer", status: "idle", adapterType: "process" });+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "failed", finishedAt: new Date(now) });+ const [environment] = await db.insert(environments).values({ name: "Remote fixture", driver: "sandbox" }).returning();+ const [lease] = await db.insert(environmentLeases).values({ companyId, environmentId: environment!.id,+ heartbeatRunId: adHoc ? null : runId, status: "failed", leasePolicy: "ephemeral", provider: "daytona",+ providerLeaseId: randomUUID(), releasedAt: new Date(now), cleanupStatus: "success", metadata: { driver: "sandbox" } }).returning();+ return { companyId, runId, environment: environment!, lease: lease! };+ }+ function verifier(leaseState = "destroyed", setupState = "destroyed") {+ return historicalCleanupVerificationService(db, {+ inspectLease: async row => ({ providerLeaseId: row.providerLeaseId, state: leaseState }),+ inspectSetup: async row => ({ providerLeaseId: row.providerLeaseId, state: setupState }),+ });+ }+ it.each([false, true])("accepts exact deletion proof without changing outcomes or granting run authority (ad-hoc %s)", async adHoc => {+ const f = await seed(adHoc); expect(await state()).toBe("present");+ const service = verifier(); expect(await service.sweep()).toMatchObject({ checked: 1, verified: 1, failed: 0 });+ expect(await state()).toBe("none");+ const [current] = await db.select().from(environmentLeases);+ expect(current).toMatchObject({ status: "failed", cleanupStatus: "success", heartbeatRunId: adHoc ? null : f.runId });+ expect(current!.metadata).not.toHaveProperty("remoteExecutionTermination");+ expect(await service.sweep()).toMatchObject({ checked: 0, verified: 0 });+ });+ it.each(["active", "stopped", "unknown"])("retains %s allocations", async status => {+ await seed(); expect(await verifier(status).sweep()).toMatchObject({ verified: 0, deferred: 1 });+ expect(await state()).toBe("present");+ });+ it.each(["old_worker", "wrong_id", "provider_error"])("does not manufacture proof from %s", async mode => {+ await seed();+ const service = historicalCleanupVerificationService(db, { inspectSetup: async () => undefined,+ inspectLease: async () => { if (mode === "provider_error") throw new Error("private credential failure");+ return mode === "old_worker" ? undefined : { providerLeaseId: "wrong", state: "destroyed" }; } });+ expect((await service.sweep()).verified).toBe(0); expect(await state()).toBe("present");+ });+ it.each(["resume", "resource", "environment", "metadata", "run", "accounting", "retry"])("loses verification when %s changes during provider inspection", async change => {+ const f = await seed();+ const service = historicalCleanupVerificationService(db, { inspectSetup: async () => undefined, inspectLease: async row => {+ if (change === "resume") await db.update(environmentLeases).set({ status: "active" }).where(eq(environmentLeases.id, row.id));+ if (change === "resource") await db.update(environmentLeases).set({ providerLeaseId: "replacement" }).where(eq(environmentLeases.id, row.id));+ if (change === "environment") await db.update(environmentLeases).set({ environmentId: null }).where(eq(environmentLeases.id, row.id));+ if (change === "metadata") await db.update(environmentLeases).set({ metadata: { driver: "sandbox", nativeWorkspaceExportResume: {} } }).where(eq(environmentLeases.id, row.id));+ if (["run", "accounting", "retry"].includes(change)) await db.update(heartbeatRuns).set(change === "run" ? { status: "running", finishedAt: null }+ : change === "accounting" ? { costAccountingPending: true } : { scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, f.runId));+ return { providerLeaseId: row.providerLeaseId, state: "destroyed" };+ } });+ expect((await service.sweep()).verified).toBe(0); expect(await state()).toBe("present");+ });+ it.each(["retained", "pending_cleanup"])("does not inspect a resource with another %s owner", async status => {+ const f = await seed();+ await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: f.environment.id,+ provider: "daytona", providerLeaseId: f.lease.providerLeaseId, status });+ expect(await verifier().sweep()).toMatchObject({ checked: 0, verified: 0 });+ expect(await state()).toBe("present");+ });+ it("invalidates a previously accepted proof when the lease row changes", async () => {+ const f = await seed(); await verifier().sweep(); expect(await state()).toBe("none");+ await db.update(environmentLeases).set({ updatedAt: new Date(0) }).where(eq(environmentLeases.id, f.lease.id));+ expect(await state()).toBe("present");+ });+ it("keeps provider work tracked through the observation and coalesces concurrent sweeps", async () => {+ await seed(); let finish!: () => void; let entered!: () => void;+ const started = new Promise<void>(resolve => { entered = resolve; });+ const paused = new Promise<void>(resolve => { finish = resolve; });+ const inspectLease = vi.fn(async (row: typeof environmentLeases.$inferSelect) => { entered(); await paused; return { providerLeaseId: row.providerLeaseId, state: "destroyed" }; });+ const service = historicalCleanupVerificationService(db, { inspectLease, inspectSetup: async () => undefined });+ const before = idleWorkSnapshot().active; const first = service.sweep(); const second = service.sweep();+ expect(first).toBe(second); await started; expect(idleWorkSnapshot().active).toBe(before + 1);+ expect(await state()).toBe("present"); finish(); await first;+ expect(inspectLease).toHaveBeenCalledTimes(1); expect(idleWorkSnapshot().active).toBe(before);+ });+ async function setup() {+ const f = await seed(); await db.delete(environmentLeases);+ const [row] = await db.insert(environmentCustomImageSetupSessions).values({ environmentId: f.environment.id,+ provider: "daytona", providerLeaseId: randomUUID(), status: "promoted", finishedAt: new Date(now), metadata: { setupRpcCompanyId: f.companyId } }).returning();+ return { ...f, setup: row! };+ }+ it("requires deletion proof for terminal image sessions and preserves their history", async () => {+ const f = await setup(); expect(await state()).toBe("present");+ expect(await verifier().sweep()).toMatchObject({ verified: 1 }); expect(await state()).toBe("none");+ expect((await db.select().from(environmentCustomImageSetupSessions))[0]).toMatchObject({ status: "promoted", providerLeaseId: f.setup.providerLeaseId });+ });+ it.each(["secret", "live", "provider", "scope"])("rejects image-session %s changes during inspection", async change => {+ await setup();+ const service = historicalCleanupVerificationService(db, { inspectLease: async () => undefined, inspectSetup: async row => {+ await db.update(environmentCustomImageSetupSessions).set(change === "secret" ? { connectionSecretRef: "secret://fixture" }+ : change === "live" ? { status: "waiting_for_user", finishedAt: null }+ : change === "provider" ? { provider: "other" } : { metadata: { setupRpcCompanyId: randomUUID() } }).where(eq(environmentCustomImageSetupSessions.id, row.id));+ return { providerLeaseId: row.providerLeaseId, state: "destroyed" };+ } });+ expect((await service.sweep()).verified).toBe(0); expect(await state()).toBe("present");+ });+ it("keeps image setup blocked when a retained lease owns the same allocation", async () => {+ const f = await setup(); await db.insert(environmentLeases).values({ companyId: f.companyId,+ environmentId: f.environment.id, provider: "daytona", providerLeaseId: f.setup.providerLeaseId, status: "retained" });+ expect(await verifier().sweep()).toMatchObject({ checked: 0 }); expect(await state()).toBe("present");+ });+});++it("removes host-owned cleanup evidence from provider metadata", () => {+ expect(withoutCleanupVerification({ paperclipCleanupVerification: { state: "destroyed" }, driver: "sandbox" })).toEqual({ driver: "sandbox" });+});diff --git a/server/src/__tests__/historical-provider-credentials.test.ts b/server/src/__tests__/historical-provider-credentials.test.tsnew file mode 100644index 0000000000..256610cb3d--- /dev/null+++ b/server/src/__tests__/historical-provider-credentials.test.ts@@ -0,0 +1,61 @@+import { createHash, randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";+import { companies, companySecrets, companySecretVersions, createDb } from "@paperclipai/db";+import type { SandboxEnvironmentConfig } from "@paperclipai/shared";+const resolve = vi.hoisted(() => vi.fn());+vi.mock("../services/secrets.js", () => ({ secretService: () => ({ resolveSecretValueForSandboxCleanup: resolve }) }));+import { historicalProviderCredentials } from "../services/historical-provider-credentials.js";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";++const value = "fixture-original-provider-key";+const hash = createHash("sha256").update(value).digest("hex");+const schema = { properties: { apiKey: { type: "string", format: "secret-ref" } } };+const context = { issueId: null, heartbeatRunId: null };+const support = await getEmbeddedPostgresTestSupport();+if (!support.supported) console.warn(`Skipping provider credential database tests: ${support.reason}`);+(support.supported ? describe : describe.skip)("historical provider credential identity", () => {+ let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("historical-credentials-"); db = createDb(database.connectionString); }, 60_000);+ afterEach(async () => { resolve.mockReset(); await db.execute(sql`SET client_min_messages = warning; TRUNCATE companies CASCADE`); });+ afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });+ async function seed() {+ const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Credential fixture", issuePrefix: `T${companyId.slice(0, 6)}` });+ const [secret] = await db.insert(companySecrets).values({ companyId, key: "provider-key", name: "Provider key" }).returning();+ await db.insert(companySecretVersions).values({ secretId: secret!.id, version: 1, material: {}, valueSha256: hash, fingerprintSha256: hash });+ resolve.mockResolvedValue(value); return { companyId, secret: secret! };+ }+ const inspect = (companyId: string, apiKey: unknown, manifest = schema) => historicalProviderCredentials(db, companyId,+ { provider: "daytona", apiKey, reuseLease: false } as SandboxEnvironmentConfig, manifest, context);+ it("uses exact recorded literal credentials without resolving another account", async () => {+ const f = await seed(); expect(await inspect(f.companyId, value)).toMatchObject({ apiKey: value }); expect(resolve).not.toHaveBeenCalled();+ });+ it("defers missing credentials instead of relying on an ambient account", async () => {+ const f = await seed(); expect(await inspect(f.companyId, undefined)).toBeUndefined(); expect(resolve).not.toHaveBeenCalled();+ });+ it.each([false, true])("pins and checks an unrotated initial secret version (binding %s)", async binding => {+ const f = await seed(); expect(await inspect(f.companyId, binding ? { type: "secret_ref", secretId: f.secret.id, version: "latest" } : f.secret.id)).toMatchObject({ apiKey: value });+ expect(resolve).toHaveBeenCalledExactlyOnceWith(f.companyId, f.secret.id, 1, { ...context, configPath: "apiKey" });+ });+ it.each(["rotated", "rolled_back", "foreign", "revoked", "deleted", "unknown_manifest", "different_pin"])("defers uncertain credential identity: %s", async mode => {+ const f = await seed(); let companyId = f.companyId;+ if (["rotated", "rolled_back"].includes(mode)) {+ await db.insert(companySecretVersions).values({ secretId: f.secret.id, version: 2, material: {}, valueSha256: hash, fingerprintSha256: hash });+ if (mode === "rotated") await db.update(companySecrets).set({ latestVersion: 2 }).where(eq(companySecrets.id, f.secret.id));+ }+ if (mode === "foreign") companyId = randomUUID();+ if (mode === "revoked") await db.update(companySecretVersions).set({ revokedAt: new Date() });+ if (mode === "deleted") await db.update(companySecrets).set({ status: "deleted" });+ const key = mode === "different_pin" ? { type: "secret_ref", secretId: f.secret.id, version: 2 } : f.secret.id;+ expect(await inspect(companyId, key, mode === "unknown_manifest" ? {} as typeof schema : schema)).toBeUndefined();+ expect(resolve).not.toHaveBeenCalled();+ });+ it("rejects externally mutable secret material whose current bytes changed accounts", async () => {+ const f = await seed(); resolve.mockResolvedValue("fixture-replacement-account-key");+ expect(await inspect(f.companyId, f.secret.id)).toBeUndefined();+ });+ it("propagates credential read errors instead of calling them resource deletion", async () => {+ const f = await seed(); resolve.mockRejectedValue(new Error("credential backend unavailable"));+ await expect(inspect(f.companyId, f.secret.id)).rejects.toThrow("credential backend unavailable");+ });+});diff --git a/server/src/__tests__/idle-connection-history.test.ts b/server/src/__tests__/idle-connection-history.test.tsnew file mode 100644index 0000000000..43b3508543--- /dev/null+++ b/server/src/__tests__/idle-connection-history.test.ts@@ -0,0 +1,156 @@+import { randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";+import {+ agents, agentWakeupRequests, chatEndpoints, chatSlackRegistrations, companies, connectionEventDeliveries, connectionGrants, connectionIntentDeliveries,+ createDb, environmentLeases, environments, heartbeatRuns, issues, issueThreadInteractions, toolApplications, toolConnections,+} from "@paperclipai/db";+import { beginIdleTrackedWork } from "../services/task-admission.js";+import { connectionIntentDeliveryService } from "../services/connection-intent-delivery.js";+import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";++const now = Date.UTC(2020, 0, 1); const ownerId = randomUUID();+const owned = (): IdleSleepDrainStatus => ({ ownerId, draining: true, startedAt: new Date(now - 1000),+ expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 });+const none = { version: 1, backgroundWork: "none" }; const present = { version: 1, backgroundWork: "present" };+const support = await getEmbeddedPostgresTestSupport();+if (!support.supported) console.warn(`Skipping connection-history database tests: ${support.reason}`);+(support.supported ? describe : describe.skip)("idle connection and local lease history", () => {+ let db: ReturnType<typeof createDb>; let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("idle-connection-history-"); db = createDb(database.connectionString); }, 60_000);+ afterEach(async () => { await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); });+ afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });+ const read = () => readIdleSleepSafety(db, owned, () => now, ownerId, async () => "none", undefined, undefined, true, true);+ async function seed(status = "done") {+ const companyId = randomUUID(); const agentId = randomUUID(); const runId = randomUUID(); const issueId = randomUUID();+ await db.insert(companies).values({ id: companyId, name: "Connection history fixture", issuePrefix: `T${companyId.slice(0, 6).toUpperCase()}` });+ await db.insert(agents).values({ id: agentId, companyId, name: "Idle owner", role: "engineer", status: "idle", adapterType: "process" });+ await db.insert(issues).values({ id: issueId, companyId, title: "Fixture", status, assigneeAgentId: agentId });+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "succeeded", finishedAt: new Date(now) });+ return { companyId, agentId, runId, issueId };+ }+ async function delivery(f: Awaited<ReturnType<typeof seed>>, deliveredAt: Date | null = new Date(now)) {+ const [card] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,+ kind: "connection_intent", status: "expired", resolvedAt: new Date(now), sourceRunId: f.runId,+ payload: { requestingAgentId: f.agentId }, result: { version: 1, outcome: "expired", connectionId: randomUUID() } }).returning();+ const [outbox] = await db.insert(connectionIntentDeliveries).values({ companyId: f.companyId, interactionId: card!.id, deliveredAt }).returning();+ return { card: card!, outbox: outbox! };+ }+ it("retains processed event history while allowing sleep", async () => {+ const f = await seed();+ const [event] = await db.insert(connectionEventDeliveries).values({ companyId: f.companyId, provider: "github", providerDeliveryId: randomUUID(), event: "push", status: "processed", processedAt: new Date(now) }).returning();+ expect(await read()).toEqual(none); expect(await db.select().from(connectionEventDeliveries)).toEqual([event]);+ });+ it.each(["received", "failed", "missing_receipt"])("retains %s event work", async state => {+ const f = await seed();+ await db.insert(connectionEventDeliveries).values({ companyId: f.companyId, provider: "github", providerDeliveryId: randomUUID(), event: "push", status: state === "missing_receipt" ? "processed" : state as "received" | "failed", processedAt: state === "missing_receipt" ? null : new Date(now) });+ expect(await read()).toEqual(present);+ });+ it("keeps downstream work awake after event application finishes", async () => {+ const f = await seed(); await db.insert(connectionEventDeliveries).values({ companyId: f.companyId, provider: "github", providerDeliveryId: randomUUID(), event: "push", status: "processed", processedAt: new Date(now) });+ await db.update(heartbeatRuns).set({ status: "queued", finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));+ expect(await read()).toEqual(present);+ });+ it("permits a delivered no-op without deleting its outbox receipt", async () => {+ const d = await delivery(await seed()); expect(await read()).toEqual(none);+ expect(await db.select().from(connectionIntentDeliveries)).toEqual([d.outbox]);+ });+ it.each(["pending", "unfinished_wake", "queued_wake", "coalesced_unfinished"])("retains %s connection delivery work", async state => {+ const f = await seed(); const d = await delivery(f, state === "pending" ? null : new Date(now));+ if (state !== "pending") await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",+ idempotencyKey: `connection-intent:${d.card.id}:expired`, status: state === "queued_wake" ? "queued" : state === "coalesced_unfinished" ? "coalesced" : "completed",+ finishedAt: state === "coalesced_unfinished" ? new Date(now) : null });+ expect(await read()).toEqual(present);+ });+ it.each(["completed", "coalesced"])("permits a delivered connection after its exact %s wake finishes", async status => {+ const f = await seed(); const d = await delivery(f);+ await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation", status, runId: f.runId,+ idempotencyKey: `connection-intent:${d.card.id}:expired`, finishedAt: new Date(now) });+ expect(await read()).toEqual(none);+ });+ it("parks an undelivered system retirement behind its still-pending response without marking delivery complete", async () => {+ const f = await seed("in_review"); const d = await delivery(f, null);+ const [question] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId, kind: "ask_user_questions", status: "pending",+ effectiveResolverPolicy: "human_only", sourceRunId: f.runId, payload: {} }).returning();+ expect(await read()).toEqual(none);+ const worker = connectionIntentDeliveryService(db, { wakeup: async () => null });+ expect(await worker.hasPending()).toBe(false);+ expect(worker.nextRunAt()).toBeTypeOf("number");+ await worker.sweepPending();+ expect(await db.select().from(connectionIntentDeliveries)).toEqual([d.outbox]);+ await db.update(issueThreadInteractions).set({ status: "answered", resolvedAt: new Date(now) }).where(eq(issueThreadInteractions.id, question!.id));+ expect(await worker.hasPending()).toBe(true);+ expect(worker.nextRunAt()).toBeNull();+ expect(await read()).toEqual(present);+ });+ async function connection(f: Awaited<ReturnType<typeof seed>>, change: Partial<typeof toolConnections.$inferInsert> = {}) {+ const [app] = await db.insert(toolApplications).values({ companyId: f.companyId, name: "Outbound tool fixture", type: "mcp_http" }).returning();+ const [c] = await db.insert(toolConnections).values({ companyId: f.companyId, applicationId: app!.id, uid: randomUUID(), name: "Saved remote tools",+ connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true, ...change }).returning(); return c!;+ }+ it("allows saved outbound MCP configuration while preserving active-call admission", async () => {+ const c = await connection(await seed()); expect(await read()).toEqual(none);+ const finish = beginIdleTrackedWork(); try { expect(await read()).toEqual({ version: 1, backgroundWork: "unknown" }); } finally { finish(); }+ expect(await db.select().from(toolConnections)).toEqual([c]);+ });+ it("retains inbound GitHub bindings until their own wake contract is supported", async () => {+ const f = await seed(); const c = await connection(f);+ await db.insert(connectionGrants).values({ companyId: f.companyId, connectionId: c.id, kind: "organization", providerTenant: { github: {+ userId: "fixture", login: "fixture", installationCount: 1, repositoryCount: 1, repositorySelection: "all", installationIds: ["fixture"], installationOwnerLogins: ["fixture"],+ } } }); expect(await read()).toEqual(present);+ });+ it.each(["tool", "channel", "ai"] as const)("permits an inert disabled %s draft", async purpose => {+ await connection(await seed(), { connectionPurpose: purpose, transport: purpose === "channel" ? "chat_sdk" : purpose === "ai" ? "runtime_auth" : "mcp_remote", status: "draft", enabled: false });+ expect(await read()).toEqual(none);+ });+ it.each(["local_stdio", "rest_api"] as const)("retains unsupported active %s integrations", async transport => {+ await connection(await seed(), { transport }); expect(await read()).toEqual(present);+ });+ async function slackDraft(command?: string) {+ const f = await seed(); const c = await connection(f, { connectionPurpose: "channel", transport: "chat_sdk", status: "draft", enabled: false });+ const [endpoint] = await db.insert(chatEndpoints).values({ companyId: f.companyId, connectionId: c.id,+ assignedAgentId: f.agentId, publicId: randomUUID(), provider: "slack",+ setup: { step: "provider_setup", ...(command ? { command } : {}) } }).returning();+ return { ...f, connection: c, endpoint: endpoint! };+ }+ it.each([undefined, "/paperclip-fixture"])("permits an unconfigured disabled Slack draft without losing setup state (%s)", async command => {+ const f = await slackDraft(command); expect(await read()).toEqual(none);+ expect(await db.select().from(chatEndpoints)).toEqual([f.endpoint]);+ });+ it.each(["active", "provider_identity", "previous_activity", "connected_step", "health", "unsupported_provider", "enabled_connection", "setup_in_progress", "registration", "unknown_command_shape"])("retains draft integration with %s", async mode => {+ const f = await slackDraft();+ if (mode === "active") await db.update(chatEndpoints).set({ status: "active" }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "provider_identity") await db.update(chatEndpoints).set({ botExternalId: "fixture-bot" }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "previous_activity") await db.update(chatEndpoints).set({ lastEventAt: new Date(now) }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "connected_step") await db.update(chatEndpoints).set({ setup: { step: "test" } }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "health") await db.update(chatEndpoints).set({ healthMessage: "Provider setup pending" }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "unsupported_provider") await db.update(chatEndpoints).set({ provider: "discord" }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "enabled_connection") await db.update(toolConnections).set({ enabled: true }).where(eq(toolConnections.id, f.connection.id));+ if (mode === "setup_in_progress") await db.update(chatEndpoints).set({ setup: { step: "provider_setup", slackSetupMethod: "managed" } }).where(eq(chatEndpoints.id, f.endpoint.id));+ if (mode === "registration") await db.insert(chatSlackRegistrations).values({ endpointId: f.endpoint.id, companyId: f.companyId,+ requestId: randomUUID(), status: "creating", manifest: {}, manifestHash: "fixture-hash", callbackUri: "https://example.test/callback", createdByUserId: "fixture-user" });+ if (mode === "unknown_command_shape") await db.update(chatEndpoints).set({ setup: sql`'{"step":"provider_setup","command":42}'::jsonb` }).where(eq(chatEndpoints.id, f.endpoint.id));+ expect(await read()).toEqual(present);+ });+ async function localLease() {+ const f = await seed(); const [environment] = await db.insert(environments).values({ name: "Local fixture", driver: "local" }).returning();+ const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: environment!.id, heartbeatRunId: f.runId,+ status: "failed", leasePolicy: "ephemeral", provider: "local", providerLeaseId: null, releasedAt: new Date(now), metadata: { driver: "local" } }).returning();+ return { ...f, lease: lease!, environment: environment! };+ }+ it("permits released local bookkeeping after the owning run finishes", async () => {+ const f = await localLease(); expect(await read()).toEqual(none);+ expect(await db.select().from(environmentLeases)).toEqual([f.lease]);+ });+ it.each(["provider_resource", "missing_release", "live_run", "unknown_driver", "pending_export", "retained"])("retains local lease work with %s evidence", async kind => {+ const f = await localLease();+ if (kind === "provider_resource") await db.update(environmentLeases).set({ providerLeaseId: "unexpected-resource" }).where(eq(environmentLeases.id, f.lease.id));+ if (kind === "missing_release") await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, f.lease.id));+ if (kind === "live_run") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));+ if (kind === "unknown_driver") await db.update(environments).set({ driver: "ssh" }).where(eq(environments.id, f.environment.id));+ if (kind === "pending_export") await db.update(environmentLeases).set({ metadata: { driver: "local", nativeWorkspaceExportResume: {} } }).where(eq(environmentLeases.id, f.lease.id));+ if (kind === "retained") await db.update(environmentLeases).set({ leasePolicy: "retain_on_failure" }).where(eq(environmentLeases.id, f.lease.id));+ expect(await read()).toEqual(present);+ });+});diff --git a/server/src/__tests__/idle-response-waits.test.ts b/server/src/__tests__/idle-response-waits.test.tsnew file mode 100644index 0000000000..58dafafad9--- /dev/null+++ b/server/src/__tests__/idle-response-waits.test.ts@@ -0,0 +1,170 @@+import { randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";+import {+ agents, companies, createDb, heartbeatRuns, issues, issueRecoveryActions,+ issueThreadInteractions, routines, routineTriggers,+ toolApplications, toolConnections,+} from "@paperclipai/db";+import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";+import { getMergeConfirmationPullRequestReferences } from "../services/issue-thread-interactions.js";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";++const now = Date.UTC(2020, 0, 1);+const ownerId = randomUUID();+const owned = (): IdleSleepDrainStatus => ({ ownerId, draining: true, startedAt: new Date(now - 1000),+ expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 });+const none = { version: 1, backgroundWork: "none" };+const present = { version: 1, backgroundWork: "present" };+const support = await getEmbeddedPostgresTestSupport();+if (!support.supported) console.warn(`Skipping response-wait database tests: ${support.reason}`);+(support.supported ? describe : describe.skip)("idle response-driven waits", () => {+ let db: ReturnType<typeof createDb>;+ let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ beforeAll(async () => {+ database = await startEmbeddedPostgresTestDatabase("idle-response-waits-");+ db = createDb(database.connectionString);+ }, 60_000);+ afterEach(async () => { await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); });+ afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });+ const read = (schedules = true) => readIdleSleepSafety(db, owned, () => now, ownerId,+ async () => "none", undefined, undefined, schedules, schedules);++ async function seed(status = "in_review") {+ const companyId = randomUUID(); const agentId = randomUUID(); const runId = randomUUID(); const issueId = randomUUID();+ await db.insert(companies).values({ id: companyId, name: "Response fixture", issuePrefix: `T${companyId.slice(0, 6).toUpperCase()}` });+ await db.insert(agents).values({ id: agentId, companyId, name: "Idle owner", role: "engineer", status: "idle", adapterType: "process" });+ await db.insert(issues).values({ id: issueId, companyId, title: "Awaiting a response", status, assigneeAgentId: agentId });+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand",+ status: "succeeded", finishedAt: new Date(now), contextSnapshot: { issueId } });+ return { companyId, agentId, runId, issueId };+ }+ async function wait(f: Awaited<ReturnType<typeof seed>>, change: Partial<typeof issueThreadInteractions.$inferInsert> = {}) {+ const [row] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,+ kind: "ask_user_questions", status: "pending", sourceRunId: f.runId, effectiveResolverPolicy: "human_only",+ continuationPolicy: "wake_assignee", payload: { version: 1 }, ...change }).returning();+ return row!;+ }++ it.each(["anyone", "not_creator", "human_only"] as const)("preserves the %s response audience while permitting an idle wait", async policy => {+ const f = await seed(); const card = await wait(f, { effectiveResolverPolicy: policy });+ expect(await read(false)).toEqual(present);+ expect(await read()).toEqual(none);+ expect(await db.select().from(issueThreadInteractions)).toEqual([card]);+ expect((await db.select().from(issues))[0]?.status).toBe("in_review");+ });+ it.each(["ask_user_questions", "request_confirmation", "request_checkbox_confirmation", "suggest_tasks"])(+ "permits a persisted %s wait after its producer finishes", async kind => {+ await wait(await seed("in_progress"), { kind });+ expect(await read()).toEqual(none);+ });+ it.each(["human_only", "anyone"])("keeps automatically polled merge confirmations awake with %s audience", async policy => {+ const f = await seed("in_progress");+ const card = { kind: "request_confirmation", title: "Merge PaperclipAI/paperclip#39?", summary: null,+ payload: { version: 1, prompt: "Merge the pull request?" } };+ expect(getMergeConfirmationPullRequestReferences(card)).toHaveLength(1);+ await wait(f, { ...card, effectiveResolverPolicy: policy });+ expect(await read()).toEqual(present);+ });+ it("does not treat an assigned task or an answered interaction as a passive wait", async () => {+ const f = await seed(); expect(await read()).toEqual(present);+ const card = await wait(f); expect(await read()).toEqual(none);+ await db.update(issueThreadInteractions).set({ status: "answered", resolvedAt: new Date(now) }).where(eq(issueThreadInteractions.id, card.id));+ expect(await read()).toEqual(present);+ });+ it("allows human-only connection setup without reusable access, and reblocks when access appears", async () => {+ const f = await seed();+ await wait(f, { kind: "connection_intent", addresseeUserId: "fixture-user", payload: {+ version: 1, serviceSlug: "fixture-service", requestingAgentId: f.agentId,+ } });+ expect(await read()).toEqual(none);+ const [app] = await db.insert(toolApplications).values({ companyId: f.companyId,+ type: "mcp_http", name: "Fixture service" }).returning();+ await db.insert(toolConnections).values({ companyId: f.companyId, applicationId: app!.id,+ uid: randomUUID(), name: "Reusable access", connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true });+ expect(await read()).toEqual(present);+ });+ it.each(["wrong_assignee", "unknown_purpose", "no_addressee", "agent_resolver"])("retains unsafe connection wait: %s", async mode => {+ const f = await seed();+ await wait(f, { kind: "connection_intent", addresseeUserId: mode === "no_addressee" ? null : "fixture-user",+ effectiveResolverPolicy: mode === "agent_resolver" ? "anyone" : "human_only", payload: {+ version: 1, serviceSlug: "fixture-service", requestingAgentId: mode === "wrong_assignee" ? randomUUID() : f.agentId,+ ...(mode === "unknown_purpose" ? { purpose: "future" } : {}),+ } });+ expect(await read()).toEqual(present);+ });+ it.each([+ { kind: "connection_intent" }, { kind: "unknown" }, { continuationPolicy: "automatic" },+ { effectiveResolverPolicy: "unknown" }, { resolvedAt: new Date(now) },+ ])("keeps unsupported waiting work awake: %j", async change => {+ const f = await seed(); await wait(f); await wait(f, change);+ expect(await read()).toEqual(present);+ });+ it("does not borrow another company's interaction or ignore a foreign pending sibling", async () => {+ const f = await seed(); const other = await seed("done");+ await wait(f, { companyId: other.companyId, sourceRunId: other.runId });+ expect(await read()).toEqual(present);+ await wait(f); expect(await read()).toEqual(present);+ });+ it.each([+ { status: "running", finishedAt: null }, { finishedAt: null }, { costAccountingPending: true },+ { scheduledRetryAt: new Date(now + 1000) },+ ])("keeps a response producer's unfinished work awake: %j", async change => {+ const f = await seed(); await wait(f);+ await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, f.runId));+ expect(await read()).toEqual(present);+ });+ it.each([+ { checkoutRunId: randomUUID() }, { executionRunId: randomUUID() },+ { executionState: { status: "pending" } }, { monitorWakeRequestedAt: new Date(now) },+ ])("does not let a response card hide issue execution: %j", async change => {+ const f = await seed(); await wait(f);+ // Checkout and execution fields reference real run rows.+ const patch = { ...change };+ if ("checkoutRunId" in patch) patch.checkoutRunId = f.runId;+ if ("executionRunId" in patch) patch.executionRunId = f.runId;+ await db.update(issues).set(patch).where(eq(issues.id, f.issueId));+ expect(await read()).toEqual(present);+ });+ it("includes response identity and audience in the future schedule generation", async () => {+ const f = await seed(); const card = await wait(f);+ const [routine] = await db.insert(routines).values({ companyId: f.companyId, title: "Later work", status: "active", assigneeAgentId: f.agentId }).returning();+ await db.insert(routineTriggers).values({ companyId: f.companyId, routineId: routine!.id, kind: "schedule",+ cronExpression: "*/15 * * * *", timezone: "UTC", nextRunAt: new Date(now + 600_000), enabled: true, setupPending: false });+ const before = await read(); expect(before.version).toBe(2);+ await db.update(issueThreadInteractions).set({ effectiveResolverPolicy: "anyone" }).where(eq(issueThreadInteractions.id, card.id));+ const after = await read(); expect(after.scheduledWake?.nextWakeAt).toBe(before.scheduledWake?.nextWakeAt);+ expect(after.scheduledWake?.generation).not.toBe(before.scheduledWake?.generation);+ });++ async function boardHold(f: Awaited<ReturnType<typeof seed>>, change: Partial<typeof issueRecoveryActions.$inferInsert> = {}) {+ const [row] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,+ kind: "stranded_assigned_issue", cause: "stranded_assigned_issue", status: "active", ownerType: "board",+ fingerprint: randomUUID(), nextAction: "Board decision required", wakePolicy: { type: "board_escalation", reason: "stranded_assigned_issue", preservesSourceAssignee: true },+ ...change }).returning();+ return row!;+ }+ it.each(["active", "escalated"])("retains visible %s board escalation history without keeping compute running", async status => {+ const f = await seed("blocked"); const action = await boardHold(f, { status });+ expect(await read()).toEqual(none);+ expect(await db.select().from(issueRecoveryActions)).toEqual([action]);+ });+ it.each([+ { ownerType: "agent" }, { ownerType: "system" }, { cause: "native_workspace_finalization_owner_unverified" },+ { timeoutAt: new Date(now + 60_000) }, { monitorPolicy: {} },+ { wakePolicy: { type: "wake_owner" } }, { wakePolicy: { type: "board_escalation", retryAt: "2099-01-01" } },+ { wakePolicy: null }, { resolvedAt: new Date(now) },+ ])("keeps automatic or unproved recovery work awake: %j", async change => {+ await boardHold(await seed("blocked"), change);+ expect(await read()).toEqual(present);+ });+ it("rejects a foreign recovery source and an escalation whose source still runs", async () => {+ const f = await seed("blocked"); const other = await seed("done");+ const action = await boardHold(f, { companyId: other.companyId });+ expect(await read()).toEqual(present);+ await db.update(issueRecoveryActions).set({ companyId: f.companyId }).where(eq(issueRecoveryActions.id, action.id));+ expect(await read()).toEqual(none);+ await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));+ expect(await read()).toEqual(present);+ });+});diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.tsindex ad10c180c7..befce2fac6 100644--- a/server/src/__tests__/idle-sleep-safety.test.ts+++ b/server/src/__tests__/idle-sleep-safety.test.ts@@ -592,13 +592,13 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp},);- it("retains remote MCP connections alongside inert AI credentials", async () => {+ it("allows saved outbound MCP connections alongside inert AI credentials", async () => {const { companyId, connection } = await seedAiConnection();await db.insert(toolConnections).values({companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(),connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true,});- expect(await read()).toEqual(present);+ expect(await read()).toEqual(none);});async function seedFinishedFailedLease() {diff --git a/server/src/__tests__/plugin-inspection-concurrency.test.ts b/server/src/__tests__/plugin-inspection-concurrency.test.tsnew file mode 100644index 0000000000..41c57323db--- /dev/null+++ b/server/src/__tests__/plugin-inspection-concurrency.test.ts@@ -0,0 +1,38 @@+import path from "node:path";+import { fileURLToPath } from "node:url";+import { expect, it } from "vitest";+import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";+import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";+import { idleWorkSnapshot } from "../services/task-admission.js";++it("bounds actual provider reads across caller timeouts, plugins, retries, and worker exit", async () => {+ const baseline = idleWorkSnapshot().active;+ const manager = createPluginWorkerManager();+ const manifest: PaperclipPluginManifestV1 = { id: "fixture.inspect", apiVersion: 1, version: "1.0.0",+ displayName: "Inspection fixture", categories: ["automation"], capabilities: [], entrypoints: { worker: "worker.cjs" } };+ const params = { driverKey: "fixture", companyId: "fixture-company", environmentId: "fixture-environment",+ config: {}, providerLeaseId: "fixture-resource", leaseMetadata: { holdInspection: true } };+ try {+ for (const pluginId of ["fixture.inspect.a", "fixture.inspect.b"]) await manager.startWorker(pluginId, {+ entrypointPath: path.join(path.dirname(fileURLToPath(import.meta.url)), "fixtures/plugin-worker-delayed.cjs"),+ manifest, config: {}, instanceInfo: { instanceId: "fixture-instance", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {},+ });+ await Promise.all(["fixture.inspect.a", "fixture.inspect.b"].map(pluginId =>+ expect(manager.call(pluginId, "environmentInspectLease", params, 20)).rejects.toThrow("timed out")));+ expect(idleWorkSnapshot().active).toBe(baseline + 2);+ for (let retry = 0; retry < 4; retry++) await expect(manager.call("fixture.inspect.a", "environmentInspectLease", params, 20)).rejects.toThrow("concurrency limit");+ for (const pluginId of ["fixture.inspect.a", "fixture.inspect.b"]) expect(await manager.call(pluginId, "health", {})).toMatchObject({+ details: { inspectionCount: 1, activeInspections: 1 },+ });+ await manager.call("fixture.inspect.a", "configChanged", { companyId: "fixture-company", config: { releaseInspections: true } });+ expect(idleWorkSnapshot().active).toBe(baseline + 1);+ await expect(manager.call("fixture.inspect.a", "environmentInspectLease", params, 20)).rejects.toThrow("timed out");+ await expect(manager.call("fixture.inspect.b", "environmentInspectLease", params, 20)).rejects.toThrow("concurrency limit");+ await manager.stopWorker("fixture.inspect.b");+ expect(idleWorkSnapshot().active).toBe(baseline + 1);+ await expect(manager.call("fixture.inspect.a", "environmentInspectLease", { ...params, leaseMetadata: {} }, 1000)).resolves.toEqual({+ providerLeaseId: "fixture-resource", state: "destroyed",+ });+ } finally { await manager.stopAll(); }+ expect(idleWorkSnapshot().active).toBe(baseline);+}, 15_000);diff --git a/server/src/__tests__/routine-status-reconciliation.test.ts b/server/src/__tests__/routine-status-reconciliation.test.tsnew file mode 100644index 0000000000..eba0718ce2--- /dev/null+++ b/server/src/__tests__/routine-status-reconciliation.test.ts@@ -0,0 +1,87 @@+import { randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";+import { companies, createDb, issues, routineRuns, routines } from "@paperclipai/db";+import { routineService } from "../services/routines.js";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";++const support = await getEmbeddedPostgresTestSupport();+if (!support.supported) console.warn(`Skipping routine status database tests: ${support.reason}`);+(support.supported ? describe : describe.skip)("routine status reconciliation", () => {+ let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("routine-status-"); db = createDb(database.connectionString); }, 60_000);+ afterEach(async () => { vi.restoreAllMocks(); await db.execute(sql`SET client_min_messages = warning; TRUNCATE companies CASCADE`); });+ afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });+ async function seed(status = "blocked") {+ const companyId = randomUUID(), runId = randomUUID();+ await db.insert(companies).values({ id: companyId, name: "Routine fixture", issuePrefix: `T${companyId.slice(0, 6)}` });+ const [routine] = await db.insert(routines).values({ companyId, title: "Routine fixture" }).returning();+ const [issue] = await db.insert(issues).values({ companyId, title: "Execution fixture", status,+ originKind: "routine_execution", originId: routine!.id, originRunId: runId }).returning();+ const [run] = await db.insert(routineRuns).values({ id: runId, companyId, routineId: routine!.id,+ linkedIssueId: issue!.id, source: "schedule", status: "issue_created", triggerPayload: { retainedInput: true } }).returning();+ return { companyId, routine: routine!, issue: issue!, run: run! };+ }+ it.each(["done", "blocked", "cancelled"])("repairs missed %s status using normal owning-service transitions", async status => {+ const f = await seed(status); const service = routineService(db);+ expect(await service.reconcileExecutionRunStatuses()).toEqual({ scanned: 1, settled: 1, deferred: 0 });+ const [run] = await db.select().from(routineRuns);+ expect(run!.status).toBe(status === "done" ? "completed" : "failed");+ expect(run!.completedAt).toBeInstanceOf(Date); expect(run!.triggerPayload).toMatchObject({ retainedInput: true });+ expect((await db.select().from(issues))[0]).toEqual(f.issue);+ expect(await service.reconcileExecutionRunStatuses()).toEqual({ scanned: 0, settled: 0, deferred: 0 });+ if (status !== "done") {+ await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issue.id));+ await service.syncRunStatusForIssue(f.issue.id);+ expect((await db.select().from(routineRuns))[0]).toMatchObject({ status: "issue_created", completedAt: null });+ }+ });+ it.each(["active_issue", "wrong_origin", "wrong_routine", "wrong_run", "wrong_company", "wrong_routine_company", "different_status"])("does not settle an unsupported handoff: %s", async mode => {+ const f = await seed();+ if (mode === "active_issue") await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issue.id));+ if (mode === "wrong_origin") await db.update(issues).set({ originKind: "manual" }).where(eq(issues.id, f.issue.id));+ if (mode === "wrong_routine") await db.update(issues).set({ originId: randomUUID() }).where(eq(issues.id, f.issue.id));+ if (mode === "wrong_run") await db.update(issues).set({ originRunId: randomUUID() }).where(eq(issues.id, f.issue.id));+ if (mode === "wrong_company") { const other = await seed("done"); await db.update(routineRuns).set({ companyId: other.companyId }).where(eq(routineRuns.id, f.run.id)); }+ if (mode === "wrong_routine_company") { const other = await seed("done");+ await db.update(routineRuns).set({ routineId: other.routine.id }).where(eq(routineRuns.id, f.run.id));+ await db.update(issues).set({ originId: other.routine.id }).where(eq(issues.id, f.issue.id)); }+ if (mode === "different_status") await db.update(routineRuns).set({ status: "received" }).where(eq(routineRuns.id, f.run.id));+ const before = await db.select().from(routineRuns);+ const result = await routineService(db).reconcileExecutionRunStatuses();+ expect(result.settled).toBe(["wrong_company", "wrong_routine_company"].includes(mode) ? 1 : 0);+ expect((await db.select().from(routineRuns)).find(row => row.id === f.run.id)).toEqual(before.find(row => row.id === f.run.id));+ if (mode === "wrong_routine_company") await expect(routineService(db).syncRunStatusForIssue(f.issue.id)).resolves.toBeNull();+ });+ it.each(["resume", "retarget", "routine_transfer"])("rechecks %s after candidate selection and before repair", async mode => {+ const f = await seed(); const transaction = db.transaction.bind(db);+ vi.spyOn(db, "transaction").mockImplementationOnce(async callback => {+ if (mode === "routine_transfer") {+ const otherId = randomUUID(); await db.insert(companies).values({ id: otherId, name: "Other fixture", issuePrefix: `T${otherId.slice(0, 6)}` });+ await db.update(routines).set({ companyId: otherId }).where(eq(routines.id, f.routine.id));+ } else await db.update(issues).set(mode === "resume" ? { status: "in_progress" } : { originRunId: randomUUID() }).where(eq(issues.id, f.issue.id));+ return transaction(callback);+ });+ expect(await routineService(db).reconcileExecutionRunStatuses()).toEqual({ scanned: 1, settled: 0, deferred: 0 });+ expect((await db.select().from(routineRuns))[0]).toEqual(f.run);+ });+ it("repairs another routine while one row is locked, then retries the deferred owner", async () => {+ const [held, other] = [await seed(), await seed()].sort((a, b) => a.run.id.localeCompare(b.run.id));+ let entered!: () => void, release!: () => void;+ const locked = new Promise<void>(resolve => { entered = resolve; });+ const gate = new Promise<void>(resolve => { release = resolve; });+ const holding = db.transaction(async tx => {+ await tx.select().from(routines).where(eq(routines.id, held!.routine.id)).for("update");+ entered(); await gate;+ });+ try {+ await Promise.race([locked, holding.then(() => { throw new Error("Lock transaction ended before entry"); })]);+ const service = routineService(db);+ expect(await service.reconcileExecutionRunStatuses()).toEqual({ scanned: 2, settled: 1, deferred: 1 });+ expect((await db.select().from(routineRuns)).find(row => row.id === other!.run.id)?.status).toBe("failed");+ expect((await db.select().from(routineRuns)).find(row => row.id === held!.run.id)?.status).toBe("issue_created");+ release(); await holding;+ expect(await service.reconcileExecutionRunStatuses()).toEqual({ scanned: 1, settled: 1, deferred: 0 });+ } finally { release(); await holding; }+ });+});diff --git a/server/src/__tests__/satisfied-connection-intents.test.ts b/server/src/__tests__/satisfied-connection-intents.test.tsindex ed37e0b633..8c8f4a5330 100644--- a/server/src/__tests__/satisfied-connection-intents.test.ts+++ b/server/src/__tests__/satisfied-connection-intents.test.ts@@ -14,6 +14,9 @@ import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-worimport { issueThreadInteractionService } from "../services/issue-thread-interactions.js";import { legacyExecutionNeedsReconciliationWithEvidence } from "../services/legacy-execution-recovery.js";import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";+import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";+import { idleWorkSnapshot } from "../services/task-admission.js";+import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";const support = await getEmbeddedPostgresTestSupport();const describePostgres = support.supported ? describe : describe.skip;@@ -86,6 +89,60 @@ describePostgres("already available connection requests", () => {} finally { unsubscribe(); }});+ it("releases a created human setup card's worker hold, then resumes when access becomes available", async () => {+ const f = await seed();+ await db.delete(issueThreadInteractions);+ await db.update(toolConnections).set({ status: "draft", enabled: false }).where(eq(toolConnections.id, f.connection.id));+ const [source] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,+ status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: f.issueId } }).returning();+ let cardId = "";+ const wakeup = vi.fn(async () => {+ await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",+ status: "queued", idempotencyKey: `connection-intent:${cardId}:expired` });+ return null;+ });+ const service = connectionIntentDeliveryService(db, { wakeup });+ const errors: unknown[] = [];+ const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { errors.push(error); } });+ let sawPassive!: () => void, sawDelivered!: () => void;+ const passive = new Promise<void>(resolve => { sawPassive = resolve; });+ const delivered = new Promise<void>(resolve => { sawDelivered = resolve; });+ const wait = async (event: Promise<void>) => {+ let timer: ReturnType<typeof setTimeout> | undefined;+ try { await Promise.race([event, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error("Connection worker did not settle")), 8000); })]); }+ finally { clearTimeout(timer); }+ await new Promise<void>(resolve => setImmediate(resolve));+ };+ try {+ await coordinator.register(DELIVERY_QUEUES.connection, { retryMs: 5000, run: () => service.sweepPending(),+ hasPending: async () => { const pending = await service.hasPending();+ if (!pending && service.nextRunAt() !== null) sawPassive();+ if (!pending && wakeup.mock.calls.length > 0) sawDelivered();+ return pending;+ }, nextRunAt: service.nextRunAt,+ }).ready;+ const card = await issueThreadInteractionService(db).createConnectionIntent({ id: f.issueId, companyId: f.companyId }, {+ payload: f.intent.payload, sourceRunId: source!.id, addresseeUserId: "connection-owner", idempotencyKey: "passive-setup-card",+ });+ cardId = card.id;+ await wait(passive);+ expect(errors).toEqual([]); expect(idleWorkSnapshot().active).toBe(0);+ expect((await db.select().from(issueThreadInteractions))[0].status).toBe("pending");+ const now = Date.now(), ownerId = randomUUID();+ const drain = (): IdleSleepDrainStatus => ({ ownerId, draining: true, startedAt: new Date(now - 1000),+ expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 });+ expect(await readIdleSleepSafety(db, drain, Date.now, ownerId, async () => "none", undefined, undefined, true, true)).toMatchObject({ backgroundWork: "none" });+ // This mutation has no queue notification: awake-only maintenance must+ // still discover new access; after sleep startup performs the same scan.+ await db.update(toolConnections).set({ status: "active", enabled: true }).where(eq(toolConnections.id, f.connection.id));+ await wait(delivered);+ expect(errors).toEqual([]); expect(wakeup).toHaveBeenCalledOnce();+ expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).not.toBeNull();+ expect((await db.select().from(agentWakeupRequests))[0]).toMatchObject({ status: "queued", idempotencyKey: `connection-intent:${cardId}:expired` });+ expect(await readIdleSleepSafety(db, drain, Date.now, ownerId, async () => "none", undefined, undefined, true, true)).toMatchObject({ backgroundWork: "present" });+ } finally { await coordinator.stop(); }+ }, 20_000);+it.each(["wrong_identity", "revoked", "disabled", "removed_member", "denied_policy"] as const)("rechecks %s after the card retires", async (reason) => {const f = await seed(); const run = await retry(f);await satisfiedConnectionIntentService(db).sweepPending();diff --git a/server/src/index.ts b/server/src/index.tsindex 264adcd558..409b35da93 100644--- a/server/src/index.ts+++ b/server/src/index.ts@@ -107,6 +107,8 @@ import { deliverNativeQuestionResponse } from "./services/native-runtime/native-import { queueIssueAssignmentWakeup } from "./services/issue-assignment-wakeup.js";import { createSecretProposalsService } from "./services/secret-proposals.js";import { environmentRuntimeService } from "./services/environment-runtime.js";+import { historicalCleanupVerificationService } from "./services/historical-cleanup-verification.js";+import { isCloudManagedInstance } from "./services/cloud-instance.js";import { createDbAdapterAuthSessionStore } from "./services/device-login-service.js";import {createDeviceLoginReaper,@@ -1327,6 +1329,7 @@ async function startServerWithDatabaseTeardown(retryMs: config.heartbeatSchedulerIntervalMs,run: () => service.sweepPending(),hasPending: () => service.hasPending(),+ ...(queue === DELIVERY_QUEUES.connection ? { nextRunAt: connectionDeliveries.nextRunAt, pausePassiveMaintenance: true } : {}),}));// Preserve the existing task-scoped activity fast path. Claims in the// delivery service arbitrate with the coordinator's startup/recovery sweep.@@ -1337,10 +1340,27 @@ async function startServerWithDatabaseTeardown(.catch(err => logger.error({ err }, "post-commit chat completion delivery failed")));});server.on("close", unsubscribeChatCompletions);+ const cleanupInspectionRuntime = environmentRuntimeService(db as any, { pluginWorkerManager });+ const environmentCustomImages = environmentCustomImageService(db as any, { pluginWorkerManager });+ const historicalCleanup = historicalCleanupVerificationService(db as any, {+ inspectLease: lease => cleanupInspectionRuntime.inspectLeaseResource(lease.id),+ inspectSetup: setup => environmentCustomImages.inspectSetupResource(setup),+ });+ let nextHistoricalCleanupCheck = 0;+ const scheduleHistoricalCleanupVerification = () => {+ if (!isCloudManagedInstance() || isIdleTaskDrainActive() || isWarmStandby() || heartbeatSchedulerStopped ||+ Date.now() < nextHistoricalCleanupCheck) return;+ nextHistoricalCleanupCheck = Date.now() + 5 * 60_000;+ // Provider reads never delay readiness or the ordinary cleanup sweep.+ trackHeartbeatSchedulerWork(historicalCleanup.sweep().then(result => {+ if (result.verified > 0) logger.info(result, "verified historical provider cleanup");+ }).catch(() => logger.warn("historical provider cleanup verification deferred")));+ };const runEnvironmentLeaseCleanupSweep = (backoffMs: number) =>environmentLeaseCleanupHeartbeat.sweepPendingCleanupLeases({ backoffMs }).then((result) => {+ scheduleHistoricalCleanupVerification();if (result.destroyed > 0 || result.capped > 0) {logger.info(result, "environment lease cleanup sweep retried orphan sandbox teardowns");}@@ -1405,8 +1425,16 @@ async function startServerWithDatabaseTeardown(drainHeartbeatExecutionFinalizers = () =>heartbeat.drainActiveRunExecutions();prepareHotRestartShutdown = heartbeat.prepareHotRestartShutdown;- const environmentCustomImages = environmentCustomImageService(db as any, { pluginWorkerManager });const routines = routineService(db as any, { pluginWorkerManager });+ let nextRoutineStatusReconcile = 0;+ const scheduleRoutineStatusReconcile = () => {+ if (!isCloudManagedInstance() || isWarmStandby() || isIdleTaskDrainActive() || heartbeatSchedulerStopped ||+ Date.now() < nextRoutineStatusReconcile) return;+ nextRoutineStatusReconcile = Date.now() + 5 * 60_000;+ trackHeartbeatSchedulerWork(routines.reconcileExecutionRunStatuses().then(result => {+ if (result.settled > 0) logger.info(result, "reconciled routine execution issue status");+ }).catch(() => logger.warn("routine execution status reconciliation deferred")));+ };const statusCards = statusCardService(db as any);const issues = issueService(db as any);const mergedPullRequestConfirmations = issueThreadInteractionService(db as any, {@@ -1725,6 +1753,7 @@ async function startServerWithDatabaseTeardown(// captures the authoritative set of running heartbeat rows.trackHeartbeatSchedulerWork((async () => {if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;+ scheduleRoutineStatusReconcile();trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired().catch((err: unknown) => {logger.error({ err }, "decision expiry sweep failed");}));diff --git a/server/src/services/cleanup-verification-proof.ts b/server/src/services/cleanup-verification-proof.tsnew file mode 100644index 0000000000..13bb993074--- /dev/null+++ b/server/src/services/cleanup-verification-proof.ts@@ -0,0 +1,32 @@+export const CLEANUP_VERIFICATION_KEY = "paperclipCleanupVerification";+export const CLEANUP_VERIFICATION_SCHEMA = "paperclip.cleanup-verification.v1";+export const CLEANUP_ROW_STAMP_FORMAT = 'YYYY-MM-DD"T"HH24:MI:SS.US';++/** Provider-supplied metadata cannot create host verification evidence. */+export function withoutCleanupVerification(metadata: Record<string, unknown> | null | undefined) {+ if (!metadata || typeof metadata !== "object" || Array.isArray(metadata)) return null;+ const { [CLEANUP_VERIFICATION_KEY]: _ignored, ...rest } = metadata;+ return rest;+}++export function cleanupRowStamp(alias: string): string {+ return `to_char(${alias}.updated_at AT TIME ZONE 'UTC', '${CLEANUP_ROW_STAMP_FORMAT}')`;+}++/** Fixed aliases only; never interpolate request input into these predicates. */+export function verifiedResourceCleanup(alias: string, kind: "lease" | "image_setup"): string {+ const proof = `${alias}.metadata->'${CLEANUP_VERIFICATION_KEY}'`;+ const company = kind === "lease" ? `${alias}.company_id::text`+ : `COALESCE(NULLIF(btrim(${alias}.metadata->>'setupRpcCompanyId'), ''), 'instance')`;+ return `jsonb_typeof(${proof}) = 'object'+ AND ${proof}->>'schema' = '${CLEANUP_VERIFICATION_SCHEMA}' AND ${proof}->>'kind' = '${kind}'+ AND ${proof}->>'resourceId' = ${alias}.id::text AND ${proof}->>'companyId' = ${company}+ AND ${proof}->>'environmentId' = ${alias}.environment_id::text+ AND ${proof}->>'provider' = ${alias}.provider+ AND ${proof}->>'providerLeaseId' = ${alias}.provider_lease_id+ AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> ''+ AND ${proof}->>'state' = 'destroyed'+ AND ${proof}->>'rowUpdatedAt' = ${cleanupRowStamp(alias)}+ ${kind === "lease" ? `AND ${proof} ? 'heartbeatRunId'+ AND ${proof}->'heartbeatRunId' = COALESCE(to_jsonb(${alias}.heartbeat_run_id::text), 'null'::jsonb)` : ""}`;+}diff --git a/server/src/services/connection-intent-delivery.ts b/server/src/services/connection-intent-delivery.tsindex 3a5c8dfba4..561cd6e7ba 100644--- a/server/src/services/connection-intent-delivery.ts+++ b/server/src/services/connection-intent-delivery.ts@@ -7,6 +7,7 @@ import type { heartbeatService } from "./heartbeat.js";import { issueService } from "./issues.js";import { issueRecoveryActionService } from "./issue-recovery-actions.js";import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js";+import { connectionDeliveryWaitingForResponse, connectionIntentWaitingForResponse } from "./idle-work-proof.js";type Heartbeat = ReturnType<typeof heartbeatService>;export async function wakeConnectionIntentAfterResolution(@@ -215,18 +216,31 @@ export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbeaeq(connectionIntentDeliveries.nextAttemptAt, claimed.nextAttemptAt), isNull(connectionIntentDeliveries.deliveredAt)));}}+ let maintenanceAt: number | null = null;async function hasPending() {- if ((await db.select({ id: connectionIntentDeliveries.interactionId }).from(connectionIntentDeliveries)- .where(isNull(connectionIntentDeliveries.deliveredAt)).limit(1)).length > 0) return true;- return (await db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions)- .innerJoin(issues, and(eq(issues.id, issueThreadInteractions.issueId), eq(issues.companyId, issueThreadInteractions.companyId)))- .where(and(eq(issueThreadInteractions.kind, "connection_intent"), eq(issueThreadInteractions.status, "pending"),- inArray(issues.status, ["in_progress", "in_review"]), isNull(issues.assigneeUserId)))- .limit(1)).length > 0;+ const [state] = await db.execute<{ active: boolean; present: boolean }>(sql`+ WITH pending_deliveries AS (+ SELECT (${sql.raw(connectionDeliveryWaitingForResponse('d'))}) IS TRUE AS passive+ FROM connection_intent_deliveries d WHERE d.delivered_at IS NULL+ ), pending_intents AS (+ SELECT (${sql.raw(connectionIntentWaitingForResponse('card', 'task'))}) IS TRUE AS passive+ FROM issue_thread_interactions card LEFT JOIN issues task ON task.id = card.issue_id AND task.company_id = card.company_id+ WHERE card.kind = 'connection_intent' AND card.status = 'pending'+ AND (task.id IS NULL OR (task.status IN ('in_progress', 'in_review') AND task.assignee_user_id IS NULL))+ ) SELECT EXISTS (SELECT 1 FROM pending_deliveries WHERE NOT passive+ UNION ALL SELECT 1 FROM pending_intents WHERE NOT passive) AS active,+ EXISTS (SELECT 1 FROM pending_deliveries UNION ALL SELECT 1 FROM pending_intents) AS present`);+ if (!state || typeof state.active !== "boolean" || typeof state.present !== "boolean") throw new Error("Connection work state is unknown");+ // Passive records need no admitted-work hold or wake alarm. While the+ // process is already awake, keep the existing reconciliation cadence so+ // new access or responses resume work even without a queue notification.+ maintenanceAt = !state.active && state.present ? Date.now() + 5_000 : null;+ return state.active;}- return { deliver, hasPending, tryDeliver: async (id: string) => { try { await deliver(id); } catch { /* Persisted delivery remains due after its lease. */ } }, sweepPending: async () => {+ return { deliver, hasPending, nextRunAt: () => maintenanceAt, tryDeliver: async (id: string) => { try { await deliver(id); } catch { /* Persisted delivery remains due after its lease. */ } }, sweepPending: async () => {await satisfiedIntents.sweepPending();- const rows = await db.select().from(connectionIntentDeliveries).where(and(isNull(connectionIntentDeliveries.deliveredAt), lte(connectionIntentDeliveries.nextAttemptAt, new Date())))+ const rows = await db.select().from(connectionIntentDeliveries).where(and(isNull(connectionIntentDeliveries.deliveredAt), lte(connectionIntentDeliveries.nextAttemptAt, new Date()),+ sql`(${sql.raw(connectionDeliveryWaitingForResponse('connection_intent_deliveries'))}) IS NOT TRUE`)).orderBy(asc(connectionIntentDeliveries.nextAttemptAt)).limit(50);let failed = 0;for (const row of rows) { try { await deliver(row.interactionId); } catch { failed += 1; } }diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.tsindex 2efb832250..08f8dc4715 100644--- a/server/src/services/delivery-work-coordinator.ts+++ b/server/src/services/delivery-work-coordinator.ts@@ -26,6 +26,9 @@ export function createDeliveryWorkCoordinator(input: {onSettled?: (hints?: ReadonlySet<string>) => void;// Optional maintenance/retry deadline restored from durable state.nextRunAt?: () => number | null;+ // An empty queue's maintenance may wait until dispatch admission reopens.+ // Notifications and unresolved producer writes still reconcile in drain.+ pausePassiveMaintenance?: boolean;}) {if (stopped) throw new Error("Delivery coordinator is stopped");if (workers.has(queue)) throw new Error(`Delivery worker already registered: ${queue}`);@@ -35,6 +38,7 @@ export function createDeliveryWorkCoordinator(input: {let workerStopped = false;let controller: AbortController | null = null;let finishIntent: (() => void) | undefined;+ let pendingWork = true;function schedule(delay: number, deadline?: number) {if (workerStopped) return;const at = deadline ?? Date.now() + delay;@@ -47,13 +51,22 @@ export function createDeliveryWorkCoordinator(input: {scheduler.cancel(queue);scheduledAt = null;if (!input.canRun() && !(input.canReconcile?.() ?? false)) { schedule(task.retryMs); return; }+ if (!input.canRun() && task.pausePassiveMaintenance && !pendingWork && !dirty &&+ !finishIntent && !databaseWorkPending(input.owner, queue)) {+ // A timer alone must not change the idle generation during a slow+ // backup. Retry without querying or admitting work; a notification+ // still preempts this deadline and retains its normal intent hold.+ schedule(task.retryMs);+ return;+ }dirty = false;const finish = beginIdleTrackedWork();const attempt = new AbortController();controller = attempt;running = Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue)).then(() => workerStopped || !input.canRun() ? undefined : task.run(attempt.signal)).then(() => workerStopped ? false : task.hasPending()).then(pending => {- if (pending || databaseWorkPending(input.owner, queue)) schedule(task.retryMs);+ pendingWork = pending || databaseWorkPending(input.owner, queue);+ if (pendingWork) schedule(task.retryMs);else { finishIntent?.(); finishIntent = undefined; }const next = task.nextRunAt?.();if (next != null) {diff --git a/server/src/services/environment-custom-images.ts b/server/src/services/environment-custom-images.tsindex f377b2bd1a..7f81a93508 100644--- a/server/src/services/environment-custom-images.ts+++ b/server/src/services/environment-custom-images.ts@@ -1,4 +1,5 @@import { randomUUID } from "node:crypto";+import { withoutCleanupVerification } from "./cleanup-verification-proof.js";import { and, desc, eq, inArray, lte } from "drizzle-orm";import type { Db } from "@paperclipai/db";import {@@ -22,6 +23,7 @@ import type {PluginEnvironmentCaptureTemplateResult,PluginEnvironmentInteractiveSetupConnectionPayload,PluginEnvironmentInteractiveSetupSession,+ PluginEnvironmentLeaseObservation,PluginEnvironmentTemplateRefKind,} from "@paperclipai/plugin-sdk";import { conflict, notFound, unprocessable } from "../errors.js";@@ -167,7 +169,7 @@ function normalizeConnectionSummary(function normalizeProviderMetadata(metadata: Record<string, unknown> | null | undefined): Record<string, unknown> | null {if (!metadata) return null;- return redactEnvironmentCustomImageValue(metadata);+ return redactEnvironmentCustomImageValue(withoutCleanupVerification(metadata) ?? {});}function metadataRecord(metadata: Record<string, unknown> | null | undefined): Record<string, unknown> {@@ -647,7 +649,7 @@ export function environmentCustomImageService(return await markSessionStatus({sessionId: session.id,status,- metadata: cancelled.metadata,+ metadata: mergeSetupSessionMetadata(session.metadata, cancelled.metadata),failureReason: status === "failed" ? "Provider failed to cancel setup session." : null,});} catch (error) {@@ -760,6 +762,22 @@ export function environmentCustomImageService(getSessionById,+ async inspectSetupResource(session: typeof environmentCustomImageSetupSessions.$inferSelect): Promise<PluginEnvironmentLeaseObservation | undefined> {+ if (!session.providerLeaseId || !session.finishedAt || session.connectionSecretRef !== null ||+ !["promoted", "cancelled", "timed_out", "failed"].includes(session.status)) return undefined;+ const environment = await requireEnvironment(session.environmentId);+ const provider = await resolveSetupProvider({ environment, storedProvider: session.provider,+ storedRpcCompanyId: readSetupRpcCompanyId(session.metadata) });+ const recordedFingerprint = readString(session.metadata?.[SOURCE_ENVIRONMENT_CONFIG_FINGERPRINT_METADATA_KEY]);+ if (!recordedFingerprint || fingerprintEnvironmentSandboxProviderConfig(provider.runtimeConfig) !== recordedFingerprint) return undefined;+ if (!options.pluginWorkerManager?.getWorker(provider.pluginId)?.supportedMethods?.includes("environmentInspectLease")) return undefined;+ return await options.pluginWorkerManager.call(provider.pluginId, "environmentInspectLease", {+ driverKey: provider.provider, companyId: provider.rpcCompanyId, environmentId: environment.id,+ issueId: null, config: provider.driverConfig, providerLeaseId: session.providerLeaseId,+ leaseMetadata: session.metadata ?? {},+ }, 15_000);+ },+startSetupSession: async (input: {environmentId: string;templateId?: string | null;diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.tsindex c488b4c591..a95d58cec6 100644--- a/server/src/services/environment-runtime.ts+++ b/server/src/services/environment-runtime.ts@@ -1,6 +1,7 @@import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";import { beginIdleTrackedWork } from "./task-admission.js";+import { historicalProviderCredentials } from "./historical-provider-credentials.js";import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";@@ -32,6 +33,7 @@ import type {PluginEnvironmentAcquireLeaseParams,PluginEnvironmentExecuteResult,PluginEnvironmentLease,+ PluginEnvironmentLeaseObservation,PluginEnvironmentRealizeWorkspaceResult,PluginEnvironmentSyncResult,PluginSyncOperation,@@ -592,6 +594,7 @@ export interface EnvironmentDriverRunnerIngressInputexport interface EnvironmentRuntimeDriver {readonly driver: string;+ inspectLeaseResource?(input: { environment: Environment | null; lease: EnvironmentLease }): Promise<PluginEnvironmentLeaseObservation | undefined>;acquireRunLease(input: EnvironmentDriverAcquireInput): Promise<EnvironmentLease>;releaseRunLease(input: EnvironmentDriverReleaseInput): Promise<EnvironmentLease | null>;resumeRunLease?(input: EnvironmentDriverLeaseInput): Promise<PluginEnvironmentLease | EnvironmentLease | null>;@@ -2610,6 +2613,28 @@ function createSandboxEnvironmentDriver(});},+ async inspectLeaseResource({ lease }) {+ const provider = lease.provider;+ const config = sandboxConfigFromLeaseMetadataLoose(lease);+ if (!provider || isBuiltinSandboxProvider(provider) || !config || config.provider !== provider ||+ !lease.providerLeaseId || !lease.environmentId || !pluginWorkerManager) return undefined;+ const pluginId = readString(lease.metadata?.pluginId);+ // Historical observations must use the recorded plugin and credentials.+ // A different installation cannot establish that this allocation is gone.+ if (!pluginId) return undefined;+ const resolved = await resolveSandboxProviderPlugin({ provider, pluginId });+ if (resolved.state !== "running" || !pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentInspectLease")) return undefined;+ const resolvedConfig = await historicalProviderCredentials(db, lease.companyId, config,+ resolved.resolved.driver.configSchema as Record<string, unknown> | undefined,+ { issueId: lease.issueId, heartbeatRunId: lease.heartbeatRunId });+ if (!resolvedConfig) return undefined;+ return await pluginWorkerManager.call(pluginId, "environmentInspectLease", {+ driverKey: provider, companyId: lease.companyId, environmentId: lease.environmentId,+ issueId: lease.issueId, config: stripSandboxProviderEnvelope(resolvedConfig as SandboxEnvironmentConfig),+ providerLeaseId: lease.providerLeaseId, leaseMetadata: lease.metadata ?? {},+ }, 15_000);+ },+async retryPendingSandboxTeardown(input) {const exportResume = hasStopOnlyCleanup(input.lease);if (!exportResume && await nativeWorkspaceExportPendingForLease(db, input.lease)) {@@ -4235,6 +4260,13 @@ export function environmentRuntimeService(// record, so a deleted or foreign-bound environment never strands it. The// caller owns the lease release; this dispatcher only runs the provider// teardown and throws when the driver teardown throws.+ async inspectLeaseResource(leaseId: string): Promise<PluginEnvironmentLeaseObservation | undefined> {+ const lease = await environmentsSvc.getLeaseById(leaseId);+ if (!lease || lease.status !== "failed" || lease.leasePolicy !== "ephemeral" || !lease.providerLeaseId) return undefined;+ const environment = lease.environmentId ? await environmentsSvc.getById(lease.environmentId) : null;+ return await getDriver(getLeaseDriverKey(lease, environment))?.inspectLeaseResource?.({ environment, lease });+ },+async retryPendingSandboxTeardown(input: {environment: Environment | null;lease: EnvironmentLease;diff --git a/server/src/services/environments.ts b/server/src/services/environments.tsindex b8922aa61d..0ac6c69fcb 100644--- a/server/src/services/environments.ts+++ b/server/src/services/environments.ts@@ -1,3 +1,4 @@+import { withoutCleanupVerification } from "./cleanup-verification-proof.js";import { randomUUID } from "node:crypto";import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "./environment-custom-image-runtime.js";@@ -1463,7 +1464,7 @@ export function environmentService(db: Db) {releasedAt: null,failureReason: null,cleanupStatus: null,- metadata: input.metadata ?? null,+ metadata: withoutCleanupVerification(input.metadata),createdAt: now,updatedAt: now,};@@ -1584,7 +1585,7 @@ export function environmentService(db: Db) {expiresAt: input.expiresAt ?? null,failureReason: null,cleanupStatus: null,- metadata: input.metadata ?? null,+ metadata: withoutCleanupVerification(input.metadata),updatedAt: now,}).where(@@ -1739,7 +1740,7 @@ export function environmentService(db: Db) {releasedAt: now,failureReason: input.failureReason,cleanupStatus: "failed",- metadata: input.metadata ?? null,+ metadata: withoutCleanupVerification(input.metadata),createdAt: now,updatedAt: now,})@@ -1759,7 +1760,7 @@ export function environmentService(db: Db) {const row = await db.update(environmentLeases).set({- metadata,+ metadata: withoutCleanupVerification(metadata),lastUsedAt: new Date(),updatedAt: new Date(),})diff --git a/server/src/services/historical-cleanup-verification.ts b/server/src/services/historical-cleanup-verification.tsnew file mode 100644index 0000000000..405c2ae69f--- /dev/null+++ b/server/src/services/historical-cleanup-verification.ts@@ -0,0 +1,150 @@+import { and, eq, inArray, sql } from "drizzle-orm";+import { environmentCustomImageSetupSessions, environmentLeases, type Db } from "@paperclipai/db";+import { beginIdleTrackedWork } from "./task-admission.js";+import { finishedIdleRun } from "./idle-work-proof.js";+import { CLEANUP_ROW_STAMP_FORMAT, CLEANUP_VERIFICATION_KEY, CLEANUP_VERIFICATION_SCHEMA, cleanupRowStamp, verifiedResourceCleanup } from "./cleanup-verification-proof.js";++type Lease = typeof environmentLeases.$inferSelect;+type Setup = typeof environmentCustomImageSetupSessions.$inferSelect;+type Inspectors = {+ inspectLease(lease: Lease): Promise<unknown>;+ inspectSetup(setup: Setup): Promise<unknown>;+};+type Result = { checked: number; verified: number; deferred: number; failed: number };++/** Read provider state only. Preserve failures and retained data; observing an+ * allocation never starts or destroys it. Older workers simply defer proof. */+export function historicalCleanupVerificationService(db: Db, inspectors: Inspectors) {+ let inFlight: Promise<Result> | undefined;+ let leaseCursor = "";+ let setupCursor = "";+ async function run(): Promise<Result> {+ const finish = beginIdleTrackedWork();+ const result: Result = { checked: 0, verified: 0, deferred: 0, failed: 0 };+ try {+ const leases = await db.select({ row: environmentLeases, stamp: sql<string>`${sql.raw(cleanupRowStamp('environment_leases'))}` })+ .from(environmentLeases).where(and(+ eq(environmentLeases.status, "failed"), eq(environmentLeases.leasePolicy, "ephemeral"),+ eq(environmentLeases.cleanupStatus, "success"),+ sql`${environmentLeases.id}::text > ${leaseCursor}`,+ sql`${environmentLeases.releasedAt} IS NOT NULL AND ${environmentLeases.environmentId} IS NOT NULL`,+ sql`${environmentLeases.provider} IS NOT NULL AND ${environmentLeases.provider} <> 'local'+ AND ${environmentLeases.providerLeaseId} IS NOT NULL`,+ sql`COALESCE(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'false'+ AND NOT (COALESCE(${environmentLeases.metadata}, '{}'::jsonb) ?| ARRAY['nativeWorkspaceExportResume', 'sandboxStopAndRetain'])`,+ sql`(${sql.raw(verifiedResourceCleanup('environment_leases', 'lease'))}) IS NOT TRUE`,+ sql`(${environmentLeases.heartbeatRunId} IS NULL OR EXISTS (SELECT 1 FROM heartbeat_runs historical_run+ WHERE historical_run.id = ${environmentLeases.heartbeatRunId} AND historical_run.company_id = ${environmentLeases.companyId}+ AND ${sql.raw(finishedIdleRun('historical_run'))}))`,+ sql`NOT EXISTS (SELECT 1 FROM environment_leases other_owner+ WHERE other_owner.id <> ${environmentLeases.id} AND other_owner.provider = ${environmentLeases.provider}+ AND other_owner.provider_lease_id = ${environmentLeases.providerLeaseId}+ AND other_owner.status IN ('active', 'retained', 'pending_cleanup'))`,+ sql`NOT EXISTS (SELECT 1 FROM environment_custom_image_setup_sessions setup_owner+ WHERE setup_owner.provider = ${environmentLeases.provider} AND setup_owner.provider_lease_id = ${environmentLeases.providerLeaseId}+ AND (setup_owner.finished_at IS NULL OR setup_owner.connection_secret_ref IS NOT NULL))`,+ )).orderBy(environmentLeases.id).limit(16);+ const setups = await db.select({ row: environmentCustomImageSetupSessions,+ stamp: sql<string>`${sql.raw(cleanupRowStamp('environment_custom_image_setup_sessions'))}` })+ .from(environmentCustomImageSetupSessions).where(and(+ inArray(environmentCustomImageSetupSessions.status, ["promoted", "cancelled", "timed_out", "failed"]),+ sql`${environmentCustomImageSetupSessions.id}::text > ${setupCursor}`,+ sql`${environmentCustomImageSetupSessions.finishedAt} IS NOT NULL+ AND ${environmentCustomImageSetupSessions.connectionSecretRef} IS NULL+ AND ${environmentCustomImageSetupSessions.providerLeaseId} IS NOT NULL`,+ sql`(${sql.raw(verifiedResourceCleanup('environment_custom_image_setup_sessions', 'image_setup'))}) IS NOT TRUE`,+ sql`NOT EXISTS (SELECT 1 FROM environment_leases lease_owner+ WHERE lease_owner.provider = ${environmentCustomImageSetupSessions.provider}+ AND lease_owner.provider_lease_id = ${environmentCustomImageSetupSessions.providerLeaseId}+ AND lease_owner.status NOT IN ('released', 'expired', 'failed'))`,+ )).orderBy(environmentCustomImageSetupSessions.id).limit(16);+ leaseCursor = leases.length === 16 ? leases.at(-1)!.row.id : "";+ setupCursor = setups.length === 16 ? setups.at(-1)!.row.id : "";+ // Keep provider concurrency bounded. Each operation is an observation;+ // every accepted receipt must also win the row-version comparison below.+ const jobs = [+ ...leases.map(snapshot => async () => inspect("lease", snapshot.row, snapshot.stamp)),+ ...setups.map(snapshot => async () => inspect("image_setup", snapshot.row, snapshot.stamp)),+ ];+ let next = 0;+ await Promise.all(Array.from({ length: Math.min(2, jobs.length) }, async () => {+ for (;;) { const index = next++; if (index >= jobs.length) return; await jobs[index]!(); }+ }));+ return result;+ } finally { finish(); }++ async function inspect(kind: "lease" | "image_setup", row: Lease | Setup, stamp: string) {+ result.checked += 1;+ try {+ const observation = await (kind === "lease" ? inspectors.inspectLease(row as Lease) : inspectors.inspectSetup(row as Setup)) as {+ providerLeaseId?: unknown; state?: unknown;+ } | null | undefined;+ if (observation?.state !== "destroyed" || !row.providerLeaseId || observation.providerLeaseId !== row.providerLeaseId) {+ result.deferred += 1; return;+ }+ const now = new Date();+ const companyId = kind === "lease" ? (row as Lease).companyId+ : typeof row.metadata?.setupRpcCompanyId === "string" && row.metadata.setupRpcCompanyId.trim()+ ? row.metadata.setupRpcCompanyId.trim() : "instance";+ const proof = { schema: CLEANUP_VERIFICATION_SCHEMA, kind, resourceId: row.id, companyId,+ environmentId: row.environmentId, provider: row.provider, providerLeaseId: row.providerLeaseId,+ heartbeatRunId: kind === "lease" ? (row as Lease).heartbeatRunId : null,+ state: "destroyed", confirmedAt: now.toISOString() };+ const metadata = sql`COALESCE(${kind === "lease" ? environmentLeases.metadata : environmentCustomImageSetupSessions.metadata}, '{}'::jsonb)+ || jsonb_build_object(${CLEANUP_VERIFICATION_KEY}::text, ${JSON.stringify(proof)}::jsonb+ || jsonb_build_object('rowUpdatedAt', to_char(${now.toISOString()}::timestamptz AT TIME ZONE 'UTC', ${CLEANUP_ROW_STAMP_FORMAT})))`;+ let updated: Array<{ id: string }>;+ if (kind === "lease") {+ const lease = row as Lease;+ updated = await db.update(environmentLeases).set({ metadata, updatedAt: now }).where(and(+ eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),+ eq(environmentLeases.status, "failed"), eq(environmentLeases.leasePolicy, "ephemeral"),+ eq(environmentLeases.cleanupStatus, "success"),+ sql`${environmentLeases.provider} IS NOT DISTINCT FROM ${lease.provider}+ AND ${environmentLeases.providerLeaseId} IS NOT DISTINCT FROM ${lease.providerLeaseId}+ AND ${environmentLeases.environmentId} IS NOT DISTINCT FROM ${lease.environmentId}::uuid+ AND ${environmentLeases.heartbeatRunId} IS NOT DISTINCT FROM ${lease.heartbeatRunId}::uuid+ AND ${environmentLeases.releasedAt} IS NOT NULL`,+ sql`(${environmentLeases.heartbeatRunId} IS NULL OR EXISTS (SELECT 1 FROM heartbeat_runs verify_run+ WHERE verify_run.id = ${environmentLeases.heartbeatRunId} AND verify_run.company_id = ${environmentLeases.companyId}+ AND ${sql.raw(finishedIdleRun('verify_run'))}))`,+ sql`${sql.raw(cleanupRowStamp('environment_leases'))} = ${stamp}+ AND ${environmentLeases.metadata} IS NOT DISTINCT FROM ${(lease.metadata === null ? null : JSON.stringify(lease.metadata))}::jsonb`,+ sql`NOT EXISTS (SELECT 1 FROM environment_leases other_owner+ WHERE other_owner.id <> ${environmentLeases.id} AND other_owner.provider = ${environmentLeases.provider}+ AND other_owner.provider_lease_id = ${environmentLeases.providerLeaseId}+ AND other_owner.status IN ('active', 'retained', 'pending_cleanup'))`,+ sql`NOT EXISTS (SELECT 1 FROM environment_custom_image_setup_sessions setup_owner+ WHERE setup_owner.provider = ${environmentLeases.provider} AND setup_owner.provider_lease_id = ${environmentLeases.providerLeaseId}+ AND (setup_owner.finished_at IS NULL OR setup_owner.connection_secret_ref IS NOT NULL))`,+ )).returning({ id: environmentLeases.id });+ } else {+ const setup = row as Setup;+ updated = await db.update(environmentCustomImageSetupSessions).set({ metadata, updatedAt: now }).where(and(+ eq(environmentCustomImageSetupSessions.id, setup.id), eq(environmentCustomImageSetupSessions.status, setup.status),+ sql`${environmentCustomImageSetupSessions.provider} IS NOT DISTINCT FROM ${setup.provider}+ AND ${environmentCustomImageSetupSessions.providerLeaseId} IS NOT DISTINCT FROM ${setup.providerLeaseId}+ AND ${environmentCustomImageSetupSessions.environmentId} = ${setup.environmentId}::uuid+ AND ${environmentCustomImageSetupSessions.finishedAt} IS NOT NULL`,+ sql`NOT EXISTS (SELECT 1 FROM environment_leases lease_owner+ WHERE lease_owner.provider = ${environmentCustomImageSetupSessions.provider}+ AND lease_owner.provider_lease_id = ${environmentCustomImageSetupSessions.providerLeaseId}+ AND lease_owner.status NOT IN ('released', 'expired', 'failed'))`,+ sql`${sql.raw(cleanupRowStamp('environment_custom_image_setup_sessions'))} = ${stamp}+ AND ${environmentCustomImageSetupSessions.metadata} IS NOT DISTINCT FROM ${(setup.metadata === null ? null : JSON.stringify(setup.metadata))}::jsonb+ AND ${environmentCustomImageSetupSessions.connectionSecretRef} IS NULL`,+ )).returning({ id: environmentCustomImageSetupSessions.id });+ }+ if (updated.length === 1) result.verified += 1;+ else result.deferred += 1;+ } catch {+ // Provider exceptions may contain credentials. Keep only a count.+ result.failed += 1;+ }+ }+ }+ return { sweep(): Promise<Result> {+ if (!inFlight) inFlight = run().finally(() => { inFlight = undefined; });+ return inFlight;+ } };+}diff --git a/server/src/services/historical-provider-credentials.ts b/server/src/services/historical-provider-credentials.tsnew file mode 100644index 0000000000..d36fe4009b--- /dev/null+++ b/server/src/services/historical-provider-credentials.ts@@ -0,0 +1,40 @@+import { createHash } from "node:crypto";+import { and, eq, sql } from "drizzle-orm";+import { companySecrets, companySecretVersions, type Db } from "@paperclipai/db";+import type { SandboxEnvironmentConfig } from "@paperclipai/shared";+import { collectSecretRefPaths, isUuidSecretRef, parseSecretRefBindingObject, readConfigValueAtPath, writeConfigValueAtPath } from "./json-schema-secret-refs.js";+import { secretService } from "./secrets.js";++/** A 404 under replacement credentials says nothing about the old resource.+ * Legacy leases did not record a secret-version manifest. Only an unrotated+ * initial version whose resolved bytes still match its immutable hash can+ * establish their credential identity. Other histories need explicit evidence.+ * Normal cleanup resolution and run-continuation rules remain unchanged. */+export async function historicalProviderCredentials(db: Db, companyId: string,+ config: SandboxEnvironmentConfig, schema: Record<string, unknown> | null | undefined,+ context: { issueId: string | null; heartbeatRunId: string | null },+): Promise<SandboxEnvironmentConfig | undefined> {+ let resolved = { ...config } as Record<string, unknown>;+ for (const path of collectSecretRefPaths(schema)) {+ const raw = readConfigValueAtPath(resolved, path);+ const binding = parseSecretRefBindingObject(raw);+ if (binding && binding.version !== "latest" && binding.version !== 1) return undefined;+ const ref = binding?.secretId ?? (typeof raw === "string" ? raw.trim() : null);+ if (!ref || !isUuidSecretRef(ref)) continue;+ const [version] = await db.select({ hash: companySecretVersions.valueSha256 })+ .from(companySecrets).innerJoin(companySecretVersions, and(+ eq(companySecretVersions.secretId, companySecrets.id), eq(companySecretVersions.version, 1)))+ .where(and(eq(companySecrets.id, ref), eq(companySecrets.companyId, companyId),+ eq(companySecrets.latestVersion, 1), eq(companySecrets.status, "active"),+ sql`${companySecretVersions.revokedAt} IS NULL`,+ sql`NOT EXISTS (SELECT 1 FROM company_secret_versions other_version+ WHERE other_version.secret_id = ${companySecrets.id} AND other_version.version <> 1)`));+ if (!version) return undefined;+ const value = await secretService(db).resolveSecretValueForSandboxCleanup(companyId, ref, 1, { ...context, configPath: path });+ if (createHash("sha256").update(value).digest("hex") !== version.hash) return undefined;+ resolved = writeConfigValueAtPath(resolved, path, value);+ }+ if (resolved.provider === "daytona" && (typeof resolved.apiKey !== "string" ||+ !resolved.apiKey.trim() || isUuidSecretRef(resolved.apiKey.trim()))) return undefined;+ return resolved as SandboxEnvironmentConfig;+}diff --git a/server/src/services/idle-issue-schedule.ts b/server/src/services/idle-issue-schedule.tsindex 431ea72f2f..e3944d75a0 100644--- a/server/src/services/idle-issue-schedule.ts+++ b/server/src/services/idle-issue-schedule.ts@@ -4,6 +4,7 @@ import type { Db } from "@paperclipai/db";import { AGENT_STATUSES, issueExecutionMonitorPolicySchema, PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared";import type { ScheduledWake } from "./idle-scheduled-work.js";import { onboardingFirstTaskAwaitingUserSql } from "./recovery/onboarding-first-task-wait.js";+import { responseDrivenIssueWait } from "./idle-work-proof.js";const processGeneration = randomUUID();const limit = 10_000;@@ -15,6 +16,7 @@ const record = (value: unknown): Record<string, unknown> | null =>export async function readIdleIssueSchedule(tx: Pick<Db, "execute">, ownerId: string, now: number): Promise<{blocked: boolean;scheduledWake?: ScheduledWake;+ generation?: string;}> {const rows = await tx.execute<{id: string; company_id: string; status: string; updated_at: Date | string;@@ -25,13 +27,18 @@ export async function readIdleIssueSchedule(tx: Pick<Db, "execute">, ownerId: stmonitor_wake_requested_at: Date | string | null; monitor_attempt_count: number;company_status: string; company_updated: Date | string; agent_status: string | null;agent_updated: Date | string | null; scoped: boolean; unsafe_dependencies: boolean; dependency_ready: boolean;- untouched_onboarding: boolean;+ untouched_onboarding: boolean; response_driven_wait: boolean;+ response_wait_generation: unknown;}>(sql`SELECT i.id, i.company_id, i.status, i.updated_at, i.assignee_agent_id, i.assignee_user_id,i.checkout_run_id, i.execution_run_id, i.execution_policy, i.execution_state, i.unblock_descriptor,i.conversation_agent_id, i.monitor_next_check_at, i.monitor_wake_requested_at, i.monitor_attempt_count,c.status AS company_status, c.updated_at AS company_updated,a.status AS agent_status, a.updated_at AS agent_updated,(i.assignee_agent_id IS NULL OR a.id IS NOT NULL) AS scoped,+ (${sql.raw(responseDrivenIssueWait('i'))}) AS response_driven_wait,+ (SELECT jsonb_agg(jsonb_build_array(w.id, w.updated_at, w.kind, w.status, w.source_run_id,+ w.effective_resolver_policy, w.continuation_policy, w.addressee_agent_id, w.addressee_user_id) ORDER BY w.id)+ FROM issue_thread_interactions w WHERE w.issue_id = i.id AND w.status = 'pending') AS response_wait_generation,(${onboardingFirstTaskAwaitingUserSql(sql`i.company_id`, sql`i.id`, sql`i.origin_kind`)}AND NOT EXISTS (SELECT 1 FROM heartbeat_runs h WHERE h.company_id = i.company_idAND (h.context_snapshot ->> 'issueId' = i.id::text OR h.context_snapshot ->> 'taskId' = i.id::text))) AS untouched_onboarding,@@ -76,17 +83,22 @@ export async function readIdleIssueSchedule(tx: Pick<Db, "execute">, ownerId: st|| (row.status === "todo" && !row.assignee_agent_id)|| (row.status === "todo" && row.assignee_agent_id && row.agent_status === "idle" && row.untouched_onboarding === true)|| (row.status === "blocked" && (row.unblock_descriptor === null || unblock?.owner === "board"))+ || (["in_progress", "in_review"].includes(row.status) && row.response_driven_wait === true)|| Boolean(row.assignee_user_id && !row.assignee_agent_id);if (!passive) return { blocked: true };}- if (!Number.isFinite(earliest)) return { blocked: false };+ const generation = createHash("sha256").update(JSON.stringify([processGeneration, ownerId, rows])).digest("hex");+ if (!Number.isFinite(earliest)) return { blocked: false, generation };return { blocked: false, scheduledWake: { version: 1, kind: "routines_and_issue_monitors",- generation: createHash("sha256").update(JSON.stringify([processGeneration, ownerId, rows])).digest("hex"),+ generation,nextWakeAt: new Date(earliest).toISOString() } };}-export function combineIdleSchedules(routine?: ScheduledWake, issue?: ScheduledWake): ScheduledWake | undefined {- if (!issue) return routine;+export function combineIdleSchedules(routine?: ScheduledWake, issue?: ScheduledWake, issueGeneration?: string): ScheduledWake | undefined {+ // A passive issue can change even when only a routine owns the next alarm.+ // Bind that waiting-state proof into the routine generation as well.+ if (!issue) return routine && issueGeneration ? { ...routine,+ generation: createHash("sha256").update(JSON.stringify([routine, issueGeneration])).digest("hex") } : routine;if (!routine) return issue;return { version: 1, kind: "routines_and_issue_monitors",generation: createHash("sha256").update(JSON.stringify([routine, issue])).digest("hex"),diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.tsindex c5dfb63049..579cbf39da 100644--- a/server/src/services/idle-sleep-safety.ts+++ b/server/src/services/idle-sleep-safety.ts@@ -2,8 +2,10 @@ import { idleWorkSnapshot } from "./task-admission.js";import { readIdleLocalWork, type IdleLocalWork } from "./idle-local-work.js";import { readIdleRoutineSchedule, type ScheduledWake } from "./idle-scheduled-work.js";import { combineIdleSchedules, readIdleIssueSchedule } from "./idle-issue-schedule.js";+import { connectionDeliveryWaitingForResponse, finishedIdleRun as finishedRun, finishedIdleWake, passiveBoardRecovery } from "./idle-work-proof.js";import type { Db } from "@paperclipai/db";import { sql } from "drizzle-orm";+import { verifiedResourceCleanup } from "./cleanup-verification-proof.js";export interface IdleSleepDrainStatus {ownerId?: string;@@ -25,16 +27,6 @@ export type InspectIdlePlugins = (hold: { ownerId: string; expiresAt: number })pluginIds: string[];}>;-// Retry promotion keeps its due date as history on the run. Only a terminal-// run finished at or after that date has consumed the schedule. Newer retry-// dates, unfinished runs and accounting work still require the instance.-function finishedRun(alias: string): string {- return `${alias}.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')- AND ${alias}.finished_at IS NOT NULL- AND (${alias}.scheduled_retry_at IS NULL OR ${alias}.scheduled_retry_at <= ${alias}.finished_at)- AND ${alias}.cost_accounting_pending = false`;-}-// A failed turn may have successfully torn down its ephemeral allocation. A// cleanup label alone is not provider termination proof. Require the exact// host-owned receipt and finished run, and retain all cleanup/export intents.@@ -42,21 +34,28 @@ function finishedRun(alias: string): string {function finishedFailedLease(alias: string): string {const receipt = `${alias}.metadata->'remoteExecutionTermination'`;return `${alias}.status = 'failed' AND ${alias}.lease_policy = 'ephemeral'- AND ${alias}.cleanup_status = 'success' AND ${alias}.released_at IS NOT NULL+ AND ${alias}.released_at IS NOT NULLAND COALESCE(${alias}.metadata->>'pendingCleanupInFlight', 'false') = 'false'AND NOT (COALESCE(${alias}.metadata, '{}'::jsonb) ?| ARRAY['nativeWorkspaceExportResume', 'sandboxStopAndRetain'])- AND ${alias}.provider IS NOT NULL AND ${alias}.provider <> 'local'- AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> ''- AND ${receipt}->>'schema' = 'paperclip.remote-termination.v1'- AND ${receipt}->>'companyId' = ${alias}.company_id::text- AND ${receipt}->>'runId' = ${alias}.heartbeat_run_id::text- AND ${receipt}->>'leaseId' = ${alias}.id::text- AND ${receipt}->>'provider' = ${alias}.provider- AND ${receipt}->>'providerLeaseId' = ${alias}.provider_lease_id- AND ${receipt}->>'state' IN ('stopped', 'destroyed')- AND EXISTS (SELECT 1 FROM heartbeat_runs r WHERE r.id = ${alias}.heartbeat_run_id+ AND (( ${alias}.provider = 'local' AND ${alias}.provider_lease_id IS NULL+ AND ${alias}.metadata->>'driver' = 'local'+ AND ${receipt} IS NULL+ AND EXISTS (SELECT 1 FROM environments local_environment+ WHERE local_environment.id = ${alias}.environment_id AND local_environment.driver = 'local'))+ OR (${alias}.cleanup_status = 'success' AND ${alias}.provider IS NOT NULL AND ${alias}.provider <> 'local'+ AND ${alias}.provider_lease_id IS NOT NULL AND ${alias}.provider_lease_id <> ''+ AND ((${receipt}->>'schema' = 'paperclip.remote-termination.v1'+ AND ${receipt}->>'companyId' = ${alias}.company_id::text+ AND ${receipt}->>'runId' = ${alias}.heartbeat_run_id::text+ AND ${receipt}->>'leaseId' = ${alias}.id::text+ AND ${receipt}->>'provider' = ${alias}.provider+ AND ${receipt}->>'providerLeaseId' = ${alias}.provider_lease_id+ AND ${receipt}->>'state' IN ('stopped', 'destroyed'))+ OR (${verifiedResourceCleanup(alias, 'lease')}))))+ AND ((${alias}.heartbeat_run_id IS NULL AND ${alias}.provider <> 'local'+ AND (${verifiedResourceCleanup(alias, 'lease')})) OR EXISTS (SELECT 1 FROM heartbeat_runs r WHERE r.id = ${alias}.heartbeat_run_idAND r.company_id = ${alias}.company_id- AND ${finishedRun("r")})`;+ AND ${finishedRun("r")}))`;}// These are deliberately conservative. Completed agent runs and workspace@@ -128,17 +127,45 @@ const WORK_CHECKS = [) IS NOT TRUE`,`SELECT 1 FROM environment_leases l WHERE status NOT IN ('released', 'expired')AND (${finishedFailedLease("l")}) IS NOT TRUE`,+ `SELECT 1 FROM environment_custom_image_setup_sessions s WHERE+ (s.status IN ('promoted', 'cancelled', 'timed_out', 'failed') AND s.finished_at IS NOT NULL+ AND s.connection_secret_ref IS NULL AND (${verifiedResourceCleanup('s', 'image_setup')})) IS NOT TRUE`,+ // An unconfigured Slack draft has no provider identity or inbound runtime.+ // Setup still wakes through normal browser admission. Retain every enabled,+ // previously active, partially connected, foreign, or unsupported endpoint.+ `SELECT 1 FROM chat_endpoints endpoint WHERE (endpoint.provider = 'slack'+ AND endpoint.status = 'draft' AND endpoint.setup->>'step' = 'provider_setup'+ AND (endpoint.setup - ARRAY['step', 'command']) = '{}'::jsonb+ AND (endpoint.setup->'command' IS NULL OR endpoint.setup->'command' = 'null'::jsonb+ OR jsonb_typeof(endpoint.setup->'command') = 'string')+ AND endpoint.activated_at IS NULL AND endpoint.provider_account_id IS NULL+ AND endpoint.bot_external_id IS NULL AND endpoint.bot_username IS NULL+ AND endpoint.last_event_at IS NULL AND endpoint.last_publication_at IS NULL+ AND endpoint.health_message IS NULL AND endpoint.last_error IS NULL+ AND NOT EXISTS (SELECT 1 FROM chat_slack_registrations registration WHERE registration.endpoint_id = endpoint.id)+ AND EXISTS (SELECT 1 FROM tool_connections draft_connection+ WHERE draft_connection.id = endpoint.connection_id AND draft_connection.company_id = endpoint.company_id+ AND draft_connection.connection_purpose = 'channel' AND draft_connection.transport = 'chat_sdk'+ AND draft_connection.status = 'draft' AND draft_connection.enabled = false)) IS NOT TRUE`,`SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`,`SELECT 1 FROM routine_runs WHERE status NOT IN ('completed', 'failed', 'skipped', 'cancelled')`,`SELECT 1 FROM plugin_jobs WHERE status <> 'paused'`,`SELECT 1 FROM plugin_job_runs WHERE status NOT IN ('succeeded', 'failed', 'cancelled')`,`SELECT 1 FROM plugin_webhook_deliveries WHERE status NOT IN ('succeeded', 'failed')`,- // Saved AI credentials have no independent worker. Login sessions, accepted- // quota/refresh work and recovery journals have their own checks. Other- // integrations still need an ingress/scheduler wake contract.- `SELECT 1 FROM tool_connections WHERE status <> 'archived'- AND NOT (status = 'active' AND connection_purpose = 'ai' AND transport = 'runtime_auth')`,- `SELECT 1 FROM issue_recovery_actions WHERE status NOT IN ('resolved', 'cancelled')`,+ // Saved outbound MCP configuration and disabled drafts have no independent+ // listener. Keep local processes, channels and GitHub webhook bindings awake;+ // accepted calls, credential recovery and delivery work have separate fences.+ `SELECT 1 FROM tool_connections c WHERE c.status <> 'archived' AND (+ (c.status = 'draft' AND c.enabled = false)+ OR (c.status = 'active' AND c.connection_purpose = 'ai' AND c.transport = 'runtime_auth')+ OR (c.status = 'active' AND c.connection_purpose = 'tool' AND c.transport = 'mcp_remote'+ AND NOT EXISTS (SELECT 1 FROM connection_grants g WHERE g.connection_id = c.id+ AND (g.company_id <> c.company_id OR g.provider_tenant->'github' IS NOT NULL)))+ ) IS NOT TRUE`,+ // An explicit board escalation has no automatic retry. Retain the visible+ // action and all execution/cleanup checks while the board decides what to do.+ `SELECT 1 FROM issue_recovery_actions a WHERE a.status NOT IN ('resolved', 'cancelled')+ AND (${passiveBoardRecovery('a')}) IS NOT TRUE`,`SELECT 1 FROM status_cards WHERE archived_at IS NULL`,`SELECT 1 FROM external_objects WHERE NOT is_terminal OR next_refresh_at IS NOT NULL`,// Login history keeps its provider reference after teardown. A terminal@@ -186,17 +213,30 @@ const WORK_CHECKS = [SELECT 1 FROM heartbeat_runs r WHERE r.id = w.run_idAND r.company_id = w.company_id AND r.agent_id = w.agent_idAND ${finishedRun("r")})))))`,+ // Cloud owns inbound event durability. A processed receipt is idempotency+ // history; any downstream run, wake or delivery is still checked separately.+ `SELECT 1 FROM connection_event_deliveries WHERE status <> 'processed' OR processed_at IS NULL`,+ // A delivered outbox row is history after its exact accepted wake finishes.+ // A system-retired connection card may also be parked behind a still-pending+ // response. Keep the outbox intact: the normal reply path resumes delivery.+ `SELECT 1 FROM connection_intent_deliveries d WHERE+ (d.delivered_at IS NULL AND (${connectionDeliveryWaitingForResponse('d')}) IS NOT TRUE)+ OR EXISTS (SELECT 1 FROM agent_wakeup_requests connection_wake+ WHERE connection_wake.company_id = d.company_id AND connection_wake.idempotency_key IN (+ 'connection-intent:' || d.interaction_id::text || ':accepted',+ 'connection-intent:' || d.interaction_id::text || ':rejected',+ 'connection-intent:' || d.interaction_id::text || ':expired')+ AND (${finishedIdleWake('connection_wake')}) IS NOT TRUE)`,// These less common work sources fail closed on any retained state. Their// terminal-state exceptions can be added with tests for the owning service....[- "chat_endpoints", "chat_deliveries", "chat_publications", "chat_actions",- "chat_completion_deliveries", "connection_event_deliveries",- "connection_intent_deliveries", "tool_action_requests", "tool_action_deliveries",+ "chat_deliveries", "chat_publications", "chat_actions",+ "chat_completion_deliveries", "tool_action_requests", "tool_action_deliveries","workspace_runtime_services","execution_workspace_runtime_leases", "pipeline_automation_executions","native_run_finalizations", "company_transfer_runs", "decisions", "decision_effect_executions","decision_archive_notification_outbox", "browser_use_sessions", "browser_use_runs",- "browser_use_browsers", "environment_custom_image_setup_sessions", "feedback_exports",+ "browser_use_browsers", "feedback_exports","company_secret_proposals","mcp_oauth_grants", "mcp_mutation_receipts", "mcp_event_subscriptions","mcp_event_deliveries", "mcp_attachment_uploads", "dot_agent_bindings",@@ -266,7 +306,7 @@ export async function readIdleSleepSafety(if (!scheduledRoutines || !ownerId || ownerId !== before.ownerId) return undefined;const schedule = await readIdleIssueSchedule(tx, ownerId, now());if (schedule.blocked) return true;- scheduledWake = combineIdleSchedules(scheduledWake, schedule.scheduledWake);+ scheduledWake = combineIdleSchedules(scheduledWake, schedule.scheduledWake, schedule.generation);}return false;}, { isolationLevel: "repeatable read", accessMode: "read only" });diff --git a/server/src/services/idle-work-proof.ts b/server/src/services/idle-work-proof.tsnew file mode 100644index 0000000000..38ef426696--- /dev/null+++ b/server/src/services/idle-work-proof.ts@@ -0,0 +1,96 @@+/** Fixed SQL aliases only. These predicates do not accept request input. */+export function finishedIdleRun(alias: string): string {+ return `${alias}.status IN ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')+ AND ${alias}.finished_at IS NOT NULL+ AND (${alias}.scheduled_retry_at IS NULL OR ${alias}.scheduled_retry_at <= ${alias}.finished_at)+ AND ${alias}.cost_accounting_pending = false`;+}++export function finishedIdleWake(alias: string): string {+ return `${alias}.finished_at IS NOT NULL AND (+ ${alias}.status IN ('completed', 'failed', 'cancelled', 'skipped', 'timed_out')+ OR (${alias}.status = 'coalesced' AND EXISTS (+ SELECT 1 FROM heartbeat_runs wake_target WHERE wake_target.id = ${alias}.run_id+ AND wake_target.company_id = ${alias}.company_id AND wake_target.agent_id = ${alias}.agent_id+ AND ${finishedIdleRun('wake_target')})))`;+}++export function connectionDeliveryWaitingForResponse(delivery: string): string {+ return `EXISTS (SELECT 1 FROM issue_thread_interactions waiting_delivery+ JOIN issues waiting_issue ON waiting_issue.id = waiting_delivery.issue_id+ AND waiting_issue.company_id = waiting_delivery.company_id+ WHERE waiting_delivery.id = ${delivery}.interaction_id AND waiting_delivery.company_id = ${delivery}.company_id+ AND waiting_delivery.kind = 'connection_intent' AND waiting_delivery.status = 'expired'+ AND waiting_delivery.resolved_at IS NOT NULL AND waiting_delivery.result->>'outcome' = 'expired'+ AND jsonb_typeof(waiting_delivery.result->'connectionId') = 'string'+ AND waiting_issue.status IN ('in_progress', 'in_review')+ AND waiting_issue.assignee_agent_id::text = waiting_delivery.payload->>'requestingAgentId'+ AND (${responseDrivenIssueWait('waiting_issue')}))`;+}++// Questions and confirmations advance on a response, not on a timer. Preserve+// the audience and continuation policy. A producer which still owns execution+// or accounting cannot certify a passive wait.+function passiveResponse(alias: string, issue: string): string {+ return `${alias}.company_id = ${issue}.company_id+ AND ${alias}.status = 'pending' AND ${alias}.resolved_at IS NULL+ AND (${alias}.kind IN ('ask_user_questions', 'request_confirmation', 'request_checkbox_confirmation', 'suggest_tasks')+ OR (${alias}.kind = 'connection_intent' AND ${alias}.effective_resolver_policy = 'human_only'+ AND ${alias}.addressee_user_id IS NOT NULL AND ${issue}.assignee_user_id IS NULL+ AND ${alias}.payload->>'version' = '1'+ AND ${alias}.payload->>'requestingAgentId' = ${issue}.assignee_agent_id::text+ AND length(${alias}.payload->>'serviceSlug') > 0+ AND (${alias}.payload->>'purpose' IS NULL OR ${alias}.payload->>'purpose' IN ('ai', 'channel'))+ -- Existing access can automatically retire a setup card. In its+ -- absence only the addressed human can finish this request. A broad+ -- purpose match deliberately keeps unfamiliar credential paths awake.+ AND NOT EXISTS (SELECT 1 FROM tool_connections response_connection+ WHERE response_connection.company_id = ${issue}.company_id+ AND response_connection.status = 'active' AND response_connection.enabled+ AND response_connection.connection_purpose = COALESCE(${alias}.payload->>'purpose', 'tool'))))+ -- Merge confirmations can advance through GitHub polling, without a+ -- response. Its recognizer requires merge intent in these four fields;+ -- conservatively retain those cards until polling has a durable wake.+ AND (${alias}.kind <> 'request_confirmation' OR concat_ws(' ', ${alias}.title, ${alias}.summary,+ ${alias}.payload->>'prompt', ${alias}.payload->>'acceptLabel') !~* 'merge')+ AND ${alias}.effective_resolver_policy IN ('anyone', 'not_creator', 'human_only')+ AND ${alias}.continuation_policy IN ('wake_assignee', 'none')+ AND (${alias}.source_run_id IS NULL OR EXISTS (+ SELECT 1 FROM heartbeat_runs response_source WHERE response_source.id = ${alias}.source_run_id+ AND response_source.company_id = ${issue}.company_id AND ${finishedIdleRun('response_source')}))`;+}++export function responseDrivenIssueWait(issue: string): string {+ return `EXISTS (SELECT 1 FROM issue_thread_interactions response_wait+ WHERE response_wait.issue_id = ${issue}.id AND ${passiveResponse('response_wait', issue)})+ AND NOT EXISTS (SELECT 1 FROM issue_thread_interactions other_wait+ WHERE other_wait.issue_id = ${issue}.id AND other_wait.status = 'pending'+ AND (${passiveResponse('other_wait', issue)}) IS NOT TRUE)`;+}++export function connectionIntentWaitingForResponse(interaction: string, issue: string): string {+ return `${interaction}.company_id = ${issue}.company_id AND ${interaction}.issue_id = ${issue}.id+ AND ${interaction}.kind = 'connection_intent' AND ${interaction}.status = 'pending'+ AND ${issue}.status IN ('in_progress', 'in_review') AND ${issue}.assignee_user_id IS NULL+ AND (${responseDrivenIssueWait(issue)})`;+}++export function passiveBoardRecovery(alias: string): string {+ return `${alias}.status IN ('active', 'escalated') AND ${alias}.resolved_at IS NULL+ AND ${alias}.owner_type = 'board' AND ${alias}.owner_agent_id IS NULL+ AND ${alias}.kind IN ('stranded_assigned_issue', 'missing_disposition')+ AND ${alias}.cause IN ('stranded_assigned_issue', 'successful_run_missing_state', 'issue_review_path_lost')+ AND ${alias}.recovery_issue_id IS NULL AND ${alias}.monitor_policy IS NULL AND ${alias}.timeout_at IS NULL+ AND jsonb_typeof(${alias}.wake_policy) = 'object'+ AND ${alias}.wake_policy->>'type' = 'board_escalation'+ AND (${alias}.wake_policy - ARRAY['type', 'reason', 'preservesSourceAssignee']) = '{}'::jsonb+ AND COALESCE(${alias}.wake_policy->>'preservesSourceAssignee', 'true') = 'true'+ AND EXISTS (SELECT 1 FROM issues recovery_source+ WHERE recovery_source.id = ${alias}.source_issue_id AND recovery_source.company_id = ${alias}.company_id+ AND recovery_source.status = 'blocked'+ AND recovery_source.checkout_run_id IS NULL AND recovery_source.execution_run_id IS NULL+ AND recovery_source.conversation_agent_id IS NULL+ AND recovery_source.monitor_next_check_at IS NULL AND recovery_source.monitor_wake_requested_at IS NULL+ AND (recovery_source.execution_state IS NULL OR recovery_source.execution_state->>'status' = 'idle')+ AND COALESCE(recovery_source.execution_policy->'monitor', 'null'::jsonb) = 'null'::jsonb)`;+}diff --git a/server/src/services/plugin-worker-manager.ts b/server/src/services/plugin-worker-manager.tsindex 5569f80c65..dd7173d9dd 100644--- a/server/src/services/plugin-worker-manager.ts+++ b/server/src/services/plugin-worker-manager.ts@@ -716,6 +716,7 @@ export interface PluginWorkerHandle {params: HostToWorkerMethods[M][0],timeoutMs?: number,executeLogSink?: ExecuteLogSink,+ onWorkerSettled?: () => void,): Promise<HostToWorkerMethods[M][1]>;/**@@ -900,6 +901,7 @@ export function createPluginWorkerHandle(// Unlike RPC promises, these survive timeouts. A late response is the// completion receipt. A worker crash cannot manufacture one.const unsettledCalls = new Map<string | number, () => void>();+ const unsettledInspections = new Set<() => void>();let activeHostHandlers = 0;let idleHold: { ownerId: string; expiresAt: number; ready: boolean } | null = null;@@ -3018,6 +3020,10 @@ export function createPluginWorkerHandle()),);+ // Read-only requests cannot continue in an exited worker. Do not infer+ // completion for execute/cleanup calls, whose provider work may outlive it.+ for (const finish of [...unsettledInspections]) finish();+// Close the one login pseudo-terminal route with a fixed non-secret exit and// clear the route one time. The pending pseudo-terminal calls// already rejected through `rejectAllPending`.@@ -3351,14 +3357,17 @@ export function createPluginWorkerHandle(params: HostToWorkerMethods[M][0],timeoutMs?: number,executeLogSink?: ExecuteLogSink,+ onWorkerSettled?: () => void,): Promise<HostToWorkerMethods[M][1]> {const rpcPromise = new Promise<HostToWorkerMethods[M][1]>((resolve, reject) => {const idleControl = method === "prepareIdleSleep" || method === "releaseIdleSleep";if (!idleControl && method !== "shutdown" && idleSleepHeld()) {+ onWorkerSettled?.();reject(new Error("Plugin is held for idle sleep"));return;}if (!childProcess?.stdin?.writable) {+ onWorkerSettled?.();reject(new Error(`Cannot call "${method}" — worker for "${pluginId}" is not running`,@@ -3368,7 +3377,20 @@ export function createPluginWorkerHandle(}const id = nextRequestId++;- if (!idleControl) unsettledCalls.set(id, beginIdleTrackedWork());+ if (!idleControl) {+ const finishIdle = beginIdleTrackedWork();+ let completed = false;+ const finish = () => {+ if (completed) return;+ completed = true;+ finishIdle();+ unsettledInspections.delete(finish);+ unsettledCalls.delete(id);+ onWorkerSettled?.();+ };+ unsettledCalls.set(id, finish);+ if (method === "environmentInspectLease") unsettledInspections.add(finish);+ }const timeout = resolveRpcCallTimeoutMs(timeoutMs, rpcTimeoutMs);const invocationScope = deriveInvocationScope(method, params);const invocation = invocationScope ? registerInvocation(invocationScope) : null;@@ -3524,15 +3546,17 @@ export function createPluginWorkerHandle(params: HostToWorkerMethods[M][0],timeoutMs?: number,executeLogSink?: ExecuteLogSink,+ onWorkerSettled?: () => void,): Promise<HostToWorkerMethods[M][1]> {if (status !== "running" && status !== "starting") {+ onWorkerSettled?.();return Promise.reject(new Error(`Cannot call "${method}" — worker for "${pluginId}" is ${status}`,),);}- return callInternal(method, params, timeoutMs, executeLogSink);+ return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled);},openLoginPtySession(input: LoginPtyOpenInput) {@@ -3729,6 +3753,7 @@ export function createPluginWorkerManager(const log = logger.child({ service: "plugin-worker-manager" });const workers = new Map<string, PluginWorkerHandle>();const configApplied = new Set<() => void>();+ let activeInspections = 0;/** Per-plugin startup locks to prevent concurrent spawn races. */const startupLocks = new Map<string, Promise<PluginWorkerHandle>>();// The one shared, process-scoped aggregate route-slot controller. The manager@@ -3879,7 +3904,16 @@ export function createPluginWorkerManager(new Error(`No worker registered for plugin "${pluginId}"`),);}- const result = handle.call(method, params, timeoutMs, executeLogSink);+ let finishInspection: (() => void) | undefined;+ if (method === "environmentInspectLease") {+ if (activeInspections >= 2) return Promise.reject(new Error("Provider inspection concurrency limit reached"));+ activeInspections += 1;+ let finished = false;+ finishInspection = () => { if (!finished) { finished = true; activeInspections -= 1; } };+ }+ // A caller timeout is not worker completion. Inspection capacity is+ // released only by its actual reply, send failure, or worker exit.+ const result = handle.call(method, params, timeoutMs, executeLogSink, finishInspection);if (method === "configChanged") void result.then(() => {for (const listener of configApplied) {try { listener(); }diff --git a/server/src/services/routines.ts b/server/src/services/routines.tsindex d92bae9560..825e79e6bf 100644--- a/server/src/services/routines.ts+++ b/server/src/services/routines.ts@@ -2065,6 +2065,95 @@ export function routineService(return run;}+ let statusReconcileCursor: string | null = null;++ async function syncRunStatusForIssue(issueId: string, executor: Db = db) {+ const issue = await executor+ .select({+ id: issues.id,+ companyId: issues.companyId,+ status: issues.status,+ originKind: issues.originKind,+ originRunId: issues.originRunId,+ })+ .from(issues)+ .where(eq(issues.id, issueId))+ .then((rows) => rows[0] ?? null);+ if (!issue || issue.originKind !== "routine_execution" || !issue.originRunId) return null;+ const run = await executor+ .select({+ id: routineRuns.id,+ status: routineRuns.status,+ failureReason: routineRuns.failureReason,+ triggerPayload: routineRuns.triggerPayload,+ })+ .from(routineRuns)+ .innerJoin(routines, and(eq(routines.id, routineRuns.routineId), eq(routines.companyId, issue.companyId)))+ .where(and(eq(routineRuns.id, issue.originRunId), eq(routineRuns.companyId, issue.companyId)))+ .then((rows) => rows[0] ?? null);+ if (!run) return null;+ if (issue.status === "done") {+ const transientFailureStatus = executionIssueTransientFailureStatusFromPayload(run.triggerPayload)+ ?? legacyExecutionIssueTransientFailureStatus(run.failureReason);+ const transientFailureClearedAt = executionIssueTransientFailureClearedAtFromPayload(run.triggerPayload);+ return finalizeRun(issue.originRunId, {+ status: "completed",+ failureReason: null,+ completedAt: new Date(),+ ...(transientFailureStatus+ ? {+ triggerPayload: {+ ...(run.triggerPayload ?? {}),+ transientFailure: {+ code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,+ status: transientFailureStatus,+ reason: executionIssueTransientFailureReason(transientFailureStatus),+ clearedAt: transientFailureClearedAt ?? new Date().toISOString(),+ },+ },+ }+ : {}),+ }, executor);+ }+ if (issue.status === "blocked" || issue.status === "cancelled") {+ const failureReason = executionIssueTransientFailureReason(issue.status);+ return finalizeRun(issue.originRunId, {+ status: "failed",+ failureReason,+ completedAt: new Date(),+ triggerPayload: {+ ...(run.triggerPayload ?? {}),+ transientFailure: {+ code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,+ status: issue.status,+ reason: failureReason,+ recordedAt: new Date().toISOString(),+ },+ },+ }, executor);+ }+ const transientFailureStatus = executionIssueTransientFailureStatusFromPayload(run.triggerPayload)+ ?? legacyExecutionIssueTransientFailureStatus(run.failureReason);+ if (run.status === "failed" && transientFailureStatus) {+ return finalizeRun(issue.originRunId, {+ status: "issue_created",+ failureReason: null,+ completedAt: null,+ triggerPayload: {+ ...(run.triggerPayload ?? {}),+ transientFailure: {+ code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,+ status: transientFailureStatus,+ reason: executionIssueTransientFailureReason(transientFailureStatus),+ clearedAt: new Date().toISOString(),+ },+ },+ }, executor);+ }+ return null;+ }++return {evaluateActivityGate,get: getRoutineById,@@ -3317,88 +3406,56 @@ export function routineService(return { triggered };},- syncRunStatusForIssue: async (issueId: string) => {- const issue = await db- .select({- id: issues.id,- status: issues.status,- originKind: issues.originKind,- originRunId: issues.originRunId,- })- .from(issues)- .where(eq(issues.id, issueId))- .then((rows) => rows[0] ?? null);- if (!issue || issue.originKind !== "routine_execution" || !issue.originRunId) return null;- const run = await db- .select({- id: routineRuns.id,- status: routineRuns.status,- failureReason: routineRuns.failureReason,- triggerPayload: routineRuns.triggerPayload,- })- .from(routineRuns)- .where(eq(routineRuns.id, issue.originRunId))- .then((rows) => rows[0] ?? null);- if (!run) return null;- if (issue.status === "done") {- const transientFailureStatus = executionIssueTransientFailureStatusFromPayload(run.triggerPayload)- ?? legacyExecutionIssueTransientFailureStatus(run.failureReason);- const transientFailureClearedAt = executionIssueTransientFailureClearedAtFromPayload(run.triggerPayload);- return finalizeRun(issue.originRunId, {- status: "completed",- failureReason: null,- completedAt: new Date(),- ...(transientFailureStatus- ? {- triggerPayload: {- ...(run.triggerPayload ?? {}),- transientFailure: {- code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,- status: transientFailureStatus,- reason: executionIssueTransientFailureReason(transientFailureStatus),- clearedAt: transientFailureClearedAt ?? new Date().toISOString(),- },- },- }- : {}),- });- }- if (issue.status === "blocked" || issue.status === "cancelled") {- const failureReason = executionIssueTransientFailureReason(issue.status);- return finalizeRun(issue.originRunId, {- status: "failed",- failureReason,- completedAt: new Date(),- triggerPayload: {- ...(run.triggerPayload ?? {}),- transientFailure: {- code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,- status: issue.status,- reason: failureReason,- recordedAt: new Date().toISOString(),- },- },- });- }- const transientFailureStatus = executionIssueTransientFailureStatusFromPayload(run.triggerPayload)- ?? legacyExecutionIssueTransientFailureStatus(run.failureReason);- if (run.status === "failed" && transientFailureStatus) {- return finalizeRun(issue.originRunId, {- status: "issue_created",- failureReason: null,- completedAt: null,- triggerPayload: {- ...(run.triggerPayload ?? {}),- transientFailure: {- code: EXECUTION_ISSUE_TRANSIENT_FAILURE_CODE,- status: transientFailureStatus,- reason: executionIssueTransientFailureReason(transientFailureStatus),- clearedAt: new Date().toISOString(),- },- },- });+ // Repair a missed issue-status hook through the same status transition as+ // normal issue writes. Never discard queued input or infer run completion+ // from age. Lock both owners and re-read the issue before applying repair.+ reconcileExecutionRunStatuses: async (): Promise<{ scanned: number; settled: number; deferred: number }> => {+ const candidates = await db.select({ id: routineRuns.id, companyId: routineRuns.companyId,+ issueId: issues.id, routineId: routineRuns.routineId })+ .from(routineRuns).innerJoin(routines, and(eq(routines.id, routineRuns.routineId), eq(routines.companyId, routineRuns.companyId)))+ .innerJoin(issues, and(eq(issues.id, routineRuns.linkedIssueId),+ eq(issues.companyId, routineRuns.companyId), eq(issues.originKind, "routine_execution"),+ sql`${issues.originRunId} = ${routineRuns.id}::text`, sql`${issues.originId} = ${routineRuns.routineId}::text`))+ .where(and(eq(routineRuns.status, "issue_created"), inArray(issues.status, ["done", "cancelled", "blocked"]),+ ...(statusReconcileCursor ? [sql`${routineRuns.id}::text > ${statusReconcileCursor}`] : [])))+ .orderBy(asc(routineRuns.id)).limit(50);+ statusReconcileCursor = candidates.length === 50 ? candidates.at(-1)!.id : null;+ let settled = 0;+ let deferred = 0;+ for (const candidate of candidates) {+ try {+ const repaired = await db.transaction(async tx => {+ await tx.execute(sql`SET LOCAL lock_timeout = '500ms'`);+ await tx.execute(sql`SET LOCAL statement_timeout = '5s'`);+ const [routine] = await tx.select({ id: routines.id }).from(routines).where(and(+ eq(routines.id, candidate.routineId), eq(routines.companyId, candidate.companyId))).for("share");+ if (!routine) return false;+ const [issue] = await tx.select().from(issues).where(and(eq(issues.id, candidate.issueId),+ eq(issues.companyId, candidate.companyId), eq(issues.originKind, "routine_execution"),+ eq(issues.originRunId, candidate.id), eq(issues.originId, candidate.routineId))).for("update");+ if (!issue || !["done", "cancelled", "blocked"].includes(issue.status)) return false;+ const [run] = await tx.select().from(routineRuns).where(and(eq(routineRuns.id, candidate.id),+ eq(routineRuns.companyId, candidate.companyId), eq(routineRuns.routineId, candidate.routineId),+ eq(routineRuns.linkedIssueId, issue.id), eq(routineRuns.status, "issue_created"))).for("update");+ if (!run) return false;+ return (await syncRunStatusForIssue(issue.id, tx as unknown as Db)) !== null;+ });+ if (repaired) settled += 1;+ } catch (error) {+ let current: unknown = error;+ let retryable = false;+ for (let depth = 0; depth < 5 && current && typeof current === "object"; depth++) {+ const cause = current as { code?: string; cause?: unknown };+ if (["55P03", "57014", "40P01", "40001"].includes(cause.code ?? "")) { retryable = true; break; }+ current = cause.cause;+ }+ if (!retryable) throw error;+ deferred += 1;+ }}- return null;+ return { scanned: candidates.length, settled, deferred };},++ syncRunStatusForIssue,};}