Skip to content

The open-source app everyone uses to manage agents at work

fix: keep sandbox termination progressing after bridge loss (#13287)

Nicky Leache926b1301712
diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md
index bc3e58f96f..121716fd55 100644
--- a/doc/execution-semantics.md
+++ b/doc/execution-semantics.md
@@ -1321,3 +1321,26 @@ card. A resolved card from that same source run, or its queued continuation, is
valid live path. A stale agent handback to a human cannot cancel the run and orphan
a queued response. This does not make an old resolved card a review path for a
later run, or prevent an explicit board reassignment.
+
+### Persistent sandbox cleanup
+
+A lost bridge cannot indefinitely prevent Daytona termination. Ordinary lease release
+and destruction wait briefly for bridge activity, then call the provider for the exact
+recorded sandbox. Drain timeout is not a stop receipt. Reusable sandboxes prefer
+stop; failed stop falls back to deletion. Stop/delete transport hangs are bounded
+and leave cleanup pending unless the provider confirms termination.
+
+The pending-cleanup sweep retains a durable attempt identity and a 15-minute
+in-flight deadline. It retries after restart, waits at least 30 seconds between
+failed attempts, and slows to 30 minutes after five failures. It reports that
+operator attention is needed at that threshold, while automatic cleanup continues.
+Provider outages never convert a live sandbox into an abandoned manual task.
+Explicit Retry can skip the cooldown after a failed cleanup, but cannot take over
+a live cleanup attempt. Active startup cancellation still stops the sandbox first.
+
+A live cleanup attempt renews its durable claim every 30 seconds. Another sweep in the same controller cannot overlap it, even if the deadline passes. Completion writes require the current attempt identity. After controller loss, cleanup can repeat destruction of the exact quarantined provider resource; providers must make that operation idempotent. A timeout or claim expiry does not prove termination.
+
+Renewal updates only the ownership deadline, never the retry cooldown. Cleanup
+does not await an outstanding renewal; a stalled database response cannot retain
+process-local cleanup ownership. Late responses still require the same active
+attempt, and completed attempts use only the persisted retry cooldown.
diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts
index 212e650ba8..c01e83a309 100644
--- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts
+++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts
@@ -253,7 +253,7 @@ describe("Daytona sandbox provider plugin", () => {
expect(manifest.version).toBe("0.1.7");
});
- it("opens a duplex channel, forwards a host write, and closes it on lease release", async () => {
+ it.each([false, true])("closes duplex routes on lease release even when bridge drain hangs: %s", async (hangDrain) => {
process.env.DAYTONA_API_KEY = "host-key";
// A fake PTY handle records each host write, drives the data stream on demand,
// and records the kill and the disconnect.
@@ -297,6 +297,8 @@ describe("Daytona sandbox provider plugin", () => {
},
} as unknown as PluginContext);
+ let finishBlocked: (() => void) | undefined;
+ let blocked: Promise<unknown> | undefined;
try {
await plugin.definition.onEnvironmentAcquireLease?.({
driverKey: "daytona",
@@ -306,7 +308,7 @@ describe("Daytona sandbox provider plugin", () => {
agentId: "agent-1",
executionWorkspaceId: "workspace-1",
adapterType: "codex_local",
- config: { image: "node:20", timeoutMs: 300000, reuseLease: true },
+ config: { image: "node:20", timeoutMs: 300000, livenessTimeoutMs: 5, reuseLease: true },
});
const open = await plugin.definition.onDuplexChannelOpen?.({
@@ -368,14 +370,24 @@ describe("Daytona sandbox provider plugin", () => {
},
]);
- // Lease release closes the channel: it kills the child and releases the
- // pseudo-terminal socket.
+ if (hangDrain) {
+ sandbox.process.executeCommand.mockImplementationOnce(async () => {
+ await new Promise<void>(resolve => { finishBlocked = resolve; });
+ return { exitCode: 0, result: "done", artifacts: { stdout: "done" } };
+ });
+ blocked = plugin.definition.onEnvironmentExecute?.({ driverKey: "daytona", companyId: "company-1",
+ environmentId: "env-1", config: { timeoutMs: 300000 }, bypassSession: true,
+ lease: { providerLeaseId: "sandbox-123", metadata: {} }, command: "printf", args: ["done"] });
+ await vi.waitFor(() => expect(finishBlocked).toBeTypeOf("function"));
+ }
+ sandbox.stop.mockImplementation(async () => { expect(disconnected).toBe(1); });
+ // Route invalidation must happen before provider stop, including after drain timeout.
await plugin.definition.onEnvironmentReleaseLease?.({
driverKey: "daytona",
companyId: "company-1",
environmentId: "env-1",
providerLeaseId: "sandbox-123",
- config: { image: "node:20", timeoutMs: 300000, reuseLease: true },
+ config: { image: "node:20", timeoutMs: 300000, livenessTimeoutMs: 5, reuseLease: true },
});
expect(killed).toBeGreaterThanOrEqual(1);
expect(disconnected).toBe(1);
@@ -390,6 +402,8 @@ describe("Daytona sandbox provider plugin", () => {
});
expect(inputs.length).toBe(inputsBefore);
} finally {
+ finishBlocked?.();
+ await blocked;
restore();
}
});
@@ -2706,6 +2720,29 @@ describe("Daytona sandbox provider plugin", () => {
await expect(plugin.definition.onEnvironmentExecute!(execParams("lease-a"))).rejects.toThrow(/no longer active/);
});
+ it.each(["release", "destroy"])("%s terminates through the provider when bridge activity never settles", async (kind) => {
+ process.env.DAYTONA_API_KEY = "host-key";
+ const sandbox = createMockSandbox({ id: "lease-a" });
+ let finish!: () => void;
+ sandbox.process.executeCommand.mockImplementation(async () => {
+ await new Promise<void>(resolve => { finish = resolve; });
+ return { exitCode: 0, result: "bash", artifacts: { stdout: "bash" } };
+ });
+ mockGet.mockResolvedValue(sandbox);
+ const execute = plugin.definition.onEnvironmentExecute?.(execParams("lease-a"));
+ await vi.waitFor(() => expect(finish).toBeTypeOf("function"));
+ const params = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
+ providerLeaseId: "lease-a", config: { timeoutMs: 300000, livenessTimeoutMs: 5, reuseLease: true } };
+ try {
+ const receipt = kind === "release"
+ ? await plugin.definition.onEnvironmentReleaseLease?.(params)
+ : await plugin.definition.onEnvironmentDestroyLease?.(params);
+ expect(receipt).toEqual({ providerLeaseId: "lease-a", state: kind === "release" ? "stopped" : "destroyed" });
+ expect(kind === "release" ? sandbox.stop : sandbox.delete).toHaveBeenCalledTimes(1);
+ await expect(plugin.definition.onEnvironmentExecute?.(execParams("lease-a"))).rejects.toThrow(/no longer active/);
+ } finally { finish(); await execute; }
+ });
+
it("waits for an in-flight execute before teardown cleanup starts", async () => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox({ id: "lease-a" });
diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts
index 1709810f37..151a22fa6a 100644
--- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts
+++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts
@@ -463,6 +463,29 @@ async function withLivenessTimeout<T>(
}
}
+/** A dead bridge must not prevent provider-level termination. Stop/delete is
+ * the receipt boundary; timing out this drain is never termination evidence. */
+async function drainSandboxBeforeTermination(sandbox: Sandbox, scope: SandboxScope) {
+ const timeoutMs = Math.max(1, Math.min(scope.config.livenessTimeoutMs || 10_000, 10_000));
+ const steps: [string, () => Promise<unknown>][] = [
+ ["sandbox.activityDrain", () => sandboxHandleActivityGates.waitForIdle(scope)],
+ ["sandbox.sessionTeardown", () => teardownSession(sandbox, scope)],
+ ["sandbox.channelTeardown", () => closeDaytonaDuplexChannelsForLease(scope.providerLeaseId)],
+ ];
+ for (const [operation, action] of steps) {
+ try { await withLivenessTimeout(operation, timeoutMs, action); }
+ catch {
+ // Each cleanup is independent; one hung bridge must not retain other routes.
+ console.warn("Sandbox bridge cleanup failed; continuing provider termination.");
+ }
+ }
+}
+
+async function terminateAtProvider<T>(scope: SandboxScope, operation: string, action: () => Promise<T>) {
+ const timeoutMs = scope.config.timeoutMs > 0 ? scope.config.timeoutMs : 300_000;
+ return withLivenessTimeout(operation, timeoutMs + LIVENESS_START_TIMEOUT_MARGIN_MS, action);
+}
+
async function ensureSandboxStarted(sandbox: Sandbox, timeoutSeconds: number): Promise<void> {
if (sandbox.state === "started") return;
// Bound the lifecycle call just past its own SDK deadline. A normal slow start
@@ -1556,6 +1579,9 @@ async function getOrCreateSession(sandbox: Sandbox, scope: SandboxScope): Promis
async function teardownSession(sandbox: Sandbox, scope: SandboxScope): Promise<void> {
const sessionId = sandboxHandleSessionStore.get(scope);
if (!sessionId) return;
+ // Retire the captured identity before awaiting the provider. A late response
+ // must not clear a new session created after this lease is resumed.
+ sandboxHandleSessionStore.clear(scope);
try {
// Wrap the session delete in a short `session.close` provider span. The
// host maps the name to `sandbox.daytona.session.close`.
@@ -1569,8 +1595,6 @@ async function teardownSession(sandbox: Sandbox, scope: SandboxScope): Promise<v
console.error(
`Failed to delete Daytona session ${sessionId} during teardown: ${formatErrorMessage(error)}`,
);
- } finally {
- sandboxHandleSessionStore.clear(scope);
}
}
@@ -1996,10 +2020,8 @@ async function closeDaytonaDuplexChannelsForLease(providerLeaseId: string): Prom
const matches = [...daytonaDuplexChannelByRoute.values()].filter(
(entry) => entry.providerLeaseId === providerLeaseId,
);
- for (const entry of matches) {
- forgetDaytonaDuplexChannel(entry);
- await entry.session.close().catch(() => undefined);
- }
+ for (const entry of matches) forgetDaytonaDuplexChannel(entry);
+ await Promise.allSettled(matches.map(entry => entry.session.close()));
}
const plugin = definePlugin({
@@ -2304,11 +2326,7 @@ const plugin = definePlugin({
await closeDaytonaDuplexChannelsForLease(params.providerLeaseId);
return { providerLeaseId: params.providerLeaseId, state: "stopped" };
}
- await sandboxHandleActivityGates.waitForIdle(scope);
- await teardownSession(sandbox, scope);
- // Close every duplex channel on this lease before the stop or the delete,
- // so no channel outlives the sandbox and no stored channel id survives.
- await closeDaytonaDuplexChannelsForLease(params.providerLeaseId);
+ await drainSandboxBeforeTermination(sandbox, scope);
// A cached stopped state is not a receipt: the resource could have been
// resumed since the handle was cached. Read provider state at this boundary.
@@ -2316,12 +2334,12 @@ const plugin = definePlugin({
if (config.reuseLease) {
if (sandbox.state !== "stopped") {
try {
- await sandbox.stop(toTimeoutSeconds(config.timeoutMs));
+ await terminateAtProvider(scope, "sandbox.stop", () => sandbox.stop(toTimeoutSeconds(config.timeoutMs)));
} catch (error) {
console.warn(
`Failed to stop Daytona sandbox during lease release: ${formatErrorMessage(error)}. Attempting delete instead.`,
);
- await sandbox.delete(toTimeoutSeconds(config.timeoutMs), true);
+ await terminateAtProvider(scope, "sandbox.delete", () => sandbox.delete(toTimeoutSeconds(config.timeoutMs), true));
return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
}
}
@@ -2331,7 +2349,7 @@ const plugin = definePlugin({
if (config.archiveOnRelease) {
try {
if (sandbox.state !== "stopped") {
- await sandbox.stop(toTimeoutSeconds(config.timeoutMs));
+ await terminateAtProvider(scope, "sandbox.stop", () => sandbox.stop(toTimeoutSeconds(config.timeoutMs)));
}
await sandbox.setAutoDeleteInterval(ARCHIVE_ON_RELEASE_AUTO_DELETE_MINUTES);
await sandbox.archive();
@@ -2343,7 +2361,7 @@ const plugin = definePlugin({
}
}
- await sandbox.delete(toTimeoutSeconds(config.timeoutMs), true);
+ await terminateAtProvider(scope, "sandbox.delete", () => sandbox.delete(toTimeoutSeconds(config.timeoutMs), true));
return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
} finally {
sandboxHandleTeardownGates.end(scope, teardownGate);
@@ -2372,12 +2390,8 @@ const plugin = definePlugin({
if (!sandbox) return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
evictSandboxHandle(scope);
- await sandboxHandleActivityGates.waitForIdle(scope);
- await teardownSession(sandbox, scope);
- // Close every duplex channel on this lease before the delete, so no channel
- // outlives the sandbox and no stored channel id survives.
- await closeDaytonaDuplexChannelsForLease(params.providerLeaseId);
- await sandbox.delete(toTimeoutSeconds(config.timeoutMs), true);
+ await drainSandboxBeforeTermination(sandbox, scope);
+ await terminateAtProvider(scope, "sandbox.delete", () => sandbox.delete(toTimeoutSeconds(config.timeoutMs), true));
return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
} finally {
sandboxHandleTeardownGates.end(scope, teardownGate);
diff --git a/server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts b/server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts
index 1a6babe409..aee3ccb40a 100644
--- a/server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts
+++ b/server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts
@@ -2,7 +2,9 @@ import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
+ activityLog,
agents,
+ heartbeatRuns,
companies,
createDb,
environmentLeases,
@@ -67,7 +69,9 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
});
afterEach(async () => {
+ await db.delete(activityLog);
await db.delete(environmentLeases);
+ await db.delete(heartbeatRuns);
await db.delete(environments);
await db.delete(agents);
await db.delete(companies);
@@ -233,7 +237,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
expect(destroyRunLease).toHaveBeenCalledTimes(20);
});
- it("test_pending_cleanup_sweep_stops_at_attempt_cap_and_warns_once", async () => {
+ it("test_pending_cleanup_sweep_continues_after_escalation_with_durable_backoff", async () => {
const { companyId, environmentId } = await seedCompanyAndEnvironment();
const leaseId = await insertPendingCleanupLease({
companyId,
@@ -249,8 +253,8 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
const first = await heartbeat.sweepPendingCleanupLeases({ backoffMs: 0 });
expect(first).toEqual({ swept: 1, destroyed: 0, capped: 1 });
- // A capped lease is not retried.
- expect(destroyRunLease).not.toHaveBeenCalled();
+ // Escalation never abandons a possibly running sandbox.
+ expect(destroyRunLease).toHaveBeenCalledTimes(1);
const metadataAfterFirst = await db
.select({ metadata: environmentLeases.metadata })
@@ -262,11 +266,238 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
// A second sweep warns no more; the lease keeps its warned flag.
const second = await heartbeat.sweepPendingCleanupLeases({ backoffMs: 0 });
- expect(second).toEqual({ swept: 1, destroyed: 0, capped: 1 });
- expect(destroyRunLease).not.toHaveBeenCalled();
+ expect(second).toEqual({ swept: 0, destroyed: 0, capped: 0 });
+ expect(destroyRunLease).toHaveBeenCalledTimes(1);
expect(vi.mocked(logger.warn)).toHaveBeenCalledTimes(1);
});
+ it("does not let malformed provider metadata defer cleanup forever", async () => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ await insertPendingCleanupLease({ companyId, environmentId,
+ updatedAt: new Date(Date.now() - 60 * 60_000), metadata: { pendingCleanupRetryAfterMs: 1e300 } });
+ const destroy = vi.fn(async () => null);
+ await heartbeatService(db, { environmentRuntime: fakeRuntime(destroy) }).sweepPendingCleanupLeases();
+ expect(destroy).toHaveBeenCalledTimes(1);
+ });
+
+ it.each([false, true])("uses the ownership deadline while cleanup is in flight (expired: %s)", async expired => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0), metadata: {
+ pendingCleanupAttemptId: "previous-boot", pendingCleanupInFlight: true,
+ pendingCleanupLeaseExpiresAtMs: Date.now() + (expired ? -60_000 : 60_000),
+ pendingCleanupRetryAfterMs: Date.now() + (expired ? 60_000 : -60_000),
+ } });
+ const teardown = vi.fn(async () => {});
+ const service = heartbeatService(db, { environmentRuntime: {
+ retryPendingSandboxTeardown: teardown,
+ } as unknown as HeartbeatEnvironmentRuntime });
+ expect((await service.sweepPendingCleanupLeases()).destroyed).toBe(expired ? 1 : 0);
+ expect(teardown).toHaveBeenCalledTimes(expired ? 1 : 0);
+ });
+
+ it("starts escalation and the long cooldown on the fifth failed attempt", async () => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertPendingCleanupLease({ companyId, environmentId,
+ updatedAt: new Date(Date.now() - 60 * 60_000), metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP - 1 } });
+ const runtime = fakeRuntime(vi.fn(async () => null));
+ await heartbeatService(db, { environmentRuntime: runtime }).sweepPendingCleanupLeases();
+ const [saved] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
+ expect(saved.metadata?.[CAP_WARNED_KEY]).toBe(true);
+ expect(Number(saved.metadata?.pendingCleanupRetryAfterMs)).toBeGreaterThan(Date.now() + 29 * 60_000);
+ });
+
+ it("retries after a restart and cooldown when the provider recovers", async () => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId,
+ updatedAt: new Date(Date.now() - 60 * 60_000),
+ metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP, pendingCleanupRetryAfterMs: Date.now() - 1 },
+ });
+ const failing = vi.fn(async () => { throw new Error("provider unavailable"); });
+ const first = heartbeatService(db, { environmentRuntime: {
+ retryPendingSandboxTeardown: failing,
+ } as unknown as HeartbeatEnvironmentRuntime });
+ await first.sweepPendingCleanupLeases();
+ expect(failing).toHaveBeenCalledTimes(1);
+ const recovered = vi.fn(async ({ lease }) => ({ providerLeaseId: lease.providerLeaseId, state: "destroyed" }));
+ const restarted = heartbeatService(db, { environmentRuntime: {
+ retryPendingSandboxTeardown: recovered,
+ } as unknown as HeartbeatEnvironmentRuntime });
+ await restarted.sweepPendingCleanupLeases();
+ expect(recovered).not.toHaveBeenCalled();
+ await db.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || '{"pendingCleanupRetryAfterMs":0}'::jsonb` })
+ .where(eq(environmentLeases.id, leaseId));
+ expect((await restarted.sweepPendingCleanupLeases()).destroyed).toBe(1);
+ expect(recovered).toHaveBeenCalledTimes(1);
+ const [saved] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
+ expect(saved.cleanupStatus).toBe("success");
+ expect(saved.status).toBe("expired");
+ });
+
+ it("does not start another cleanup while a previous attempt remains in flight", async () => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId,
+ updatedAt: new Date(Date.now() - 60 * 60_000),
+ metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP },
+ });
+ let finish!: () => void;
+ const started = Promise.withResolvers<void>();
+ const teardown = vi.fn(async () => { started.resolve(); await new Promise<void>(resolve => { finish = resolve; }); });
+ const first = heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: teardown } as unknown as HeartbeatEnvironmentRuntime });
+ const second = heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: teardown } as unknown as HeartbeatEnvironmentRuntime });
+ const running = first.sweepPendingCleanupLeases();
+ await started.promise;
+ try {
+ await db.update(environmentLeases).set({
+ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ pendingCleanupLeaseExpiresAtMs: Date.now() - 1 })}::jsonb`,
+ }).where(eq(environmentLeases.id, leaseId));
+ await second.sweepPendingCleanupLeases();
+ expect(teardown).toHaveBeenCalledTimes(1);
+ const [saved] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
+ expect(saved.metadata?.pendingCleanupInFlight).toBe(true);
+ } finally { finish(); await running; }
+ });
+
+ it.each([false, true])("explicit Retry bypasses cooldown but preserves in-flight ownership (%s)", async inFlight => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const agentId = randomUUID(), runId = randomUUID();
+ await db.insert(agents).values({ id: agentId, companyId, name: "Cleanup owner" });
+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "failed" });
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0),
+ metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP, pendingCleanupInFlight: inFlight,
+ pendingCleanupAttemptId: "previous-attempt", pendingCleanupRetryAfterMs: Date.now() + 15 * 60_000 },
+ });
+ await db.update(environmentLeases).set({ heartbeatRunId: runId }).where(eq(environmentLeases.id, leaseId));
+ const unrelatedLease = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0) });
+ const teardown = vi.fn(async () => {});
+ const service = heartbeatService(db, { environmentRuntime: {
+ retryPendingSandboxTeardown: teardown,
+ } as unknown as HeartbeatEnvironmentRuntime });
+ const result = await service.sweepPendingCleanupLeases({ explicitRetry: {
+ companyId, runId, actorId: "cleanup-reviewer",
+ } });
+ expect(result.destroyed).toBe(inFlight ? 0 : 1);
+ expect(teardown).toHaveBeenCalledTimes(inFlight ? 0 : 1);
+ expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, unrelatedLease)))[0].status).toBe("pending_cleanup");
+ const metadata = await readMetadata(leaseId);
+ if (inFlight) expect(metadata?.pendingCleanupAttemptId).toBe("previous-attempt");
+ else {
+ expect(metadata?.pendingCleanupManualAttemptId).toEqual(expect.any(String));
+ expect(metadata?.pendingCleanupAttemptId).not.toBe("previous-attempt");
+ const events = await db.select().from(activityLog).where(eq(activityLog.runId, runId));
+ expect(events).toEqual(expect.arrayContaining([expect.objectContaining({ action: "environment_lease.cleanup_retried" })]));
+ }
+ });
+
+ it("renews cleanup ownership while the provider remains blocked", async () => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0) });
+ const started = Promise.withResolvers<void>(), finish = Promise.withResolvers<void>();
+ const service = heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: async () => {
+ started.resolve(); await finish.promise;
+ } } as unknown as HeartbeatEnvironmentRuntime });
+ vi.useFakeTimers({ toFake: ["setInterval", "clearInterval"] });
+ const running = service.sweepPendingCleanupLeases();
+ await started.promise;
+ try {
+ await db.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || '{"pendingCleanupLeaseExpiresAtMs":1}'::jsonb` }).where(eq(environmentLeases.id, leaseId));
+ await vi.advanceTimersByTimeAsync(30000);
+ await vi.waitFor(async () => expect((await readMetadata(leaseId))?.pendingCleanupLeaseExpiresAtMs).toBeGreaterThan(Date.now() + 14 * 60_000));
+ } finally { finish.resolve(); await running; vi.useRealTimers(); }
+ });
+
+ it.each([
+ { renewalFails: false, retryBeforeSettlement: false },
+ { renewalFails: true, retryBeforeSettlement: false },
+ { renewalFails: false, retryBeforeSettlement: true },
+ { renewalFails: true, retryBeforeSettlement: true },
+ ])("preserves progress and cooldown with hung renewal (fails: $renewalFails, retry first: $retryBeforeSettlement)", async ({ renewalFails, retryBeforeSettlement }) => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0),
+ metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP - 1, [CAP_WARNED_KEY]: true },
+ });
+ const providerStarted = Promise.withResolvers<void>(), providerFinished = Promise.withResolvers<void>();
+ const renewalStarted = Promise.withResolvers<void>(), releaseRenewal = Promise.withResolvers<void>();
+ const renewalFinished = Promise.withResolvers<void>();
+ let holdNextRenewal = false, renewalSettled = false;
+ const realUpdate = db.update.bind(db);
+ const updateSpy = vi.spyOn(db, "update").mockImplementation(((table: unknown) => {
+ const builder = (realUpdate as (t: unknown) => unknown)(table) as { set: (values: Record<string, unknown>) => unknown };
+ const realSet = builder.set.bind(builder);
+ builder.set = values => {
+ const query = realSet(values) as { where: (predicate: unknown) => unknown };
+ if (table === environmentLeases && Object.keys(values).length === 1 && values.metadata && holdNextRenewal) {
+ holdNextRenewal = false;
+ const realWhere = query.where.bind(query);
+ query.where = predicate => (async () => {
+ renewalStarted.resolve();
+ await releaseRenewal.promise;
+ try {
+ if (renewalFails) throw new Error("renewal connection failed");
+ return await realWhere(predicate);
+ } finally { renewalSettled = true; renewalFinished.resolve(); }
+ })();
+ }
+ return query;
+ };
+ return builder;
+ }) as unknown as typeof db.update);
+ vi.useFakeTimers({ toFake: ["Date", "setInterval", "clearInterval"] });
+ const teardown = vi.fn(async () => {
+ providerStarted.resolve(); await providerFinished.promise; throw new Error("provider unavailable");
+ });
+ const service = heartbeatService(db, { environmentRuntime: {
+ retryPendingSandboxTeardown: teardown,
+ } as unknown as HeartbeatEnvironmentRuntime });
+ const running = service.sweepPendingCleanupLeases();
+ try {
+ await providerStarted.promise;
+ holdNextRenewal = true;
+ await vi.advanceTimersByTimeAsync(30_000);
+ await renewalStarted.promise;
+ providerFinished.resolve();
+ await running;
+ expect(renewalSettled).toBe(false);
+ const metadata = await readMetadata(leaseId);
+ expect(metadata?.pendingCleanupInFlight).toBe(false);
+ expect(metadata?.pendingCleanupRetryAfterMs).toBeGreaterThan(Date.now() + 29 * 60_000);
+ expect((await service.sweepPendingCleanupLeases()).swept).toBe(0);
+ if (retryBeforeSettlement) {
+ // Advance only the clock: renewal remains blocked across another attempt.
+ vi.setSystemTime(Date.now() + 31 * 60_000);
+ expect((await service.sweepPendingCleanupLeases()).swept).toBe(1);
+ expect(teardown).toHaveBeenCalledTimes(2);
+ expect(renewalSettled).toBe(false);
+ }
+ const afterRetry = await readMetadata(leaseId);
+ releaseRenewal.resolve();
+ await renewalFinished.promise;
+ expect(await readMetadata(leaseId)).toEqual(afterRetry);
+ } finally {
+ providerFinished.resolve(); releaseRenewal.resolve(); await running;
+ await renewalFinished.promise;
+ vi.useRealTimers(); updateSpy.mockRestore();
+ }
+ });
+
+ it.each([false, true])("ignores a superseded cleanup completion (throws: %s)", async throws => {
+ const { companyId, environmentId } = await seedCompanyAndEnvironment();
+ const leaseId = await insertOrphanEphemeralLease({ companyId, environmentId, updatedAt: new Date(0) });
+ const started = Promise.withResolvers<void>(), finish = Promise.withResolvers<void>();
+ const service = heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: async () => {
+ started.resolve(); await finish.promise; if (throws) throw new Error("old attempt failed");
+ } } as unknown as HeartbeatEnvironmentRuntime });
+ const running = service.sweepPendingCleanupLeases();
+ await started.promise;
+ try {
+ await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success",
+ metadata: { pendingCleanupAttemptId: "newer-attempt", remoteExecutionTermination: { proof: "newer-receipt" } },
+ }).where(eq(environmentLeases.id, leaseId));
+ } finally { finish.resolve(); await running; }
+ const [saved] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
+ expect(saved.status).toBe("expired");
+ expect(saved.metadata?.remoteExecutionTermination).toEqual({ proof: "newer-receipt" });
+ });
+
// Two sweep ticks can overlap. Without an atomic claim, both read the same
// attempt count, both destroy the same lease, and the retry cap counts one
// attempt for two destroys. The atomic claim must let only one sweep destroy
@@ -772,7 +1003,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
.mocked(logger.warn)
.mock.calls.filter(
(call) =>
- call[1] === "environment lease reached the pending_cleanup retry cap; left for manual cleanup",
+ call[1] === "environment lease needs operator attention; automatic cleanup continues with backoff",
);
expect(capWarnings.length).toBeLessThanOrEqual(1);
});
@@ -1069,7 +1300,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
.mocked(logger.warn)
.mock.calls.filter(
(call) =>
- call[1] === "environment lease reached the pending_cleanup retry cap; left for manual cleanup",
+ call[1] === "environment lease needs operator attention; automatic cleanup continues with backoff",
);
expect(capWarnings.length).toBe(0);
@@ -1079,7 +1310,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
companyId,
environmentId,
updatedAt: new Date(Date.now() - 60 * 60 * 1000),
- metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP - 1 },
+ metadata: { [ATTEMPTS_KEY]: ATTEMPT_CAP - 2 },
});
await heartbeat.sweepPendingCleanupLeases({ backoffMs: 0 });
const belowCapMetadata = await readMetadata(belowCapLeaseId);
@@ -1198,7 +1429,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
// The value clamped to the cap, so the lease took the cap branch, not a
// destroy. The warn-once flag is set exactly once.
expect(result.capped).toBe(1);
- expect(destroyedLeaseIds).not.toContain(outOfRangeLeaseId);
+ expect(destroyedLeaseIds).toContain(outOfRangeLeaseId);
const outOfRangeMetadata = await readMetadata(outOfRangeLeaseId);
expect(outOfRangeMetadata?.[CAP_WARNED_KEY]).toBe(true);
diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts
index 2deb410986..5efccf36af 100644
--- a/server/src/services/environment-runtime.ts
+++ b/server/src/services/environment-runtime.ts
@@ -3136,6 +3136,8 @@ function createSandboxEnvironmentDriver(
input.lease.id,
cleanupStatus === "success" ? "expired" : "pending_cleanup",
{
+ ...(input.lease.status === "pending_cleanup" && typeof metadata.pendingCleanupAttemptId === "string"
+ ? { expectedPendingCleanupAttemptId: metadata.pendingCleanupAttemptId } : {}),
failureReason: input.failureReason,
cleanupStatus,
...(cleanupStatus === "success" && termination ? { remoteExecutionTermination: termination } : {}),
diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts
index e641125b2f..ea01905eb3 100644
--- a/server/src/services/environments.ts
+++ b/server/src/services/environments.ts
@@ -1613,6 +1613,7 @@ export function environmentService(db: Db) {
id: string,
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed" | "retained" | "pending_cleanup"> = "released",
options?: {
+ expectedPendingCleanupAttemptId?: string;
failureReason?: string;
cleanupStatus?: EnvironmentLeaseCleanupStatus;
remoteExecutionTermination?: Record<string, unknown>;
@@ -1634,7 +1635,10 @@ export function environmentService(db: Db) {
? sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ remoteExecutionTermination: options.remoteExecutionTermination })}::jsonb`
: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'`,
})
- .where(eq(environmentLeases.id, id))
+ .where(and(eq(environmentLeases.id, id), options?.expectedPendingCleanupAttemptId
+ ? and(eq(environmentLeases.status, "pending_cleanup"),
+ sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.expectedPendingCleanupAttemptId}`)
+ : undefined))
.returning()
.then((rows) => rows[0] ?? null);
return row ? toEnvironmentLease(row) : null;
diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts
index abb5961d5f..67514c3081 100644
--- a/server/src/services/heartbeat.ts
+++ b/server/src/services/heartbeat.ts
@@ -658,7 +658,8 @@ const NATIVE_OWNERSHIP_UNVERIFIED_MESSAGE =
"Native execution ownership could not be verified; automatic recovery is blocked";
// The reaper sweeps at most this many pending_cleanup leases per tick.
const PENDING_CLEANUP_SWEEP_PAGE_SIZE = 20;
-// The reaper stops retrying a pending_cleanup lease after this many attempts.
+const pendingCleanupAttemptsInFlight = new Set<string>();
+// Escalate and slow cleanup after this many attempts; never abandon a live lease.
const PENDING_CLEANUP_SWEEP_ATTEMPT_CAP = 5;
// The reaper stores its retry state under these keys in the lease metadata.
const PENDING_CLEANUP_ATTEMPTS_METADATA_KEY = "pendingCleanupRetryAttempts";
@@ -699,6 +700,23 @@ function pendingCleanupAttemptsSql() {
end`;
}
+function pendingCleanupRetryDueSql(explicitRetry = false) {
+ // Ownership renewal and retry cooldown are separate clocks. A late renewal
+ // cannot change when a completed attempt may retry. Older claims used the
+ // retry field for both clocks, so retain that fallback until they settle.
+ const deadline = sql`case
+ when ${environmentLeases.metadata}->>'pendingCleanupInFlight' = 'true'
+ and ${environmentLeases.metadata} ? 'pendingCleanupLeaseExpiresAtMs'
+ then ${environmentLeases.metadata}->'pendingCleanupLeaseExpiresAtMs'
+ else ${environmentLeases.metadata}->'pendingCleanupRetryAfterMs' end`;
+ return sql`case
+ when ${explicitRetry} and coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true' then true
+ when jsonb_typeof(${deadline}) = 'number'
+ then (${deadline})::numeric <= ${Date.now()}
+ or (${deadline})::numeric > ${Date.now() + 30 * 60_000 + 1_000}
+ else true end`;
+}
+
// Choose the `jsonb_set` target root. A provider can write a scalar or array
// metadata root. `jsonb_set` fails on a non-object root, so the reader uses the
// stored metadata only when its root is an object. A NULL, scalar, or array root
@@ -10140,7 +10158,7 @@ export function heartbeatService(
state: "acknowledged", acknowledgedAt: new Date().toISOString(),
proof: "provider_termination_receipt" },
conversationContinuation: CONVERSATION_CONTINUATION_POLICY,
- })}::jsonb`,
+ })}::jsonb`,
updatedAt: new Date(),
}).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.status, "cancelled")));
@@ -18091,25 +18109,29 @@ export function heartbeatService(
// only matches when the lease is still pending_cleanup and its stored attempt
// count still equals `expectedAttempts`. Two concurrent sweeps read the same
// count, but Postgres serializes the two updates on the row and only the first
- // matches the guard. The loser gets zero rows and skips the lease. This bounds
- // the retries to the cap and stops a second destroy of the same lease.
- // Returns true only for the sweep that won the claim.
+ // matches the guard. The loser gets zero rows and skips the lease. The persisted in-flight deadline also prevents a later tick
+ // from starting another attempt while this one is still running.
+ // Returns the attempt identity only for the sweep that won the claim.
//
- // The update writes only the attempts key with `jsonb_set`. It never writes a
- // copied metadata object, so a concurrent write to an unrelated metadata key
+ // The update patches cleanup ownership fields without writing a copied
+ // metadata object, so a concurrent write to an unrelated metadata key
// survives. The guard reads the stored count through the safe SQL reader, so a
// malformed value never throws.
async function claimPendingCleanupRetryAttempt(
leaseId: string,
expectedAttempts: number,
manualAttempt?: { previousId: unknown },
- ): Promise<boolean> {
+ ): Promise<string | null> {
const now = new Date();
+ const attemptId = randomUUID();
const claimed = await db
.update(environmentLeases)
.set({
metadata: sql`jsonb_set(${pendingCleanupMetadataObjectSql()}, array[${PENDING_CLEANUP_ATTEMPTS_METADATA_KEY}], to_jsonb(${expectedAttempts + 1}::int), true)
- || ${JSON.stringify(manualAttempt ? { pendingCleanupManualAttemptId: randomUUID() } : {})}::jsonb`,
+ || ${JSON.stringify({ ...(manualAttempt ? { pendingCleanupManualAttemptId: randomUUID() } : {}),
+ pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true,
+ pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
+ })}::jsonb`,
lastUsedAt: now,
updatedAt: now,
})
@@ -18119,10 +18141,11 @@ export function heartbeatService(
eq(environmentLeases.status, "pending_cleanup"),
sql`${pendingCleanupAttemptsSql()} = ${expectedAttempts}`,
manualAttempt ? sql`coalesce(${environmentLeases.metadata}->'pendingCleanupManualAttemptId', 'null'::jsonb) is not distinct from ${JSON.stringify(manualAttempt.previousId ?? null)}::jsonb` : undefined,
+ pendingCleanupRetryDueSql(Boolean(manualAttempt)),
),
)
.returning({ id: environmentLeases.id });
- return claimed.length > 0;
+ return claimed.length > 0 ? attemptId : null;
}
// Atomically claim the one-time cap warning for a lease. The update only
@@ -18298,12 +18321,13 @@ export function heartbeatService(
// sweep. The backoff equals the reaper staleness threshold, so a lease waits
// for that period between attempts. The sweep reads and writes the attempt
// count in the lease metadata. It warns once when a lease reaches the attempt
- // cap and then stops the retries for that lease.
+ // threshold, then continues with slower retries. Live attempts renew their
+ // cleanup claim; after controller loss, exact-resource destruction may repeat.
async function sweepPendingCleanupLeases(opts?: {
backoffMs?: number;
/** One cleanup attempt per explicit user Retry, for this failed run only.
- * A later user Retry may try again after a provider failure; automatic
- * sweeps retain their exhausted budget and never gain extra attempts.
+ * A later user Retry may bypass the cooldown after a provider failure,
+ * but cannot take over an in-flight cleanup attempt.
*/
explicitRetry?: { companyId: string; runId: string; actorId: string; reason?: "retry_failed_run" | "queued_comment_interrupt" };
}): Promise<{
@@ -18346,6 +18370,7 @@ export function heartbeatService(
eq(environmentLeases.status, "pending_cleanup"),
opts?.explicitRetry ? eq(environmentLeases.companyId, opts.explicitRetry.companyId) : undefined,
opts?.explicitRetry ? eq(environmentLeases.heartbeatRunId, opts.explicitRetry.runId) : undefined,
+ pendingCleanupRetryDueSql(Boolean(opts?.explicitRetry)),
backoffMs > 0 ? lte(environmentLeases.updatedAt, cutoff) : undefined,
),
)
@@ -18355,25 +18380,10 @@ export function heartbeatService(
let destroyed = 0;
let capped = 0;
for (const row of rows) {
+ if (pendingCleanupAttemptsInFlight.has(row.id)) continue;
const metadata = { ...(row.metadata ?? {}) } as Record<string, unknown>;
const attempts = readPendingCleanupRetryAttempts(metadata);
- if (attempts >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP && !opts?.explicitRetry) {
- capped += 1;
- // Warn once, then leave the lease for manual cleanup. The atomic claim
- // keeps the warning to one log line even when two sweeps overlap.
- if (metadata[PENDING_CLEANUP_CAP_WARNED_METADATA_KEY] !== true) {
- const warned = await claimPendingCleanupCapWarning(row.id);
- if (warned) {
- logger.warn(
- { leaseId: row.id, environmentId: row.environmentId, attempts },
- "environment lease reached the pending_cleanup retry cap; left for manual cleanup",
- );
- }
- }
- continue;
- }
-
const environment = row.environmentId
? await environmentsSvc.getById(row.environmentId)
: null;
@@ -18426,19 +18436,34 @@ export function heartbeatService(
// Atomically claim the attempt before the retry. Only the winning sweep
// increments the count and tears the sandbox down, so an overlapping sweep
- // never tears the same sandbox down twice or exceeds the attempt cap. The
- // claim records the attempt before the retry, so a thrown driver error
- // still counts against the cap.
+ // cannot start another attempt while this one holds the cleanup lease.
+ // The deadline survives a server restart; the counter saturates at the
+ // escalation threshold while attempt identities remain unique.
const claimed = await claimPendingCleanupRetryAttempt(row.id, attempts,
opts?.explicitRetry ? { previousId: metadata.pendingCleanupManualAttemptId } : undefined);
if (!claimed) continue;
- if (opts?.explicitRetry) await logActivity(db, {
- companyId: row.companyId, actorType: "user", actorId: opts.explicitRetry.actorId,
- action: "environment_lease.cleanup_retried", entityType: "environment_lease", entityId: row.id,
- runId: opts.explicitRetry.runId, details: { attempt: attempts + 1, reason: opts.explicitRetry.reason ?? "retry_failed_run" },
- });
+ pendingCleanupAttemptsInFlight.add(row.id);
+ lease.metadata = { ...lease.metadata, pendingCleanupAttemptId: claimed };
+ let activeRenewal: Promise<void> | null = null;
+ const renewal = setInterval(() => {
+ if (activeRenewal) return;
+ activeRenewal = db.update(environmentLeases).set({
+ metadata: sql`jsonb_set(${pendingCleanupMetadataObjectSql()}, '{pendingCleanupLeaseExpiresAtMs}', to_jsonb(${Date.now() + 15 * 60_000}::bigint))`,
+ }).where(and(eq(environmentLeases.id, row.id), eq(environmentLeases.status, "pending_cleanup"),
+ sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${claimed}`,
+ sql`${environmentLeases.metadata}->>'pendingCleanupInFlight' = 'true'`))
+ .then(() => {})
+ .catch(() => logger.warn({ leaseId: row.id }, "cleanup ownership renewal failed"))
+ .finally(() => { activeRenewal = null; });
+ }, 30_000);
+ renewal.unref();
try {
+ if (opts?.explicitRetry) await logActivity(db, {
+ companyId: row.companyId, actorType: "user", actorId: opts.explicitRetry.actorId,
+ action: "environment_lease.cleanup_retried", entityType: "environment_lease", entityId: row.id,
+ runId: opts.explicitRetry.runId, details: { attempt: attempts + 1, reason: opts.explicitRetry.reason ?? "retry_failed_run" },
+ });
if (useRecordedTeardown) {
// Tear the sandbox down from the recorded provider config and the
// cleanup-authorized secret versions. Preserve any provider receipt;
@@ -18447,12 +18472,13 @@ export function heartbeatService(
environment,
lease,
});
- await environmentsSvc.releaseLease(lease.id, "expired", {
+ const released = await environmentsSvc.releaseLease(lease.id, "expired", {
+ expectedPendingCleanupAttemptId: claimed,
cleanupStatus: "success",
failureReason: "pending_cleanup_retry",
remoteExecutionTermination: remoteTerminationReceipt(lease, receipt),
});
- destroyed += 1;
+ if (released) destroyed += 1;
} else if (environment) {
const result = await environmentRuntime.destroyRunLease({
environment,
@@ -18466,11 +18492,12 @@ export function heartbeatService(
} catch {
// The recorded-data teardown throws on failure, so revert the lease to
// pending_cleanup for a later sweep. The claimed attempt still counts
- // against the cap, so the retries stay bounded. The `destroyRunLease`
+ // for backoff, so requests stay bounded. The `destroyRunLease`
// path reverts the lease itself, so this revert only runs for the
// recorded-data teardown path.
if (useRecordedTeardown) {
await environmentsSvc.releaseLease(lease.id, "pending_cleanup", {
+ expectedPendingCleanupAttemptId: claimed,
cleanupStatus: "failed",
failureReason: "pending_cleanup_retry",
});
@@ -18486,7 +18513,36 @@ export function heartbeatService(
},
"pending_cleanup lease retry failed",
);
+ } finally {
+ clearInterval(renewal);
+ // A hung renewal must not retain process-local cleanup ownership.
+ // Late renewals are attempt-fenced and never write the retry cooldown.
+ pendingCleanupAttemptsInFlight.delete(row.id);
+ }
+ if (attempts + 1 >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP) {
+ capped += 1;
+ // Warn once, then continue automatic cleanup with backoff. The atomic claim
+ // keeps the warning to one log line even when two sweeps overlap.
+ if (metadata[PENDING_CLEANUP_CAP_WARNED_METADATA_KEY] !== true) {
+ const warned = await claimPendingCleanupCapWarning(row.id);
+ if (warned) {
+ logger.warn(
+ { leaseId: row.id, environmentId: row.environmentId, attempts },
+ "environment lease needs operator attention; automatic cleanup continues with backoff",
+ );
+ }
+ }
}
+ // Persist the cooldown independently of process memory. A crash before
+ // this write leaves the bounded in-flight lease for a later sweep.
+ await db.update(environmentLeases).set({
+ metadata: sql`${pendingCleanupMetadataObjectSql()} || ${JSON.stringify({
+ pendingCleanupInFlight: false,
+ pendingCleanupRetryAfterMs: Date.now() + (attempts + 1 >= PENDING_CLEANUP_SWEEP_ATTEMPT_CAP
+ ? 30 * 60_000 : Math.min(30 * 60_000, Math.max(30_000, backoffMs))),
+ })}::jsonb`,
+ }).where(and(eq(environmentLeases.id, lease.id),
+ sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${claimed}`));
if (lease.heartbeatRunId) {
// Delivery failure must not revert successful provider cleanup. A new
// message can still use the persisted receipt on its next admission.