Skip to content

The open-source app everyone uses to manage agents at work

fix(recovery): let preservation owners reopen retained sources (#17053)

Dottaeaacadeba3e2
diff --git a/doc/task-recovery.md b/doc/task-recovery.md
index bc59048df5..9198e53c9e 100644
--- a/doc/task-recovery.md
+++ b/doc/task-recovery.md
@@ -33,6 +33,12 @@ The source recovery action holds a validated `systemRecovery` record. Its
revision and generation fence late bookkeeping writes. Physical exports still
use the existing workspace ownership lock and controller stop receipts.
+The export owner can reopen its exact stopped sandbox through the environment
+runtime's recovery path. This path checks the current source action, run,
+environment, allocation, stop-only intent, and physical ownership claim before
+calling the provider. The retained-source fence still blocks ordinary resume
+and destruction. Recovery does not clear it to gain access to the saved files.
+
Execution containment and file preservation are independent facts:
- Another agent needs authoritative containment of the old executor.
@@ -113,6 +119,8 @@ Run the focused PostgreSQL/filesystem suites from `server/`:
`server/scripts/workspace-recovery-provider.test.mjs` is an opt-in controlled
Daytona probe. It requires `PAPERCLIP_RECOVERY_LIVE=1`, `DAYTONA_API_KEY`,
`PAPERCLIP_DAYTONA_SDK_PATH` and an immutable `PAPERCLIP_RECOVERY_IMAGE`.
+An immutable provider snapshot UUID in `PAPERCLIP_RECOVERY_SNAPSHOT_ID` can
+replace the image digest. The probe checks its ID and active state first.
It interrupts export,
joins the first controller, starts a second controller and verifies files,
preservation receipts, agent-start count and external-operation count before
diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md
index 2b37935279..77ea407fd9 100644
--- a/maps/agent-access-map/recovery.md
+++ b/maps/agent-access-map/recovery.md
@@ -212,6 +212,12 @@ Gaps: An acknowledged retry can remain pending or fail; report actual useful res
Capability ID: `recovery/workspace`.
+The internal preservation resume path grants no board or agent execution API.
+It requires the exact source action, company, run, lease and environment, a
+held physical workspace owner, and its current stop-only cleanup claim. The
+ordinary resume and destruction fences still apply to protected sources. The
+assembled-runtime regression checks this separation before any provider call.
+
Review: `operation-reviewed`.
Each workspace repair/export/reconstruction/physical-owner operation has its own proof and authority.
diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md
index 8f9f00902b..4fb2ff30d4 100644
--- a/maps/feature-map/recovery.md
+++ b/maps/feature-map/recovery.md
@@ -222,6 +222,14 @@ cover quiet live recovery, read-only access, stale or missing stop proof,
source-only repair, and preserved task status. These tests use fixtures; separately
verify original files and their attached artifact before recording a real repair.
+For a source with a retained-file protection marker, exercise recovery through
+the assembled environment runtime. Its exact preservation owner may reopen the
+stopped allocation to read saved files; ordinary resume stays blocked. Reject a
+changed claim, missing stop intent, wrong lease or wrong environment before a
+provider call. Verify saved and newer local files, one preservation receipt, and
+no additional agent turn. The PostgreSQL regression covers this boundary; the
+opt-in provider probe also uses this runtime path with the protection marker.
+
Repeat source-only repair with a `succeeded` legacy run that still has a retained
file obligation. Its result must remain successful, its current task owner and
status must stay unchanged, and later saved input must become eligible for normal
diff --git a/server/scripts/workspace-recovery-provider.test.mjs b/server/scripts/workspace-recovery-provider.test.mjs
index a9902ae580..e4a7411dae 100644
--- a/server/scripts/workspace-recovery-provider.test.mjs
+++ b/server/scripts/workspace-recovery-provider.test.mjs
@@ -15,6 +15,7 @@ import { NOOP_PLUGIN_TRACER } from "@paperclipai/plugin-sdk";
import { startEmbeddedPostgresTestDatabase } from "../src/__tests__/helpers/embedded-postgres.js";
import { readSystemRecovery } from "../src/services/system-recovery-state.js";
import { recoverRetainedWorkspace } from "../src/services/workspace-recovery.js";
+import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../src/services/environment-runtime.js";
import { attachPreservedWorkspaceSnapshots } from "../src/services/workspace-recovery-snapshot.js";
import { performSyncOut } from "../../packages/plugins/sandbox-providers/daytona/src/file-sync.js";
@@ -68,7 +69,11 @@ it.skipIf(!live || !worker)("provider controller", async () => {
},
syncOut: operations => performSyncOut({ sandbox, operations, remoteDir: f.remoteCwd, timeoutSeconds: 120 }),
} };
- const environmentRuntime = {
+ const driver = {
+ driver: "sandbox",
+ acquireRunLease: async () => { throw new Error("Recovery must not acquire another sandbox"); },
+ releaseRunLease: async () => { throw new Error("Recovery must not destroy its retained source"); },
+ resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [] }),
resumeRunLease: async () => {
await sandbox.start(120); await sandbox.refreshData();
if (sandbox.state !== "started") throw new Error("Provider did not confirm start");
@@ -80,6 +85,7 @@ it.skipIf(!live || !worker)("provider controller", async () => {
return { providerLeaseId: sandbox.id, state: "stopped" };
},
};
+ const environmentRuntime = environmentRuntimeService(db, { drivers: [driver] });
await recoverRetainedWorkspace({ db, ...f, environmentRuntime });
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
const state = readSystemRecovery(action.evidence);
@@ -90,7 +96,9 @@ it.skipIf(!live || !worker)("provider controller", async () => {
it.skipIf(!live || Boolean(worker))("preserves provider files across an interrupted export and controller restart without restarting the old agent", async () => {
const image = process.env.PAPERCLIP_RECOVERY_IMAGE;
- if (!image?.includes("@sha256:")) throw new Error("An immutable qualification image is required");
+ const snapshotId = process.env.PAPERCLIP_RECOVERY_SNAPSHOT_ID;
+ if (!image?.includes("@sha256:") && !/^[0-9a-f-]{36}$/i.test(snapshotId ?? ""))
+ throw new Error("An immutable qualification image or snapshot ID is required");
const directory = await mkdtemp(path.join(os.tmpdir(), "paperclip-provider-recovery-"));
const priorHome = process.env.PAPERCLIP_HOME;
process.env.PAPERCLIP_HOME = path.join(directory, "home");
@@ -100,11 +108,16 @@ it.skipIf(!live || Boolean(worker))("preserves provider files across an interrup
const db = createDb(database.connectionString);
const daytona = await client();
let sandbox;
- const evidence = { image, interrupted: false, restored: false, oldAgentStarts: null, externalOperations: null,
+ const evidence = { image, snapshotId, interrupted: false, restored: false, oldAgentStarts: null, externalOperations: null,
prematureCleanup: false, controllerProcesses: [], snapshotCount: 0 };
try {
- sandbox = await daytona.create({ image, autoStopInterval: 10, autoArchiveInterval: 60,
- resources: { cpu: 2, memory: 4, disk: 10 }, labels: { purpose: "paperclip-recovery-qualification" } }, { timeout: 300 });
+ if (snapshotId) {
+ const snapshot = await daytona.snapshot.get(snapshotId);
+ expect(snapshot.id).toBe(snapshotId);
+ expect(snapshot.state).toBe("active");
+ }
+ sandbox = await daytona.create({ ...(snapshotId ? { snapshot: snapshotId } : { image, resources: { cpu: 2, memory: 4, disk: 10 } }),
+ autoStopInterval: 10, autoArchiveInterval: 60, labels: { purpose: "paperclip-recovery-qualification" } }, { timeout: 300 });
const remoteCwd = "/tmp/paperclip-recovery-fixture";
await sandbox.process.executeCommand(`mkdir -p ${remoteCwd}`, undefined, undefined, 30);
await sandbox.process.createSession("old-agent");
@@ -124,7 +137,8 @@ it.skipIf(!live || Boolean(worker))("preserves provider files across an interrup
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
environmentId: environment.id, provider: "daytona", providerLeaseId: sandbox.id,
leasePolicy: "retain_on_failure", status: "released", releasedAt: new Date(),
- metadata: { remoteCwd, driver: "sandbox", pluginId: randomUUID(), sandboxProviderPlugin: true } }).returning();
+ metadata: { remoteCwd, driver: "sandbox", pluginId: randomUUID(), sandboxProviderPlugin: true,
+ workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId } } }).returning();
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [lease.id] };
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreRecovery: recovery, workspaceRestoreFailure: "restore_failed" } })
.where(eq(heartbeatRuns.id, runId));
diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts
index 7d14dbf7c8..da016c9249 100644
--- a/server/src/services/environment-runtime.ts
+++ b/server/src/services/environment-runtime.ts
@@ -4,6 +4,8 @@ import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourcePro
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { historicalProviderCredentials } from "./historical-provider-credentials.js";
+import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
+import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
@@ -4569,6 +4571,19 @@ export function environmentRuntimeService(
return await driver.resumeRunLease(input);
},
+ async resumeWorkspaceRecoveryLease(input: {
+ source: WorkspaceRecoverySourceIdentity; leaseId: string; environmentId: string;
+ ownership: NativeWorkspaceFinalizationOwnership;
+ }): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
+ const { lease, environment } = await ownedWorkspaceRecoveryLease(db, { ...input.source,
+ leaseId: input.leaseId, environmentId: input.environmentId, ownership: input.ownership });
+ const driver = requireDriverKey(getLeaseDriverKey(lease, environment as Environment));
+ if (!driver.resumeRunLease) throw new Error(`Environment driver "${driver.driver}" does not support lease resume.`);
+ // Maintenance reopens the protected source only to preserve its files.
+ // Ordinary resume and destructive cleanup keep their existing fences.
+ return await driver.resumeRunLease({ environment: environment as Environment, lease: lease as EnvironmentLease });
+ },
+
async destroyRunLease(input: EnvironmentDriverLeaseInput): Promise<EnvironmentLease | null> {
if (await nativeWorkspaceExportPendingForLease(db, input.lease)) return null;
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
diff --git a/server/src/services/workspace-recovery-source.ts b/server/src/services/workspace-recovery-source.ts
index cd5e0b61f2..ab8517aa88 100644
--- a/server/src/services/workspace-recovery-source.ts
+++ b/server/src/services/workspace-recovery-source.ts
@@ -3,11 +3,31 @@ import { environmentLeases, environments, heartbeatRuns, issueRecoveryActions, n
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { SystemRecoveryFailure } from "./system-recovery-state.js";
+import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
+import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string };
+/** The preservation owner may reopen only its exact stop-only allocation.
+ * This does not release the source's ordinary execution or destruction fence. */
+export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecoverySourceIdentity & {
+ leaseId: string; environmentId: string; ownership: NativeWorkspaceFinalizationOwnership;
+}) {
+ await input.ownership.assertHeld();
+ const inspected = await inspectWorkspaceRecoverySource(db, input);
+ const source = inspected.sources.find(item => item.lease.id === input.leaseId);
+ if (!source || source.environment.id !== input.environmentId || source.lease.status !== "pending_cleanup"
+ || source.lease.metadata?.pendingCleanupInFlight !== true
+ || source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token
+ || !readStopOnlyCleanup(source.lease)) {
+ throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
+ }
+ await input.ownership.assertHeld();
+ return source;
+}
+
/** Resolve only the immutable source recorded by the failed run. Never discover
* a candidate by recency, mutable adapter configuration, or a provider search. */
export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity) {
diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts
index 2576dbe394..6b10de5b61 100644
--- a/server/src/services/workspace-recovery.test.ts
+++ b/server/src/services/workspace-recovery.test.ts
@@ -1,6 +1,7 @@
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js";
import { environmentService } from "./environments.js";
+import { classifyEnvironmentCapabilities, environmentRuntimeService } from "./environment-runtime.js";
import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { workspaceOperationService } from "./workspace-operations.js";
@@ -111,7 +112,11 @@ describe("automatic retained workspace recovery", () => {
fingerprint: runId, nextAction: "Manual repair required", evidence: { runId, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery } });
const runtime = { resumeRunLease: vi.fn(async (_input: { lease: EnvironmentLease }) => ({ providerLeaseId, metadata: { remoteCwd: remote } })),
retryPendingSandboxTeardown: vi.fn(async () => ({ providerLeaseId, state: "stopped" })) };
- const input = { db, companyId, issueId, runId, actionId, environmentRuntime: runtime as never };
+ const environmentRuntime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
+ acquireRunLease: vi.fn(), releaseRunLease: vi.fn(), ...runtime,
+ resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [], declared: null, narrowing: null }),
+ }] });
+ const input = { db, companyId, issueId, runId, actionId, environmentRuntime };
return { ...input, input, runtime, host, remote, lease, providerLeaseId };
}
const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
@@ -283,6 +288,52 @@ describe("automatic retained workspace recovery", () => {
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.phase, "workspace_recovery")))).toHaveLength(1);
}, 30_000);
+ it("recovers a protected retained source through the assembled environment runtime", async () => {
+ const f = await seed();
+ await db.update(environmentLeases).set({ metadata: { ...f.lease.metadata,
+ workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: f.runId },
+ remoteExecutionTermination: remoteTerminationReceipt(f.lease, { providerLeaseId: f.providerLeaseId, state: "stopped" }),
+ } }).where(eq(environmentLeases.id, f.lease.id));
+ const runtime = f.input.environmentRuntime;
+ const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+ const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId!));
+ expect(await runtime.resumeRunLease({ environment: environment as never, lease: lease as EnvironmentLease })).toBeNull();
+ expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+ await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
+ expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
+ expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
+ expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
+ expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
+ await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
+ expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
+ expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+ expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
+ }, 30_000);
+
+ it.each(["changed_claim", "missing_stop_intent", "wrong_lease", "wrong_environment"])(
+ "does not reopen a retained source without exact recovery authority: %s", async kind => {
+ const f = await seed();
+ const runtime = f.input.environmentRuntime;
+ const resume = runtime.resumeWorkspaceRecoveryLease.bind(runtime);
+ vi.spyOn(runtime, "resumeWorkspaceRecoveryLease").mockImplementationOnce(async input => {
+ if (kind === "changed_claim") await db.update(environmentLeases).set({ metadata: sql`
+ ${environmentLeases.metadata} || '{"pendingCleanupAttemptId":"other-owner"}'::jsonb`
+ }).where(eq(environmentLeases.id, f.lease.id));
+ if (kind === "missing_stop_intent") await db.update(environmentLeases).set({ metadata: sql`
+ ${environmentLeases.metadata} - 'sandboxStopAndRetain'`
+ }).where(eq(environmentLeases.id, f.lease.id));
+ return resume({ ...input,
+ ...(kind === "wrong_lease" ? { leaseId: randomUUID() } : {}),
+ ...(kind === "wrong_environment" ? { environmentId: randomUUID() } : {}),
+ });
+ });
+ await recoverRetainedWorkspace(f.input);
+ expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("source_binding_invalid");
+ expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+ expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
+ expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+ }, 30_000);
+
it("keeps verified copyback files in the preserved workspace on ordinary Retry", async () => {
const f = await seed();
await recoverRetainedWorkspace(f.input);
diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts
index 5b880d5b65..ec050a5b51 100644
--- a/server/src/services/workspace-recovery.ts
+++ b/server/src/services/workspace-recovery.ts
@@ -169,8 +169,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
}).catch(() => undefined); }, 30_000);
let stopSettled = false;
try {
- const opened = await input.environmentRuntime.resumeRunLease({ environment: environment as unknown as Environment,
- lease: cleanupLease as EnvironmentLease });
+ const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
+ leaseId: cleanupLease.id, environmentId: environment.id, ownership });
if (opened?.providerLeaseId !== lease.providerLeaseId || opened.metadata?.remoteCwd !== remoteCwd)
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
await inspectWorkspaceRecoverySource(db, input);