Skip to content

The open-source app everyone uses to manage agents at work

fix(runtime): continue from saved sources after workspace loss (#16690)

Dotta7c7d2d8d2cfc
diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md
index 3d861bb82a..570c221d0e 100644
--- a/doc/native-workspace-finalization-recovery.md
+++ b/doc/native-workspace-finalization-recovery.md
@@ -359,3 +359,27 @@ A retained source receipt records control-plane evidence; it does not prove the
provider still holds the original bytes or that copy-back succeeded. Operators
must verify and recover the exact source separately. Later successful runs and
completed tasks do not certify recovery of earlier workspace files.
+
+### Continue after confirmed source destruction
+
+When an old export failed and its exact sandbox has a confirmed destruction
+receipt, the task can continue from surviving attachments. This is reconstruction,
+not recovery of the missing edits. The task's recovery notice exposes **Continue
+from saved sources** to board operators with `runtime:manage`. Select the surviving
+attachments, describe the missing work, and record the observed external action
+outcomes. A retained or unverified sandbox must use the existing export or file
+recovery path instead.
+
+`POST /api/issues/{id}/recovery-actions/reconstruct-workspace` records this choice.
+It validates the source run, accepted result, prepared descriptor, exact destroyed
+lease, current task owner and status version. It rejects active copyback owners,
+competing recovery, newer executions, and attachments from another task or company.
+The transaction records attachment hashes and a durable fresh-session delivery
+intent. The old run, result, descriptor and failed finalization remain intact.
+Repeated submissions adopt the original intent. Normal admission rechecks source
+identity, attachments, authority, budgets, holds and task eligibility. It preserves
+the retry count and does not grant a new work window or replay completed actions.
+The agent must verify the selected attachments before rebuilding files.
+Queued successors recheck their exact reconstruction receipt, sources and current
+execution gates when claimed and immediately before provider handoff. A later
+human block cannot be overwritten by automatic checkout.
diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md
index 016f34aef4..5a764e09a7 100644
--- a/maps/agent-access-map/recovery.md
+++ b/maps/agent-access-map/recovery.md
@@ -176,6 +176,7 @@ Review: `baseline`.
API: `present` — representative operation IDs:
- `POST /api/issues/{id}/recovery-actions/retry-workspace-export`.
+- `POST /api/issues/{id}/recovery-actions/reconstruct-workspace`.
- `POST /api/issues/{id}/recovery-actions/resume-workspace-finalization`.
- `POST /api/issues/{id}/recovery-actions/resolve` for retained-source file repair.
- `POST /api/execution-workspaces/{id}/reconcile-branch`.
@@ -184,6 +185,10 @@ Agent access: `restricted` — Board/runtime management APIs; agent reads diagno
performs permitted workspace work.
Policy: Export retry and branch reconciliation require board runtime management.
+Reconstruction also requires board runtime management. It verifies exact source
+destruction and same-task attachment hashes, records lost work and external action
+outcomes, and admits one fresh continuation through normal execution gates. The
+old failed export remains failed. A retained or uncertain source cannot use it.
Retained legacy file-only repair requires the board, exact stopped-source receipts,
and recovered-file evidence. It can record repair for a successful legacy turn
without changing its result or authorizing replay. Queued and running sources
diff --git a/maps/agent-access-map/workspaces.md b/maps/agent-access-map/workspaces.md
index 6d050c32fd..ffaef2f422 100644
--- a/maps/agent-access-map/workspaces.md
+++ b/maps/agent-access-map/workspaces.md
@@ -2,6 +2,14 @@
Product map: [Execution workspaces, services, and files](../feature-map/workspaces.md).
+Lost-source reconstruction adds the board-only
+`POST /api/issues/{id}/recovery-actions/reconstruct-workspace` action. It requires
+`runtime:manage` and ordinary task visibility. Agents cannot authorize it. The
+result is a durable fresh-session admission request, subject to current execution
+gates. It grants no additional tool access, budget, work window or permission to
+repeat completed actions. Attachments must belong to the same task and company.
+The original failed run and lost-export evidence remain unchanged.
+
Review date: 2026-10-12. Source commit: `58e16dbddfc9c8d0cf60867f9d9847f0feda055b`.
Native REST transport reconciled in this change; see the
[shared transport contract](README.md#native-rest-transport).
diff --git a/maps/feature-map/workspaces.md b/maps/feature-map/workspaces.md
index 4bda0d21c3..b1508f6778 100644
--- a/maps/feature-map/workspaces.md
+++ b/maps/feature-map/workspaces.md
@@ -18,6 +18,14 @@ Agent access: [companion](../agent-access-map/workspaces.md).
- `files-git`: inspect workspace files and repository state used for the deliverable.
- `closure`: close/reopen or reconcile through permitted controls without losing work.
+For a historical failed export whose exact sandbox was destroyed, the task recovery
+notice offers **Continue from saved sources** to an authorized board operator.
+Select surviving task attachments, record lost work and external action outcomes,
+then rebuild through fresh-session admission. The old failed run remains failed.
+Retained sandboxes keep their existing export recovery path. Automated coverage:
+[reconstruction](../../server/src/services/native-runtime/native-workspace-reconstruction.test.ts).
+Live staging acceptance remains separate from these regression tests.
+
## How to get to it (user POV)
### `task-workspace`
diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts
index af3387712b..7d9a9799b5 100644
--- a/packages/shared/src/index.ts
+++ b/packages/shared/src/index.ts
@@ -2001,6 +2001,7 @@ export {
resolveIssueRecoveryActionSchema,
isValidExistingBranchName,
retryWorkspaceExportSchema,
+ reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts
index c84678fec6..2a8fb29bbc 100644
--- a/packages/shared/src/types/execution-projection.ts
+++ b/packages/shared/src/types/execution-projection.ts
@@ -14,6 +14,9 @@ export interface ExecutionBlocker {
/** Required workspace files must be recovered before another provider turn. */
workspaceRepairRequired?: boolean;
canRecordWorkspaceRepair?: boolean;
+ canReconstructWorkspace?: boolean;
+ workspaceReconstructionPending?: boolean;
+ workspaceRecoveryStatusVersion?: number;
savedMessageCount?: number;
}
@@ -77,4 +80,12 @@ export interface ExecutionReconciliation {
outcomeEvidence: string;
/** Operator evidence bound to this failed run; required after workspace restore failure. */
workspaceRepairEvidence?: string;
+ workspaceReconstruction?: {
+ schema: "paperclip.workspace-reconstruction.v1";
+ leaseId: string;
+ resultId: string;
+ descriptorSha256: string;
+ missingWork: string;
+ sources: Array<{ attachmentId: string; sha256: string; byteSize: number; filename: string | null }>;
+ };
}
diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts
index 8e224de092..1684fdeaa4 100644
--- a/packages/shared/src/validators/index.ts
+++ b/packages/shared/src/validators/index.ts
@@ -441,6 +441,7 @@ export {
resolveIssueRecoveryActionSchema,
isValidExistingBranchName,
retryWorkspaceExportSchema,
+ reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
diff --git a/packages/shared/src/validators/issue.ts b/packages/shared/src/validators/issue.ts
index 23c0fd7e10..cf398d1f8b 100644
--- a/packages/shared/src/validators/issue.ts
+++ b/packages/shared/src/validators/issue.ts
@@ -573,6 +573,17 @@ export const retryWorkspaceExportSchema = z.object({
repairNote: z.string().trim().min(20).max(12000),
}).strict();
+export const reconstructWorkspaceSchema = z.object({
+ actionId: z.string().guid(),
+ runId: z.string().guid(),
+ expectedStatusVersion: z.number().int().nonnegative(),
+ attachmentIds: z.array(z.string().guid()).min(1).max(20).refine(ids => new Set(ids).size === ids.length),
+ missingWork: z.string().trim().min(20).max(12000),
+ actionOutcome: z.enum(["completed", "not_performed", "mixed"]),
+ outcomeEvidence: z.string().trim().min(20).max(12000),
+ acknowledgeLostWork: z.literal(true),
+}).strict();
+
export const resolveIssueRecoveryActionSchema = z
.object({
workspaceBaseRef: z.object({
diff --git a/server/src/__tests__/satisfied-connection-intents.test.ts b/server/src/__tests__/satisfied-connection-intents.test.ts
index 8c8f4a5330..6b5fd7da05 100644
--- a/server/src/__tests__/satisfied-connection-intents.test.ts
+++ b/server/src/__tests__/satisfied-connection-intents.test.ts
@@ -7,7 +7,7 @@ import { activityLog, agentWakeupRequests, agents, approvals, authUsers, compani
} from "@paperclipai/db";
import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapters/postgres.js";
import { connectionIntentDeliveryService } from "../services/connection-intent-delivery.js";
-import { findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js";
+import { connectionContinuationPendingResponse, findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { materializeNativeInteractionResponses } from "../services/native-runtime/native-interaction-bridge.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
@@ -64,6 +64,28 @@ describePostgres("already available connection requests", () => {
expect((await db.select().from(issueThreadInteractions))[0].status).toBe("pending");
});
+ it.each(["ready", "revoked", "question", "approval"] as const)("checks retired connection cards while a continuation is todo: %s", async state => {
+ const f = await seed();
+ const run = await retry(f);
+ await satisfiedConnectionIntentService(db).sweepPending();
+ expect((await db.select().from(issueThreadInteractions))[0].status).toBe("expired");
+ await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId));
+ if (state === "revoked") await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id));
+ if (state === "question") await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
+ kind: "ask_user_questions", status: "pending", addresseeUserId: "connection-owner", payload: { version: 1, questions: [] } });
+ if (state === "approval") {
+ const [approval] = await db.insert(approvals).values({ companyId: f.companyId, type: "tool_action", status: "pending", payload: {} }).returning();
+ await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId: approval.id });
+ }
+ const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+ expect(await findSatisfiedToolConnection(db, task, f.intent, run.id)).toBeNull();
+ expect(await connectionContinuationPendingResponse(db, task, "connection-owner", run.id))
+ .toBe(state === "ready" ? null : state === "approval" ? "approval" : "interaction");
+ expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId,
+ runId: run.id, retryReasonOverride: "native_safe_replacement", now: new Date() }))
+ .toMatchObject({ allowed: state === "ready" });
+ });
+
it("keeps a retry under another user's identity waiting for its response", async () => {
const f = await seed(); const run = await retry(f);
await db.update(heartbeatRuns).set({ responsibleUserId: "someone-else" }).where(eq(heartbeatRuns.id, run.id));
diff --git a/server/src/modules/run-dispatch/adapters/postgres.ts b/server/src/modules/run-dispatch/adapters/postgres.ts
index 947e7d3386..2807033653 100644
--- a/server/src/modules/run-dispatch/adapters/postgres.ts
+++ b/server/src/modules/run-dispatch/adapters/postgres.ts
@@ -1,5 +1,6 @@
import { hasConversationContinuationPolicy } from "../../../services/conversation-continuation.js";
import { getExecutionBlocker } from "../../../services/execution-blocker.js";
+import { validateReconstructedWorkspaceDispatch } from "../../../services/native-runtime/native-workspace-reconstruction.js";
import { getNativeReviewAssignment } from "../../../services/native-runtime/native-review-participant.js";
import { and, asc, eq, gte, inArray, lte, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
@@ -981,10 +982,27 @@ export function createPostgresRunDispatchAdapter(
};
}
- async function decideCurrentRunStaleness(tx: Db, run: HeartbeatRun, now: Date) {
+ async function decideCurrentRunStaleness(tx: Db, run: HeartbeatRun, now: Date, publications?: ActivityPublication[]) {
const contextSnapshot = parseObject(run.contextSnapshot);
const issueId = readNonEmptyString(contextSnapshot.issueId);
if (!issueId) return { issueId: null, facts: null, executionBlocker: null, decision: { stale: false as const } };
+ if (typeof contextSnapshot.workspaceReconstructionActionId === "string") {
+ try {
+ await validateReconstructedWorkspaceDispatch(tx, run);
+ const current = await loadGateFacts({ runId: run.id, companyId: run.companyId, agentId: run.agentId,
+ responsibleUserId: run.responsibleUserId, conversationContinuation: true, contextSnapshot,
+ scheduledRetryReason: run.scheduledRetryReason, retryReasonOverride: "native_safe_replacement", wakeupRequestId: run.wakeupRequestId,
+ }, now, tx, publications);
+ if (!current.agentFound || !decideScheduledRetryGate(current.facts, now).allowed) throw new Error("reconstruction_execution_not_eligible");
+ }
+ catch {
+ return { issueId, facts: null, executionBlocker: null, decision: { stale: true as const,
+ errorCode: "execution_reconciliation_required" as const,
+ reason: "Workspace reconstruction changed before execution. Inspect the saved sources and current task before continuing.",
+ details: { issueId, recoveryActionId: contextSnapshot.workspaceReconstructionActionId },
+ } };
+ }
+ }
const recovery = await getExecutionBlocker(tx, run.companyId, issueId, { conversationResetCommentId: deriveCommentId(contextSnapshot) });
if (recovery) return { issueId, facts: null, executionBlocker: recovery, decision: { stale: true as const,
errorCode: "execution_reconciliation_required" as const, reason: recovery.nextAction,
@@ -1070,9 +1088,9 @@ export function createPostgresRunDispatchAdapter(
async function cancelStaleQueuedRun(
input: CancelStaleQueuedRunInput,
): Promise<CancelStaleQueuedRunOutcome> {
- const cancelLockedRun = async (tx: Db, run: HeartbeatRun) => {
+ const cancelLockedRun = async (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => {
if (run.status !== input.expectedStatus) return { outcome: "lost_race" as const };
- const { issueId, facts, decision, executionBlocker } = await decideCurrentRunStaleness(tx, run, input.now);
+ const { issueId, facts, decision, executionBlocker } = await decideCurrentRunStaleness(tx, run, input.now, publications);
if (!decision.stale || !issueId) {
if (input.expectedStatus === "queued" && facts?.isInteractionWake) {
// Preserve the authority accepted under the issue/run locks. Later
@@ -1109,7 +1127,7 @@ export function createPostgresRunDispatchAdapter(
async function dispatchResolvedInteractionIfCurrent<T>(
input: DispatchResolvedInteractionInput<T>,
): Promise<DispatchResolvedInteractionOutcome<T>> {
- const dispatchLockedRun = async (tx: Db, run: HeartbeatRun) => {
+ const dispatchLockedRun = async (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => {
if (run.status !== input.expectedStatus || run.resultJson?.cancellation || run.resultJson?.startupCancellation) {
return { dispatched: false as const, cancellation: { outcome: "lost_race" as const } };
}
@@ -1117,6 +1135,7 @@ export function createPostgresRunDispatchAdapter(
tx,
run,
input.now,
+ publications,
);
const decision =
!initialDecision.stale && issueId
diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts
index eacaea637c..6852b42280 100644
--- a/server/src/routes/issues.ts
+++ b/server/src/routes/issues.ts
@@ -19,6 +19,8 @@ import { createIssueReadTiming } from "../services/issue-read-timing.js";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js";
import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js";
+import { reconstructNativeWorkspace, unreconciledNativeExportPredicate } from "../services/native-runtime/native-workspace-reconstruction.js";
+import { reconstructWorkspaceSchema } from "@paperclipai/shared";
import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js";
import { deliverConversationComments, isConversation } from "../services/agent-conversations.js";
import { issueRecoveryActionReadModel } from "../services/issue-recovery-actions.js";
@@ -9501,11 +9503,11 @@ export function issueRoutes(
});
const retainedHolds = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
- sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
- or(
+ or(unreconciledNativeExportPredicate(), and(
+ sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, or(
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
- ),
+ ))),
)).orderBy(desc(issueRecoveryActions.createdAt));
res.json({
active,
@@ -9537,6 +9539,17 @@ export function issueRoutes(
res.status(202).json(receipt);
});
+ router.post("/issues/:id/recovery-actions/reconstruct-workspace", validate(reconstructWorkspaceSchema), async (req, res) => {
+ assertBoard(req);
+ const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found");
+ if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return;
+ const decision = await access.decide({ actor: req.actor, action: "runtime:manage", resource: { type: "company", companyId: issue.companyId } });
+ if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
+ const receipt = await reconstructNativeWorkspace({ db, companyId: issue.companyId, issueId: issue.id,
+ actorId: getActorInfo(req).actorId, request: req.body });
+ res.status(202).json(receipt);
+ });
+
router.post(
"/issues/:id/recovery-actions/resolve",
validate(resolveIssueRecoveryActionSchema),
diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts
index 120312ba87..d51610850f 100644
--- a/server/src/routes/openapi.ts
+++ b/server/src/routes/openapi.ts
@@ -243,6 +243,7 @@ import {
createAcceptedPlanDecompositionSchema,
resolveIssueRecoveryActionSchema,
retryWorkspaceExportSchema,
+ reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
cancelIssueThreadInteractionSchema,
// Secret provider configs and remote import
@@ -10668,6 +10669,15 @@ registerCurrentRoute({
responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict },
});
+registerCurrentRoute({
+ method: "post",
+ path: "/api/issues/{id}/recovery-actions/reconstruct-workspace",
+ tags: ["issues"],
+ summary: "Continue from saved task attachments after confirmed destruction of the original workspace",
+ body: reconstructWorkspaceSchema,
+ responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict },
+});
+
registerCurrentRoute({
method: "post",
path: "/api/issues/{id}/recovery-actions/resolve",
diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts
index 38d615dfbd..2d2ae87c85 100644
--- a/server/src/services/execution-blocker.ts
+++ b/server/src/services/execution-blocker.ts
@@ -8,6 +8,9 @@ import { canRetryStoppedRun } from "./cancelled-native-startup.js";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
+import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js";
+import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
+import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
export async function hasPendingRunCleanup(db: Db, companyId: string, runId: string): Promise<boolean> {
const [state] = await db.execute<{ pending: boolean }>(sql`SELECT
@@ -40,16 +43,18 @@ export async function hasPendingRunCleanup(db: Db, companyId: string, runId: str
export function executionBlockerPredicate() {
return and(
not(conversationRecoveryActionPredicate()!),
- inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
- or(inArray(issueRecoveryActions.status, ["active", "escalated"]),
- sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`),
+ or(unreconciledNativeExportPredicate(), and(
+ inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
+ or(inArray(issueRecoveryActions.status, ["active", "escalated"]),
+ sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`),
+ )),
);
}
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { conversationResetCommentId?: string | null }): Promise<ExecutionBlocker | null> {
const [conversation] = await db.select({ agentId: issues.conversationAgentId,
boundaryId: issues.conversationBoundaryCommentId, status: issues.status,
- assigneeAgentId: issues.assigneeAgentId }).from(issues).where(and(
+ assigneeAgentId: issues.assigneeAgentId, statusVersion: issues.statusVersion }).from(issues).where(and(
eq(issues.companyId, companyId), eq(issues.id, issueId),
)).limit(1);
// Resetting model context cannot repair or recover a workspace. This hold
@@ -59,6 +64,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
eq(issueRecoveryActions.sourceIssueId, issueId),
executionBlockerPredicate(),
or(
+ unreconciledNativeExportPredicate(),
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
),
@@ -94,6 +100,17 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
)).limit(1) : [];
const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace);
+ const reconstructableExport = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"].includes(action.cause);
+ const exportSource = reconstructableExport ? readNativeWorkspaceSyncReference(run?.runnerProfileJson?.nativeWorkspaceSync) : null;
+ const [exportLease] = exportSource ? await db.select().from(environmentLeases).where(and(
+ eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run!.id), eq(environmentLeases.id, exportSource.leaseId),
+ eq(environmentLeases.providerLeaseId, exportSource.providerLeaseId),
+ )).limit(1) : [];
+ const canReconstructWorkspace = Boolean(exportLease && hasRemoteTerminationReceipt(exportLease)
+ && (exportLease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed"
+ && !action.evidence.executionReconciliation && conversation?.status === "blocked" && conversation.assigneeAgentId === run?.agentId);
+ const reconstructionPending = Boolean((action.evidence.executionReconciliation as { workspaceReconstruction?: unknown } | undefined)?.workspaceReconstruction
+ && action.evidence.continuationDelivery === "pending");
const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
@@ -163,12 +180,14 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
// A stopped reviewer can differ from the task owner who receives the work back.
agentId: run?.agentId ?? null,
cause: action.cause,
- nextAction,
+ nextAction: reconstructionPending ? "Continuation from saved sources is queued and waiting for execution checks." : nextAction,
runStatus: run?.status ?? null,
runError: runError?.slice(0, 1024) ?? null,
canContinue: eligibleContinuation && !chatBinding,
canRetry: canRetry && !chatBinding,
...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
savedMessageCount: saved.length,
+ ...(canReconstructWorkspace ? { canReconstructWorkspace, workspaceRecoveryStatusVersion: conversation!.statusVersion } : {}),
+ ...(reconstructionPending ? { workspaceReconstructionPending: true } : {}),
};
}
diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts
index aabb3e1b78..46686f7d7b 100644
--- a/server/src/services/explicit-native-continuation.ts
+++ b/server/src/services/explicit-native-continuation.ts
@@ -144,6 +144,9 @@ export async function admitExplicitNativeContinuation(input: {
if (!actions.length) return null;
const blocker = await getExecutionBlocker(db, companyId, issueId);
if (blocker && blocker.recoveryActionId === null) return null;
+ if (blocker?.workspaceRepairRequired && ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"].includes(blocker.cause)) {
+ return blocked("workspace_reconstruction_required", "Recover the saved files or record reconstruction before continuing. Your message is saved.");
+ }
const [pendingInteraction] = await db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and(
eq(issueThreadInteractions.companyId, companyId), eq(issueThreadInteractions.issueId, issueId),
eq(issueThreadInteractions.status, "pending"),
diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts
index 81632b1b83..474559cd29 100644
--- a/server/src/services/heartbeat.ts
+++ b/server/src/services/heartbeat.ts
@@ -2173,7 +2173,7 @@ export function heartbeatService(
await issuesSvc.checkout(
issueId,
agent.id,
- ["todo", "backlog", "blocked"],
+ typeof context.workspaceReconstructionActionId === "string" ? ["todo"] : ["todo", "backlog", "blocked"],
run.id,
);
context[PAPERCLIP_HARNESS_CHECKOUT_KEY] = true;
@@ -2844,6 +2844,7 @@ export function heartbeatService(
if (
!issueId ||
(!isResolvedInteractionContinuationWakeContext(context) &&
+ typeof context.workspaceReconstructionActionId !== "string" &&
!["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? ""))
) {
return { dispatched: true, resultPromise: dispatch(() => {}) };
diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts
index be493ce9b4..758910670f 100644
--- a/server/src/services/heartbeat/queue.ts
+++ b/server/src/services/heartbeat/queue.ts
@@ -33,6 +33,7 @@ import {
} from "../explicit-native-continuation.js";
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
import { getExecutionBlocker, hasPendingRunCleanup } from "../execution-blocker.js";
+import { validateNativeWorkspaceReconstruction } from "../native-runtime/native-workspace-reconstruction.js";
import { isConversationAdapter } from "../conversation-continuation.js";
import { recordExecutionWait } from "../execution-wait.js";
import {
@@ -3102,6 +3103,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId),
));
+ if (decision.workspaceReconstruction) {
+ await validateNativeWorkspaceReconstruction(tx as unknown as Db, {
+ companyId: issue.companyId, issueId: issue.id, runId: sourceRunId, actionId,
+ });
+ reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
+ enrichedContextSnapshot.workspaceReconstructionActionId = action.id;
+ }
if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) {
if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20)
return { kind: "skipped" as const };
@@ -3170,7 +3178,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// A bound chat request has already rechecked its current principal
// above. Treat its new user message like a board comment, but keep
// failed-run retry actions on their separate exact-request path.
- if (executionBlocker && !(await admitExplicitNativeContinuation({
+ const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
+ && executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
+ if (executionBlocker && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: opts.requestedByActorType, actorId: opts.requestedByActorId,
reason: durableRequest && !failedChatRetry ? "issue_commented" : reason,
@@ -3968,7 +3978,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
});
- if (!explicitContinuation && executionBlocker) return deferBlockedExecution(executionBlocker);
+ if (!explicitContinuation && executionBlocker && !ownsReconstructionContinuation) return deferBlockedExecution(executionBlocker);
if (explicitContinuation) {
enrichedContextSnapshot.forceFreshSession = true;
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
@@ -4013,7 +4023,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// in separate turns, or finalization will deliver it a second time.
const pendingComments =
(executionWaitRequestId || canCoalesceComments) &&
- !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id))
+ (ownsReconstructionContinuation || !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)))
? await tx
.select()
.from(agentWakeupRequests)
@@ -4095,6 +4105,14 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.returning()
.then((rows) => rows[0]);
+ if (ownsReconstructionContinuation) {
+ await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
+ continuationDelivery: "delivered", continuationRunId: newRun.id,
+ })}::jsonb` }).where(and(eq(issueRecoveryActions.companyId, issue.companyId),
+ eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
+ sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
+ }
+
await tx
.update(agentWakeupRequests)
.set({
diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.test.ts b/server/src/services/native-runtime/native-workspace-reconstruction.test.ts
new file mode 100644
index 0000000000..52a65d4b78
--- /dev/null
+++ b/server/src/services/native-runtime/native-workspace-reconstruction.test.ts
@@ -0,0 +1,222 @@
+import { randomUUID } from "node:crypto";
+import express from "express";
+import request from "supertest";
+import { eq, sql } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { activityLog, agents, agentWakeupRequests, assets, authUsers, companies, completionContracts, createDb, environmentLeases, heartbeatRuns, issueAttachments, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
+import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
+import { errorHandler } from "../../middleware/index.js";
+import { issueRoutes } from "../../routes/issues.js";
+import { getExecutionBlocker } from "../execution-blocker.js";
+import { admitExplicitNativeContinuation } from "../explicit-native-continuation.js";
+import { deliverReconciledExecutions } from "../execution-recovery-resolution.js";
+import { heartbeatService } from "../heartbeat.js";
+import { remoteTerminationReceipt } from "../remote-execution-termination.js";
+import { reconstructNativeWorkspace } from "./native-workspace-reconstruction.js";
+import { createRunDispatch } from "../../modules/run-dispatch/index.js";
+import { drainHeartbeatRunsToQuiescence } from "../../__tests__/helpers/drain-heartbeat-runs.js";
+const providerExecute = vi.hoisted(() => vi.fn(async () => ({ exitCode: 0, signal: null, timedOut: false,
+ errorMessage: null, summary: "Bounded reconstruction completed.", provider: "test", model: "test" })));
+vi.mock("../../adapters/index.js", async () => ({
+ ...await vi.importActual<typeof import("../../adapters/index.js")>("../../adapters/index.js"),
+ getServerAdapter: () => ({ supportsLocalAgentJwt: false, execute: providerExecute }),
+}));
+
+describe("reconstruction after confirmed native workspace loss", () => {
+ let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+ let db: ReturnType<typeof createDb>;
+ let runningHeartbeat: ReturnType<typeof heartbeatService> | null = null;
+ beforeAll(async () => {
+ temporary = await startEmbeddedPostgresTestDatabase("workspace-reconstruction-");
+ db = createDb(temporary.connectionString);
+ }, 30_000);
+ afterAll(async () => { await temporary.cleanup(); });
+ afterEach(async () => {
+ if (runningHeartbeat) await drainHeartbeatRunsToQuiescence(db, runningHeartbeat);
+ runningHeartbeat = null;
+ providerExecute.mockClear();
+ await db.transaction(async tx => {
+ await tx.execute(sql`set local client_min_messages = warning`);
+ await tx.execute(sql`truncate companies cascade`);
+ }); });
+
+ async function seed() {
+ const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), resultId = randomUUID(),
+ contractId = randomUUID(), leaseId = randomUUID(), actionId = randomUUID(), providerLeaseId = randomUUID(), userId = randomUUID(), attachmentId = randomUUID(), assetId = randomUUID();
+ await db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.test`, emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
+ await db.insert(companies).values({ id: companyId, name: "Reconstruction", issuePrefix: `REC${companyId.replaceAll("-", "").slice(0, 6)}`, defaultResponsibleUserId: userId, budgetMonthlyCents: 10000 });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Worker", status: "idle", adapterType: "paperclip_runner", budgetMonthlyCents: 5000, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
+ await db.insert(issues).values({ id: issueId, companyId, title: "Finish bounded work", status: "blocked", assigneeAgentId: agentId });
+ await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { objective: "Finish bounded work" }, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "failed", runtimeMode: "native", nativeIssueId: issueId, nativePhase: "terminal_failure", completionContractId: contractId,
+ createdAt: new Date("2026-05-01T00:00:00Z"), finishedAt: new Date("2026-05-01T01:30:00Z"), responsibleUserId: userId, scheduledRetryAttempt: 2,
+ contextSnapshot: { issueId, originalWorkDeadline: "2026-05-01T01:30:00Z" }, errorCode: "native_session_interrupted", processPid: process.pid,
+ runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId, providerLeaseId, remoteCwd: "/work", resourceDisposition: "destroy" } } });
+ await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted", resultJson: { work: "Unfinished local edits; no published PR" }, canonicalSha256: resultId });
+ const cause = "native_workspace_sync_out_unrecoverable";
+ await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure", resultId, failureCode: cause });
+ const identity = { id: leaseId, companyId, heartbeatRunId: runId, provider: "daytona", providerLeaseId };
+ await db.insert(environmentLeases).values({ ...identity, issueId, status: "expired", releasedAt: new Date(), cleanupStatus: "success", metadata: { remoteExecutionTermination: remoteTerminationReceipt(identity, { providerLeaseId, state: "destroyed" }) } });
+ await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: agentId, cause, fingerprint: runId, evidence: { runId }, nextAction: "Recover saved files", status: "resolved", outcome: "restored", resolutionNote: "new_source_execution_path" });
+ await db.insert(assets).values({ id: assetId, companyId, provider: "local", objectKey: assetId, contentType: "text/plain", byteSize: 100, sha256: "c".repeat(64), originalFilename: "saved-baseline.ts" });
+ await db.insert(issueAttachments).values({ id: attachmentId, companyId, issueId, assetId });
+ const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
+ const body = { actionId, runId, expectedStatusVersion: issue.statusVersion, attachmentIds: [attachmentId], missingWork: "The unexported patch is lost. Rebuild only from this earlier verified baseline.", actionOutcome: "not_performed" as const, outcomeEvidence: "The saved execution receipt and remote repository show no published PR or submitted external change.", acknowledgeLostWork: true as const };
+ const input = { db, companyId, issueId, actorId: userId, request: body };
+ return { ...input, body, agentId, leaseId, resultId, runId, actionId, providerLeaseId, attachmentId, assetId };
+ }
+ function app(actor: unknown = { type: "board", source: "local_implicit" }) {
+ const app = express(); app.use(express.json());
+ app.use((req, _res, next) => { (req as unknown as { actor: unknown }).actor = actor; next(); });
+ app.use("/api", issueRoutes(db, {} as never)); app.use(errorHandler); return app;
+ }
+ it("keeps a falsely resolved export visible and prevents conversation reset from hiding it", async () => {
+ const f = await seed();
+ expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true, canReconstructWorkspace: true });
+ const commentId = randomUUID();
+ await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: f.actorId, body: "/new" });
+ await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: f.actorId, conversationState: "active" }).where(eq(issues.id, f.issueId));
+ expect(await getExecutionBlocker(db, f.companyId, f.issueId, { conversationResetCommentId: commentId })).not.toBeNull();
+ expect(await admitExplicitNativeContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
+ actorType: "user", actorId: f.actorId, reason: "issue_commented", commentId, successorRunId: randomUUID() })).toBeNull();
+ const response = await request(app()).get(`/api/issues/${f.issueId}/recovery-actions`).expect(200);
+ expect(response.body.actions).toEqual(expect.arrayContaining([expect.objectContaining({ id: f.actionId })]));
+ });
+ it("records partial sources atomically, preserves failed history and budgets, and deduplicates concurrent submissions", async () => {
+ const f = await seed();
+ const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+ await Promise.all([reconstructNativeWorkspace(f), reconstructNativeWorkspace(f)]);
+ expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before);
+ expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
+ expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].schemaStatus).toBe("accepted");
+ const [record] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
+ expect(record).toMatchObject({ outcome: "handed_back", evidence: { continuationDelivery: "pending", executionReconciliation: { workspaceReconstruction: { missingWork: f.body.missingWork, sources: [{ attachmentId: f.attachmentId, sha256: "c".repeat(64) }] } } } });
+ expect(record.evidence.executionReconciliation).not.toHaveProperty("workspaceRepairEvidence");
+ expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
+ expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issueId))).toHaveLength(1);
+ expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
+ expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0].budgetMonthlyCents).toBe(10000);
+ expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0].budgetMonthlyCents).toBe(5000);
+ });
+ it.each(["retained", "receipt", "lease", "owner", "coordinator", "copyback", "newer", "status", "version", "attachment", "result"] as const)("rejects stale or unsafe reconstruction: %s", async kind => {
+ const f = await seed();
+ const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
+ if (kind === "retained") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: f.providerLeaseId, state: "stopped" }) } }).where(eq(environmentLeases.id, f.leaseId));
+ if (kind === "receipt") await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...(lease.metadata!.remoteExecutionTermination as object), runId: randomUUID() } } }).where(eq(environmentLeases.id, f.leaseId));
+ if (kind === "lease") await db.update(environmentLeases).set({ status: "pending_cleanup" }).where(eq(environmentLeases.id, f.leaseId));
+ if (kind === "owner") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId));
+ if (kind === "coordinator") await db.update(nativeRunFinalizations).set({ leaseOwner: "other-worker" }).where(eq(nativeRunFinalizations.runId, f.runId));
+ if (kind === "copyback") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { token: "other-worker" } } }).where(eq(heartbeatRuns.id, f.runId));
+ if (kind === "newer") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, contextSnapshot: { issueId: f.issueId }, status: "succeeded" });
+ if (kind === "status") await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
+ if (kind === "version") f.request.expectedStatusVersion += 1;
+ if (kind === "attachment") f.request.attachmentIds = [(await seed()).attachmentId];
+ if (kind === "result") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.id, f.resultId));
+ await expect(reconstructNativeWorkspace(f)).rejects.toThrow();
+ expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issueId))).toHaveLength(0);
+ expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence).not.toHaveProperty("executionReconciliation");
+ });
+ it("exposes board recovery through the assembled API and rejects missing loss acknowledgement", async () => {
+ const f = await seed();
+ await request(app()).post(`/api/issues/${f.issueId}/recovery-actions/reconstruct-workspace`).send({ ...f.body, acknowledgeLostWork: false }).expect(400);
+ await request(app({ type: "agent", companyId: f.companyId, agentId: f.agentId, runId: f.runId })).post(`/api/issues/${f.issueId}/recovery-actions/reconstruct-workspace`).send(f.body).expect(403);
+ await request(app()).post(`/api/issues/${f.issueId}/recovery-actions/reconstruct-workspace`).send(f.body).expect(202);
+ });
+ it.each(["status", "attachment", "paused"] as const)("keeps delivery pending when %s changes after reconstruction is recorded", async kind => {
+ const f = await seed();
+ await reconstructNativeWorkspace(f);
+ if (kind === "status") await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issueId));
+ if (kind === "attachment") await db.delete(issueAttachments).where(eq(issueAttachments.id, f.attachmentId));
+ if (kind === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
+ const heartbeat = heartbeatService(db, { runtimeEnv: {} });
+ await deliverReconciledExecutions(db, heartbeat.wakeup);
+ expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agentId))).toHaveLength(1);
+ expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence.continuationDelivery).toBe("pending");
+ });
+ it("admits one fresh continuation through public heartbeat, keeps retry budget, and adopts a lost acknowledgement", async () => {
+ const f = await seed();
+ await reconstructNativeWorkspace(f);
+ await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
+ const heartbeat = heartbeatService(db, { runtimeEnv: {} });
+ const wake = vi.fn(heartbeat.wakeup);
+ let successorId: string | undefined;
+ let admissionError: unknown;
+ await deliverReconciledExecutions(db, async (...args) => {
+ let successor;
+ try { successor = await wake(...args); } catch (error) { admissionError = error; throw error; }
+ expect(successor).not.toBeNull(); successorId = successor!.id;
+ expect(successor).toMatchObject({ retryOfRunId: f.runId, scheduledRetryAttempt: 2, contextSnapshot: { forceFreshSession: true, previousRunId: f.runId } });
+ throw new Error("lost acknowledgement");
+ });
+ if (admissionError) throw admissionError;
+ const queued = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
+ expect(successorId, JSON.stringify(queued.map(row => ({ status: row.status, reason: row.reason, payload: row.payload })))).toBeDefined();
+ await deliverReconciledExecutions(db, wake);
+ expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, `execution-reconciliation:${f.actionId}`))).toHaveLength(1);
+ expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: successorId });
+ expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+ });
+ it.each(["unchanged", "blocked", "attachment", "paused", "forged_successor"] as const)("rechecks a queued reconstruction at the final provider handoff: %s", async kind => {
+ const f = await seed();
+ await reconstructNativeWorkspace(f);
+ await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
+ const heartbeat = heartbeatService(db, { runtimeEnv: {} });
+ await deliverReconciledExecutions(db, heartbeat.wakeup);
+ const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
+ const successorId = String(action.evidence.continuationRunId);
+ expect(successorId).not.toBe("undefined");
+ await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, successorId));
+ await db.update(issues).set({ status: "in_progress", executionRunId: successorId }).where(eq(issues.id, f.issueId));
+ if (kind === "blocked") await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issueId));
+ if (kind === "attachment") await db.delete(issueAttachments).where(eq(issueAttachments.id, f.attachmentId));
+ if (kind === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
+ if (kind === "forged_successor") await db.update(issueRecoveryActions).set({ evidence: { ...action.evidence, continuationRunId: randomUUID() } }).where(eq(issueRecoveryActions.id, f.actionId));
+ const dispatch = vi.fn(async () => "provider entered");
+ const outcome = await createRunDispatch(db).dispatchResolvedInteractionIfCurrent({ companyId: f.companyId,
+ runId: successorId, expectedStatus: "running", dispatch });
+ expect(outcome.dispatched).toBe(kind === "unchanged");
+ expect(dispatch).toHaveBeenCalledTimes(kind === "unchanged" ? 1 : 0);
+ if (!outcome.dispatched) expect(outcome.cancellation).toMatchObject({ outcome: "cancelled" });
+ });
+ it.each([false, true])("runs the assembled heartbeat without overwriting a block after claim: %s", async blockAfterClaim => {
+ const f = await seed();
+ await db.update(agents).set({ adapterType: "codex_local" }).where(eq(agents.id, f.agentId));
+ await reconstructNativeWorkspace(f);
+ const [busy] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }).returning();
+ const heartbeat = runningHeartbeat = heartbeatService(db, { runtimeEnv: { PAPERCLIP_IN_WORKTREE: "false" } });
+ await deliverReconciledExecutions(db, heartbeat.wakeup);
+ const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
+ const successorId = String(action.evidence.continuationRunId);
+ expect(successorId).not.toBe("undefined");
+ if (blockAfterClaim) await db.execute(sql.raw(`
+ create or replace function block_reconstruction_after_claim() returns trigger as $$
+ begin
+ if NEW.id = '${successorId}'::uuid and NEW.status = 'running' then
+ update issues set status = 'blocked' where id = '${f.issueId}'::uuid;
+ end if;
+ return NEW;
+ end; $$ language plpgsql;
+ create trigger block_reconstruction_after_claim after update of status on heartbeat_runs
+ for each row execute function block_reconstruction_after_claim();
+ `));
+ try {
+ await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, busy.id));
+ await heartbeat.resumeQueuedRuns();
+ const deadline = Date.now() + 15000;
+ let current;
+ do {
+ [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successorId));
+ if (["succeeded", "failed", "cancelled"].includes(current.status)) break;
+ await new Promise(resolve => setTimeout(resolve, 25));
+ } while (Date.now() < deadline);
+ expect(providerExecute).toHaveBeenCalledTimes(blockAfterClaim ? 0 : 1);
+ if (blockAfterClaim) {
+ expect(current).toMatchObject({ status: "cancelled", errorCode: "execution_reconciliation_required" });
+ expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("blocked");
+ } else expect(current.status).toBe("succeeded");
+ } finally {
+ if (blockAfterClaim) await db.execute(sql`drop trigger block_reconstruction_after_claim on heartbeat_runs; drop function block_reconstruction_after_claim()`);
+ await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
+ }
+ }, 30_000);
+});
diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.ts b/server/src/services/native-runtime/native-workspace-reconstruction.ts
new file mode 100644
index 0000000000..a238dd95b2
--- /dev/null
+++ b/server/src/services/native-runtime/native-workspace-reconstruction.ts
@@ -0,0 +1,152 @@
+import { and, eq, gt, inArray, ne, or, sql } from "drizzle-orm";
+import { assets, environmentLeases, heartbeatRuns, issueAttachments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
+import { reconstructWorkspaceSchema, type ExecutionReconciliation } from "@paperclipai/shared";
+import type { z } from "zod";
+import { conflict } from "../../errors.js";
+import { persistActivity, publishActivity } from "../activity-log.js";
+import { markExecutionReconciliation } from "../execution-recovery-resolution.js";
+import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
+import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
+import { nativeSha256 } from "./canonical.js";
+
+const causes = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
+const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
+const changed = () => conflict("The lost workspace recovery changed. Refresh the task before continuing.", { code: "workspace_reconstruction_stale" });
+
+export function unreconciledNativeExportPredicate() {
+ return and(
+ inArray(issueRecoveryActions.cause, causes),
+ or(and(
+ or(inArray(issueRecoveryActions.status, ["active", "escalated"]), and(
+ eq(issueRecoveryActions.status, "resolved"), eq(issueRecoveryActions.resolutionNote, "new_source_execution_path"),
+ )),
+ sql`${issueRecoveryActions.evidence}->'executionReconciliation'->'workspaceReconstruction' is null`,
+ ), and(eq(issueRecoveryActions.status, "resolved"),
+ sql`${issueRecoveryActions.evidence}->'executionReconciliation'->'workspaceReconstruction'->>'schema' = 'paperclip.workspace-reconstruction.v1'`,
+ sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
+ )),
+ sql`exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
+ and ${heartbeatRuns.nativeIssueId} = ${issueRecoveryActions.sourceIssueId}
+ and ${heartbeatRuns.id}::text = ${issueRecoveryActions.evidence}->>'runId'
+ and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
+ and ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared')`,
+ );
+}
+
+export async function validateNativeWorkspaceReconstruction(db: Db, input: {
+ companyId: string; issueId: string; runId: string; actionId: string;
+ successorRunId?: string;
+}) {
+ const [issue] = await db.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
+ const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
+ const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))).for("update");
+ const [action] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update");
+ if (!issue || !run || !coordinator || !action || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id
+ || run.status !== "failed" || run.nativePhase !== "terminal_failure" || coordinator.issueId !== issue.id
+ || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner || coordinator.failureDetail?.successorRunId
+ || object(run.runnerProfileJson).nativeWorkspaceFinalizationOwner
+ || action.kind !== "active_run_watchdog" || action.ownerType !== "board" || !causes.includes(action.cause)
+ || action.evidence.runId !== run.id || coordinator.failureCode !== action.cause
+ || !["active", "escalated", "resolved"].includes(action.status)
+ || (action.status === "resolved" && action.resolutionNote !== "new_source_execution_path" && !object(action.evidence.executionReconciliation).workspaceReconstruction)
+ || issue.assigneeAgentId !== run.agentId || action.returnOwnerAgentId !== run.agentId
+ || !(input.successorRunId ? ["todo", "in_progress"] : ["blocked", "todo"]).includes(issue.status)
+ || (issue.executionRunId && issue.executionRunId !== input.successorRunId)
+ || (issue.checkoutRunId && issue.checkoutRunId !== input.successorRunId)) throw changed();
+ const reference = readNativeWorkspaceSyncReference(object(run.runnerProfileJson).nativeWorkspaceSync);
+ if (!reference || reference.state !== "prepared") throw changed();
+ const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, run.id))).for("update");
+ const lease = leases.find(row => row.id === reference.leaseId);
+ if (!lease || lease.issueId !== issue.id || lease.providerLeaseId !== reference.providerLeaseId
+ || !leases.every(hasRemoteTerminationReceipt) || object(lease.metadata?.remoteExecutionTermination).state !== "destroyed"
+ || leases.some(row => row.metadata?.pendingCleanupInFlight === true)) {
+ throw conflict("Paperclip has not confirmed destruction of the original workspace. Recover its files or retry its export first.", { code: "workspace_reconstruction_source_not_destroyed" });
+ }
+ const [otherLease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
+ ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
+ inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]),
+ )).limit(1);
+ const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
+ eq(heartbeatRuns.companyId, input.companyId), ne(heartbeatRuns.id, run.id),
+ input.successorRunId ? ne(heartbeatRuns.id, input.successorRunId) : undefined,
+ or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
+ or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])),
+ )).limit(1);
+ const [otherAction] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
+ eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
+ ne(issueRecoveryActions.id, action.id), inArray(issueRecoveryActions.status, ["active", "escalated"]),
+ )).limit(1);
+ const [result] = await db.select().from(nativeRunResults).where(and(
+ eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, issue.id), eq(nativeRunResults.runId, run.id),
+ eq(nativeRunResults.id, coordinator.resultId ?? "00000000-0000-0000-0000-000000000000"),
+ eq(nativeRunResults.completionContractId, run.completionContractId ?? "00000000-0000-0000-0000-000000000000"), eq(nativeRunResults.schemaStatus, "accepted"),
+ )).limit(1);
+ if (otherLease || newer || otherAction || !result) throw changed();
+ const decision = action.evidence.executionReconciliation as ExecutionReconciliation | undefined;
+ if (decision?.workspaceReconstruction) {
+ const reconstruction = decision.workspaceReconstruction;
+ if ((!input.successorRunId && issue.status !== "todo") || decision.runId !== run.id || reconstruction.schema !== "paperclip.workspace-reconstruction.v1"
+ || (input.successorRunId && (action.evidence.continuationRunId !== input.successorRunId || action.evidence.continuationDelivery !== "delivered"))
+ || reconstruction.leaseId !== lease.id || reconstruction.resultId !== result.id
+ || reconstruction.descriptorSha256 !== reference.descriptorSha256 || !reconstruction.sources?.length) throw changed();
+ const files = await db.select({ attachmentId: issueAttachments.id, sha256: assets.sha256 }).from(issueAttachments)
+ .innerJoin(assets, and(eq(assets.id, issueAttachments.assetId), eq(assets.companyId, input.companyId)))
+ .where(and(eq(issueAttachments.companyId, input.companyId), eq(issueAttachments.issueId, input.issueId),
+ inArray(issueAttachments.id, reconstruction.sources.map(source => source.attachmentId)))).for("share");
+ if (files.length !== reconstruction.sources.length || reconstruction.sources.some(source => !files.some(file => file.attachmentId === source.attachmentId && file.sha256 === source.sha256))) throw changed();
+ }
+ return { issue, run, action, reference, lease, result };
+}
+
+export async function validateReconstructedWorkspaceDispatch(db: Db, run: typeof heartbeatRuns.$inferSelect) {
+ const context = object(run.contextSnapshot);
+ if (typeof context.workspaceReconstructionActionId !== "string") return;
+ const actionId = context.workspaceReconstructionActionId;
+ if (typeof context.issueId !== "string" || typeof context.previousRunId !== "string"
+ || context.source !== "execution.reconciled" || context.recoveryActionId !== actionId || context.forceFreshSession !== true
+ || run.retryOfRunId !== context.previousRunId) throw changed();
+ const source = await validateNativeWorkspaceReconstruction(db, { companyId: run.companyId, issueId: context.issueId,
+ runId: context.previousRunId, actionId, successorRunId: run.id });
+ if (source.run.agentId !== run.agentId || !object(source.action.evidence.executionReconciliation).workspaceReconstruction) throw changed();
+}
+
+export async function reconstructNativeWorkspace(input: {
+ db: Db; companyId: string; issueId: string; actorId: string;
+ request: z.infer<typeof reconstructWorkspaceSchema>;
+}) {
+ const request = reconstructWorkspaceSchema.parse(input.request);
+ const requestSha256 = nativeSha256(request);
+ const outcome = await input.db.transaction(async tx => {
+ const db = tx as unknown as Db;
+ const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
+ const [prior] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, request.actionId)));
+ if (prior?.evidence.workspaceReconstructionRequest === requestSha256) return { replayed: true, publication: null };
+ if (!issue || issue.status !== "blocked" || issue.statusVersion !== request.expectedStatusVersion || prior?.evidence.executionReconciliation) throw changed();
+ const source = await validateNativeWorkspaceReconstruction(db, { ...input, ...request });
+ const files = await tx.select({ attachmentId: issueAttachments.id, sha256: assets.sha256, byteSize: assets.byteSize, filename: assets.originalFilename })
+ .from(issueAttachments).innerJoin(assets, and(eq(assets.id, issueAttachments.assetId), eq(assets.companyId, input.companyId)))
+ .where(and(eq(issueAttachments.companyId, input.companyId), eq(issueAttachments.issueId, input.issueId), inArray(issueAttachments.id, request.attachmentIds)));
+ if (files.length !== request.attachmentIds.length || files.some(file => !/^[a-f0-9]{64}$/.test(file.sha256) || file.byteSize <= 0)) {
+ throw conflict("Choose saved attachments from this task before reconstructing its workspace.", { code: "workspace_reconstruction_sources_invalid" });
+ }
+ const decision: ExecutionReconciliation = { runId: request.runId, providerStopped: true,
+ actionOutcome: request.actionOutcome, outcomeEvidence: request.outcomeEvidence,
+ workspaceReconstruction: { schema: "paperclip.workspace-reconstruction.v1", leaseId: source.lease.id,
+ resultId: source.result.id, descriptorSha256: source.reference.descriptorSha256, missingWork: request.missingWork, sources: files } };
+ await markExecutionReconciliation(db, source.action, decision, input.actorId);
+ const now = new Date();
+ await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "handed_back", resolvedAt: now, updatedAt: now,
+ resolutionNote: "workspace_reconstruction_authorized", nextAction: "Continue from saved sources. The original unexported edits are lost; verify the selected attachments before rebuilding. Do not repeat completed external actions or reset the original work limits.",
+ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ workspaceReconstructionRequest: requestSha256 })}::jsonb`,
+ wakePolicy: null, monitorPolicy: null,
+ }).where(eq(issueRecoveryActions.id, source.action.id));
+ await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
+ blockedOwnerNotifiedAt: null, updatedAt: now }).where(eq(issues.id, issue.id));
+ const publication = await persistActivity(db, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
+ action: "issue.workspace_reconstruction_authorized", entityType: "issue", entityId: input.issueId, runId: request.runId,
+ details: { recoveryActionId: request.actionId, ...decision } });
+ return { replayed: false, publication };
+ });
+ if (outcome.publication) publishActivity(outcome.publication.publication);
+ return { runId: request.runId, actionId: request.actionId, status: "queued" as const, replayed: outcome.replayed };
+}
diff --git a/server/src/services/satisfied-connection-intents.ts b/server/src/services/satisfied-connection-intents.ts
index 49d295ae97..f6a3eb39ef 100644
--- a/server/src/services/satisfied-connection-intents.ts
+++ b/server/src/services/satisfied-connection-intents.ts
@@ -11,10 +11,10 @@ type Task = Pick<typeof issues.$inferSelect, "id" | "companyId" | "projectId" |
type Intent = Pick<typeof issueThreadInteractions.$inferSelect, "id" | "companyId" | "issueId" | "kind" | "status" | "payload" | "addresseeUserId"> & Partial<Pick<typeof issueThreadInteractions.$inferSelect, "sourceRunId">>;
/** Verify existing access only. This never installs tools, grants access, or adopts an identity. */
-export async function findSatisfiedToolConnection(db: Db, task: Task, intent: Intent, policyRunId?: string | null) {
+async function findAvailableToolConnection(db: Db, task: Task, intent: Intent, policyRunId?: string | null) {
if (intent.kind !== "connection_intent" || intent.status !== "pending"
|| intent.companyId !== task.companyId || intent.issueId !== task.id
- || task.assigneeUserId || !["in_progress", "in_review"].includes(task.status)) return null;
+ || task.assigneeUserId) return null;
const parsed = connectionIntentPayloadSchema.safeParse(intent.payload);
if (!parsed.success) return null;
const payload = parsed.data;
@@ -42,6 +42,11 @@ export async function findSatisfiedToolConnection(db: Db, task: Task, intent: In
return access.allowedCount + access.askFirstCount > 0 ? connection : null;
}
+export async function findSatisfiedToolConnection(db: Db, task: Task, intent: Intent, policyRunId?: string | null) {
+ if (!["in_progress", "in_review"].includes(task.status)) return null;
+ return findAvailableToolConnection(db, task, intent, policyRunId);
+}
+
/** Recheck both live requests and system-retired cards against the next run. */
export async function connectionContinuationPendingResponse(db: Db, task: Task, responsibleUserId: string | null,
policyRunId?: string | null): Promise<"interaction" | "approval" | null> {
@@ -55,7 +60,7 @@ export async function connectionContinuationPendingResponse(db: Db, task: Task,
if (interaction.status !== "pending" && !retired) continue;
if (retired && (interaction.payload as { requestingAgentId?: string }).requestingAgentId !== task.assigneeAgentId) continue;
if (interaction.addresseeUserId !== responsibleUserId) return "interaction";
- const connection = await findSatisfiedToolConnection(db, task, { ...interaction, status: "pending" }, policyRunId);
+ const connection = await findAvailableToolConnection(db, task, { ...interaction, status: "pending" }, policyRunId);
// Historical system retirements describe a service need, not a permanent
// binding to an account the user may later replace. Delivery and native
// response projection still validate their own exact recorded connection.
diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts
index 40881632c3..af057efd11 100644
--- a/ui/src/api/issues.ts
+++ b/ui/src/api/issues.ts
@@ -287,6 +287,9 @@ export const issuesApi = {
api.post<{ runId: string; resultId: string; status: "queued" }>(`/issues/${id}/recovery-actions/resume-workspace-finalization`, data),
retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) =>
api.post<{ runId: string; resultId: string; leaseId: string; status: "queued" }>(`/issues/${id}/recovery-actions/retry-workspace-export`, data),
+ reconstructWorkspace: (id: string, data: { actionId: string; runId: string; expectedStatusVersion: number; attachmentIds: string[];
+ missingWork: string; actionOutcome: "completed" | "not_performed" | "mixed"; outcomeEvidence: string; acknowledgeLostWork: true }) =>
+ api.post<{ runId: string; actionId: string; status: "queued"; replayed: boolean }>(`/issues/${id}/recovery-actions/reconstruct-workspace`, data),
recoveryActions: (id: string) =>
api.get<{ active: IssueRecoveryAction | null; actions: IssueRecoveryAction[] }>(`/issues/${id}/recovery-actions`),
resolveRecoveryAction: (
diff --git a/ui/src/components/ExecutionBlockerNotice.tsx b/ui/src/components/ExecutionBlockerNotice.tsx
index f6f82de177..676d3c347f 100644
--- a/ui/src/components/ExecutionBlockerNotice.tsx
+++ b/ui/src/components/ExecutionBlockerNotice.tsx
@@ -1,6 +1,7 @@
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import type { ExecutionBlocker, Issue } from "@paperclipai/shared";
import { WorkspaceFileRecovery } from "./WorkspaceFileRecovery";
+import { WorkspaceReconstructionRecovery } from "./WorkspaceReconstructionRecovery";
import { agentsApi } from "../api/agents";
import { activityApi } from "../api/activity";
import { issuesApi } from "../api/issues";
@@ -50,15 +51,16 @@ export function ExecutionBlockerNotice({ companyId, issueId, issueStatus, canMan
return (
<div role="status" aria-label="Task recovery" className={cn("task-context-notice flex flex-wrap items-center justify-between gap-2", className)}>
<div className="min-w-0 flex-1 break-words">
- <p>{modelRejected ? "Model unavailable." : "Recovery needed."}{blocker.runError ? ` ${blocker.runError}` : ""}</p>
+ <p>{blocker.workspaceReconstructionPending ? "Reconstruction queued." : modelRejected ? "Model unavailable." : "Recovery needed."}{blocker.runError && !blocker.workspaceReconstructionPending ? ` ${blocker.runError}` : ""}</p>
{modelRejected && <p>Choose a supported model or clear the task's model override, then retry.</p>}
{blocker.workspaceRepairRequired ? (
<>
{/* The stored next action is an operator runbook step; lead with what happened. This hold also covers
unsafe archives and local runs, so do not claim retries or a retained sandbox here. */}
<p>
- Paperclip could not restore the agent's workspace files after its last run. A board operator must
- recover the files before the agent can continue.
+ {blocker.workspaceReconstructionPending ? blocker.nextAction : blocker.canReconstructWorkspace
+ ? "The workspace was destroyed before its edits were saved. A board operator can continue from surviving source attachments."
+ : "Paperclip could not restore the agent's workspace files after its last run. A board operator must recover the files before the agent can continue."}
</p>
<details>
<summary className="cursor-pointer">Technical details</summary>
@@ -94,6 +96,8 @@ export function ExecutionBlockerNotice({ companyId, issueId, issueStatus, canMan
{recoveryError && <p role="alert" className="w-full text-destructive">Unable to load recovery details: {recoveryError.message}</p>}
{issueStatus && <WorkspaceFileRecovery key={`${blocker.recoveryActionId}:${blocker.runId}`} issueId={issueId}
issueStatus={issueStatus} blocker={blocker} canManage={canManage} onRecorded={onRetried} />}
+ {blocker.canReconstructWorkspace && <WorkspaceReconstructionRecovery key={`reconstruct:${blocker.recoveryActionId}:${blocker.runId}`}
+ issueId={issueId} blocker={blocker} canManage={canManage} onRecorded={onRetried} />}
</div>
);
}
diff --git a/ui/src/components/WorkspaceReconstructionRecovery.test.tsx b/ui/src/components/WorkspaceReconstructionRecovery.test.tsx
new file mode 100644
index 0000000000..eb025f2f8a
--- /dev/null
+++ b/ui/src/components/WorkspaceReconstructionRecovery.test.tsx
@@ -0,0 +1,74 @@
+// @vitest-environment jsdom
+import { act } from "react";
+import { createRoot, type Root } from "react-dom/client";
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { WorkspaceReconstructionRecovery } from "./WorkspaceReconstructionRecovery";
+import { queryKeys } from "../lib/queryKeys";
+const api = vi.hoisted(() => ({ listAttachments: vi.fn(), reconstructWorkspace: vi.fn() }));
+vi.mock("../api/issues", () => ({ issuesApi: api }));
+
+describe("rebuild a destroyed workspace from partial sources", () => {
+ let root: Root, container: HTMLDivElement, client: QueryClient;
+ const onRecorded = vi.fn();
+ beforeEach(() => {
+ vi.clearAllMocks();
+ api.listAttachments.mockResolvedValue([{ id: "source", originalFilename: "baseline.ts", byteSize: 100 }]);
+ api.reconstructWorkspace.mockResolvedValue({ status: "queued" });
+ container = document.createElement("div"); document.body.append(container); root = createRoot(container);
+ client = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } });
+ });
+ afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); });
+ async function mount(canManage = true) {
+ await act(async () => root.render(<QueryClientProvider client={client}><WorkspaceReconstructionRecovery issueId="task" canManage={canManage}
+ blocker={{ recoveryActionId: "recovery", runId: "lost-run", agentId: "agent", cause: "native_workspace_sync_out_unrecoverable", nextAction: "", workspaceRecoveryStatusVersion: 3 }} onRecorded={onRecorded} />
+ </QueryClientProvider>));
+ await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
+ }
+ async function fill() {
+ await act(async () => {
+ for (const area of container.querySelectorAll("textarea")) {
+ Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(area, "Verified source and receipts show what must be rebuilt without repeating published work.");
+ area.dispatchEvent(new Event("input", { bubbles: true }));
+ }
+ const checkboxes = container.querySelectorAll<HTMLButtonElement>('[role="checkbox"]');
+ checkboxes[0].click(); checkboxes[1].click();
+ });
+ }
+ it("requires surviving sources, lost-work acknowledgement and outcome evidence without claiming original-file recovery", async () => {
+ await mount();
+ const button = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Rebuild and continue")!;
+ expect(button.disabled).toBe(true);
+ expect(container.textContent).toContain("missing edits have not been recovered");
+ expect(container.textContent).toContain("Existing work limits still apply");
+ await fill(); expect(button.disabled).toBe(false);
+ await act(async () => button.click());
+ expect(api.reconstructWorkspace).toHaveBeenCalledWith("task", expect.objectContaining({ runId: "lost-run", expectedStatusVersion: 3, attachmentIds: ["source"], acknowledgeLostWork: true, actionOutcome: "mixed" }));
+ expect(onRecorded).toHaveBeenCalledOnce();
+ });
+ it("preserves the reconstruction decision when current source checks reject it", async () => {
+ api.reconstructWorkspace.mockRejectedValue(new Error("The source owner changed."));
+ await mount(); await fill();
+ await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Rebuild and continue")!.click());
+ await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
+ expect(container.querySelector('[role="alert"]')?.textContent).toContain("source owner changed");
+ expect(container.querySelector("textarea")?.value).toContain("Verified source");
+ expect(onRecorded).not.toHaveBeenCalled();
+ });
+ it("offers no reconstruction form to a reader without runtime management authority", async () => {
+ await mount(false); expect(container.querySelector("button")).toBeNull();
+ expect(api.listAttachments).not.toHaveBeenCalled(); expect(api.reconstructWorkspace).not.toHaveBeenCalled();
+ });
+ it("refreshes uploaded sources with the task attachment cache and drops deleted selections", async () => {
+ await mount(); await fill();
+ await act(async () => client.setQueryData(queryKeys.issues.attachments("task"), [
+ { id: "new-source", originalFilename: "uploaded-source.ts", byteSize: 200 },
+ ]));
+ await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
+ expect(container.textContent).toContain("uploaded-source.ts");
+ expect(container.textContent).not.toContain("baseline.ts");
+ const button = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Rebuild and continue")!;
+ expect(button.disabled).toBe(true);
+ expect(api.reconstructWorkspace).not.toHaveBeenCalled();
+ });
+});
diff --git a/ui/src/components/WorkspaceReconstructionRecovery.tsx b/ui/src/components/WorkspaceReconstructionRecovery.tsx
new file mode 100644
index 0000000000..b3a1da8bcf
--- /dev/null
+++ b/ui/src/components/WorkspaceReconstructionRecovery.tsx
@@ -0,0 +1,67 @@
+import { useId, useState } from "react";
+import { useMutation, useQuery } from "@tanstack/react-query";
+import type { ExecutionBlocker } from "@paperclipai/shared";
+import { issuesApi } from "../api/issues";
+import { queryKeys } from "../lib/queryKeys";
+import { Button } from "./ui/button";
+import { Checkbox } from "./ui/checkbox";
+import { Label } from "./ui/label";
+import { Textarea } from "./ui/textarea";
+import { NativeSelect } from "./ui/select";
+
+export function WorkspaceReconstructionRecovery({ issueId, blocker, canManage, onRecorded }: {
+ issueId: string; blocker: ExecutionBlocker; canManage: boolean; onRecorded: () => void;
+}) {
+ const id = useId();
+ const [selected, setSelected] = useState<string[]>([]);
+ const [missingWork, setMissingWork] = useState("");
+ const [outcomeEvidence, setOutcomeEvidence] = useState("");
+ const [actionOutcome, setActionOutcome] = useState<"completed" | "not_performed" | "mixed">("mixed");
+ const [acknowledged, setAcknowledged] = useState(false);
+ const files = useQuery({ queryKey: queryKeys.issues.attachments(issueId),
+ queryFn: () => issuesApi.listAttachments(issueId), enabled: canManage });
+ const currentSelection = selected.filter(id => files.data?.some(file => file.id === id));
+ const submit = useMutation({
+ mutationFn: () => issuesApi.reconstructWorkspace(issueId, {
+ actionId: blocker.recoveryActionId!, runId: blocker.runId!, expectedStatusVersion: blocker.workspaceRecoveryStatusVersion!,
+ attachmentIds: currentSelection, missingWork: missingWork.trim(), actionOutcome, outcomeEvidence: outcomeEvidence.trim(), acknowledgeLostWork: true,
+ }), onSuccess: onRecorded,
+ });
+ return <details className="w-full">
+ <summary className="cursor-pointer">Continue from saved sources</summary>
+ <p>The original workspace was destroyed before its edits were saved. Its last run will remain failed.</p>
+ {canManage ? <div className="flex flex-col gap-2">
+ <p>Select task attachments to rebuild from and record what remains missing. Paperclip starts a fresh session through its normal execution checks. Existing work limits still apply.</p>
+ <fieldset disabled={submit.isPending} className="flex flex-col gap-2">
+ <legend>Saved source attachments</legend>
+ {files.data?.map(file => <div key={file.id} className="flex items-center gap-2">
+ <Checkbox id={`${id}-${file.id}`} checked={selected.includes(file.id)} onCheckedChange={checked =>
+ setSelected(current => checked === true ? [...current, file.id] : current.filter(value => value !== file.id))} />
+ <Label htmlFor={`${id}-${file.id}`}>{file.originalFilename ?? "Attachment"} ({file.byteSize} bytes)</Label>
+ </div>)}
+ {files.isPending && <p>Loading saved attachments…</p>}
+ {files.data?.length === 0 && <p>Attach the surviving source files to this task first.</p>}
+ <Label htmlFor={`${id}-missing`}>Missing work and reconstruction plan</Label>
+ <Textarea id={`${id}-missing`} maxLength={12000} value={missingWork} onChange={event => setMissingWork(event.target.value)} />
+ <Label htmlFor={`${id}-outcome`}>Previously completed external actions</Label>
+ <NativeSelect id={`${id}-outcome`} value={actionOutcome} onChange={event => setActionOutcome(event.target.value as typeof actionOutcome)}>
+ <option value="mixed">Some completed, some not performed</option>
+ <option value="not_performed">None performed</option>
+ <option value="completed">All completed</option>
+ </NativeSelect>
+ <Label htmlFor={`${id}-evidence`}>Evidence and actions that must not be repeated</Label>
+ <Textarea id={`${id}-evidence`} maxLength={12000} value={outcomeEvidence} onChange={event => setOutcomeEvidence(event.target.value)} />
+ <div className="flex items-center gap-2">
+ <Checkbox id={`${id}-ack`} checked={acknowledged} onCheckedChange={checked => setAcknowledged(checked === true)} />
+ <Label htmlFor={`${id}-ack`}>Continue from these sources. The missing edits have not been recovered.</Label>
+ </div>
+ </fieldset>
+ <div className="flex justify-end">
+ <Button disabled={submit.isPending || !acknowledged || currentSelection.length === 0 || missingWork.trim().length < 20 || outcomeEvidence.trim().length < 20}
+ onClick={() => submit.mutate()}>{submit.isPending ? "Queuing…" : "Rebuild and continue"}</Button>
+ </div>
+ {files.isError && <p role="alert" className="text-destructive">{files.error.message}</p>}
+ {submit.isError && <p role="alert" className="text-destructive">{submit.error.message}</p>}
+ </div> : <p>A board member with runtime access can continue from the surviving task attachments.</p>}
+ </details>;
+}