The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
feat: expire closed git worktrees with unsaved work behind restore points (#17248)
Devin Foley6d44b80a5edb
diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.mdindex 53504cd79e..66d70867c5 100644--- a/doc/DEVELOPING.md+++ b/doc/DEVELOPING.md@@ -2583,6 +2583,12 @@ Environment overrides:negative, empty, or non-numeric value falls back to the default.`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates.+- `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` lets git cache expiration remove closed+ worktrees with unpushed or uncommitted work after saving an encrypted restore point+ (see `doc/recoverable-disk-cleanup.md`). It is off by default.+ `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=<days>` sets how long the issue tree+ must be closed first. The default is `3`. A negative, empty, or non-numeric value falls+ back to the default.- `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS=<days>` sets how long a git workspacerestore point is kept after its issue tree closes (see`doc/recoverable-disk-cleanup.md`). The default is `90` and the minimum is `1`. Adiff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.mdindex 915c404e83..42ba628c1b 100644--- a/doc/recoverable-disk-cleanup.md+++ b/doc/recoverable-disk-cleanup.md@@ -95,6 +95,40 @@ bytes, cleared task sessions, and skip reasons.## Git workspace restore points+Git cache expiration (`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true`) normally removes a+closed issue's worktree only when its remote can reproduce it exactly. With+`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`, it can also remove a worktree that has+unpushed commits, uncommitted or untracked files, or history-only commits, once its issue+tree has been closed for `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`).+Before removal it saves a restore point:++- a git bundle of commits the remote does not have, against a remote-verified base,+ including commits only reachable from the reflog or `ORIG_HEAD`;+- deterministic snapshots of the index and working tree, including untracked files and+ non-default file modes. Ignored files are left out, and so are untracked `.env` and+ `.env.*` files, which provisioning recreates with current secrets.++The restore point is encrypted with a per-restore-point key wrapped by the instance+secrets key, uploaded to file storage, read back and hash-checked, then activated in+the same transaction that archives the workspace with a versioned recipe+(`paperclip.git-cache-restoration.v2`). Older servers reject that recipe instead of+restoring without the saved work. Under the expiration locks, the worktree must still+match the captured snapshot exactly; otherwise expiration rolls back, keeps the+worktree, and drops the restore point. Nested managed checkouts, submodules, special+git state and ignored files other than `.env` still keep a worktree from expiring.+Restore points are refused when the file storage is `s3` and the secrets master key+does not come from `PAPERCLIP_SECRETS_MASTER_KEY`.++Reopening the issue, or the next run that recreates the worktree, imports the bundle,+re-adds the worktree at the saved commit, applies the restore point and verifies the+result. The bundle is always imported, so history-only commits are present again as+objects in the repository; they are not recreated as branches or reflog entries, and the+restore point keeps them until it expires. A draft commit message left by an aborted+commit still keeps the worktree from expiring, because restore points do not save it. A worktree that already matches is only marked applied, and a worktree that+diverged is never overwritten: the restore fails closed and can be retried. A missing,+corrupt or expired restore point also fails closed.++Restore points are encrypted saves of the parts of a git worktree that its remotecannot reproduce: unpushed commits and uncommitted files. They let a closed issue'sworktree be removed and restored later. The table `execution_workspace_restore_points`diff --git a/server/src/__tests__/git-restore-point-expiration-config.test.ts b/server/src/__tests__/git-restore-point-expiration-config.test.tsnew file mode 100644index 0000000000..5d1aacc399--- /dev/null+++ b/server/src/__tests__/git-restore-point-expiration-config.test.ts@@ -0,0 +1,25 @@+import { afterEach, describe, expect, it, vi } from "vitest";+import { loadConfig } from "../config.ts";++describe("git restore point expiration config parsing", () => {+ afterEach(() => {+ vi.unstubAllEnvs();+ });++ it("is disabled by default with a three-day delay", () => {+ vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", undefined);+ vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", undefined);+ expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: false, gitRestorePointTerminalDelayDays: 3 });+ });++ it("accepts explicit settings", () => {+ vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", "true");+ vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", "0");+ expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: true, gitRestorePointTerminalDelayDays: 0 });+ });++ it.each(["", "-1", "many"])("falls back to three days for %j", (value) => {+ vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", value);+ expect(loadConfig()).toMatchObject({ gitRestorePointTerminalDelayDays: 3 });+ });+});diff --git a/server/src/__tests__/git-restore-point-expiration.test.ts b/server/src/__tests__/git-restore-point-expiration.test.tsnew file mode 100644index 0000000000..fd27b0a24a--- /dev/null+++ b/server/src/__tests__/git-restore-point-expiration.test.ts@@ -0,0 +1,244 @@+import { promises as fs } from "node:fs";+import os from "node:os";+import path from "node:path";+import { execFileSync } from "node:child_process";+import { randomUUID } from "node:crypto";+import { pathToFileURL } from "node:url";+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";+import { eq, sql } from "drizzle-orm";+import { companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, issues } from "@paperclipai/db";+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";+import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";+import { ensurePersistedExecutionWorkspaceAvailable, realizeExecutionWorkspace } from "../services/workspace-runtime.js";+import { executionWorkspaceService } from "../services/execution-workspaces.js";+import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";+import { getStorageService } from "../storage/index.js";++const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "[email protected]",+ GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "[email protected]", GIT_TERMINAL_PROMPT: "0" };+const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();+const exists = (target: string) => fs.lstat(target).then(() => true, () => false);+const DAY_MS = 24 * 60 * 60 * 1000;++const support = await getEmbeddedPostgresTestSupport();+const describeEmbeddedPostgres = support.supported ? describe : describe.skip;+describeEmbeddedPostgres("git restore point expiration", () => {+ let db: ReturnType<typeof createDb>;+ let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;+ let root: string;+ const previousHome = process.env.PAPERCLIP_HOME;+ const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;+ beforeAll(async () => {+ database = await startEmbeddedPostgresTestDatabase("git-restore-point-expiration-");+ db = createDb(database.connectionString);+ root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-restore-point-expiration-"));+ process.env.PAPERCLIP_HOME = path.join(root, "home");+ process.env.PAPERCLIP_INSTANCE_ID = "default";+ vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));+ });+ afterEach(async () => {+ await db.delete(executionWorkspaceRestorePoints);+ await db.execute(sql`TRUNCATE TABLE companies CASCADE`);+ });+ afterAll(async () => {+ vi.unstubAllEnvs();+ if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;+ if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;+ await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true });+ });++ async function fixture() {+ const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID();+ const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git");+ await fs.mkdir(source, { recursive: true });+ git(source, "init", "--initial-branch=main");+ await fs.writeFile(path.join(source, "marker.txt"), "commit A\n");+ await fs.writeFile(path.join(source, "remove-me.txt"), "tracked\n");+ await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n");+ git(source, "add", "."); git(source, "commit", "-m", "A");+ git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote);+ await db.insert(companies).values({ id: companyId, name: "Restore point fixture", issuePrefix: `RP${companyId.slice(0, 6)}` });+ await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) });+ const repoUrl = pathToFileURL(remote).href;+ const baseCwd = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null });+ const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" };+ const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "RP-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId },+ config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true });+ await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active",+ providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName,+ metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha,+ isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } });+ await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));+ const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);+ const restorePoint = (overrides: Partial<{ notBefore: Date; keyDurable: boolean }> = {}) => ({+ storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true, ...overrides,+ });+ const expire = async (options: Partial<Parameters<typeof expireReconstructableGitWorkspace>[2]> = {}) =>+ expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: false, now: new Date(), restorePoint: restorePoint(), ...options });+ const reopen = () => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId,+ issue: { id: issueId, companyId, projectId: null }, actor: { actorType: "user", agentId: null },+ authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });+ const ledger = () => db.select().from(executionWorkspaceRestorePoints).where(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId));+ const runtimeRestore = async () => ensurePersistedExecutionWorkspaceAvailable({ db, base,+ workspace: (await executionWorkspaceService(db).getStoredById(companyId, workspaceId))!,+ issue: { id: issueId, identifier: "RP-1", title: "Resumed task" },+ agent: { id: null, name: "Fixture", companyId }, skipHostProvision: true });+ return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, row, restorePoint, expire, reopen, ledger, runtimeRestore };+ }++ async function makeUnsavedWork(cwd: string) {+ await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n");+ git(cwd, "commit", "-am", "B unpushed");+ const unpushed = git(cwd, "rev-parse", "HEAD");+ await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n");+ git(cwd, "add", "marker.txt");+ await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n");+ await fs.rm(path.join(cwd, "remove-me.txt"));+ await fs.mkdir(path.join(cwd, "notes"));+ await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n");+ await fs.writeFile(path.join(cwd, ".env"), "TOKEN=old\n");+ return unpushed;+ }++ it("expires a worktree with unpushed and uncommitted work and restores it exactly on reopen", async () => {+ const f = await fixture();+ const unpushed = await makeUnsavedWork(f.cwd);+ expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null });+ expect(await exists(f.cwd)).toBe(false);+ const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;+ expect(recipe).toMatchObject({ schema: "paperclip.git-cache-restoration.v2", commit: unpushed,+ restorePoint: { head: unpushed, headOnRemote: false, appliedAt: null } });+ expect(await f.ledger()).toMatchObject([{ status: "active", objectKey: recipe.restorePoint.objectKey }]);++ expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });+ expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);+ expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n");+ expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit");+ await expect(fs.access(path.join(f.cwd, "remove-me.txt"))).rejects.toMatchObject({ code: "ENOENT" });+ expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");+ await expect(fs.access(path.join(f.cwd, ".env"))).rejects.toMatchObject({ code: "ENOENT" });+ const restored = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;+ expect(restored.restorePoint.appliedAt).toEqual(expect.any(String));+ });++ it("restores through the runtime path and never re-applies over later work", async () => {+ const f = await fixture();+ const unpushed = await makeUnsavedWork(f.cwd);+ expect(await f.expire()).toMatchObject({ expired: true });+ await db.update(executionWorkspaces).set({ status: "active", closedAt: null }).where(eq(executionWorkspaces.id, f.workspaceId));+ await f.runtimeRestore();+ expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);+ expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");+ const applied = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;+ expect(applied.restorePoint.appliedAt).toEqual(expect.any(String));+ await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "later work\n");+ await f.runtimeRestore();+ expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("later work\n");+ });++ it("imports history-only commits on reopen even when the base no longer has them", async () => {+ const f = await fixture();+ await fs.writeFile(path.join(f.cwd, "lost.txt"), "rewritten away\n");+ git(f.cwd, "add", "lost.txt");+ git(f.cwd, "commit", "-m", "rewritten away");+ const lost = git(f.cwd, "rev-parse", "HEAD");+ git(f.cwd, "reset", "-q", "--hard", "HEAD~1");+ expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true });+ git(f.baseCwd, "reflog", "expire", "--expire=now", "--all");+ git(f.baseCwd, "gc", "-q", "--prune=now");+ expect(() => git(f.baseCwd, "cat-file", "-e", `${lost}^{commit}`)).toThrow();+ expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });+ expect(git(f.baseCwd, "cat-file", "-t", lost)).toBe("commit");+ });++ it("keeps a worktree whose draft commit message a restore point cannot save", async () => {+ const f = await fixture();+ await fs.writeFile(path.join(f.cwd, "marker.txt"), "staged edit\n");+ git(f.cwd, "add", "marker.txt");+ const gitDir = git(f.cwd, "rev-parse", "--absolute-git-dir");+ await fs.writeFile(path.join(gitDir, "COMMIT_EDITMSG"), "Draft message from an aborted commit\n");+ expect(await f.expire()).toMatchObject({ expired: false, skipReason: "uncommitted_git_message" });+ expect(await exists(f.cwd)).toBe(true);+ expect(await f.ledger()).toEqual([]);+ });++ it("marks a restore point for cleanup when its upload fails", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ const storage = getStorageService();+ const failing = { ...storage, putFile: async () => { throw new Error("storage unavailable"); } };+ expect(await f.expire({ restorePoint: { ...f.restorePoint(), storage: failing as typeof storage } }))+ .toMatchObject({ expired: false });+ expect(await exists(f.cwd)).toBe(true);+ expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);+ expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();+ });++ it("does not expire unsaved work before the restore point delay", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ expect(await f.expire({ restorePoint: f.restorePoint({ notBefore: new Date(Date.now() + DAY_MS) }) }))+ .toMatchObject({ expired: false, skipReason: "restore_point_not_due" });+ expect(await exists(f.cwd)).toBe(true);+ expect(await f.ledger()).toEqual([]);+ });++ it("keeps today's behavior when restore points are disabled", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ expect(await f.expire({ restorePoint: undefined })).toMatchObject({ expired: false, skipReason: "dirty_or_untracked_files" });+ expect(await exists(f.cwd)).toBe(true);+ });++ it("refuses to capture when the encryption key is not durable", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ expect(await f.expire({ restorePoint: f.restorePoint({ keyDurable: false }) }))+ .toMatchObject({ expired: false, skipReason: "restore_point_key_not_durable" });+ expect(await exists(f.cwd)).toBe(true);+ });++ it("keeps the worktree and drops the restore point when contents change after capture", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ const result = await f.expire({ beforeDelete: async () => { await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "changed after capture\n"); } });+ expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });+ expect(await exists(f.cwd)).toBe(true);+ expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("changed after capture\n");+ const after = await f.row();+ expect(after.status).toBe("active");+ expect(after.metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });+ expect((after.metadata?.gitCacheRestoration as Record<string, unknown>).restorePoint).toBeUndefined();+ expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);+ });++ it("fails closed when the stored restore point is missing", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ expect(await f.expire()).toMatchObject({ expired: true });+ const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;+ await getStorageService().deleteObject(f.companyId, recipe.restorePoint.objectKey);+ expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });+ const failed = await f.row();+ expect(failed.status).toBe("archived");+ expect(failed.metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { objectKey: recipe.restorePoint.objectKey, appliedAt: null } });+ await expect(fs.access(path.join(f.cwd, "notes", "todo.md"))).rejects.toMatchObject({ code: "ENOENT" });+ expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });+ });++ it("reports a restore point candidate in dry run without storing anything", async () => {+ const f = await fixture();+ await makeUnsavedWork(f.cwd);+ expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, restorePoint: true, skipReason: null });+ expect(await exists(f.cwd)).toBe(true);+ expect(await f.ledger()).toEqual([]);+ expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();+ });++ it("still expires a fully reproducible worktree without a restore point", async () => {+ const f = await fixture();+ expect(await f.expire()).toMatchObject({ expired: true, restorePoint: false });+ expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });+ expect(await f.ledger()).toEqual([]);+ });+});diff --git a/server/src/config.ts b/server/src/config.tsindex 23b1ee84c5..bc974203ec 100644--- a/server/src/config.ts+++ b/server/src/config.ts@@ -92,6 +92,8 @@ export interface Config {nativeHarnessBackupExpirationDryRun: boolean;nativeHarnessBackupExpirationDays: number;gitRestorePointRetentionDays: number;+ gitRestorePointsEnabled: boolean;+ gitRestorePointTerminalDelayDays: number;serveUi: boolean;uiDevMiddleware: boolean;secretsProvider: SecretProvider;@@ -339,6 +341,7 @@ export function loadConfig(): Config {const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60);const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7));const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90));+ const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3);const bindValidationErrors = validateConfiguredBindMode({deploymentMode,deploymentExposure,@@ -397,6 +400,8 @@ export function loadConfig(): Config {nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true",nativeHarnessBackupExpirationDays,gitRestorePointRetentionDays,+ gitRestorePointsEnabled: process.env.PAPERCLIP_GIT_RESTORE_POINTS_ENABLED === "true",+ gitRestorePointTerminalDelayDays,serveUi:process.env.SERVE_UI !== undefined? process.env.SERVE_UI === "true"diff --git a/server/src/index.ts b/server/src/index.tsindex 6b6de3cb71..d8f26a3dbf 100644--- a/server/src/index.ts+++ b/server/src/index.ts@@ -13,6 +13,7 @@ import {nativeHarnessBackupExpirationService,} from "./services/native-runtime/native-harness-backup-expiration.js";import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js";+import { markGitRestorePointExpired } from "./services/git-restore-point-storage.js";import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";import { registerBudgetEnforcementWork, registerCostAccountingWork } from "./services/budget-enforcement-worker.js";@@ -1479,6 +1480,11 @@ async function startServerWithDatabaseTeardown(gitCacheExpirationEnabled: config.gitCacheExpirationEnabled,gitBaseExpirationEnabled: config.gitBaseExpirationEnabled,gitCacheExpirationDryRun: config.gitCacheExpirationDryRun,+ gitRestorePoints: config.gitRestorePointsEnabled ? {+ storage: storageService,+ terminalDelayMs: config.gitRestorePointTerminalDelayDays * 24 * 60 * 60 * 1000,+ keyDurable: config.storageProvider !== "s3" || Boolean(process.env.PAPERCLIP_SECRETS_MASTER_KEY?.trim()),+ } : undefined,workspaceReaperCooldownDays: config.workspaceReaperCooldownDays,workspaceCompactionDelayMs: config.workspaceCompactionDelayHours === null? null@@ -1501,6 +1507,7 @@ async function startServerWithDatabaseTeardown(const gitRestorePointWorker = registerGitRestorePointWork(deliveryWork, gitRestorePointRetentionService(db as any, {storage: storageService,retentionDays: config.gitRestorePointRetentionDays,+ beforeExpire: (restorePoint) => markGitRestorePointExpired(db as any, restorePoint),}), (result) => {if (result.abandoned > 0 || result.expired > 0 || result.orphaned > 0 || result.deleted > 0 || result.failed > 0) {logger.info(result, "git restore point retention");diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.tsindex 91f7a0d03d..13830397ce 100644--- a/server/src/services/execution-workspaces.ts+++ b/server/src/services/execution-workspaces.ts@@ -5,6 +5,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js";import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js";import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";import { expireReconstructableGitWorkspace, hasPendingGitExpiration } from "./git-workspace-expiration.js";+import type { StorageService } from "../storage/types.js";import { readGitCacheRestoration, restoreGitCacheBase, recoverGitCacheExpirationLocks, GitCacheRestorationFailure } from "./git-workspace-restoration.js";import { parseObject } from "../adapters/utils.js";import { resolveCoreTrustPreset } from "./trust-preset-resolver.js";@@ -289,6 +290,7 @@ export type ExecutionWorkspaceServiceOptions = {gitCacheExpirationEnabled?: boolean;gitBaseExpirationEnabled?: boolean;gitCacheExpirationDryRun?: boolean;+ gitRestorePoints?: { storage: StorageService; terminalDelayMs: number; keyDurable: boolean };terminalWorkspaceInspectionBackoffMs?: number;resolvePullRequestDetails?: PullRequestMergeDetailsResolver;now?: () => Date;@@ -3343,6 +3345,11 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServicconst expiration = await expireReconstructableGitWorkspace(db, workspace, {dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false,now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup,+ restorePoint: opts.gitRestorePoints ? {+ storage: opts.gitRestorePoints.storage,+ notBefore: new Date(latestTerminalAt + opts.gitRestorePoints.terminalDelayMs),+ keyDurable: opts.gitRestorePoints.keyDurable,+ } : undefined,});const expirationDueAt = now().getTime() + 60 * 60 * 1000;terminalWorkspaceDeadlines.set(workspace.id,@@ -3989,6 +3996,17 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServicif (updated.length !== 1) throw new GitCacheRestorationFailure();nextMetadata.gitCacheRestoration = { ...currentRecipe as Record<string, unknown>, contentsRestoredAt: at };},+ persistRestorePointAppliedAt: async at => {+ const currentRecipe = nextMetadata.gitCacheRestoration as Record<string, unknown>;+ const appliedRecipe = { ...currentRecipe, restorePoint: { ...currentRecipe.restorePoint as Record<string, unknown>, appliedAt: at } };+ const updated = await db.update(executionWorkspaces).set({+ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration}', ${JSON.stringify(appliedRecipe)}::jsonb)`,+ }).where(and(eq(executionWorkspaces.id, row.id), eq(executionWorkspaces.companyId, row.companyId),+ sql`${executionWorkspaces.metadata}->'gitCacheRestoration' = ${JSON.stringify(currentRecipe)}::jsonb`))+ .returning({ id: executionWorkspaces.id });+ if (updated.length !== 1) throw new GitCacheRestorationFailure();+ nextMetadata.gitCacheRestoration = appliedRecipe;+ },prepareRestoredLocalSource: async sourceCwd => {const sourceIdentity = await fs.realpath(sourceCwd);const targetIdentity = await canonicalRepositoryRoot(row.cwd!);diff --git a/server/src/services/git-restore-point-storage.ts b/server/src/services/git-restore-point-storage.tsnew file mode 100644index 0000000000..f0aaa5b2f5--- /dev/null+++ b/server/src/services/git-restore-point-storage.ts@@ -0,0 +1,251 @@+import { createHash, randomUUID } from "node:crypto";+import { and, eq, inArray, ne, sql } from "drizzle-orm";+import { executionWorkspaceRestorePoints, executionWorkspaces, type Db } from "@paperclipai/db";+import type { StoredSecretVersionMaterial } from "../secrets/types.js";+import type { StorageService } from "../storage/types.js";+import {+ applyRestorePoint,+ captureRestorePoint,+ currentRestorePointSnapshot,+ decodeRestorePointPayload,+ importRestorePoint,+ openRestorePoint,+ RestorePointRestoreFailure,+ sealRestorePoint,+ type RestorePointLimits,+ type RestorePointManifest,+} from "./git-restore-point.js";+import { notifyGitRestorePointWork } from "./git-restore-point-ledger.js";+import { gitCacheCommand } from "./git-workspace-restoration.js";++export const GIT_RESTORE_POINT_SCHEMA = "paperclip.git-restore-point.v1";+const NAMESPACE = "git-restore-points";+const MAX_RESTORE_POINT_BYTES = 512 * 1024 * 1024;++export type GitRestorePointRef = {+ schema: typeof GIT_RESTORE_POINT_SCHEMA;+ id: string;+ objectKey: string;+ sha256: string;+ byteSize: number;+ key: StoredSecretVersionMaterial;+ generation: number;+ head: string;+ index: string | null;+ worktree: string | null;+ modes: Array<{ path: string; mode: number }>;+ headOnRemote: boolean;+ appliedAt?: string | null;+ expiredAt?: string;+};++type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];++function restorePointObjectKey(companyId: string, workspaceId: string, generation: number, id: string) {+ return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/${id}.pcrp`;+}++export function validateGitRestorePointRef(value: unknown, workspace: { companyId: string; id: string }): GitRestorePointRef {+ const ref = value as GitRestorePointRef | null;+ const sha = /^[0-9a-f]{40}$/;+ if (!ref || ref.schema !== GIT_RESTORE_POINT_SCHEMA || typeof ref.id !== "string"+ || ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id)+ || !/^[0-9a-f]{64}$/.test(ref.sha256) || !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_RESTORE_POINT_BYTES+ || !ref.key || typeof ref.key !== "object" || Array.isArray(ref.key)+ || !Number.isSafeInteger(ref.generation) || ref.generation < 1+ || !sha.test(ref.head) || (ref.index !== null && !sha.test(ref.index)) || (ref.worktree !== null && !sha.test(ref.worktree))+ || (ref.index === null) !== (ref.worktree === null) || typeof ref.headOnRemote !== "boolean"+ || !Array.isArray(ref.modes) || ref.modes.length > 20_000 || ref.modes.some((entry) => !entry || typeof entry.path !== "string"+ || entry.path.startsWith("/") || entry.path.split("/").includes("..") || !Number.isInteger(entry.mode) || entry.mode < 0 || entry.mode > 0o777)+ || (ref.appliedAt != null && !Number.isFinite(Date.parse(ref.appliedAt)))+ || (ref.expiredAt !== undefined && !Number.isFinite(Date.parse(ref.expiredAt)))) {+ throw new RestorePointRestoreFailure("invalid_restore_point_reference");+ }+ return ref;+}++export async function storeGitRestorePoint(input: {+ db: Db;+ storage: StorageService;+ workspace: { companyId: string; id: string; sourceIssueId: string | null };+ cwd: string;+ generation: number;+ upstreamCommits: readonly string[];+ headOnRemote: boolean;+ limits?: Partial<RestorePointLimits>;+ signal?: AbortSignal;+}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {+ const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal });+ if (!captured) return null;+ const id = randomUUID();+ const identity = { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: id, generation: input.generation };+ const sealed = await sealRestorePoint(captured.payload, identity);+ const objectKey = restorePointObjectKey(input.workspace.companyId, input.workspace.id, input.generation, id);+ const [ledger] = await input.db.transaction(async (tx) => {+ await notifyGitRestorePointWork(tx, `workspace:${input.workspace.id}`);+ return tx.insert(executionWorkspaceRestorePoints).values({+ companyId: input.workspace.companyId,+ executionWorkspaceId: input.workspace.id,+ sourceIssueId: input.workspace.sourceIssueId,+ generation: input.generation,+ objectKey,+ sha256: sealed.sha256,+ byteSize: sealed.ciphertext.length,+ status: "pending",+ }).returning({ id: executionWorkspaceRestorePoints.id });+ });+ try {+ await input.storage.putFile({+ objectKey,+ companyId: input.workspace.companyId,+ namespace: NAMESPACE,+ originalFilename: null,+ contentType: "application/octet-stream",+ body: sealed.ciphertext,+ });+ const stored = await readObject(input.storage, input.workspace.companyId, objectKey, input.signal);+ if (createHash("sha256").update(stored).digest("hex") !== sealed.sha256) {+ throw new RestorePointRestoreFailure("stored_restore_point_mismatch");+ }+ } catch (error) {+ await abandonGitRestorePoint(input.db, ledger!.id).catch(() => undefined);+ throw error;+ }+ const { manifest } = captured;+ return {+ ledgerId: ledger!.id,+ manifest,+ ref: {+ schema: GIT_RESTORE_POINT_SCHEMA,+ id,+ objectKey,+ sha256: sealed.sha256,+ byteSize: sealed.ciphertext.length,+ key: sealed.key,+ generation: input.generation,+ head: manifest.head,+ index: manifest.index,+ worktree: manifest.worktree,+ modes: manifest.modes,+ headOnRemote: input.headOnRemote,+ appliedAt: null,+ },+ };+}++export async function activateGitRestorePoint(tx: Tx, input: { ledgerId: string; workspaceId: string; at: Date }) {+ await notifyGitRestorePointWork(tx, `workspace:${input.workspaceId}`);+ await tx.update(executionWorkspaceRestorePoints)+ .set({ status: "superseded", updatedAt: input.at })+ .where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, input.workspaceId),+ ne(executionWorkspaceRestorePoints.id, input.ledgerId),+ inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));+ const activated = await tx.update(executionWorkspaceRestorePoints)+ .set({ status: "active", activatedAt: input.at, deleteAfter: input.at, updatedAt: input.at })+ .where(and(eq(executionWorkspaceRestorePoints.id, input.ledgerId), eq(executionWorkspaceRestorePoints.status, "pending")))+ .returning({ id: executionWorkspaceRestorePoints.id });+ if (activated.length !== 1) throw new RestorePointRestoreFailure("restore_point_not_pending");+}++export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null; modes: Array<{ path: string; mode: number }> }) {+ return snapshot.head === ref.head && snapshot.index === ref.index && snapshot.worktree === ref.worktree+ && JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes);+}++export async function abandonGitRestorePointById(db: Db, workspaceId: string, objectKey: string) {+ await db.update(executionWorkspaceRestorePoints)+ .set({ status: "superseded", updatedAt: new Date() })+ .where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId), eq(executionWorkspaceRestorePoints.objectKey, objectKey),+ inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));+}++export async function abandonGitRestorePoint(db: Db, ledgerId: string) {+ await db.update(executionWorkspaceRestorePoints)+ .set({ status: "superseded", updatedAt: new Date() })+ .where(and(eq(executionWorkspaceRestorePoints.id, ledgerId), inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));+}++async function readObject(storage: StorageService, companyId: string, objectKey: string, signal?: AbortSignal) {+ const { stream } = await storage.getObject(companyId, objectKey);+ const chunks: Buffer[] = [];+ let total = 0;+ for await (const chunk of stream) {+ signal?.throwIfAborted();+ const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);+ total += buffer.length;+ if (total > MAX_RESTORE_POINT_BYTES) throw new RestorePointRestoreFailure("restore_point_too_large");+ chunks.push(buffer);+ }+ return Buffer.concat(chunks);+}++export async function loadGitRestorePoint(input: {+ storage: StorageService;+ workspace: { companyId: string; id: string };+ ref: GitRestorePointRef;+ signal?: AbortSignal;+}): Promise<{ payload: Buffer; manifest: RestorePointManifest }> {+ if (input.ref.expiredAt) throw new RestorePointRestoreFailure("restore_point_expired");+ const ciphertext = await readObject(input.storage, input.workspace.companyId, input.ref.objectKey, input.signal)+ .catch((error) => {+ if (error instanceof RestorePointRestoreFailure) throw error;+ throw new RestorePointRestoreFailure("restore_point_unavailable");+ });+ if (ciphertext.length !== input.ref.byteSize) throw new RestorePointRestoreFailure("ciphertext_size_mismatch");+ const payload = await openRestorePoint({+ ciphertext,+ sha256: input.ref.sha256,+ key: input.ref.key,+ identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: input.ref.id, generation: input.ref.generation },+ });+ const { manifest } = decodeRestorePointPayload(payload);+ if (manifest.head !== input.ref.head || manifest.index !== input.ref.index || manifest.worktree !== input.ref.worktree+ || JSON.stringify(manifest.modes) !== JSON.stringify(input.ref.modes)) {+ throw new RestorePointRestoreFailure("restore_point_manifest_mismatch");+ }+ return { payload, manifest };+}++export async function ensureGitRestorePointCommit(input: {+ storage: StorageService;+ workspace: { companyId: string; id: string };+ ref: GitRestorePointRef;+ repositoryCwd: string;+ signal?: AbortSignal;+}) {+ if (input.ref.headOnRemote) return;+ const present = await importNeeded(input.repositoryCwd, input.ref.head);+ if (!present) return;+ const { payload } = await loadGitRestorePoint(input);+ await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal });+}++async function importNeeded(repositoryCwd: string, commit: string) {+ return !await gitCacheCommand(repositoryCwd, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false);+}++export async function applyGitRestorePointToWorktree(input: {+ storage: StorageService;+ workspace: { companyId: string; id: string };+ ref: GitRestorePointRef;+ repositoryCwd: string;+ worktreePath: string;+ signal?: AbortSignal;+}) {+ const current = await currentRestorePointSnapshot(input.worktreePath, input.signal);+ const matches = restorePointSnapshotMatches(input.ref, current);+ if (!matches && (current.head !== input.ref.head || current.index !== null || current.worktree !== null || current.modes.length > 0)) {+ throw new RestorePointRestoreFailure("worktree_diverged_from_restore_point");+ }+ const { payload } = await loadGitRestorePoint(input);+ const manifest = await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal });+ if (!matches) await applyRestorePoint({ cwd: input.worktreePath, manifest, signal: input.signal });+}++export async function markGitRestorePointExpired(db: Db, restorePoint: { companyId: string; executionWorkspaceId: string; objectKey: string }) {+ await db.update(executionWorkspaces).set({+ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,expiredAt}', ${JSON.stringify(new Date().toISOString())}::jsonb)`,+ updatedAt: new Date(),+ }).where(and(eq(executionWorkspaces.id, restorePoint.executionWorkspaceId), eq(executionWorkspaces.companyId, restorePoint.companyId),+ sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'objectKey' = ${restorePoint.objectKey}`));+}diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.tsindex 397b27b67e..e55aab0988 100644--- a/server/src/services/git-workspace-expiration.ts+++ b/server/src/services/git-workspace-expiration.ts@@ -7,11 +7,14 @@ import { and, eq, ne, sql } from "drizzle-orm";import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases,workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db";import { createGitRemoteAuthProvider } from "./git-credentials.js";+import type { StorageService } from "../storage/types.js";+import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js";+import { abandonGitRestorePoint, abandonGitRestorePointById, activateGitRestorePoint, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js";import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";import type { GitRemoteAuthProvider } from "./git-credentials.js";-import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, readGitCacheRestoration,- verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";+import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,+ verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {if (!shouldContinue()) throw new Error("scan_stopped");@@ -41,7 +44,29 @@ type Workspace = typeof executionWorkspaces.$inferSelect;type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];type Expiration = GitCacheExpiration;export type GitExpirationResult = { candidate: boolean; expired: boolean; reclaimedBytes: number;- candidateBytes: number; baseExpired: boolean; skipReason: string | null };+ candidateBytes: number; baseExpired: boolean; skipReason: string | null; restorePoint: boolean };++export type GitRestorePointExpirationOptions = {+ storage: StorageService;+ notBefore: Date;+ keyDurable: boolean;+ limits?: Partial<RestorePointLimits>;+};++const RESTORE_POINT_RELAXABLE = new Set(["dirty_or_untracked_files", "unique_ignored_files", "unique_original_head",+ "unique_reflog_commits", "upstream_commit_unverified"]);++async function restorePointBoundary(recipe: GitCacheRestoration, headOnRemote: boolean, auth: GitRemoteAuthProvider, signal?: AbortSignal) {+ if (headOnRemote) return [recipe.commit];+ const upstream = await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--verify", "-q", "refs/remotes/origin/HEAD"], signal)+ .catch(() => "");+ const tip = upstream.trim();+ if (!tip) return [];+ const base = (await gitCacheCommand(recipe.worktreePath, ["merge-base", "HEAD", tip], signal).catch(() => "")).trim();+ if (!/^[a-f0-9]{40}$/.test(base)) return [];+ try { await verifyRemoteGitCommit(recipe.repoUrl, base, auth, signal); return [base]; }+ catch (error) { if (signal?.aborted) throw error; return []; }+}/** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */export async function isGitWorkspaceExpirationEligible(tx: Tx, workspace: Workspace, now: Date) {@@ -156,9 +181,11 @@ function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuthexport async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspace, options: {dryRun: boolean; expireBase: boolean; now: Date; signal?: AbortSignal;beforeDelete?: (workspace: Workspace) => Promise<void>;+ restorePoint?: GitRestorePointExpirationOptions;}): Promise<GitExpirationResult> {const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0,- candidateBytes: 0, baseExpired: false, skipReason: null };+ candidateBytes: 0, baseExpired: false, skipReason: null, restorePoint: false };+ let pendingRestorePoint: string | null = null;if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" };try {return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () =>@@ -176,8 +203,24 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspconst checkouts = new Map<string, ManagedCheckoutContents>();const identity = { companyId: workspace.companyId, workspaceId: workspace.id };const proveManagedCheckout = managedCheckoutProver(db, workspace, auth, checkouts);- let recipe = state ? readGitCacheRestoration(workspace)! : await inspectGitCacheRecipe(workspace,- auth, options.signal, { proveManagedCheckout });+ let headOnRemote = true;+ let needsRestorePoint = false;+ let recipe: GitCacheRestoration;+ if (state) recipe = readGitCacheRestoration(workspace)!;+ else {+ try { recipe = await inspectGitCacheRecipe(workspace, auth, options.signal, { proveManagedCheckout }); }+ catch (error) {+ if (!options.restorePoint || !RESTORE_POINT_RELAXABLE.has(skipReason(error))) throw error;+ if (options.now.getTime() < options.restorePoint.notBefore.getTime()) throw new Error("restore_point_not_due");+ if (!options.restorePoint.keyDurable) throw new Error("restore_point_key_not_durable");+ checkouts.clear();+ const inspected = await inspectGitCacheRecipeForRestorePoint(workspace, auth, options.signal, { proveManagedCheckout });+ recipe = inspected.recipe;+ headOnRemote = inspected.headOnRemote;+ needsRestorePoint = true;+ result.restorePoint = true;+ }+ }if (!state) recipe.contents = await captureGitWorkspaceContents(recipe.worktreePath, identity, [...checkouts.values()]);const present = await exists(recipe.worktreePath);if (!state && !present) throw new Error("missing_unverified_worktree");@@ -199,16 +242,34 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspif (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) {throw new Error("unproven_worktree_admin");}+ let activateLedger: string | null = null;+ if (needsRestorePoint) {+ const stored = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage,+ workspace: { companyId: workspace.companyId, id: workspace.id, sourceIssueId: workspace.sourceIssueId },+ cwd: recipe.worktreePath, generation, headOnRemote, limits: options.restorePoint!.limits, signal: options.signal,+ upstreamCommits: await restorePointBoundary(recipe, headOnRemote, auth, options.signal) });+ if (stored) {+ pendingRestorePoint = stored.ledgerId;+ activateLedger = stored.ledgerId;+ recipe = { ...recipe, schema: GIT_CACHE_RESTORATION_SCHEMA_V2, restorePoint: stored.ref };+ } else if (!headOnRemote) throw new Error("upstream_commit_unverified");+ }state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false,worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`),worktreeAdmin, baseTrash: null, completed: false, lockToken: randomBytes(32).toString("hex") };if (await exists(state.worktreeTrash)) throw new Error("unknown_worktree_trash");// Commit the immutable restoration recipe BEFORE touching disk. The separate// lifecycle-lock connection remains held through deletion and final state.- [workspace] = await db.update(executionWorkspaces).set({ status: "archived", closedAt: options.now,+ const archive = { status: "archived", closedAt: options.now,cleanupEligibleAt: options.now, cleanupReason: GIT_CACHE_EXPIRATION_REASON, updatedAt: options.now,metadata: { ...workspace.metadata, lifecycleGeneration: generation, [GIT_CACHE_RESTORATION_KEY]: recipe, [EXPIRATION_KEY]: state },- }).where(eq(executionWorkspaces.id, workspace.id)).returning();+ };+ [workspace] = await db.transaction(async (archiveTx) => {+ const rows = await archiveTx.update(executionWorkspaces).set(archive).where(eq(executionWorkspaces.id, workspace!.id)).returning();+ if (activateLedger) await activateGitRestorePoint(archiveTx, { ledgerId: activateLedger, workspaceId: workspace!.id, at: options.now });+ return rows;+ });+ pendingRestorePoint = null;}if (!workspace) throw new Error("workspace_changed");const persist = async () => {@@ -229,7 +290,10 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksptry {checkouts.clear();await assertReproducibleGitFiles(recipe.worktreePath, options.signal, { proveManagedCheckout, preserveSettings: true,+ restorePoint: Boolean(recipe.restorePoint),verifyFetchedCommit: (commit, signal) => verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal) });+ if (recipe.restorePoint && !restorePointSnapshotMatches(recipe.restorePoint,+ await currentRestorePointSnapshot(recipe.worktreePath, options.signal))) throw new Error("restore_point_contents_changed");for (const checkout of checkouts.values()) {const cwd = path.join(recipe.worktreePath, ".paperclip-repositories", checkout.name);locks.push(await acquireGitExpirationLocks(lockLocations.filter(location => location.checkout === checkout.name), state.lockToken));@@ -241,11 +305,13 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspif ((await gitCacheCommand(recipe.worktreePath, ["rev-parse", "HEAD"], options.signal)).trim() !== recipe.commit) throw new Error("head_changed");} catch (error) {for (const lock of [...locks].reverse()) await lock.release();+ const { restorePoint: _abandoned, ...reproducible } = recipe;const metadata: Record<string, unknown> = { ...workspace.metadata, lifecycleGeneration: state.generation + 1,- [GIT_CACHE_RESTORATION_KEY]: { ...recipe, restoredAt: new Date().toISOString() } };+ [GIT_CACHE_RESTORATION_KEY]: { ...reproducible, schema: "paperclip.git-cache-restoration.v1", restoredAt: new Date().toISOString() } };delete metadata[EXPIRATION_KEY];await db.update(executionWorkspaces).set({ status: "active", closedAt: null, cleanupEligibleAt: null,cleanupReason: null, metadata, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id));+ if (recipe.restorePoint) await abandonGitRestorePointById(db, workspace.id, recipe.restorePoint.objectKey);throw error;}await fs.rename(recipe.worktreePath, state.worktreeTrash);@@ -284,8 +350,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspreturn result;})), true);} catch (error) {+ if (pendingRestorePoint) await abandonGitRestorePoint(db, pendingRestorePoint).catch(() => undefined);options.signal?.throwIfAborted();- result.skipReason = skipReason(error);+ result.skipReason = error instanceof RestorePointSkip ? error.reason : skipReason(error);return result;}}diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.tsindex c05a14bbfc..c5c3c2774f 100644--- a/server/src/services/git-workspace-restoration.ts+++ b/server/src/services/git-workspace-restoration.ts@@ -13,14 +13,18 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js";import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js";import { recordedIsolatedRemoteSource, verifiedRecordedIsolatedRemoteSource, materializeIsolatedRemoteWorkspaceSource } from "./isolated-remote-workspace-source.js";import { findRegenerableWorkspaceDirs } from "./workspace-compaction.js";+import { ensureGitRestorePointCommit, validateGitRestorePointRef, type GitRestorePointRef } from "./git-restore-point-storage.js";+import { getStorageService } from "../storage/index.js";+import type { StorageService } from "../storage/types.js";import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync";const execFile = promisify(execFileCallback);const shaPattern = /^[a-f0-9]{40}$/;export const GIT_CACHE_RESTORATION_KEY = "gitCacheRestoration";export const GIT_CACHE_EXPIRATION_REASON = "reproducible_git_cache_expired";+export const GIT_CACHE_RESTORATION_SCHEMA_V2 = "paperclip.git-cache-restoration.v2";export type GitCacheRestoration = {- schema: "paperclip.git-cache-restoration.v1";+ schema: "paperclip.git-cache-restoration.v1" | typeof GIT_CACHE_RESTORATION_SCHEMA_V2;companyId: string;workspaceId: string;sourceIssueId: string;@@ -36,6 +40,7 @@ export type GitCacheRestoration = {restoredAt?: string;contentsRestoredAt?: string | null;contents?: GitWorkspaceContents;+ restorePoint?: GitRestorePointRef;};export type GitCacheExpiration = {schema: "paperclip.git-cache-expiration.v1"; generation: number; workspaceRemoved: boolean;@@ -80,7 +85,7 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestoraticonst value = workspace.metadata?.[GIT_CACHE_RESTORATION_KEY];if (value === undefined) return null;const recipe = value as GitCacheRestoration | null;- if (!recipe || recipe.schema !== "paperclip.git-cache-restoration.v1"+ if (!recipe || !restorationSchemaMatches(recipe)|| !isRuntimeOwnedGitBranch(workspace.metadata)|| recipe.gitBranchOwnershipVersion !== 1|| recipe.companyId !== workspace.companyId || recipe.workspaceId !== workspace.id@@ -96,9 +101,18 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestoratithrow new GitCacheRestorationFailure();}validateGitWorkspaceContents(recipe.contents);+ if (recipe.restorePoint) {+ try { validateGitRestorePointRef(recipe.restorePoint, workspace); } catch { throw new GitCacheRestorationFailure(); }+ }return recipe;}+function restorationSchemaMatches(recipe: GitCacheRestoration) {+ return recipe.schema === "paperclip.git-cache-restoration.v1"+ ? recipe.restorePoint === undefined+ : recipe.schema === GIT_CACHE_RESTORATION_SCHEMA_V2 && recipe.restorePoint !== undefined;+}+export async function gitCacheCommand(cwd: string, args: string[], signal?: AbortSignal) {const result = await execFile("git", ["-c", "core.fsmonitor=false", "-C", cwd, ...args], {timeout: 60_000, signal, maxBuffer: 16 * 1024 * 1024,@@ -293,14 +307,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSignproveManagedCheckout?: ManagedCheckoutProver;preserveSettings?: boolean;verifyFetchedCommit?: (commit: string, signal?: AbortSignal) => Promise<void>;+ restorePoint?: boolean;} = {}) {- if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files");+ if (!options.restorePoint && (await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files");if ((await gitCacheCommand(cwd, ["ls-files", "-v"], signal)).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags");const localConfig = await gitCacheCommand(cwd, ["config", "--local", "--name-only", "--list"], signal);if (localConfig.split("\n").filter(Boolean).some(key => !/^(core\.(repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode)|remote\.origin\.(url|fetch)|branch\.[^.]+\.(remote|merge))$/.test(key))) throw new Error("unsupported_local_git_config");const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal });const ignored = (await gitCacheCommand(cwd, ["ls-files", "--others", "--ignored", "--exclude-standard", "-z"], signal)).split("\0").filter(Boolean);- const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`)));+ const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`))+ && !(options.restorePoint && /^\.env(\..*)?$/.test(path.posix.basename(file))));const checkouts = new Set<string>();for (const file of unexplained) {const [namespace, name, rest] = file.split("/", 3);@@ -323,7 +339,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSignif (await fs.lstat(path.join(root, name)).then(() => true, (e: NodeJS.ErrnoException) => { if(e.code === "ENOENT") return false; throw e; })) throw new Error("unsupported_local_git_state");}}- const original = await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null);+ const original = options.restorePoint ? null : await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null);if (original) await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", original.trim(), "HEAD"], signal).catch(() => { throw new Error("unique_original_head"); });if (gitDir !== common) {@@ -343,7 +359,12 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSignconst message = await fs.readFile(path.join(gitDir, "COMMIT_EDITMSG"), "utf8").catch((error: NodeJS.ErrnoException) => {if (error.code === "ENOENT") return null; throw error;});- if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) throw new Error("uncommitted_git_message");+ if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) {+ const saved = options.restorePoint+ ? (await gitCacheCommand(cwd, ["log", "-g", "--format=%B%x00", "HEAD"], signal).catch(() => "")).split("\0").map(entry => entry.trim())+ : [];+ if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message");+ }const logs = await fs.readdir(path.join(gitDir, "logs")).catch((error: NodeJS.ErrnoException) => {if (error.code === "ENOENT") return []; throw error;});@@ -352,7 +373,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign// Packed stashes/replacement refs are just as important as loose refs.if ((await gitCacheCommand(cwd, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], signal)).trim()) throw new Error("unique_git_refs");// Rewrites can leave commits reachable only from this worktree's reflog.- for (const commit of new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) {+ for (const commit of options.restorePoint ? [] : new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) {await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", commit, "HEAD"], signal).catch(() => { throw new Error("unique_reflog_commits"); });}return regenerable;@@ -360,6 +381,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSignexport async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal,options: { proveManagedCheckout?: ManagedCheckoutProver } = {}): Promise<GitCacheRestoration> {+ return (await inspectRecipe(workspace, resolveGitAuth, signal, options, false)).recipe;+}++export async function inspectGitCacheRecipeForRestorePoint(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal,+ options: { proveManagedCheckout?: ManagedCheckoutProver } = {}) {+ return inspectRecipe(workspace, resolveGitAuth, signal, options, true);+}++async function inspectRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal: AbortSignal | undefined,+ options: { proveManagedCheckout?: ManagedCheckoutProver }, restorePoint: boolean): Promise<{ recipe: GitCacheRestoration; headOnRemote: boolean }> {if (workspace.mode !== "isolated_workspace" || workspace.strategyType !== "git_worktree"|| !isRuntimeOwnedGitBranch(workspace.metadata) || !workspace.sourceIssueId || !workspace.repoUrl|| !workspace.cwd || !workspace.branchName || (workspace.providerRef && workspace.providerRef !== workspace.cwd)) throw new Error("unproven_workspace_ownership");@@ -378,14 +409,19 @@ export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuthif (!(await fs.lstat(path.join(workspace.cwd, ".git"))).isFile()) throw new Error("not_linked_worktree");if ((await gitCacheCommand(workspace.cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim() !== path.join(await fs.realpath(source), ".git")) throw new Error("source_identity_changed");if ((await gitCacheCommand(workspace.cwd, ["symbolic-ref", "--short", "HEAD"], signal)).trim() !== workspace.branchName) throw new Error("branch_identity_changed");- await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true,+ await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true, restorePoint,verifyFetchedCommit: (commit, verificationSignal) => verifyRemoteGitCommit(workspace.repoUrl!, commit, resolveGitAuth, verificationSignal) });const commit = (await gitCacheCommand(workspace.cwd, ["rev-parse", "HEAD"], signal)).trim();- await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal);- return { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id,+ let headOnRemote = true;+ try { await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal); }+ catch (error) {+ if (!restorePoint || signal?.aborted) throw error;+ headOnRemote = false;+ }+ return { headOnRemote, recipe: { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id,sourceIssueId: workspace.sourceIssueId, repoUrl: workspace.repoUrl, commit, branchName: workspace.branchName,sourceKind: recordedIsolatedRemoteSource(workspace) ? "isolated_remote" : "managed_project", projectId: workspace.projectId,- sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() };+ sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() } };}export function readGitCacheExpiration(workspace: Workspace): GitCacheExpiration | null {@@ -466,7 +502,8 @@ export async function finishGitCacheTrash(workspace: Workspace) {}/** Called only after authorization and while holding the source/lifecycle fences. */-export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider) {+export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider,+ storage?: StorageService) {if (readGitCacheRestoration(workspace)?.commit !== recipe.commit) throw new GitCacheRestorationFailure();try {await finishGitCacheTrash(workspace);@@ -491,6 +528,10 @@ export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCache: (await materializeManagedProjectWorkspace(recipe.sourcePath, { repoUrl: recipe.repoUrl, resolveGitAuth })).cwd;await verifiedGitCacheSource(workspace, recipe.sourcePath);if (!cwd) throw new GitCacheRestorationFailure();+ if (recipe.restorePoint) {+ await ensureGitRestorePointCommit({ storage: storage ?? getStorageService(), workspace: { companyId: workspace.companyId, id: workspace.id },+ ref: recipe.restorePoint, repositoryCwd: cwd });+ }await ensureSavedGitCommit(cwd, recipe.repoUrl, recipe.commit, resolveGitAuth);return cwd;} catch { throw new GitCacheRestorationFailure(); }@@ -514,7 +555,7 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record<string,const raw = input.metadata?.[GIT_CACHE_RESTORATION_KEY];if (raw === undefined) return null;const value = raw as GitCacheRestoration | null;- if (!value || value.schema !== "paperclip.git-cache-restoration.v1" || !shaPattern.test(value.commit)+ if (!value || !restorationSchemaMatches(value) || !shaPattern.test(value.commit)|| value.companyId !== input.companyId || value.workspaceId !== input.workspaceId|| value.worktreePath !== input.cwd || value.sourcePath !== input.baseCwd|| value.repoUrl !== input.repoUrl || value.branchName !== input.branchName@@ -522,6 +563,11 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record<string,|| (value.restoredAt !== undefined && !Number.isFinite(Date.parse(value.restoredAt)))|| (value.contentsRestoredAt != null && !Number.isFinite(Date.parse(value.contentsRestoredAt)))) throw new GitCacheRestorationFailure();validateGitWorkspaceContents(value.contents);- return { commit: value.commit, restored: value.restoredAt !== undefined, contents: value.contents,+ let restorePoint: GitRestorePointRef | null = null;+ if (value.restorePoint) {+ try { restorePoint = validateGitRestorePointRef(value.restorePoint, { companyId: input.companyId, id: input.workspaceId ?? "" }); }+ catch { throw new GitCacheRestorationFailure(); }+ }+ return { commit: value.commit, restored: value.restoredAt !== undefined, contents: value.contents, restorePoint,contentsRestored: value.contentsRestoredAt === undefined ? value.restoredAt !== undefined : value.contentsRestoredAt !== null };}diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.tsindex 2ff23d1956..3171e5e5a3 100644--- a/server/src/services/workspace-runtime.ts+++ b/server/src/services/workspace-runtime.ts@@ -1,6 +1,9 @@import { insertIssueComment } from "./issue-comment-writes.js";import { restoreGitWorkspaceContents } from "./git-workspace-contents.js";import { savedGitWorkspaceRestoration, ensureSavedGitCommit, GitCacheRestorationFailure } from "./git-workspace-restoration.js";+import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit } from "./git-restore-point-storage.js";+import { getStorageService } from "../storage/index.js";+import type { StorageService } from "../storage/types.js";import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";export { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";@@ -3779,6 +3782,8 @@ function isPersistedTaskDirectory(workspace: {export async function ensurePersistedExecutionWorkspaceAvailable(input: {prepareRestoredLocalSource?: (sourceCwd: string) => Promise<() => Promise<void>>;persistContentsRestoredAt?: (at: string | null) => Promise<void>;+ persistRestorePointAppliedAt?: (at: string) => Promise<void>;+ restorePointStorage?: StorageService;db?: Db | null;base: ExecutionWorkspaceInput;workspace: {@@ -3880,6 +3885,27 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {currentRecipe = { ...currentRecipe as Record<string, unknown>, contentsRestoredAt: at };input.workspace.metadata!.gitCacheRestoration = currentRecipe;};+ const applyRestorePoint = async (repositoryCwd: string, worktreePath: string) => {+ const ref = restoration?.restorePoint;+ if (!ref || ref.appliedAt || !input.workspace.id) return;+ await applyGitRestorePointToWorktree({ storage: input.restorePointStorage ?? getStorageService(),+ workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref, repositoryCwd, worktreePath })+ .catch(() => { throw new GitCacheRestorationFailure(); });+ const at = new Date().toISOString();+ if (input.persistRestorePointAppliedAt) await input.persistRestorePointAppliedAt(at);+ else {+ if (!input.db) throw new GitCacheRestorationFailure();+ const updated = await input.db.update(executionWorkspaces).set({+ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,appliedAt}', ${JSON.stringify(at)}::jsonb)`,+ }).where(and(eq(executionWorkspaces.id, input.workspace.id), eq(executionWorkspaces.companyId, input.agent.companyId),+ sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'id' = ${ref.id}`))+ .returning({ id: executionWorkspaces.id });+ if (updated.length !== 1) throw new GitCacheRestorationFailure();+ }+ ref.appliedAt = at;+ currentRecipe = { ...currentRecipe as Record<string, unknown>, restorePoint: { ...ref } };+ input.workspace.metadata!.gitCacheRestoration = currentRecipe;+ };const restoreContents = async () => {if (!restoration?.contents || !input.workspace.id) return;await restoreGitWorkspaceContents({ cwd, companyId: input.agent.companyId, workspaceId: input.workspace.id,@@ -3980,6 +4006,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings];realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha;await input.afterSourceCopy?.(repoRoot, realized.worktreePath ?? cwd);+ await applyRestorePoint(repoRoot, realized.worktreePath ?? cwd);if (restoration && !restoration.contentsRestored) await restoreContents();await provisionExecutionWorktree({skipHostProvision: input.skipHostProvision,@@ -4009,6 +4036,11 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {await fs.mkdir(path.dirname(worktreePath), { recursive: true });await runGit(["worktree", "prune"], repoRoot).catch(() => {});if (savedCommit) {+ if (restoration?.restorePoint && input.workspace.id) {+ await ensureGitRestorePointCommit({ storage: input.restorePointStorage ?? getStorageService(),+ workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref: restoration.restorePoint, repositoryCwd: repoRoot })+ .catch(() => { throw new GitCacheRestorationFailure(); });+ }await ensureSavedGitCommit(repoRoot, input.workspace.repoUrl ?? input.base.repoUrl!, savedCommit, input.resolveGitAuth ?? (async () => null));const existingTip = await resolveBaseRefSha(repoRoot, `refs/heads/${branchName}`);// Once restored, subsequent task commits own this branch. The saved commit@@ -4083,6 +4115,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {});await input.afterSourceCopy?.(repoRoot, worktreePath);+ await applyRestorePoint(repoRoot, worktreePath);await restoreContents();await provisionExecutionWorktree({skipHostProvision: input.skipHostProvision,