Skip to content

The open-source app everyone uses to manage agents at work

fix: reuse verified seeds after warm workspace fallback (#16398)

Dotta5bc4e878e791
diff --git a/doc/plans/2026-10-09-task-workspaces.md b/doc/plans/2026-10-09-task-workspaces.md
index c12873ddc1..e0cfdaf0e9 100644
--- a/doc/plans/2026-10-09-task-workspaces.md
+++ b/doc/plans/2026-10-09-task-workspaces.md
@@ -67,7 +67,7 @@ Preserve simultaneous local/SSH tasks in an explicitly configured identical phys
Concurrent remote sharing requires proven common physical placement, separate session authority and coherent snapshot capability. Unsupported new concurrent sharing returns a typed unavailable result; do not instantiate separate clones and label them shared. Already-admitted descriptors remain recoverable unchanged. Local authoritative roots need no replica copyback; SSH retains its actual existing persistence contract.
-Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest.
+Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. If warm identity validation requires a host-current transfer, that fallback retains the same company/workspace seed-cache scope, so a verified generation can supply the transfer without a second Git clone and archive rebuild. Missing or invalid generations still rebuild from the host. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest.
A returned sparse snapshot must describe a coherent tree before any host mutation. Every nested entry requires an immediate parent marked as a directory in that same returned snapshot, including baseline-unchanged files. Checking every directory also verifies recursive ancestry. Top-level entries use the implicit root. Invalid trees fail without overwriting retained host files and keep recovery evidence. The check uses the existing indexed manifest interface; it adds no host-path scans or unbounded in-memory inventory.
diff --git a/server/src/__tests__/native-workspace-sync-history.test.ts b/server/src/__tests__/native-workspace-sync-history.test.ts
index 56b4a66c2f..41d391df2d 100644
--- a/server/src/__tests__/native-workspace-sync-history.test.ts
+++ b/server/src/__tests__/native-workspace-sync-history.test.ts
@@ -1,12 +1,13 @@
import { execFile } from "node:child_process";
-import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
-import { afterEach, describe, expect, it } from "vitest";
+import { afterEach, describe, expect, it, vi } from "vitest";
import { heartbeatRuns, type Db } from "@paperclipai/db";
import type { EnvironmentLease } from "@paperclipai/shared";
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
+import * as gitWorkspaceSync from "@paperclipai/adapter-utils/git-workspace-sync";
import { runLocalGit } from "@paperclipai/adapter-utils/git-workspace-sync";
import { prepareNativeWorkspaceSync, nativeWorkspaceSyncInternals } from "../services/native-runtime/native-workspace-sync.js";
@@ -37,6 +38,7 @@ describe("native warm workspace Git history", () => {
const oldInstance = process.env.PAPERCLIP_INSTANCE_ID;
const roots: string[] = [];
afterEach(async () => {
+ vi.restoreAllMocks();
if (oldHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = oldHome;
if (oldInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
@@ -44,8 +46,9 @@ describe("native warm workspace Git history", () => {
await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true })));
});
- it.each(["unchanged", "host_commit", "host_branch", "host_origin", "remote_commit", "nested_commit", "plain_nested_commit"] as const)(
+ it.each(["unchanged", "host_commit", "host_branch", "host_origin", "remote_commit", "remote_commit_corrupt_seed", "remote_commit_missing_seed", "nested_commit", "plain_nested_commit"] as const)(
"checks Git identity before reusing a warm sandbox: %s", async (change) => {
+ const packClone = vi.spyOn(gitWorkspaceSync, "withShallowGitWorkspaceClone");
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-warm-history-"));
roots.push(root);
process.env.PAPERCLIP_HOME = path.join(root, "home");
@@ -110,19 +113,30 @@ describe("native warm workspace Git history", () => {
await git(host, "checkout", "-b", "other");
} else if (change === "host_origin") {
await git(host, "remote", "add", "origin", "https://github.com/paperclipai/new-source.git");
- } else if (change === "remote_commit") {
+ } else if (change === "remote_commit" || change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") {
await git(remote, "-c", "user.name=Test", "-c", "[email protected]", "commit", "--allow-empty", "-m", "remote only");
}
+ if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") {
+ const cache = path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "workspace");
+ const [generation] = (await readdir(cache)).filter(name => /^[a-f0-9]{64}$/.test(name));
+ expect(generation).toBeDefined();
+ const archive = path.join(cache, generation!, "workspace.tar");
+ await rm(archive);
+ if (change === "remote_commit_corrupt_seed") await writeFile(archive, "invalid cached archive");
+ }
const hostRepo = change === "plain_nested_commit" ? path.join(host, nestedPath) : host;
const remoteRepo = change === "plain_nested_commit" ? path.join(remote, nestedPath) : remote;
const expectedHead = await git(hostRepo, "rev-parse", "HEAD");
const expectedBranch = await git(hostRepo, "symbolic-ref", "--short", "HEAD");
const nestedHead = change === "nested_commit" || change === "plain_nested_commit" ? await git(path.join(host, nestedPath), "rev-parse", "HEAD") : null;
runnerProfileJson = {};
+ packClone.mockClear();
const second = (await prepareNativeWorkspaceSync({ ...input, runId: "second", target: {
...target, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" },
} }))!;
try {
+ if (change === "remote_commit") expect(packClone).not.toHaveBeenCalled();
+ if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") expect(packClone).toHaveBeenCalled();
expect(second.mode).toBe(change === "unchanged" ? "adopt_remote" : "host_current");
expect(await git(remoteRepo, "rev-parse", "HEAD")).toBe(expectedHead);
expect(await git(remoteRepo, "symbolic-ref", "--short", "HEAD")).toBe(expectedBranch);
diff --git a/server/src/services/native-runtime/native-workspace-sync.ts b/server/src/services/native-runtime/native-workspace-sync.ts
index 6841f9f132..192be86c37 100644
--- a/server/src/services/native-runtime/native-workspace-sync.ts
+++ b/server/src/services/native-runtime/native-workspace-sync.ts
@@ -971,6 +971,8 @@ export async function prepareNativeWorkspaceSync(input: {
acquisition,
hasPriorStamp: priorStamp !== null,
});
+ const seedCacheCompanyDirectory = path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME, "generations", requireSafeSegment(input.companyId, "company_id"));
+ const seedCacheDirectory = path.join(seedCacheCompanyDirectory, requireSafeSegment(input.workspaceId, "workspace_id"));
runtime = await prepareRuntime({
runId: input.runId,
target,
@@ -978,8 +980,8 @@ export async function prepareNativeWorkspaceSync(input: {
onCheckpoint: input.onCheckpoint,
workspaceLocalDir: input.workspaceLocalDir,
mode,
- seedCacheCompanyDirectory: path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME, "generations", requireSafeSegment(input.companyId, "company_id")),
- seedCacheDirectory: path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME, "generations", requireSafeSegment(input.companyId, "company_id"), requireSafeSegment(input.workspaceId, "workspace_id")),
+ seedCacheCompanyDirectory,
+ seedCacheDirectory,
durableSeed: {
workspaceArchivePath: seedPaths.workspaceArchivePath,
gitArchivePath: seedPaths.gitArchivePath,
@@ -1005,6 +1007,8 @@ export async function prepareNativeWorkspaceSync(input: {
target,
workspaceLocalDir: input.workspaceLocalDir,
mode,
+ seedCacheCompanyDirectory,
+ seedCacheDirectory,
baseline: currentSnapshot.baseline,
gitSnapshot: currentSnapshot.gitSnapshot,
repositories: currentSnapshot.repositories,