Skip to content

The open-source app everyone uses to manage agents at work

Add lease-scoped Runner task hooks to environment plugins (#15406)

Nicky Leach33cc567f75ae
diff --git a/doc/plugins/PLUGIN_AUTHORING_GUIDE.md b/doc/plugins/PLUGIN_AUTHORING_GUIDE.md
index f4b4c06a8d..e6913a203d 100644
--- a/doc/plugins/PLUGIN_AUTHORING_GUIDE.md
+++ b/doc/plugins/PLUGIN_AUTHORING_GUIDE.md
@@ -791,3 +791,10 @@ The host resets plugin state on account/company changes and keeps its built-in
menu when no unique contribution exists, discovery fails, the module is missing,
or rendering throws. The slot is a React-only contract; do not use a custom
element export. This replaces only the menu, not company policy or authorization.
+
+## Runner task worker hooks
+
+A plugin can declare `supportsTasks` on an environment driver and implement
+`onEnvironmentTask`. See the [task API](PLUGIN_ENVIRONMENT_TASKS.md) for the
+operations, validation, and lease rules. This hook does not add a shell or file
+transfer API. Native agent execution is a separate integration.
diff --git a/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md b/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md
new file mode 100644
index 0000000000..efbd7b3aa4
--- /dev/null
+++ b/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md
@@ -0,0 +1,95 @@
+# Environment task API
+
+An environment driver can provide a Runner task through a worker hook.
+The task API has no shell command or file transfer operation.
+
+This API is a server and worker contract. It does not select an environment
+for a native agent or connect a Runner to a run. Those integrations are separate.
+
+## Declare the hook
+
+Set `supportsTasks: true` on the environment driver in the plugin manifest.
+Implement `onEnvironmentTask` with the SDK types:
+
+```ts
+import type {
+ PluginEnvironmentTaskParams,
+ PluginEnvironmentTaskResult,
+} from "@paperclipai/plugin-sdk";
+
+type TaskHook = (
+ params: PluginEnvironmentTaskParams,
+) => Promise<PluginEnvironmentTaskResult>;
+```
+
+Core requires the `environment.drivers.register` capability, a ready plugin,
+the manifest declaration, and the worker's `environmentTask` method.
+The SDK validates each operation and its result before it returns the result.
+See [the schemas](../../packages/plugins/sdk/src/environment-tasks.ts) for all fields.
+
+## Operations
+
+| Operation | Result | Required behavior |
+| --- | --- | --- |
+| `describe` | `description` | Return the installed Runner version and SHA-256 digest. |
+| `submit` | `accepted` | Start the same opaque task at most once. Use the structured launch inputs. |
+| `connection` | `connection` | Return an authenticated WSS endpoint and its generation. |
+| `status` | `status` | Report the phase and observed process termination. |
+| `credentials` | `credentials` | Return the selected account after process termination. |
+| `stop` | `accepted` | Stop the task and its descendants. Fence a delayed submission. |
+| `complete` | `accepted` | Release task resources after Core saves required results. |
+
+Each result must contain the task ID from the request. Core rejects a result
+with a different ID or result kind. A WSS URL must have no user information,
+query, or fragment. Return authentication in the secret headers.
+
+`submit` contains the PRP version range, Runner identity, bootstrap ticket,
+authorized project IDs, and optional launch limits. It can also contain a saved
+task ID for session reuse and a typed Codex or Claude credential. Core does not
+supply an executable command or a guest path.
+
+## Lease authority
+
+Core calls `environmentRuntimeService.task` with a company ID, a saved lease ID,
+and an operation. It derives the plugin and driver from the lease. A change to
+the environment configuration cannot redirect cleanup to a new provider.
+
+Before submission, Core checks the active lease, run, issue, and every project
+in the company. It checks the submitting run ID separately from a retained PRP
+identity. Session reuse requires the same agent, environment, workspace, and
+provider. It also requires a retained source task and proof of termination.
+
+An internal setup probe can use a private, time-limited lease without a run row.
+Core checks the saved probe identity, rejects project mounts, and limits the
+Runner lifetime to the remaining probe time.
+
+## Failure and cleanup
+
+A failed submission reply does not prove that the task failed to start.
+Core saves the dispatch marker before it calls the worker. Retry against the
+same task ID. Do not allocate another task to resolve an uncertain reply.
+
+An accepted stop does not prove termination. Set `executionStopped: true`
+only after the Runner and all descendants have stopped. A terminal phase alone
+does not permit Core to release execution authority.
+
+Core fences new submissions when cleanup starts. It stops the saved task,
+checks termination, saves credential results, and completes the task before it
+releases or destroys the environment lease. An uncertain result leaves a saved
+lease for the cleanup sweep. Cleanup can use that lease after the environment
+or run row is deleted.
+
+## Credential writeback
+
+Account selection and credential revision have separate identities. Keep the
+selected account fixed for the task. A credential result of `null` means that
+cancellation prevented task admission. It must not mean that an export failed.
+
+Core saves the account grant, secret reference, and session epoch on the lease.
+It saves no credential bytes or temporary path in lease metadata. After process
+termination, it can recover a Codex refresh even if Core restarted. It uses the
+existing account comparison and secret service. A revoked grant, replaced
+account, or changed session epoch discards the old refresh.
+
+Keep file synchronization and backup outside this API. The provider must retain
+recoverable credential state until task completion succeeds.
diff --git a/packages/plugins/sdk/src/define-plugin.ts b/packages/plugins/sdk/src/define-plugin.ts
index 3d7d81a31b..13096cbcf8 100644
--- a/packages/plugins/sdk/src/define-plugin.ts
+++ b/packages/plugins/sdk/src/define-plugin.ts
@@ -1,3 +1,4 @@
+import type { PluginEnvironmentTaskParams, PluginEnvironmentTaskResult } from "./environment-tasks.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
/**
@@ -401,6 +402,9 @@ export interface PluginDefinition {
params: PluginEnvironmentValidateConfigParams,
): Promise<PluginEnvironmentValidationResult>;
+ /** Provide Runner execution: prepare resources, start Runner, connect via PRP, and manage task lifecycle. */
+ onEnvironmentTask?(params: PluginEnvironmentTaskParams): Promise<PluginEnvironmentTaskResult>;
+
/** Called to test reachability or readiness of a plugin-hosted environment. */
onEnvironmentProbe?(
params: PluginEnvironmentProbeParams,
diff --git a/packages/plugins/sdk/src/environment-tasks.ts b/packages/plugins/sdk/src/environment-tasks.ts
new file mode 100644
index 0000000000..ae2cd9e7ee
--- /dev/null
+++ b/packages/plugins/sdk/src/environment-tasks.ts
@@ -0,0 +1,114 @@
+import { z } from "zod";
+import type { PluginEnvironmentDriverBaseParams, PluginEnvironmentLease } from "./protocol.js";
+
+// Provider IDs are opaque; providers validate their own addressing constraints.
+const providerId = z.string().min(1);
+// PRP identity.schema.json stableId.
+const runnerIdentity = z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$/);
+const identifier = z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,95}$/);
+const secureUrl = z.string().url().refine(value => {
+ const url = new URL(value);
+ return url.protocol === "wss:" && !url.username && !url.password && !url.search && !url.hash;
+}, "Expected a credential-free WSS URL");
+
+const credentialValue = z.string().min(1).max(65_536).refine(value => !value.includes("\0"));
+/** Only provider authentication is transferred. Paths and executable settings remain provider-owned. */
+export const environmentTaskCredentialSchema = z.discriminatedUnion("type", [
+ z.object({ type: z.literal("codex-auth-json"), authJson: credentialValue }).strict(),
+ z.object({ type: z.literal("claude-env"), environment: z.object({
+ CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
+ }).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential") }).strict(),
+]);
+export type PluginEnvironmentTaskCredential = z.infer<typeof environmentTaskCredentialSchema>;
+
+/** One plugin-provided Runner execution attempt with a PRP connection. Secrets are transient RPC input. */
+export const environmentTaskOperationSchema = z.discriminatedUnion("kind", [
+ z.object({ kind: z.literal("describe") }).strict(),
+ z.object({
+ kind: z.literal("submit"),
+ projectIds: z.array(z.string().uuid()).max(64).refine(ids => new Set(ids).size === ids.length, "Duplicate project IDs"),
+ runner: z.object({
+ /** Inclusive PRP version range supported by the submitting client. */
+ protocolMin: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
+ protocolMax: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
+ harness: identifier,
+ runnerId: runnerIdentity, leaseId: runnerIdentity, runId: runnerIdentity,
+ sessionId: runnerIdentity, turnId: runnerIdentity, itemId: runnerIdentity,
+ }).strict().refine(value => value.protocolMin <= value.protocolMax, "Invalid PRP version range"),
+ bootstrapTicket: z.string().min(1).max(65_536),
+ /** Opaque prior provider task; Core authorizes the saved session before supplying it. */
+ resumeFrom: z.string().min(1).optional(),
+ /** Require the externally installed account to match this Core-selected credential revision. */
+ credentialIdentity: z.string().min(1).max(1024).optional(),
+ /** Stable account selection; distinct from its current credential revision. */
+ accountIdentity: z.string().min(1).max(1024).optional(),
+ credential: environmentTaskCredentialSchema.optional(),
+ /** Core supplies these protected launch inputs for native agent runs. */
+ launch: z.object({
+ runnerVersion: z.string().min(1), runnerDigest: z.string().regex(/^sha256:[a-f0-9]{64}$/),
+ maxOutboxBytes: z.number().int().positive(), p0ReserveBytes: z.number().int().positive(),
+ maxRuntimeMs: z.number().int().nonnegative().optional(), maxLifetimeMs: z.number().int().nonnegative().optional(),
+ reconnectGraceMs: z.number().int().nonnegative().optional(),
+ lifecyclePolicy: z.object({ mode: z.literal("per_turn"), idleTimeoutMs: z.null() }).strict(),
+ }).strict().optional(),
+ }).strict(),
+ z.object({ kind: z.literal("credentials"), accountIdentity: z.string().min(1).max(1024) }).strict(),
+ z.object({ kind: z.literal("status") }).strict(),
+ z.object({ kind: z.literal("connection") }).strict(),
+ z.object({ kind: z.literal("complete") }).strict(),
+ z.object({ kind: z.literal("stop") }).strict(),
+]);
+
+export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
+ z.object({ kind: z.literal("accepted"), taskId: providerId }).strict(),
+ z.object({ kind: z.literal("credentials"), taskId: providerId, /** Null proves that cancellation fenced admission before the task could start. It must not mean an unavailable export. */
+ credential: environmentTaskCredentialSchema.nullable() }).strict(),
+ z.object({ kind: z.literal("description"), taskId: providerId,
+ runner: z.object({ version: z.string().min(1), digest: z.string().regex(/^sha256:[a-f0-9]{64}$/) }).strict(),
+ }).strict(),
+ z.object({
+ kind: z.literal("status"), taskId: providerId,
+ phase: z.enum(["preparing", "running", "completed", "failed", "cancelled", "interrupted"]),
+ exitCode: z.number().int().optional(),
+ /** Provider observed the task and all descendants stopped; terminal phase alone is insufficient. */
+ executionStopped: z.boolean().optional(),
+ /** The provider retains this task's suspended session for a later launch. */
+ sessionStateAvailable: z.boolean().optional(),
+ }).strict(),
+ z.object({
+ kind: z.literal("connection"), taskId: providerId,
+ endpoint: z.object({
+ kind: z.literal("authenticated_websocket"), websocketUrl: secureUrl,
+ generation: providerId,
+ secretHeaders: z.array(z.object({
+ name: z.string().regex(/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/),
+ value: z.string().min(1).max(65_536).regex(/^[^\r\n]+$/),
+ }).strict()).max(16),
+ }).strict(),
+ }).strict(),
+]);
+
+export type PluginEnvironmentTaskOperation = z.infer<typeof environmentTaskOperationSchema>;
+export type PluginEnvironmentTaskResult = z.infer<typeof environmentTaskResultSchema>;
+
+export interface PluginEnvironmentTaskParams extends Omit<PluginEnvironmentDriverBaseParams, "environmentId"> {
+ /** Null during cleanup after the environment is deleted. */
+ environmentId: string | null;
+ lease: PluginEnvironmentLease;
+ /** Provider-issued task identifier persisted in the lease, stable across ambiguous submission retries. */
+ taskId: string;
+ /** Null during cleanup after the associated run is deleted. */
+ runId: string | null;
+ agentId: string | null;
+ /** Company-validated projects to prepare on submit; empty during cleanup. */
+ projectIds: string[];
+ operation: PluginEnvironmentTaskOperation;
+}
+
+/** An accepted stop is not proof of process termination. */
+export function parseEnvironmentTaskResult(operation: PluginEnvironmentTaskOperation, taskId: string, value: unknown): PluginEnvironmentTaskResult {
+ const result = environmentTaskResultSchema.parse(value);
+ const expected = operation.kind === "describe" ? "description" : ["status", "connection", "credentials"].includes(operation.kind) ? operation.kind : "accepted";
+ if (result.taskId !== taskId || result.kind !== expected) throw new Error("Invalid plugin-provided Runner execution response");
+ return result;
+}
diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.ts
index 8698d6ba4b..9b74f554d8 100644
--- a/packages/plugins/sdk/src/index.ts
+++ b/packages/plugins/sdk/src/index.ts
@@ -463,4 +463,6 @@ export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-er
export type { AiConnectionPool, AiConnectionPoolConfig, AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, AiConnectionRouterSelection } from "@paperclipai/shared";
+export { environmentTaskOperationSchema, environmentTaskResultSchema, environmentTaskCredentialSchema, parseEnvironmentTaskResult } from "./environment-tasks.js";
+export type { PluginEnvironmentTaskOperation, PluginEnvironmentTaskParams, PluginEnvironmentTaskResult, PluginEnvironmentTaskCredential } from "./environment-tasks.js";
export type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts
index 83e8d88b09..b6cb01a679 100644
--- a/packages/plugins/sdk/src/protocol.ts
+++ b/packages/plugins/sdk/src/protocol.ts
@@ -1,3 +1,4 @@
+import type { PluginEnvironmentTaskParams, PluginEnvironmentTaskResult } from "./environment-tasks.js";
import type { PluginAgentRuntimeState } from "./types.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
@@ -1393,6 +1394,7 @@ export interface HostToWorkerMethods {
params: PluginEnvironmentValidateConfigParams,
result: PluginEnvironmentValidationResult,
];
+ environmentTask: [params: PluginEnvironmentTaskParams, result: PluginEnvironmentTaskResult];
environmentProbe: [
params: PluginEnvironmentProbeParams,
result: PluginEnvironmentProbeResult,
@@ -1521,6 +1523,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[]
"resolveExternalObject",
"refreshExternalObjects",
"environmentValidateConfig",
+ "environmentTask",
"environmentProbe",
"environmentAcquireLease",
"environmentResumeLease",
diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.ts
index 3de92ac521..8b98a83e64 100644
--- a/packages/plugins/sdk/src/worker-rpc-host.ts
+++ b/packages/plugins/sdk/src/worker-rpc-host.ts
@@ -1,3 +1,4 @@
+import { environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskParams } from "./environment-tasks.js";
import type { AgentLifecycleRequest, AgentLifecycleResult } from "@paperclipai/shared";
import type { AiConnectionRouterRequest, AiConnectionRouterResult } from "@paperclipai/shared";
import { environmentCreationCleanupErrorData } from "./environment-creation-cleanup.js";
@@ -1682,6 +1683,13 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
case "environmentValidateConfig":
return handleEnvironmentValidateConfig(params as PluginEnvironmentValidateConfigParams);
+ case "environmentTask": {
+ if (!plugin.definition.onEnvironmentTask) throw methodNotImplemented("environmentTask");
+ const input = params as PluginEnvironmentTaskParams;
+ const operation = environmentTaskOperationSchema.parse(input.operation);
+ return parseEnvironmentTaskResult(operation, input.taskId, await plugin.definition.onEnvironmentTask({ ...input, operation }));
+ }
+
case "environmentProbe":
return handleEnvironmentProbe(params as PluginEnvironmentProbeParams);
@@ -1801,6 +1809,7 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
if (plugin.definition.onResolveExternalObject) supportedMethods.push("resolveExternalObject");
if (plugin.definition.onRefreshExternalObjects) supportedMethods.push("refreshExternalObjects");
if (plugin.definition.onEnvironmentValidateConfig) supportedMethods.push("environmentValidateConfig");
+ if (plugin.definition.onEnvironmentTask) supportedMethods.push("environmentTask");
if (plugin.definition.onEnvironmentProbe) supportedMethods.push("environmentProbe");
if (plugin.definition.onEnvironmentAcquireLease) supportedMethods.push("environmentAcquireLease");
if (plugin.definition.onEnvironmentResumeLease) supportedMethods.push("environmentResumeLease");
diff --git a/packages/shared/src/types/plugin.ts b/packages/shared/src/types/plugin.ts
index 384f9e9b4c..bbfb2c97a0 100644
--- a/packages/shared/src/types/plugin.ts
+++ b/packages/shared/src/types/plugin.ts
@@ -191,6 +191,8 @@ export interface SandboxProviderCapabilities {
}
export interface PluginEnvironmentDriverDeclaration {
+ /** Provides Runner execution, resource preparation, and PRP connections via environmentTask. */
+ supportsTasks?: boolean;
/** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */
driverKey: string;
/**
diff --git a/packages/shared/src/validators/plugin.ts b/packages/shared/src/validators/plugin.ts
index ca6fa07963..5cc7d174c4 100644
--- a/packages/shared/src/validators/plugin.ts
+++ b/packages/shared/src/validators/plugin.ts
@@ -177,6 +177,7 @@ export type SandboxProviderCapabilitiesInput = z.infer<typeof sandboxProviderCap
// validation error, not a silently dropped field. A misspelled capability name
// (for example `supportsLoginPTY`) fails validation instead of dropping.
export const pluginEnvironmentDriverDeclarationSchema = z.object({
+ supportsTasks: z.boolean().optional(),
driverKey: z.string().min(1).regex(
/^[a-z0-9][a-z0-9._-]*$/,
"Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
diff --git a/server/src/__tests__/environment-task-authority.integration.test.ts b/server/src/__tests__/environment-task-authority.integration.test.ts
new file mode 100644
index 0000000000..4ddda4f3c8
--- /dev/null
+++ b/server/src/__tests__/environment-task-authority.integration.test.ts
@@ -0,0 +1,108 @@
+import { randomUUID } from "node:crypto";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { eq } from "drizzle-orm";
+import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins } from "@paperclipai/db";
+import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
+import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
+import { environmentRuntimeService } from "../services/environment-runtime.js";
+import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+
+const support = await getEmbeddedPostgresTestSupport();
+const databaseSuite = support.supported ? describe : describe.skip;
+
+databaseSuite("environment task lease authority", () => {
+ let db: ReturnType<typeof createDb>;
+ let stopDb: (() => Promise<void>) | undefined;
+ beforeAll(async () => {
+ const started = await startEmbeddedPostgresTestDatabase("environment-task-authority");
+ stopDb = started.stop; db = createDb(started.connectionString);
+ });
+ afterAll(async () => { await stopDb?.(); });
+
+ it("authorizes a new task lease only after scoped prior execution has stopped", async () => {
+ const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
+ const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
+ await db.insert(companies).values({ id: companyId, name: "Lease transition", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", role: "engineer", adapterType: "paperclip_runner" });
+ await db.insert(environments).values({ id: environmentId, companyId, name: `Fixture ${environmentId}`, driver: "plugin", config: {} });
+ await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-continuation", version: "1.0.0", status: "ready", manifestJson: {
+ capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
+ } as never });
+ await db.insert(heartbeatRuns).values([
+ { id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" },
+ { id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" },
+ ]);
+ const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true };
+ const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" };
+ await db.insert(environmentLeases).values([
+ { ...prior, status: "released", leasePolicy: "ephemeral", cleanupStatus: "success", releasedAt: new Date(),
+ metadata: { ...pin, runnerTaskSessionRetained: true, runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session" }, remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) } },
+ { id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task",
+ status: "active", leasePolicy: "ephemeral", metadata: pin },
+ ]);
+ const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId };
+ const authorize = () => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity });
+ await expect(authorize()).resolves.toBe("old-task");
+ for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) {
+ await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId,
+ priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow("not authorized");
+ }
+ await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId));
+ await expect(authorize()).rejects.toThrow("not authorized");
+ await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId));
+ await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow("not authorized");
+ await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId));
+ await expect(authorize()).rejects.toThrow("not authorized");
+ await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId));
+ await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId));
+ await expect(authorize()).rejects.toThrow("not authorized");
+ await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
+ const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
+ call: async (_id: string, _method: string, params: any) => ({ kind: "accepted", taskId: params.taskId }),
+ } as unknown as PluginWorkerManager;
+ const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
+ const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
+ turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
+ await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
+ kind: "submit", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
+ } })).resolves.toMatchObject({ kind: "accepted" });
+ const [second] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, nextLeaseId));
+ await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
+ metadata: { ...second.metadata, runnerTaskSessionRetained: true,
+ remoteExecutionTermination: remoteTerminationReceipt(second, { providerLeaseId: "new-task", state: "stopped" }) },
+ }).where(eq(environmentLeases.id, nextLeaseId));
+ await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId));
+ const thirdRunId = randomUUID(), thirdLeaseId = randomUUID();
+ await db.insert(heartbeatRuns).values({ id: thirdRunId, companyId, agentId, invocationSource: "manual", status: "running" });
+ await db.insert(environmentLeases).values({ id: thirdLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
+ status: "active", provider: prior.provider, providerLeaseId: "third-task", metadata: pin });
+ await expect(resolvePluginTaskResume(db, { companyId, leaseId: thirdLeaseId,
+ priorIdentity: { ...priorIdentity, runId: nextRunId } })).resolves.toBe("new-task");
+ const submit = { kind: "submit", projectIds: [], bootstrapTicket: "fixture", runner: { ...runner, runId: thirdRunId }, resumeFrom: "new-task" } as const;
+ await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [] } })).resolves.toMatchObject({ kind: "accepted" });
+ await expect(runtime.task({ companyId, leaseId: thirdLeaseId, operation: { ...submit, projectIds: [], runner } })).rejects.toThrow("identity mismatch");
+ // Same-run recovery must choose the latest task in the saved lineage, not an earlier physical lease.
+ const [third] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, thirdLeaseId));
+ await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
+ metadata: { ...third.metadata, runnerTaskSessionRetained: true,
+ remoteExecutionTermination: remoteTerminationReceipt(third, { providerLeaseId: "third-task", state: "stopped" }) },
+ }).where(eq(environmentLeases.id, thirdLeaseId));
+ const retryLeaseId = randomUUID();
+ await db.insert(environmentLeases).values({ id: retryLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
+ status: "active", provider: prior.provider, providerLeaseId: "retry-task", metadata: pin });
+ await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: { ...submit, projectIds: [], resumeFrom: "third-task" } })).resolves.toMatchObject({ kind: "accepted" });
+ const [retry] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, retryLeaseId));
+ await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
+ metadata: { ...retry.metadata, runnerTaskSessionRetained: true,
+ remoteExecutionTermination: remoteTerminationReceipt(retry, { providerLeaseId: "retry-task", state: "stopped" }) },
+ }).where(eq(environmentLeases.id, retryLeaseId));
+ const finalLeaseId = randomUUID();
+ await db.insert(environmentLeases).values({ id: finalLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
+ status: "active", provider: prior.provider, providerLeaseId: "final-task", metadata: pin });
+ await expect(resolvePluginTaskResume(db, { companyId, leaseId: finalLeaseId,
+ priorIdentity: { ...priorIdentity, runId: thirdRunId } })).resolves.toBe("retry-task");
+
+ });
+
+});
diff --git a/server/src/__tests__/environment-task-runtime.test.ts b/server/src/__tests__/environment-task-runtime.test.ts
new file mode 100644
index 0000000000..fec9d2c2d7
--- /dev/null
+++ b/server/src/__tests__/environment-task-runtime.test.ts
@@ -0,0 +1,199 @@
+import { beforeEach, describe, expect, it, vi } from "vitest";
+import { PgDialect } from "drizzle-orm/pg-core";
+import type { Db } from "@paperclipai/db";
+import { environmentTaskOperationSchema, parseEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
+import { executeEnvironmentTask } from "../services/environment-task-runtime.js";
+
+const state = vi.hoisted(() => ({ plugin: {} as any }));
+vi.mock("../services/plugin-registry.js", () => ({ pluginRegistryService: () => ({ getById: vi.fn(async () => state.plugin) }) }));
+const leaseId = "10000000-0000-4000-8000-000000000001";
+const projectId = "10000000-0000-4000-8000-000000000002";
+const secondProjectId = "10000000-0000-4000-8000-000000000003";
+const row = () => ({
+ lease: { id: leaseId, companyId: "company", environmentId: "environment", providerLeaseId: "attempt-1", heartbeatRunId: "run", issueId: "issue", status: "active", expiresAt: null as Date | null,
+ metadata: { driver: "plugin", pluginId: "original", driverKey: "tasks" } },
+ environment: { id: "environment", config: { pluginKey: "test.provider", driverKey: "tasks", driverConfig: {} } } as { id: string; config: Record<string, unknown> } | null,
+ run: { id: "run", agentId: "agent", status: "running" } as { id: string; agentId: string; status: string } | null,
+});
+function database(value: ReturnType<typeof row> | null = row(), projectRows = [{ id: projectId }]) {
+ const results = [value ? [value] : [], [{ id: "issue" }]];
+ const query: any = { from: () => query, leftJoin: () => query, where: vi.fn(() => query), limit: () => Promise.resolve(results.shift()), then: (resolve: (rows: typeof projectRows) => unknown) => Promise.resolve(projectRows).then(resolve) };
+ const update: any = { set: () => update, where: () => update,
+ returning: async () => [{ id: leaseId }], then: (resolve: (value: unknown) => unknown) => Promise.resolve([]).then(resolve) };
+ return { db: { select: () => query, update: () => update } as unknown as Db, query };
+}
+function worker(result: unknown = { kind: "accepted", taskId: "attempt-1" }) {
+ return { getWorker: vi.fn(() => ({ supportedMethods: ["environmentTask"] })), call: vi.fn(async () => result) };
+}
+const submit = { kind: "submit" as const, projectIds: [projectId], runner: { protocolMin: 1, protocolMax: 2, harness: "codex", runnerId: "runner", leaseId, runId: "run", sessionId: "session", turnId: "turn", itemId: "item" }, bootstrapTicket: "transient-test-ticket" };
+
+beforeEach(() => {
+ state.plugin = { id: "original", pluginKey: "test.provider", status: "ready", manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } };
+});
+describe("plugin-provided Runner execution", () => {
+ it("accepts artifact identity and excludes filesystem transfer from the contract", () => {
+ const operation = environmentTaskOperationSchema.parse({ kind: "describe" });
+ const description = { kind: "description", taskId: "attempt-1", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
+ expect(parseEnvironmentTaskResult(operation, "attempt-1", description)).toEqual(description);
+ for (const field of ["resources", "environment", "connection", "codexCommand"]) {
+ expect(() => parseEnvironmentTaskResult(operation, "attempt-1", { ...description, [field]: {} })).toThrow();
+ }
+ for (const kind of ["checkpoint", "read_file", "check_project_auth"]) {
+ expect(environmentTaskOperationSchema.safeParse({ kind }).success).toBe(false);
+ }
+ });
+ it("allows only the private, unexpired setup identity on an ad-hoc lease", async () => {
+ const value = row();
+ Object.assign(value.lease, { heartbeatRunId: null, issueId: null }); value.run = null;
+ Object.assign(value.lease.metadata, { runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session", expiresAtMs: Date.now() + 60_000 } });
+ const operation = { ...submit, projectIds: [], runner: { ...submit.runner, runId: "probe-run", sessionId: "probe-session" },
+ launch: { runnerVersion: "fixture", runnerDigest: `sha256:${"a".repeat(64)}`,
+ maxOutboxBytes: 262144, p0ReserveBytes: 65536, maxRuntimeMs: 0, maxLifetimeMs: 0,
+ lifecyclePolicy: { mode: "per_turn" as const, idleTimeoutMs: null },
+ } };
+ const workers = worker();
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation })).resolves.toMatchObject({ kind: "accepted" });
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ runId: "probe-run", projectIds: [] }), 15_000);
+ const lifetime = (workers.call.mock.calls[0] as unknown as [string, string, { operation: typeof operation }])[2].operation.launch.maxRuntimeMs;
+ expect(lifetime).toBeGreaterThan(0); expect(lifetime).toBeLessThanOrEqual(60_000);
+ expect((workers.call.mock.calls[0] as unknown as [string, string, { operation: typeof operation }])[2].operation.launch).not.toHaveProperty("maxLifetimeMs");
+ for (const invalid of [{ ...operation, projectIds: [projectId] }, { ...operation, runner: { ...operation.runner, sessionId: "other" } }, submit]) {
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: invalid })).rejects.toThrow("identity mismatch");
+ }
+ Object.assign(value.lease.metadata, { runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session", expiresAtMs: Date.now() - 1 } });
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation })).rejects.toThrow("not active");
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } })).resolves.toMatchObject({ kind: "accepted" });
+ });
+ it("dispatches with host-derived scope and a persisted attempt identity", async () => {
+ const { db, query } = database(); const workers = worker();
+ await expect(executeEnvironmentTask(db, workers as never, { companyId: "company", leaseId, operation: submit })).resolves.toEqual({ kind: "accepted", taskId: "attempt-1" });
+ const scope = new PgDialect().sqlToQuery(query.where.mock.calls[0][0]);
+ expect(scope.sql).toContain('"environment_leases"."company_id" =');
+ expect(scope.sql).toContain('"environment_leases"."id" =');
+ expect(scope.params).toEqual([leaseId, "company"]);
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({
+ taskId: "attempt-1", companyId: "company", agentId: "agent", projectIds: [projectId], runId: "run", operation: submit,
+ }), 15_000);
+ });
+ it("validates every requested project in the task company before dispatch", async () => {
+ const operation = { ...submit, projectIds: [projectId, secondProjectId] };
+ const { db, query } = database(row(), [{ id: projectId }, { id: secondProjectId }]);
+ const workers = worker();
+ await executeEnvironmentTask(db, workers as never, { companyId: "company", leaseId, operation });
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ projectIds: operation.projectIds }), 15_000);
+ const scope = new PgDialect().sqlToQuery(query.where.mock.calls[2][0]);
+ expect(scope.sql).toContain('"projects"."company_id" =');
+ expect(scope.params).toEqual(["company", projectId, secondProjectId]);
+ const rejected = worker();
+ await expect(executeEnvironmentTask(database(row(), [{ id: projectId }]).db, rejected as never, { companyId: "company", leaseId, operation })).rejects.toThrow("project unavailable");
+ expect(rejected.call).not.toHaveBeenCalled();
+ });
+ it("validates unique project IDs and allows tasks with no project mounts", async () => {
+ for (const projectIds of [[projectId, projectId], ["invalid"], Array(65).fill(projectId)]) {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, projectIds }).success).toBe(false);
+ }
+ const workers = worker();
+ await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: { ...submit, projectIds: [] } });
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ projectIds: [] }), 15_000);
+ });
+ it("rejects an absent or cross-company lease before calling a worker", async () => {
+ const workers = worker();
+ await expect(executeEnvironmentTask(database(null).db, workers as never, { companyId: "other", leaseId, operation: submit })).rejects.toThrow("lease unavailable");
+ expect(workers.call).not.toHaveBeenCalled();
+ });
+ it("rejects mismatched Runner binding and inactive submission", async () => {
+ const workers = worker();
+ await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: { ...submit, runner: { ...submit.runner, runId: "other" } } })).rejects.toThrow("identity mismatch");
+ const value = row(); value.lease.status = "released";
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("not active");
+ expect(workers.call).not.toHaveBeenCalled();
+ });
+ it("uses the pinned provider for cleanup after environment edits", async () => {
+ const value = row(); value.lease.status = "released"; value.environment!.config.pluginKey = "replacement";
+ const workers = worker();
+ await executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } });
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({ config: {}, operation: { kind: "stop" } }), 15_000);
+ });
+ it("keeps cleanup available after environment, run, and issue deletion", async () => {
+ const value = row(); value.environment = null; value.run = null;
+ for (const kind of ["stop", "complete", "status"] as const) {
+ const workers = worker(kind === "status" ? { kind: "status", taskId: "attempt-1", phase: "cancelled" } : undefined);
+ await executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind } });
+ expect(workers.call).toHaveBeenCalledWith("original", "environmentTask", expect.objectContaining({
+ config: {}, environmentId: null, runId: null, agentId: null, projectIds: [],
+ }), 15_000);
+ }
+ await expect(executeEnvironmentTask(database(value).db, worker() as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("not active");
+ });
+ it("preserves opaque provider IDs in requests and receipts", async () => {
+ const value = row(); value.lease.providerLeaseId = `provider:attempt.${"x".repeat(120)}`;
+ const workers = worker({ kind: "accepted", taskId: value.lease.providerLeaseId });
+ await expect(executeEnvironmentTask(database(value).db, workers as never, { companyId: "company", leaseId, operation: { kind: "stop" } })).resolves.toMatchObject({ taskId: value.lease.providerLeaseId });
+ });
+ it("returns a validated connection only while the lease and run are active", async () => {
+ const connection = { kind: "connection", taskId: "attempt-1", endpoint: {
+ kind: "authenticated_websocket", websocketUrl: "wss://runner.example.test/connect", generation: "attempt-1",
+ secretHeaders: [{ name: "Authorization", value: "Bearer fixture" }],
+ } };
+ const input = { companyId: "company", leaseId, operation: { kind: "connection" as const } };
+ await expect(executeEnvironmentTask(database().db, worker(connection) as never, input)).resolves.toEqual(connection);
+ for (const state of ["expired", "finished", "deleted"] as const) {
+ const value = row();
+ if (state === "expired") value.lease.expiresAt = new Date(0);
+ if (state === "finished") value.run!.status = "succeeded";
+ if (state === "deleted") value.environment = null;
+ const workers = worker(connection);
+ await expect(executeEnvironmentTask(database(value).db, workers as never, input)).rejects.toThrow("not active");
+ expect(workers.call).not.toHaveBeenCalled();
+ }
+ for (const endpoint of [
+ { ...connection.endpoint, websocketUrl: "ws://runner.example.test/connect" },
+ { ...connection.endpoint, websocketUrl: "wss://runner.example.test/connect?token=fixture" },
+ { ...connection.endpoint, secretHeaders: [{ name: "Authorization", value: "fixture\r\nInjected: true" }] },
+ { ...connection.endpoint, secretHeaders: [{ name: "Invalid Header", value: "fixture" }] },
+ ]) {
+ await expect(executeEnvironmentTask(database().db, worker({ ...connection, endpoint }) as never, input)).rejects.toThrow("reconcile the same task");
+ }
+ });
+ it.each(["manifest", "worker"])("requires live %s support", async kind => {
+ const workers = worker();
+ if (kind === "manifest") state.plugin.manifestJson.environmentDrivers[0].supportsTasks = false;
+ else workers.getWorker.mockReturnValue({ supportedMethods: [] });
+ await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("provider unavailable");
+ expect(workers.call).not.toHaveBeenCalled();
+ });
+ it("rejects wrong task receipts and sanitizes worker errors", async () => {
+ await expect(executeEnvironmentTask(database().db, worker({ kind: "accepted", taskId: "other" }) as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow("reconcile the same task");
+ const workers = worker(); workers.call.mockRejectedValue(new Error("private-credential"));
+ await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit })).rejects.toThrow(/^Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying$/);
+ });
+ it("accepts PRP identity characters and length limits", () => {
+ for (const field of ["runnerId", "leaseId", "runId", "sessionId", "turnId", "itemId"]) {
+ for (const value of ["session:part.1", "a".repeat(160)]) {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, [field]: value } }).success).toBe(true);
+ }
+ for (const value of ["a".repeat(161), "../escape", "a/b", ""]) {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, [field]: value } }).success).toBe(false);
+ }
+ }
+ });
+ it("validates the client's inclusive PRP version range", () => {
+ for (const range of [{ protocolMin: 1, protocolMax: 1 }, { protocolMin: 1, protocolMax: 2 }, { protocolMin: 3, protocolMax: 5 }]) {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, ...range } }).success).toBe(true);
+ }
+ for (const range of [
+ { protocolMin: 0, protocolMax: 2 }, { protocolMin: 2, protocolMax: 1 },
+ { protocolMin: 1.5, protocolMax: 2 }, { protocolMin: 1, protocolMax: 2.5 },
+ { protocolMin: 1, protocolMax: Number.MAX_SAFE_INTEGER + 1 },
+ { protocolMin: undefined, protocolMax: 2 }, { protocolMin: 1, protocolMax: undefined },
+ ]) {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, ...range } }).success).toBe(false);
+ }
+ });
+ it("validates operation and result shape without making acceptance mean readiness", () => {
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, surprise: true }).success).toBe(false);
+ expect(environmentTaskOperationSchema.safeParse({ ...submit, runner: { ...submit.runner, unknownField: true } }).success).toBe(false);
+ expect(() => parseEnvironmentTaskResult({ kind: "status" }, "attempt-1", { kind: "accepted", taskId: "attempt-1" })).toThrow();
+ expect(parseEnvironmentTaskResult(submit, "attempt-1", { kind: "accepted", taskId: "attempt-1" }).kind).toBe("accepted");
+ });
+});
diff --git a/server/src/__tests__/plugin-environment-driver-seam.test.ts b/server/src/__tests__/plugin-environment-driver-seam.test.ts
index 56c13d091e..1e691fd8a0 100644
--- a/server/src/__tests__/plugin-environment-driver-seam.test.ts
+++ b/server/src/__tests__/plugin-environment-driver-seam.test.ts
@@ -8,6 +8,7 @@ import {
parseMessage,
serializeMessage,
} from "../../../packages/plugins/sdk/src/protocol.js";
+import type { PluginEnvironmentTaskParams } from "../../../packages/plugins/sdk/src/environment-tasks.js";
import { definePlugin } from "../../../packages/plugins/sdk/src/define-plugin.js";
import { startWorkerRpcHost } from "../../../packages/plugins/sdk/src/worker-rpc-host.js";
import { pluginManifestV1Schema, type PaperclipPluginManifestV1 } from "@paperclipai/shared";
@@ -94,6 +95,54 @@ describe("plugin environment driver seam", () => {
});
});
+ it("negotiates and validates plugin-provided Runner execution RPCs", async () => {
+ const calls: unknown[] = [];
+ const plugin = definePlugin({
+ async setup() {},
+ async onEnvironmentTask(input) {
+ calls.push(input);
+ return { kind: "accepted", taskId: input.taskId };
+ },
+ });
+ const stdin = new PassThrough();
+ const stdout = new PassThrough();
+ const host = startWorkerRpcHost({ plugin, stdin, stdout });
+ const responses: unknown[] = [];
+ stdout.on("data", chunk => {
+ for (const line of String(chunk).split("\n").filter(Boolean)) responses.push(parseMessage(line));
+ });
+ try {
+ const manifest = { ...baseManifest, environmentDrivers: [{ ...baseManifest.environmentDrivers![0], supportsTasks: true }] };
+ expect(pluginManifestV1Schema.parse(manifest).environmentDrivers![0].supportsTasks).toBe(true);
+ stdin.write(serializeMessage(createRequest("initialize", {
+ manifest, config: {}, instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" }, apiVersion: 1,
+ }, 1)));
+ await waitForResponses(responses, 1);
+ const initialized = responses[0];
+ expect(isJsonRpcSuccessResponse(initialized)).toBe(true);
+ if (!isJsonRpcSuccessResponse(initialized)) return;
+ expect(initialized.result.supportedMethods).toContain("environmentTask");
+ const projectIds = ["10000000-0000-4000-8000-000000000001", "10000000-0000-4000-8000-000000000002"];
+ const params: PluginEnvironmentTaskParams = { driverKey: "fake-plugin", companyId: "company", environmentId: "environment", config: {},
+ taskId: "attempt", runId: "run", agentId: "agent", projectIds, lease: { providerLeaseId: "attempt" },
+ operation: { kind: "submit", projectIds, bootstrapTicket: "transient-test-ticket", runner: {
+ protocolMin: 1, protocolMax: 2, harness: "codex", runnerId: "runner", leaseId: "lease", runId: "run", sessionId: "session:part.1", turnId: "turn", itemId: "item",
+ } } };
+
+ stdin.write(serializeMessage(createRequest("environmentTask", params, 2)));
+ await waitForResponses(responses, 2);
+ expect(responses[1]).toMatchObject({ result: { kind: "accepted", taskId: "attempt" } });
+ expect(calls[0]).toMatchObject({ projectIds, operation: { kind: "submit", projectIds } });
+ stdin.write(serializeMessage(createRequest("environmentTask", { ...params, operation: { kind: "status" } }, 3)));
+ await waitForResponses(responses, 3);
+ expect(isJsonRpcErrorResponse(responses[2])).toBe(true); // Wrong response kind.
+ stdin.write(serializeMessage(createRequest("environmentTask", { ...params, operation: { kind: "shell", command: "echo test" } }, 4)));
+ await waitForResponses(responses, 4);
+ expect(isJsonRpcErrorResponse(responses[3])).toBe(true);
+ expect(calls).toHaveLength(2); // Invalid operation did not reach the plugin.
+ } finally { host.stop(); }
+ });
+
it("dispatches environment driver worker hooks and reports support", async () => {
const plugin = definePlugin({
async setup() {},
diff --git a/server/src/__tests__/plugin-task-credential-writeback.test.ts b/server/src/__tests__/plugin-task-credential-writeback.test.ts
new file mode 100644
index 0000000000..8ef383854c
--- /dev/null
+++ b/server/src/__tests__/plugin-task-credential-writeback.test.ts
@@ -0,0 +1,117 @@
+import { randomUUID } from "node:crypto";
+import { mkdtemp, realpath, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { and, eq } from "drizzle-orm";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import { agents, companies, companyMemberships, createDb, environmentLeases, environments, heartbeatRuns, plugins } from "@paperclipai/db";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+import { aiConnectionService } from "../services/ai-connections.js";
+import { prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";
+import { registerPluginTaskCredentialHandoff } from "../services/plugin-task-credential-writeback.js";
+import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js";
+import { secretService } from "../services/secrets.js";
+import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
+
+vi.mock("../services/subscription-refresh.js", () => ({ refreshSubscriptionConnection: async () => {} }));
+const support = await getEmbeddedPostgresTestSupport();
+const suite = support.supported ? describe : describe.skip;
+const auth = (token: string, year = 2020) => JSON.stringify({ tokens: { account_id: "fixture-account", access_token: "fixture-access", refresh_token: token }, last_refresh: `${year}-01-01T00:00:00Z` });
+
+suite("durable plugin task credential handoff", () => {
+ let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>, home: string;
+ beforeAll(async () => {
+ home = await realpath(await mkdtemp(join(tmpdir(), "plugin-task-secret-")));
+ vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "plugin-task-fixture");
+ database = await startEmbeddedPostgresTestDatabase("plugin-task-credential-"); db = createDb(database.connectionString);
+ }, 90_000);
+ afterAll(async () => { await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); });
+ async function fixture(dispatched = true) {
+ const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), pluginId = randomUUID(), environmentId = randomUUID(), leaseId = randomUUID();
+ const owner = `fixture-${randomUUID()}`;
+ await db.insert(companies).values({ id: companyId, name: "Task credential fixture", issuePrefix: `C${companyId.slice(0, 6)}` });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
+ await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
+ const service = aiConnectionService(db);
+ await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Fixture Codex", loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("fixture-original"));
+ const [selection] = await service.quotaAccounts(companyId, owner);
+ const managed = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "paperclip_runner",
+ binding: { mode: "responsible_user", provider: "openai", method: "subscription" }, config: { provider: "codex" } });
+ expect(managed.credentialWriteback).toBeDefined();
+ await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin", status: "active", config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
+ await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-tasks", version: "1.0.0", status: "ready", manifestJson: {
+ capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }],
+ } as never });
+ await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
+ await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId, provider: `plugin:${pluginId}:tasks`, providerLeaseId: "fixture-task", status: "active",
+ metadata: { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskDispatched: dispatched, runnerTaskExecutionStopped: false } });
+ await registerPluginTaskCredentialHandoff(db, { companyId, leaseId }, managed.sessionIdentity, managed.credentialWriteback!);
+ // Recovery must not depend on the original controller or its private home.
+ await rm(managed.home!, { recursive: true, force: true });
+ const calls: string[] = [];
+ let failExport = false;
+ const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
+ call: async (_plugin: string, _method: string, params: any) => {
+ calls.push(params.operation.kind);
+ expect(params.companyId).toBe(companyId); expect(params.taskId).toBe("fixture-task");
+ if (params.operation.kind === "status") return { kind: "status", taskId: "fixture-task", phase: "cancelled", executionStopped: true };
+ if (params.operation.kind === "credentials") {
+ expect(params.operation.accountIdentity).toBe(managed.sessionIdentity);
+ if (failExport) throw new Error("fixture export unavailable");
+ return { kind: "credentials", taskId: "fixture-task", credential: { type: "codex-auth-json", authJson: auth("fixture-refreshed", 2021) } };
+ }
+ return { kind: "accepted", taskId: "fixture-task" };
+ } } as unknown as PluginWorkerManager;
+ const lease = () => db.select().from(environmentLeases).where(and(eq(environmentLeases.id, leaseId), eq(environmentLeases.companyId, companyId))).then(rows => rows[0]!);
+ return { companyId, agentId, runId, environmentId, leaseId, owner, selection: selection!, managed, workers, calls, lease,
+ failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!),
+ settle: () => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }) };
+ }
+ it("persists refreshed credentials before completing a stopped task, after controller files disappear", async () => {
+ const f = await fixture();
+ expect(await f.settle()).toBe(true);
+ expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
+ expect(f.calls.indexOf("credentials")).toBeLessThan(f.calls.indexOf("complete"));
+ const saved = await f.lease();
+ expect(saved.metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
+ expect(JSON.stringify(saved.metadata)).not.toContain("fixture-refreshed");
+ expect(JSON.stringify(saved.metadata)).not.toContain(f.managed.home!);
+ });
+ it("keeps a failed transfer pending and recovers through a fresh cleanup after run and environment deletion", async () => {
+ const f = await fixture(); f.failExport(true);
+ await expect(f.settle()).rejects.toThrow("termination is unconfirmed");
+ expect(f.calls).not.toContain("complete");
+ expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "pending" });
+ expect(await f.credential()).toBe(auth("fixture-original"));
+ await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+ await db.delete(environments).where(eq(environments.id, f.environmentId));
+ f.failExport(false);
+ expect(await f.settle()).toBe(true);
+ expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
+ expect(f.calls.at(-1)).toBe("complete");
+ });
+ it("does not overwrite an explicit account replacement", async () => {
+ const f = await fixture();
+ await secretService(db).rotate(f.managed.credentialWriteback!.secretId, { value: auth("fixture-reconnected", 2022) }, { userId: f.owner });
+ expect(await f.settle()).toBe(true);
+ expect(f.calls).not.toContain("credentials");
+ expect(await f.credential()).toBe(auth("fixture-reconnected", 2022));
+ });
+ it("allows concurrent cleanup to complete the same durable handoff", async () => {
+ const f = await fixture();
+ expect(await Promise.all([f.settle(), f.settle()])).toEqual([true, true]);
+ expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
+ expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
+ });
+ it("retires an undispatched task without exporting credentials and blocks a late submit", async () => {
+ const f = await fixture(false);
+ expect(await f.settle()).toBe(false);
+ expect(f.calls).toEqual([]);
+ expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
+ await expect(executeEnvironmentTask(db, f.workers, { companyId: f.companyId, leaseId: f.leaseId, operation: {
+ kind: "submit", projectIds: [], bootstrapTicket: "fixture-ticket", runner: { harness: "codex", protocolMin: 1, protocolMax: 3,
+ runnerId: "runner", leaseId: f.leaseId, runId: f.runId, sessionId: "session", turnId: "turn", itemId: "item" },
+ } })).rejects.toThrow("not active");
+ expect(f.calls).toEqual([]);
+ });
+});
diff --git a/server/src/services/ai-connection-runtime.ts b/server/src/services/ai-connection-runtime.ts
index c201f2451f..74a4173cb7 100644
--- a/server/src/services/ai-connection-runtime.ts
+++ b/server/src/services/ai-connection-runtime.ts
@@ -1,3 +1,4 @@
+import type { PluginTaskCredentialWritebackBinding } from "./plugin-task-credential-writeback.js";
import { OPENROUTER_ATTRIBUTION_HEADERS, openRouterClaudeHeaders } from "@paperclipai/shared/openrouter-attribution";
import { createHash } from "node:crypto";
import { HttpError, unprocessable } from "../errors.js";
@@ -385,6 +386,10 @@ export async function prepareManagedAiRuntime(
const sessionIdentity = `${selection.grant.id}:${input.responsibleUserId ?? "shared"}:${noAuth ? "no-auth" : `${credentialRef!.secretId}:${freshness!.epoch}`}`;
return {
sessionIdentity,
+ ...(subscriptionFile && input.binding.provider === "openai" && credentialRef && freshness ? {
+ credentialWriteback: { grantId: selection.grant.id, secretId: credentialRef.secretId, sessionEpoch: freshness.epoch,
+ versionSelector: credentialRef.versionSelector ?? "latest" } satisfies PluginTaskCredentialWritebackBinding,
+ } : {}),
config: {
...input.config,
...projected?.config,
diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts
index 35a5717509..7d14dbf7c8 100644
--- a/server/src/services/environment-runtime.ts
+++ b/server/src/services/environment-runtime.ts
@@ -1,3 +1,5 @@
+import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
+import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import { beginIdleTrackedWork } from "./task-admission.js";
@@ -3594,16 +3596,16 @@ function createPluginEnvironmentDriver(
return { plugin, driver };
}
- async function resolvePluginDriverForRelease(input: EnvironmentDriverReleaseInput) {
+ async function resolvePluginDriverForRelease(input: { environment: Environment | null; lease: EnvironmentDriverReleaseInput["lease"] }) {
const metadata = input.lease.metadata ?? {};
const metadataPluginId = readString(metadata.pluginId);
const metadataPluginKey = readString(metadata.pluginKey);
const metadataDriverKey = readString(metadata.driverKey);
- const currentConfig = tryParseCurrentPluginConfig(input.environment);
+ const currentConfig = input.environment ? tryParseCurrentPluginConfig(input.environment) : null;
if (!metadataPluginId && !metadataPluginKey && !metadataDriverKey) {
if (!currentConfig) {
- throw new Error(`Expected plugin environment config for driver "${input.environment.driver}".`);
+ throw new Error(`Expected pinned plugin environment config.`);
}
const { plugin } = await resolvePluginDriver(currentConfig);
return {
@@ -3698,6 +3700,8 @@ function createPluginEnvironmentDriver(
metadata: {
...(input.agentId ? { agentId: input.agentId } : {}),
providerMetadata: providerLease.metadata ?? {},
+ environmentId: input.environment.id,
+ ...(input.adapterType === "paperclip_runner" && driver.supportsTasks === true ? { runnerTaskLease: true } : {}),
driver: input.environment.driver,
executionWorkspaceMode: input.executionWorkspaceMode,
pluginId: plugin.id,
@@ -3708,6 +3712,9 @@ function createPluginEnvironmentDriver(
},
async releaseRunLease(input) {
+ await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
+ const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
+ (input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await workerManager.call(plugin.id, "environmentReleaseLease", {
driverKey,
@@ -3718,17 +3725,18 @@ function createPluginEnvironmentDriver(
providerLeaseId: input.lease.providerLeaseId,
leaseMetadata: input.lease.metadata ?? undefined,
});
- return await environmentsSvc.releaseLease(input.lease.id, input.status, { cleanupStatus: "success" });
+ return await environmentsSvc.releaseLease(input.lease.id, input.status, {
+ cleanupStatus: "success",
+ ...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
+ { providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
+ });
},
async resumeRunLease(input) {
if (!input.lease.providerLeaseId) {
throw new Error(`Plugin environment lease "${input.lease.id}" does not have a provider lease id to resume.`);
}
- const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
- ...input,
- status: "released",
- });
+ const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
return await resumePluginEnvironmentLease({
db,
workerManager,
@@ -3746,10 +3754,10 @@ function createPluginEnvironmentDriver(
},
async destroyRunLease(input) {
- const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
- ...input,
- status: "failed",
- });
+ await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"destroy"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
+ const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
+ (input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
+ const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await destroyPluginEnvironmentLease({
db,
workerManager,
@@ -3769,14 +3777,37 @@ function createPluginEnvironmentDriver(
cleanupStatus: "success",
...(input.lease.status === "pending_cleanup" && typeof input.lease.metadata?.pendingCleanupAttemptId === "string"
? { expectedPendingCleanupAttemptId: input.lease.metadata.pendingCleanupAttemptId } : {}),
+ ...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
+ { providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
});
},
+ async isPendingCleanupWorkerReady(input) {
+ try {
+ const { plugin } = await resolvePluginDriverForRelease(input);
+ const methods = workerManager.getWorker(plugin.id)?.supportedMethods ?? [];
+ return methods.includes(input.lease.metadata?.pluginLeaseCleanupMethod === "destroy" ? "environmentDestroyLease" : "environmentReleaseLease") &&
+ (input.lease.metadata?.runnerTaskDispatched !== true || methods.includes("environmentTask"));
+ } catch { return false; }
+ },
+
+ async retryPendingSandboxTeardown(input) {
+ const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
+ (input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
+ const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
+ const environmentId = input.environment?.id ?? readString(input.lease.metadata?.environmentId);
+ if (!environmentId) throw new Error("Plugin lease cleanup has no saved environment identity");
+ const receipt = await workerManager.call(plugin.id, input.lease.metadata?.pluginLeaseCleanupMethod === "destroy" ? "environmentDestroyLease" : "environmentReleaseLease", {
+ driverKey, companyId: input.lease.companyId, environmentId, issueId: input.lease.issueId,
+ config: driverConfig, providerLeaseId: input.lease.providerLeaseId, leaseMetadata: input.lease.metadata ?? undefined,
+ });
+ return taskStopped ? { providerLeaseId: input.lease.providerLeaseId, state: "stopped" } : receipt;
+ },
+
async realizeWorkspace(input) {
const { plugin, pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
environment: input.environment,
lease: input.lease,
- status: "released",
});
return await realizePluginEnvironmentWorkspace({
db,
@@ -3807,7 +3838,6 @@ function createPluginEnvironmentDriver(
const { plugin, pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease({
environment: input.environment,
lease: input.lease,
- status: "released",
});
return await executePluginEnvironmentCommand({
db,
@@ -4051,6 +4081,17 @@ export function environmentRuntimeService(
return resolveSandboxDuplexBridgeInput(experimental);
},
+ async supportsTasks(input: { lease: EnvironmentLease }) {
+ return options.pluginWorkerManager
+ ? pluginEnvironmentSupportsTasks(db, options.pluginWorkerManager, input.lease) : false;
+ },
+
+ /** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
+ async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation }) {
+ if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
+ return executeEnvironmentTask(db, options.pluginWorkerManager, input);
+ },
+
async acquireRunLease(input: {
companyId: string;
environment: Environment;
@@ -4198,7 +4239,15 @@ export function environmentRuntimeService(
}
continue;
}
- if (!environment) continue;
+ if (!environment) {
+ if (leaseSnapshot.metadata?.driver === "plugin" && options.pluginWorkerManager) {
+ const receipt = await requireDriverKey("plugin").retryPendingSandboxTeardown!({ environment: null, lease: leaseSnapshot });
+ await environmentsSvc.releaseLease(leaseSnapshot.id, status, { cleanupStatus: "success",
+ remoteExecutionTermination: remoteTerminationReceipt(leaseSnapshot, receipt),
+ });
+ }
+ continue;
+ }
if (
providerResourceDisposition === "keep_running" &&
leaseSnapshot.leasePolicy === "reuse_by_environment"
diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts
new file mode 100644
index 0000000000..a8b080a37b
--- /dev/null
+++ b/server/src/services/environment-task-runtime.ts
@@ -0,0 +1,202 @@
+import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
+import { and, eq, inArray, or, sql } from "drizzle-orm";
+import { environmentLeases, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
+import { environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
+import { pluginRegistryService } from "./plugin-registry.js";
+import { hasRemoteTerminationReceipt, remoteExecutionHasStopped } from "./remote-execution-termination.js";
+import type { PluginWorkerManager } from "./plugin-worker-manager.js";
+
+export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
+ const metadata = lease.metadata ?? {};
+ if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") return false;
+ const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
+ const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
+ const methods = workers.getWorker(metadata.pluginId)?.supportedMethods ?? [];
+ return plugin?.status === "ready" && driver?.supportsTasks === true &&
+ plugin.manifestJson.capabilities.includes("environment.drivers.register") &&
+ methods.includes("environmentTask");
+}
+
+/** Server-only dispatch for plugin-provided Runner execution.
+ * The caller supplies an authorized company and a persisted lease,
+ * never a plugin identity, resource endpoint, or agent/project authority.
+ * A timeout is ambiguous: retry the same operation against the same lease.
+ */
+export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManager, input: {
+ companyId: string;
+ leaseId: string;
+ operation: PluginEnvironmentTaskOperation;
+}) {
+ const operation = environmentTaskOperationSchema.parse(input.operation);
+ const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
+ .from(environmentLeases)
+ .leftJoin(environments, eq(environments.id, environmentLeases.environmentId))
+ .leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
+ .where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
+ if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
+ const { lease, environment, run } = row;
+ const taskId = row.lease.providerLeaseId;
+ const metadata = lease.metadata ?? {};
+ const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
+ const activeProbe = lease.heartbeatRunId === null && lease.issueId === null && probe
+ && typeof probe.runId === "string" && typeof probe.sessionId === "string"
+ && typeof probe.expiresAtMs === "number" && probe.expiresAtMs > Date.now();
+ if ((operation.kind === "describe" || operation.kind === "submit" || operation.kind === "connection") &&
+ (metadata.runnerTaskRetiring === true || !environment || (!activeProbe && run?.status !== "running") || lease.status !== "active" || (lease.expiresAt && lease.expiresAt.getTime() <= Date.now()))) throw new Error("Plugin-provided Runner execution lease is not active");
+ if (operation.kind === "submit" && (operation.runner.runId !== (activeProbe ? probe.runId : run?.id) || (!operation.resumeFrom && operation.runner.leaseId !== lease.id)
+ || (activeProbe && (operation.runner.sessionId !== probe.sessionId || operation.projectIds.length > 0)))) {
+ throw new Error("Plugin-provided Runner execution identity mismatch");
+ }
+ if (operation.kind === "submit" && operation.resumeFrom) {
+ const sources = await db.select({ runId: environmentLeases.heartbeatRunId }).from(environmentLeases).where(and(
+ eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.provider, lease.provider!),
+ eq(environmentLeases.providerLeaseId, operation.resumeFrom),
+ )).limit(2);
+ const priorRunId = sources.length === 1 ? sources[0]!.runId : null;
+ if (activeProbe || !priorRunId || operation.resumeFrom !== await resolvePluginTaskResume(db, {
+ companyId: input.companyId, leaseId: lease.id,
+ priorIdentity: { runnerInstanceId: operation.runner.runnerId, environmentLeaseId: operation.runner.leaseId,
+ runId: priorRunId, normalizedSessionId: operation.runner.sessionId },
+ })) throw new Error("Plugin Runner resume identity mismatch");
+ }
+ if (metadata.driver !== "plugin" || typeof metadata.pluginId !== "string" || typeof metadata.driverKey !== "string") {
+ throw new Error("Plugin-provided Runner execution lease has no pinned plugin driver");
+ }
+ const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
+ const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
+ if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
+ !plugin.manifestJson.capabilities.includes("environment.drivers.register") ||
+ !workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
+ throw new Error("Plugin-provided Runner execution provider unavailable");
+ }
+ const issue = operation.kind === "submit" && lease.issueId ? await db.select({ id: issues.id }).from(issues)
+ .where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
+ if (operation.kind === "submit" && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
+ const projectIds = operation.kind === "submit" ? operation.projectIds : [];
+ if (projectIds.length > 0) {
+ const authorizedProjects = await db.select({ id: projects.id }).from(projects)
+ .where(and(eq(projects.companyId, input.companyId), inArray(projects.id, projectIds)));
+ if (authorizedProjects.length !== projectIds.length) throw new Error("Plugin-provided Runner execution project unavailable");
+ }
+ // Provider identity comes from the lease. Editing the environment must never
+ // redirect status or cleanup to a replacement plugin.
+ const config = (environment?.config ?? {}) as Record<string, unknown>;
+ const driverConfig = config.pluginKey === plugin.pluginKey && config.driverKey === metadata.driverKey
+ ? (config.driverConfig as Record<string, unknown> | undefined) ?? {} : {};
+ if (operation.kind === "submit" && activeProbe) {
+ const remainingMs = Number(probe.expiresAtMs) - Date.now();
+ if (remainingMs <= 0 || !operation.launch) throw new Error("Plugin-provided Runner probe lease is not active");
+ // Runner itself must stop if Core disappears during a setup probe.
+ const requestedLimits = [operation.launch.maxRuntimeMs, operation.launch.maxLifetimeMs]
+ .filter((value): value is number => typeof value === "number" && value > 0);
+ operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
+ delete operation.launch.maxLifetimeMs;
+ }
+ if (operation.kind === "submit") {
+ // No ticket, headers, environment values, or launch paths belong in metadata.
+ const marked = await db.update(environmentLeases).set({
+ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskLease: true, runnerTaskDispatched: true, runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: operation.runner, runnerTaskResumeFrom: operation.resumeFrom ?? null })}::jsonb`,
+ }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
+ eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({ id: environmentLeases.id });
+ if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
+ }
+ try {
+ if (operation.kind === "complete") {
+ await settlePluginTaskCredentialHandoff(db, { companyId: input.companyId, leaseId: lease.id }, metadata, async accountIdentity => {
+ const snapshot = await executeEnvironmentTask(db, workers, { ...input, operation: { kind: "credentials", accountIdentity } });
+ if (snapshot.kind !== "credentials") throw new Error("Plugin Runner credential snapshot unavailable");
+ return snapshot.credential;
+ });
+ }
+ const result = await workers.call(plugin.id, "environmentTask", {
+ driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
+ issueId: lease.issueId, config: driverConfig,
+ taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
+ lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined }, operation,
+ }, 15_000);
+ const parsed = parseEnvironmentTaskResult(operation, taskId, result);
+ if (parsed.kind === "status" && parsed.executionStopped === true) {
+ await db.update(environmentLeases).set({
+ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true, runnerTaskSessionRetained: parsed.sessionStateAvailable === true })}::jsonb`,
+ }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
+ }
+ if (operation.kind === "complete") {
+ await db.update(environmentLeases).set({
+ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskDispatched":false}'::jsonb`,
+ }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
+ sql`${environmentLeases.metadata}->>'runnerTaskExecutionStopped' = 'true'`));
+ }
+ return parsed;
+ } catch {
+ // Worker errors and schema diagnostics can contain request credentials.
+ throw new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
+ }
+}
+
+/** Failure leaves the dispatch marker and provider handle available to cleanup retries. */
+export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string }) {
+ const [current] = await db.update(environmentLeases).set({
+ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
+ }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
+ .returning({ metadata: environmentLeases.metadata });
+ if (current?.metadata?.runnerTaskDispatched !== true) {
+ if (current?.metadata) await settlePluginTaskCredentialHandoff(db, { companyId: lease.companyId, leaseId: lease.id }, current.metadata,
+ async () => { throw new Error("Undispatched task must not request credentials"); });
+ return current?.metadata?.runnerTaskExecutionStopped === true;
+ }
+ const scope = { companyId: lease.companyId, leaseId: lease.id };
+ try {
+ await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
+ const state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
+ if (state.kind !== "status" || state.executionStopped !== true) throw new Error("Plugin Runner termination is unconfirmed");
+ await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
+ return true;
+ } catch {
+ await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
+ failureReason: "plugin_runner_task_cleanup_unconfirmed", releasedAt: new Date(), updatedAt: new Date(),
+ }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)));
+ throw new Error("Plugin Runner termination is unconfirmed; the saved lease requires cleanup");
+ }
+}
+
+/** A saved provider task can resume only within the same authorized agent workspace. */
+export async function resolvePluginTaskResume(db: Db, input: {
+ companyId: string; leaseId: string;
+ priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string };
+}): Promise<string> {
+ const rows = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
+ .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, input.companyId)))
+ .where(and(eq(environmentLeases.companyId, input.companyId),
+ or(eq(environmentLeases.id, input.leaseId), eq(heartbeatRuns.id, input.priorIdentity.runId))));
+ const next = rows.find(row => row.lease.id === input.leaseId);
+ // A retained PRP lease is logical; later provider tasks have different physical lease IDs.
+ const candidates = rows.filter(row => {
+ const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
+ return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId && row.lease.metadata?.runnerTaskSessionRetained === true &&
+ identity?.runnerId === input.priorIdentity.runnerInstanceId && identity.leaseId === input.priorIdentity.environmentLeaseId &&
+ identity.sessionId === input.priorIdentity.normalizedSessionId && identity.runId === input.priorIdentity.runId &&
+ !rows.some(successor => successor.lease.id !== input.leaseId &&
+ successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId);
+ });
+ const old = candidates.length === 1 ? candidates[0] : undefined;
+ const identity = old?.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
+ if (!old || !next || !old.lease.providerLeaseId || !identity ||
+ identity.runnerId !== input.priorIdentity.runnerInstanceId || identity.leaseId !== input.priorIdentity.environmentLeaseId ||
+ identity.sessionId !== input.priorIdentity.normalizedSessionId || identity.runId !== input.priorIdentity.runId ||
+ old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin" ||
+ !next.lease.providerLeaseId || next.lease.providerLeaseId === old.lease.providerLeaseId ||
+ (old.run.id !== next.run.id && !["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(old.run.status)) ||
+ old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId ||
+ old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId ||
+ !old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId ||
+ !next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider ||
+ old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey ||
+ next.lease.status !== "active" || next.lease.releasedAt || next.run.status !== "running" ||
+ (next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now()) ||
+ !(old.run.id === next.run.id
+ ? rows.filter(row => row.lease.id !== next.lease.id).every(row => hasRemoteTerminationReceipt(row.lease))
+ : await remoteExecutionHasStopped(db, input.companyId, old.run.id))) {
+ throw new Error("Plugin Runner retained session is not authorized");
+ }
+ return old.lease.providerLeaseId;
+}
diff --git a/server/src/services/plugin-task-credential-writeback.ts b/server/src/services/plugin-task-credential-writeback.ts
new file mode 100644
index 0000000000..e13630b5f5
--- /dev/null
+++ b/server/src/services/plugin-task-credential-writeback.ts
@@ -0,0 +1,143 @@
+import { aiConnectionMetadataSchema } from "@paperclipai/shared";
+import { and, eq, sql } from "drizzle-orm";
+import { companySecrets, connectionGrants, environmentLeases, toolConnections, type Db } from "@paperclipai/db";
+import { z, type PluginEnvironmentTaskCredential } from "@paperclipai/plugin-sdk";
+import { mkdtemp, writeFile, rm, realpath } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { isDeepStrictEqual } from "node:util";
+import { decideCodexAuthMerge, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
+import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
+import { aiConnectionService } from "./ai-connections.js";
+import { secretService } from "./secrets.js";
+
+const bindingSchema = z.object({
+ grantId: z.string().uuid(),
+ secretId: z.string().uuid(),
+ sessionEpoch: z.number().int().nonnegative(),
+ versionSelector: z.union([z.literal("latest"), z.number().int().positive()]),
+}).strict();
+export type PluginTaskCredentialWritebackBinding = z.infer<typeof bindingSchema>;
+const handoffSchema = z.object({
+ accountIdentity: z.string().min(1),
+ binding: bindingSchema,
+ state: z.enum(["pending", "complete"]),
+}).strict();
+const key = "runnerTaskCredentialHandoff";
+type Scope = { companyId: string; leaseId: string };
+
+/** Save authority, never credential bytes or a controller-local temporary path. */
+export async function registerPluginTaskCredentialHandoff(
+ db: Db,
+ scope: Scope,
+ accountIdentity: string,
+ binding: PluginTaskCredentialWritebackBinding,
+) {
+ const handoff = handoffSchema.parse({ accountIdentity, binding, state: "pending" });
+ await db.transaction(async tx => {
+ const [lease] = await tx.select().from(environmentLeases).where(and(
+ eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
+ )).for("update");
+ if (!lease || lease.status !== "active" || lease.metadata?.runnerTaskLease !== true ||
+ lease.metadata.runnerTaskRetiring === true) {
+ throw new Error("Plugin task credential lease unavailable");
+ }
+ const saved = lease.metadata[key];
+ if (saved) {
+ const prior = handoffSchema.parse(saved);
+ if (prior.accountIdentity !== accountIdentity || !isDeepStrictEqual(prior.binding, handoff.binding)) {
+ throw new Error("Plugin task credential authority changed");
+ }
+ return;
+ }
+ await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, [key]: handoff } })
+ .where(eq(environmentLeases.id, lease.id));
+ });
+}
+
+/** Runs for live completion and the cleanup sweep, including after Core restarts. */
+export async function settlePluginTaskCredentialHandoff(
+ db: Db,
+ scope: Scope,
+ metadata: Record<string, unknown>,
+ fetchCredential: (accountIdentity: string) => Promise<PluginEnvironmentTaskCredential | null>,
+) {
+ if (!metadata[key]) return;
+ const handoff = handoffSchema.parse(metadata[key]);
+ if (handoff.state === "complete") return;
+ if (metadata.runnerTaskDispatched === true && metadata.runnerTaskExecutionStopped !== true) {
+ throw new Error("Plugin task credential process termination is unconfirmed");
+ }
+ const binding = handoff.binding;
+ const home = await mkdtemp(join(await realpath(tmpdir()), "paperclip-task-credential-"));
+ try {
+ // Revocation or an explicit account replacement intentionally discards an old refresh.
+ const account = async (database: Db) => {
+ const [grant] = await database.select().from(connectionGrants).where(and(
+ eq(connectionGrants.companyId, scope.companyId), eq(connectionGrants.id, binding.grantId),
+ )).for("update");
+ if (!grant || grant.status !== "active") return null;
+ const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
+ if (!ref || ref.secretId !== binding.secretId ||
+ (ref.versionSelector ?? "latest") !== binding.versionSelector) return null;
+ const [secret] = await database.select().from(companySecrets).where(and(
+ eq(companySecrets.companyId, scope.companyId), eq(companySecrets.id, binding.secretId),
+ )).for("update");
+ if (!secret || secret.aiSessionEpoch !== binding.sessionEpoch) return null;
+ const [connection] = await database.select().from(toolConnections).where(and(
+ eq(toolConnections.companyId, scope.companyId), eq(toolConnections.id, grant.connectionId),
+ ));
+ const ai = aiConnectionMetadataSchema.safeParse(connection?.config.ai);
+ if (!connection || !ai.success || ai.data.provider !== "openai" || ai.data.method !== "subscription") return null;
+ return { connection, grant };
+ };
+ const mutate = <T>(operation: (database: Db) => Promise<T>) => withAccountHomeSecretMutationLock(
+ undefined, scope.companyId, () => db.transaction(tx => operation(tx as unknown as Db)),
+ );
+ const current = metadata.runnerTaskDispatched === true && await mutate(account);
+ if (current) {
+ // Do not hold database or account locks while the provider exports its credential.
+ const credential = await fetchCredential(handoff.accountIdentity);
+ if (credential !== null) {
+ if (credential.type !== "codex-auth-json") throw new Error("Plugin task credential provider mismatch");
+ const incoming = join(home, "incoming.json"), destination = join(home, "current.json");
+ await writeFile(incoming, credential.authJson, { mode: 0o600 });
+ const writeBack = () => mutate(async database => {
+ const selected = await account(database);
+ if (!selected) return;
+ const value = await aiConnectionService(database).credential(selected);
+ await writeFile(destination, value, { mode: 0o600 });
+ if (await decideCodexAuthMerge(incoming, destination, { errorLabel: "AI account refresh" }) !== 10) return;
+ await secretService(database).rotate(binding.secretId, {
+ value: credential.authJson, preserveAiSessionEpoch: true,
+ }, { userId: selected.grant.subjectUserId });
+ });
+ for (let attempt = 0; ; attempt++) {
+ try {
+ await writeBack();
+ break;
+ } catch (error) {
+ if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
+ }
+ }
+ }
+ }
+ const completed = await db.update(environmentLeases).set({
+ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [key]: { ...handoff, state: "complete" } })}::jsonb`,
+ }).where(and(
+ eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
+ sql`${environmentLeases.metadata}->${key} = ${JSON.stringify(handoff)}::jsonb`,
+ )).returning({ id: environmentLeases.id });
+ if (completed.length !== 1) {
+ const [current] = await db.select({ metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
+ eq(environmentLeases.companyId, scope.companyId), eq(environmentLeases.id, scope.leaseId),
+ ));
+ if (!isDeepStrictEqual(handoffSchema.safeParse(current?.metadata?.[key]).data, { ...handoff, state: "complete" })) {
+ throw new Error("Plugin task credential handoff changed");
+ }
+ }
+ } finally {
+ // The provider retains the recoverable credential until completion succeeds.
+ await rm(home, { recursive: true, force: true });
+ }
+}