Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(eve): upgrade older handoff checkpoints instead of stranding sessions (#4191)

Andrew Barbaa53f2808ccc7
diff --git a/.changeset/handoff-checkpoint-upgrades.md b/.changeset/handoff-checkpoint-upgrades.md
new file mode 100644
index 000000000..38d647e60
--- /dev/null
+++ b/.changeset/handoff-checkpoint-upgrades.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Long-lived sessions now move to newer deployments after an eve upgrade: a deployment upgrades handoff checkpoints written by eve 0.66.0 and later instead of leaving the session on its old deployment, and stops idle subagent sessions those releases kept for reuse. Refused handoffs now log why on both deployments. Callbacks from remote agents to sessions created by eve 0.66–0.68 no longer fail with `Unsupported callback kind`.
diff --git a/docs/concepts/execution-model-and-durability.mdx b/docs/concepts/execution-model-and-durability.mdx
index 20c051060..f8a14007e 100644
--- a/docs/concepts/execution-model-and-durability.mdx
+++ b/docs/concepts/execution-model-and-durability.mdx
@@ -25,7 +25,9 @@ The Workflow SDK is not inherently tied to Vercel. In local development and in a
When a Vercel production deployment changes, a new delivery to an idle session hands the session's settled state to the exact deployment that accepted the request. This applies whether the delivery arrives through the session ID or through any channel continuation address the session has claimed. The session keeps its original ID and event stream, while the next model turn uses that deployment's current system-role instructions, model, and tools. Static user-role instructions stay pinned in the durable conversation history and are not seeded again. A successful handoff restarts the session's original configured [`sessionTimeoutMs`](../agent-config#runtime-limits) duration; disabled timeouts stay disabled. Failed or skipped handoffs keep the existing deadline.
-Before taking ownership, the target deployment validates the checkpoint's stored tasks and subagent handles and checks that no pending work remains. This includes settled tasks. If that validation fails, the previous owner recovers the session and processes the triggering delivery. Task and handle state preserve unknown metadata fields, while known fields and lifecycle rules must still be valid. Authored session state remains opaque to this validation.
+Before taking ownership, the target deployment upgrades the checkpoint when an older eve release wrote it, then checks that no pending work remains. A deployment accepts checkpoints from eve 0.66.0 and later, so upgrading eve does not leave long-lived sessions, such as a Slack thread, on their old deployment. An older release cannot read a newer checkpoint: after a rollback, a session that already moved to the newer deployment stays there. Extension state saved before eve 0.69 moves to its mount when exactly one mount uses that extension package and the extension still defines that state. Otherwise eve cannot choose an owner, and the session stays on its current deployment. Idle subagent sessions that eve 0.66–0.68 kept for later reuse are stopped after the handoff. Authored session state remains opaque to this validation.
+
+If the target deployment rejects the session, the previous owner keeps it and processes the triggering delivery. When the target deployment cannot read the checkpoint, its Vercel runtime logs show a `session handoff refused` warning with the checkpoint version and the reason. Owners on this release or later also write a `session handoff failed` warning to their own runtime logs, with the target deployment and the reason.
eve does not move live work between deployments. A session with a running turn, pending human input or authorization, queued messages, or active tasks and subagents stays on its current deployment. After that work settles, a later delivery can move the idle session; the earlier deployment signal is not remembered.
diff --git a/docs/extensions.md b/docs/extensions.md
index 353747bc3..825e95f17 100644
--- a/docs/extensions.md
+++ b/docs/extensions.md
@@ -210,10 +210,11 @@ Extension state belongs to the logical mount path (for example, `extensions/crm`
### Upgrade from package-scoped extension state
-Deployments before this release stored extension state under package-prefixed keys. eve does not migrate that state to mount-owned keys or reset it during restore.
+Deployments before eve 0.69 stored extension state under package-prefixed keys, such as `acme-crm.requests`. When a session from one of those deployments hands off to a newer deployment, eve moves each package-prefixed value to the mount that uses that package. eve never resets that state during restore.
-- Session handoffs across this upgrade boundary are rejected in both directions, including for agents without extensions. Keep each session's owning deployment available until the session finishes, or start a new session on the deployment you want to use.
-- Local context snapshots use a separate state-layout check. Older snapshots for agents with extensions are incompatible; snapshots for agents without extensions can restore if they contain no unrecognized state keys.
+- The value moves only when exactly one mount uses the package and the extension still defines that state name. Otherwise eve cannot tell which mount owns the value, so the handoff is rejected and the session stays on its current deployment. Keep that deployment available until the session finishes, or start a new session on the deployment you want to use.
+- Older deployments cannot read sessions saved by a newer release, so a session that already moved does not hand back after a rollback.
+- Local context snapshots follow the same rule: an older snapshot restores when every saved state key still has an owner.
### Use an extension in a workspace
diff --git a/packages/eve/src/context/serialize.test.ts b/packages/eve/src/context/serialize.test.ts
index 0d1758382..0e7fc0de6 100644
--- a/packages/eve/src/context/serialize.test.ts
+++ b/packages/eve/src/context/serialize.test.ts
@@ -4,7 +4,7 @@ import { createRuntimeAdapterRegistry } from "#runtime/channels/registry.js";
import { compileFromMemory } from "#internal/testing/compile-from-memory.js";
import { ContextContainer } from "#context/container.js";
import { ContextKey } from "#context/key.js";
-import { defineState } from "#public/definitions/state.js";
+import { defineMountedState, defineState } from "#public/definitions/state.js";
import type { CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";
import { BundleKey, ChannelKey } from "#runtime/sessions/runtime-context-keys.js";
import { deserializeContext, serializeContext } from "#context/serialize.js";
@@ -61,25 +61,32 @@ describe("state layout admission", () => {
}
});
- it("admits an unmarked checkpoint only when its graph has no mounts", async () => {
+ it("moves legacy extension state to the one mount that owns its package", async () => {
const { manifest } = await compileFromMemory({ model: "openai/gpt-5.4" });
const bundle = { compiledArtifactsSource: { kind: "bundled" } } as CompiledBundle;
- const deserialize = vi.spyOn(BundleKey.codec!, "deserialize").mockResolvedValue(bundle);
+ const deserialize = vi.spyOn(BundleKey.codec!, "deserialize").mockImplementation(async () => {
+ defineMountedState("extensions/crm", "requests", () => 0);
+ return bundle;
+ });
+ const mount = (mountId: string) =>
+ ({ mountId, packageName: "@acme/crm" }) as (typeof manifest.extensionMounts)[number];
const loadManifest = vi
.spyOn(manifestLoader, "loadCompiledManifest")
- .mockResolvedValue(manifest);
+ .mockResolvedValue({ ...manifest, extensionMounts: [mount("extensions/crm")] });
try {
- const restored = await deserializeContext({ "eve.bundle": {} });
- expect(restored.get(BundleKey)).toBe(bundle);
+ expect((await deserializeContext({ "eve.bundle": {} })).get(BundleKey)).toBe(bundle);
+ const restored = await deserializeContext({ "eve.bundle": {}, "acme-crm.requests": 4 });
+ expect(
+ [...restored.entries()].map(([key, value]) => [key.name, value]).filter(([, v]) => v === 4),
+ ).toEqual([["eve:mount.v1:extensions%2Fcrm:requests", 4]]);
+ // Two mounts of one package shared the legacy key, so neither can claim it.
loadManifest.mockResolvedValue({
...manifest,
- extensionMounts: [
- { mountId: "extensions/crm" } as (typeof manifest.extensionMounts)[number],
- ],
+ extensionMounts: [mount("extensions/crm"), mount("extensions/crm-eu")],
});
- await expect(deserializeContext({ "eve.bundle": {} })).rejects.toThrow(
- "Incompatible context state layout",
- );
+ await expect(
+ deserializeContext({ "eve.bundle": {}, "acme-crm.requests": 4 }),
+ ).rejects.toThrow('Incompatible context state layout for key "acme-crm.requests"');
} finally {
deserialize.mockRestore();
loadManifest.mockRestore();
diff --git a/packages/eve/src/context/serialize.ts b/packages/eve/src/context/serialize.ts
index a19f8da39..56929f189 100644
--- a/packages/eve/src/context/serialize.ts
+++ b/packages/eve/src/context/serialize.ts
@@ -2,7 +2,8 @@ import { type AlsContext, ContextContainer } from "#context/container.js";
import { resolveKey } from "#context/key.js";
import { createLogger, logError } from "#internal/logging.js";
import { loadCompiledManifest } from "#runtime/loaders/manifest.js";
-import { BundleKey } from "#runtime/sessions/runtime-context-keys.js";
+import { mountedStateKeyName } from "#public/definitions/state.js";
+import { BundleKey, type CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";
const log = createLogger("context.serialize");
const STATE_LAYOUT_KEY = "eve.stateLayout";
@@ -40,18 +41,21 @@ export function serializeContext(ctx: AlsContext): Record<string, unknown> {
* Each entry is matched to a registered {@link ContextKey} by name.
* Unknown entries (no registered key) are dropped with a warning.
*/
-export async function deserializeContext(data: Record<string, unknown>): Promise<ContextContainer> {
+export async function deserializeContext(
+ serialized: Record<string, unknown>,
+): Promise<ContextContainer> {
+ let data = serialized;
const ctx = new ContextContainer();
const serializedBundle = data[BundleKey.name];
if (data[STATE_LAYOUT_KEY] !== undefined && data[STATE_LAYOUT_KEY] !== STATE_LAYOUT_VERSION) {
- throw incompatibleStateLayout();
+ throw new IncompatibleStateLayoutError();
}
if (
data[STATE_LAYOUT_KEY] === undefined &&
Object.keys(data).some((name) => data[name] !== undefined && name.startsWith("eve:mount."))
) {
- throw incompatibleStateLayout();
+ throw new IncompatibleStateLayoutError();
}
if (serializedBundle !== undefined) {
const codec = BundleKey.codec;
@@ -59,30 +63,7 @@ export async function deserializeContext(data: Record<string, unknown>): Promise
throw new Error('Context key "eve.bundle" is missing a codec.');
}
const bundle = await codec.deserialize(serializedBundle, ctx);
- if (data[STATE_LAYOUT_KEY] === undefined) {
- const manifest = await loadCompiledManifest({
- compiledArtifactsSource: bundle.compiledArtifactsSource,
- });
- if (
- [manifest, ...manifest.subagents.map((subagent) => subagent.agent)].some(
- (node) => node.extensionMounts.length > 0,
- )
- ) {
- throw incompatibleStateLayout();
- }
- }
- if (
- data[STATE_LAYOUT_KEY] === undefined &&
- Object.keys(data).some(
- (name) =>
- name !== BundleKey.name &&
- name !== STATE_LAYOUT_KEY &&
- data[name] !== undefined &&
- resolveKey(name) === undefined,
- )
- ) {
- throw incompatibleStateLayout();
- }
+ if (data[STATE_LAYOUT_KEY] === undefined) data = await adoptLegacyStateLayout(data, bundle);
ctx.set(BundleKey, bundle);
}
@@ -105,8 +86,64 @@ export async function deserializeContext(data: Record<string, unknown>): Promise
return ctx;
}
-function incompatibleStateLayout(): Error {
- return new Error(
- "Incompatible context state layout. Restore this session with its original deployment or start a new session; legacy extension state cannot be assigned to mounts automatically.",
+/**
+ * Admits a context written before mount-scoped state (eve 0.68 and earlier)
+ * only when every value keeps an owner in this deployment. Extension state
+ * moves to its mount's key when exactly one mount uses its package namespace.
+ */
+async function adoptLegacyStateLayout(
+ data: Record<string, unknown>,
+ bundle: CompiledBundle,
+): Promise<Record<string, unknown>> {
+ const manifest = await loadCompiledManifest({
+ compiledArtifactsSource: bundle.compiledArtifactsSource,
+ });
+ const mounts = [manifest, ...manifest.subagents.map((subagent) => subagent.agent)].flatMap(
+ (node) => node.extensionMounts,
+ );
+ const adopted: Record<string, unknown> = {};
+ for (const [name, value] of Object.entries(data)) {
+ if (value === undefined) continue;
+ if (name === BundleKey.name || resolveKey(name) !== undefined) {
+ adopted[name] = value;
+ continue;
+ }
+ const owners = mounts.flatMap((mount) => {
+ const prefix = `${legacyPackageStateNamespace(mount.packageName)}.`;
+ return name.startsWith(prefix)
+ ? [mountedStateKeyName(mount.mountId, name.slice(prefix.length))]
+ : [];
+ });
+ const [mounted] = owners;
+ if (
+ owners.length !== 1 ||
+ mounted === undefined ||
+ resolveKey(mounted) === undefined ||
+ data[mounted] !== undefined
+ ) {
+ throw new IncompatibleStateLayoutError(name);
+ }
+ adopted[mounted] = value;
+ }
+ return adopted;
+}
+
+/** The `defineState` prefix extensions used before state was scoped to mounts. */
+function legacyPackageStateNamespace(packageName: string): string {
+ return (
+ packageName
+ .replace(/^@/, "")
+ .replace(/[^a-zA-Z0-9._-]+/g, "-")
+ .replace(/^-+|-+$/g, "") || "extension"
);
}
+
+/** A context whose saved state this deployment cannot place; retrying never changes the answer. */
+export class IncompatibleStateLayoutError extends Error {
+ constructor(key?: string) {
+ super(
+ `Incompatible context state layout${key === undefined ? "" : ` for key "${key}"`}. Restore this session with its original deployment or start a new session; state with no unambiguous owner in this deployment cannot be carried forward.`,
+ );
+ this.name = "IncompatibleStateLayoutError";
+ }
+}
diff --git a/packages/eve/src/execution/legacy-remote-agent/protocol.ts b/packages/eve/src/execution/legacy-remote-agent/protocol.ts
index a4bb7436b..1d6da88cf 100644
--- a/packages/eve/src/execution/legacy-remote-agent/protocol.ts
+++ b/packages/eve/src/execution/legacy-remote-agent/protocol.ts
@@ -95,6 +95,23 @@ export async function forwardLegacySessionInput(
return true;
}
+/**
+ * Renames a `task.*` callback that a remote agent sends a protocol-1 caller to
+ * the payload kind the caller's session reads. The callback URL resolves to the
+ * caller's newest deployment, while the session waiting on it may still run on
+ * an eve 0.66–0.68 build; both read the renamed kinds. The caller still
+ * validates the result.
+ */
+export function renameLegacyTaskCallback(body: unknown): unknown {
+ if (!isObject(body)) return body;
+ const { taskId: _taskId, ...payload } = body;
+ if (body.kind === "task.input-requested") return { ...payload, kind: "subagent-input-request" };
+ if (body.kind === "task.authorization") {
+ return { ...payload, kind: "subagent-authorization-event" };
+ }
+ return body;
+}
+
/** Route a protocol-1 caller posts its answers to a remote agent's question or approval. */
export const legacyTaskInputRoute = {
handler: handleLegacyTaskInputRequest,
diff --git a/packages/eve/src/execution/session/checkpoint-migrations.test.ts b/packages/eve/src/execution/session/checkpoint-migrations.test.ts
new file mode 100644
index 000000000..6a0d37bf2
--- /dev/null
+++ b/packages/eve/src/execution/session/checkpoint-migrations.test.ts
@@ -0,0 +1,163 @@
+import { describe, expect, it } from "vitest";
+
+import { migrateSessionCheckpoint } from "#execution/session/checkpoint-migrations.js";
+import { SESSION_CHECKPOINT_VERSION } from "#execution/session/handoff.js";
+
+describe("migrateSessionCheckpoint", () => {
+ it("upgrades a checkpoint written by an eve 0.66 owner to the current shape", () => {
+ expect(migrateSessionCheckpoint(eve066Checkpoint())).toEqual({
+ kind: "current",
+ childRunIdsToStop: ["child-local"],
+ checkpoint: {
+ history: [
+ { content: "Alice asks for a report.", kind: "user", role: "user" },
+ { content: [{ text: "Started the report.", type: "text" }], role: "assistant" },
+ { content: "The report is ready.", kind: "task.result", role: "user" },
+ ],
+ serializedContext: {
+ "eve.auth": null,
+ "eve.bundle": { source: { kind: "bundled" } },
+ "eve.channel": { kind: "http", state: {} },
+ "eve.legacyRemoteAgentCaller": { taskId: "caller-task" },
+ "eve.sessionCallback": CALLBACK,
+ "eve.sessionTitle": "Alice asks for a report.",
+ },
+ sessionState: {
+ continuationToken: "",
+ emissionState: { sessionStarted: true, sequence: 1, stepIndex: 0, turnId: "" },
+ hasProxyInputRequests: false,
+ sessionId: "session-1",
+ snapshot: {
+ session: {
+ agent: { system: "Be helpful." },
+ continuationToken: "",
+ sandboxState: { session: null },
+ sessionId: "session-1",
+ state: { "eve.harness.requestEnvelopeTokens": 2483.5 },
+ },
+ },
+ version: 2,
+ },
+ sessionTimeoutMs: false,
+ version: SESSION_CHECKPOINT_VERSION,
+ },
+ });
+ });
+
+ it.each([
+ [
+ "a version 3 workflow tool run is unsettled",
+ { settledTask: false },
+ "workflow tool run registry version 3 holds unsettled runs",
+ ],
+ [
+ "a subagent session is working",
+ { subagentPhase: "running" },
+ "a subagent session is still working",
+ ],
+ ])("keeps the session on its owner while %s", (_name, options, detail) => {
+ expect(migrateSessionCheckpoint(eve066Checkpoint(options))).toEqual({
+ kind: "incompatible",
+ detail: `checkpoint version 8: ${detail}`,
+ });
+ });
+
+ it.each([7, SESSION_CHECKPOINT_VERSION + 1])("refuses checkpoint version %s", (version) => {
+ expect(migrateSessionCheckpoint({ ...eve066Checkpoint(), version })).toMatchObject({
+ kind: "incompatible",
+ });
+ });
+});
+
+const CALLBACK = {
+ callId: "call-0",
+ subagentName: "research",
+ token: "callback-token",
+ url: "https://caller.example/eve/v1/callback/callback-token",
+};
+
+/** Trimmed from a handoff checkpoint captured from an eve 0.66.1 owner. */
+function eve066Checkpoint({
+ settledTask = true,
+ subagentPhase = "parked",
+}: { readonly settledTask?: boolean; readonly subagentPhase?: string } = {}) {
+ return {
+ mode: "conversation",
+ serializedContext: {
+ "eve.auth": null,
+ "eve.bundle": { source: { kind: "bundled" } },
+ "eve.channel": { kind: "http", state: {} },
+ "eve.mode": "conversation",
+ "eve.runtime.taskDeliveryPolicy": "auto",
+ "eve.sessionCallback": { ...CALLBACK, taskId: "caller-task" },
+ "eve.sessionTitle": "Alice asks for a report.",
+ "eve.turnTaskDelivery": "none",
+ },
+ sessionState: {
+ continuationToken: "",
+ emissionState: { sessionStarted: true, sequence: 1, stepIndex: 0, turnId: "" },
+ hasProxyInputRequests: false,
+ sessionId: "session-1",
+ snapshot: {
+ session: {
+ agent: { system: "Be helpful." },
+ continuationToken: "",
+ history: [
+ { content: "Alice asks for a report.", kind: "user", role: "user" },
+ { content: [{ text: "Started the report.", type: "text" }], role: "assistant" },
+ { content: "The report is ready.", kind: "execution.background_task", role: "user" },
+ ],
+ sandboxState: { session: null },
+ sessionId: "session-1",
+ state: {
+ "eve.agent.handles": {
+ handles: [
+ {
+ address: {
+ continuationToken: "local-token",
+ kind: "agent/local",
+ sessionId: "child-local",
+ },
+ phase: subagentPhase,
+ },
+ {
+ address: {
+ kind: "agent/remote",
+ sessionId: "child-remote",
+ url: "https://remote.example",
+ },
+ phase: "available",
+ },
+ ],
+ },
+ "eve.harness.requestEnvelopeTokens": 2483.5,
+ "eve.workflowTool": {
+ runs: [
+ {
+ address: { hookToken: "hook-1", runId: "run-1" },
+ callId: "call-1",
+ lifetime: "session",
+ origin: { stepIndex: 0, turnId: "turn_0" },
+ task: {
+ dispatchContext: { auth: { current: null, initiator: null } },
+ metadata: { kind: "tool", name: "report" },
+ outcome: settledTask
+ ? { lastOutput: { type: "result", value: "ready" }, status: "completed" }
+ : undefined,
+ taskId: "task-1",
+ },
+ toolName: "report",
+ },
+ ],
+ version: 3,
+ },
+ },
+ taskId: "task-0",
+ },
+ },
+ version: 1,
+ },
+ sessionTimeoutMs: false,
+ version: 8,
+ };
+}
diff --git a/packages/eve/src/execution/session/checkpoint-migrations.ts b/packages/eve/src/execution/session/checkpoint-migrations.ts
new file mode 100644
index 000000000..5af9569bb
--- /dev/null
+++ b/packages/eve/src/execution/session/checkpoint-migrations.ts
@@ -0,0 +1,244 @@
+import { SESSION_CHECKPOINT_VERSION, type SessionCheckpoint } from "#execution/session/handoff.js";
+import { isObject } from "#shared/guards.js";
+
+/**
+ * Oldest checkpoint a successor upgrades (eve 0.66.0). Earlier checkpoints
+ * predate the sandbox provider and dynamic skill manifest contracts and lack
+ * data the current reader requires.
+ */
+export const MIN_SESSION_CHECKPOINT_VERSION = 8;
+
+export type SessionCheckpointMigration =
+ | {
+ readonly kind: "current";
+ readonly checkpoint: SessionCheckpoint;
+ /** Idle child sessions the current build no longer tracks; the successor stops them. */
+ readonly childRunIdsToStop: readonly string[];
+ }
+ | { readonly kind: "incompatible"; readonly detail: string };
+
+type CheckpointRecord = Record<string, unknown>;
+
+interface UpgradeEffects {
+ readonly childRunIdsToStop: string[];
+}
+
+/**
+ * One pure upgrade per checkpoint version, keyed by the version it reads.
+ * Bumping `SESSION_CHECKPOINT_VERSION` requires adding the step from the
+ * previous version: a successor must always accept its predecessors'
+ * checkpoints. A step refuses state the current build cannot continue, such
+ * as work still in flight; the owner then keeps the session. Steps cover the
+ * shapes released builds wrote.
+ */
+const CHECKPOINT_UPGRADES: Readonly<
+ Record<number, (checkpoint: CheckpointRecord, effects: UpgradeEffects) => CheckpointRecord>
+> = {
+ // Run mode was removed (eve 0.67).
+ 8: (checkpoint) => {
+ const { mode: _mode, ...rest } = checkpoint;
+ return {
+ ...rest,
+ serializedContext: omitKeys(readRecord(rest, "serializedContext"), ["eve.mode"]),
+ };
+ },
+ // eve 0.69 removed background tasks, subagent handles, activity artifacts, and
+ // cached connection search results, and moved protocol-1 caller detection.
+ 9: (checkpoint, effects) => {
+ const sessionState = readRecord(checkpoint, "sessionState");
+ const snapshot = readRecord(sessionState, "snapshot");
+ const { taskId: _taskId, ...session } = readRecord(snapshot, "session");
+ return {
+ ...checkpoint,
+ serializedContext: upgradeLegacyCaller(
+ omitKeys(readRecord(checkpoint, "serializedContext"), [
+ "eve.activityObserver",
+ "eve.activityPendingBlockers",
+ "eve.activityRootTurnId",
+ "eve.connectionSearchResults",
+ "eve.internal.backgroundToolExecution",
+ "eve.runtime.taskDeliveryPolicy",
+ "eve.turnTaskDelivery",
+ ]),
+ ),
+ sessionState: {
+ ...sessionState,
+ snapshot: {
+ ...snapshot,
+ session: {
+ ...session,
+ history: readArray(session, "history").map(renameBackgroundTaskMessage),
+ state: upgradeSessionState(session.state, effects),
+ },
+ },
+ },
+ };
+ },
+ // History moved out of the durable session snapshot (eve 0.70).
+ 10: (checkpoint) => {
+ const sessionState = readRecord(checkpoint, "sessionState");
+ const snapshot = readRecord(sessionState, "snapshot");
+ const { history, ...session } = readRecord(snapshot, "session");
+ if (sessionState.version !== 1) refuse("durable session version is not 1");
+ if (!Array.isArray(history)) refuse("session history is missing");
+ return {
+ ...checkpoint,
+ history,
+ sessionState: { ...sessionState, snapshot: { ...snapshot, session }, version: 2 },
+ };
+ },
+};
+
+/**
+ * Upgrades a checkpoint written by an older eve build to the current shape.
+ * Pure, so the session workflow can run it before validation. Newer
+ * checkpoints and those older than {@link MIN_SESSION_CHECKPOINT_VERSION} are
+ * incompatible.
+ */
+export function migrateSessionCheckpoint(checkpoint: unknown): SessionCheckpointMigration {
+ if (!isObject(checkpoint)) return { kind: "incompatible", detail: "checkpoint is not an object" };
+ const { version } = checkpoint;
+ if (
+ typeof version !== "number" ||
+ !Number.isSafeInteger(version) ||
+ version < MIN_SESSION_CHECKPOINT_VERSION ||
+ version > SESSION_CHECKPOINT_VERSION
+ ) {
+ return {
+ kind: "incompatible",
+ detail: `checkpoint version ${JSON.stringify(version)} is outside the supported range ${MIN_SESSION_CHECKPOINT_VERSION}-${SESSION_CHECKPOINT_VERSION}`,
+ };
+ }
+ const effects: UpgradeEffects = { childRunIdsToStop: [] };
+ let current: CheckpointRecord = checkpoint;
+ try {
+ for (let from = version; from < SESSION_CHECKPOINT_VERSION; from++) {
+ const upgrade = CHECKPOINT_UPGRADES[from];
+ if (upgrade === undefined) refuse(`no upgrade from checkpoint version ${from}`);
+ current = { ...upgrade(current, effects), version: from + 1 };
+ }
+ } catch (error) {
+ if (!(error instanceof CheckpointRefusal)) throw error;
+ return { kind: "incompatible", detail: `checkpoint version ${version}: ${error.message}` };
+ }
+ if (!isCurrentCheckpoint(current)) {
+ return { kind: "incompatible", detail: `checkpoint version ${version} is incomplete` };
+ }
+ return { kind: "current", checkpoint: current, childRunIdsToStop: effects.childRunIdsToStop };
+}
+
+function isCurrentCheckpoint(value: unknown): value is SessionCheckpoint {
+ return (
+ isObject(value) &&
+ value.version === SESSION_CHECKPOINT_VERSION &&
+ Array.isArray(value.history) &&
+ isObject(value.serializedContext) &&
+ isObject(value.sessionState)
+ );
+}
+
+class CheckpointRefusal extends Error {}
+
+function refuse(detail: string): never {
+ throw new CheckpointRefusal(detail);
+}
+
+function readRecord(value: CheckpointRecord, key: string): CheckpointRecord {
+ const field = value[key];
+ if (!isObject(field)) refuse(`${key} is not an object`);
+ return field;
+}
+
+function readArray(value: CheckpointRecord, key: string): unknown[] {
+ const field = value[key];
+ if (!Array.isArray(field)) refuse(`${key} is not an array`);
+ return field;
+}
+
+function omitKeys(record: CheckpointRecord, keys: readonly string[]): CheckpointRecord {
+ return Object.fromEntries(Object.entries(record).filter(([key]) => !keys.includes(key)));
+}
+
+/**
+ * Earlier builds forwarded a session's questions to its caller as `task.*`
+ * callbacks only when the callback named a task. The current build reads that
+ * choice from the protocol-1 caller record instead.
+ */
+function upgradeLegacyCaller(context: CheckpointRecord): CheckpointRecord {
+ const callback = context["eve.sessionCallback"];
+ if (!isObject(callback)) return context;
+ const { taskId, ...current } = callback;
+ return {
+ ...context,
+ "eve.legacyRemoteAgentCaller":
+ typeof taskId === "string" && taskId.length > 0 ? { taskId } : {},
+ "eve.sessionCallback": current,
+ };
+}
+
+// Task notifications are framework-authored task results; history validation rejects the old kind.
+function renameBackgroundTaskMessage(message: unknown): unknown {
+ return isObject(message) &&
+ message.role === "user" &&
+ message.kind === "execution.background_task"
+ ? { ...message, kind: "task.result" }
+ : message;
+}
+
+function upgradeSessionState(state: unknown, effects: UpgradeEffects): unknown {
+ if (!isObject(state)) return state;
+ return omitKeys(state, [
+ ...dropSettledWorkflowToolRuns(state),
+ ...stopIdleSubagents(state, effects),
+ ]);
+}
+
+/**
+ * Version 3 of the workflow tool run registry also recorded session-owned
+ * background tasks. Settled ones already reported to the conversation and have
+ * no current reader. Any other run means work is still in flight.
+ */
+function dropSettledWorkflowToolRuns(state: CheckpointRecord): string[] {
+ const registry = state["eve.workflowTool"];
+ if (!isObject(registry) || registry.version !== 3) return [];
+ const runs = registry.runs;
+ if (
+ !Array.isArray(runs) ||
+ !runs.every(
+ (run) =>
+ isObject(run) &&
+ run.lifetime === "session" &&
+ isObject(run.task) &&
+ run.task.outcome !== undefined,
+ )
+ ) {
+ refuse("workflow tool run registry version 3 holds unsettled runs");
+ }
+ return ["eve.workflowTool"];
+}
+
+/**
+ * Earlier builds kept idle subagent sessions resumable through handles. The
+ * current build has no reader for them, so the successor stops their local runs.
+ * Remote sessions end on their own deployment.
+ */
+function stopIdleSubagents(state: CheckpointRecord, effects: UpgradeEffects): string[] {
+ const store = state["eve.agent.handles"];
+ if (store === undefined) return [];
+ if (!isObject(store) || !Array.isArray(store.handles)) refuse("subagent handles are malformed");
+ for (const handle of store.handles) {
+ if (!isObject(handle) || (handle.phase !== "parked" && handle.phase !== "available")) {
+ refuse("a subagent session is still working");
+ }
+ const address = handle.address;
+ if (
+ isObject(address) &&
+ (address.kind === "agent/local" || address.kind === "agent/self") &&
+ typeof address.sessionId === "string" &&
+ address.sessionId.length > 0
+ ) {
+ effects.childRunIdsToStop.push(address.sessionId);
+ }
+ }
+ return ["eve.agent.handles"];
+}
diff --git a/packages/eve/src/execution/session/entry-handoff.integration.test.ts b/packages/eve/src/execution/session/entry-handoff.integration.test.ts
index 9e7cb3f2d..992f95faa 100644
--- a/packages/eve/src/execution/session/entry-handoff.integration.test.ts
+++ b/packages/eve/src/execution/session/entry-handoff.integration.test.ts
@@ -20,6 +20,8 @@ import { createWorkflowRuntime, waitForCommandHookOwner } from "#execution/workf
import { buildSerializedContext, handoffFollowUp } from "#internal/testing/entry-test-helpers.js";
import { captureConsoleOutput, workflowSdkNotice } from "#internal/testing/log-records.js";
+const HANDOFF_LOG = "[eve:execution.handoff]";
+
describe("workflowEntry integration", () => {
describe("deployment handoff", () => {
it("recovers the original owner when target rejects nested state", async () => {
@@ -164,8 +166,15 @@ describe("workflowEntry integration", () => {
expect.stringContaining(workflowSdkNotice.unpinnedDelivery),
);
expect(output.lines).toContainEqual(expect.stringContaining(workflowSdkNotice.maxRetries));
+ expect(output.lines).toContainEqual(
+ expect.stringContaining(`${HANDOFF_LOG} session handoff failed`),
+ );
expect(
- output.unexpected(workflowSdkNotice.unpinnedDelivery, workflowSdkNotice.maxRetries),
+ output.unexpected(
+ workflowSdkNotice.unpinnedDelivery,
+ workflowSdkNotice.maxRetries,
+ HANDOFF_LOG,
+ ),
).toEqual([]);
});
@@ -192,10 +201,10 @@ describe("workflowEntry integration", () => {
});
const rewritten = new Map<string, Promise<unknown>>();
const candidateIds = new Set<string>();
- // Rewrites only the candidate checkpoint version. Owner and candidate both
- // run this build, so the test covers successor validation and memoization
- // on a current owner — not an owner workflow still on a pre-upgrade build.
- const olderCheckpointInput = (runId: string, encoded: unknown): Promise<unknown> => {
+ // Rewrites only the candidate checkpoint version to one a newer eve build
+ // wrote, as after a rollback. Owner and candidate both run this build, so
+ // the test covers successor validation and memoization on a current owner.
+ const newerCheckpointInput = (runId: string, encoded: unknown): Promise<unknown> => {
let pending = rewritten.get(runId);
if (pending === undefined) {
pending = (async () => {
@@ -204,7 +213,7 @@ describe("workflowEntry integration", () => {
];
expect(args[0].kind).toBe("handoff");
candidateIds.add(runId);
- Object.assign(args[0].checkpoint, { version: SESSION_CHECKPOINT_VERSION - 1 });
+ Object.assign(args[0].checkpoint, { version: SESSION_CHECKPOINT_VERSION + 1 });
const operations: Promise<void>[] = [];
const result = await dehydrateWorkflowArguments(args, runId, undefined, operations);
@@ -220,7 +229,7 @@ describe("workflowEntry integration", () => {
const [runId] = args;
const event = args[1] as (typeof args)[1] | RunCreatedEventRequest;
if (event.eventType === "run_created" && event.eventData.deploymentId === "dpl_b") {
- event.eventData.input = await olderCheckpointInput(runId, event.eventData.input);
+ event.eventData.input = await newerCheckpointInput(runId, event.eventData.input);
}
return createEvent(...args);
});
@@ -231,7 +240,7 @@ describe("workflowEntry integration", () => {
runInput?: { deploymentId?: string; input: unknown };
};
if (message.runId !== undefined && message.runInput?.deploymentId === "dpl_b") {
- message.runInput.input = await olderCheckpointInput(
+ message.runInput.input = await newerCheckpointInput(
message.runId,
message.runInput.input,
);
@@ -339,7 +348,117 @@ describe("workflowEntry integration", () => {
expect(output.lines).toContainEqual(
expect.stringContaining(workflowSdkNotice.unpinnedDelivery),
);
+ // Both sides explain the refusal once: the successor names the version it
+ // cannot read and the owner names the target it keeps the session from.
+ expect(
+ output.lines.filter((line) => line.startsWith(`${HANDOFF_LOG} session handoff refused`)),
+ ).toHaveLength(1);
+ expect(
+ output.lines.filter((line) => line.startsWith(`${HANDOFF_LOG} session handoff failed`)),
+ ).toHaveLength(1);
// No validation retry: the SDK never reports an exhausted step.
+ expect(output.unexpected(workflowSdkNotice.unpinnedDelivery, HANDOFF_LOG)).toEqual([]);
+ });
+
+ it("continues an eve 0.66 checkpoint on a current successor and stops its idle children", async () => {
+ const output = captureConsoleOutput();
+ const runtime = await createTestRuntime({ agent: { name: "handoff-eve-066-owner" } });
+ await runtime.run(async () => {
+ const anchor = await start(workflowEntry, [
+ {
+ kind: "initial",
+ ownerDeploymentId: "dpl_a",
+ sessionTimeoutMs: false,
+ input: { message: "Alice opens a research session." },
+ serializedContext: buildSerializedContext({
+ acceptedDeploymentId: "dpl_a",
+ channelKind: "http",
+ }),
+ },
+ ]);
+ const stream = captureTurnEvents(anchor);
+ const world = await getWorld();
+ const workflowRuntime = createWorkflowRuntime({
+ compiledArtifactsSource: createBundledRuntimeCompiledArtifactsSource(),
+ });
+ const rewritten = new Map<string, Promise<unknown>>();
+ // The owner runs this build; only the candidate input changes to what an
+ // eve 0.66 owner sends, so the successor must upgrade the checkpoint.
+ const eve066Input = (runId: string, encoded: unknown): Promise<unknown> => {
+ let pending = rewritten.get(runId);
+ if (pending === undefined) {
+ pending = (async () => {
+ const args = (await hydrateWorkflowArguments(encoded, runId, undefined)) as [unknown];
+ args[0] = toEve066HandoffInput(args[0] as HandoffWorkflowEntryInput);
+ const operations: Promise<void>[] = [];
+ const result = await dehydrateWorkflowArguments(args, runId, undefined, operations);
+ await Promise.all(operations);
+ return result;
+ })();
+ rewritten.set(runId, pending);
+ }
+ return pending;
+ };
+ const createEvent = world.events.create.bind(world.events);
+ const created = vi.spyOn(world.events, "create").mockImplementation(async (...args) => {
+ const [runId] = args;
+ const event = args[1] as (typeof args)[1] | RunCreatedEventRequest;
+ if (event.eventType === "run_created" && event.eventData.deploymentId === "dpl_b") {
+ event.eventData.input = await eve066Input(runId, event.eventData.input);
+ }
+ return createEvent(...args);
+ });
+ const queue = world.queue.bind(world);
+ const queued = vi.spyOn(world, "queue").mockImplementation(async (...args) => {
+ const message = args[1] as {
+ runId?: string;
+ runInput?: { deploymentId?: string; input: unknown };
+ };
+ if (message.runId !== undefined && message.runInput?.deploymentId === "dpl_b") {
+ message.runInput.input = await eve066Input(message.runId, message.runInput.input);
+ }
+ return queue(...args);
+ });
+ try {
+ await stream.nextTurn();
+ await waitForParkedTurnStep(anchor.runId);
+
+ await workflowRuntime.dispatchSession({
+ command: handoffFollowUp("dpl_b", "Bob requests the next research step.", "eve-066"),
+ sessionId: anchor.runId,
+ });
+ expect((await stream.nextTurn()).at(-1)?.type).toBe("session.waiting");
+ const successor = await waitForCommandHookOwner(
+ sessionInboxHookToken(sessionCommandHookToken(anchor.runId)),
+ );
+ expect(rewritten.has(successor.runId)).toBe(true);
+ const successorSteps = await world.steps.list({ runId: successor.runId });
+ expect(
+ successorSteps.data.some((step) =>
+ step.stepName.endsWith("//stopUntrackedChildSessionsStep"),
+ ),
+ ).toBe(true);
+ await vi.waitFor(
+ async () => {
+ const [state] = await readTurnStepStates(successor.runId);
+ const userMessages = (state?.history ?? [])
+ .filter((message) => message.role === "user")
+ .map((message) => JSON.stringify(message.content));
+ // The successor's turn continues the history the old owner carried.
+ expect(userMessages).toEqual([
+ expect.stringContaining("Alice opens a research session."),
+ expect.stringContaining("Bob requests the next research step."),
+ ]);
+ },
+ { timeout: 5000 },
+ );
+ } finally {
+ created.mockRestore();
+ queued.mockRestore();
+ stream.dispose();
+ await anchor.cancel();
+ }
+ });
expect(output.unexpected(workflowSdkNotice.unpinnedDelivery)).toEqual([]);
});
@@ -603,3 +722,50 @@ describe("workflowEntry integration", () => {
});
});
});
+
+/** The handoff input an eve 0.66 owner sends, per a captured checkpoint version 8. */
+function toEve066HandoffInput(input: HandoffWorkflowEntryInput): unknown {
+ const { checkpoint, handoffVersion: _handoffVersion, ...fields } = input;
+ const { history, serializedContext, sessionState, ...checkpointFields } = checkpoint;
+ const { "eve.stateLayout": _stateLayout, ...context } = serializedContext;
+ return {
+ ...fields,
+ checkpoint: {
+ ...checkpointFields,
+ mode: "conversation",
+ serializedContext: {
+ ...context,
+ "eve.mode": "conversation",
+ "eve.runtime.taskDeliveryPolicy": "auto",
+ "eve.turnTaskDelivery": "none",
+ },
+ sessionState: {
+ ...sessionState,
+ snapshot: {
+ session: {
+ ...sessionState.snapshot.session,
+ history,
+ state: {
+ ...sessionState.snapshot.session.state,
+ // A finished subagent the old owner kept resumable; its run no longer exists.
+ "eve.agent.handles": {
+ handles: [
+ {
+ address: {
+ continuationToken: "child-token",
+ kind: "agent/local",
+ sessionId: "wrun_child_from_eve_066",
+ },
+ phase: "parked",
+ },
+ ],
+ },
+ },
+ },
+ },
+ version: 1,
+ },
+ version: 8,
+ },
+ };
+}
diff --git a/packages/eve/src/execution/session/entry.ts b/packages/eve/src/execution/session/entry.ts
index 42a7cc636..174b41d71 100644
--- a/packages/eve/src/execution/session/entry.ts
+++ b/packages/eve/src/execution/session/entry.ts
@@ -21,8 +21,11 @@ import {
} from "#execution/session-inbox/address.js";
import {
signalSessionOwnerActivationStep,
+ stopUntrackedChildSessionsStep,
validateSessionCheckpointStep,
} from "#execution/session/handoff-steps.js";
+import { migrateSessionCheckpoint } from "#execution/session/checkpoint-migrations.js";
+import type { SessionCheckpoint } from "#execution/session/handoff.js";
import type {
HandoffWorkflowEntryInput,
InitialWorkflowEntryInput,
@@ -158,19 +161,25 @@ async function bootInitialOwner(
async function bootHandoffOwner(
input: HandoffWorkflowEntryInput,
): Promise<BootOutcome | undefined> {
- const { checkpoint, sessionId } = input;
- const serializedContext = stampSessionIdentity(checkpoint.serializedContext, sessionId);
+ const { sessionId } = input;
const inbox = createSessionInbox(sessionId);
+ let checkpoint: SessionCheckpoint;
+ let childRunIdsToStop: readonly string[];
+ let serializedContext: Record<string, unknown>;
try {
- const validation = await validateSessionCheckpointStep({ checkpoint });
- if (validation.kind === "incompatible") {
+ // The previous owner may run an older eve build; read its checkpoint in this build's shape.
+ const migration = migrateSessionCheckpoint(input.checkpoint);
+ const validation = await validateSessionCheckpointStep({ migration, sessionId });
+ if (migration.kind === "incompatible" || validation.kind === "incompatible") {
const payloads = await inbox.release();
await signalSessionOwnerActivationStep({
- activation: { kind: "incompatible", payloads, reason: validation.reason },
+ activation: { kind: "incompatible", payloads, reason: "checkpoint-version" },
token: input.activationToken,
});
return undefined;
}
+ ({ checkpoint, childRunIdsToStop } = migration);
+ serializedContext = stampSessionIdentity(checkpoint.serializedContext, sessionId);
await inbox.claimSessionHooks(
sessionHookTokens({ serializedContext, sessionState: checkpoint.sessionState }),
);
@@ -186,6 +195,9 @@ async function bootHandoffOwner(
});
return undefined;
}
+ if (childRunIdsToStop.length > 0) {
+ await stopUntrackedChildSessionsStep({ runIds: childRunIdsToStop, sessionId });
+ }
return {
inbox,
session: {
diff --git a/packages/eve/src/execution/session/handoff-steps.test.ts b/packages/eve/src/execution/session/handoff-steps.test.ts
index 18175b3ca..9376be5a0 100644
--- a/packages/eve/src/execution/session/handoff-steps.test.ts
+++ b/packages/eve/src/execution/session/handoff-steps.test.ts
@@ -1,20 +1,32 @@
import { createTestSessionState } from "#internal/testing/session-state.js";
import { beforeEach, describe, expect, it, vi } from "vitest";
+import { WorkflowRunNotFoundError } from "#compiled/@workflow/errors/index.js";
+import { IncompatibleStateLayoutError } from "#context/serialize.js";
import { SESSION_CHECKPOINT_VERSION, type SessionCheckpoint } from "#execution/session/handoff.js";
-import { validateSessionCheckpointStep } from "#execution/session/handoff-steps.js";
+import {
+ stopUntrackedChildSessionsStep,
+ validateSessionCheckpointStep,
+} from "#execution/session/handoff-steps.js";
import { BundleKey } from "#runtime/sessions/runtime-context-keys.js";
const deserializeContextMock = vi.fn();
const readDurableSessionMock = vi.fn();
+const cancelRunMock = vi.fn();
-vi.mock("#context/serialize.js", () => ({
+vi.mock("#context/serialize.js", async (importOriginal) => ({
+ ...(await importOriginal()),
deserializeContext: (...args: unknown[]) => deserializeContextMock(...args),
}));
vi.mock("#execution/durable-session-store.js", async (importOriginal) => ({
...(await importOriginal()),
readDurableSession: (...args: unknown[]) => readDurableSessionMock(...args),
}));
+vi.mock("#internal/workflow/runtime.js", async (importOriginal) => ({
+ ...(await importOriginal()),
+ cancelRun: (...args: unknown[]) => cancelRunMock(...args),
+ getWorld: async () => ({}),
+}));
describe("validateSessionCheckpointStep", () => {
beforeEach(() => {
@@ -27,7 +39,7 @@ describe("validateSessionCheckpointStep", () => {
readDurableSessionMock.mockReturnValue({});
const checkpoint = createCheckpoint();
- await expect(validateSessionCheckpointStep({ checkpoint })).resolves.toEqual({ kind: "valid" });
+ await expect(validate(checkpoint)).resolves.toEqual({ kind: "valid" });
expect(require).toHaveBeenCalledWith(BundleKey);
expect(readDurableSessionMock).toHaveBeenCalledWith(checkpoint.sessionState);
@@ -50,39 +62,80 @@ describe("validateSessionCheckpointStep", () => {
},
},
});
- await expect(validateSessionCheckpointStep({ checkpoint: createCheckpoint() })).rejects.toThrow(
+ await expect(validate(createCheckpoint())).rejects.toThrow(
"Corrupt workflow tool run registry",
);
});
- it.each([5, 6, 7, 8, 9, 10, 12])(
- "reports checkpoint version %s as incompatible before reading nested state",
- async (version) => {
- const checkpoint = createCheckpoint();
- // Simulate an incompatible checkpoint received over the wire.
- Object.assign(checkpoint, { version });
-
- await expect(validateSessionCheckpointStep({ checkpoint })).resolves.toEqual({
- kind: "incompatible",
- reason: "checkpoint-version",
- });
- expect(deserializeContextMock).not.toHaveBeenCalled();
- expect(readDurableSessionMock).not.toHaveBeenCalled();
- },
- );
+ it("reports a checkpoint that could not be upgraded without reading it", async () => {
+ const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+ const detail = "checkpoint version 7 is outside the supported range";
+
+ await expect(
+ validateSessionCheckpointStep({
+ migration: { kind: "incompatible", detail },
+ sessionId: "session-1",
+ }),
+ ).resolves.toEqual({ kind: "incompatible", reason: "checkpoint-version" });
+ expect(deserializeContextMock).not.toHaveBeenCalled();
+ expect(warn).toHaveBeenCalledWith(
+ expect.stringContaining("cannot read the checkpoint"),
+ expect.objectContaining({ detail, sessionId: "session-1" }),
+ );
+ warn.mockRestore();
+ });
+
+ it("answers rather than retries when saved state has no owner here", async () => {
+ const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+ deserializeContextMock.mockRejectedValue(new IncompatibleStateLayoutError("acme-crm.requests"));
+
+ await expect(validate(createCheckpoint())).resolves.toEqual({
+ kind: "incompatible",
+ reason: "checkpoint-version",
+ });
+ expect(warn).toHaveBeenCalledWith(
+ expect.stringContaining("cannot read the checkpoint"),
+ expect.objectContaining({ detail: expect.stringContaining('"acme-crm.requests"') }),
+ );
+ warn.mockRestore();
+ });
it.each([undefined, -1, NaN, Infinity, "30000", true])(
"rejects an invalid renewal duration (%s)",
async (sessionTimeoutMs) => {
const checkpoint = { ...createCheckpoint(), sessionTimeoutMs } as SessionCheckpoint;
- await expect(validateSessionCheckpointStep({ checkpoint })).rejects.toThrow(
- "invalid timeout duration",
- );
+ await expect(validate(checkpoint)).rejects.toThrow("invalid timeout duration");
expect(deserializeContextMock).not.toHaveBeenCalled();
},
);
});
+describe("stopUntrackedChildSessionsStep", () => {
+ it("stops every child except the session itself and tolerates finished runs", async () => {
+ cancelRunMock.mockReset();
+ cancelRunMock.mockImplementation(async (_world: unknown, runId: string) => {
+ if (runId === "child-gone") throw new WorkflowRunNotFoundError(runId);
+ });
+
+ await stopUntrackedChildSessionsStep({
+ runIds: ["child-gone", "session-1", "child-parked"],
+ sessionId: "session-1",
+ });
+
+ expect(cancelRunMock.mock.calls.map(([, runId]) => runId)).toEqual([
+ "child-gone",
+ "child-parked",
+ ]);
+ });
+});
+
+function validate(checkpoint: SessionCheckpoint) {
+ return validateSessionCheckpointStep({
+ migration: { kind: "current", checkpoint, childRunIdsToStop: [] },
+ sessionId: "session-1",
+ });
+}
+
function createCheckpoint(): SessionCheckpoint {
return {
version: SESSION_CHECKPOINT_VERSION,
diff --git a/packages/eve/src/execution/session/handoff-steps.ts b/packages/eve/src/execution/session/handoff-steps.ts
index 297b0763a..f8b039fd1 100644
--- a/packages/eve/src/execution/session/handoff-steps.ts
+++ b/packages/eve/src/execution/session/handoff-steps.ts
@@ -1,17 +1,26 @@
import { getBlockingWorkflowToolRuns } from "#harness/workflow-tool-runs.js";
-import { deserializeContext } from "#context/serialize.js";
+import {
+ EntityConflictError,
+ RunExpiredError,
+ WorkflowRunNotFoundError,
+} from "#compiled/@workflow/errors/index.js";
+import { deserializeContext, IncompatibleStateLayoutError } from "#context/serialize.js";
+import type { SessionCheckpointMigration } from "#execution/session/checkpoint-migrations.js";
import { readDurableSession, type DurableSessionState } from "#execution/durable-session-store.js";
import {
SESSION_CHECKPOINT_VERSION,
- type SessionCheckpoint,
type SessionOwnerActivation,
} from "#execution/session/handoff.js";
-import { resumeHook } from "#internal/workflow/runtime.js";
+import { createLogger, logError } from "#internal/logging.js";
+import { cancelRun, getWorld, resumeHook } from "#internal/workflow/runtime.js";
import { getResolvedRuntimeAgentNode } from "#runtime/graph.js";
import { BundleKey } from "#runtime/sessions/runtime-context-keys.js";
import { getSandboxEnvironmentRuntime } from "#shared/sandbox-environment.js";
+import { walkCauseChain } from "#shared/errors.js";
import { isObject } from "#shared/guards.js";
+const log = createLogger("execution.handoff");
+
/** Parses retained work with this deployment's code before deciding whether it can move. */
export function isSessionStateIdleForHandoff(sessionState: DurableSessionState): boolean {
const { state } = readDurableSession(sessionState);
@@ -52,26 +61,41 @@ export type SessionCheckpointValidation =
| { readonly kind: "incompatible"; readonly reason: "checkpoint-version" };
/**
- * Validates a checkpoint and resolves the target deployment's compiled bundle.
+ * Validates an upgraded checkpoint and resolves the target deployment's
+ * compiled bundle.
*
- * A version mismatch is a settled answer about this deployment, not a fault, so
- * it returns rather than throws: retrying the step can never change it.
+ * A checkpoint this deployment cannot read is a settled answer, not a fault,
+ * so it returns rather than throws: retrying the step can never change it.
*/
export async function validateSessionCheckpointStep(input: {
- readonly checkpoint: SessionCheckpoint;
+ readonly migration: SessionCheckpointMigration;
+ readonly sessionId: string;
}): Promise<SessionCheckpointValidation> {
"use step";
- const { checkpoint } = input;
- if (checkpoint.version !== SESSION_CHECKPOINT_VERSION) {
+ const { migration } = input;
+ const unreadable = (detail: string): SessionCheckpointValidation => {
+ log.warn("session handoff refused: this deployment cannot read the checkpoint", {
+ detail,
+ readableCheckpointVersion: SESSION_CHECKPOINT_VERSION,
+ sessionId: input.sessionId,
+ });
return { kind: "incompatible", reason: "checkpoint-version" };
- }
+ };
+ if (migration.kind === "incompatible") return unreadable(migration.detail);
+ const { checkpoint } = migration;
const timeout = checkpoint.sessionTimeoutMs;
if (
timeout !== false &&
(typeof timeout !== "number" || !Number.isFinite(timeout) || timeout < 0)
)
throw new Error("Session checkpoint contains an invalid timeout duration.");
- const context = await deserializeContext(checkpoint.serializedContext);
+ let context: Awaited<ReturnType<typeof deserializeContext>>;
+ try {
+ context = await deserializeContext(checkpoint.serializedContext);
+ } catch (error) {
+ if (error instanceof IncompatibleStateLayoutError) return unreadable(error.message);
+ throw error;
+ }
const bundle = context.require(BundleKey);
const session = readDurableSession(checkpoint.sessionState);
const sandboxState = session.sandboxState?.session;
@@ -97,6 +121,52 @@ export async function validateSessionCheckpointStep(input: {
return { kind: "valid" };
}
+/**
+ * Stops idle subagent sessions an older owner kept resumable, after this owner
+ * activates. Best effort: a child that cannot be stopped ends at its own timeout.
+ */
+export async function stopUntrackedChildSessionsStep(input: {
+ readonly runIds: readonly string[];
+ readonly sessionId: string;
+}): Promise<void> {
+ "use step";
+ const world = await getWorld();
+ for (const runId of input.runIds) {
+ if (runId === input.sessionId) continue;
+ try {
+ await cancelRun(world, runId, { cancelReason: "Session upgraded" });
+ } catch (error) {
+ const settled = [...walkCauseChain(error)].some(
+ (cause) =>
+ EntityConflictError.is(cause) ||
+ RunExpiredError.is(cause) ||
+ WorkflowRunNotFoundError.is(cause),
+ );
+ if (!settled) {
+ logError(log, "could not stop a subagent session after handoff", error, {
+ childRunId: runId,
+ sessionId: input.sessionId,
+ });
+ }
+ }
+ }
+}
+
+/** Records why the owner kept a session it tried to move, so a stuck handoff is visible. */
+export async function reportSessionHandoffRetainedStep(input: {
+ readonly error?: string;
+ readonly reason: "activation-failed" | "checkpoint-incompatible";
+ readonly sessionId: string;
+ readonly sourceDeploymentId: string;
+ readonly targetDeploymentId: string;
+}): Promise<void> {
+ "use step";
+ log.warn("session handoff failed; the current owner keeps the session", {
+ ...input,
+ checkpointVersion: SESSION_CHECKPOINT_VERSION,
+ });
+}
+
export async function signalSessionOwnerActivationStep(input: {
readonly activation: SessionOwnerActivation;
readonly token: string;
diff --git a/packages/eve/src/execution/session/handoff.test.ts b/packages/eve/src/execution/session/handoff.test.ts
index 8ede9bea1..93544686c 100644
--- a/packages/eve/src/execution/session/handoff.test.ts
+++ b/packages/eve/src/execution/session/handoff.test.ts
@@ -7,11 +7,14 @@ import type { TurnSelection } from "#execution/session/input-queue.js";
import type { SessionInboxHandle, SessionInboxPayload } from "#execution/session-inbox/inbox.js";
const isSessionIdleForHandoffStepMock = vi.fn(async (..._args: unknown[]) => true);
+const reportSessionHandoffRetainedStepMock = vi.fn();
const startSessionOwnerStepMock = vi.fn();
const createHookMock = vi.fn();
vi.mock("#execution/session/handoff-steps.js", () => ({
isSessionIdleForHandoffStep: (...args: unknown[]) => isSessionIdleForHandoffStepMock(...args),
+ reportSessionHandoffRetainedStep: (...args: unknown[]) =>
+ reportSessionHandoffRetainedStepMock(...args),
}));
vi.mock("#execution/workflow-runtime.js", () => ({
startSessionOwnerStep: (...args: unknown[]) => startSessionOwnerStepMock(...args),
@@ -173,6 +176,18 @@ describe("SessionHandoff", () => {
reason: "known-incompatible",
});
expect(startSessionOwnerStepMock).toHaveBeenCalledTimes(2);
+ // Each refusing deployment is reported once; remembered skips stay quiet.
+ expect(reportSessionHandoffRetainedStepMock.mock.calls).toEqual(
+ ["deployment-b", "deployment-c"].map((targetDeploymentId) => [
+ {
+ error: undefined,
+ reason: "checkpoint-incompatible",
+ sessionId: "session-1",
+ sourceDeploymentId: "deployment-a",
+ targetDeploymentId,
+ },
+ ]),
+ );
});
it("attempts handoff on every turn when the owner only sees a generic activation failure", async () => {
@@ -189,6 +204,9 @@ describe("SessionHandoff", () => {
reason: "activation-failed",
});
expect(startSessionOwnerStepMock).toHaveBeenCalledTimes(2);
+ expect(reportSessionHandoffRetainedStepMock).toHaveBeenCalledWith(
+ expect.objectContaining({ error: "unsupported checkpoint", reason: "activation-failed" }),
+ );
});
it("abandons transfer when input was accepted during release", async () => {
diff --git a/packages/eve/src/execution/session/handoff.ts b/packages/eve/src/execution/session/handoff.ts
index 8b5fbf11a..2a8a0759f 100644
--- a/packages/eve/src/execution/session/handoff.ts
+++ b/packages/eve/src/execution/session/handoff.ts
@@ -11,15 +11,20 @@ import type { TurnSelection } from "#execution/session/input-queue.js";
import type { WorkflowEntryResult } from "#execution/session/entry-input.js";
import { startSessionOwnerStep } from "#execution/workflow-runtime.js";
import { sessionHandoffMarkerToken } from "#execution/session-inbox/address.js";
-import { isSessionIdleForHandoffStep } from "#execution/session/handoff-steps.js";
+import {
+ isSessionIdleForHandoffStep,
+ reportSessionHandoffRetainedStep,
+} from "#execution/session/handoff-steps.js";
import type { AgentWorkflowRetentionDefinition } from "#shared/agent-definition.js";
+import { isObject } from "#shared/guards.js";
/**
* Cross-deployment checkpoint contract. The successor may run a different eve
* build than the owner that produced it; bump when any field changes shape so
- * an incompatible successor rejects the handoff instead of misreading state.
- * Mount-owned state requires a new version so older readers reject it rather
- * than dropping unrecognized state keys during reverse handoff.
+ * an older successor rejects the handoff instead of misreading state.
+ *
+ * A newer successor must always accept older checkpoints: every bump adds the
+ * upgrade from the previous version in `checkpoint-migrations.ts`.
*/
export const SESSION_CHECKPOINT_VERSION = 11;
@@ -136,6 +141,10 @@ export class SessionHandoff {
createHook<never>({ token: sessionHandoffMarkerToken(token) }),
);
await Promise.all(markers.map((marker) => claimHookOwnership(marker)));
+ let retained: {
+ readonly error?: unknown;
+ readonly reason: "activation-failed" | "checkpoint-incompatible";
+ };
try {
const acceptedDuringRelease = await inbox.release();
if (acceptedDuringRelease.length > 0) {
@@ -150,20 +159,23 @@ export class SessionHandoff {
targetDeploymentId,
);
if (activation.kind === "active") return { kind: "transferred" };
+ acceptedByFailedCandidate = activation.payloads;
if (activation.kind === "incompatible") {
this.incompatibleTargetDeploymentIds.add(targetDeploymentId);
- await this.recover(tokens, activation.payloads);
- return { kind: "retained", reason: "checkpoint-incompatible" };
+ retained = { reason: "checkpoint-incompatible" };
+ } else {
+ retained = { error: activation.error, reason: "activation-failed" };
}
- acceptedByFailedCandidate = activation.payloads;
- } catch {
+ } catch (error) {
// The current owner remains authoritative until activation.
+ retained = { error, reason: "activation-failed" };
}
await this.recover(tokens, acceptedByFailedCandidate);
- return { kind: "retained", reason: "activation-failed" };
} finally {
await Promise.all(markers.map((marker) => disposeHook(marker)));
}
+ await this.reportRetained(retained.reason, targetDeploymentId, retained.error);
+ return { kind: "retained", reason: retained.reason };
}
/** After a transfer, the original run parks until the final owner reports the session result. */
@@ -216,6 +228,22 @@ export class SessionHandoff {
this.input.inbox.restore(payloads);
}
+ private async reportRetained(
+ reason: "activation-failed" | "checkpoint-incompatible",
+ targetDeploymentId: string,
+ error?: unknown,
+ ): Promise<void> {
+ const message =
+ isObject(error) && typeof error.message === "string" ? error.message : undefined;
+ await reportSessionHandoffRetainedStep({
+ error: message,
+ reason,
+ sessionId: this.input.sessionId,
+ sourceDeploymentId: this.input.deploymentId,
+ targetDeploymentId,
+ });
+ }
+
private async ensureAnchor(): Promise<void> {
// Only the original run outlives successors; intermediate owners exit.
if (!this.input.isInitialOwner || this.anchor !== undefined) return;
diff --git a/packages/eve/src/public/definitions/state.ts b/packages/eve/src/public/definitions/state.ts
index dbf6c7355..5bd338cbe 100644
--- a/packages/eve/src/public/definitions/state.ts
+++ b/packages/eve/src/public/definitions/state.ts
@@ -56,10 +56,12 @@ export function defineMountedState<T>(
name: string,
initial: () => T,
): StateHandle<T> {
- return createStateHandle(
- `eve:mount.v1:${encodeURIComponent(mountId)}:${encodeURIComponent(name)}`,
- initial,
- );
+ return createStateHandle(mountedStateKeyName(mountId, name), initial);
+}
+
+/** Durable context key name of one mount's state slot. */
+export function mountedStateKeyName(mountId: string, name: string): string {
+ return `eve:mount.v1:${encodeURIComponent(mountId)}:${encodeURIComponent(name)}`;
}
function createStateHandle<T>(name: string, initial: () => T): StateHandle<T> {
diff --git a/packages/eve/src/subagents/callback-route.test.ts b/packages/eve/src/subagents/callback-route.test.ts
index 006fb5814..3db0f2092 100644
--- a/packages/eve/src/subagents/callback-route.test.ts
+++ b/packages/eve/src/subagents/callback-route.test.ts
@@ -109,15 +109,28 @@ describe("session callback route", () => {
kind: "subagent-authorization-event",
subagentName: "research",
};
- const response = await handleSessionCallbackRequest(
- new Request("https://app.example.com/eve/v1/callback/tok123", {
- body: JSON.stringify(payload),
- method: "POST",
- }),
- createRouteContext({ token: "tok123" }),
- );
- expect(response.status).toBe(202);
- expect(resumeHookMock).toHaveBeenCalledWith("tok123", payload);
+ // A remote agent sends the protocol-1 form to a session created by eve 0.66–0.68,
+ // whose callback URL now resolves to this deployment.
+ const legacy = {
+ ...payload,
+ childContinuationToken: "child-inbox",
+ kind: "task.authorization",
+ taskId: "task-1",
+ };
+ for (const body of [payload, legacy]) {
+ const response = await handleSessionCallbackRequest(
+ new Request("https://app.example.com/eve/v1/callback/tok123", {
+ body: JSON.stringify(body),
+ method: "POST",
+ }),
+ createRouteContext({ token: "tok123" }),
+ );
+ expect(response.status).toBe(202);
+ }
+ expect(resumeHookMock.mock.calls).toEqual([
+ ["tok123", payload],
+ ["tok123", payload],
+ ]);
});
it.each(["approval.candidate", "approval.settled"])(
diff --git a/packages/eve/src/subagents/callback-route.ts b/packages/eve/src/subagents/callback-route.ts
index 648e29012..1c2c6a81f 100644
--- a/packages/eve/src/subagents/callback-route.ts
+++ b/packages/eve/src/subagents/callback-route.ts
@@ -1,3 +1,4 @@
+import { renameLegacyTaskCallback } from "#execution/legacy-remote-agent/protocol.js";
import { resumeHook } from "#internal/workflow/runtime.js";
import { z } from "#compiled/zod/index.js";
import type { RouteContext } from "#public/definitions/channel.js";
@@ -135,7 +136,7 @@ export async function handleSessionCallbackRequest(
let body: unknown;
try {
- body = await request.json();
+ body = renameLegacyTaskCallback(await request.json());
} catch {
return Response.json({ error: "Invalid JSON body.", ok: false }, { status: 400 });
}
diff --git a/scripts/guard-invariants.mjs b/scripts/guard-invariants.mjs
index 3098716ed..540e3637d 100644
--- a/scripts/guard-invariants.mjs
+++ b/scripts/guard-invariants.mjs
@@ -489,6 +489,7 @@ const LEGACY_REMOTE_AGENT_INGRESS_FILES = new Set([
"packages/eve/src/eve-channel/index.ts",
"packages/eve/src/eve-channel/request.ts",
"packages/eve/src/execution/forward-session-input.ts",
+ "packages/eve/src/subagents/callback-route.ts",
]);
// `from "…"` covers imports and re-exports; `import "…"` and `import("…")` cover side effects and dynamic imports.
const LEGACY_REMOTE_AGENT_IMPORT_RE = /\b(?:from|import)\s*\(?\s*["'][^"']*legacy-remote-agent\//;