Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(eve): replay turn-scoped dynamic tools across processes (#3912)

Casey Gowrie2998dc160989
diff --git a/.changeset/bright-tools-replay.md b/.changeset/bright-tools-replay.md
new file mode 100644
index 000000000..f97aeea09
--- /dev/null
+++ b/.changeset/bright-tools-replay.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Turn-scoped dynamic tools now restore their durable callbacks when a turn resumes in a different process, so tools resolved at `turn.started` remain callable throughout the turn.
diff --git a/docs/guides/dynamic-capabilities.md b/docs/guides/dynamic-capabilities.md
index 0e9654250..b393a2c15 100644
--- a/docs/guides/dynamic-capabilities.md
+++ b/docs/guides/dynamic-capabilities.md
@@ -330,7 +330,10 @@ A parked call binds to its callback within its session, lifecycle scope, and res
- Editing a callback body while keeping its resolver entry and tool names is safe: replaying a parked call runs the latest deployed code with the closure values snapshotted when the call was made.
- If a persisted session-scoped callback has no registered implementation (a fresh process, a redeploy, or an expired in-process binding), eve re-runs `session.started` resolvers once to rebind it, then replays.
-- If the owning resolver no longer returns that tool, replay fails closed with an explicit error instead of invoking something else. Ordinary turn-scoped and step-scoped tools are not rebound; a parked call to a missing one errors. Framework-provided resolvers such as memory provider-tool wrappers opt into the same generic missing-callback rebind while preserving their locked scope.
+- If an active turn resumes without a registered turn-scoped callback, eve re-runs the owning `turn.started` resolver to restore the callback while preserving the tool set and closure captured earlier in that turn. If an authored resolver no longer returns the tool, the turn can continue, but calling that tool fails closed. Framework-provided resolvers such as memory provider-tool wrappers require all of their callbacks to be restored and fail the continuation if their locked tool set changed.
+- Step-scoped callbacks are restored from the persisted step immediately before eve replays that step.
+
+A recovery rebind is not a new lifecycle event, but it can run resolver code again. Keep `session.started` and `turn.started` resolvers idempotent and return the same tool identities for the same persisted scope.
### Naming
@@ -347,13 +350,13 @@ A dynamic connection, tool, or skill whose name matches an **authored** one **ov
### Events
-| Event | Resolver runs | Tools available for |
-| ----------------- | ----------------------------------------------------- | ------------------------------- |
-| `session.started` | At session start; may be redelivered during recovery¹ | Every model call in the session |
-| `turn.started` | Once per turn | Every model call in the turn |
-| `step.started` | Before each model call | That model call |
+| Event | Resolver runs | Tools available for |
+| ----------------- | -------------------------------------------------------- | ------------------------------- |
+| `session.started` | At session start; may be redelivered during recovery¹ | Every model call in the session |
+| `turn.started` | Once per turn; may re-run to restore a missing callback¹ | Every model call in the turn |
+| `step.started` | Before each model call | That model call |
-¹ Workflow recovery can redeliver a resolver event, so keep resolvers idempotent. Replaying a parked callback does not depend on running the resolver again — except for the one-shot rebind described under [Identity and redeploys](#identity-and-redeploys).
+¹ Workflow recovery can redeliver an event or re-run a resolver to restore a missing callback, so keep resolvers idempotent. Rebinding restores the persisted tool set; it does not make newly returned tools available in the active turn.
At `turn.started`, model, tool, skill, and subagent resolvers receive the visible conversation history and incoming message in `ctx.messages`, oldest first. Request context is included, and history projection still applies. Read these messages from the handler's second argument; the event itself contains turn metadata. Instruction resolvers use the separate snapshot described under [Dynamic instructions](#dynamic-instructions).
diff --git a/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts b/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts
index e3676fbe1..87dc88554 100644
--- a/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts
+++ b/e2e/fixtures/agent-tools-sandbox/agent/lib/mock-responder.ts
@@ -12,6 +12,17 @@ const SKILL_DIRECTIVE = /load the `([^`]+)` skill/iu;
*/
export function respond(request: MockModelRequest): MockModelResponse | string {
const message = request.lastUserMessage ?? "";
+ if (message.includes("DYNAMIC-TURN-REPLAY-START")) {
+ const gate = request.toolResults.find((result) => result.name === "dynamic-turn-replay-gate");
+ if (gate === undefined) {
+ return { toolCalls: [{ input: {}, name: "dynamic-turn-replay-gate" }] };
+ }
+ const probe = request.toolResults.find((result) => result.name === "dynamic_turn_replay_probe");
+ return probe === undefined
+ ? { toolCalls: [{ input: {}, name: "dynamic_turn_replay_probe" }] }
+ : formatOutput(probe.output);
+ }
+
let lastAuthoredUserIndex = -1;
let lastToolResultIndex = -1;
for (let index = 0; index < request.messages.length; index += 1) {
diff --git a/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay-gate.ts b/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay-gate.ts
new file mode 100644
index 000000000..26ccac137
--- /dev/null
+++ b/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay-gate.ts
@@ -0,0 +1,31 @@
+import {
+ defineWorkflowTool,
+ type WorkflowToolContext,
+ type WorkflowToolDefinition,
+} from "eve/tools";
+
+export async function dynamicTurnReplayGate(_input: unknown, ctx: WorkflowToolContext) {
+ "use workflow";
+
+ const answer = await ctx.ask({
+ display: "confirmation",
+ options: [
+ { id: "approve", label: "Continue", style: "primary" },
+ { id: "cancel", label: "Cancel" },
+ ],
+ prompt: "Resume the turn-scoped dynamic tool check?",
+ });
+ return { approved: answer.status === "answered" && answer.optionId === "approve" };
+}
+
+const tool: WorkflowToolDefinition<
+ Record<string, unknown>,
+ { approved: boolean }
+> = defineWorkflowTool({
+ description:
+ "Pause the dynamic turn replay check until the user approves. Only call for DYNAMIC-TURN-REPLAY-START.",
+ inputSchema: { type: "object" },
+ execute: dynamicTurnReplayGate,
+});
+
+export default tool;
diff --git a/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay.ts b/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay.ts
new file mode 100644
index 000000000..ca913b0fc
--- /dev/null
+++ b/e2e/fixtures/agent-tools-sandbox/agent/tools/dynamic-turn-replay.ts
@@ -0,0 +1,21 @@
+import { defineDynamic, defineTool } from "eve/tools";
+
+const TOKEN = "dynamic-turn-replay-ok-V6N";
+
+export default defineDynamic({
+ events: {
+ "turn.started": (_event, ctx) => {
+ const resolvedSessionId = ctx.session.id;
+ return {
+ dynamic_turn_replay_probe: defineTool({
+ description:
+ "Verify a turn-scoped dynamic callback after a durable workflow resumes. Only call for DYNAMIC-TURN-REPLAY-START.",
+ inputSchema: { type: "object" },
+ execute() {
+ return { resolvedSessionId, token: TOKEN };
+ },
+ }),
+ };
+ },
+ },
+});
diff --git a/e2e/fixtures/agent-tools-sandbox/evals/sandbox/redeploy.eval.ts b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/redeploy.eval.ts
index c75c2d93c..8367832d7 100644
--- a/e2e/fixtures/agent-tools-sandbox/evals/sandbox/redeploy.eval.ts
+++ b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/redeploy.eval.ts
@@ -39,6 +39,7 @@ const ALIAS_SETTLE_MATCHES = 5;
const FILE_PATH = "/workspace/redeploy-note.txt";
const FILE_TOKEN = "sandbox-redeploy-ok-K4W";
+const DYNAMIC_TURN_REPLAY_TOKEN = "dynamic-turn-replay-ok-V6N";
const INSTRUCTIONS_PATH = resolve("agent", "instructions.md");
const INSTRUCTIONS_MARKER = "redeploy-instructions-marker-T8B";
@@ -82,6 +83,18 @@ export default defineEval({
const originalInstructions = await readFile(INSTRUCTIONS_PATH, "utf8");
try {
+ // Resolve a turn-scoped dynamic tool, then park that same turn inside a
+ // workflow tool. Resuming after the deployment switch below must restore
+ // the dynamic callback in the new process before the model calls it.
+ const dynamicParked = await t.send("DYNAMIC-TURN-REPLAY-START");
+ const dynamicSession = dynamicParked.session;
+ dynamicSession.requireInputRequest({
+ display: "confirmation",
+ optionIds: ["approve", "cancel"],
+ toolName: "dynamic-turn-replay-gate",
+ });
+ dynamicParked.calledTool("dynamic-turn-replay-gate", { status: "pending", count: 1 });
+
// t0: write a marker file into this session's sandbox workspace.
const write = await t.send(
`Run the bash command \`printf %s ${FILE_TOKEN} > ${FILE_PATH}\`. ` +
@@ -99,6 +112,14 @@ export default defineEval({
await deployToAlias(t, alias, "instructions");
await waitForAliasToServe(t, INSTRUCTIONS_MARKER);
+ const dynamicResumed = await dynamicSession.respondAll("approve");
+ dynamicResumed.expectOk();
+ dynamicResumed.calledTool("dynamic_turn_replay_probe", {
+ count: 1,
+ output: new RegExp(DYNAMIC_TURN_REPLAY_TOKEN),
+ });
+ dynamicResumed.messageIncludes(DYNAMIC_TURN_REPLAY_TOKEN);
+
// t2: the same session reattaches to the same sandbox.
const persist = await session.send(
`Run the bash command \`cat ${FILE_PATH}\` and reply with the file contents verbatim.`,
diff --git a/packages/eve/src/context/dynamic-tool-lifecycle.test.ts b/packages/eve/src/context/dynamic-tool-lifecycle.test.ts
index d3c4560ba..4d78a0c85 100644
--- a/packages/eve/src/context/dynamic-tool-lifecycle.test.ts
+++ b/packages/eve/src/context/dynamic-tool-lifecycle.test.ts
@@ -759,6 +759,101 @@ describe("dispatchDynamicToolEvent", () => {
}
});
+ it("rebinds an authored turn resolver after a cold start", async () => {
+ const ctx = createCtx();
+ const captured = "turn-start-capture";
+ const handler = vi.fn(() => ({
+ tool: createReplayableTool("turn tool", () => ({ captured })),
+ }));
+ const resolver = createResolver("authored", ["turn.started"], handler);
+
+ await dispatchDynamicToolEvent({
+ ctx,
+ resolvers: [resolver],
+ messages: [],
+ event: makeEvent("turn.started"),
+ });
+ simulateColdStart(ctx);
+
+ await rebindMissingCompiledDynamicToolCallbacks({
+ ctx,
+ event: makeEvent("turn.started"),
+ messages: [],
+ resolvers: [resolver],
+ });
+
+ expect(handler).toHaveBeenCalledTimes(2);
+ expect(
+ lookupDurableDynamicCallback(
+ callbackOwner("tool", {
+ entryKey: "tool",
+ resolverSlug: "authored",
+ scope: "turn",
+ sessionId: ctx.require(SessionIdKey),
+ }),
+ "execute",
+ ),
+ ).toBeDefined();
+ const [tool] = buildDynamicTools(ctx);
+ await expect(tool!.execute!({}, executeOptions)).resolves.toEqual({ captured });
+ });
+
+ it("leaves an authored turn tool to fail when best-effort rebind cannot restore it", async () => {
+ const ctx = createCtx();
+ const handler = vi.fn(() => ({ tool: createReplayableTool() }));
+ const resolver = createResolver("authored", ["turn.started"], handler);
+
+ await dispatchDynamicToolEvent({
+ ctx,
+ resolvers: [resolver],
+ messages: [],
+ event: makeEvent("turn.started"),
+ });
+ simulateColdStart(ctx);
+ handler.mockReturnValue(null as never);
+
+ await expect(
+ rebindMissingCompiledDynamicToolCallbacks({
+ ctx,
+ event: makeEvent("turn.started"),
+ messages: [],
+ resolvers: [resolver],
+ }),
+ ).resolves.toBeUndefined();
+
+ const [tool] = buildDynamicTools(ctx);
+ await expect(tool!.execute!({}, executeOptions)).rejects.toThrow(
+ 'Dynamic tool "tool" cannot replay its execute callback',
+ );
+ });
+
+ it("requires an opted-in turn resolver to restore its callback", async () => {
+ const ctx = createCtx();
+ const handler = vi.fn(() => ({ tool: createReplayableTool() }));
+ const resolver = {
+ ...createResolver("framework", ["turn.started"], handler),
+ rebindMissingCallbacks: true,
+ };
+
+ await dispatchDynamicToolEvent({
+ ctx,
+ resolvers: [resolver],
+ messages: [],
+ event: makeEvent("turn.started"),
+ });
+ simulateColdStart(ctx);
+ handler.mockReturnValue(null as never);
+
+ await expect(
+ rebindMissingCompiledDynamicToolCallbacks({
+ ctx,
+ event: makeEvent("turn.started"),
+ messages: [],
+ resolvers: [resolver],
+ }),
+ ).rejects.toThrow("Dynamic tool callback rebind did not restore: tool");
+ });
+
it("replaces old step-function turn metadata without requiring cold-rebind opt-in", async () => {
let ctx = createCtx();
const serialized = serializeContext(ctx);
diff --git a/packages/eve/src/context/dynamic-tool-lifecycle.ts b/packages/eve/src/context/dynamic-tool-lifecycle.ts
index 7924a181e..8002b42c2 100644
--- a/packages/eve/src/context/dynamic-tool-lifecycle.ts
+++ b/packages/eve/src/context/dynamic-tool-lifecycle.ts
@@ -547,7 +547,7 @@ export async function refreshDynamicSessionToolsForRuntimeRevision(input: {
input.ctx.set(SessionDynamicToolRuntimeRevisionKey, input.runtimeRevision);
}
-/** Re-registers callbacks for compiled resolvers that explicitly support cold replay. */
+/** Re-registers missing callbacks while preserving the active turn's persisted tool set. */
export async function rebindMissingCompiledDynamicToolCallbacks(input: {
readonly ctx: AlsContext;
readonly event: UnstampedMessageStreamEvent;
@@ -566,16 +566,20 @@ export async function rebindMissingCompiledDynamicToolCallbacks(input: {
);
if (needsResolution.length === 0) return;
const resolverSlugs = new Set(needsResolution.map((entry) => entry.resolverSlug));
+ const matching = input.resolvers.filter((resolver) => resolverSlugs.has(resolver.slug));
+ // Framework resolvers opt into strict recovery because their persisted tool
+ // set represents locked runtime state. Authored resolvers rebind best-effort;
+ // a missing callback stays local to execution instead of blocking the turn.
+ const strictResolverSlugs = new Set(
+ matching
+ .filter((resolver) => resolver.rebindMissingCallbacks === true)
+ .map((resolver) => resolver.slug),
+ );
const oldResolverSlugs = new Set(
persisted
.filter((entry) => !isCurrentDynamicToolMetadata(entry))
.map((entry) => entry.resolverSlug),
);
- const matching = input.resolvers.filter(
- (resolver) =>
- resolverSlugs.has(resolver.slug) &&
- (oldResolverSlugs.has(resolver.slug) || resolver.rebindMissingCallbacks === true),
- );
if (matching.length === 0) {
input.ctx.set(TurnDynamicToolMetadataKey, toCurrentDynamicToolMetadataList(persisted));
return;
@@ -585,11 +589,14 @@ export async function rebindMissingCompiledDynamicToolCallbacks(input: {
input.ctx,
async () => await resolveToolsFromEvent(input.ctx, matching, input.event, input.messages),
);
+ // Keep the active turn's persisted tools and closures. Re-resolution only
+ // registers callback implementations; newly returned tools are not advertised.
const updated = toCurrentDynamicToolMetadataList(persisted, resolved.metadata);
input.ctx.set(TurnDynamicToolMetadataKey, updated);
const unresolved = updated.filter(
(entry) =>
+ (strictResolverSlugs.has(entry.resolverSlug) || oldResolverSlugs.has(entry.resolverSlug)) &&
needsResolution.some(
(candidate) =>
candidate.resolverSlug === entry.resolverSlug && candidate.name === entry.name,
diff --git a/packages/eve/src/execution/session/turn-step.test.ts b/packages/eve/src/execution/session/turn-step.test.ts
index 83178dcbd..e4950d515 100644
--- a/packages/eve/src/execution/session/turn-step.test.ts
+++ b/packages/eve/src/execution/session/turn-step.test.ts
@@ -25,6 +25,7 @@ import {
SessionDynamicToolRuntimeRevisionKey,
SessionIdKey,
SessionTraceSeedKey,
+ TurnDynamicToolMetadataKey,
TurnDeliveryIdsKey,
TurnTaskDeliveryKey,
HistoryStateKey,
@@ -46,6 +47,7 @@ import {
createMessageAppendedEvent,
createMessageCompletedEvent,
createResultCompletedEvent,
+ createTurnStartedEvent,
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
import { getCompiledRuntimeAgentBundle } from "#runtime/sessions/compiled-agent-cache.js";
@@ -1637,6 +1639,88 @@ describe("turnStep", () => {
}
});
+ it("does not rebind the previous turn before the next turn starts", async () => {
+ const execute = stampDurableDynamicCallback(async () => ({ ok: true }), {
+ callback: async () => ({ ok: true }),
+ closure: {},
+ });
+ const handler = vi.fn(() => ({
+ tool: defineTool({
+ description: "Turn tool",
+ inputSchema: { type: "object" },
+ execute,
+ }),
+ }));
+ const dynamicToolResolver = {
+ eventNames: ["turn.started"],
+ events: { "turn.started": handler },
+ logicalPath: "agent/tools/turn.ts",
+ slug: "turn",
+ sourceId: "test:turn",
+ sourceKind: "module",
+ } as never;
+ const compiledBundle = {
+ ...createStubBundle(),
+ resolvedAgent: { config: {}, dynamicToolResolvers: [dynamicToolResolver] },
+ } as never;
+ vi.mocked(getCompiledRuntimeAgentBundle).mockResolvedValue(compiledBundle);
+ const session = createStubSession({
+ state: {
+ "eve.harness.emission": {
+ sequence: 1,
+ sessionStarted: true,
+ stepIndex: 0,
+ turnId: "",
+ },
+ },
+ });
+ installSessionStoreMocks([session]);
+
+ const ctx = new ContextContainer();
+ ctx.set(AuthKey, null);
+ ctx.set(BundleKey, compiledBundle);
+ ctx.set(ChannelKey, threadContextAdapter);
+ ctx.set(ContinuationTokenKey, "http:thread-context");
+ ctx.set(SessionIdKey, "session-1");
+ ctx.set(SessionDynamicToolRuntimeRevisionKey, "deployment:dpl_current");
+ ctx.set(TurnDynamicToolMetadataKey, [
+ {
+ callbacks: { execute: { closure: {} } },
+ description: "Previous turn tool",
+ entryKey: "tool",
+ inputSchema: { type: "object" },
+ name: "tool",
+ resolverSlug: "turn",
+ },
+ ]);
+ vi.stubEnv("VERCEL_DEPLOYMENT_ID", "dpl_current");
+ vi.mocked(createExecutionNodeStep).mockImplementation((input) => {
+ return async (stepSession): Promise<StepResult> => {
+ await input.handleEvent?.(
+ createTurnStartedEvent({ sequence: 1, turnId: "turn_1" }),
+ stepSession.history,
+ );
+ return { next: { done: true, output: "ok" }, session: stepSession };
+ };
+ });
+
+ await turnStep({
+ input: { kind: "deliver", payloads: [{ message: "next turn" }] },
+ sessionWritable: createTestWritable(),
+ serializedContext: serializeContext(ctx),
+ sessionState: createStubSessionState({
+ emissionState: {
+ sequence: 1,
+ sessionStarted: true,
+ stepIndex: 0,
+ turnId: "",
+ },
+ }),
+ });
+
+ expect(handler).toHaveBeenCalledOnce();
+ });
+
it("prepares resumed-session history before dynamic runtime refresh", async () => {
const hidden = {
content: "HIDE_FROM_RUNTIME_REFRESH",
diff --git a/packages/eve/src/execution/session/turn-step.ts b/packages/eve/src/execution/session/turn-step.ts
index a4156839c..29033db1b 100644
--- a/packages/eve/src/execution/session/turn-step.ts
+++ b/packages/eve/src/execution/session/turn-step.ts
@@ -433,15 +433,17 @@ async function runSessionStep(input: TurnStepInput): Promise<DurableStepResult>
}),
),
]);
- await rebindMissingCompiledDynamicToolCallbacks({
- ctx,
- event: createTurnStartedEvent({
- sequence: initialEmissionState.sequence,
- turnId: activeTurnId(initialEmissionState),
- }),
- messages: history.initial.messages,
- resolvers: dynamicToolResolvers,
- });
+ if (!isHarnessBetweenTurns(initialSession)) {
+ await rebindMissingCompiledDynamicToolCallbacks({
+ ctx,
+ event: createTurnStartedEvent({
+ sequence: initialEmissionState.sequence,
+ turnId: activeTurnId(initialEmissionState),
+ }),
+ messages: history.initial.messages,
+ resolvers: dynamicToolResolvers,
+ });
+ }
}
} catch (error) {
await failChannelDeliveries(error);