Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(eve): withdraw a serve tool's pending questions when it replies (#3953)

Andrew Barbae167dd3ed355
diff --git a/.changeset/serve-reply-withdraws-questions.md b/.changeset/serve-reply-withdraws-questions.md
new file mode 100644
index 000000000..5409cdb33
--- /dev/null
+++ b/.changeset/serve-reply-withdraws-questions.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+`ctx.reply()` in a `serve` workflow tool now withdraws the `ctx.ask()` questions still pending for the calls it settles, so they resolve as `cancelled` and channels stop offering them. Before, such a question stayed pending after its call had a result: it kept steering from interrupting a model step and could take the person's next plain message as its answer.
diff --git a/docs/tools/tasks.md b/docs/tools/tasks.md
index 9387c0e1a..84c7db695 100644
--- a/docs/tools/tasks.md
+++ b/docs/tools/tasks.md
@@ -104,7 +104,8 @@ A workflow tool body receives its signals from its context or, in a `serve` body
every call in one stretch of work shares one signal, and the next stretch gets a new one. Pass it
to the steps and `ctx.agent` sends that should stop.
- **`ctx.ask(request)`** withdraws the question when the call's `abortSignal` aborts, and
- `ctx.ask(request, { signal })` also when `signal` does. The answer resolves as
+ `ctx.ask(request, { signal })` also when `signal` does. In a `serve` body, `ctx.reply()` also
+ withdraws the questions still pending for the calls it settles. The answer resolves as
`{ status: "cancelled" }`, and the stream reports `input.resolved` with `outcome: "cancelled"` so
channels stop offering the question. A person's answer that reached the session first still wins:
the ask resolves as `answered`, so branch on `status`, not on the signal.
diff --git a/docs/tools/workflows.mdx b/docs/tools/workflows.mdx
index 4d3c39838..d5ddae045 100644
--- a/docs/tools/workflows.mdx
+++ b/docs/tools/workflows.mdx
@@ -346,6 +346,8 @@ described in [Tasks](/docs/tools/tasks). What differs:
Each settled call reports its own `task.settled`, and the model receives the reply once. The task
then goes idle until its next call. A reply with
no call left to settle is dropped, and `ctx.ask` throws while no call is waiting for a result.
+ A reply also withdraws the questions still pending for the calls it settles, but doesn't abort
+ their `abortSignal`, so work the body continues after replying keeps running.
Returning settles the calls still waiting and ends the task; a throw fails them. `serve` is not
a generator, so it can't `yield` progress.
- **Working and idle:** a resumable task is working while one of its calls has no result, and idle
@@ -448,7 +450,8 @@ The answer's `status` tells the body what happened:
- `answered`: the person picked an option (`optionId`) or typed a reply (`text`).
- `cancelled`: the request was withdrawn before anyone answered it, because the `signal` passed to
- `ctx.ask` or the call's `abortSignal` aborted. The session emits `input.resolved` with
+ `ctx.ask` or the call's `abortSignal` aborted, or because a `serve` body replied to the call. The
+ session emits `input.resolved` with
`outcome: "cancelled"`, so clients stop offering the question.
- `unavailable`: the session cannot request input, such as a scheduled run. `ctx.ask` returns this
immediately instead of waiting for an answer no one can give.
diff --git a/packages/eve/src/execution/tools/workflow/ask.ts b/packages/eve/src/execution/tools/workflow/ask.ts
index a5e28687e..1d72e747b 100644
--- a/packages/eve/src/execution/tools/workflow/ask.ts
+++ b/packages/eve/src/execution/tools/workflow/ask.ts
@@ -78,6 +78,8 @@ const UNAVAILABLE: ToolInputResponse = { status: "unavailable" };
interface PendingAsk {
readonly resolve: (response: ToolInputResponse) => void;
readonly reject: (error: unknown) => void;
+ /** Sends the session the request to withdraw the question; unset once sent. */
+ withdraw: (() => Promise<void>) | undefined;
}
/**
@@ -88,24 +90,59 @@ interface PendingAsk {
export class WorkflowToolRunAsks {
private opened = 0;
private readonly pending = new Map<string, PendingAsk>();
+ /** Withdrawal requests on their way to the run's inbox. */
+ private readonly withdrawing = new Set<Promise<void>>();
private readonly runId: string;
constructor(runId: string) {
this.runId = runId;
}
- /** Opens a question under a request ID that replays deterministically with the run. */
- open(): { readonly answer: Promise<ToolInputResponse>; readonly requestId: string } {
+ /**
+ * Opens a question under a request ID that replays deterministically with
+ * the run. `send` sends the request and returns how to send its withdrawal.
+ */
+ open(send: (requestId: string) => () => Promise<void>): {
+ readonly answer: Promise<ToolInputResponse>;
+ readonly requestId: string;
+ } {
this.opened += 1;
const requestId = `${this.runId}-ask-${String(this.opened)}`;
+ let entry!: PendingAsk;
const answer = new Promise<ToolInputResponse>((resolve, reject) => {
- this.pending.set(requestId, { reject, resolve });
+ entry = { reject, resolve, withdraw: undefined };
});
+ this.pending.set(requestId, entry);
+ entry.withdraw = send(requestId);
return { answer, requestId };
}
- isPending(requestId: string): boolean {
- return this.pending.has(requestId);
+ /**
+ * Asks the session to withdraw a pending question, at most once. The
+ * question stays pending until the session decides it, so an answer the
+ * session accepted first still wins.
+ */
+ withdraw(requestId: string): void {
+ const entry = this.pending.get(requestId);
+ const withdraw = entry?.withdraw;
+ if (entry === undefined || withdraw === undefined) return;
+ entry.withdraw = undefined;
+ const sending = withdraw().catch((error: unknown) => this.fail(requestId, error));
+ this.withdrawing.add(sending);
+ void sending.then(() => this.withdrawing.delete(sending));
+ }
+
+ /** Asks the session to withdraw every pending question not already withdrawn. */
+ withdrawAll(): void {
+ for (const requestId of this.pending.keys()) this.withdraw(requestId);
+ }
+
+ /**
+ * Waits until every withdrawal asked for has reached the run's inbox, so the
+ * run's outcome, which drops its questions silently, cannot overtake one.
+ */
+ async flush(): Promise<void> {
+ while (this.withdrawing.size > 0) await Promise.all(this.withdrawing);
}
/** Applies the session's decision. The first one wins; a repeated one is dropped. */
@@ -153,25 +190,21 @@ export function ask(
if (signals.some((signal) => signal.aborted)) return Promise.resolve(CANCELLED);
const { asks, control, owner } = context;
- const { answer, requestId } = asks.open();
// A `serve` task's current call changes; the question stays the call's that asked it.
const from = context.from;
- const sent = owner.send({
- kind: "request",
- from,
- replyTo: requestId,
- request: { control, kind: "ask", request },
+ const { answer, requestId } = asks.open((replyTo) => {
+ const sent = owner.send({
+ kind: "request",
+ from,
+ replyTo,
+ request: { control, kind: "ask", request },
+ });
+ sent.catch((error: unknown) => asks.fail(replyTo, error));
+ // The withdrawal must not overtake the request it withdraws.
+ return () => sent.then(() => owner.send({ control, from, kind: "withdraw", replyTo }));
});
- sent.catch((error: unknown) => asks.fail(requestId, error));
- const requestWithdrawal = (): void => {
- if (!asks.isPending(requestId)) return;
- // The withdrawal must not overtake the request it withdraws.
- const withdrawal = sent.then(() =>
- owner.send({ control, from, kind: "withdraw", replyTo: requestId }),
- );
- withdrawal.catch((error: unknown) => asks.fail(requestId, error));
- };
+ const requestWithdrawal = (): void => asks.withdraw(requestId);
for (const signal of signals) {
signal.addEventListener("abort", requestWithdrawal, { once: true });
}
diff --git a/packages/eve/src/execution/tools/workflow/serve.test.ts b/packages/eve/src/execution/tools/workflow/serve.test.ts
index 60119f2dc..83117fcf7 100644
--- a/packages/eve/src/execution/tools/workflow/serve.test.ts
+++ b/packages/eve/src/execution/tools/workflow/serve.test.ts
@@ -13,6 +13,7 @@ import type {
import { startServeBody } from "#execution/tools/workflow/serve.js";
import { workflowToolRunWorkflow } from "#execution/tools/workflow/workflow.js";
import type { JsonValue } from "#shared/json.js";
+import type { ToolInputResponse } from "#tools/definition.js";
import type {
WorkflowServeCall,
WorkflowServeContext,
@@ -127,6 +128,13 @@ function aborted(signal: AbortSignal): Promise<void> {
return new Promise((resolve) => signal.addEventListener("abort", () => resolve()));
}
+/** The messages of one kind the run sent its inbox, oldest first. */
+function sent<K extends WorkflowToolRunMessage["kind"]>(kind: K) {
+ return mocks.deliver.mock.calls.flatMap(([, message]) =>
+ message.kind === kind ? [message as Extract<WorkflowToolRunMessage, { readonly kind: K }>] : [],
+ );
+}
+
it("serves every call to its task, one stretch of work at a time, until the session ends", async () => {
const received: WorkflowServeCall<JsonValue>[] = [];
const pendings: Promise<WorkflowServeCall<JsonValue>>[] = [];
@@ -248,6 +256,67 @@ it("serves every call to its task, one stretch of work at a time, until the sess
expect(fourth?.aborted).toBe(true);
});
+it("withdraws a question once its call has a reply, without aborting the stretch", async () => {
+ mocks.deliver.mockClear();
+ mocks.deliver.mockResolvedValue(undefined);
+ const replyNow = Promise.withResolvers<void>();
+ let stretch: AbortSignal | undefined;
+ let answer: PromiseLike<ToolInputResponse> | undefined;
+ mocks.serve.mockImplementation(
+ async (receive: WorkflowServeReceive<JsonValue>, ctx: WorkflowServeContext<JsonValue>) => {
+ stretch = (await receive()).abortSignal;
+ answer = ctx.ask({ prompt: "Should the plan include a rollback drill?" });
+ await replyNow.promise;
+ ctx.reply("plan without a rollback drill");
+ await answer;
+ return await receive();
+ },
+ );
+ const started = startServeBody(input);
+
+ await vi.waitFor(() => expect(sent("request")).toHaveLength(1));
+ const requestId = sent("request")[0]?.replyTo ?? "";
+ // Alice's second call arrives before the reply, so it joins the stretch and the reply leaves it working.
+ started.control.apply(call("call-2", "Add a rollback step."));
+ replyNow.resolve();
+ await vi.waitFor(() => expect(sent("withdraw")).toHaveLength(1));
+
+ expect(sent("withdraw")).toMatchObject([{ from: { callId: "call-1" }, replyTo: requestId }]);
+ expect(stretch?.aborted).toBe(false);
+ // Cancelling the second call aborts the stretch, and the question's withdrawal was already asked for.
+ started.control.apply({ kind: "cancel", reason: "The task was cancelled." });
+ expect(stretch?.aborted).toBe(true);
+ started.asks.settle({ kind: "withdrawn", requestId });
+ await expect(answer).resolves.toEqual({ status: "cancelled" });
+ expect(sent("withdraw")).toHaveLength(1);
+ started.control.apply({ kind: "end", reason: "The session ended." });
+ await started.outcome;
+});
+
+it("holds the outcome of a body that returns right after replying until its withdrawals reach the run", async () => {
+ mocks.deliver.mockClear();
+ const withdrawalLands = Promise.withResolvers<void>();
+ mocks.deliver.mockImplementation(async (_inbox, message) => {
+ if (message.kind === "withdraw") await withdrawalLands.promise;
+ });
+ mocks.serve.mockImplementation(
+ async (receive: WorkflowServeReceive<JsonValue>, ctx: WorkflowServeContext<JsonValue>) => {
+ await receive();
+ void ctx.ask({ prompt: "Should the plan include a rollback drill?" });
+ ctx.reply("plan without a rollback drill");
+ return null;
+ },
+ );
+ const started = startServeBody(input);
+ let finished = false;
+ void started.outcome.then(() => (finished = true));
+
+ await vi.waitFor(() => expect(sent("withdraw")).toHaveLength(1));
+ expect(finished).toBe(false);
+ withdrawalLands.resolve();
+ await expect(started.outcome).resolves.toEqual({ output: null, status: "completed" });
+});
+
it.each([
{
body: "ignores the cancel",
diff --git a/packages/eve/src/execution/tools/workflow/serve.ts b/packages/eve/src/execution/tools/workflow/serve.ts
index 406d04032..ad465b2fa 100644
--- a/packages/eve/src/execution/tools/workflow/serve.ts
+++ b/packages/eve/src/execution/tools/workflow/serve.ts
@@ -78,6 +78,8 @@ class WorkflowServeCalls implements WorkflowBodyControl {
/**
* The body's questions. A cancel keeps the run, so it withdraws them
* through the stretch's `abortSignal`, and the session decides each one.
+ * A reply withdraws them too, but leaves the signal alone: it means
+ * cancelled, and the body may keep working after it replies.
*/
readonly asks: WorkflowToolRunAsks;
private readonly run = new AbortController();
@@ -188,6 +190,10 @@ class WorkflowServeCalls implements WorkflowBodyControl {
if (settled.length === 0) return;
this.waiting = [];
if (this.arrived.length === 0) this.stretch = undefined;
+ // `ctx.ask()` needs a waiting call, and a cancel already withdrew the
+ // questions of the calls it settled, so every question not yet withdrawn
+ // was asked for a call this settles.
+ this.asks.withdrawAll();
const previous = this.replies;
this.replies = previous.then(() => this.deliverReply(settled, output));
}
@@ -199,8 +205,9 @@ class WorkflowServeCalls implements WorkflowBodyControl {
);
}
- /** Waits for in-flight replies, so the outcome can never overtake them. */
+ /** Waits for in-flight replies and withdrawals, so the outcome can never overtake them. */
async flush(): Promise<void> {
+ await this.asks.flush();
await this.replies;
}