Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): report session usage on session events and in eval derived facts (#4137)

Philipp Comansff12e851fd26
diff --git a/.changeset/session-usage-on-waiting.md b/.changeset/session-usage-on-waiting.md
new file mode 100644
index 000000000..492f86e05
--- /dev/null
+++ b/.changeset/session-usage-on-waiting.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+`session.waiting`, `turn.waiting`, `session.failed`, and `session.completed` now carry `usage`, the session's running token usage and cost including what the agents it delegated to spent. Eval results expose it as `derived.usage`, per session and summed for the eval, so a delegating eval no longer reports only its own model calls.
diff --git a/docs/concepts/sessions-runs-and-streaming.md b/docs/concepts/sessions-runs-and-streaming.md
index 239b4317c..4ac6892f9 100644
--- a/docs/concepts/sessions-runs-and-streaming.md
+++ b/docs/concepts/sessions-runs-and-streaming.md
@@ -119,7 +119,7 @@ A nested action is a call a tool makes on the model's behalf, such as each conne
Note: consider the privacy, confidentiality, and user-experience implications for displaying, storing, or transmitting reasoning events in your application.
-`message.completed` can fire more than once in a turn: the agent often emits interim assistant text before a tool call. To tell tool-call narration from a terminal reply, check `message.completed.data.finishReason`. `step.completed.data.finishReason` mirrors the step outcome, and usage lives on `step.completed`.
+`message.completed` can fire more than once in a turn: the agent often emits interim assistant text before a tool call. To tell tool-call narration from a terminal reply, check `message.completed.data.finishReason`. `step.completed.data.finishReason` mirrors the step outcome, and each step's usage lives on `step.completed`. `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` carry `data.usage`: the session's running total of its own model calls plus what the agents it delegated to spent. `costUsd` is absent when no model call reported a cost, and compaction calls are not counted.
A delegated subagent publishes progress on its own child-session stream. When a subagent tool call or a workflow tool's `ctx.agent` opens a session, the parent emits `agent.started` with the call's `callId` and `turnId`, its `taskId` when the call runs as a task, the child's `sessionId`, and a `streamPath` that a client follows with `session.agent(started).stream()`. A subagent tool call is a [task](#task-events) whose `task.started` has `kind: "agent"`: its `task.started` and `task.settled` carry the call, and `task.settled.data.output` is the child's reply. A call that fails before the child session opens has no `agent.started`. A settled call does not end the child session; later calls with the same `taskId` reach it.
diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdx
index 66c20b26c..c7b4f7025 100644
--- a/docs/evals/reporters.mdx
+++ b/docs/evals/reporters.mdx
@@ -99,6 +99,8 @@ An eval can create several sessions, and a long session can produce several trac
`result.result.derived` holds the facts the runner derived from the eval's sessions, such as tool calls and `models`: the distinct ids of the models the steps started with, in session order. Each entry in `result.result.sessions` holds the same facts for one session, in first-use order, so a reporter can tell the primary session's models from those of the other sessions the eval created or attached. A delegated subagent runs in its own session, so its models appear only when the eval attaches that session with `t.target.attachSession`. Compaction and `auto` routing calls are not included.
+`derived.usage` is the token usage and cost from each session's latest `session.waiting`, `turn.waiting`, `session.failed`, or `session.completed`, including what its delegated agents spent; on a turn, it is the session's total so far. The eval's `usage` counts each captured session once, by its latest usage, and leaves out sessions another captured session opened, because that session already counts their spend. It is absent when a counted session reported no usage. It leaves out compaction calls, the turns an eval sends directly to a child session it attached, and delegated spend counted after a session's last such event.
+
Reporter callbacks for an eval stay ordered—eval start, traced session starts, then completion—even when several evals run concurrently. Reach for a custom reporter only when a destination isn't covered. The per-run artifacts under `.eve/evals/` retain the trace contexts alongside the result for ad-hoc inspection.
## What to read next
diff --git a/e2e/fixtures/agent-subagents/evals/delegated-usage-limit.shared.ts b/e2e/fixtures/agent-subagents/evals/delegated-usage-limit.shared.ts
index 8c2a3f9bf..f7d1814be 100644
--- a/e2e/fixtures/agent-subagents/evals/delegated-usage-limit.shared.ts
+++ b/e2e/fixtures/agent-subagents/evals/delegated-usage-limit.shared.ts
@@ -9,7 +9,8 @@ import { SURVEY_WORKER_INPUT_TOKENS } from "../constants";
* worker's one model call reports more input tokens than the parent's default
* session budget, so the parent's next model call stops at its own
* session-limit prompt, whose used figure includes the worker's tokens.
- * Approving the prompt lets the parent report the survey. Returns the session
+ * Approving the prompt lets the parent report the survey, and the parent's
+ * `session.waiting` usage then includes the worker's tokens. Returns the session
* for checks on how the parent delegated.
*/
export async function expectSurveyCountedAgainstParent(t: EveEvalContext, message: string) {
@@ -33,5 +34,12 @@ export async function expectSurveyCountedAgainstParent(t: EveEvalContext, messag
const resumed = await session.respond([{ optionId: "continue", requestId: request.requestId }]);
resumed.expectOk();
resumed.messageIncludes("SURVEY-REPLY Alice's tide survey has 12 stations.");
+ t.eventsSatisfy("the parent's session.waiting usage counts the worker's tokens", (events) => {
+ const waiting = events.filter((event) => event.type === "session.waiting").at(-1);
+ return (
+ waiting?.type === "session.waiting" &&
+ (waiting.data.usage?.inputTokens ?? 0) >= SURVEY_WORKER_INPUT_TOKENS
+ );
+ });
return session;
}
diff --git a/packages/eve/extension-contracts/compatibility/channel/v43.ts b/packages/eve/extension-contracts/compatibility/channel/v43.ts
new file mode 100644
index 000000000..9a260ece2
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/channel/v43.ts
@@ -0,0 +1,12 @@
+import { defineChannel } from "#public/channels/index.js";
+
+// Epoch 43 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 44 adds it as optional.
+// Channels that read only the continuation token see no change.
+export default defineChannel({
+ routes: [],
+ events: {
+ "session.waiting"(event) {
+ console.info("session waiting", { continuationToken: event.continuationToken });
+ },
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/connection/v36.ts b/packages/eve/extension-contracts/compatibility/connection/v36.ts
new file mode 100644
index 000000000..42312a47c
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/connection/v36.ts
@@ -0,0 +1,10 @@
+import { defineMcpClientConnection } from "#public/connections/index.js";
+
+// Epoch 36 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 37 adds it as optional.
+export default defineMcpClientConnection({
+ description: "Search the incident tracker.",
+ url: "https://incidents.example.com/mcp",
+ approval({ toolName }) {
+ return toolName.endsWith("__search") ? "approved" : "user-approval";
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/dynamicInstructions/v30.ts b/packages/eve/extension-contracts/compatibility/dynamicInstructions/v30.ts
new file mode 100644
index 000000000..1423d1022
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/dynamicInstructions/v30.ts
@@ -0,0 +1,11 @@
+import { defineDynamic, defineInstructions } from "#public/instructions/index.js";
+
+// Epoch 30 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 31 adds it as optional.
+export default defineDynamic({
+ events: {
+ "turn.started": (_event, ctx) =>
+ defineInstructions({
+ markdown: `Triage the incident reported in session ${ctx.session.id}.`,
+ }),
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/dynamicSkill/v29.ts b/packages/eve/extension-contracts/compatibility/dynamicSkill/v29.ts
new file mode 100644
index 000000000..6fd3757ad
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/dynamicSkill/v29.ts
@@ -0,0 +1,12 @@
+import { defineDynamic, defineSkill } from "#public/skills/index.js";
+
+// Epoch 29 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 30 adds it as optional.
+export default defineDynamic({
+ events: {
+ "turn.started": (_event, ctx) =>
+ defineSkill({
+ description: "Triage the active incident.",
+ markdown: `Triage the incident reported in session ${ctx.session.id}.`,
+ }),
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/dynamicTool/v68.ts b/packages/eve/extension-contracts/compatibility/dynamicTool/v68.ts
new file mode 100644
index 000000000..bf345f288
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/dynamicTool/v68.ts
@@ -0,0 +1,14 @@
+import { defineDynamic, defineTool } from "#public/tools/index.js";
+
+// Epoch 68 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 69 adds it as optional.
+export default defineDynamic({
+ events: {
+ "turn.started": (_event, ctx) => ({
+ session: defineTool({
+ description: "Return the active session identifier.",
+ inputSchema: { type: "object", properties: {} },
+ execute: () => ({ sessionId: ctx.session.id }),
+ }),
+ }),
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/hook/v35.ts b/packages/eve/extension-contracts/compatibility/hook/v35.ts
new file mode 100644
index 000000000..c834812ac
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/hook/v35.ts
@@ -0,0 +1,14 @@
+import { defineHook } from "#public/hooks/index.js";
+
+// Epoch 35 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 36 adds it as optional.
+// Hooks that read only the continuation token see no change.
+export default defineHook({
+ events: {
+ "session.waiting"(event, ctx) {
+ console.info("session waiting", {
+ continuationToken: event.data.continuationToken,
+ sessionId: ctx.session.id,
+ });
+ },
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/schedule/v24.ts b/packages/eve/extension-contracts/compatibility/schedule/v24.ts
new file mode 100644
index 000000000..863ae5b4c
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/schedule/v24.ts
@@ -0,0 +1,22 @@
+import { defineChannel } from "#public/channels/index.js";
+import { defineSchedule } from "#public/schedules/index.js";
+
+// Epoch 24 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 25 adds it as optional.
+const reports = defineChannel({
+ routes: [],
+ events: {
+ "session.waiting"(event) {
+ console.info("report waiting", { continuationToken: event.continuationToken });
+ },
+ },
+ receive(input, { from }) {
+ return from("nightly-report").send(input.message, { auth: input.auth });
+ },
+});
+
+export default defineSchedule({
+ cron: "0 2 * * *",
+ async run({ to, appAuth, waitUntil }) {
+ waitUntil(to(reports, {}).send("Prepare the nightly report.", { auth: appAuth }));
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/subagent/v27.ts b/packages/eve/extension-contracts/compatibility/subagent/v27.ts
new file mode 100644
index 000000000..40252659c
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/subagent/v27.ts
@@ -0,0 +1,7 @@
+import { defineAgent } from "#public/index.js";
+
+// Epoch 27 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 28 adds it as optional.
+export default defineAgent({
+ description: "Research the incidents behind a metric change.",
+ model: "openai/gpt-5.6-sol",
+});
diff --git a/packages/eve/extension-contracts/compatibility/tool/v71.ts b/packages/eve/extension-contracts/compatibility/tool/v71.ts
new file mode 100644
index 000000000..c301b213c
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/tool/v71.ts
@@ -0,0 +1,10 @@
+import { z } from "zod";
+import { defineTool } from "#public/tools/index.js";
+
+// Epoch 71 `session.waiting`, `session.failed`, `session.completed`, and `turn.waiting` events had no `usage`; epoch 72 adds it as optional.
+export default defineTool({
+ description: "Summarize the open incidents for a service.",
+ inputSchema: z.object({ service: z.string() }),
+ outputSchema: z.object({ service: z.string(), summary: z.string() }),
+ execute: ({ service }) => ({ service, summary: `No open incidents for ${service}.` }),
+});
diff --git a/packages/eve/extension-contracts/reports/channel/v44.json b/packages/eve/extension-contracts/reports/channel/v44.json
new file mode 100644
index 000000000..af730e2ea
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/channel/v44.json
@@ -0,0 +1,21 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "channel",
+ "epoch": 44,
+ "sha256": "8ba8b7ebd8b86414be4aa8504210b4fb82090acbf0d5abe8f79903c7bd6085da",
+ "exports": [
+ "DELETE",
+ "GET",
+ "HEAD",
+ "OPTIONS",
+ "PATCH",
+ "POST",
+ "PUT",
+ "WS",
+ "createWebSocketUpgradeServer",
+ "defineChannel",
+ "disableRoute",
+ "isChannel",
+ "isDisabledRouteSentinel"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/connection/v37.json b/packages/eve/extension-contracts/reports/connection/v37.json
new file mode 100644
index 000000000..dc86f6cab
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/connection/v37.json
@@ -0,0 +1,16 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "connection",
+ "epoch": 37,
+ "sha256": "aaf23a1863d0322eb8f9da99da90573147c9b94da0ab8f8d13fda49d59bf935e",
+ "exports": [
+ "ConnectionAuthorizationFailedError",
+ "ConnectionAuthorizationRequiredError",
+ "defineDynamic",
+ "defineInteractiveAuthorization",
+ "defineMcpClientConnection",
+ "defineOpenAPIConnection",
+ "isConnectionAuthorizationFailedError",
+ "isConnectionAuthorizationRequiredError"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/dynamicInstructions/v31.json b/packages/eve/extension-contracts/reports/dynamicInstructions/v31.json
new file mode 100644
index 000000000..a487a6ced
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/dynamicInstructions/v31.json
@@ -0,0 +1,7 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "dynamicInstructions",
+ "epoch": 31,
+ "sha256": "08058f0b64f4b76a1adcb69c09377ba21da64340fdc24733e4c945fffde7caec",
+ "exports": ["defineDynamic"]
+}
diff --git a/packages/eve/extension-contracts/reports/dynamicSkill/v30.json b/packages/eve/extension-contracts/reports/dynamicSkill/v30.json
new file mode 100644
index 000000000..036cdf304
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/dynamicSkill/v30.json
@@ -0,0 +1,7 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "dynamicSkill",
+ "epoch": 30,
+ "sha256": "3e1844f05fd14258de1479ed8bd4dc03043c0e5b87d435a6d720185f3396054b",
+ "exports": ["defineDynamic"]
+}
diff --git a/packages/eve/extension-contracts/reports/dynamicTool/v69.json b/packages/eve/extension-contracts/reports/dynamicTool/v69.json
new file mode 100644
index 000000000..1c63f1c86
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/dynamicTool/v69.json
@@ -0,0 +1,16 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "dynamicTool",
+ "epoch": 69,
+ "sha256": "ea9fbe35180bfd756fa4e7285618639cf5bf216b7e9fa488f46f1ca805cac499",
+ "exports": [
+ "DynamicToolEntry",
+ "DynamicToolEvents",
+ "DynamicToolResult",
+ "DynamicToolSet",
+ "auto",
+ "defineDurableCallback",
+ "defineDurableSchema",
+ "defineDynamic"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/hook/v36.json b/packages/eve/extension-contracts/reports/hook/v36.json
new file mode 100644
index 000000000..9b3985202
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/hook/v36.json
@@ -0,0 +1,7 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "hook",
+ "epoch": 36,
+ "sha256": "c0ff9bd9f4b2ccdd462f9fd274a3871ddf43c124953e7ba698558cef053b57e2",
+ "exports": ["defineHook"]
+}
diff --git a/packages/eve/extension-contracts/reports/schedule/v25.json b/packages/eve/extension-contracts/reports/schedule/v25.json
new file mode 100644
index 000000000..9bfaf2dd3
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/schedule/v25.json
@@ -0,0 +1,15 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "schedule",
+ "epoch": 25,
+ "sha256": "697ab0eefacea4e72d1e837aac5e746d1c7992936a1ac7885800cae49ddbac78",
+ "exports": [
+ "ScheduleDefinition",
+ "ScheduleHandlerArgs",
+ "ScheduleRunHandler",
+ "ScheduleToFn",
+ "TypedReceiveTarget",
+ "defineSchedule",
+ "isScheduleAuth"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/subagent/v28.json b/packages/eve/extension-contracts/reports/subagent/v28.json
new file mode 100644
index 000000000..52bdfdf74
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/subagent/v28.json
@@ -0,0 +1,23 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "subagent",
+ "epoch": 28,
+ "sha256": "f9c443883beceedc286257a76db8cbeaeeed3563ce13873a9c91482404bbe2ab",
+ "exports": [
+ "AgentCompactionDefinition",
+ "AgentDefinition",
+ "AgentModelDefinition",
+ "AgentStaticModelDefinition",
+ "DefinedAgent",
+ "DynamicLocalSubagentDefinition",
+ "DynamicSubagentDefinition",
+ "RemoteAgentDefinition",
+ "RemoteAgentDefinitionInput",
+ "WorkspaceAgentDefinition",
+ "WorkspaceAgentTransport",
+ "defineAgent",
+ "defineDynamic",
+ "defineRemoteAgent",
+ "defineWorkspaceAgent"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/tool/v72.json b/packages/eve/extension-contracts/reports/tool/v72.json
new file mode 100644
index 000000000..451f8071e
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/tool/v72.json
@@ -0,0 +1,29 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "tool",
+ "epoch": 72,
+ "sha256": "b25340648af6b8a5671bb6b0d41d36320ab6fbc3f69aa7fb62f434d871ea127a",
+ "exports": [
+ "AgentRouterInput",
+ "AgentRouterTool",
+ "WorkflowStepToolContext",
+ "WorkflowTool",
+ "WorkflowToolDefinition",
+ "WorkflowToolInput",
+ "WorkflowToolOptions",
+ "agentRouter",
+ "defaultWebSearch",
+ "defineTool",
+ "defineWorkflowTool",
+ "disableTool",
+ "evaluate",
+ "isDisabledToolSentinel",
+ "isWebSearchToolDefinition",
+ "serializeModelInputSchema",
+ "toolOutput",
+ "toolOutputPart",
+ "toolResultFrom",
+ "webSearch",
+ "workflow"
+ ]
+}
diff --git a/packages/eve/src/acp/adapter.ts b/packages/eve/src/acp/adapter.ts
index 2813560f4..17d763aaa 100644
--- a/packages/eve/src/acp/adapter.ts
+++ b/packages/eve/src/acp/adapter.ts
@@ -211,7 +211,8 @@ export class EveAcpAdapter {
let unsupportedEvent: RequestError | undefined;
for await (const event of response) {
- const turnId = "data" in event && "turnId" in event.data ? event.data.turnId : undefined;
+ const data = "data" in event ? event.data : undefined;
+ const turnId = data !== undefined && "turnId" in data ? data.turnId : undefined;
if (typeof turnId === "string") active.turnId = turnId;
if (event.type === "input.requested") inputRequests.push(...event.data.requests);
diff --git a/packages/eve/src/channel/adapter.test.ts b/packages/eve/src/channel/adapter.test.ts
index 9189251c2..612f248ef 100644
--- a/packages/eve/src/channel/adapter.test.ts
+++ b/packages/eve/src/channel/adapter.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { describe, expect, it } from "vitest";
import type { ChannelAdapter, ChannelAdapterContext, FetchFileResult } from "#channel/adapter.js";
@@ -141,10 +142,14 @@ describe("ChannelAdapter helpers", () => {
},
};
- const event = await callAdapterEventHandler(adapter, createSessionWaitingEvent(), context);
+ const event = await callAdapterEventHandler(
+ adapter,
+ createSessionWaitingEvent(TEST_USAGE),
+ context,
+ );
expect(event).toEqual({
- data: { continuationToken: "C1:T1", wait: "next-user-message" },
+ data: { continuationToken: "C1:T1", usage: TEST_USAGE, wait: "next-user-message" },
type: "session.waiting",
});
expect(observedToken).toBe("temporary");
@@ -163,12 +168,12 @@ describe("ChannelAdapter helpers", () => {
const event = await callAdapterEventHandler(
{ kind: "http" },
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
context,
);
expect(event).toEqual({
- data: { continuationToken: "session-1", wait: "next-user-message" },
+ data: { continuationToken: "session-1", usage: TEST_USAGE, wait: "next-user-message" },
type: "session.waiting",
});
});
diff --git a/packages/eve/src/cli/dev/tui/runner.integration.test.ts b/packages/eve/src/cli/dev/tui/runner.integration.test.ts
index 9e9888920..e15a11b3b 100644
--- a/packages/eve/src/cli/dev/tui/runner.integration.test.ts
+++ b/packages/eve/src/cli/dev/tui/runner.integration.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { afterEach, describe, expect, it, vi } from "vitest";
import { Client } from "#client/client.js";
import {
@@ -113,7 +114,7 @@ describe("eve dev conversation", () => {
stepIndex: 0,
turnId: "wake",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
await vi.waitFor(() =>
expect(assistantText(tui.latest())).toEqual([
@@ -195,7 +196,7 @@ describe("eve dev conversation", () => {
stepIndex: 0,
turnId: "wake",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
await vi.waitFor(() => expect(readToolApproval).toHaveBeenCalledOnce());
@@ -246,7 +247,7 @@ describe("eve dev conversation", () => {
taskId: "task_1",
turnId: "turn_1",
}),
- createTurnWaitingEvent({ sequence: 1, turnId: "turn_1" }),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 1, turnId: "turn_1" }),
],
"delivery_1",
);
@@ -286,7 +287,7 @@ describe("eve dev conversation", () => {
status: "completed",
}),
createTurnCompletedEvent({ sequence: 2, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
await vi.waitFor(() => expect(tui.latest().working).toBe(false));
await vi.waitFor(() =>
@@ -359,14 +360,17 @@ describe("eve dev conversation", () => {
}),
]),
approval("task_1"),
- createTurnWaitingEvent({ sequence: 2, turnId: "turn_1" }),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 2, turnId: "turn_1" }),
],
"delivery_1",
);
await vi.waitFor(() => expect(readToolApproval).toHaveBeenCalledOnce());
// Bob's approval arrives while Alice is still deciding on the first one.
server.emit(
- [approval("task_2"), createTurnWaitingEvent({ sequence: 2, turnId: "turn_1" })],
+ [
+ approval("task_2"),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 2, turnId: "turn_1" }),
+ ],
"delivery_1",
);
await vi.waitFor(() => expect(tui.latest().conversation.inputs.approval_task_2).toBeDefined());
@@ -415,7 +419,10 @@ describe("eve dev conversation", () => {
await vi.waitFor(() => expect(server.requestsTo("POST", "/cancel")).toHaveLength(1));
expect(server.requestsTo("POST", "/cancel")[0]?.body).toMatchObject({ turnId: "turn_1" });
server.emit(
- [createTurnCancelledEvent({ sequence: 1, turnId: "turn_1" }), createSessionWaitingEvent()],
+ [
+ createTurnCancelledEvent({ sequence: 1, turnId: "turn_1" }),
+ createSessionWaitingEvent(TEST_USAGE),
+ ],
"delivery_1",
);
await vi.waitFor(() => expect(tui.latest().working).toBe(false));
@@ -438,7 +445,10 @@ describe("eve dev conversation", () => {
);
return () =>
server.emit(
- [createTurnCancelledEvent({ sequence: 2, turnId }), createSessionWaitingEvent()],
+ [
+ createTurnCancelledEvent({ sequence: 2, turnId }),
+ createSessionWaitingEvent(TEST_USAGE),
+ ],
delivery,
);
};
@@ -476,6 +486,7 @@ describe("eve dev conversation", () => {
[
createTurnStartedEvent({ sequence: 0, turnId: "turn_1" }),
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "SANDBOX_LOST",
message: "The sandbox stopped.",
sessionId: "session_1",
diff --git a/packages/eve/src/cli/dev/tui/runner.test.ts b/packages/eve/src/cli/dev/tui/runner.test.ts
index bdb05b5f6..e39632503 100644
--- a/packages/eve/src/cli/dev/tui/runner.test.ts
+++ b/packages/eve/src/cli/dev/tui/runner.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { Client, ClientError, type AgentInfoResult } from "#client/index.js";
@@ -832,7 +833,7 @@ describe("EveTUIRunner gateway-auth failure rendering", () => {
async function failuresFor(appRoot?: string): Promise<readonly AgentTUIFailure[]> {
const server = new FakeEveServer(({ turnId }) => [
{ type: "step.failed", data: { ...gatewayFailure, sequence: 0, stepIndex: 0, turnId } },
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
vi.stubGlobal("fetch", server.fetch);
const views: AgentTUIConversationView[] = [];
@@ -1280,7 +1281,7 @@ describe("EveTUIRunner registry handoffs", () => {
createTurnStartedEvent({ sequence: 0, turnId }),
...registryResult("slack-add", "channel/slack"),
...registryResult("linear-add", "connection/linear"),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
const handle = vi.fn(async (_command: { name: string; argument: string }) => ({
message: "done",
@@ -1302,7 +1303,7 @@ describe("EveTUIRunner registry handoffs", () => {
const order: string[] = [];
const server = serve(({ body, turnId }) =>
body?.inputResponses !== undefined
- ? (order.push("respond"), [createSessionWaitingEvent()])
+ ? (order.push("respond"), [createSessionWaitingEvent(TEST_USAGE)])
: [
createTurnStartedEvent({ sequence: 0, turnId }),
...registryResult("registry-add", "channel/slack"),
@@ -1324,7 +1325,7 @@ describe("EveTUIRunner registry handoffs", () => {
stepIndex: 0,
turnId,
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
],
);
await new EveTUIRunner({
@@ -1378,7 +1379,7 @@ describe("EveTUIRunner registry handoffs", () => {
[
createTurnStartedEvent({ sequence: 0, turnId: "child_turn" }),
...registryResult("registry-add", "channel/slack", "registry_add"),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
],
undefined,
"child_1",
@@ -1392,7 +1393,7 @@ describe("EveTUIRunner registry handoffs", () => {
taskId: "task_1",
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
],
"delivery_1",
"session_1",
@@ -1526,7 +1527,7 @@ describe("EveTUIRunner requests", () => {
it("submits the chosen session-limit continuation", async () => {
const server = serve(({ body, turnId }) =>
body?.inputResponses !== undefined
- ? [createSessionWaitingEvent()]
+ ? [createSessionWaitingEvent(TEST_USAGE)]
: [
createTurnStartedEvent({ sequence: 0, turnId }),
createInputRequestedEvent({
@@ -1551,7 +1552,7 @@ describe("EveTUIRunner requests", () => {
stepIndex: 0,
turnId,
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
],
);
const readInputQuestion = vi.fn(async () => ({ optionId: "continue" }));
@@ -1585,7 +1586,7 @@ describe("EveTUIRunner requests", () => {
stepIndex: 0,
turnId,
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
const readInputQuestion = vi.fn(async () => undefined);
const readInput = vi.fn().mockResolvedValueOnce({ type: "submit", text: "Summarize." });
@@ -1718,6 +1719,7 @@ describe("EveTUIRunner remote session failures", () => {
serve(({ turnId }) => [
createTurnStartedEvent({ sequence: 0, turnId }),
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "HookConflictError",
message: "HookConflictError: token in use",
sessionId: "session_1",
diff --git a/packages/eve/src/cli/dev/tui/terminal-renderer.test.ts b/packages/eve/src/cli/dev/tui/terminal-renderer.test.ts
index 86839b70f..110899759 100644
--- a/packages/eve/src/cli/dev/tui/terminal-renderer.test.ts
+++ b/packages/eve/src/cli/dev/tui/terminal-renderer.test.ts
@@ -13,7 +13,7 @@ import {
import { initialConversationState, reduceConversation } from "#client/conversation-reducer.js";
import type { EveAgentReducerEvent } from "#client/reducer.js";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import {
createActionsRequestedEvent,
createMessageAppendedEvent,
@@ -3836,7 +3836,7 @@ describe("TerminalRenderer conversation", () => {
turnId: "turn_1",
}),
),
- stamped(createTurnWaitingEvent({ sequence: 3, turnId: "turn_1" })),
+ stamped(createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 3, turnId: "turn_1" })),
];
renderer.renderConversation(conversationOf(working, { working: true }));
const during = screen.snapshot();
diff --git a/packages/eve/src/cli/dev/tui/test/fake-eve-server.ts b/packages/eve/src/cli/dev/tui/test/fake-eve-server.ts
index 3edc9e273..40ddcd0c6 100644
--- a/packages/eve/src/cli/dev/tui/test/fake-eve-server.ts
+++ b/packages/eve/src/cli/dev/tui/test/fake-eve-server.ts
@@ -10,6 +10,14 @@ import {
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
+/** What the fake server reports a session spent, like a real one does on `session.waiting`. */
+const FAKE_USAGE = {
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ inputTokens: 1200,
+ outputTokens: 150,
+};
+
export interface FakeEveRequest {
readonly method: string;
readonly path: string;
@@ -186,7 +194,7 @@ export function reply(text: string): FakeEveTurn {
turnId,
}),
createTurnCompletedEvent({ sequence: 3, turnId }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(FAKE_USAGE),
];
}
diff --git a/packages/eve/src/cli/dev/tui/transcript.test.ts b/packages/eve/src/cli/dev/tui/transcript.test.ts
index c5f6b355f..8a2501deb 100644
--- a/packages/eve/src/cli/dev/tui/transcript.test.ts
+++ b/packages/eve/src/cli/dev/tui/transcript.test.ts
@@ -5,7 +5,7 @@ import {
type ConversationEvent,
} from "#client/conversation-reducer.js";
import type { ConversationState } from "#client/conversation-state.js";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import {
createActionInputAppendedEvent,
createActionResultEvent,
@@ -305,7 +305,7 @@ describe("ConversationTranscript", () => {
taskStarted("call_1", "summarize"),
toolResult("call_1", "summarize"),
toolCall("wait_1", "task_wait"),
- event(createTurnWaitingEvent({ sequence: 3, turnId: "turn_1" })),
+ event(createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 3, turnId: "turn_1" })),
]);
const transcript = new ConversationTranscript();
const working = view(state, true);
diff --git a/packages/eve/src/client/conversation-reducer.test.ts b/packages/eve/src/client/conversation-reducer.test.ts
index 8208f0177..626232050 100644
--- a/packages/eve/src/client/conversation-reducer.test.ts
+++ b/packages/eve/src/client/conversation-reducer.test.ts
@@ -5,7 +5,7 @@ import {
type ConversationEvent,
} from "#client/conversation-reducer.js";
import { openConversationInputs, type ConversationState } from "#client/conversation-state.js";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import {
createAgentStartedEvent,
createInputRequestedEvent,
@@ -172,7 +172,7 @@ describe("conversation reducer tasks and agent sessions", () => {
it("marks a parked turn waiting until it resumes or ends", () => {
const parked = reduce([
createTurnStartedEvent({ sequence: 0, turnId: "t1" }),
- createTurnWaitingEvent({ sequence: 0, turnId: "t1" }),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 0, turnId: "t1" }),
]);
expect(parked.turns.t1).toEqual({ turnId: "t1", status: "active", waiting: true });
expect(parked.activeTurnId).toBe("t1");
@@ -183,7 +183,7 @@ describe("conversation reducer tasks and agent sessions", () => {
expect(resumed.turns.t1).toEqual({ turnId: "t1", status: "active" });
const ended = reduce(
[
- createTurnWaitingEvent({ sequence: 0, turnId: "t1" }),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 0, turnId: "t1" }),
createTurnCompletedEvent({ sequence: 0, turnId: "t1" }),
],
resumed,
diff --git a/packages/eve/src/client/conversation-state.test.ts b/packages/eve/src/client/conversation-state.test.ts
index e5d3e94a1..fb516a967 100644
--- a/packages/eve/src/client/conversation-state.test.ts
+++ b/packages/eve/src/client/conversation-state.test.ts
@@ -9,7 +9,7 @@ import {
isAgentSessionCaughtUp,
type ConversationState,
} from "#client/conversation-state.js";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import {
createAgentStartedEvent,
createAuthorizationCompletedEvent,
@@ -170,7 +170,7 @@ describe("agent tool sessions", () => {
description: "Connect Alice's notes",
webhookUrl: "https://example.com/callback",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
expect(caughtUp()).toBe(false);
state = observe(state, [
diff --git a/packages/eve/src/client/eve-agent-store.test.ts b/packages/eve/src/client/eve-agent-store.test.ts
index 73fecbeec..a63e5594a 100644
--- a/packages/eve/src/client/eve-agent-store.test.ts
+++ b/packages/eve/src/client/eve-agent-store.test.ts
@@ -9,7 +9,7 @@ import { Client } from "#client/client.js";
import { defaultMessageReducer } from "#client/message-reducer.js";
import { conversationReducer } from "#client/conversation-reducer.js";
import type { ConversationState } from "#client/conversation-state.js";
-import { stampTestEvent, stampTestEvents } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent, stampTestEvents } from "#internal/testing/events.js";
import {
createApprovalSettledEvent,
createAuthorizationCompletedEvent,
@@ -48,7 +48,7 @@ function turnEvents(): MessageStreamEvent[] {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
}
@@ -75,7 +75,7 @@ function streamingTurnEvents(): MessageStreamEvent[] {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
}
@@ -390,7 +390,7 @@ describe("EveAgentStore lifecycle", () => {
vi.spyOn(globalThis, "fetch").mockImplementation(async (_request, init) =>
init?.method === "POST"
? await start.promise
- : streamResponse(stampTestEvents([createSessionWaitingEvent()])),
+ : streamResponse(stampTestEvents([createSessionWaitingEvent(TEST_USAGE)])),
);
const store = createStore<readonly string[]>({
initialSession: { sessionId: "session_1", streamIndex: 0 },
@@ -456,7 +456,7 @@ describe("EveAgentStore held turns", () => {
taskId: "task_1",
turnId: "turn_1",
}),
- createTurnWaitingEvent({ sequence: 0, turnId: "turn_1" }),
+ createTurnWaitingEvent({ usage: TEST_USAGE, sequence: 0, turnId: "turn_1" }),
]).map((event) => ({ ...event, meta: { ...event.meta, deliveryIds: ["delivery_1"] } }));
for (const event of parked) live.emit(event);
await sending;
@@ -482,7 +482,7 @@ describe("EveAgentStore held turns", () => {
turnId: "turn_1",
}),
createTurnCompletedEvent({ sequence: 1, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]).map((event) => ({ ...event, meta: { ...event.meta, id: `resumed-${event.meta.id}` } }));
live.emit(resumed[0]!);
await vi.waitFor(() => expect(store.snapshot.data.inputs.req_1?.status).toBe("settled"));
@@ -523,7 +523,10 @@ describe("EveAgentStore agent-session following", () => {
]
: []),
...(ended
- ? [createTurnCompletedEvent({ sequence: 1, turnId: "turn_1" }), createSessionWaitingEvent()]
+ ? [
+ createTurnCompletedEvent({ sequence: 1, turnId: "turn_1" }),
+ createSessionWaitingEvent(TEST_USAGE),
+ ]
: []),
]);
@@ -647,7 +650,7 @@ describe("EveAgentStore agent-session following", () => {
const sending = store.send({ message: "New question" });
const confirmation = stampTestEvents([
createMessageReceivedEvent({ message: "New question", sequence: 1, turnId: "turn_2" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]).map((event, index) => ({
...event,
meta: { ...event.meta, id: `confirmation_${index}`, deliveryIds: ["delivery_1"] },
@@ -722,7 +725,9 @@ describe("EveAgentStore prewarming", () => {
.spyOn(globalThis, "fetch")
.mockRejectedValueOnce(error)
.mockResolvedValueOnce(startedResponse())
- .mockResolvedValueOnce(streamResponse(stampTestEvents([createSessionWaitingEvent()])));
+ .mockResolvedValueOnce(
+ streamResponse(stampTestEvents([createSessionWaitingEvent(TEST_USAGE)])),
+ );
const onError = vi.fn();
const store = createStore({ reducer: defaultMessageReducer() });
store.setCallbacks({ onError });
@@ -829,7 +834,9 @@ describe("EveAgentStore prewarming", () => {
const fetchMock = vi
.spyOn(globalThis, "fetch")
.mockReturnValueOnce(accepted.promise)
- .mockResolvedValueOnce(streamResponse(stampTestEvents([createSessionWaitingEvent()])));
+ .mockResolvedValueOnce(
+ streamResponse(stampTestEvents([createSessionWaitingEvent(TEST_USAGE)])),
+ );
const onFinish = vi.fn();
const onSessionChange = vi.fn();
const prepareSend = vi.fn();
@@ -926,7 +933,7 @@ describe("EveAgentStore prewarming", () => {
stepIndex: 0,
turnId: "turn_scheduled",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]).map((event) => ({
...event,
meta: { ...event.meta, deliveryIds: ["scheduled-delivery"] },
@@ -1190,7 +1197,7 @@ describe("EveAgentStore stream overlap", () => {
it("applies a pre-v20 event whose envelope has no id", async () => {
const legacy = preV20MessageCompletedEvent();
- const boundary = stampTestEvents([createSessionWaitingEvent()])[0]!;
+ const boundary = stampTestEvents([createSessionWaitingEvent(TEST_USAGE)])[0]!;
vi.spyOn(globalThis, "fetch")
.mockResolvedValueOnce(startedResponse())
.mockResolvedValueOnce(streamResponse([legacy, boundary]));
@@ -1338,7 +1345,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_0",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]).map((event, index) =>
index < 2 ? event : { ...event, meta: { ...event.meta, deliveryIds: ["delivery_1"] } },
);
@@ -1521,7 +1528,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const initialEvents = [received!, started!];
const live = controlledStreamResponse();
@@ -1555,6 +1562,7 @@ describe("EveAgentStore session resume", () => {
it("publishes a hydrated terminal failure with error status", async () => {
const failed = stampTestEvents([
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "SESSION_FAILED",
message: "Session failed.",
sessionId: "session_1",
@@ -1621,7 +1629,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createMessageReceivedEvent({ message: "Again", sequence: 0, turnId: "turn_2" }),
createTurnStartedEvent({ sequence: 1, turnId: "turn_2" }),
createMessageCompletedEvent({
@@ -1631,7 +1639,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_2",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const settled = events.slice(0, 3);
const live = controlledStreamResponse();
@@ -1678,7 +1686,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_3",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
const fetchMock = vi
.spyOn(globalThis, "fetch")
@@ -1707,7 +1715,7 @@ describe("EveAgentStore session resume", () => {
turnId: "turn_1",
webhookUrl: "https://agent.example.com/callback",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationCompletedEvent({
attemptId: "attempt_1",
name: "linear",
@@ -1716,7 +1724,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const live = controlledStreamResponse();
live.response.headers.set("x-eve-stream-tail-index", "1");
@@ -1754,7 +1762,7 @@ describe("EveAgentStore session resume", () => {
webhookUrl: `https://agent.example.com/callback/${attemptId}`,
}),
),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationCompletedEvent({
attemptId: "first",
name: "linear",
@@ -1763,7 +1771,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationCompletedEvent({
attemptId: "second",
name: "linear",
@@ -1772,7 +1780,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const live = controlledStreamResponse();
live.response.headers.set("x-eve-stream-tail-index", "2");
@@ -1808,7 +1816,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationRequiredEvent({
authorization: { url: "https://idp.example.com/authorize" },
description: "Linear",
@@ -1818,7 +1826,7 @@ describe("EveAgentStore session resume", () => {
turnId: "turn_2",
webhookUrl: "https://agent.example.com/eve/v1/connections/linear/callback/hook",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationCompletedEvent({
name: "linear",
outcome: "authorized",
@@ -1826,7 +1834,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_2",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const settled = events.slice(0, 3);
const fetchMock = vi
@@ -1855,7 +1863,7 @@ describe("EveAgentStore session resume", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const live = controlledStreamResponse();
const requests: string[] = [];
@@ -1939,7 +1947,7 @@ describe("EveAgentStore steering", () => {
1,
),
);
- live.emit(stampTestEvent(createSessionWaitingEvent(), 2));
+ live.emit(stampTestEvent(createSessionWaitingEvent(TEST_USAGE), 2));
await Promise.all([first, answering]);
expect(store.snapshot.status).toBe("ready");
expect(store.snapshot.conversation.inputs["color-a"]?.status).toBe("settled");
@@ -2073,7 +2081,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
for (const event of events.slice(0, 3)) active.emit(event);
await vi.waitFor(() =>
@@ -2110,7 +2118,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 1,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]).map((event) => ({
...event,
meta: { ...event.meta, deliveryIds: ["delivery_1"] },
@@ -2151,7 +2159,7 @@ describe("EveAgentStore steering", () => {
turnId: "turn_1",
}),
createMessageReceivedEvent({ message: "Second", sequence: 3, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]).map((event, index) => ({
...event,
meta: { ...event.meta, deliveryIds: [index === 3 ? "second" : "first"] },
@@ -2206,7 +2214,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]).map((event, index) => ({
...event,
meta: {
@@ -2246,7 +2254,7 @@ describe("EveAgentStore steering", () => {
const events = stampTestEvents([
createMessageReceivedEvent({ message: "First", sequence: 0, turnId: "turn_1" }),
createTurnStartedEvent({ sequence: 1, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createMessageReceivedEvent({ message: "Use Linear", sequence: 0, turnId: "turn_2" }),
createTurnStartedEvent({ sequence: 1, turnId: "turn_2" }),
createAuthorizationRequiredEvent({
@@ -2258,7 +2266,7 @@ describe("EveAgentStore steering", () => {
turnId: "turn_2",
webhookUrl: "https://agent.example.com/callback",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createAuthorizationCompletedEvent({
attemptId: "attempt_1",
name: "linear",
@@ -2267,7 +2275,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 0,
turnId: "turn_2",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]).map((event, index) => ({
...event,
meta: { ...event.meta, deliveryIds: [index < 3 ? "first-delivery" : "delivery_1"] },
@@ -2319,7 +2327,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
createMessageReceivedEvent({ message: "Instead", sequence: 0, turnId: "turn_2" }),
createTurnStartedEvent({ sequence: 1, turnId: "turn_2" }),
createMessageCompletedEvent({
@@ -2329,7 +2337,7 @@ describe("EveAgentStore steering", () => {
stepIndex: 0,
turnId: "turn_2",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]).map((event, index) => ({
...event,
meta: { ...event.meta, deliveryIds: [index < 4 ? "first-delivery" : "delivery_1"] },
@@ -2394,6 +2402,7 @@ describe("EveAgentStore terminal failure", () => {
it("publishes a live terminal failure with error status", async () => {
const failed = stampTestEvents([
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "SESSION_FAILED",
message: "Session failed.",
sessionId: "session_1",
@@ -2426,7 +2435,7 @@ describe("EveAgentStore cancellation", () => {
const [turnStarted, turnCancelled, boundary] = stampTestEvents([
createTurnStartedEvent({ sequence: 0, turnId: "turn_1" }),
createTurnCancelledEvent({ sequence: 1, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const fetchMock = vi
.spyOn(globalThis, "fetch")
@@ -2553,7 +2562,7 @@ describe("EveAgentStore session controls", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
vi.spyOn(globalThis, "fetch").mockImplementation(async (_input, init) =>
init?.method === "POST"
diff --git a/packages/eve/src/client/eve-agent-store.ts b/packages/eve/src/client/eve-agent-store.ts
index 01230be7a..d50614f76 100644
--- a/packages/eve/src/client/eve-agent-store.ts
+++ b/packages/eve/src/client/eve-agent-store.ts
@@ -627,7 +627,9 @@ export class EveAgentStore<TData> {
const settled =
isSettledSessionTail(this.#events, conversation) && conversation.activeTurnId === undefined;
if (this.#status !== "resuming" && this.#error === undefined) {
- if ("data" in event && "turnId" in event.data) this.#status = "streaming";
+ if ("data" in event && event.data !== undefined && "turnId" in event.data) {
+ this.#status = "streaming";
+ }
if (this.#activeTurn === undefined && settled) this.#status = "ready";
}
this.#callbacks.onSessionChange?.(this.#session?.state);
diff --git a/packages/eve/src/client/message-response.test.ts b/packages/eve/src/client/message-response.test.ts
index 46007cd27..541f0c25d 100644
--- a/packages/eve/src/client/message-response.test.ts
+++ b/packages/eve/src/client/message-response.test.ts
@@ -1,7 +1,7 @@
import { describe, expect, it, vi } from "vitest";
import { MessageResponse } from "#client/message-response.js";
-import { stampTestEvents } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvents } from "#internal/testing/events.js";
import {
createSessionWaitingEvent,
createTurnStartedEvent,
@@ -29,7 +29,7 @@ describe("MessageResponse cancellation", () => {
const settle = createDeferred<void>();
const [turnStarted, boundary] = stampTestEvents([
createTurnStartedEvent({ sequence: 0, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const cancelTurn = vi.fn(async () => acceptedCancellation());
const response = new MessageResponse({
@@ -62,7 +62,7 @@ describe("MessageResponse cancellation", () => {
const settle = createDeferred<void>();
const [turnStarted, boundary] = stampTestEvents([
createTurnStartedEvent({ sequence: 0, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const cancelTurn = vi
.fn<(turnId: string) => Promise<ReturnType<typeof acceptedCancellation>>>()
@@ -91,7 +91,7 @@ describe("MessageResponse cancellation", () => {
it("drops a queued cancellation when the response settles without starting a turn", async () => {
const [boundary] = stampTestEvents([
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
] as UnstampedMessageStreamEvent[]);
const cancelTurn = vi.fn(async () => acceptedCancellation());
const response = new MessageResponse({
diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.ts
index efb5022d1..fe9efe275 100644
--- a/packages/eve/src/compiler/extension-compatibility.ts
+++ b/packages/eve/src/compiler/extension-compatibility.ts
@@ -22,10 +22,10 @@ interface ExtensionCapabilityContract {
const EXTENSION_CAPABILITY_CONTRACTS = {
extension: { current: 1, supported: [1], dropped: {} },
tool: {
- current: 71,
+ current: 72,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 29, 30, 31, 32, 34, 35, 54, 55, 63, 64, 65, 66, 67,
- 68, 69, 70, 71,
+ 68, 69, 70, 71, 72,
],
dropped: {
14: "TaskExec.delegated was removed; migrate to workflow-backed background tools",
@@ -72,10 +72,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
dynamicTool: {
- current: 68,
+ current: 69,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 31, 32, 33, 52, 54,
- 60, 61, 62, 63, 64, 65, 66, 67, 68,
+ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,
],
dropped: {
21: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
@@ -114,10 +114,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
channel: {
- current: 43,
+ current: 44,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 29, 31, 36, 37, 38, 39, 40, 41, 42,
- 43,
+ 43, 44,
],
dropped: {
12: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
@@ -139,8 +139,8 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
schedule: {
- current: 24,
- supported: [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 23, 24],
+ current: 25,
+ supported: [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 23, 24, 25],
dropped: {
5: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
16: 'Background task delivery was removed: sends no longer accept taskDeliveryPolicy and message.completed always carries text. input.resolved reports outcome "cancelled" when a workflow tool withdraws a ctx.ask() request, and stream events gained agent.started, published when a workflow run opens a ctx.agent session, and turn.waiting, which parks the open turn instead of emitting turn.completed and session.waiting.',
@@ -148,8 +148,8 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
subagent: {
- current: 27,
- supported: [3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 23, 24, 25, 26, 27],
+ current: 28,
+ supported: [3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 23, 24, 25, 26, 27, 28],
dropped: {
1: "Persistent subagent sessions are now the default and the experimental opt-in was removed",
2: "Persistent subagent sessions are now the default and the experimental opt-in was removed",
@@ -162,10 +162,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
connection: {
- current: 36,
+ current: 37,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 26, 32, 33, 34,
- 35, 36,
+ 35, 36, 37,
],
dropped: {
9: "Dynamic connection resolvers no longer receive conversation or channel continuation data",
@@ -180,8 +180,8 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
hook: {
- current: 35,
- supported: [20, 21, 22, 23, 25, 27, 31, 32, 33, 34, 35],
+ current: 36,
+ supported: [20, 21, 22, 23, 25, 27, 31, 32, 33, 34, 35, 36],
dropped: {
1: "Model identity moved from session.started runtime metadata to step.started call attribution.",
2: "Model identity moved from session.started runtime metadata to step.started call attribution.",
@@ -211,9 +211,9 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
skill: { current: 2, supported: [1, 2], dropped: {} },
dynamicSkill: {
- current: 29,
+ current: 30,
supported: [
- 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 25, 26, 27, 28, 29,
+ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 25, 26, 27, 28, 29, 30,
],
dropped: {
13: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
@@ -224,9 +224,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
instructions: { current: 2, supported: [1, 2], dropped: {} },
dynamicInstructions: {
- current: 30,
+ current: 31,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 26, 27, 28, 29, 30,
+ 31,
],
dropped: {
14: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
diff --git a/packages/eve/src/context/hook-lifecycle.integration.test.ts b/packages/eve/src/context/hook-lifecycle.integration.test.ts
index 61c7d4128..4a4631207 100644
--- a/packages/eve/src/context/hook-lifecycle.integration.test.ts
+++ b/packages/eve/src/context/hook-lifecycle.integration.test.ts
@@ -9,7 +9,7 @@ import {
createTurnStartedEvent,
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js";
import { ContextContainer, contextStorage } from "./container.js";
import { dispatchStreamEventHooks } from "./hook-lifecycle.js";
@@ -113,6 +113,7 @@ describe("dispatchStreamEventHooks", () => {
createStepStartedEvent({ sequence: 0, turnId: "turn_0", stepIndex: 0, modelId: "test" }),
{ type: "session.completed" } as const,
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "TEST_FAILURE",
message: "Runtime failure",
sessionId: "session_test",
diff --git a/packages/eve/src/evals/runner/derive-run-facts.test.ts b/packages/eve/src/evals/runner/derive-run-facts.test.ts
index 96c36f354..626858168 100644
--- a/packages/eve/src/evals/runner/derive-run-facts.test.ts
+++ b/packages/eve/src/evals/runner/derive-run-facts.test.ts
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import type { UnstampedMessageStreamEvent } from "#protocol/message.js";
import type { JsonObject } from "#shared/json.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import { stampTestEvents } from "#internal/testing/events.js";
import { deriveRunFacts, type DeriveRunFactsOptions } from "#evals/runner/derive-run-facts.js";
import type { EveEvalDerivedFacts } from "#evals/types.js";
@@ -66,6 +67,22 @@ function stepStarted(modelId: string, stepIndex: number): UnstampedMessageStream
return { type: "step.started", data: { modelId, sequence: 1, stepIndex, turnId: "t1" } };
}
+const USAGE = { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 40, outputTokens: 8 };
+
+function sessionFailed(usage: TokenUsage): UnstampedMessageStreamEvent {
+ return {
+ type: "session.failed",
+ data: { code: "MODEL_CALL_FAILED", message: "failed", sessionId: "s1", usage },
+ };
+}
+
+function sessionWaiting(usage: TokenUsage | undefined): UnstampedMessageStreamEvent {
+ return {
+ type: "session.waiting",
+ data: { continuationToken: "", wait: "next-user-message", ...(usage && { usage }) },
+ };
+}
+
function taskStarted(
callId: string,
name: string,
@@ -443,6 +460,50 @@ describe("deriveRunFacts", () => {
expect(facts.models).toEqual(["anthropic/claude-sonnet-4.6", "openai/gpt-5.1"]);
});
+ it("reports the usage of the session's latest session.waiting", () => {
+ const facts = derive([
+ sessionWaiting({ cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 10, outputTokens: 2 }),
+ sessionWaiting({
+ cacheReadTokens: 1,
+ cacheWriteTokens: 0,
+ costUsd: 0.5,
+ inputTokens: 40,
+ outputTokens: 8,
+ }),
+ ]);
+ expect(facts.usage).toEqual({
+ cacheReadTokens: 1,
+ cacheWriteTokens: 0,
+ costUsd: 0.5,
+ inputTokens: 40,
+ outputTokens: 8,
+ });
+ });
+
+ it("reports the usage of a session.failed or session.completed that came after it", () => {
+ const failed = { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 90, outputTokens: 9 };
+ const completed = { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 70, outputTokens: 7 };
+ expect([
+ derive([sessionWaiting(USAGE), sessionFailed(failed)]).usage,
+ derive([sessionWaiting(USAGE), { type: "session.completed", data: { usage: completed } }])
+ .usage,
+ ]).toEqual([failed, completed]);
+ });
+
+ it("reports the usage of a turn.waiting that parked the turn after it", () => {
+ const parked = { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 55, outputTokens: 5 };
+ expect(
+ derive([
+ sessionWaiting(USAGE),
+ { type: "turn.waiting", data: { sequence: 2, turnId: "t2", usage: parked } },
+ ]).usage,
+ ).toEqual(parked);
+ });
+
+ it("reports no usage when the latest session event carried none", () => {
+ expect(derive([sessionWaiting(USAGE), sessionWaiting(undefined)]).usage).toBeUndefined();
+ });
+
it("captures failure code from session.failed event", () => {
const events: UnstampedMessageStreamEvent[] = [
{
diff --git a/packages/eve/src/evals/runner/derive-run-facts.ts b/packages/eve/src/evals/runner/derive-run-facts.ts
index 7d9425d9f..38bee80d0 100644
--- a/packages/eve/src/evals/runner/derive-run-facts.ts
+++ b/packages/eve/src/evals/runner/derive-run-facts.ts
@@ -6,6 +6,7 @@ import type {
import { LOAD_SKILL_TOOL_NAME } from "#runtime/skills/fragment-context.js";
import type { InputRequest } from "#shared/input.js";
import type { JsonObject, JsonValue } from "#shared/json.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import type { EveEvalDerivedFacts, EveEvalSubagentCall, EveEvalToolCall } from "#evals/types.js";
interface MutableToolCall {
@@ -68,6 +69,7 @@ export function deriveRunFacts(
let messageCount = 0;
let reasoningBlockCount = 0;
const models = new Set<string>();
+ let usage: TokenUsage | undefined;
let failureCode: string | undefined;
const ensureToolCall = (callId: string, name: string, input: JsonObject): MutableToolCall => {
@@ -157,7 +159,19 @@ export function deriveRunFacts(
break;
}
+ case "session.waiting":
+ case "turn.waiting": {
+ usage = event.data.usage;
+ break;
+ }
+
+ case "session.completed": {
+ usage = event.data?.usage;
+ break;
+ }
+
case "session.failed": {
+ usage = event.data.usage;
failureCode = event.data.code;
break;
}
@@ -180,6 +194,7 @@ export function deriveRunFacts(
messageCount,
reasoningBlockCount,
models: [...models],
+ usage,
failureCode,
};
}
diff --git a/packages/eve/src/evals/runner/execute-task.test.ts b/packages/eve/src/evals/runner/execute-task.test.ts
index 9ea5697f1..4e1593f29 100644
--- a/packages/eve/src/evals/runner/execute-task.test.ts
+++ b/packages/eve/src/evals/runner/execute-task.test.ts
@@ -10,6 +10,7 @@ import {
import { stampTestEvents } from "#internal/testing/events.js";
import { executeTask } from "#evals/runner/execute-task.js";
import type { EveEval, EveEvalContext } from "#evals/types.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import { createEvalTargetHandle } from "#evals/target.js";
import { satisfies } from "#evals/expect/index.js";
import { z } from "zod";
@@ -522,6 +523,157 @@ describe("executeTask", () => {
});
});
+ it("sums the usage of every session except those another captured session opened", async () => {
+ const server = createScriptedServer(
+ [
+ {
+ sessionId: "parent",
+ events: [
+ turnStarted("turn_1"),
+ agentStarted("child", "researcher"),
+ messageCompleted("parent done", "turn_1"),
+ sessionWaiting(usage(100)),
+ ],
+ },
+ {
+ sessionId: "independent",
+ events: [
+ turnStarted("turn_3"),
+ messageCompleted("done", "turn_3"),
+ sessionWaiting(usage(7)),
+ ],
+ },
+ ],
+ {
+ streams: [
+ {
+ sessionId: "child",
+ events: [
+ turnStarted("turn_2"),
+ messageCompleted("done", "turn_2"),
+ sessionWaiting(usage(60)),
+ ],
+ },
+ ],
+ },
+ );
+ vi.spyOn(globalThis, "fetch").mockImplementation(server.fetch);
+
+ const { result } = await executeTask({
+ client: new Client({ host: target.url }),
+ target,
+ evaluation: createTestEval(async (t) => {
+ await t.send("parent");
+ await t.target.attachSession("child");
+ await t.send("independent");
+ }, "delegated-usage"),
+ });
+
+ expect(result.derived.usage).toEqual(usage(107, 2));
+ });
+
+ it("counts a session captured twice once, by its latest usage", async () => {
+ const server = createScriptedServer([], {
+ streams: [
+ {
+ sessionId: "scheduled",
+ events: [
+ turnStarted("turn_1"),
+ messageCompleted("first", "turn_1"),
+ sessionWaiting(usage(10)),
+ ],
+ },
+ {
+ sessionId: "scheduled",
+ events: [
+ turnStarted("turn_2"),
+ messageCompleted("second", "turn_2"),
+ sessionWaiting(usage(30)),
+ ],
+ },
+ ],
+ });
+ vi.spyOn(globalThis, "fetch").mockImplementation(server.fetch);
+
+ const { result } = await executeTask({
+ client: new Client({ host: target.url }),
+ target,
+ evaluation: createTestEval(async (t) => {
+ await t.target.attachSession("scheduled");
+ await t.target.attachSession("scheduled");
+ }, "recaptured-usage"),
+ });
+
+ expect(result.derived.usage).toEqual(usage(30, 1));
+ });
+
+ it("sums cost over the counted sessions that reported one", async () => {
+ const server = createScriptedServer([
+ {
+ sessionId: "priced",
+ events: [
+ turnStarted("turn_1"),
+ messageCompleted("done", "turn_1"),
+ sessionWaiting(usage(5, 1, 0.5)),
+ ],
+ },
+ {
+ sessionId: "unpriced",
+ events: [
+ turnStarted("turn_2"),
+ messageCompleted("done", "turn_2"),
+ sessionWaiting(usage(7)),
+ ],
+ },
+ ]);
+ vi.spyOn(globalThis, "fetch").mockImplementation(server.fetch);
+
+ const { result } = await executeTask({
+ client: new Client({ host: target.url }),
+ target,
+ evaluation: createTestEval(async (t) => {
+ await t.send("priced");
+ await t.send("unpriced");
+ }, "partial-cost"),
+ });
+
+ expect(result.derived.usage?.costUsd).toBe(0.5);
+ });
+
+ it("reports no usage when a counted session reported none", async () => {
+ const server = createScriptedServer([
+ {
+ sessionId: "reported",
+ events: [
+ turnStarted("turn_1"),
+ messageCompleted("done", "turn_1"),
+ sessionWaiting({
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ inputTokens: 5,
+ outputTokens: 1,
+ }),
+ ],
+ },
+ {
+ sessionId: "unreported",
+ events: [turnStarted("turn_2"), messageCompleted("done", "turn_2"), sessionCompleted()],
+ },
+ ]);
+ vi.spyOn(globalThis, "fetch").mockImplementation(server.fetch);
+
+ const { result } = await executeTask({
+ client: new Client({ host: target.url }),
+ target,
+ evaluation: createTestEval(async (t) => {
+ await t.send("reported");
+ await t.send("unreported");
+ }, "unreported-usage"),
+ });
+
+ expect(result.derived.usage).toBeUndefined();
+ });
+
it("records assertions against individual turns without leaking other turns", async () => {
const server = createScriptedServer([
{
@@ -1067,9 +1219,13 @@ function turnCompleted(turnId: string): UnstampedMessageStreamEvent {
return { data: { sequence: 3, turnId }, type: "turn.completed" };
}
-function sessionWaiting(): UnstampedMessageStreamEvent {
+function usage(inputTokens: number, outputTokens = 1, costUsd?: number): TokenUsage {
+ return { cacheReadTokens: 0, cacheWriteTokens: 0, costUsd, inputTokens, outputTokens };
+}
+
+function sessionWaiting(usage?: TokenUsage): UnstampedMessageStreamEvent {
return {
- data: { continuationToken: "session-id", wait: "next-user-message" },
+ data: { continuationToken: "session-id", wait: "next-user-message", ...(usage && { usage }) },
type: "session.waiting",
};
}
diff --git a/packages/eve/src/evals/runner/execute-task.ts b/packages/eve/src/evals/runner/execute-task.ts
index be1061327..c81d16b11 100644
--- a/packages/eve/src/evals/runner/execute-task.ts
+++ b/packages/eve/src/evals/runner/execute-task.ts
@@ -2,6 +2,7 @@ import { runUntilAborted } from "#evals/abort.js";
import type { Client } from "#client/client.js";
import type { MessageStreamEvent, RuntimeIdentity } from "#protocol/message.js";
import { toErrorMessage } from "#shared/errors.js";
+import { addTokenUsage, type TokenUsage } from "#shared/token-usage.js";
import type {
AssertionResult,
EveEval,
@@ -172,10 +173,32 @@ function combineDerivedFacts(sessions: readonly EveEvalSessionResult[]): EveEval
messageCount: sum(sessions, (session) => session.derived.messageCount),
reasoningBlockCount: sum(sessions, (session) => session.derived.reasoningBlockCount),
models: [...new Set(sessions.flatMap((session) => session.derived.models))],
+ usage: evalUsage(sessions),
failureCode,
};
}
+/**
+ * The eval's usage: each captured session's latest usage, counted once however often the eval
+ * captured it, less the sessions another captured session opened, whose spend that session already
+ * counts. No usage when a counted session reported none.
+ */
+function evalUsage(sessions: readonly EveEvalSessionResult[]): TokenUsage | undefined {
+ const opened = new Set(
+ sessions.flatMap((session) =>
+ session.events.flatMap((event) =>
+ event.type === "agent.started" ? [event.data.sessionId] : [],
+ ),
+ ),
+ );
+ const latestById = new Map(sessions.map((session) => [session.sessionId, session.derived.usage]));
+ const counted = [...latestById].flatMap(([id, usage]) =>
+ id !== undefined && opened.has(id) ? [] : [usage],
+ );
+ if (!counted.every((usage): usage is TokenUsage => usage !== undefined)) return undefined;
+ return counted.reduce(addTokenUsage);
+}
+
function selectPrimarySessionId(sessions: readonly EveEvalSessionResult[]): string | undefined {
return sessions.find((session) => session.primary)?.sessionId ?? sessions[0]?.sessionId;
}
diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.ts
index 7a610dc04..80a0a4a5c 100644
--- a/packages/eve/src/evals/types.ts
+++ b/packages/eve/src/evals/types.ts
@@ -1,3 +1,4 @@
+import type { TokenUsage } from "#shared/token-usage.js";
import type { Experimental_EvaluationModel as EvaluationModel } from "ai";
import type { StandardSchemaV1 } from "#compiled/@standard-schema/spec/index.js";
@@ -94,6 +95,14 @@ export interface EveEvalDerivedFacts {
* attaches that session. Compaction and `auto` routing calls are not included.
*/
readonly models: readonly string[];
+ /**
+ * Token usage from the latest `session.waiting`, `turn.waiting`, `session.failed`, or
+ * `session.completed`: the
+ * session's own model calls plus what the agents it delegated to spent, so on a turn it is the
+ * session's total so far. For an eval, each captured session counts once, by its latest usage,
+ * except sessions another captured session opened. Absent when a counted session reported none.
+ */
+ readonly usage?: TokenUsage;
readonly failureCode?: string;
}
diff --git a/packages/eve/src/execution/session-event-stream.test.ts b/packages/eve/src/execution/session-event-stream.test.ts
index 7491d4073..3daaefed0 100644
--- a/packages/eve/src/execution/session-event-stream.test.ts
+++ b/packages/eve/src/execution/session-event-stream.test.ts
@@ -1,6 +1,6 @@
import { afterEach, describe, expect, it, vi } from "vitest";
-import { stampTestEvent } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvent } from "#internal/testing/events.js";
import { createSessionWaitingEvent, type MessageStreamEvent } from "#protocol/message.js";
import { sessions } from "#public/server/index.js";
@@ -45,7 +45,9 @@ function mockSessionStream(
}
const storedEvents = (count: number) =>
- Array.from({ length: count }, (_, index) => stampTestEvent(createSessionWaitingEvent(), index));
+ Array.from({ length: count }, (_, index) =>
+ stampTestEvent(createSessionWaitingEvent(TEST_USAGE), index),
+ );
async function collect(iterable: AsyncIterable<MessageStreamEvent>) {
const events: MessageStreamEvent[] = [];
diff --git a/packages/eve/src/execution/session/finalization.test.ts b/packages/eve/src/execution/session/finalization.test.ts
index c9987844d..36c98bf1c 100644
--- a/packages/eve/src/execution/session/finalization.test.ts
+++ b/packages/eve/src/execution/session/finalization.test.ts
@@ -3,6 +3,8 @@ import { finalizeSession } from "#execution/session/finalization.js";
import { createDurableSessionState } from "#execution/durable-session-store.js";
import { setTurnUsageState, takeSessionUsageDelta } from "#harness/turn-tag-state.js";
import type { HarnessSession } from "#harness/types.js";
+import { emitTerminalSessionCompletionStep } from "#execution/terminal-session-completion-step.js";
+import { emitTerminalSessionFailureStep } from "#execution/terminal-session-failure-step.js";
import { notifyTurnCallerStep } from "#subagents/parent-notification.js";
vi.mock("#execution/terminate-child-sessions-step.js", () => ({
@@ -80,6 +82,29 @@ describe("session finalization with an unsettled caller", () => {
},
);
+ it.each([
+ { kind: "expired" as const, step: emitTerminalSessionCompletionStep },
+ {
+ kind: "failed" as const,
+ error: new Error("Owner failed"),
+ step: emitTerminalSessionFailureStep,
+ },
+ ])("ends a $kind session with its usage on the terminal event", async ({ step, ...outcome }) => {
+ await finalizeSession(outcome, {
+ caller: undefined,
+ cursor: { serializedContext: {}, sessionState: sessionWithUnreportedUsage() },
+ sessionWritable: new WritableStream(),
+ });
+
+ expect(vi.mocked(step).mock.calls[0]?.[0].usage).toEqual({
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ costUsd: undefined,
+ inputTokens: 250,
+ outputTokens: 0,
+ });
+ });
+
it("does not notify again when an already settled conversation expires", async () => {
const result = await finalizeSession(
{ kind: "expired" },
diff --git a/packages/eve/src/execution/session/finalization.ts b/packages/eve/src/execution/session/finalization.ts
index 5371d3964..80c66f18c 100644
--- a/packages/eve/src/execution/session/finalization.ts
+++ b/packages/eve/src/execution/session/finalization.ts
@@ -7,7 +7,7 @@ import type { TurnOutcome } from "#execution/session/turn-step-types.js";
import { normalizeSerializableError } from "#execution/workflow-errors.js";
import type { WorkflowEntryResult } from "#execution/session/entry-input.js";
import type { TokenUsage } from "#shared/token-usage.js";
-import { getSessionTokenUsage, takeSessionUsageDelta, toUsage } from "#harness/turn-tag-state.js";
+import { getSessionUsage, takeSessionUsageDelta } from "#harness/turn-tag-state.js";
import { notifyTurnCallerStep } from "#subagents/parent-notification.js";
/** The three ways a session ends. `done` already emitted its terminal event inside the turn. */
@@ -37,10 +37,13 @@ export async function finalizeSession(
if (sessionState !== undefined) {
await terminateChildSessionsStep({ sessionState });
}
+ const session = sessionState?.snapshot.session;
+ const usage = session === undefined ? undefined : getSessionUsage(session);
if (outcome.kind === "expired") {
await emitTerminalSessionCompletionStep({
sessionWritable: context.sessionWritable,
serializedContext,
+ usage,
});
} else if (outcome.kind === "failed") {
await emitTerminalSessionFailureStep({
@@ -48,6 +51,7 @@ export async function finalizeSession(
sessionWritable: context.sessionWritable,
serializedContext,
turnId: outcome.turnId,
+ usage,
});
}
@@ -94,7 +98,7 @@ function settledResult(
session === undefined || context.caller === undefined
? {}
: {
- sessionUsage: toUsage(getSessionTokenUsage(session)),
+ sessionUsage: getSessionUsage(session),
turnUsage: takeSessionUsageDelta(session).delta,
};
switch (outcome.kind) {
diff --git a/packages/eve/src/execution/session/turn-step.test.ts b/packages/eve/src/execution/session/turn-step.test.ts
index a8e2156ca..a7d99f3f2 100644
--- a/packages/eve/src/execution/session/turn-step.test.ts
+++ b/packages/eve/src/execution/session/turn-step.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { createTestSessionState } from "#internal/testing/session-state.js";
import type { ModelMessage } from "ai";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
@@ -2810,6 +2811,7 @@ describe("emitTerminalSessionFailureStep", () => {
const errorLog = vi.spyOn(console, "error").mockImplementation(() => {});
await emitTerminalSessionFailureStep({
+ usage: TEST_USAGE,
error,
sessionWritable: createTestWritable(),
serializedContext: serialized,
@@ -2859,6 +2861,7 @@ describe("emitTerminalSessionFailureStep", () => {
});
await emitTerminalSessionFailureStep({
+ usage: TEST_USAGE,
error,
sessionWritable: createTestWritable(),
serializedContext: serialized,
@@ -2907,6 +2910,7 @@ describe("emitTerminalSessionFailureStep", () => {
// a secondary failure during notification.
await expect(
emitTerminalSessionFailureStep({
+ usage: TEST_USAGE,
error: new Error("inner"),
sessionWritable: createTestWritable(),
serializedContext: serialized,
diff --git a/packages/eve/src/execution/session/turn-waiting-step.ts b/packages/eve/src/execution/session/turn-waiting-step.ts
index 7913c976d..fd866402d 100644
--- a/packages/eve/src/execution/session/turn-waiting-step.ts
+++ b/packages/eve/src/execution/session/turn-waiting-step.ts
@@ -3,6 +3,8 @@ import {
withSessionStateDelta,
type SessionStateTransition,
} from "#execution/session/state-delta.js";
+import { readDurableSession } from "#execution/durable-session-store.js";
+import { getSessionUsage } from "#harness/turn-tag-state.js";
import { createTurnWaitingEvent } from "#protocol/message.js";
/** Publishes `turn.waiting` for the open turn the session workflow just parked. */
@@ -13,6 +15,7 @@ export async function publishTurnWaitingStep(
return await withSessionStateDelta(target, async (input) => {
const { sequence, turnId } = input.sessionState.emissionState;
- return await publishSessionEvents(input, [createTurnWaitingEvent({ sequence, turnId })]);
+ const usage = getSessionUsage(readDurableSession(input.sessionState));
+ return await publishSessionEvents(input, [createTurnWaitingEvent({ sequence, turnId, usage })]);
});
}
diff --git a/packages/eve/src/execution/settle-cancelled-turn-step.ts b/packages/eve/src/execution/settle-cancelled-turn-step.ts
index c5a42c02c..dcb3a6ec2 100644
--- a/packages/eve/src/execution/settle-cancelled-turn-step.ts
+++ b/packages/eve/src/execution/settle-cancelled-turn-step.ts
@@ -18,7 +18,11 @@ import type { HarnessModelMessage } from "#harness/messages.js";
import { clearPendingSessionLimitPrompt } from "#harness/input-requests.js";
import { getHarnessEmissionState, setHarnessEmissionState } from "#harness/emission.js";
import { removeBlockingWorkflowToolRuns } from "#harness/workflow-tool-runs.js";
-import { getTurnUsageState, takeSessionUsageDelta } from "#harness/turn-tag-state.js";
+import {
+ getSessionUsage,
+ getTurnUsageState,
+ takeSessionUsageDelta,
+} from "#harness/turn-tag-state.js";
import type { TokenUsage } from "#shared/token-usage.js";
export interface CancelledTurnSettleResult {
@@ -65,7 +69,12 @@ export async function settleCancelledTurn(
const durableState = step.durableSession.state;
const { published, result: usage } = await publishFromSessionStep(step, {
origin: "own",
- publish: (emit) => emitCancelledTurn(emit, getHarnessEmissionState(durableState)),
+ publish: (emit) =>
+ emitCancelledTurn(
+ emit,
+ getHarnessEmissionState(durableState),
+ getSessionUsage(step.durableSession),
+ ),
updateSession(session, emissionState) {
// `clearPendingSessionLimitPrompt`: cancellation settles with the step's
// input snapshot, which can resurrect an already-answered session-limit
diff --git a/packages/eve/src/execution/terminal-session-completion-step.ts b/packages/eve/src/execution/terminal-session-completion-step.ts
index 5820d10b8..76000c52a 100644
--- a/packages/eve/src/execution/terminal-session-completion-step.ts
+++ b/packages/eve/src/execution/terminal-session-completion-step.ts
@@ -1,12 +1,18 @@
import { publishTerminalSessionEvent } from "#execution/publish-session-events.js";
import { createSessionCompletedEvent } from "#protocol/message.js";
+import type { TokenUsage } from "#shared/token-usage.js";
/** Emits a terminal `session.completed` outside a turn. */
export async function emitTerminalSessionCompletionStep(input: {
readonly sessionWritable: WritableStream<Uint8Array>;
readonly serializedContext: Record<string, unknown>;
+ readonly usage: TokenUsage | undefined;
}): Promise<void> {
"use step";
- await publishTerminalSessionEvent({ ...input, event: createSessionCompletedEvent() });
+ await publishTerminalSessionEvent({
+ sessionWritable: input.sessionWritable,
+ serializedContext: input.serializedContext,
+ event: createSessionCompletedEvent(input.usage),
+ });
}
diff --git a/packages/eve/src/execution/terminal-session-failure-step.ts b/packages/eve/src/execution/terminal-session-failure-step.ts
index ec0c568b3..ff956af11 100644
--- a/packages/eve/src/execution/terminal-session-failure-step.ts
+++ b/packages/eve/src/execution/terminal-session-failure-step.ts
@@ -2,6 +2,7 @@ import { publishTerminalSessionEvent } from "#execution/publish-session-events.j
import { summarizeKnownError } from "#harness/semantic-errors/index.js";
import { createLogger, formatError } from "#internal/logging.js";
import { createSessionFailedEvent } from "#protocol/message.js";
+import type { TokenUsage } from "#shared/token-usage.js";
const log = createLogger("execution.workflow-entry");
@@ -11,6 +12,7 @@ export async function emitTerminalSessionFailureStep(input: {
readonly sessionWritable: WritableStream<Uint8Array>;
readonly serializedContext: Record<string, unknown>;
readonly turnId?: string;
+ readonly usage: TokenUsage | undefined;
}): Promise<void> {
"use step";
@@ -41,7 +43,7 @@ export async function emitTerminalSessionFailureStep(input: {
await publishTerminalSessionEvent({
errorId: typeof details.errorId === "string" ? details.errorId : undefined,
- event: createSessionFailedEvent({ code, details, message, sessionId }),
+ event: createSessionFailedEvent({ code, details, message, sessionId, usage: input.usage }),
sessionWritable: input.sessionWritable,
serializedContext: input.serializedContext,
turnId: input.turnId,
diff --git a/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts b/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts
index 6c97ee831..b738f9f49 100644
--- a/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts
+++ b/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts
@@ -234,7 +234,9 @@ describe("workflow tools", () => {
// The answer resumes the same turn, which completes once.
const turnIds = new Set(
answered.flatMap((event) =>
- "data" in event && "turnId" in event.data ? [event.data.turnId] : [],
+ "data" in event && event.data !== undefined && "turnId" in event.data
+ ? [event.data.turnId]
+ : [],
),
);
expect([...turnIds]).toEqual([parked!.data.turnId]);
diff --git a/packages/eve/src/harness/cancelled-turn-emission.test.ts b/packages/eve/src/harness/cancelled-turn-emission.test.ts
index 9efb92bb9..936cd0ec5 100644
--- a/packages/eve/src/harness/cancelled-turn-emission.test.ts
+++ b/packages/eve/src/harness/cancelled-turn-emission.test.ts
@@ -3,14 +3,23 @@ import { describe, expect, it } from "vitest";
import { emitCancelledTurn } from "#harness/cancelled-turn-emission.js";
import type { UnstampedMessageStreamEvent } from "#protocol/message.js";
+const USAGE = {
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ costUsd: 0.5,
+ inputTokens: 120,
+ outputTokens: 30,
+};
+
describe("emitCancelledTurn", () => {
- it("emits turn.cancelled → session.waiting and no failure events", async () => {
+ it("emits turn.cancelled → session.waiting with the session's usage and no failure events", async () => {
const events: UnstampedMessageStreamEvent[] = [];
const next = await emitCancelledTurn(
async (event) => {
events.push(event);
},
{ sessionStarted: true, sequence: 3, stepIndex: 2, turnId: "turn_3" },
+ USAGE,
);
expect(events.map((event) => event.type)).toEqual(["turn.cancelled", "session.waiting"]);
@@ -19,7 +28,7 @@ describe("emitCancelledTurn", () => {
type: "turn.cancelled",
});
expect(events[1]).toEqual({
- data: { continuationToken: "", wait: "next-user-message" },
+ data: { continuationToken: "", usage: USAGE, wait: "next-user-message" },
type: "session.waiting",
});
expect(next).toEqual({ sessionStarted: true, sequence: 4, stepIndex: 0, turnId: "" });
@@ -36,6 +45,7 @@ describe("emitCancelledTurn", () => {
events.push(event);
},
{ sessionStarted: false, sequence: 0, stepIndex: 0, turnId: "" },
+ USAGE,
);
expect(events[0]).toMatchObject({
diff --git a/packages/eve/src/harness/cancelled-turn-emission.ts b/packages/eve/src/harness/cancelled-turn-emission.ts
index 948779cb9..26e8f6b58 100644
--- a/packages/eve/src/harness/cancelled-turn-emission.ts
+++ b/packages/eve/src/harness/cancelled-turn-emission.ts
@@ -1,5 +1,6 @@
import { createSessionWaitingEvent, createTurnCancelledEvent } from "#protocol/message.js";
import type { HarnessEmitFn } from "#harness/types.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import { activeTurnId } from "#harness/active-turn-id.js";
import type { HarnessEmissionState } from "#harness/emission.js";
@@ -15,6 +16,7 @@ import type { HarnessEmissionState } from "#harness/emission.js";
export async function emitCancelledTurn(
emitFn: HarnessEmitFn,
state: HarnessEmissionState,
+ usage: TokenUsage,
): Promise<HarnessEmissionState> {
await emitFn(
createTurnCancelledEvent({
@@ -22,7 +24,7 @@ export async function emitCancelledTurn(
turnId: activeTurnId(state),
}),
);
- await emitFn(createSessionWaitingEvent());
+ await emitFn(createSessionWaitingEvent(usage));
return {
sessionStarted: true,
diff --git a/packages/eve/src/harness/emission.test.ts b/packages/eve/src/harness/emission.test.ts
index 54f79dfb8..5d1aad5db 100644
--- a/packages/eve/src/harness/emission.test.ts
+++ b/packages/eve/src/harness/emission.test.ts
@@ -54,7 +54,12 @@ describe("emitTurnEpilogue", () => {
const emit = createEmitStub();
const history = [{ content: "settled reply", role: "assistant" as const }];
- await emitTurnEpilogue(emit, EMISSION_STATE, history);
+ await emitTurnEpilogue(emit, EMISSION_STATE, history, {
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ inputTokens: 0,
+ outputTokens: 0,
+ });
expect(emit).toHaveBeenNthCalledWith(
1,
diff --git a/packages/eve/src/harness/emission.ts b/packages/eve/src/harness/emission.ts
index 860b8a4dd..c63bfff59 100644
--- a/packages/eve/src/harness/emission.ts
+++ b/packages/eve/src/harness/emission.ts
@@ -35,6 +35,7 @@ import {
createTurnStartedEvent,
} from "#protocol/message.js";
import type { JsonObject } from "#shared/json.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import {
createRuntimeToolResultFromStepResult,
createRuntimeToolResultFromToolError,
@@ -183,7 +184,7 @@ async function emitStepAndTurnFailed(
export async function emitFailedStep(
emitFn: HarnessEmitFn,
state: HarnessEmissionState,
- input: FailedStepPayload & { readonly sessionId: string },
+ input: FailedStepPayload & { readonly sessionId: string; readonly usage: TokenUsage },
): Promise<void> {
await emitStepAndTurnFailed(emitFn, state, input);
await emitFn(createSessionFailedEvent(input));
@@ -196,10 +197,10 @@ export async function emitFailedStep(
export async function emitRecoverableFailedTurn(
emitFn: HarnessEmitFn,
state: HarnessEmissionState,
- input: FailedStepPayload & { readonly continuationToken: string },
+ input: FailedStepPayload & { readonly continuationToken: string; readonly usage: TokenUsage },
): Promise<HarnessEmissionState> {
await emitStepAndTurnFailed(emitFn, state, input);
- await emitFn(createSessionWaitingEvent());
+ await emitFn(createSessionWaitingEvent(input.usage));
return {
sessionStarted: state.sessionStarted,
@@ -220,13 +221,14 @@ export function advanceStep(state: HarnessEmissionState): HarnessEmissionState {
}
/**
- * Emits `turn.completed` and `session.waiting`.
+ * Emits `turn.completed` and `session.waiting` with the session's `usage`.
* Returns updated emission state with an incremented sequence.
*/
export async function emitTurnEpilogue(
emitFn: HarnessEmitFn,
state: HarnessEmissionState,
messages: readonly ModelMessage[],
+ usage: TokenUsage,
): Promise<HarnessEmissionState> {
await emitFn(
createTurnCompletedEvent({
@@ -235,7 +237,7 @@ export async function emitTurnEpilogue(
}),
messages,
);
- await emitFn(createSessionWaitingEvent());
+ await emitFn(createSessionWaitingEvent(usage));
return {
sessionStarted: state.sessionStarted,
diff --git a/packages/eve/src/harness/issue-3494-diagnosis.integration.test.ts b/packages/eve/src/harness/issue-3494-diagnosis.integration.test.ts
index 43d98ba7f..09ec9e76e 100644
--- a/packages/eve/src/harness/issue-3494-diagnosis.integration.test.ts
+++ b/packages/eve/src/harness/issue-3494-diagnosis.integration.test.ts
@@ -142,7 +142,11 @@ it("continues the second step past an older approval batch", async () => {
emittedBySecondStep: events.slice(before).map((event) => event.type),
firstTurnEvents: events
.filter(
- (event) => "data" in event && "turnId" in event.data && event.data.turnId === "turn_1",
+ (event) =>
+ "data" in event &&
+ event.data !== undefined &&
+ "turnId" in event.data &&
+ event.data.turnId === "turn_1",
)
.map((event) => event.type),
};
diff --git a/packages/eve/src/harness/session-limit-enforcement.ts b/packages/eve/src/harness/session-limit-enforcement.ts
index 88aac4b96..83a458fc9 100644
--- a/packages/eve/src/harness/session-limit-enforcement.ts
+++ b/packages/eve/src/harness/session-limit-enforcement.ts
@@ -26,6 +26,7 @@ import {
bumpSessionRuntimeUsageLimits,
getSessionUsageLimitViolation,
getSessionTokenUsage,
+ getSessionUsage,
type SessionUsageLimitViolation,
} from "#harness/turn-tag-state.js";
import type { HarnessSession, StepResult, ToolLoopHarnessConfig } from "#harness/types.js";
@@ -143,7 +144,12 @@ async function parkOnSessionUsageLimit(input: {
}),
);
- emissionState = await emitTurnEpilogue(input.emit, emissionState, parkedSession.history);
+ emissionState = await emitTurnEpilogue(
+ input.emit,
+ emissionState,
+ parkedSession.history,
+ getSessionUsage(parkedSession),
+ );
return {
next: null,
@@ -195,6 +201,7 @@ async function failSessionUsageLimit(input: {
details,
message,
sessionId: input.session.sessionId,
+ usage: getSessionUsage(input.session),
});
}
diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts
index 88da0f7e0..55b5992ce 100644
--- a/packages/eve/src/harness/tool-loop.test.ts
+++ b/packages/eve/src/harness/tool-loop.test.ts
@@ -75,6 +75,7 @@ import { PendingSkillAnnouncementKey } from "#context/dynamic-skill-lifecycle.js
import { deserializeContext, serializeContext } from "#context/serialize.js";
import { stashToolInterrupt } from "#harness/tool-interrupts.js";
import { appendMissingToolResultMessages, createToolLoopHarness } from "#harness/tool-loop.js";
+import { countRunUsage } from "#execution/agent-sessions/usage.js";
import { createTask, writeTaskTable } from "#execution/tasks/table.js";
import { SessionLimitDeclinedError, TurnCancelledError } from "#harness/turn-cancellation.js";
import {
@@ -420,6 +421,28 @@ function setDelegatedParent(ctx: ContextContainer): void {
});
}
+const DELEGATED_SPEND = {
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ costUsd: 0.25,
+ inputTokens: 100,
+ outputTokens: 20,
+};
+
+/** A session whose delegated agents already spent {@link DELEGATED_SPEND}. */
+function withDelegatedSpend(overrides?: Parameters<typeof createTestSession>[0]): HarnessSession {
+ return countRunUsage(createTestSession(overrides), DELEGATED_SPEND);
+}
+
+function lastSessionEvent<T extends "session.waiting" | "session.failed">(
+ events: readonly UnstampedMessageStreamEvent[],
+ type: T,
+): Extract<UnstampedMessageStreamEvent, { type: T }> | undefined {
+ return events.findLast(
+ (event): event is Extract<UnstampedMessageStreamEvent, { type: T }> => event.type === type,
+ );
+}
+
function createEventCollector(): {
emit: HarnessEmitFn;
events: UnstampedMessageStreamEvent[];
@@ -1712,6 +1735,81 @@ describe("createToolLoopHarness", () => {
});
});
+ it("reports the session's running usage, delegated spend included, when the turn ends", async () => {
+ setupMockAgent({
+ finishReason: "stop",
+ providerMetadata: { gateway: { cost: "0.125" } },
+ response: { messages: [{ content: "Hello!", role: "assistant" }] },
+ text: "Hello!",
+ toolCalls: [],
+ toolResults: [],
+ usage: {
+ inputTokenDetails: { cacheReadTokens: 2, cacheWriteTokens: 1 },
+ inputTokens: 7,
+ outputTokens: 3,
+ },
+ });
+ const { emit, events } = createEventCollector();
+
+ await createToolLoopHarness(createTestConfig(emit))(withDelegatedSpend(), { message: "Hi" });
+
+ expect(lastSessionEvent(events, "session.waiting")?.data.usage).toEqual({
+ cacheReadTokens: 2,
+ cacheWriteTokens: 1,
+ costUsd: 0.375,
+ inputTokens: 107,
+ outputTokens: 23,
+ });
+ });
+
+ it("reports no session cost on session.waiting when no model call reported one", async () => {
+ setupMockAgent({
+ finishReason: "stop",
+ response: { messages: [{ content: "Hello!", role: "assistant" }] },
+ text: "Hello!",
+ toolCalls: [],
+ toolResults: [],
+ usage: { inputTokens: 7, outputTokens: 3 },
+ });
+ const { emit, events } = createEventCollector();
+
+ await createToolLoopHarness(createTestConfig(emit))(createTestSession(), { message: "Hi" });
+
+ expect(lastSessionEvent(events, "session.waiting")?.data.usage).toEqual({
+ cacheReadTokens: 0,
+ cacheWriteTokens: 0,
+ costUsd: undefined,
+ inputTokens: 7,
+ outputTokens: 3,
+ });
+ });
+
+ it("reports the session's usage on session.waiting when a failed model call parks the turn", async () => {
+ setupMockAgentError(new Error("Model blew up"));
+ const { emit, events } = createEventCollector();
+
+ await createToolLoopHarness(createTestConfig(emit))(withDelegatedSpend(), { message: "Hi" });
+
+ expect(lastSessionEvent(events, "session.waiting")?.data.usage).toEqual(DELEGATED_SPEND);
+ });
+
+ it("reports the session's usage on session.failed when model selection fails", async () => {
+ const emit: HarnessEmitFn = async (event) => {
+ events.push(event);
+ if (event.type === "turn.started") {
+ throw new DynamicModelSelectionError(new Error("flag service unavailable"));
+ }
+ };
+ const events: UnstampedMessageStreamEvent[] = [];
+
+ await createToolLoopHarness(createTestConfig(emit))(
+ withDelegatedSpend({ outputSchema: { type: "object" } }),
+ { message: "Hi" },
+ );
+
+ expect(lastSessionEvent(events, "session.failed")?.data.usage).toEqual(DELEGATED_SPEND);
+ });
+
it.each([
{
details: {
@@ -4674,7 +4772,7 @@ describe("createToolLoopHarness", () => {
);
});
- it("parks a delegated turn held by its working tasks with turn.waiting", async () => {
+ it("parks a delegated turn held by its working tasks with turn.waiting and the session's usage", async () => {
setupMockAgent({
finishReason: "stop",
response: { messages: [{ content: "Checking now.", role: "assistant" }] },
@@ -4692,7 +4790,7 @@ describe("createToolLoopHarness", () => {
);
const result = await contextStorage.run(ctx, () =>
- runStep(writeTaskTable(createTestSession(), table), { message: "Delegated turn" }),
+ runStep(writeTaskTable(withDelegatedSpend(), table), { message: "Delegated turn" }),
);
expect(result.held).toBeDefined();
@@ -4703,7 +4801,7 @@ describe("createToolLoopHarness", () => {
}),
);
expect(events.at(-1)).toEqual({
- data: { sequence: 0, turnId: "turn_0" },
+ data: { sequence: 0, turnId: "turn_0", usage: DELEGATED_SPEND },
type: "turn.waiting",
});
expect(events.map((event) => event.type)).not.toContain("session.waiting");
diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts
index 0b64bfb52..98a38d382 100644
--- a/packages/eve/src/harness/tool-loop.ts
+++ b/packages/eve/src/harness/tool-loop.ts
@@ -87,10 +87,12 @@ import { collectDeferredCalls, dispatchApprovedWorkflowCalls } from "#harness/wo
import { estimateTokens } from "#harness/token-estimate.js";
import {
accumulateTurnUsage,
+ getSessionUsage,
getTurnUsageState,
setTurnUsageState,
type TokenUsageDelta,
} from "#harness/turn-tag-state.js";
+import type { TokenUsage } from "#shared/token-usage.js";
import {
applySessionLimitContinuation,
enforceSessionUsageLimit,
@@ -554,6 +556,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
details: { errorId },
message,
sessionId: session.sessionId,
+ usage: getSessionUsage(session),
});
return {
@@ -594,7 +597,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
turnId: activeTurnId(emissionState),
}),
);
- await emit?.(createSessionWaitingEvent());
+ await emit?.(createSessionWaitingEvent(getSessionUsage(session)));
return { next: null, session };
}
@@ -632,7 +635,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
}
}
- await emit?.(createSessionWaitingEvent());
+ await emit?.(createSessionWaitingEvent(getSessionUsage(session)));
return { next: null, session };
}
@@ -934,7 +937,12 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
]),
state: memoryCommit?.state ?? parkedSession.state,
};
- emissionState = await emitTurnEpilogue(emit, emissionState, parkedSession.history);
+ emissionState = await emitTurnEpilogue(
+ emit,
+ emissionState,
+ parkedSession.history,
+ getSessionUsage(parkedSession),
+ );
return {
next: null,
session: setHarnessEmissionState(parkedSession, emissionState),
@@ -943,7 +951,12 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
if (resolvedCoordination.outcome === "resolved") {
if (emit) {
- emissionState = await emitTurnEpilogue(emit, emissionState, parkedSession.history);
+ emissionState = await emitTurnEpilogue(
+ emit,
+ emissionState,
+ parkedSession.history,
+ getSessionUsage(parkedSession),
+ );
parkedSession = setHarnessEmissionState(parkedSession, emissionState);
}
return { next: null, session: parkedSession };
@@ -954,7 +967,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
isHarnessBetweenTurns(pending.session) &&
getPendingAuthorization(pending.session.state) === undefined
) {
- await emit?.(createSessionWaitingEvent());
+ await emit?.(createSessionWaitingEvent(getSessionUsage(pending.session)));
}
return { next: null, session: pending.session };
}
@@ -1858,6 +1871,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
emissionState = await emitRecoverableFailedTurn(emit, emissionState, {
code: "WORKFLOW_STREAM_WRITE_FAILED",
continuationToken: session.continuationToken,
+ usage: getSessionUsage(session),
details: { ...streamWriteDetails, errorId },
message: toErrorMessage(finalError),
});
@@ -1918,6 +1932,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
details,
message: errorMessage,
sessionId: session.sessionId,
+ usage: getSessionUsage(session),
});
// Delegated runs need a failed terminal result so their caller sees a
// failed `subagent-result`; top-level conversation failures already
@@ -1937,6 +1952,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
emissionState = await emitRecoverableFailedTurn(emit, emissionState, {
code: "MODEL_CALL_FAILED",
continuationToken: session.continuationToken,
+ usage: getSessionUsage(session),
details,
message: errorMessage,
});
@@ -2744,7 +2760,12 @@ async function handleStepResult(input: {
}),
);
- emissionState = await emitTurnEpilogue(emit, emissionState, parkedSession.history);
+ emissionState = await emitTurnEpilogue(
+ emit,
+ emissionState,
+ parkedSession.history,
+ getSessionUsage(parkedSession),
+ );
parkedSession = setHarnessEmissionState(parkedSession, emissionState);
}
@@ -2796,7 +2817,12 @@ async function handleStepResult(input: {
// above: the session keeps serving ordinary turns while the challenge
// is open, so the stream must close its turn boundary — clients wait
// on `session.waiting` and would otherwise hang on the parked turn.
- emissionState = await emitTurnEpilogue(emit, emissionState, authorizationHistory);
+ emissionState = await emitTurnEpilogue(
+ emit,
+ emissionState,
+ authorizationHistory,
+ getSessionUsage(baseSession),
+ );
}
return {
@@ -2862,6 +2888,7 @@ async function handleStepResult(input: {
createTurnWaitingEvent({
sequence: emissionState.sequence,
turnId: emissionState.turnId,
+ usage: getSessionUsage(nextSession),
}),
);
}
@@ -3006,6 +3033,7 @@ async function emitStructuredResult(
emissionState: ReturnType<typeof getHarnessEmissionState>,
structured: JsonValue,
history: readonly HarnessModelMessage[],
+ usage: TokenUsage,
): Promise<ReturnType<typeof getHarnessEmissionState>> {
await emit(
createResultCompletedEvent({
@@ -3015,7 +3043,7 @@ async function emitStructuredResult(
turnId: emissionState.turnId,
}),
);
- return emitTurnEpilogue(emit, emissionState, history);
+ return emitTurnEpilogue(emit, emissionState, history, usage);
}
/**
@@ -3038,7 +3066,12 @@ async function finishTurn(input: {
if (schema === undefined) {
if (emit) {
- emissionState = await emitTurnEpilogue(emit, emissionState, session.history);
+ emissionState = await emitTurnEpilogue(
+ emit,
+ emissionState,
+ session.history,
+ getSessionUsage(session),
+ );
session = setHarnessEmissionState(session, emissionState);
}
const settledTurn = { output: stepOutput ?? "" } satisfies SettledTurn;
@@ -3054,6 +3087,7 @@ async function finishTurn(input: {
emissionState = await emitRecoverableFailedTurn(emit, emissionState, {
...OUTPUT_SCHEMA_NOT_FULFILLED,
continuationToken: session.continuationToken,
+ usage: getSessionUsage(session),
});
session = setHarnessEmissionState(session, emissionState);
}
@@ -3066,7 +3100,13 @@ async function finishTurn(input: {
session = persistStructuredAssistantTurn(session, history, structured);
if (emit) {
- emissionState = await emitStructuredResult(emit, emissionState, structured, session.history);
+ emissionState = await emitStructuredResult(
+ emit,
+ emissionState,
+ structured,
+ session.history,
+ getSessionUsage(session),
+ );
session = setHarnessEmissionState(session, emissionState);
}
const settledTurn = { output: structured } satisfies SettledTurn;
diff --git a/packages/eve/src/harness/turn-tag-state.ts b/packages/eve/src/harness/turn-tag-state.ts
index 0002738f5..8444c228e 100644
--- a/packages/eve/src/harness/turn-tag-state.ts
+++ b/packages/eve/src/harness/turn-tag-state.ts
@@ -78,6 +78,11 @@ export function getSessionTokenUsage(session: Pick<HarnessSession, "state">): To
return getTurnUsageState(session.state)?.session ?? ZERO_TOKEN_USAGE;
}
+/** The session's usage so far, delegated spend included. */
+export function getSessionUsage(session: Pick<HarnessSession, "state">): TokenUsage {
+ return toUsage(getSessionTokenUsage(session));
+}
+
/** Projects a {@link TokenUsageTotals} down to the cross-cutting {@link TokenUsage} shape. */
export function toUsage(totals: TokenUsageTotals): TokenUsage {
return {
diff --git a/packages/eve/src/instrumentation/native-events.test.ts b/packages/eve/src/instrumentation/native-events.test.ts
index 81b16b7b4..c72373cd4 100644
--- a/packages/eve/src/instrumentation/native-events.test.ts
+++ b/packages/eve/src/instrumentation/native-events.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { describe, expect, it } from "vitest";
import { ContextContainer, contextStorage } from "#context/container.js";
@@ -119,7 +120,7 @@ describe("createInstrumentationHandleEvent", () => {
}),
);
await handleEvent(createTurnCompletedEvent({ sequence: 0, turnId: "turn-1" }));
- await handleEvent(createSessionWaitingEvent());
+ await handleEvent(createSessionWaitingEvent(TEST_USAGE));
expect(order).toEqual([
"durable:session.started",
@@ -177,7 +178,7 @@ describe("createInstrumentationHandleEvent", () => {
turnId: "turn-1",
})!;
- await handleEvent(createSessionWaitingEvent());
+ await handleEvent(createSessionWaitingEvent(TEST_USAGE));
expect(events).toEqual([
{
diff --git a/packages/eve/src/internal/testing/events.ts b/packages/eve/src/internal/testing/events.ts
index 5314c916d..31ae2b272 100644
--- a/packages/eve/src/internal/testing/events.ts
+++ b/packages/eve/src/internal/testing/events.ts
@@ -216,3 +216,12 @@ export function stampTestEvents(
): MessageStreamEvent[] {
return events.map((event, index) => stampTestEvent(event, index));
}
+
+/** Token usage a session reports in tests that build session events without caring about its value. */
+export const TEST_USAGE = {
+ cacheReadTokens: 800,
+ cacheWriteTokens: 0,
+ costUsd: 0.0042,
+ inputTokens: 1200,
+ outputTokens: 150,
+};
diff --git a/packages/eve/src/protocol/message.test.ts b/packages/eve/src/protocol/message.test.ts
index 00d01ae79..e8f6a0eba 100644
--- a/packages/eve/src/protocol/message.test.ts
+++ b/packages/eve/src/protocol/message.test.ts
@@ -15,7 +15,6 @@ import {
createMessageReceivedEvent,
createReasoningAppendedEvent,
createResultCompletedEvent,
- createSessionWaitingEvent,
createStepStartedEvent,
createAgentStartedEvent,
createTurnCancelledEvent,
@@ -301,16 +300,6 @@ describe("message stream protocol", () => {
});
});
- it("publishes the channel-local continuation token on session.waiting", () => {
- expect(createSessionWaitingEvent("slack:C1:T1")).toEqual({
- data: {
- continuationToken: "C1:T1",
- wait: "next-user-message",
- },
- type: "session.waiting",
- });
- });
-
it("creates turn.cancelled events", () => {
expect(createTurnCancelledEvent({ sequence: 2, turnId: "turn_2" })).toEqual({
data: { sequence: 2, turnId: "turn_2" },
diff --git a/packages/eve/src/protocol/message.ts b/packages/eve/src/protocol/message.ts
index 2895611f9..53e1f39e0 100644
--- a/packages/eve/src/protocol/message.ts
+++ b/packages/eve/src/protocol/message.ts
@@ -18,8 +18,8 @@ import type {
RuntimeToolResultActionResult,
} from "#shared/action-types.js";
import type { InputRequest, InputResponse } from "#shared/input.js";
-import { toChannelLocalContinuationToken } from "#shared/continuation-token.js";
import type { JsonObject, JsonValue } from "#shared/json.js";
+import type { TokenUsage } from "#shared/token-usage.js";
export const EVE_SESSION_ID_HEADER = "x-eve-session-id";
export const EVE_STREAM_FORMAT_HEADER = "x-eve-stream-format";
@@ -619,6 +619,12 @@ export interface TurnWaitingStreamEvent {
data: {
sequence: number;
turnId: string;
+ /**
+ * The session's token usage so far: its own model calls plus what the
+ * agents it delegated to spent. `costUsd` is absent when no model call
+ * reported a cost. Absent on events from eve versions before it was added.
+ */
+ usage?: TokenUsage;
};
type: "turn.waiting";
}
@@ -774,6 +780,12 @@ export interface SessionWaitingStreamEvent {
data: {
/** Channel-local continuation token, or the immutable session ID for an ID-only session. */
continuationToken: string;
+ /**
+ * The session's token usage so far: its own model calls plus what the
+ * agents it delegated to spent. `costUsd` is absent when no model call
+ * reported a cost. Absent on events from eve versions before it was added.
+ */
+ usage?: TokenUsage;
wait: "next-user-message";
};
type: "session.waiting";
@@ -788,6 +800,12 @@ export interface SessionFailedStreamEvent {
details?: JsonObject;
message: string;
sessionId: string;
+ /**
+ * The session's token usage so far: its own model calls plus what the
+ * agents it delegated to spent. `costUsd` is absent when no model call
+ * reported a cost. Absent on events from eve versions before it was added.
+ */
+ usage?: TokenUsage;
};
type: "session.failed";
}
@@ -796,6 +814,14 @@ export interface SessionFailedStreamEvent {
* Stream event emitted when the session completes successfully.
*/
export interface SessionCompletedStreamEvent {
+ /** Absent when the session's usage was unknown, and on events from eve versions before it was added. */
+ data?: {
+ /**
+ * The session's token usage: its own model calls plus what the agents it
+ * delegated to spent. `costUsd` is absent when no model call reported a cost.
+ */
+ usage: TokenUsage;
+ };
type: "session.completed";
}
@@ -1683,11 +1709,13 @@ export function createTurnCompletedEvent(input: {
export function createTurnWaitingEvent(input: {
readonly sequence: number;
readonly turnId: string;
+ readonly usage: TokenUsage | undefined;
}): TurnWaitingStreamEvent {
return {
data: {
sequence: input.sequence,
turnId: input.turnId,
+ ...(input.usage !== undefined && { usage: input.usage }),
},
type: "turn.waiting",
};
@@ -1796,11 +1824,12 @@ export function createCompactionCompletedEvent(input: {
* wait.
*/
export function createSessionWaitingEvent(
- namespacedContinuationToken: string = "",
+ usage: TokenUsage | undefined,
): SessionWaitingStreamEvent {
return {
data: {
- continuationToken: toChannelLocalContinuationToken(namespacedContinuationToken),
+ continuationToken: "",
+ ...(usage !== undefined && { usage }),
wait: "next-user-message",
},
type: "session.waiting",
@@ -1815,6 +1844,7 @@ export function createSessionFailedEvent(input: {
readonly details?: JsonObject;
readonly message: string;
readonly sessionId: string;
+ readonly usage: TokenUsage | undefined;
}): SessionFailedStreamEvent {
return {
data: {
@@ -1822,6 +1852,7 @@ export function createSessionFailedEvent(input: {
details: input.details,
message: input.message,
sessionId: input.sessionId,
+ ...(input.usage !== undefined && { usage: input.usage }),
},
type: "session.failed",
};
@@ -1830,8 +1861,12 @@ export function createSessionFailedEvent(input: {
/**
* Creates the `session.completed` event for one terminal session completion.
*/
-export function createSessionCompletedEvent(): SessionCompletedStreamEvent {
- return { type: "session.completed" };
+export function createSessionCompletedEvent(
+ usage: TokenUsage | undefined,
+): SessionCompletedStreamEvent {
+ return usage === undefined
+ ? { type: "session.completed" }
+ : { data: { usage }, type: "session.completed" };
}
/**
diff --git a/packages/eve/src/react/use-eve-agent.test.ts b/packages/eve/src/react/use-eve-agent.test.ts
index 75d2f7b93..c47e8fa69 100644
--- a/packages/eve/src/react/use-eve-agent.test.ts
+++ b/packages/eve/src/react/use-eve-agent.test.ts
@@ -19,7 +19,7 @@ import {
createTurnFailedEvent,
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
-import { stampTestEvents } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvents } from "#internal/testing/events.js";
import type { ClientSessionState } from "#client/types.js";
function createStartedMessageResponse(sessionId: string, continuationToken: string): Response {
@@ -195,7 +195,7 @@ describe("useEveAgent", () => {
const cancel = vi.fn();
const body = new ReadableStream<Uint8Array>({
start(controller) {
- const [waiting] = stampTestEvents([createSessionWaitingEvent()]);
+ const [waiting] = stampTestEvents([createSessionWaitingEvent(TEST_USAGE)]);
controller.enqueue(new TextEncoder().encode(`${JSON.stringify(waiting)}\n`));
},
cancel,
@@ -242,7 +242,7 @@ describe("useEveAgent", () => {
let creates = 0;
const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation(async (_request, init) => {
if (init?.method === "POST") return createStartedMessageResponse(`session_${++creates}`, "");
- return createEagerStreamResponse([createSessionWaitingEvent()]);
+ return createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]);
});
let agent: UseEveAgentHelpers<EveMessageData> | undefined;
function Chat({
@@ -293,7 +293,7 @@ describe("useEveAgent", () => {
const fetchMock = vi
.spyOn(globalThis, "fetch")
.mockReturnValueOnce(accepted.promise)
- .mockResolvedValueOnce(createEagerStreamResponse([createSessionWaitingEvent()]));
+ .mockResolvedValueOnce(createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]));
let agent: UseEveAgentHelpers<EveMessageData> | undefined;
function Chat({ prewarm }: { readonly prewarm: boolean }) {
agent = useEveAgent({ prewarm });
@@ -356,7 +356,7 @@ describe("useEveAgent", () => {
let creates = 0;
const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation(async (_request, init) => {
if (init?.method === "POST") return createStartedMessageResponse(`session_${++creates}`, "");
- return createEagerStreamResponse([createSessionWaitingEvent()]);
+ return createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]);
});
let agent: UseEveAgentHelpers<EveMessageData> | undefined;
let setPrewarm: ((value: boolean) => void) | undefined;
@@ -473,7 +473,7 @@ describe("useEveAgent", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const startResponse = createDeferred<Response>();
@@ -596,7 +596,7 @@ describe("useEveAgent", () => {
.mockImplementation(async (_request, init) =>
init?.method === "POST"
? await startResponse.promise
- : createEagerStreamResponse([createSessionWaitingEvent()]),
+ : createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]),
);
let randomWord = "jazz";
@@ -653,7 +653,7 @@ describe("useEveAgent", () => {
.mockImplementation(async (_request, init) =>
init?.method === "POST"
? await startResponse.promise
- : createEagerStreamResponse([createSessionWaitingEvent()]),
+ : createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]),
);
let helpers: UseEveAgentHelpers<EveMessageData> | undefined;
@@ -732,7 +732,7 @@ describe("useEveAgent", () => {
stepIndex: 0,
turnId: "turn_2",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
vi.spyOn(globalThis, "fetch")
@@ -812,6 +812,7 @@ describe("useEveAgent", () => {
turnId: "turn_1",
}),
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "MODEL_CALL_FAILED",
message: "Bad Request",
sessionId: "session_1",
@@ -884,7 +885,7 @@ describe("useEveAgent", () => {
sequence: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const startResponse = createDeferred<Response>();
@@ -948,7 +949,7 @@ describe("useEveAgent", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
const fetchMock = vi
.spyOn(globalThis, "fetch")
@@ -994,7 +995,7 @@ describe("useEveAgent", () => {
init?.method === "POST"
? await startResponse.promise
: await startResponse.promise.then(() =>
- createEagerStreamResponse([createSessionWaitingEvent()]),
+ createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]),
),
);
diff --git a/packages/eve/src/shared/token-usage.ts b/packages/eve/src/shared/token-usage.ts
index 0b5a32e70..28e61d0f1 100644
--- a/packages/eve/src/shared/token-usage.ts
+++ b/packages/eve/src/shared/token-usage.ts
@@ -14,3 +14,17 @@ const tokenUsageSchema = z.object({
export const tokenUsageWithCostSchema = tokenUsageSchema.extend({
costUsd: z.number().finite().nonnegative().optional(),
});
+
+/** Adds two usages; the cost is absent only when neither reported one. */
+export function addTokenUsage(a: TokenUsage, b: TokenUsage): TokenUsage {
+ return {
+ cacheReadTokens: a.cacheReadTokens + b.cacheReadTokens,
+ cacheWriteTokens: a.cacheWriteTokens + b.cacheWriteTokens,
+ costUsd:
+ a.costUsd === undefined && b.costUsd === undefined
+ ? undefined
+ : (a.costUsd ?? 0) + (b.costUsd ?? 0),
+ inputTokens: a.inputTokens + b.inputTokens,
+ outputTokens: a.outputTokens + b.outputTokens,
+ };
+}
diff --git a/packages/eve/src/subagents/hitl-proxy.integration.test.ts b/packages/eve/src/subagents/hitl-proxy.integration.test.ts
index a2118de9e..076a83296 100644
--- a/packages/eve/src/subagents/hitl-proxy.integration.test.ts
+++ b/packages/eve/src/subagents/hitl-proxy.integration.test.ts
@@ -267,7 +267,14 @@ describe("subagent HITL proxy → Slack-style text-approve regression (Finding #
// `turn.waiting`; the call that asked is still running, so the turn
// neither completes nor resets.
expect(events.slice(1)).toEqual([
- { data: { sequence: 3, turnId: "turn_3" }, type: "turn.waiting" },
+ {
+ data: {
+ sequence: 3,
+ turnId: "turn_3",
+ usage: { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 0, outputTokens: 0 },
+ },
+ type: "turn.waiting",
+ },
]);
expect(events[0]?.type).toBe("input.requested");
diff --git a/packages/eve/src/subagents/hitl-proxy.ts b/packages/eve/src/subagents/hitl-proxy.ts
index c46895f41..69c5e9036 100644
--- a/packages/eve/src/subagents/hitl-proxy.ts
+++ b/packages/eve/src/subagents/hitl-proxy.ts
@@ -1,4 +1,5 @@
import type { SessionInboxAddress } from "#execution/session-inbox/address.js";
+import { getSessionUsage } from "#harness/turn-tag-state.js";
import type {
DeliverPayload,
SubagentAuthorizationEventHookPayload,
@@ -69,7 +70,13 @@ export async function emitProxiedAuthorizationEvent(input: {
async function emitTurnWaiting(emit: HarnessEmitFn, session: HarnessSessionBase): Promise<void> {
const turn = getHarnessEmissionState(session.state);
- await emit(createTurnWaitingEvent({ sequence: turn.sequence, turnId: turn.turnId }));
+ await emit(
+ createTurnWaitingEvent({
+ sequence: turn.sequence,
+ turnId: turn.turnId,
+ usage: getSessionUsage(session),
+ }),
+ );
}
// ---------------------------------------------------------------------------
diff --git a/packages/eve/src/svelte/use-eve-agent.test.ts b/packages/eve/src/svelte/use-eve-agent.test.ts
index b41b0fd6a..1cd8eac52 100644
--- a/packages/eve/src/svelte/use-eve-agent.test.ts
+++ b/packages/eve/src/svelte/use-eve-agent.test.ts
@@ -12,7 +12,7 @@ import {
createSessionWaitingEvent,
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
-import { stampTestEvents } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvents } from "#internal/testing/events.js";
function createStartedMessageResponse(sessionId: string, continuationToken: string): Response {
return new Response(JSON.stringify({ continuationToken, ok: true, sessionId }), {
@@ -109,7 +109,7 @@ describe("useEveAgent (Svelte rune binding)", () => {
vi.stubGlobal("window", {});
const events = [
createMessageReceivedEvent({ message: "Hello", sequence: 0, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const fetchMock = vi
.spyOn(globalThis, "fetch")
@@ -133,7 +133,7 @@ describe("useEveAgent (Svelte rune binding)", () => {
vi.stubGlobal("window", {});
const events = [
createMessageReceivedEvent({ message: "Hello", sequence: 0, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
vi.spyOn(globalThis, "fetch")
.mockResolvedValueOnce(createStartedMessageResponse("session_1", "http:session_1"))
diff --git a/packages/eve/src/vue/use-eve-agent.test.ts b/packages/eve/src/vue/use-eve-agent.test.ts
index 070357c52..fb1487cc9 100644
--- a/packages/eve/src/vue/use-eve-agent.test.ts
+++ b/packages/eve/src/vue/use-eve-agent.test.ts
@@ -23,7 +23,7 @@ import {
createSessionFailedEvent,
type UnstampedMessageStreamEvent,
} from "#protocol/message.js";
-import { stampTestEvents } from "#internal/testing/events.js";
+import { TEST_USAGE, stampTestEvents } from "#internal/testing/events.js";
function createStartedMessageResponse(sessionId: string, continuationToken: string): Response {
return new Response(JSON.stringify({ continuationToken, ok: true, sessionId }), {
@@ -175,7 +175,7 @@ describe("EveAgentStore (Vue composable backing store)", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const startResponse = createDeferred<Response>();
@@ -257,6 +257,7 @@ describe("EveAgentStore (Vue composable backing store)", () => {
turnId: "turn_1",
}),
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "MODEL_CALL_FAILED",
message: "Bad Request",
sessionId: "session_1",
@@ -328,7 +329,7 @@ describe("EveAgentStore (Vue composable backing store)", () => {
vi.spyOn(globalThis, "fetch").mockImplementation(async (_request, init) =>
init?.method === "POST"
? await startResponse.promise
- : createEagerStreamResponse([createSessionWaitingEvent()]),
+ : createEagerStreamResponse([createSessionWaitingEvent(TEST_USAGE)]),
);
const store = createStore<readonly string[]>({
@@ -373,7 +374,7 @@ describe("useEveAgent (Vue composable wiring)", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
vi.spyOn(globalThis, "fetch")
.mockResolvedValueOnce(createBoundedStreamResponse(events))
@@ -411,7 +412,7 @@ describe("useEveAgent (Vue composable wiring)", () => {
stepIndex: 0,
turnId: "turn_1",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const startResponse = createDeferred<Response>();
@@ -452,7 +453,7 @@ describe("useEveAgent (Vue composable wiring)", () => {
.mockResolvedValueOnce(
createEagerStreamResponse([
createMessageReceivedEvent({ message: "After", sequence: 0, turnId: "turn_1" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]),
);
diff --git a/packages/eve/test/client.test.ts b/packages/eve/test/client.test.ts
index e8e4b00fd..b147b1477 100644
--- a/packages/eve/test/client.test.ts
+++ b/packages/eve/test/client.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { afterEach, describe, expect, expectTypeOf, it, vi } from "vitest";
import {
@@ -88,7 +89,7 @@ function createEagerStreamResponse(
): Response {
let turnId = "turn_001";
for (const event of events) {
- if ("data" in event && "turnId" in event.data) {
+ if ("data" in event && event.data !== undefined && "turnId" in event.data) {
turnId = event.data.turnId;
break;
}
@@ -134,7 +135,7 @@ function singleTurnEvents(input: {
turnId: input.turnId,
}),
createTurnCompletedEvent({ sequence: input.sequence, turnId: input.turnId }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
}
@@ -312,6 +313,7 @@ describe("Session.send (result)", () => {
const events: UnstampedMessageStreamEvent[] = [
createTurnStartedEvent({ sequence: 1, turnId: "turn_001" }),
createSessionFailedEvent({
+ usage: TEST_USAGE,
code: "internal_error",
message: "Something went wrong",
sessionId: "session_001",
@@ -340,7 +342,7 @@ describe("Session.send (result)", () => {
turnId: "turn_001",
}),
createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }),
- createSessionCompletedEvent(),
+ createSessionCompletedEvent(TEST_USAGE),
];
vi.spyOn(globalThis, "fetch")
@@ -380,7 +382,7 @@ describe("Session.send (result)", () => {
turnId: "turn_001",
}),
createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const fetchMock = vi
@@ -409,7 +411,7 @@ describe("Session.send (result)", () => {
stepIndex: 0,
turnId: "turn_001",
}),
- createSessionCompletedEvent(),
+ createSessionCompletedEvent(TEST_USAGE),
];
const secondEvents = singleTurnEvents({
message: "New conversation",
@@ -476,7 +478,7 @@ describe("Session.send (stream)", () => {
turnId: "turn_001",
}),
);
- stream.pushEvent(createSessionWaitingEvent());
+ stream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
await iterationPromise;
@@ -500,7 +502,7 @@ describe("Session.send (stream)", () => {
expect(res.sessionId).toBe("session_001");
setTimeout(() => {
- stream.pushEvent(createSessionWaitingEvent());
+ stream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
for await (const _ of res) {
@@ -544,7 +546,7 @@ describe("Session.send (reconnection)", () => {
turnId: "turn_001",
}),
createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const fetchMock = vi
@@ -714,7 +716,7 @@ describe("Session.stream", () => {
stepIndex: 0,
turnId: "turn_002",
}),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
];
const fetchMock = vi
diff --git a/packages/eve/test/dev-send-message.test.ts b/packages/eve/test/dev-send-message.test.ts
index e9da7d1ef..f13a5640d 100644
--- a/packages/eve/test/dev-send-message.test.ts
+++ b/packages/eve/test/dev-send-message.test.ts
@@ -1,3 +1,4 @@
+import { TEST_USAGE } from "#internal/testing/events.js";
import { getVercelOidcToken } from "#compiled/@vercel/oidc/index.js";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
@@ -136,7 +137,7 @@ describe("sendDevelopmentMessage", () => {
}),
);
firstStream.pushEvent(createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }));
- firstStream.pushEvent(createSessionWaitingEvent());
+ firstStream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
const first = await firstPromise;
@@ -169,7 +170,7 @@ describe("sendDevelopmentMessage", () => {
}),
);
secondStream.pushEvent(createTurnCompletedEvent({ sequence: 2, turnId: "turn_002" }));
- secondStream.pushEvent(createSessionWaitingEvent());
+ secondStream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
const second = await secondPromise;
@@ -221,7 +222,7 @@ describe("sendDevelopmentMessage", () => {
type: "turn.completed",
},
{
- data: { continuationToken: "", wait: "next-user-message" },
+ data: { continuationToken: "", usage: TEST_USAGE, wait: "next-user-message" },
type: "session.waiting",
},
]);
@@ -281,7 +282,7 @@ describe("sendDevelopmentMessage", () => {
turnId: "turn_002",
}),
createTurnCompletedEvent({ sequence: 2, turnId: "turn_002" }),
- createSessionWaitingEvent(),
+ createSessionWaitingEvent(TEST_USAGE),
]);
const fetchMock = vi
.spyOn(globalThis, "fetch")
@@ -322,7 +323,7 @@ describe("sendDevelopmentMessage", () => {
}),
);
firstStream.pushEvent(createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }));
- firstStream.pushEvent(createSessionWaitingEvent());
+ firstStream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
const first = await firstPromise;
@@ -416,7 +417,7 @@ describe("sendDevelopmentMessage", () => {
type: "turn.completed",
},
{
- data: { continuationToken: "", wait: "next-user-message" },
+ data: { continuationToken: "", usage: TEST_USAGE, wait: "next-user-message" },
type: "session.waiting",
},
]);
@@ -474,7 +475,7 @@ describe("sendDevelopmentMessage", () => {
}),
);
stream.pushEvent(createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }));
- stream.pushEvent(createSessionWaitingEvent());
+ stream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
const result = await messagePromise;
@@ -533,7 +534,7 @@ describe("sendDevelopmentMessage", () => {
}),
);
stream.pushEvent(createTurnCompletedEvent({ sequence: 1, turnId: "turn_001" }));
- stream.pushEvent(createSessionWaitingEvent());
+ stream.pushEvent(createSessionWaitingEvent(TEST_USAGE));
}, 0);
await messagePromise;