The Open Framework for Building Agents
refactor(eve): document task ownership, remove dead invocation tracing, test task ordering (#3981)
Andrew Barbae00593210cc2
diff --git a/.changeset/remove-nested-invocation-traces.md b/.changeset/remove-nested-invocation-traces.mdnew file mode 100644index 000000000..0ca3b5c4c--- /dev/null+++ b/.changeset/remove-nested-invocation-traces.md@@ -0,0 +1,5 @@+---+"eve": patch+---++Traces no longer give each `ctx.agent()` call in a workflow tool its own `agent.action` caller span, and a workflow tool that calls agents stays an `agent.action` span instead of becoming `invoke_workflow <tool>` with `gen_ai.workflow.name`. Sessions opened with `ctx.agent()` now link to the calling tool's `agent.action` span, and their usage no longer appears on a separate caller span; it still counts toward the parent session's usage.diff --git a/docs/reference/cli.md b/docs/reference/cli.mdindex ffb5d6586..6c7066b6e 100644--- a/docs/reference/cli.md+++ b/docs/reference/cli.md@@ -340,7 +340,7 @@ Span rows carry inline metrics when the span recorded them — `↑input`/`↓ouEvery subagent activation starts its own trace. The first child's `invoke_agent` root links to the dispatching caller with `eve.link.type=agent.dispatch`; remote agents preserve that caller in W3C `tracestate` even when HTTP `traceparent` advances through platform ingress. Later turns also start fresh traces without repeating the initial caller link. All related sessions retain the same `gen_ai.conversation.id`, and `agent.subagent.name` labels the child invocation.-Each `agent()` call inside an authored workflow has its own `agent.action` caller span, including sequential and parallel calls. A workflow tool that coordinates one of those calls has an enclosing `invoke_workflow <tool>` span; a workflow tool without agent calls remains `agent.action`. The tool also keeps its `execute_tool <tool>` span. Only agent execution uses `invoke_agent`.+Each workflow tool call has one `agent.action` span and keeps its `execute_tool <tool>` span. Sessions the tool opens with `ctx.agent` link to that call's `agent.action` span as their dispatching caller; individual `ctx.agent` calls get no span of their own. Only agent execution uses `invoke_agent`.Outbound MCP `tools/call` requests add MCP semantic attributes to the matching `execute_tool` span. When eve has no tool span to enrich, it creates a `CLIENT` `tools/call <tool>` span. Each `tools/list` discovery also has a `CLIENT` span. Configured OpenTelemetry propagation fields are injected into MCP `params._meta` for JSON-RPC bodies up to 1 MiB and propagation metadata up to 8 KiB; larger requests are sent unchanged. eve removes its audience and session-lineage baggage before forwarding, and keeps the input/output content-capture policy local.diff --git a/docs/tools/tasks.md b/docs/tools/tasks.mdindex 84c7db695..8409c7989 100644--- a/docs/tools/tasks.md+++ b/docs/tools/tasks.md@@ -287,6 +287,11 @@ task's start and end instead. Seenote cover every task in the session. Any later turn can continue an idle resumable task by its`taskId`, and that call runs with its own caller's auth. Only the turn's own caller steers it;anonymous callers share one identity.+ eve doesn't check which caller started a task: in a session with several people, such as a+ shared Slack thread, the model working for one person can continue or `task_cancel` a task+ another person started. A continued `serve` task keeps the state its body built for earlier+ calls, so [key per-caller data on the principal](/docs/tools/workflows#resumable-tasks-serve)+ and enforce per-person access inside the tool.- **Working tasks.** A session runs at most 32 working tasks at once. An idle resumable task doesn'tcount, and a call that makes it work again does.- **No time limits.** Tasks have no timeout of their own. The session lifetime,diff --git a/docs/tools/workflows.mdx b/docs/tools/workflows.mdxindex d5ddae045..f3d71b5a0 100644--- a/docs/tools/workflows.mdx+++ b/docs/tools/workflows.mdx@@ -337,6 +337,13 @@ described in [Tasks](/docs/tools/tasks). What differs:different caller than the one that started the task. `getToken`, `requireAuth`, and connectionsin a step use the current call's auth. `ctx.agents` lists theagents the current call may open, including dynamic agents selected in its turn.+- **Body state spans callers:** variables the body keeps between calls, such as `plan` in the+ example, may hold what an earlier call from another caller produced. Key per-caller data on+ the whole principal in `ctx.session.auth.current`: `authenticator`, `issuer`, `principalType`,+ and `principalId` together, because a `principalId` is unique only within its authenticator and+ issuer. Anonymous callers, whose `current` is `null` or whose `principalType` is `"anonymous"`,+ share one identity, so don't keep per-person data for them. Any caller in the session can+ continue or cancel the task; see [Tasks](/docs/tools/tasks#ownership-and-limits).- **Agents belong to the call that opened them:** a session the body opens with `ctx.agent` whileserving a call is that call's child. Its `session.parent` names the call's `callId` and turn, andits trace continues from the call's tool span. The session keeps that lineage for its laterdiff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.tsindex 42f749a42..b55c39fab 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts@@ -102,38 +102,6 @@ describe("buildConversationItems", () => {expect(items[2]?.result).toBe('{"temperatureF":72}');});- it.each([- { error: false, result: '{"deployed":true}', statusCode: 0 },- { error: true, result: undefined, statusCode: 2 },- ])("keeps $error workflow invocations as tool cards", ({ error, result, statusCode }) => {- const workflow = span(- "e".repeat(16),- "invoke_workflow deploy",- 2500,- 3250,- "b".repeat(16),- {- "agent.action.name": "deploy",- "gen_ai.operation.name": "invoke_workflow",- "gen_ai.tool.call.arguments": '{"service":"api"}',- "gen_ai.tool.call.result": result,- "gen_ai.workflow.name": "deploy",- },- statusCode,- );- const items = buildConversationItems(trace([...weatherTurn(), workflow]));- const item = items.find((candidate) => candidate.span.spanId === workflow.spanId);-- expect(item).toMatchObject({- args: '{"service":"api"}',- durationMs: 750,- error,- kind: "tool",- name: "deploy",- result,- });- });-it("renders provider-executed tool results as tool cards after the assistant", () => {// Provider-executed tools (e.g. web_search) never get an execute_tool// span; their outcomes live on the model span's tool_results attribute.diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.tsindex 22c3e2777..c97a61246 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts@@ -161,21 +161,14 @@ export function buildConversationItems(trace: LocalTrace): ConversationItem[] {}continue;}- if (- span.name === "agent.action" ||- stringAttribute(span, "gen_ai.operation.name") === "invoke_workflow"- ) {+ if (span.name === "agent.action") {entries.push({item: {kind: "tool",args: stringAttribute(span, "gen_ai.tool.call.arguments"),durationMs: spanDurationMs(span),error: span.statusCode === 2,- name: stripTerminalControls(- stringAttribute(span, "agent.action.name") ??- stringAttribute(span, "gen_ai.workflow.name") ??- "action",- ),+ name: stripTerminalControls(stringAttribute(span, "agent.action.name") ?? "action"),result: unwrapJsonString(stringAttribute(span, "gen_ai.tool.call.result")),span,subagent,diff --git a/packages/eve/src/execution/session/turn.test.ts b/packages/eve/src/execution/session/turn.test.tsindex 1ac9d6231..f7fdaeef9 100644--- a/packages/eve/src/execution/session/turn.test.ts+++ b/packages/eve/src/execution/session/turn.test.ts@@ -20,6 +20,7 @@ import {} from "#execution/tasks/table.js";import { getSessionTokenUsage } from "#harness/turn-tag-state.js";import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js";+import { interruptWorkflowToolRun } from "#execution/tools/workflow/interrupt.js";import type { TokenUsage } from "#shared/token-usage.js";vi.mock("#compiled/@workflow/core/index.js", async (importOriginal) => ({@@ -37,6 +38,9 @@ vi.mock("#execution/cancel-descendant-turns-step.js", () => ({vi.mock("#execution/route-child-delivery.js", () => ({routeDeliverToChildren: vi.fn(),}));+vi.mock("#execution/tools/workflow/interrupt.js", () => ({+ interruptWorkflowToolRun: vi.fn(),+}));vi.mock("#execution/session/turn-waiting-step.js", () => ({publishTurnWaitingStep: vi.fn(async ({ serializedContext, sessionState }) => ({serializedContext,@@ -727,6 +731,91 @@ describe("SessionExecution checkpoints", () => {expect(queue.pendingCount).toBe(0);});+ it("routes an answer to its question before the message beside it interrupts the waited call", async () => {+ const base = { ...state(""), hasProxyInputRequests: true };+ const sessionState: DurableSessionState = {+ ...base,+ snapshot: {+ session: registerWorkflowToolRun(base.snapshot.session, {+ address: { hookToken: "deploy-control", runId: "deploy-run" },+ callId: "deploy-call",+ origin: { stepIndex: 0, turnId: "turn_0" },+ toolName: "deploy",+ }),+ },+ };+ const correction: DeliverHookPayload = {+ kind: "deliver",+ payloads: [{ message: "Also include Bob's service." }],+ };+ const answerAndCorrection: DeliverHookPayload = {+ kind: "deliver",+ payloads: [+ { inputResponses: [{ requestId: "region", text: "us-east-1" }] },+ { message: "Also include Bob's service." },+ ],+ };+ const runtimePayloads: SessionInboxPayload[] = [answerAndCorrection, { kind: "cancel" }];+ const inbox: SessionInbox = {+ claimedTokens: [],+ claimSessionHook: vi.fn(),+ claimSessionHooks: vi.fn(),+ drain: vi.fn(() => []),+ hasPending: vi.fn(() => false),+ whenPending: () => new Promise<void>(() => {}),+ next: vi.fn(async () => runtimePayloads.shift()),+ onDelivery: vi.fn(() => () => {}),+ onAgentStarted: () => () => {},+ onInterrupt: vi.fn(() => () => {}),+ restore: vi.fn(),+ };+ vi.mocked(turnStep)+ .mockReset()+ .mockResolvedValue({+ action: "park",+ hasPendingAuthorization: false,+ hasPendingInputBatch: false,+ pendingCoordinationCallIds: ["deploy-call"],+ serializedContext: {},+ sessionState,+ });+ vi.mocked(dispatchCoordinationStep).mockReset().mockResolvedValue({+ results: [],+ serializedContext: {},+ sessionState,+ });+ const order: string[] = [];+ vi.mocked(routeDeliverToChildren).mockImplementation(async (input) => {+ order.push("route");+ return {+ kind: "continue",+ remainder: correction,+ serializedContext: input.serializedContext,+ sessionState: input.sessionState,+ };+ });+ vi.mocked(interruptWorkflowToolRun)+ .mockReset()+ .mockImplementation(async () => {+ order.push("interrupt");+ });++ await expect(+ createExecution({ inbox, sessionState }).runTurn({+ delivery: { kind: "deliver", payloads: [{ message: "Deploy the release." }] },+ }),+ ).resolves.toEqual({ cancelled: true, kind: "park" });++ expect(routeDeliverToChildren).toHaveBeenCalledWith(+ expect.objectContaining({ delivery: answerAndCorrection }),+ );+ expect(interruptWorkflowToolRun).toHaveBeenCalledWith({+ hookToken: "deploy-control",+ runId: "deploy-run",+ });+ expect(order).toEqual(["route", "interrupt"]);+ });+it("reports turn.waiting once when task_wait parks on a working task", async () => {const base = state("");const working = createTask(readTaskTable(undefined), {diff --git a/packages/eve/src/execution/tasks/session.test.ts b/packages/eve/src/execution/tasks/session.test.tsnew file mode 100644index 000000000..eb6abd169--- /dev/null+++ b/packages/eve/src/execution/tasks/session.test.ts@@ -0,0 +1,133 @@+import { beforeEach, describe, expect, it, vi } from "vitest";++import type { DurableSession } from "#execution/durable-session-store.js";+import type { SessionInbox } from "#execution/session-inbox/inbox.js";+import { SessionStateCursor } from "#execution/session/state-cursor.js";+import { answerTaskCancel } from "#execution/tasks/session.js";+import {+ createTask,+ markTaskRunStarted,+ readTaskTable,+ recordTaskRun,+ writeTaskTable,+ type TaskTable,+} from "#execution/tasks/table.js";+import {+ getProxyInputRequests,+ upsertProxyInputRequestState,+} from "#harness/proxy-input-requests.js";+import { createTestSessionState } from "#internal/testing/session-state.js";++// No workflow runtime runs here: the run's cancel hook is a stub, and the stream is recorded.+vi.mock("#internal/workflow/runtime.js", () => ({ resumeHook: vi.fn(async () => {}) }));+const { published } = vi.hoisted(() => ({ published: [] as unknown[] }));+vi.mock("#execution/publish-session-events.js", () => {+ const publish = async (+ target: { readonly serializedContext: Record<string, unknown>; readonly sessionState: unknown },+ events: readonly unknown[],+ ) => {+ published.push(...events);+ return { serializedContext: target.serializedContext, sessionState: target.sessionState };+ };+ return { publishSessionEvents: publish, relaySessionEvents: publish };+});++const REQUEST_EVENT = { sequence: 3, stepIndex: 1, turnId: "turn_1" };++beforeEach(() => {+ published.length = 0;+});++describe("answerTaskCancel", () => {+ it("withdraws the questions of the task() run it cancels, and only those", async () => {+ const research = startedTask(readTaskTable(undefined), "research");+ const summarize = startedTask(research.table, "summarize");+ let session: DurableSession = writeTaskTable(+ createTestSessionState({ sessionId: "session-1" }).snapshot.session,+ summarize.table,+ );+ session = withQuestion(session, "research-run");+ session = withQuestion(session, "summarize-run");+ const cursor = new SessionStateCursor({+ inbox: { claimSessionHooks: vi.fn() } as Partial<SessionInbox> as SessionInbox,+ serializedContext: {},+ sessionState: {+ ...createTestSessionState({ sessionId: "session-1" }),+ snapshot: { session },+ },+ sessionWritable: new WritableStream<Uint8Array>(),+ });++ await answerTaskCancel(cursor, {+ callId: "cancel-call",+ kind: "task_cancel",+ taskId: research.taskId,+ });++ const committed = cursor.sessionState.snapshot.session;+ expect([...getProxyInputRequests(committed.state).keys()]).toEqual(["summarize-run-ask-1"]);+ expect(published).toEqual([+ {+ data: {+ ...REQUEST_EVENT,+ resolutions: [+ { kind: "question", outcome: "cancelled", requestId: "research-run-ask-1" },+ ],+ },+ type: "input.resolved",+ },+ {+ data: {+ callId: "research-call",+ status: "cancelled",+ taskId: research.taskId,+ turnId: "turn_1",+ },+ type: "task.settled",+ },+ ]);+ });+});++/** A working `task()` task whose run has started. */+function startedTask(+ table: TaskTable,+ name: string,+): { readonly table: TaskTable; readonly taskId: string } {+ const created = createTask(table, {+ callId: `${name}-call`,+ kind: "tool",+ name,+ resumable: false,+ turnId: "turn_1",+ });+ const runId = `${name}-run`;+ const recorded = recordTaskRun(created.table, created.taskId, {+ hookToken: `${runId}-control`,+ runId,+ });+ return {+ table: markTaskRunStarted(recorded, created.taskId, runId).table,+ taskId: created.taskId,+ };+}++function withQuestion(session: DurableSession, runId: string): DurableSession {+ const requestId = `${runId}-ask-1`;+ const state = upsertProxyInputRequestState({+ entries: [+ [+ requestId,+ {+ childContinuationToken: requestId,+ event: REQUEST_EVENT,+ kind: "question",+ workflowAsk: { control: `${runId}-control`, question: {}, runId },+ },+ ],+ ],+ forChildContinuationToken: requestId,+ state: session.state,+ });+ return { ...session, state };+}diff --git a/packages/eve/src/execution/tools/workflow/ask.test.ts b/packages/eve/src/execution/tools/workflow/ask.test.tsindex eb7d8e27f..e577ecd46 100644--- a/packages/eve/src/execution/tools/workflow/ask.test.ts+++ b/packages/eve/src/execution/tools/workflow/ask.test.ts@@ -1,3 +1,4 @@+import { getEventListeners } from "node:events";import { describe, expect, it, vi } from "vitest";import {@@ -10,28 +11,51 @@ import type { ToolContext } from "#tools/definition.js";describe("ask", () => {it("resolves as unavailable without waiting when the session cannot request input", async () => {- const ctx = {} as ToolContext;- const send = vi.fn();- attachWorkflowToolRunContext(ctx, {- agentContext: { capabilities: { requestInput: false } } as AgentSessionContext,- asks: new WorkflowToolRunAsks("run"),- auth: { current: null, initiator: null },- control: "control",- from: {- callId: "call",- input: {},- runId: "run",- sequence: 1,- stepIndex: 0,- toolName: "ask_question",- turnId: "turn",- },- owner: { send, sent: 0 },- });+ const { ctx, send } = runContext({ requestInput: false });await expect(ask(ctx, { prompt: "Which region?" })).resolves.toEqual({status: "unavailable",});expect(send).not.toHaveBeenCalled();});++ it("removes its abort listeners once answered", async () => {+ const call = new AbortController();+ const caller = new AbortController();+ const { asks, ctx } = runContext({ abortSignal: call.signal, requestInput: true });++ const answer = ask(ctx, { prompt: "Which region?" }, { signal: caller.signal });+ asks.settle({+ kind: "answer",+ requestId: "run-ask-1",+ response: { status: "answered", text: "us-east-1" },+ });+ await expect(answer).resolves.toEqual({ status: "answered", text: "us-east-1" });++ expect(getEventListeners(call.signal, "abort")).toHaveLength(0);+ expect(getEventListeners(caller.signal, "abort")).toHaveLength(0);+ });});++function runContext(input: { readonly abortSignal?: AbortSignal; readonly requestInput: boolean }) {+ const ctx = { abortSignal: input.abortSignal ?? new AbortController().signal } as ToolContext;+ const asks = new WorkflowToolRunAsks("run");+ const send = vi.fn(async (_message: { readonly kind: string }) => {});+ attachWorkflowToolRunContext(ctx, {+ agentContext: { capabilities: { requestInput: input.requestInput } } as AgentSessionContext,+ asks,+ auth: { current: null, initiator: null },+ control: "control",+ from: {+ callId: "call",+ input: {},+ runId: "run",+ sequence: 1,+ stepIndex: 0,+ toolName: "ask_question",+ turnId: "turn",+ },+ owner: { send, sent: 0 },+ });+ return { asks, ctx, send };+}diff --git a/packages/eve/src/instrumentation/runtime.ts b/packages/eve/src/instrumentation/runtime.tsindex b13465fa9..1934580ac 100644--- a/packages/eve/src/instrumentation/runtime.ts+++ b/packages/eve/src/instrumentation/runtime.ts@@ -136,8 +136,6 @@ export interface BoundInstrumentationSession {/** Process-wide runtime consumed by every harness execution surface. */export interface InstrumentationRuntime {- /** Materializes settled invocation spans without draining the exporter pipeline. */- readonly flushSettledInvocations?: () => Promise<void>;readonly forceFlush: () => Promise<void>;readonly hooks: InstrumentationHooks;readonly idGenerator?: AgentSpanIdGenerator;diff --git a/packages/eve/src/internal/testing/in-memory-agent-trace-state-store.ts b/packages/eve/src/internal/testing/in-memory-agent-trace-state-store.tsindex 560e87c81..a259acc9f 100644--- a/packages/eve/src/internal/testing/in-memory-agent-trace-state-store.ts+++ b/packages/eve/src/internal/testing/in-memory-agent-trace-state-store.ts@@ -1,6 +1,5 @@import type {AgentActionTraceState,- AgentInvocationTraceState,AgentSessionTraceState,AgentTraceStateStore,AgentTurnTraceState,@@ -9,7 +8,6 @@ import type {export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {readonly #actionAnchors = new Map<string, AgentActionTraceState>();readonly #actions = new Map<string, AgentActionTraceState>();- readonly #invocations = new Map<string, AgentInvocationTraceState>();readonly #sessions = new Map<string, AgentSessionTraceState>();readonly #turns = new Map<string, AgentTurnTraceState>();@@ -31,18 +29,6 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {}}- deleteInvocation(idempotencyKey: string): void {- this.#invocations.delete(idempotencyKey);- }-- deleteInvocations(sessionId: string, turnId?: string): void {- for (const [key, state] of this.#invocations) {- if (state.sessionId === sessionId && (turnId === undefined || state.turnId === turnId)) {- this.#invocations.delete(key);- }- }- }-deleteSession(sessionId: string): void {this.#sessions.delete(sessionId);}@@ -68,19 +54,6 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {);}- findInvocations(- sessionId?: string,- turnId?: string,- parentActionCallId?: string,- ): readonly AgentInvocationTraceState[] {- return [...this.#invocations.values()].filter(- (state) =>- (sessionId === undefined || state.sessionId === sessionId) &&- (turnId === undefined || state.turnId === turnId) &&- (parentActionCallId === undefined || state.parentActionCallId === parentActionCallId),- );- }-getAction(idempotencyKey: string): AgentActionTraceState | undefined {return this.#actions.get(idempotencyKey);}@@ -101,10 +74,6 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {this.#actionAnchors.set(idempotencyKey, state);}- setInvocation(idempotencyKey: string, state: AgentInvocationTraceState): void {- this.#invocations.set(idempotencyKey, state);- }-setSession(sessionId: string, state: AgentSessionTraceState): void {this.#sessions.set(sessionId, state);}diff --git a/packages/eve/src/tracing/agent-action-instrumentation.ts b/packages/eve/src/tracing/agent-action-instrumentation.tsindex 9d6c28b92..f294d115a 100644--- a/packages/eve/src/tracing/agent-action-instrumentation.ts+++ b/packages/eve/src/tracing/agent-action-instrumentation.ts@@ -14,7 +14,6 @@ import type {InstrumentationActionTerminalEvent,InstrumentationAttemptScope,InstrumentationProviderDefinition,- InstrumentationSessionTransitionEvent,} from "#instrumentation/lifecycle.js";import { actionIdempotencyKey, attemptIdempotencyKey } from "#instrumentation/lifecycle.js";import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";@@ -22,16 +21,11 @@ import { contentAttribute, textContentAttribute } from "#tracing/agent-otel-contimport { setAgentUsage } from "#tracing/agent-otel-usage.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";-import type {- AgentActionTraceState,- AgentActionTraceTerminalState,- AgentInvocationTraceState,- AgentTraceStateStore,-} from "#tracing/agent-trace-state.js";+import type { AgentActionTraceState, AgentTraceStateStore } from "#tracing/agent-trace-state.js";import { normalizeChannelAudience } from "#shared/channel-audience.js";import { isSampledTrace } from "#tracing/sampled-trace.js";import { withChannelAudience } from "#tracing/channel-audience-context.js";-import { AGENT_SPAN_NAMES, workflowInvocationSpanName } from "#tracing/agent-span-contract.js";+import { AGENT_SPAN_NAMES } from "#tracing/agent-span-contract.js";import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";interface AgentActionInstrumentation {@@ -41,8 +35,6 @@ interface AgentActionInstrumentation {>;deleteForSession(sessionId: string): void | PromiseLike<void>;failForAttempt(scope: InstrumentationAttemptScope, error: unknown): Promise<void>;- flushSettledInvocations(): Promise<void>;- flushForSessionTransition(event: InstrumentationSessionTransitionEvent): Promise<void>;contextFor(sessionId: string,turnId: string,@@ -79,7 +71,6 @@ export function createAgentActionInstrumentation(input: {callId: event.callId,channelAudience: normalizeChannelAudience(event.scope.channelAudience),inputAttribute: input.recordInputs ? contentAttribute(event.input) : undefined,- isWorkflowTool: event.isWorkflowTool === true,kind: event.kind,name: event.name,parent: {@@ -116,11 +107,6 @@ export function createAgentActionInstrumentation(input: {const startSpan = (state: AgentActionTraceState): Span => {const invocation = isAgentInvocation(state.kind);- const workflowName = state.kind === "tool-call" ? state.workflowName : undefined;- const spanName =- workflowName === undefined- ? AGENT_SPAN_NAMES.action- : workflowInvocationSpanName(workflowName);const span = input.idGenerator.withSpanId(state.spanId, () =>input.tracer.startSpan(AGENT_SPAN_NAMES.action,@@ -134,20 +120,11 @@ export function createAgentActionInstrumentation(input: {"agent.step.attempt": state.attemptIndex,"agent.step.index": state.stepIndex,"agent.turn.id": state.turnId,- ...agentSpanNamingAttributes(- spanName,- workflowName === undefined ? undefined : "invoke_workflow",- ),+ ...agentSpanNamingAttributes(AGENT_SPAN_NAMES.action),...agentTraceIdentityAttributes({rootSessionId: state.rootSessionId,sessionId: state.sessionId,}),- ...(workflowName === undefined- ? undefined- : {- "gen_ai.operation.name": "invoke_workflow",- "gen_ai.workflow.name": workflowName,- }),...(invocation? {"gen_ai.agent.name": state.name,@@ -161,7 +138,6 @@ export function createAgentActionInstrumentation(input: {contextFromActionState(state),),);- if (workflowName !== undefined) updateSpanName(span, spanName);if (!invocation && state.inputAttribute !== undefined) {span.setAttribute("gen_ai.tool.call.arguments", state.inputAttribute);}@@ -179,7 +155,6 @@ export function createAgentActionInstrumentation(input: {async deleteForSession(sessionId) {await input.stateStore.deleteActions(sessionId);await input.stateStore.deleteActionAnchors(sessionId);- await input.stateStore.deleteInvocations(sessionId);},async failForAttempt(scope, error) {const keys = byAttempt.get(scope.attemptId);@@ -194,16 +169,6 @@ export function createAgentActionInstrumentation(input: {await input.stateStore.deleteAction(key);}},- flushSettledInvocations: () => flushInvocations(),- async flushForSessionTransition(event) {- const terminal =- event.type === "session.completed"- ? { outcome: "abandoned" as const }- : event.type === "session.failed"- ? { error: event.error, outcome: "failed" as const }- : undefined;- await flushInvocations(event.sessionId, terminal);- },events: {"action.completed": onTerminal,"action.failed": onTerminal,@@ -211,20 +176,6 @@ export function createAgentActionInstrumentation(input: {},};- async function flushInvocations(- sessionId?: string,- fallback?: AgentActionTraceTerminalState,- ): Promise<void> {- for (const invocation of await input.stateStore.findInvocations(sessionId)) {- const terminal = invocation.terminal ?? fallback;- if (terminal === undefined) continue;- finishInvocationSpan(invocation, terminal);- await input.stateStore.deleteInvocation(- actionIdempotencyKey(invocation.sessionId, invocation.turnId, invocation.callId),- );- }- }-function forget(idempotencyKey: string): void {for (const [attemptId, keys] of byAttempt) {keys.delete(idempotencyKey);@@ -256,24 +207,6 @@ export function createAgentActionInstrumentation(input: {}span.end(event.acceptedAtMs);}-- function finishInvocationSpan(- state: AgentInvocationTraceState,- terminal: AgentActionTraceTerminalState,- ): void {- const span = startSpan(state);- span.setAttribute("agent.action.outcome", terminal.outcome);- if (terminal.usage !== undefined) {- setAgentUsage(span, terminal.usage);- }- if (terminal.outcome === "failed") {- recordError(- span,- input.recordOutputs && state.recordOutputs === true ? terminal.error : undefined,- );- }- span.end(terminal.acceptedAtMs);- }}function actionContext(state: AgentActionTraceState): AgentActionContext {@@ -303,11 +236,6 @@ function isAgentInvocation(kind: InstrumentationActionKind): boolean {return kind === "subagent-call" || kind === "remote-agent-call";}-function updateSpanName(span: Span, name: string): void {- const updateName = Reflect.get(span, "updateName");- if (typeof updateName === "function") Reflect.apply(updateName, span, [name]);-}-function recordActionError(span: Span, error: unknown, errorType?: string): void {if (error instanceof Error || error === undefined) {recordError(span, error, errorType);diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.test.ts b/packages/eve/src/tracing/agent-invocation-coordinator.test.tsindex f3f7ca893..c0de53c7b 100644--- a/packages/eve/src/tracing/agent-invocation-coordinator.test.ts+++ b/packages/eve/src/tracing/agent-invocation-coordinator.test.ts@@ -2,15 +2,13 @@ import { describe, expect, it } from "vitest";import { ContextContainer, contextStorage } from "#context/container.js";import { SessionTraceSeedKey } from "#context/keys.js";-import { deserializeContext, serializeContext } from "#context/serialize.js";-import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";-import { settleAgentInvocationTrace } from "#tracing/agent-invocation-terminal.js";-import { deriveAgentActionSpanId } from "#tracing/agent-span-id-generator.js";+import { serializeContext } from "#context/serialize.js";+import { resolveToolCallAgentTrace } from "#tracing/agent-invocation-coordinator.js";import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";import type { ConversationContext } from "#shared/conversation-context.js";-const outerKey = actionIdempotencyKey("session-1", "turn-1", "workflow");+const actionKey = actionIdempotencyKey("session-1", "turn-1", "workflow");const conversation: ConversationContext = {audience: "private",channel: { kind: "http" },@@ -18,253 +16,54 @@ const conversation: ConversationContext = {principalType: "anonymous",};-const sessionState = {- "eve.workflowTool": {- version: 4,- runs: [- {- callId: "workflow",- toolName: "coordinate",- origin: { turnId: "turn-1", stepIndex: 0 },- address: { runId: "workflow-run", hookToken: "workflow-hook" },- },- ],- },-};--describe("agent invocation trace coordinator", () => {- it("keeps parallel invocation identities and caller contexts independent", async () => {- let serializedContext = await contextWithActionAnchor();-- const first = prepare(serializedContext, "workflow:first");- serializedContext = first.serializedContext;- const second = prepare(serializedContext, "workflow:second");- serializedContext = second.serializedContext;-- expect(first.dispatch.parentTraceContext?.spanId).toBe(- deriveAgentActionSpanId("session-1", "turn-1", "workflow:first"),- );- expect(second.dispatch.parentTraceContext?.spanId).toBe(- deriveAgentActionSpanId("session-1", "turn-1", "workflow:second"),- );-- const invocations = await readInvocations(serializedContext);- expect(invocations).toEqual([- expect.objectContaining({- callId: "workflow:first",- parentActionCallId: "workflow",- }),- expect.objectContaining({- callId: "workflow:second",- parentActionCallId: "workflow",- }),- ]);- await expect(readActionAnchor(serializedContext)).resolves.toMatchObject({- isWorkflowTool: true,- workflowName: "coordinate",- });- });-- it("replays one invocation with the same caller coordinates", async () => {- const serializedContext = await contextWithActionAnchor();-- const first = prepare(serializedContext, "workflow:first");- const replay = prepare(first.serializedContext, "workflow:first");-- expect(replay.dispatch).toEqual(first.dispatch);- await expect(readInvocations(replay.serializedContext)).resolves.toHaveLength(1);- expect(prepare(serializedContext, "workflow:first").serializedContext).toEqual(- first.serializedContext,- );- });-- it.each([0, 1])(- "falls back to the turn when a nested caller anchor is missing (%s)",- async (traceFlags) => {+describe("resolveToolCallAgentTrace", () => {+ it.each([true, false])(+ "parents to the call's action span, or its turn, capped to the live audience (recorded: %s)",+ async (recorded) => {const context = new ContextContainer();- const parent = { ...outerAction().parent, traceFlags };- contextStorage.run(context, () => {- new ContextAgentTraceStateStore().setTurn("session-1", "turn-1", {- context: parent,+ const action = workflowAction();+ context.set(SessionTraceSeedKey, {+ ...action.parent,+ decision: { action: "record", recordInputs: true, recordOutputs: true },+ forwardedTracePolicy: {+ originAudience: "public",+ ceiling: { recordInputs: true, recordOutputs: true },+ },+ });+ await contextStorage.run(context, () => {+ const store = new ContextAgentTraceStateStore();+ store.setTurn("session-1", "turn-1", {+ context: action.parent,rootSessionId: "session-1",sequence: 0,startTimeMs: 1,});+ if (recorded) store.setActionAnchor(actionKey, action);});- const prepared = prepare(serializeContext(context), "workflow:nested");- expect(prepared.dispatch.parentTraceContext).toMatchObject(parent);- await expect(readInvocations(prepared.serializedContext)).resolves.toEqual([]);- },- );-- it.each([- ["subagent-call", true],- ["subagent-call", false],- ["remote-agent-call", true],- ["remote-agent-call", false],- ] as const)("caps %s context with or without a recorded caller (%s)", async (kind, recorded) => {- const context = new ContextContainer();- const action = outerAction();- context.set(SessionTraceSeedKey, {- ...action.parent,- decision: { action: "record", recordInputs: true, recordOutputs: true },- forwardedTracePolicy: {- originAudience: "public",- ceiling: { recordInputs: true, recordOutputs: true },- },- });- await contextStorage.run(context, () => {- const store = new ContextAgentTraceStateStore();- store.setTurn("session-1", "turn-1", {- context: action.parent,- rootSessionId: "session-1",- sequence: 0,- startTimeMs: 1,- });- if (recorded) {- store.setAction(outerKey, action);- store.setActionAnchor(outerKey, action);- }- });- const serializedContext = await serializeContext(context);- const prepared = prepareAgentInvocationTrace({- conversation,- invocation: {+ const dispatch = resolveToolCallAgentTrace({callId: "workflow",- kind,- name: "research",- },- ownerId: "workflow-run",- startTimeMs: 2,- serializedContext,- sessionId: "session-1",- sessionState,- turnId: "turn-1",- });- expect(prepared.dispatch.parentTraceContext).toMatchObject({- ...action.parent,- spanId: recorded ? action.spanId : action.parent.spanId,- decision: { action: "record", recordInputs: false, recordOutputs: false },- });- expect(prepared.dispatch.originAudience).toBe("public");- const restored = await deserializeContext(prepared.serializedContext);- await contextStorage.run(restored, () => {- const store = new ContextAgentTraceStateStore();- expect(store.getAction(outerKey)?.kind).toBe(recorded ? kind : undefined);- expect(store.findActionAnchor("session-1", "turn-1", "workflow")?.kind).toBe(- recorded ? kind : undefined,- );- expect(store.findActionAnchor("session-1", "turn-1", "workflow")?.isWorkflowTool).toBe(- undefined,- );- expect(store.findActionAnchor("session-1", "turn-1", "workflow")?.workflowName).toBe(- undefined,- );- expect(store.findInvocations("session-1")).toHaveLength(0);- });- });-- it("does not infer an outer call from caller-chosen invocation IDs", async () => {- const prepared = prepareAgentInvocationTrace({- invocation: { callId: "workflow:unowned", kind: "subagent-call", name: "research" },- ownerId: "other-run",- startTimeMs: 2,- serializedContext: await contextWithActionAnchor(),- sessionId: "session-1",- sessionState,- turnId: "turn-1",- });- expect(prepared.dispatch.parentTraceContext).toBeUndefined();- await expect(readInvocations(prepared.serializedContext)).resolves.toEqual([]);- });-- it("records settlement on the matching invocation only", async () => {- const first = prepare(await contextWithActionAnchor(), "workflow:first");- const second = prepare(first.serializedContext, "workflow:second");- const serializedContext = settleAgentInvocationTrace({- acceptedAtMs: 3,- result: {- callId: "workflow:first",- kind: "subagent-result",- origin: "child",- outcome: {- kind: "parked",- result: { kind: "succeeded", output: "done" },- usageDelta: {- cacheReadTokens: 1,- cacheWriteTokens: 2,- costUsd: 0.012,- inputTokens: 3,- outputTokens: 4,- },- },- output: "done",- subagentName: "research",- },- serializedContext: second.serializedContext,- sessionId: "session-1",- });-- const invocations = await readInvocations(serializedContext);- expect(invocations.find((invocation) => invocation.callId === "workflow:first")).toMatchObject({- terminal: {- acceptedAtMs: 3,- outcome: "completed",- usage: { costUsd: 0.012, inputTokens: 3, outputTokens: 4 },- },- });- expect(- invocations.find((invocation) => invocation.callId === "workflow:second")?.terminal,- ).toBeUndefined();- });-});+ conversation,+ serializedContext: await serializeContext(context),+ sessionId: "session-1",+ turnId: "turn-1",+ });-function prepare(serializedContext: Record<string, unknown>, callId: string) {- return prepareAgentInvocationTrace({- conversation,- invocation: {- callId,- kind: "subagent-call",- name: "research",+ expect(dispatch.parentTraceContext).toMatchObject({+ ...action.parent,+ spanId: recorded ? action.spanId : action.parent.spanId,+ decision: { action: "record", recordInputs: false, recordOutputs: false },+ });+ expect(dispatch.originAudience).toBe("public");},- ownerId: "workflow-run",- startTimeMs: 2,- serializedContext,- sessionId: "session-1",- sessionState,- turnId: "turn-1",- });-}--async function contextWithActionAnchor(): Promise<Record<string, unknown>> {- const context = new ContextContainer();- await contextStorage.run(context, () => {- new ContextAgentTraceStateStore().setActionAnchor(outerKey, outerAction());- });- return await serializeContext(context);-}--async function readInvocations(serializedContext: Record<string, unknown>) {- const context = await deserializeContext(serializedContext);- return contextStorage.run(context, () =>- new ContextAgentTraceStateStore().findInvocations("session-1", "turn-1", "workflow"),- );-}--async function readActionAnchor(serializedContext: Record<string, unknown>) {- const context = await deserializeContext(serializedContext);- return contextStorage.run(context, () =>- new ContextAgentTraceStateStore().findActionAnchor("session-1", "turn-1", "workflow"),);-}+});-function outerAction() {+function workflowAction() {return {attemptIndex: 0,callId: "workflow",channelAudience: "private" as const,- isWorkflowTool: true,kind: "tool-call" as const,name: "coordinate",parent: {diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.ts b/packages/eve/src/tracing/agent-invocation-coordinator.tsindex 65d8e36b2..936891113 100644--- a/packages/eve/src/tracing/agent-invocation-coordinator.ts+++ b/packages/eve/src/tracing/agent-invocation-coordinator.ts@@ -1,26 +1,16 @@import type { SessionTraceContext } from "#channel/types.js";import { ConversationIdKey } from "#context/keys.js";import { readConversationId } from "#shared/conversation-identity.js";-import type { RuntimeSubagentResult } from "#shared/action-types.js";-import type { SessionStateMap } from "#harness/types.js";-import { getBlockingWorkflowToolRuns } from "#harness/workflow-tool-runs.js";-import { type ChannelAudience } from "#shared/channel-audience.js";+import type { ChannelAudience } from "#shared/channel-audience.js";import type { ConversationContext } from "#shared/conversation-context.js";import {applyLiveDeliveryAudienceCeiling,readForwardedTraceAssertion,} from "#shared/forwarded-trace-policy.js";-import { deriveAgentActionSpanId } from "#tracing/agent-span-id-generator.js";import {readActionTraceContext,readTurnTraceContext,- recordActionInvocationKind,- recordNestedAgentInvocation,} from "#tracing/agent-trace-context-store.js";-import {- invocationError,- recordNestedAgentInvocationTerminal,-} from "#tracing/agent-invocation-terminal.js";export interface AgentChildTraceDispatch {readonly conversationId?: string;@@ -28,102 +18,6 @@ export interface AgentChildTraceDispatch {readonly parentTraceContext?: SessionTraceContext;}-export function prepareAgentInvocationTrace(input: {- readonly conversation?: ConversationContext;- readonly invocation: {- readonly callId: string;- readonly kind: "remote-agent-call" | "subagent-call";- readonly name: string;- };- readonly ownerId: string;- readonly serializedContext: Record<string, unknown>;- readonly sessionId: string;- readonly sessionState?: SessionStateMap;- readonly startTimeMs: number;- readonly turnId: string;-}): {- readonly dispatch: AgentChildTraceDispatch;- fail(result: RuntimeSubagentResult): Record<string, unknown>;- readonly serializedContext: Record<string, unknown>;-} {- const conversationId = readConversationId(input.serializedContext[ConversationIdKey.name]);- const parentActionCallId = getBlockingWorkflowToolRuns(input.sessionState).find(- (run) => run.address.runId === input.ownerId,- )?.callId;- const { turnId } = input;- const parentTurnContext = readTurnTraceContext(input.serializedContext, input.sessionId, turnId);- const conversation = input.conversation;- const liveAudience = conversation?.audience ?? "unknown";- const environment = conversation?.environment ?? "production";- const outerTrace =- (parentActionCallId === undefined- ? undefined- : readActionTraceContext(- input.serializedContext,- input.sessionId,- turnId,- parentActionCallId,- )) ?? parentTurnContext;- const outputDecision =- outerTrace?.decision === undefined- ? undefined- : applyLiveDeliveryAudienceCeiling(- outerTrace.decision,- liveAudience,- outerTrace.forwardedTracePolicy,- environment,- );- const serializedContext =- parentActionCallId === undefined- ? input.serializedContext- : parentActionCallId === input.invocation.callId- ? recordActionInvocationKind({- callId: input.invocation.callId,- kind: input.invocation.kind,- serializedContext: input.serializedContext,- sessionId: input.sessionId,- turnId,- })- : recordNestedAgentInvocation({- callId: input.invocation.callId,- kind: input.invocation.kind,- name: input.invocation.name,- outerCallId: parentActionCallId,- recordOutputs: outputDecision?.action === "record" && outputDecision.recordOutputs,- serializedContext: input.serializedContext,- sessionId: input.sessionId,- spanId: deriveAgentActionSpanId(input.sessionId, turnId, input.invocation.callId),- startTimeMs: input.startTimeMs,- turnId,- });- const callerTraceContext = readActionTraceContext(- serializedContext,- input.sessionId,- turnId,- input.invocation.callId,- );- return {- dispatch: toChildTraceDispatch({- conversation,- conversationId,- stored: callerTraceContext ?? outerTrace,- }),- fail: (result) =>- recordNestedAgentInvocationTerminal({- callId: input.invocation.callId,- serializedContext,- sessionId: input.sessionId,- terminal: {- acceptedAtMs: Date.now(),- error: invocationError(result.output),- outcome: "failed",- },- turnId,- }),- serializedContext,- };-}-/*** The trace dispatch for sessions opened with `ctx.agent` while a run serves* one workflow tool call: the call's tool span in the calling turn, read fromdiff --git a/packages/eve/src/tracing/agent-invocation-terminal.ts b/packages/eve/src/tracing/agent-invocation-terminal.tsdeleted file mode 100644index 77f22c1c2..000000000--- a/packages/eve/src/tracing/agent-invocation-terminal.ts+++ /dev/null@@ -1,121 +0,0 @@-import type { RuntimeSubagentChildResult } from "#shared/action-types.js";-import type { AgentActionTraceTerminalState } from "#tracing/agent-trace-state.js";-import {- AGENT_TRACE_CONTEXT_KEY,- deserializeAgentTraceContextState,- serializeAgentTraceContextState,-} from "#tracing/agent-trace-context-codec.js";-import { truncateTelemetryText } from "#tracing/telemetry-budget.js";-import { boundedTraceError } from "#tracing/bounded-error.js";-import { contextStorage } from "#context/container.js";-import { deserializeContext, serializeContext } from "#context/serialize.js";-import { getInstrumentationRuntime } from "#instrumentation/runtime.js";-import { createLogger, formatError } from "#internal/logging.js";--const log = createLogger("tracing.agent-invocation-terminal");--export function recordNestedAgentInvocationTerminal(input: {- readonly callId: string;- readonly serializedContext: Record<string, unknown>;- readonly sessionId: string;- readonly terminal: AgentActionTraceTerminalState;- readonly turnId?: string;-}): Record<string, unknown> {- const raw = input.serializedContext[AGENT_TRACE_CONTEXT_KEY];- if (raw === undefined) return input.serializedContext;- const state = deserializeAgentTraceContextState(raw);- const entry = Object.entries(state.invocations).find(- ([, invocation]) =>- invocation.sessionId === input.sessionId &&- (input.turnId === undefined || invocation.turnId === input.turnId) &&- invocation.callId === input.callId,- );- if (entry === undefined) return input.serializedContext;- const [key, invocation] = entry;- if (invocation.terminal !== undefined) return input.serializedContext;- return {- ...input.serializedContext,- [AGENT_TRACE_CONTEXT_KEY]: serializeAgentTraceContextState({- ...state,- invocations: {- ...state.invocations,- [key]: {- ...invocation,- terminal: {- ...input.terminal,- error:- invocation.recordOutputs === true && input.terminal.error !== undefined- ? invocationError(input.terminal.error)- : undefined,- },- },- },- }),- };-}--export function settleAgentInvocationTrace(input: {- readonly acceptedAtMs: number;- readonly result: RuntimeSubagentChildResult;- readonly serializedContext: Record<string, unknown>;- readonly sessionId: string;-}): Record<string, unknown> {- const turnResult = input.result.outcome.result;- const usage = input.result.usage ?? input.result.outcome.usageDelta;- return recordNestedAgentInvocationTerminal({- callId: input.result.callId,- serializedContext: input.serializedContext,- sessionId: input.sessionId,- terminal: {- acceptedAtMs: input.acceptedAtMs,- error: turnResult.kind === "failed" ? invocationError(turnResult.error) : undefined,- outcome:- turnResult.kind === "succeeded"- ? "completed"- : turnResult.kind === "cancelled"- ? "cancelled"- : "failed",- usage: {- costUsd: usage.costUsd,- inputTokenDetails: {- cacheReadTokens: usage.cacheReadTokens,- cacheWriteTokens: usage.cacheWriteTokens,- },- inputTokens: usage.inputTokens,- outputTokens: usage.outputTokens,- },- },- });-}--/** Called inside dispatch/settlement steps, so replay reuses the flushed context. */-export async function flushAgentInvocationTraces(- serializedContext: Record<string, unknown>,-): Promise<Record<string, unknown>> {- if (serializedContext[AGENT_TRACE_CONTEXT_KEY] === undefined) return serializedContext;- const runtime = getInstrumentationRuntime();- if (runtime?.flushSettledInvocations === undefined) return serializedContext;- try {- const ctx = await deserializeContext(serializedContext);- await contextStorage.run(ctx, runtime.flushSettledInvocations);- return {- ...serializedContext,- [AGENT_TRACE_CONTEXT_KEY]: serializeContext(ctx)[AGENT_TRACE_CONTEXT_KEY],- };- } catch (error) {- log.warn("could not materialize settled invocation traces", { error: formatError(error) });- return serializedContext;- }-}--export function invocationError(value: unknown): Error {- if (value instanceof Error) {- return boundedTraceError(value);- }- if (typeof value === "object" && value !== null && "message" in value) {- return new Error(truncateTelemetryText(String(value.message), 4096));- }- return new Error(- typeof value === "string" ? truncateTelemetryText(value, 4096) : "Agent invocation failed.",- );-}diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.tsindex af31911fb..166fb38ba 100644--- a/packages/eve/src/tracing/agent-otel-provider.test.ts+++ b/packages/eve/src/tracing/agent-otel-provider.test.ts@@ -2,10 +2,8 @@ import { SpanKind, SpanStatusCode, trace as apiTrace } from "@opentelemetry/api"import {BasicTracerProvider,InMemorySpanExporter,- SamplingDecision,SimpleSpanProcessor,type ReadableSpan,- type Sampler,type SpanProcessor,} from "@opentelemetry/sdk-trace-base";import { describe, expect, it, vi } from "vitest";@@ -29,7 +27,6 @@ import {type AgentOtelInstrumentationInput,} from "#tracing/agent-otel-provider.js";import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";-import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";import type { AgentTraceStateStore } from "#tracing/agent-trace-state.js";import { InMemoryAgentTraceStateStore } from "#internal/testing/in-memory-agent-trace-state-store.js";@@ -99,13 +96,11 @@ function createRuntime(}),extraSpanProcessors: readonly SpanProcessor[] = [],samplesTrace?: AgentOtelInstrumentationInput["samplesTrace"],- sampler?: Sampler,): TestRuntime {const exporter = new InMemorySpanExporter();const idGenerator = new AgentSpanIdGenerator();const provider = new BasicTracerProvider({idGenerator,- sampler,spanProcessors: [new SimpleSpanProcessor(exporter), ...extraSpanProcessors],});const tracer = provider.getTracer("eve.agent");@@ -1449,23 +1444,12 @@ describe("createAgentOtelInstrumentation", () => {});});- it("projects production workflow actions and classifies only GenAI composition", async () => {- const samplerCalls: {- readonly attributes: Readonly<Record<string, unknown>>;- readonly name: string;- }[] = [];- const sampler: Sampler = {- shouldSample(_context, _traceId, name, _kind, attributes) {- samplerCalls.push({ attributes, name });- return { decision: SamplingDecision.RECORD_AND_SAMPLED };- },- toString: () => "recording-sampler",- };+ it("projects production workflow tool calls as agent.action spans", async () => {const context = new ContextContainer();let runtime: TestRuntime | undefined;await contextStorage.run(context, async () => {- runtime = createRuntime(new ContextAgentTraceStateStore(), undefined, [], undefined, sampler);+ runtime = createRuntime(new ContextAgentTraceStateStore());const scope: InstrumentationAttemptScope = {attemptId: "session-workflow:turn-1:0:0",attemptIndex: 0,@@ -1545,217 +1529,41 @@ describe("createAgentOtelInstrumentation", () => {}),);- const nested = prepareAgentInvocationTrace({- invocation: {- callId: "workflow:child",- kind: "subagent-call",- name: "research",- },- ownerId: "workflow-run",- serializedContext: serializeContext(context),- sessionId: scope.sessionId,- sessionState: {- "eve.workflowTool": {- version: 4,- runs: [- {- callId: "workflow",- toolName: "coordinate",- origin: { turnId: "turn-1", stepIndex: 0 },- address: { runId: "workflow-run", hookToken: "workflow-hook" },- },- ],- },- },- startTimeMs: 2,- turnId: scope.turnId,- });- const settled = nested.fail({- callId: "workflow:child",- isError: true,- kind: "subagent-result",- origin: "dispatch",- output: "unavailable",- subagentName: "research",- });- const restored = await deserializeContext(settled);- await contextStorage.run(restored, async () => {- const terminal = createInstrumentationHandleEvent({- handleEvent: async () => {},- hooks: runtime!.hooks,- sessionId: scope.sessionId,- })!;- for (const [callId, toolName] of [- ["workflow", "coordinate"],- ["wait", "wait"],- ] as const) {- await terminal(- createActionResultEvent({- result: {- callId,- kind: "tool-result",- output: { done: true },- toolName,- },- sequence: 0,- stepIndex: 0,- turnId: scope.turnId,- }),- );- }- await runtime!.hooks.publish({- idempotencyKey: attemptIdempotencyKey(scope),- scope,- type: "step.attempt.completed",- });- await completeTurn(runtime!.hooks, scope.sessionId, scope.turnId);+ for (const [callId, toolName] of [+ ["workflow", "coordinate"],+ ["wait", "wait"],+ ] as const) {+ await handleEvent(+ createActionResultEvent({+ result: {+ callId,+ kind: "tool-result",+ output: { done: true },+ toolName,+ },+ sequence: 0,+ stepIndex: 0,+ turnId: scope.turnId,+ }),+ );+ }+ await runtime.hooks.publish({+ idempotencyKey: attemptIdempotencyKey(scope),+ scope,+ type: "step.attempt.completed",});+ await completeTurn(runtime.hooks, scope.sessionId, scope.turnId);});await runtime!.provider.forceFlush();- const spans = runtime!.exporter.getFinishedSpans();- const workflow = byName(spans, "invoke_workflow coordinate")[0]!;- const wait = spans.find((span) => span.attributes["agent.action.call_id"] === "wait")!;- const sampledWorkflow = samplerCalls.find(- ({ attributes }) => attributes["agent.action.call_id"] === "workflow",- )!;-- expect(workflow.attributes).toMatchObject({- "gen_ai.operation.name": "invoke_workflow",- "gen_ai.workflow.name": "coordinate",- });- expect(wait.name).toBe("agent.action");- expect(wait.attributes).not.toHaveProperty("gen_ai.operation.name");- expect(sampledWorkflow).toMatchObject({- attributes: {- "gen_ai.operation.name": "invoke_workflow",- "gen_ai.workflow.name": "coordinate",- },- name: "agent.action",- });- });-- it("keeps a composed workflow intact while emitting independent nested callers", async () => {- const stateStore = new InMemoryAgentTraceStateStore();- const runtime = createRuntime(stateStore);- const scope: InstrumentationAttemptScope = {- attemptId: "session-nested:turn-1:0:0",- attemptIndex: 0,- functionId: "weather",- sessionId: "session-nested",- stepIndex: 0,- turnId: "turn-1",- };- const actionKey = actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow");- const toolKey = `tool:${scope.attemptId}:workflow:0`;-- await publishTurnStarted({- hooks: runtime.hooks,- sessionId: scope.sessionId,- turnId: scope.turnId,- turnSequence: 0,- });- await runtime.hooks.publish({- idempotencyKey: attemptIdempotencyKey(scope),- operation: { modelId: "model", operationId: "ai.streamText", provider: "test" },- scope,- type: "step.attempt.started",- });- await runtime.hooks.publish({- callId: "workflow",- idempotencyKey: actionKey,- input: {},- isWorkflowTool: true,- kind: "tool-call",- name: "coordinate",- scope,- type: "action.started",- });- await runtime.hooks.publish({- callId: "workflow",- idempotencyKey: toolKey,- input: {},- scope,- toolName: "coordinate",- type: "tool.call.started",- });- await runtime.hooks.publish({- idempotencyKey: toolKey,- output: { output: "done", type: "result" },- scope,- type: "tool.call.completed",- });-- const action = stateStore.getAction(actionKey)!;- const anchor = { ...action, workflowName: "coordinate" };- stateStore.setAction(actionKey, anchor);- stateStore.setActionAnchor(actionKey, anchor);- stateStore.setInvocation(- actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow:first"),- {- ...anchor,- callId: "workflow:first",- kind: "subagent-call",- name: "research",- parent: { ...anchor.parent, spanId: anchor.spanId },- parentActionCallId: "workflow",- spanId: "b".repeat(16),- terminal: { outcome: "completed" },- },- );- stateStore.setInvocation(- actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow:second"),- {- ...anchor,- callId: "workflow:second",- kind: "remote-agent-call",- name: "review",- parent: { ...anchor.parent, spanId: anchor.spanId },- parentActionCallId: "workflow",- spanId: "d".repeat(16),- terminal: { error: new Error("review failed"), outcome: "failed" },- },- );- await runtime.hooks.publish({- idempotencyKey: actionKey,- outcome: "completed",- output: { output: "done", type: "result" },- scope,- type: "action.completed",- });- await runtime.hooks.publish({- idempotencyKey: attemptIdempotencyKey(scope),- scope,- type: "step.attempt.completed",- });- await completeTurn(runtime.hooks, scope.sessionId, scope.turnId);- await runtime.provider.forceFlush();-- const spans = runtime.exporter.getFinishedSpans();- const outer = byName(spans, "invoke_workflow coordinate")[0]!;- const tool = byName(spans, "execute_tool coordinate")[0]!;- const first = spans.find(- (span) => span.attributes["agent.action.call_id"] === "workflow:first",- )!;- const second = spans.find(- (span) => span.attributes["agent.action.call_id"] === "workflow:second",- )!;- expect(byName(spans, "agent.action")).toHaveLength(2);- expect(outer.attributes).toMatchObject({- "agent.action.call_id": "workflow",- "agent.action.kind": "tool-call",- "agent.action.name": "coordinate",- "gen_ai.operation.name": "invoke_workflow",- "gen_ai.workflow.name": "coordinate",- "operation.name": "invoke_workflow",- "resource.name": "invoke_workflow coordinate",- });- expect(first.attributes).not.toHaveProperty("gen_ai.operation.name");- expect(second.attributes).not.toHaveProperty("gen_ai.operation.name");- expect(tool.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);- expect(first.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);- expect(second.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);- expect(second.status.code).toBe(SpanStatusCode.ERROR);+ const actions = byName(runtime!.exporter.getFinishedSpans(), "agent.action");+ expect(actions.map((span) => span.attributes["agent.action.call_id"]).sort()).toEqual([+ "wait",+ "workflow",+ ]);+ for (const action of actions) {+ expect(action.attributes).toMatchObject({ "agent.action.kind": "tool-call" });+ }});it.each(["private", "unknown"] as const)(diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.tsindex 7d53fa6b9..7449a8526 100644--- a/packages/eve/src/tracing/agent-otel-provider.ts+++ b/packages/eve/src/tracing/agent-otel-provider.ts@@ -302,7 +302,6 @@ export function createAgentOtelInstrumentation(const onSessionTransition = async (event: InstrumentationSessionTransitionEvent,): Promise<void> => {- await actions.flushForSessionTransition(event);if (event.type === "session.failed" && event.turnId !== undefined) {await input.stateStore.updateTurn(event.sessionId, event.turnId, (turn) => ({...turn,@@ -534,7 +533,6 @@ export function createAgentOtelInstrumentation(return {hook: {- flush: actions.flushSettledInvocations,events: {...channelDeliveries,"action.completed": actions.events["action.completed"],diff --git a/packages/eve/src/tracing/agent-span-contract.ts b/packages/eve/src/tracing/agent-span-contract.tsindex 311abc8c2..2e0d6c914 100644--- a/packages/eve/src/tracing/agent-span-contract.ts+++ b/packages/eve/src/tracing/agent-span-contract.ts@@ -19,10 +19,6 @@ export function agentInvocationSpanName(agentName: string | undefined): string {return agentName === undefined ? "invoke_agent" : `invoke_agent ${agentName}`;}-export function workflowInvocationSpanName(workflowName: string): string {- return `invoke_workflow ${workflowName}`;-}-export function modelSpanName(modelId: string): string {return `chat ${modelId}`;}diff --git a/packages/eve/src/tracing/agent-span-id-generator.ts b/packages/eve/src/tracing/agent-span-id-generator.tsindex f2702db72..ca569cb8b 100644--- a/packages/eve/src/tracing/agent-span-id-generator.ts+++ b/packages/eve/src/tracing/agent-span-id-generator.ts@@ -1,7 +1,5 @@import { createHash } from "node:crypto";-import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";-/*** Id generator shared by `registerOTel` and the agent OTel provider. A span* whose lifetime crosses durable worker boundaries (`invoke_agent`) is emitted@@ -70,10 +68,6 @@ function deriveAgentSpanId(key: string): string {return /^0+$/u.test(spanId) ? "0000000000000001" : spanId;}-export function deriveAgentActionSpanId(sessionId: string, turnId: string, callId: string): string {- return deriveAgentSpanId(`action:${actionIdempotencyKey(sessionId, turnId, callId)}`);-}-const HEX_DIGITS = "0123456789abcdef";function randomHexId(length: number): string {diff --git a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.tsindex 646495629..379550c6c 100644--- a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts+++ b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts@@ -5,7 +5,6 @@ import {type ReadableSpan,} from "@opentelemetry/sdk-trace-base";import { describe, expect, it, vi } from "vitest";-import { SpanStatusCode } from "#compiled/@opentelemetry/api/index.js";import { JsonTraceSerializer } from "#compiled/@opentelemetry/otlp-transformer/index.js";import { ContextContainer, contextStorage } from "#context/container.js";@@ -35,11 +34,7 @@ import { createAgentOtelInstrumentation } from "#tracing/agent-otel-provider.js"import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";import { AGENT_TRACE_CONTEXT_KEY } from "#tracing/agent-trace-context-codec.js";-import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";-import {- flushAgentInvocationTraces,- settleAgentInvocationTrace,-} from "#tracing/agent-invocation-terminal.js";+import { resolveToolCallAgentTrace } from "#tracing/agent-invocation-coordinator.js";import * as instrumentation from "#instrumentation/runtime.js";import {assembleLocalTrace,@@ -92,13 +87,7 @@ function createRuntime() {idGenerator,ownsAgentSpans: true,otelSettings: { recordInputs: true, recordOutputs: true, traceChannelRequests: false },- flushSettledInvocations: async () => {- await agent.hook.flush?.();- },- forceFlush: async () => {- await agent.hook.flush?.();- await provider.forceFlush();- },+ forceFlush: () => provider.forceFlush(),shutdown: () => provider.shutdown(),};}@@ -136,139 +125,6 @@ function scopeFor(sessionId: string, audience: "public" | "private"): Instrument}describe("exported agent telemetry contract", () => {- it("flushes 1000 settled invocations without closing the session or retaining their errors", async () => {- const runtime = createRuntime();- const exporterFlush = vi.spyOn(runtime.provider, "forceFlush").mockImplementation(() => {- throw new Error("Exporter unavailable");- });- const registered = vi- .spyOn(instrumentation, "getInstrumentationRuntime")- .mockReturnValue(runtime);- let ctx = contextFor("public");- const store = new ContextAgentTraceStateStore();- const count = 1000;- try {- for (let index = 0; index < count; index++) {- const callId = `nested-${index}`;- contextStorage.run(ctx, () =>- store.setInvocation(actionIdempotencyKey("parent", "turn_0", callId), {- attemptIndex: 0,- callId,- channelAudience: "public",- kind: "subagent-call",- name: "child",- parent: { spanId: "1".repeat(16), traceFlags: 1, traceId: "2".repeat(32) },- parentActionCallId: "outer",- recordOutputs: true,- rootSessionId: "parent",- sessionId: "parent",- spanId: runtime.idGenerator.deriveSpanId(callId),- startTimeMs: 1,- stepIndex: 0,- turnId: "turn_0",- terminal: {- acceptedAtMs: 2,- error: new Error("private " + "x".repeat(16384)),- outcome: index % 2 === 0 ? "failed" : "cancelled",- },- }),- );- const flushed = await flushAgentInvocationTraces(serializeContext(ctx));- expect(JSON.stringify(flushed)).not.toContain("private ");- ctx = await deserializeContext(flushed);- contextStorage.run(ctx, () => expect(store.findInvocations("parent")).toEqual([]));- }- expect(runtime.exporter.getFinishedSpans()).toHaveLength(count);- for (const span of runtime.exporter.getFinishedSpans()) {- expect(span.status.code).toBe(- span.attributes["agent.action.outcome"] === "failed"- ? SpanStatusCode.ERROR- : SpanStatusCode.UNSET,- );- }- await flushAgentInvocationTraces(serializeContext(ctx));- expect(runtime.exporter.getFinishedSpans()).toHaveLength(count);- expect(exporterFlush).not.toHaveBeenCalled();- } finally {- exporterFlush.mockRestore();- registered.mockRestore();- await runtime.shutdown();- }- });-- it.each(["throw", "reject"] as const)(- "preserves the settlement context when invocation materialization fails (%s)",- async (failure) => {- const logs = captureLogRecords();- const runtime = createRuntime();- const flush = vi.spyOn(runtime, "flushSettledInvocations").mockImplementation(() => {- const error = new Error("Span processor unavailable");- if (failure === "throw") throw error;- return Promise.reject(error);- });- const registered = vi- .spyOn(instrumentation, "getInstrumentationRuntime")- .mockReturnValue(runtime);- const serializedContext = { [AGENT_TRACE_CONTEXT_KEY]: {} };- try {- await expect(flushAgentInvocationTraces(serializedContext)).resolves.toBe(- serializedContext,- );- expect(flush).toHaveBeenCalledOnce();- expect(logs.records).toContainEqual(- expect.objectContaining({- level: "warn",- message: "could not materialize settled invocation traces",- }),- );- } finally {- registered.mockRestore();- await runtime.shutdown();- }- },- );-- it.each(["session.completed", "session.failed"] as const)(- "only marks unfinished invocations as errors on %s",- async (type) => {- const runtime = createRuntime();- const ctx = contextFor("public");- const hooks = runtime.hooks.forTrace!(traceContext("parent", "public"));- await contextStorage.run(ctx, async () => {- const store = new ContextAgentTraceStateStore();- store.setInvocation(actionIdempotencyKey("parent", "turn_0", "nested"), {- attemptIndex: 0,- callId: "nested",- kind: "subagent-call",- name: "child",- parent: { spanId: "1".repeat(16), traceFlags: 1, traceId: "2".repeat(32) },- parentActionCallId: "outer",- rootSessionId: "parent",- sessionId: "parent",- spanId: "3".repeat(16),- startTimeMs: 1,- stepIndex: 0,- turnId: "turn_0",- });- await hooks.publish({- error: new Error("private failure"),- idempotencyKey: sessionIdempotencyKey("parent"),- sessionId: "parent",- type,- });- expect(store.findInvocations("parent")).toEqual([]);- });- const [span] = runtime.exporter.getFinishedSpans();- expect(span?.attributes["agent.action.outcome"]).toBe(- type === "session.failed" ? "failed" : "abandoned",- );- expect(span?.status.code).toBe(- type === "session.failed" ? SpanStatusCode.ERROR : SpanStatusCode.UNSET,- );- await runtime.shutdown();- },- );-it.each(["public", "private"] as const)("round-trips the normalized %s v4 trace forest through OTLP",async (audience) => {@@ -284,7 +140,7 @@ describe("exported agent telemetry contract", () => {rootSessionId: "parent",sessionId: "parent",})!;- let dispatch: ReturnType<typeof prepareAgentInvocationTrace>;+ let dispatch: ReturnType<typeof resolveToolCallAgentTrace>;await contextStorage.run(parent, async () => {await binding.preparePreamble({ sequence: 0, sessionStarted: false });await binding.instrumentChannelDelivery({@@ -357,42 +213,26 @@ describe("exported agent telemetry contract", () => {scope,type: "input.resolved",});- dispatch = prepareAgentInvocationTrace({+ dispatch = resolveToolCallAgentTrace({+ callId: "workflow",conversation: parent.get(ConversationContextKey),- invocation: { callId: "nested", kind: "subagent-call", name: "child" },- ownerId: "workflow-run",- startTimeMs: Date.now(),serializedContext: serializeContext(parent),sessionId: "parent",turnId: "turn_0",- sessionState: {- "eve.workflowTool": {- version: 4,- runs: [- {- callId: "workflow",- toolName: "coordinate",- origin: { turnId: "turn-1", stepIndex: 0 },- address: { runId: "workflow-run", hookToken: "hook" },- },- ],- },- },});});- parent = await deserializeContext(dispatch!.serializedContext);const child = contextFor(audience);- child.set(ConversationIdKey, dispatch!.dispatch.conversationId!);+ child.set(ConversationIdKey, dispatch!.conversationId!);child.set(ParentSessionKey, {- callId: "nested",+ callId: "workflow",rootSessionId: "parent",sessionId: "parent",turn: { id: "turn_0", sequence: 0 },});- if (dispatch!.dispatch.parentTraceContext !== undefined) {- child.set(ParentTraceContextKey, dispatch!.dispatch.parentTraceContext);+ if (dispatch!.parentTraceContext !== undefined) {+ child.set(ParentTraceContextKey, dispatch!.parentTraceContext);child.set(SessionTraceSeedKey, {- ...dispatch!.dispatch.parentTraceContext,+ ...dispatch!.parentTraceContext,spanId: runtime.idGenerator.allocateSpanId(),traceId: runtime.idGenerator.generateTraceId(),});@@ -453,38 +293,7 @@ describe("exported agent telemetry contract", () => {type: "session.waiting",});});- const settled = settleAgentInvocationTrace({- acceptedAtMs: Date.now(),- serializedContext: serializeContext(parent),- sessionId: "parent",- result: {- callId: "nested",- kind: "subagent-result",- origin: "child",- subagentName: "child",- output: "private reply",- outcome: {- kind: "terminal",- result: { kind: "succeeded", output: "private reply" },- usageDelta: {- inputTokens: 10,- outputTokens: 5,- cacheReadTokens: 4,- cacheWriteTokens: 2,- },- },- },- });- if (audience === "private") expect(JSON.stringify(settled)).not.toContain("private reply");- parent = await deserializeContext(settled);await contextStorage.run(parent, async () => {- await runtime.forceFlush();- expect(- runtime.exporter- .getFinishedSpans()- .filter((span) => span.attributes["agent.invocation.role"] === "caller"),- ).toHaveLength(1);- expect(new ContextAgentTraceStateStore().findInvocations("parent")).toEqual([]);await hooks.publish({idempotencyKey: toolCallIdempotencyKey(scope, "workflow", 0),output: { type: "result", output: "private output" },@@ -623,7 +432,9 @@ describe("exported agent telemetry contract", () => {);}expect(new TextDecoder().decode(bytes)).not.toContain("auth-only-secret");- const caller = parsed.find((span) => span.attributes["agent.invocation.role"] === "caller")!;+ const workflow = parsed.find(+ (span) => span.attributes["agent.action.call_id"] === "workflow",+ )!;const activation = parsed.find((span) => span.name === "invoke_agent child" && isAgentTurnSpan(span),)!;@@ -631,14 +442,14 @@ describe("exported agent telemetry contract", () => {(span) => span.name === "invoke_agent parent" && isAgentTurnSpan(span),)!;expect(activation.parentSpanId).toBeUndefined();- expect(activation.traceId).not.toBe(caller.traceId);+ expect(activation.traceId).not.toBe(workflow.traceId);expect(parentActivation.attributes).toMatchObject({"agent.channel.delivery.id": "delivery","agent.channel.kind": "http","agent.channel.name": "web",});expect(activation.attributes).toMatchObject({- "agent.parent_call.id": "nested",+ "agent.parent_call.id": "workflow","agent.parent_run.id": "parent","agent.run.id": "child","gen_ai.usage.input_tokens": 10,@@ -646,20 +457,17 @@ describe("exported agent telemetry contract", () => {"agent.usage.input_tokens": 10,"agent.usage.output_tokens": 5,});- expect(caller.attributes["gen_ai.usage.input_tokens"]).toBeUndefined();- expect(caller.attributes["gen_ai.usage.output_tokens"]).toBeUndefined();expect(normalizeTraceForest(parsed, exported)).toEqual(["conversation original-conversation","trace parent:turn_0 outcome=completed channel=http:web delivery=delivery"," invoked from external via channel.request"," invoke_agent parent"," agent.step",- " invoke_workflow coordinate",- " agent.action child role=caller",+ " agent.action coordinate"," agent.approval approved"," execute_tool coordinate","trace child:turn_0 outcome=completed",- " invoked from parent:turn_0/agent.action child role=caller via agent.dispatch",+ " invoked from parent:turn_0/agent.action coordinate via agent.dispatch"," invoke_agent child"," agent.step"," chat test",@@ -976,9 +784,6 @@ function normalizeTraceForest(}function spanLabel(span: LocalTraceSpan): string {- if (span.attributes["gen_ai.operation.name"] === "invoke_workflow") {- return span.name;- }if (span.name === "agent.action") {const role = span.attributes["agent.invocation.role"] === "caller" ? " role=caller" : "";return `agent.action ${String(span.attributes["agent.action.name"])}${role}`;diff --git a/packages/eve/src/tracing/agent-trace-context-codec.ts b/packages/eve/src/tracing/agent-trace-context-codec.tsindex 7439f5698..60a982418 100644--- a/packages/eve/src/tracing/agent-trace-context-codec.ts+++ b/packages/eve/src/tracing/agent-trace-context-codec.ts@@ -2,8 +2,6 @@ import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";import type {AgentActionTraceState,- AgentActionTraceTerminalState,- AgentInvocationTraceState,AgentSessionTraceState,AgentTurnTraceState,} from "#tracing/agent-trace-state.js";@@ -18,35 +16,18 @@ export const AGENT_TRACE_CONTEXT_KEY = "eve.harness.agentTrace";export interface AgentTraceContextState {readonly actionAnchors: Readonly<Record<string, AgentActionTraceState>>;readonly actions: Readonly<Record<string, AgentActionTraceState>>;- readonly invocations: Readonly<Record<string, AgentInvocationTraceState>>;readonly sessions: Readonly<Record<string, AgentSessionTraceState>>;readonly turns: Readonly<Record<string, AgentTurnTraceState>>;}export function emptyAgentTraceContextState(): AgentTraceContextState {- return { actionAnchors: {}, actions: {}, invocations: {}, sessions: {}, turns: {} };+ return { actionAnchors: {}, actions: {}, sessions: {}, turns: {} };}export function serializeAgentTraceContextState(state: AgentTraceContextState): unknown {return {actionAnchors: state.actionAnchors,actions: state.actions,- invocations: Object.fromEntries(- Object.entries(state.invocations).map(([id, value]) => [- id,- {- ...value,- terminal:- value.terminal === undefined- ? undefined- : {- ...value.terminal,- error:- value.recordOutputs === true ? serializeError(value.terminal.error) : undefined,- },- },- ]),- ),sessions: Object.fromEntries(Object.entries(state.sessions).map(([id, value]) => [id,@@ -87,7 +68,6 @@ export function deserializeAgentTraceContextState(data: unknown): AgentTraceContreturn {actionAnchors: deserializeRecord(data.actionAnchors, deserializeAction),actions: deserializeRecord(data.actions, deserializeAction),- invocations: deserializeRecord(data.invocations, deserializeInvocation),sessions: deserializeRecord(data.sessions, deserializeSession),turns: deserializeRecord(data.turns, deserializeTurn),};@@ -180,7 +160,6 @@ function deserializeAction(value: unknown): AgentActionTraceState | undefined {callId: value.callId,channelAudience: normalizeChannelAudience(value.channelAudience),inputAttribute: typeof value.inputAttribute === "string" ? value.inputAttribute : undefined,- isWorkflowTool: value.isWorkflowTool === true ? true : undefined,kind: value.kind,name: value.name,parent: value.parent,@@ -190,49 +169,9 @@ function deserializeAction(value: unknown): AgentActionTraceState | undefined {startTimeMs: value.startTimeMs,stepIndex: value.stepIndex,turnId: value.turnId,- workflowName: typeof value.workflowName === "string" ? value.workflowName : undefined,};}-function deserializeInvocation(value: unknown): AgentInvocationTraceState | undefined {- const action = deserializeAction(value);- if (- action === undefined ||- (action.kind !== "subagent-call" && action.kind !== "remote-agent-call") ||- !isRecord(value) ||- typeof value.parentActionCallId !== "string"- ) {- return undefined;- }- return {- attemptIndex: action.attemptIndex,- callId: action.callId,- channelAudience: action.channelAudience,- kind: action.kind,- name: action.name,- parent: action.parent,- parentActionCallId: value.parentActionCallId,- recordOutputs: value.recordOutputs === true,- rootSessionId: action.rootSessionId,- sessionId: action.sessionId,- spanId: action.spanId,- startTimeMs: action.startTimeMs,- stepIndex: action.stepIndex,- terminal: deserializeActionTerminal(value.terminal),- turnId: action.turnId,- };-}--function deserializeActionTerminal(value: unknown): AgentActionTraceTerminalState | undefined {- if (!isRecord(value) || !isActionOutcome(value.outcome)) return undefined;- return {- acceptedAtMs: typeof value.acceptedAtMs === "number" ? value.acceptedAtMs : undefined,- error: deserializeError(value.error),- outcome: value.outcome,- usage: isRecord(value.usage) ? value.usage : undefined,- } as AgentActionTraceTerminalState;-}-function deserializeModelUsage(value: unknown,): NonNullable<AgentTurnTraceState["modelUsage"]> | undefined {@@ -313,16 +252,6 @@ function isActionKind(value: unknown): value is AgentActionTraceState["kind"] {);}-function isActionOutcome(value: unknown): value is AgentActionTraceTerminalState["outcome"] {- return (- value === "abandoned" ||- value === "cancelled" ||- value === "completed" ||- value === "failed" ||- value === "rejected"- );-}-function isTurnTerminalType(value: unknown,): value is "turn.cancelled" | "turn.completed" | "turn.failed" {diff --git a/packages/eve/src/tracing/agent-trace-context-store.ts b/packages/eve/src/tracing/agent-trace-context-store.tsindex b7bbde888..b23f0267a 100644--- a/packages/eve/src/tracing/agent-trace-context-store.ts+++ b/packages/eve/src/tracing/agent-trace-context-store.ts@@ -7,7 +7,6 @@ import type { ContextAccessor } from "#context/key.js";import { SessionTraceSeedKey, type SessionTraceSeed } from "#context/keys.js";import type {AgentActionTraceState,- AgentInvocationTraceState,AgentSessionTraceState,AgentTraceStateStore,AgentTurnTraceState,@@ -69,20 +68,7 @@ function pruneTraceOwnership(calls.has(action.callId),),);- const retainedCalls = new Set(- Object.values(actionAnchors)- .filter((action) => action.sessionId === sessionId)- .map((action) => action.callId),- );- const invocations = Object.fromEntries(- Object.entries(state.invocations).filter(- ([, invocation]) =>- invocation.sessionId !== sessionId ||- invocation.terminal !== undefined ||- retainedCalls.has(invocation.parentActionCallId),- ),- );- context.set(AgentTraceContextKey, { ...state, actionAnchors, invocations });+ context.set(AgentTraceContextKey, { ...state, actionAnchors });}export function readSessionTraceDecision(@@ -129,7 +115,7 @@ export function readActionTraceContext(if (raw === undefined) return undefined;const state = deserializeAgentTraceContextState(raw);const key = actionIdempotencyKey(sessionId, turnId, callId);- const action = state.invocations[key] ?? state.actions[key] ?? state.actionAnchors[key];+ const action = state.actions[key] ?? state.actionAnchors[key];if (action === undefined) return undefined;return withTraceDecision(serializedContext,@@ -143,112 +129,6 @@ export function readActionTraceContext();}-export function recordNestedAgentInvocation(input: {- readonly callId: string;- readonly kind: "remote-agent-call" | "subagent-call";- readonly name: string;- readonly outerCallId: string;- readonly recordOutputs?: boolean;- readonly serializedContext: Record<string, unknown>;- readonly sessionId: string;- readonly spanId: string;- readonly startTimeMs: number;- readonly turnId: string;-}): Record<string, unknown> {- const raw = input.serializedContext[AgentTraceContextKey.name];- if (raw === undefined) return input.serializedContext;- const state = deserializeAgentTraceContextState(raw);- const outerKey = actionIdempotencyKey(input.sessionId, input.turnId, input.outerCallId);- const outer = state.actions[outerKey] ?? state.actionAnchors[outerKey];- if (outer === undefined) return input.serializedContext;- const key = actionIdempotencyKey(input.sessionId, input.turnId, input.callId);- const promote = (action: AgentActionTraceState): AgentActionTraceState =>- action.isWorkflowTool === true && action.kind === "tool-call"- ? { ...action, workflowName: action.name }- : action;- const actions =- state.actions[outerKey] === undefined- ? state.actions- : { ...state.actions, [outerKey]: promote(state.actions[outerKey]) };- const actionAnchors =- state.actionAnchors[outerKey] === undefined- ? state.actionAnchors- : { ...state.actionAnchors, [outerKey]: promote(state.actionAnchors[outerKey]) };- if (state.invocations[key] !== undefined) {- return {- ...input.serializedContext,- [AgentTraceContextKey.name]: serializeAgentTraceContextState({- ...state,- actionAnchors,- actions,- }),- };- }- const invocation: AgentInvocationTraceState = {- attemptIndex: outer.attemptIndex,- callId: input.callId,- channelAudience: outer.channelAudience,- kind: input.kind,- name: input.name,- parent: {- spanId: outer.spanId,- traceFlags: outer.parent.traceFlags,- traceId: outer.parent.traceId,- },- parentActionCallId: input.outerCallId,- recordOutputs: input.recordOutputs === true,- rootSessionId: outer.rootSessionId,- sessionId: outer.sessionId,- spanId: input.spanId,- startTimeMs: input.startTimeMs,- stepIndex: outer.stepIndex,- turnId: outer.turnId,- };- return {- ...input.serializedContext,- [AgentTraceContextKey.name]: serializeAgentTraceContextState({- ...state,- actionAnchors,- actions,- invocations: { ...state.invocations, [key]: invocation },- }),- };-}--export function recordActionInvocationKind(input: {- readonly callId: string;- readonly kind: "remote-agent-call" | "subagent-call";- readonly serializedContext: Record<string, unknown>;- readonly sessionId: string;- readonly turnId: string;-}): Record<string, unknown> {- const raw = input.serializedContext[AgentTraceContextKey.name];- if (raw === undefined) return input.serializedContext;- const state = deserializeAgentTraceContextState(raw);- const update = (action: AgentActionTraceState): AgentActionTraceState =>- action.sessionId === input.sessionId &&- action.turnId === input.turnId &&- action.callId === input.callId- ? {- ...action,- isWorkflowTool: undefined,- kind: input.kind,- workflowName: undefined,- }- : action;- return {- ...input.serializedContext,- [AgentTraceContextKey.name]: serializeAgentTraceContextState({- ...state,- actionAnchors: Object.fromEntries(- Object.entries(state.actionAnchors).map(([key, action]) => [key, update(action)]),- ),- actions: Object.fromEntries(- Object.entries(state.actions).map(([key, action]) => [key, update(action)]),- ),- }),- };-}function withTraceDecision(serializedContext: Readonly<Record<string, unknown>>,context: SpanContext,@@ -310,27 +190,6 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {});}- deleteInvocation(idempotencyKey: string): void {- updateState((state) => {- const invocations = { ...state.invocations };- delete invocations[idempotencyKey];- return { ...state, invocations };- });- }-- deleteInvocations(sessionId: string, turnId?: string): void {- updateState((state) => ({- ...state,- invocations: Object.fromEntries(- Object.entries(state.invocations).filter(- ([, invocation]) =>- invocation.sessionId !== sessionId ||- (turnId !== undefined && invocation.turnId !== turnId),- ),- ),- }));- }-deleteSession(sessionId: string): void {updateState((state) => {const sessions = { ...state.sessions };@@ -366,21 +225,6 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {);}- findInvocations(- sessionId?: string,- turnId?: string,- parentActionCallId?: string,- ): readonly AgentInvocationTraceState[] {- return Object.values(- contextStorage.getStore()?.get(AgentTraceContextKey)?.invocations ?? {},- ).filter(- (invocation) =>- (sessionId === undefined || invocation.sessionId === sessionId) &&- (turnId === undefined || invocation.turnId === turnId) &&- (parentActionCallId === undefined || invocation.parentActionCallId === parentActionCallId),- );- }-getAction(idempotencyKey: string): AgentActionTraceState | undefined {return contextStorage.getStore()?.get(AgentTraceContextKey)?.actions[idempotencyKey];}@@ -407,13 +251,6 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {}));}- setInvocation(idempotencyKey: string, value: AgentInvocationTraceState): void {- updateState((state) => ({- ...state,- invocations: { ...state.invocations, [idempotencyKey]: value },- }));- }-setSession(sessionId: string, value: AgentSessionTraceState): void {updateState((state) => ({...state,diff --git a/packages/eve/src/tracing/agent-trace-state.ts b/packages/eve/src/tracing/agent-trace-state.tsindex 842614f94..ade39d6fa 100644--- a/packages/eve/src/tracing/agent-trace-state.ts+++ b/packages/eve/src/tracing/agent-trace-state.ts@@ -2,13 +2,11 @@ import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";import type {InstrumentationActionKind,- InstrumentationActionOutcome,InstrumentationParentLineage,InstrumentationPrincipalSummary,InstrumentationTraceContext,InstrumentationTurnFailedEvent,InstrumentationTurnSettledEvent,- InstrumentationUsage,} from "#instrumentation/lifecycle.js";import type { ChannelAudience } from "#shared/channel-audience.js";import type { InstrumentationDecision } from "#shared/instrumentation-decision.js";@@ -57,7 +55,6 @@ export interface AgentActionTraceState {readonly callId: string;readonly channelAudience?: ChannelAudience;readonly inputAttribute?: string;- readonly isWorkflowTool?: boolean;readonly kind: InstrumentationActionKind;readonly name: string;readonly parent: InstrumentationTraceContext;@@ -67,24 +64,6 @@ export interface AgentActionTraceState {readonly startTimeMs: number;readonly stepIndex: number;readonly turnId: string;- readonly workflowName?: string;-}--export interface AgentInvocationTraceState extends Omit<- AgentActionTraceState,- "inputAttribute" | "isWorkflowTool" | "kind" | "workflowName"-> {- readonly kind: "remote-agent-call" | "subagent-call";- readonly parentActionCallId: string;- readonly recordOutputs?: boolean;- readonly terminal?: AgentActionTraceTerminalState;-}--export interface AgentActionTraceTerminalState {- readonly acceptedAtMs?: number;- readonly error?: unknown;- readonly outcome: InstrumentationActionOutcome;- readonly usage?: InstrumentationUsage;}/** Provider-owned serializable storage for durable agent trace state. */@@ -92,8 +71,6 @@ export interface AgentTraceStateStore {deleteAction(idempotencyKey: string): void | PromiseLike<void>;deleteActionAnchors(sessionId: string): void | PromiseLike<void>;deleteActions(sessionId: string, turnId?: string): void | PromiseLike<void>;- deleteInvocation(idempotencyKey: string): void | PromiseLike<void>;- deleteInvocations(sessionId: string, turnId?: string): void | PromiseLike<void>;deleteSession(sessionId: string): void | PromiseLike<void>;deleteTurn(sessionId: string, turnId: string): void | PromiseLike<void>;findAction(@@ -105,11 +82,6 @@ export interface AgentTraceStateStore {turnId: string,callId: string,): AgentActionTraceState | undefined | PromiseLike<AgentActionTraceState | undefined>;- findInvocations(- sessionId?: string,- turnId?: string,- parentActionCallId?: string,- ): readonly AgentInvocationTraceState[] | PromiseLike<readonly AgentInvocationTraceState[]>;getAction(idempotencyKey: string,): AgentActionTraceState | undefined | PromiseLike<AgentActionTraceState | undefined>;@@ -122,7 +94,6 @@ export interface AgentTraceStateStore {): AgentTurnTraceState | undefined | PromiseLike<AgentTurnTraceState | undefined>;setAction(idempotencyKey: string, state: AgentActionTraceState): void | PromiseLike<void>;setActionAnchor(idempotencyKey: string, state: AgentActionTraceState): void | PromiseLike<void>;- setInvocation(idempotencyKey: string, state: AgentInvocationTraceState): void | PromiseLike<void>;setSession(sessionId: string, state: AgentSessionTraceState): void | PromiseLike<void>;setTurn(sessionId: string, turnId: string, state: AgentTurnTraceState): void | PromiseLike<void>;/** Atomically updates an existing turn and does nothing after that turn is deleted. */diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts b/packages/eve/src/tracing/install-instrumentation-runtime.test.tsindex 81915e698..79f0cfa5e 100644--- a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts+++ b/packages/eve/src/tracing/install-instrumentation-runtime.test.ts@@ -6,9 +6,8 @@ import { sessionIdempotencyKey, turnIdempotencyKey } from "#instrumentation/lifeimport { installInstrumentationRuntime } from "#tracing/install-instrumentation-runtime.js";import { otelIntegration, collectOtelPipeline } from "#tracing/otel-declaration.js";-const { forceFlush, invocationFlush, internalTerminalState, shutdown } = vi.hoisted(() => ({+const { forceFlush, internalTerminalState, shutdown } = vi.hoisted(() => ({forceFlush: vi.fn(async () => undefined),- invocationFlush: vi.fn(async () => undefined),internalTerminalState: vi.fn(),shutdown: vi.fn(async () => undefined),}));@@ -31,7 +30,6 @@ vi.mock("#tracing/otel-registration.js", async (importOriginal) => {vi.mock("#tracing/agent-otel-provider.js", () => ({createAgentOtelInstrumentation: () => ({hook: {- flush: invocationFlush,events: {"turn.completed": (_event: unknown, ctx: { state: { get(): unknown } }) => {internalTerminalState(ctx.state.get());@@ -51,7 +49,6 @@ const RUNTIME_GLOBAL_KEY = Symbol.for("eve.instrumentation-runtime");describe("installInstrumentationRuntime", () => {beforeEach(() => {forceFlush.mockClear();- invocationFlush.mockClear();internalTerminalState.mockClear();shutdown.mockClear();delete (globalThis as Record<symbol, unknown>)[RUNTIME_GLOBAL_KEY];@@ -85,22 +82,6 @@ describe("installInstrumentationRuntime", () => {expect(providerShutdown).toHaveBeenCalledOnce();});- it("materializes settled invocations without draining authored providers or the exporter", async () => {- const providerFlush = vi.fn();- const runtime = installInstrumentationRuntime({- collected: collectOtelPipeline([otelIntegration()]),- frameworkVersion: "test",- providers: [{ flush: providerFlush, name: "test" }],- serviceName: "weather",- });-- await runtime.flushSettledInvocations!();-- expect(invocationFlush).toHaveBeenCalledOnce();- expect(providerFlush).not.toHaveBeenCalled();- expect(forceFlush).not.toHaveBeenCalled();- });-it("enables memory operations only for OTel or a provider that handles them", () => {const withoutMemory = installInstrumentationRuntime({collected: collectOtelPipeline([]),diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.ts b/packages/eve/src/tracing/install-instrumentation-runtime.tsindex eb83a723b..30304d07a 100644--- a/packages/eve/src/tracing/install-instrumentation-runtime.ts+++ b/packages/eve/src/tracing/install-instrumentation-runtime.ts@@ -42,7 +42,6 @@ export function installInstrumentationRuntime(input: {const serialBefore: InstrumentationProviderDefinition[] = [];const serialAfter: InstrumentationProviderDefinition[] = [];let otelRuntime: RegisteredOtelPipeline | undefined;- let flushSettledInvocations: InstrumentationRuntime["flushSettledInvocations"];let prepareSessionTrace: InstrumentationRuntime["prepareSessionTrace"];let prepareTurnTrace: InstrumentationRuntime["prepareTurnTrace"];let runInContext: InstrumentationRuntime["runInContext"] = (_operation, execute) => execute();@@ -65,9 +64,6 @@ export function installInstrumentationRuntime(input: {});// The span must exist before authored providers observe the lifecycle event.serialBefore.push({ ...agentOtel.hook, stateNamespace: "internal:otel" });- flushSettledInvocations = async () => {- await agentOtel.hook.flush?.();- };prepareSessionTrace = agentOtel.prepareSessionTrace;prepareTurnTrace = agentOtel.prepareTurnTrace;runInContext = agentOtel.runInContext;@@ -81,7 +77,6 @@ export function installInstrumentationRuntime(input: {const allProviders = [...serialBefore, ...input.providers, ...serialAfter];let shutdown: Promise<void> | undefined;return registerInstrumentationRuntime({- flushSettledInvocations,forceFlush: async () => {await settleAll(allProviders.map((provider) => () => provider.flush?.()));await settleAll(otelRuntime === undefined ? [] : [otelRuntime.forceFlush]);