The Open Framework for Building Agents
feat(eve): separate dispatch from bounded activation traces (#2966)
Chad Hietala879d4e540383
diff --git a/.changeset/trace-workflow-agent-dispatch.md b/.changeset/trace-workflow-agent-dispatch.mdnew file mode 100644index 000000000..780610b93--- /dev/null+++ b/.changeset/trace-workflow-agent-dispatch.md@@ -0,0 +1,5 @@+---+"eve": minor+---++Advance `agent.trace.schema.version` from 3 to 4 and remove `agent.session` and `agent.channel.delivery`: update dashboards to use per-activation `invoke_agent` spans, linked to callers and grouped by `gen_ai.conversation.id`. Single turn-bound channel deliveries annotate their activation and link it to the active upstream request or function span; `traceChannelRequests` only adds an eve-owned server span as the link target. Other delivery lifecycles do not emit agent spans. Dispatch uses `agent.action` and `execute_tool`, preserves standard GenAI usage totals and trace-content restrictions, and rejects baggage overflow; settlement materializes spans without draining exporters, and conversation IDs remain available without instrumentation.diff --git a/docs/guides/instrumentation-providers.md b/docs/guides/instrumentation-providers.mdindex 2fc529f8a..d5237eee3 100644--- a/docs/guides/instrumentation-providers.md+++ b/docs/guides/instrumentation-providers.md@@ -235,6 +235,9 @@ An ordinary tool emits both `action.*` and `tool.call.*` events. Use `action.*`Handlers for different providers run concurrently and are failure-isolated. Do not depend on provider execution order. Use `flush` to drain buffered records and `shutdown` to release resources.+The [agent trace contract](./instrumentation#agent-trace-contract) defines span+names, identities, and usage counters shared by the exporters and local viewers.+## What to read next- [Observability](./instrumentation): the default `instrumentation.ts` API and trace hierarchydiff --git a/docs/guides/instrumentation.md b/docs/guides/instrumentation.mdindex 800bc8da8..52f499d0b 100644--- a/docs/guides/instrumentation.md+++ b/docs/guides/instrumentation.md@@ -56,6 +56,8 @@ Three more fields control what the AI SDK records inside those spans (see the AIeve records metadata without model or tool inputs and outputs by default. Enable either content category only after reviewing the exporter and its data-retention path.+In the provider layout, eve stamps each span with `gen_ai.conversation.id`, which stays fixed across local and remote activations, including independent remote root workflows. Query this attribute to find the conversation's exported, retained traces; it does not grant access or control trace parenting. On Vercel, `vercel.session_id` additionally identifies the current Workflow run.+You are responsible for ensuring any observability or eval provider is approved for the data exported to it.The third configurable surface, [runtime context events](#runtime-context), attaches per-model-call values to these spans.@@ -79,11 +81,9 @@ and outcome fields. Content providers additionally receive only eve's knownmessage, context, input-response, and output-schema fields; adapter-specificpayload fields are never projected.-The built-in OpenTelemetry provider maps each pair to an-`agent.channel.delivery` consumer span under the durable session window. When-`traceChannelRequests: true` creates an inbound HTTP server span, the delivery-span links to it with `eve.link.type=channel.request` rather than using the-short-lived request span as its parent.+For a single delivery that starts a turn, the built-in OpenTelemetry provider records the channel kind, channel name, delivery ID, optional request ID, and captured input on that turn's `invoke_agent` activation. The activation remains a root in its own trace and links to the active upstream request or function span with `eve.link.type=channel.request`. Set `traceChannelRequests: true` to create an eve-owned HTTP server span as that link target; when the option is false, an already-active upstream span remains the target.++Deliveries that do not map one-to-one to an activation still produce instrumentation lifecycle events, but the built-in OpenTelemetry provider does not emit a separate delivery span.## Callback delivery errors@@ -103,12 +103,12 @@ not mark the active span as failed.The built-in OpenTelemetry provider preserves each span's OTel name and adds`operation.name` and `resource.name` for Datadog's operation/resource mapping:-| OTel span name | `operation.name` | `resource.name` |-| ------------------------------------------------------------------------------------------------------------------ | ----------------- | ---------------------- |-| `invoke_agent weather` | `invoke_agent` | `invoke_agent weather` |-| `execute_tool search` | `execute_tool` | `execute_tool search` |-| `chat <model>` | `chat` | `chat <model>` |-| `agent.session`, `agent.step`, `agent.action`, `agent.approval`, `agent.channel.delivery`, `agent.channel.request` | Same as span name | Same as span name |+| OTel span name | `operation.name` | `resource.name` |+| ----------------------------------------------------------------------- | ----------------- | ---------------------- |+| `invoke_agent weather` | `invoke_agent` | `invoke_agent weather` |+| `execute_tool search` | `execute_tool` | `execute_tool search` |+| `chat <model>` | `chat` | `chat <model>` |+| `agent.step`, `agent.action`, `agent.approval`, `agent.channel.request` | Same as span name | Same as span name |These attributes apply to eve-owned spans in local tracing and the[instrumentation provider layout](./instrumentation-providers). They do not@@ -137,6 +137,56 @@ that discard span events, including Sentry's [direct OTLPintake](https://docs.sentry.io/concepts/otlp/direct/traces/). These attributesremain available when model and tool content is redacted.+## Agent trace contract++The provider layout and zero-config local tracing emit the following spans.+The legacy `instrumentation.ts` layout still uses its authored OTel setup.++| Span | Meaning |+| ----------------------- | -------------------------------------------------------- |+| `invoke_agent <agent>` | One agent activation in its own trace |+| `agent.step` | One model attempt |+| `agent.action` | Durable action lifecycle, including dispatch and waiting |+| `execute_tool <tool>` | In-process tool execution beneath its action |+| `agent.approval` | Approval waiting beneath its action |+| `agent.channel.request` | Optional HTTP request span in the provider layout |++Schema v4 removes the session-long `agent.session` root and duplicate agent session and lineage attributes. Every eve span carries `agent.trace.schema.version=4` and `gen_ai.conversation.id`; Vercel deployments additionally carry `vercel.session_id`.+Only activations use the `invoke_agent` operation. Dispatch lifecycle spans use+`agent.action` with `agent.invocation.role=caller`; the built-in `agent` tool+executes under `execute_tool agent`. Turn IDs such as `turn_0` are local to a session,+so correlate turns by session ID and turn ID together.++Each activation owns a fresh trace identity, including local and remote+subagents. The first child activation links to its caller with+`eve.link.type=agent.dispatch`; incoming `traceparent` provides that link, not+the child trace ID. Later turns do not reuse the original caller link.+Conversation baggage is independent of execution lineage and trace-policy+ceilings, which remain in force across the boundary.++Channel delivery does not allocate or replace the activation, and a later turn+receives a new trace. Samplers+see the activation name and attributes after its session coordinates are+available. Sampling must be deterministic for the same trace and operation+because durable reconstruction can evaluate it again.++Activation spans retain `gen_ai.usage.input_tokens` and+`gen_ai.usage.output_tokens` alongside `agent.usage.*` totals for that activation.+Model spans carry `gen_ai.usage.*`; step and dispatch spans report+`agent.usage.*`. Do not sum usage across these levels.++Nested dispatch spans are materialized when the child settles and handed to+span processors; successful materialization removes their completed records+from durable state. Settlement does not drain exporters or authored providers.+Materialization failures are logged without failing settlement, and exporter+draining follows the runtime's normal flush lifecycle.++Cancelled or abandoned dispatches retain+their outcome without error status. Error messages and stacks on eve spans count as+output content, including errors reconstructed after a worker replacement.+Metadata-only capture retains failure status without those details. Error logging+outside eve's instrumented execution retains its existing exception content.+## Runtime context_Runtime context_ is an [AI SDK concept](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text): a user-defined object that flows through a generation lifecycle. eve exposes it through `events["step.started"]`, a callback that runs once eve has assembled the model input for an attempt and returns `{ runtimeContext }`. Because eve registers the AI SDK's OpenTelemetry integration with runtime context enabled, those returned values ride onto the model-call span and its children. The field is named `runtimeContext`, not `metadata`, because AI SDK v7 carries per-call attributes on runtime context rather than a dedicated metadata field.@@ -198,7 +248,7 @@ eve creates the `ai.eve.turn` parent span per turn and passes enriched telemetryThis hierarchy applies when eve passes telemetry to the AI SDK. When the `otel()` provider layout is declared and eve owns the agent spans, eve names its invocation span `invoke_agent <agent>` and its model-attempt spans `agent.step`. Session, turn, step, and channel context is injected as the framework half of the runtime context (`eve.version`, `eve.session.id`, `eve.environment`, `eve.turn.id`, `eve.turn.sequence`, `eve.step.index`, `eve.channel.kind`) and rides onto the spans alongside any values your `events["step.started"]` callback returns under `runtimeContext`.-Set `traceChannelRequests: true` on `defineInstrumentation` to also wrap each inbound channel HTTP request in a single OpenTelemetry `SERVER` span named for the registered route, which parents the turn tree above (and any `hook.resume` and outgoing HTTP spans):+Set `traceChannelRequests: true` on `defineInstrumentation` to also wrap each inbound channel HTTP request in a single OpenTelemetry `SERVER` span named for the registered route. In the authored hierarchy above, this span parents the turn tree and any `hook.resume` or outgoing HTTP spans. In the provider layout, `invoke_agent` remains a separate trace root and links to the request span.```textPOST /eve/v1/session/:sessionId@@ -238,7 +288,7 @@ These tags power the **Agent Runs** tab in the Vercel dashboard. When you deploy## Local traces-Without an `instrumentation.ts`, `eve dev` records spans to disk — one trace per session, with turns, model steps, and tool calls. Read them two ways:+Without an `instrumentation.ts`, `eve dev` records spans to disk with one bounded trace per turn, including its model steps and tool calls. Read them two ways:- [`/traces`](dev-tui#logs-and-traces) in the dev TUI: a live trace viewer that replays captured content as a conversation.- [`eve traces`](../reference/cli#eve-traces): a span tree in the terminal, `eve traces ls` to list. Works after `eve dev` exits.diff --git a/docs/guides/remote-agents.md b/docs/guides/remote-agents.mdindex 559b86032..fd75ecc4a 100644--- a/docs/guides/remote-agents.md+++ b/docs/guides/remote-agents.md@@ -141,9 +141,14 @@ Forwarding is explicit on both sides. The receiver names which forwarders it truRemote tracing uses ordinary `traceparent` propagation. Trace context is not an authorization grant: it cannot assert eve parent lineage, change `rootSessionId`, or remove the normal root-session token cap.-eve also carries the original `gen_ai.conversation.id` in `eve.conversation.id` baggage so you can find related traces across local and remote agents. This observability identifier does not require principal forwarding or shared execution lineage; it grants no session access and does not change trace-content policy. Only session-create requests with a callback may supply this baggage; top-level requests ignore it and establish their own conversation ID. Callback metadata is caller-supplied correlation, not verified identity.+eve also carries the original `gen_ai.conversation.id` in `eve.conversation.id` baggage so you can find related traces across local and remote agents. This observability identifier does not require principal forwarding or shared execution lineage; it grants no session access and does not change trace-content policy. Session execution establishes this ID even without an instrumentation runtime. Only session-create requests with a callback may supply this baggage; top-level requests ignore it and establish their own conversation ID. Callback metadata is caller-supplied correlation, not verified identity.-eve replaces configured `traceparent` and conversation baggage only when it has a framework value to send. Incoming baggage has an 8 KiB limit; the audience and conversation readers share whitespace, percent-decoding, and duplicate-key validation. Conversation IDs are limited to 1 KiB and exclude control characters and line separators.+eve replaces configured `traceparent` and conversation baggage only when it has a framework value to send. Incoming baggage has an 8 KiB limit; the audience and conversation readers share whitespace, percent-decoding, and duplicate-key validation. Conversation IDs are limited to 1 KiB and exclude control characters and line separators. If adding an audience assertion or conversation ID would exceed 8 KiB, eve rejects the dispatch before sending a request instead of dropping the member. Reduce `remote.headers.baggage` to leave room for these framework values.++In the [provider trace contract](./instrumentation#agent-trace-contract), each+child activation starts a separate trace. The first activation uses the incoming+`traceparent` as an `agent.dispatch` span link rather than adopting the caller's+trace ID.> ⚠️ **Upgrade both deployments before resuming persistent remote sessions.** A sender with continuation forwarding includes `forwardedPrincipal` on each authenticated follow-up. A receiver that supports forwarding only on session creation rejects that continuation with HTTP 400. eve does not retry without the field because that would run the follow-up as the transport service principal and silently change caller authority. The parent retains the child handle after this failure, so you can retry the same session after upgrading the receiver.diff --git a/docs/reference/cli.md b/docs/reference/cli.mdindex c8bc556ff..5f6e461c8 100644--- a/docs/reference/cli.md+++ b/docs/reference/cli.md@@ -304,23 +304,25 @@ eve traces --verbose # expand every span with all attributes and eventseve traces --json # dump the full trace as JSON```-Reads the immutable OTLP/JSON segments under `.eve/traces/v1`, so `eve dev` need not be running. Accepts a full trace id, an `agent.session.id`, or an unambiguous prefix of either. Malformed segments are skipped without hiding valid spans from the same trace.+Reads the immutable OTLP/JSON segments under `.eve/traces/v1`, so `eve dev` need not be running. Accepts a full trace id, a `gen_ai.conversation.id`, or an unambiguous prefix of either. Malformed segments are skipped without hiding valid spans from the same trace.Span rows carry inline metrics when the span recorded them — `↑input`/`↓output` token counts, gateway cost, and the tool name for `execute_tool` spans — and the header aggregates models, token totals, cost, and error count across the trace's step spans. `--verbose` expands each span under its tree row: status (with the error message on failures), timing, ids, every attribute (prompts, responses, and tool payloads as transcripts or pretty-printed JSON), and every span event with its offset from span start. `--json` prints the same records as JSON, one object per selected trace.-A local subagent keeps its own session id but records into the parent trace. Its `invoke_agent` span is parented to the dispatching action. The attributes `agent.root_run.id`, `agent.parent_run.id`, and `agent.parent_call.id` identify the root session, immediate parent session, and dispatch call; `agent.subagent.name` labels the child invocation. Either session id resolves to that trace. Remote agents propagate caller trace context over `traceparent` without sharing execution lineage.+Every subagent activation starts its own trace. The first child's `invoke_agent` root links to the dispatching caller with `eve.link.type=agent.dispatch`; remote agents carry that caller context over `traceparent`. Later turns also start fresh traces without repeating the initial caller link. All related sessions retain the same `gen_ai.conversation.id`, and `agent.subagent.name` labels the child invocation.-A durable session keeps one persisted trace context across turns and worker resumptions. Independently replayed attempts can still produce another trace; passing the session id shows every trace it produced, oldest first.+Each `agent()` call inside an authored workflow has its own `agent.action` caller span, including sequential, parallel, and background calls. The workflow tool keeps its own `agent.action` and `execute_tool` spans. Only agent execution uses `invoke_agent`.-Every span carries a real duration except `agent.session`: an idle session never closes, so it is recorded as a zero-duration marker and the span tree shows its descendant extent instead. A turn's `invoke_agent` span is written when the turn settles, so a running turn shows only its steps.+A durable conversation produces one bounded trace per turn. Worker replacements reuse the prepared context for the same turn, while a later turn or an independently replayed attempt starts a fresh trace. Passing the conversation ID shows every trace it produced, oldest first.++Every span carries a real duration. A turn's root `invoke_agent` span is written when the turn settles, so a running turn shows only its steps.Model and `execute_tool` spans omit their inputs and outputs by default. Set `EVE_TRACES_CONTENT=on` to capture system prompts, prompt messages, and response text for models, plus call arguments and results for tools. Each captured value is capped at 32 KB.-Step spans carry token counts, and cost when Vercel AI Gateway served the call. Both follow the [OTel GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) (`gen_ai.usage.*`), so a third-party backend reads them without mapping.+Step spans carry token counts under `agent.usage.*`, and cost when Vercel AI Gateway served the call. Model spans also expose `gen_ai.usage.*` token counters. The CLI sums step-level counters only, so model and delegated-call totals are not counted twice.### Retention-eve sweeps the store when a session finishes and when the dev server starts, evicting oldest-first past the bounds below — except that the newest traces and anything written in the last five minutes are always kept, so a sweep will exceed the size budget rather than drop a trace you just recorded. Set the bounds in `.env.local`, which `eve dev` loads automatically; each accepts `off` to disable it individually.+eve sweeps the store when an activation's writes finish, when a session finishes, and when the dev server starts. An open conversation does not pin every completed turn's trace. Sweeps evict oldest-first past the bounds below, except that active traces, the newest traces, and anything written in the last five minutes are kept. A sweep can therefore exceed the size budget. Set the bounds in `.env.local`, which `eve dev` loads automatically; each accepts `off` to disable it individually.| Variable | Default | Effect || ---------------------------- | -------------------- | ----------------------------------------------------------------------------------- |diff --git a/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.tsindex e5d5c24c3..f0831470a 100644--- a/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts+++ b/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts@@ -9,6 +9,18 @@ export default defineEval({turn.expectOk();turn.calledTool("fanout_agents", { count: 1, status: "completed" });turn.event("subagent.called", { data: { name: "workflow-marker" }, count: 2 });+ turn.eventsSatisfy("parallel invocations keep distinct calls and child sessions", (events) => {+ const calls = events.flatMap((event) =>+ event.type === "subagent.called" && event.data.name === "workflow-marker"+ ? [event.data]+ : [],+ );+ return (+ calls.length === 2 &&+ new Set(calls.map((call) => call.callId)).size === 2 &&+ new Set(calls.map((call) => call.childSessionId)).size === 2+ );+ });turn.messageIncludes("api:replica-0");turn.messageIncludes("api:replica-1");turn.eventsSatisfy("both children start before the waiting tool resolves", (events) => {diff --git a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.tsindex 3f7098171..0089b87a2 100644--- a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts+++ b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts@@ -44,6 +44,7 @@ export interface Tracer {}export interface Context {+ getValue(key: symbol): unknown;setValue(key: symbol, value: unknown): Context;}diff --git a/packages/eve/src/cli/commands/trace-detail.test.ts b/packages/eve/src/cli/commands/trace-detail.test.tsindex d1801caaa..4e4c9b9e9 100644--- a/packages/eve/src/cli/commands/trace-detail.test.ts+++ b/packages/eve/src/cli/commands/trace-detail.test.ts@@ -59,7 +59,7 @@ describe("summarizeLocalTrace", () => {"agent.model.id": "gpt-5","agent.usage.input_tokens": 1000,"agent.usage.output_tokens": 100,- "gen_ai.usage.cache_read.input_tokens": 800,+ "agent.usage.cache_read_tokens": 800,"gen_ai.usage.cost": 0.01,},}),diff --git a/packages/eve/src/cli/commands/trace-detail.ts b/packages/eve/src/cli/commands/trace-detail.tsindex 472601357..c3deef204 100644--- a/packages/eve/src/cli/commands/trace-detail.ts+++ b/packages/eve/src/cli/commands/trace-detail.ts@@ -11,6 +11,7 @@ import { formatAttributeContent } from "#cli/dev/tui/traces/trace-content.js";import { formatElapsed } from "#cli/format-elapsed.js";import { sanitizeForTerminal } from "#cli/ui/output.js";import type { LocalTraceSpan } from "#tracing/local-trace-reader.js";+import { AGENT_USAGE_ATTRIBUTES } from "#tracing/agent-span-contract.js";/** Usage and cost totals aggregated over a trace's `agent.step` spans. */export interface LocalTraceSummary {@@ -65,8 +66,14 @@ export function summarizeLocalTrace(spans: readonly LocalTraceSpan[]): LocalTracif (span.name !== "agent.step") continue;inputTokens += numberAttribute(span, "agent.usage.input_tokens") ?? 0;outputTokens += numberAttribute(span, "agent.usage.output_tokens") ?? 0;- cacheReadTokens += numberAttribute(span, "gen_ai.usage.cache_read.input_tokens") ?? 0;- cacheWriteTokens += numberAttribute(span, "gen_ai.usage.cache_creation.input_tokens") ?? 0;+ cacheReadTokens +=+ numberAttribute(span, AGENT_USAGE_ATTRIBUTES.cacheReadTokens) ??+ numberAttribute(span, "gen_ai.usage.cache_read.input_tokens") ??+ 0;+ cacheWriteTokens +=+ numberAttribute(span, AGENT_USAGE_ATTRIBUTES.cacheWriteTokens) ??+ numberAttribute(span, "gen_ai.usage.cache_creation.input_tokens") ??+ 0;const cost = spanCostUsd(span);if (cost !== undefined) costUsd = (costUsd ?? 0) + cost;}diff --git a/packages/eve/src/cli/commands/trace.integration.test.ts b/packages/eve/src/cli/commands/trace.integration.test.tsindex f1483241d..5ce9ef89d 100644--- a/packages/eve/src/cli/commands/trace.integration.test.ts+++ b/packages/eve/src/cli/commands/trace.integration.test.ts@@ -25,7 +25,7 @@ describe("eve traces", () => {TRACE_ONE,span("a", "agent.turn", 10, 20, undefined, {"agent.name": "weather",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(@@ -33,7 +33,7 @@ describe("eve traces", () => {TRACE_TWO,span("b", "agent.turn", 30, 40, undefined, {"agent.name": "research",- "agent.session.id": "session-two",+ "gen_ai.conversation.id": "session-two",}),);await writeFile(@@ -48,7 +48,7 @@ describe("eve traces", () => {const traces = await listLocalTraces(root);expect(traces.map((trace) => trace.traceId)).toEqual([TRACE_TWO, TRACE_ONE]);- expect(traces[0]).toMatchObject({ agentName: "research", sessionId: "session-two" });+ expect(traces[0]).toMatchObject({ agentName: "research", conversationId: "session-two" });});it("resolves trace ids, session ids, and unambiguous prefixes", async () => {@@ -57,14 +57,14 @@ describe("eve traces", () => {root,TRACE_ONE,span("a", "agent.turn", 10, 20, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(root,TRACE_TWO,span("b", "agent.turn", 20, 30, undefined, {- "agent.session.id": "session-two",+ "gen_ai.conversation.id": "session-two",}),);const traces = await listLocalTraces(root);@@ -82,14 +82,14 @@ describe("eve traces", () => {root,TRACE_TWO,span("b", "agent.session", 100, 100, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(root,TRACE_ONE,span("a", "agent.session", 10, 10, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);const traces = await listLocalTraces(root);@@ -112,20 +112,20 @@ describe("eve traces", () => {root,TRACE_ONE,span(session, "agent.session", 10, 10, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(root,TRACE_ONE,span(child, "agent.turn", 20, 30, session, {- "agent.session.id": "child-one",+ "gen_ai.conversation.id": "child-one",}),);const traces = await listLocalTraces(root);// The opener still names the trace, so `eve trace ls` reads unchanged.- expect(traces[0]!.sessionId).toBe("session-one");+ expect(traces[0]!.conversationId).toBe("session-one");expect(ids(resolveLocalTraces(traces, "child-one"))).toEqual([TRACE_ONE]);expect(ids(resolveLocalTraces(traces, "session-one"))).toEqual([TRACE_ONE]);});@@ -140,7 +140,7 @@ describe("eve traces", () => {TRACE_ONE,span(turn, "invoke_agent weather", 10, 100, undefined, {"agent.name": "weather",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one","agent.turn.id": "turn-1","gen_ai.operation.name": "invoke_agent",}),@@ -149,7 +149,7 @@ describe("eve traces", () => {root,TRACE_ONE,span(step, "agent.step", 20, 90, turn, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one","agent.step.attempt": 0,"agent.step.index": 0,}),@@ -160,7 +160,7 @@ describe("eve traces", () => {span(action, "agent.action", 30, 80, step, {"agent.action.kind": "tool-call","agent.action.name": "\u001B[31mweather\u001B[0m",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(root, TRACE_ONE, {@@ -188,24 +188,24 @@ describe("eve traces", () => {root,TRACE_ONE,span(session, "agent.session", 10, 10, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);await writeSegment(root,TRACE_ONE,- span(turn, "agent.turn", 10, 95, session, { "agent.session.id": "session-one" }),+ span(turn, "agent.turn", 10, 95, session, { "gen_ai.conversation.id": "session-one" }),);await writeSegment(root,TRACE_ONE,- span(step, "agent.step", 20, 90, turn, { "agent.session.id": "session-one" }),+ span(step, "agent.step", 20, 90, turn, { "gen_ai.conversation.id": "session-one" }),);await writeSegment(root,TRACE_ONE,span(orphanMarker, "user.marker", 95, 95, turn, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);const output = collectingLogger();@@ -226,7 +226,7 @@ describe("eve traces", () => {TRACE_ONE,span("a", "agent.step", 10, 80, undefined, {"agent.model.id": "gpt-5",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one","agent.step.attempt": 0,"agent.step.index": 0,"agent.usage.input_tokens": 1400,@@ -261,7 +261,7 @@ describe("eve traces", () => {await writeSegment(root, TRACE_ONE, {...span("a", "agent.step", 10, 80, undefined, {"agent.model.id": "gpt-5",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one","agent.step.index": 0,}),events: [@@ -302,7 +302,7 @@ describe("eve traces", () => {await writeSegment(root, TRACE_ONE, {...span("a", "agent.step", 10, 80, undefined, {"agent.model.id": "gpt-5",- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),events: [{ name: "step.started", timeUnixNano: "10000000" }],});@@ -347,7 +347,7 @@ describe("eve traces", () => {root,TRACE_ONE,span("a", "agent.turn", 10, 20, undefined, {- "agent.session.id": "session-one",+ "gen_ai.conversation.id": "session-one",}),);const latest = collectingLogger();@@ -356,7 +356,7 @@ describe("eve traces", () => {const json = collectingLogger();await runTraceListCommand(json.logger, root, { json: true });expect(JSON.parse(json.out[0]!)).toEqual([- expect.objectContaining({ sessionId: "session-one", spanCount: 1, traceId: TRACE_ONE }),+ expect.objectContaining({ conversationId: "session-one", spanCount: 1, traceId: TRACE_ONE }),]);});diff --git a/packages/eve/src/cli/commands/trace.ts b/packages/eve/src/cli/commands/trace.tsindex 3100da7a6..afaf6872b 100644--- a/packages/eve/src/cli/commands/trace.ts+++ b/packages/eve/src/cli/commands/trace.ts@@ -29,8 +29,8 @@ interface SpanExtent {}/**- * Resolves an exact trace/session id or an unambiguous prefix. One session id- * can match several traces, since a long session windows into more than one.+ * Resolves an exact trace/conversation id or an unambiguous prefix. One+ * conversation ID can match several traces, since each turn is separate.*/export function resolveLocalTraces(traces: readonly LocalTrace[],@@ -41,16 +41,16 @@ export function resolveLocalTraces(const references = raw === normalized ? [raw] : [raw, normalized];const exactTrace = traces.find((trace) => references.includes(trace.traceId));if (exactTrace !== undefined) return [exactTrace];- const exactSessions = traces.filter((trace) =>- trace.sessionIds.some((sessionId) => references.includes(sessionId)),+ const exactConversations = traces.filter((trace) =>+ trace.conversationIds.some((conversationId) => references.includes(conversationId)),);- if (exactSessions.length > 0) return orderWindows(exactSessions);+ if (exactConversations.length > 0) return orderWindows(exactConversations);const matches = traces.filter((trace) =>references.some((candidate) =>trace.traceId.startsWith(candidate) ||- trace.sessionIds.some((sessionId) => sessionId.startsWith(candidate)),+ trace.conversationIds.some((conversationId) => conversationId.startsWith(candidate)),),);if (matches.length === 0) {@@ -59,8 +59,8 @@ export function resolveLocalTraces();}if (matches.length === 1) return matches;- const sessions = new Set(matches.map((trace) => trace.sessionId ?? trace.traceId));- if (sessions.size > 1) throw ambiguousTraceError(matches, reference);+ const conversations = new Set(matches.map((trace) => trace.conversationId ?? trace.traceId));+ if (conversations.size > 1) throw ambiguousTraceError(matches, reference);return orderWindows(matches);}@@ -86,7 +86,7 @@ export async function runTraceListCommand(traces.map((trace) => ({agentName: trace.agentName ?? null,durationMs: durationMs(trace.startTimeNs, trace.endTimeNs),- sessionId: trace.sessionId ?? null,+ conversationId: trace.conversationId ?? null,spanCount: trace.spans.length,startedAt: toDate(trace.startTimeNs).toISOString(),traceId: trace.traceId,@@ -104,13 +104,13 @@ export async function runTraceListCommand(const rows = traces.map((trace) => [trace.traceId,- sanitizeForTerminal(trace.sessionId ?? "unknown"),+ sanitizeForTerminal(trace.conversationId ?? "unknown"),sanitizeForTerminal(trace.agentName ?? "unknown"),toDate(trace.startTimeNs).toISOString(),formatElapsed(durationMs(trace.startTimeNs, trace.endTimeNs)),String(trace.spans.length),]);- const headers = ["TRACE", "SESSION", "AGENT", "STARTED", "DURATION", "SPANS"];+ const headers = ["TRACE", "CONVERSATION", "AGENT", "STARTED", "DURATION", "SPANS"];const widths = headers.map((header, index) =>Math.max(header.length, ...rows.map((row) => row[index]!.length)),);@@ -167,7 +167,7 @@ function traceHeaderRows(trace: LocalTrace): { label: string; value: string }[]const summary = summarizeLocalTrace(trace.spans);return [{ label: "Trace ID", value: trace.traceId },- { label: "Session ID", value: trace.sessionId ?? "unknown" },+ { label: "Conversation ID", value: trace.conversationId ?? "unknown" },{ label: "Agent", value: trace.agentName ?? "unknown" },{ label: "Started", value: toDate(trace.startTimeNs).toISOString() },{@@ -197,8 +197,8 @@ function serializeTraceForJson(trace: LocalTrace): Record<string, unknown> {return {agentName: trace.agentName ?? null,durationMs: durationMs(trace.startTimeNs, trace.endTimeNs),- sessionId: trace.sessionId ?? null,- sessionIds: trace.sessionIds,+ conversationId: trace.conversationId ?? null,+ conversationIds: trace.conversationIds,spanCount: trace.spans.length,spans: trace.spans.map((span) => ({attributes: span.attributes,@@ -283,11 +283,8 @@ function renderSpanTree(/*** Extent of each span's own range unioned with its descendants', keyed by span id.*- * eve records a span with no guaranteed close — an `agent.session` root,- * whose session may idle forever — as a zero-duration marker, because- * a span object cannot cross a durable worker boundary to be ended later.- * The tree falls back to this extent for those rows so it shows where the- * time went.+ * Older schemas recorded zero-duration session markers. Include descendant+ * time when rendering those stored traces.*/function subtreeExtents(spans: readonly LocalTraceSpan[],@@ -350,9 +347,9 @@ function ambiguousTraceError(traces: readonly LocalTrace[], reference: string):`"${sanitizeForTerminal(reference)}" matches ${traces.length} local traces:`,...traces.map((trace) =>- ` ${trace.traceId} ${sanitizeForTerminal(trace.sessionId ?? "unknown session")}`,+ ` ${trace.traceId} ${sanitizeForTerminal(trace.conversationId ?? "unknown conversation")}`,),- "Pass a longer prefix or the full trace/session id.",+ "Pass a longer prefix or the full trace/conversation id.",].join("\n"),);}diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.tsindex 8c7f9789e..681332cd4 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts@@ -35,31 +35,19 @@ function span(function trace(spans: readonly LocalTraceSpan[]): LocalTrace {return {endTimeNs: spans.reduce((m, s) => (s.endTimeNs > m ? s.endTimeNs : m), 0n),- sessionIds: [],+ conversationIds: [],spans,startTimeNs: spans.reduce((m, s) => (s.startTimeNs < m ? s.startTimeNs : m), BASE),traceId: "t".repeat(32),};}-function delivery(- spanId: string,- startMs: number,- turnId: string,- message: string,-): LocalTraceSpan {- return span(spanId, "agent.channel.delivery", startMs, startMs, undefined, {- "agent.channel.delivery.input": JSON.stringify({ message }),- "agent.turn.id": turnId,- });-}-function weatherTurn(): LocalTraceSpan[] {const turn = span("a".repeat(16), "invoke_agent weather", 0, 0, undefined, {+ "agent.channel.delivery.input": JSON.stringify({ message: "weather in sf?" }),"agent.turn.id": "turn_0","gen_ai.operation.name": "invoke_agent",});- const inbound = delivery("0".repeat(16), 0, "turn_0", "weather in sf?");const step = span("b".repeat(16), "agent.step", 10, 5000, turn.spanId, {});const messages = JSON.stringify([{ role: "user", content: "weather in nyc?" },@@ -78,22 +66,24 @@ function weatherTurn(): LocalTraceSpan[] {"gen_ai.tool.call.arguments": '{"city":"sf"}',"gen_ai.tool.call.result": '{"temperatureF":72}',});- return [turn, inbound, step, model, action];+ return [turn, step, model, action];}describe("buildConversationItems", () => {- it("reads the user message from the channel delivery", () => {+ it("reads the user message from activation delivery metadata", () => {const items = buildConversationItems(trace(weatherTurn()));expect(items[0]?.kind).toBe("user");expect(items[0]?.text).toBe("weather in sf?");- expect(items[0]?.span.name).toBe("agent.channel.delivery");+ expect(items[0]?.span.name).toBe("invoke_agent weather");});- it("skips metadata-only channel deliveries", () => {+ it("skips metadata-only activation deliveries", () => {const items = buildConversationItems(trace([- span("a".repeat(16), "agent.channel.delivery", 0, 0, undefined, {+ span("a".repeat(16), "invoke_agent weather", 0, 0, undefined, {"agent.channel.delivery.input": JSON.stringify({ context: ["sidebar"] }),+ "agent.turn.id": "turn_0",+ "gen_ai.operation.name": "invoke_agent",}),]),);@@ -173,11 +163,11 @@ describe("buildConversationItems", () => {6000,parentAction.spanId,{+ "agent.channel.delivery.input": JSON.stringify({ message: "delegated task" }),"agent.turn.id": "turn_child","gen_ai.operation.name": "invoke_agent",},);- const childDelivery = delivery("3".repeat(16), 6000, "turn_child", "delegated task");const childStep = span("1".repeat(16), "agent.step", 6010, 8000, childTurn.spanId, {});const childModel = span("2".repeat(16),@@ -191,7 +181,7 @@ describe("buildConversationItems", () => {},);const items = buildConversationItems(- trace([...parent, parentAction, childTurn, childDelivery, childStep, childModel]),+ trace([...parent, parentAction, childTurn, childStep, childModel]),);const parentItems = items.filter((item) => item.subagent === undefined);const childItems = items.filter((item) => item.subagent !== undefined);@@ -204,14 +194,32 @@ describe("buildConversationItems", () => {});});+ it("labels a separately rooted child without requiring the caller trace", () => {+ const spans = weatherTurn().map((item) =>+ item.attributes["gen_ai.operation.name"] === "invoke_agent"+ ? {+ ...item,+ attributes: {+ ...item.attributes,+ "agent.subagent.name": "research",+ },+ }+ : item,+ );+ const items = buildConversationItems(trace(spans));+ expect(items.length).toBeGreaterThan(0);+ expect(items.every((item) => item.subagent?.name === "research")).toBe(true);+ });+it("interleaves a subagent's cards between the parent's dispatch and reply", () => {// The parent parks while the child runs: step 1 dispatches, the child// works, step 2 replies with the result. Cards must read in that order,// not with the child appended after the parent's whole turn.- const turn = span("a".repeat(16), "agent.turn", 0, 0, undefined, {+ const turn = span("a".repeat(16), "invoke_agent parent", 0, 0, undefined, {+ "agent.channel.delivery.input": JSON.stringify({ message: "run the subagent" }),"agent.turn.id": "turn_0",+ "gen_ai.operation.name": "invoke_agent",});- const parentDelivery = delivery("0".repeat(16), 0, "turn_0", "run the subagent");const step1 = span("b".repeat(16), "agent.step", 10, 20, turn.spanId, {});const dispatch = span("c".repeat(16), "ai.streamText.doStream", 12, 18, step1.spanId, {"ai.prompt.messages": JSON.stringify([{ role: "user", content: "run the subagent" }]),@@ -223,10 +231,18 @@ describe("buildConversationItems", () => {"agent.action.name": "echo-marker","agent.turn.id": "turn_0",});- const childTurn = span("f".repeat(16), "agent.turn", 30, 30, parentAction.spanId, {- "agent.turn.id": "turn_child",- });- const childDelivery = delivery("5".repeat(16), 30, "turn_child", "delegated task");+ const childTurn = span(+ "f".repeat(16),+ "invoke_agent echo-marker",+ 30,+ 30,+ parentAction.spanId,+ {+ "agent.channel.delivery.input": JSON.stringify({ message: "delegated task" }),+ "agent.turn.id": "turn_child",+ "gen_ai.operation.name": "invoke_agent",+ },+ );const childStep = span("1".repeat(16), "agent.step", 32, 40, childTurn.spanId, {});const childModel = span("2".repeat(16), "ai.streamText.doStream", 34, 38, childStep.spanId, {"ai.prompt.messages": JSON.stringify([{ role: "user", content: "delegated task" }]),@@ -237,19 +253,7 @@ describe("buildConversationItems", () => {"ai.response.text": "The subagent said: delegated reply",});const items = buildConversationItems(- trace([- turn,- parentDelivery,- step1,- dispatch,- parentAction,- childTurn,- childDelivery,- childStep,- childModel,- step2,- reply,- ]),+ trace([turn, step1, dispatch, parentAction, childTurn, childStep, childModel, step2, reply]),);expect(items.map((item) => [item.kind, item.subagent !== undefined])).toEqual([["user", false],diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.tsindex 53b5f60cc..5e7c8f8ee 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts@@ -1,14 +1,15 @@/*** Conversation view for the `/traces` viewer: the same trace, re-told as a* flow of user messages, assistant replies, and tool calls instead of a- * latency waterfall. Durable delivery and action spans provide the user and- * tool cards directly; model response spans provide assistant cards.+ * latency waterfall. Activation and action spans provide the user and tool+ * cards directly; model response spans provide assistant cards.*/import { formatElapsed } from "#cli/format-elapsed.js";import { clipVisible, stripTerminalControls, visibleLength } from "#cli/ui/terminal-text.js";import type { LocalTrace, LocalTraceSpan } from "#tracing/local-trace-reader.js";import { compareLocalTraceSpans, isAgentTurnSpan } from "#tracing/local-trace-reader.js";+import { agentTurnIdentity } from "#tracing/agent-span-contract.js";import { formatCompactTokenCount } from "../stream-format.js";import type { Theme } from "../theme.js";@@ -43,12 +44,12 @@ export interface ConversationItem {readonly error: boolean;}-/** Dispatch lineage for a subagent turn, read from its parent action span. */+/** Dispatch lineage available on a subagent activation or its caller. */export interface ConversationSubagent {/** Subagent name, when the turn arrived through the subagent adapter. */readonly name?: string;/** Turn id of the dispatching parent turn. */- readonly parentTurnId: string;+ readonly parentTurnId?: string;/** Tool call id of the dispatch, when recorded. */readonly parentCallId?: string;}@@ -56,13 +57,13 @@ export interface ConversationSubagent {/** Max rendered lines for one collapsed card payload (args, result, or text). */const CARD_PAYLOAD_LINES = 3;-/** Builds the conversation flow from eve's durable delivery, model, and action spans. */+/** Builds the conversation flow from eve's activation, model, and action spans. */export function buildConversationItems(trace: LocalTrace): ConversationItem[] {const byId = new Map(trace.spans.map((span) => [span.spanId, span]));const subagents = new Map<string, ConversationSubagent>();for (const span of trace.spans) {if (!isAgentTurnSpan(span)) continue;- const turnId = stringAttribute(span, "agent.turn.id");+ const turnId = agentTurnIdentity(span);const subagent = turnSubagent(span, byId);if (turnId !== undefined && subagent !== undefined) subagents.set(turnId, subagent);}@@ -87,14 +88,15 @@ export function buildConversationItems(trace: LocalTrace): ConversationItem[] {for (const span of trace.spans) {const subagent = subagentFor(span, subagents, byId);- if (span.name === "agent.channel.delivery") {- const text = deliveryText(stringAttribute(span, "agent.channel.delivery.input"));+ const deliveryInput = stringAttribute(span, "agent.channel.delivery.input");+ if (deliveryInput !== undefined) {+ const text = deliveryText(deliveryInput);if (text === undefined) continue;entries.push({item: {kind: "user",- durationMs: spanDurationMs(span),- error: span.statusCode === 2,+ durationMs: 0,+ error: false,span,subagent,text,@@ -185,20 +187,26 @@ function subagentFor(): ConversationSubagent | undefined {let current: LocalTraceSpan | undefined = span;while (current !== undefined) {- const turnId = stringAttribute(current, "agent.turn.id");+ const turnId = agentTurnIdentity(current);if (turnId !== undefined) return subagents.get(turnId);current = current.parentSpanId === undefined ? undefined : byId.get(current.parentSpanId);}return undefined;}-/** Reads subagent identity from the action span that directly parents its turn. */function turnSubagent(turn: LocalTraceSpan,byId: ReadonlyMap<string, LocalTraceSpan>,): ConversationSubagent | undefined {+ const subagentName = stringAttribute(turn, "agent.subagent.name");+ if (subagentName !== undefined) {+ const name = subagentName ?? stringAttribute(turn, "gen_ai.agent.name");+ return {+ name: name === undefined ? undefined : stripTerminalControls(name),+ };+ }const parent = turn.parentSpanId === undefined ? undefined : byId.get(turn.parentSpanId);- if (parent?.name !== "agent.action") return undefined;+ if (parent === undefined || parent.name !== "agent.action") return undefined;const kind = stringAttribute(parent, "agent.action.kind");if (kind !== "subagent-call" && kind !== "remote-agent-call") return undefined;const parentTurnId = stringAttribute(parent, "agent.turn.id");diff --git a/packages/eve/src/cli/dev/tui/traces/trace-view.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-view.test.tsindex db6b2f8b5..6d57838b4 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-view.test.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-view.test.ts@@ -72,8 +72,8 @@ function viewerState(const trace: LocalTrace = {agentName: "weather",endTimeNs: ends.reduce((a, b) => (b > a ? b : a), 0n),- sessionId: "session-123",- sessionIds: ["session-123"],+ conversationId: "session-123",+ conversationIds: ["session-123"],spans,startTimeNs: starts.reduce((a, b) => (b < a ? b : a), starts[0] ?? 0n),traceId,@@ -99,7 +99,7 @@ describe("renderTraceViewer", () => {const header = stripAnsi(frame.rows[1]!);expect(header).toContain("traces");expect(header).toContain("weather");- expect(header).toContain("session session-123");+ expect(header).toContain("conversation session-123");expect(header).toContain(clockTimeOf(BASE));expect(header).toContain("5 spans");expect(header).toContain("[1/1]");diff --git a/packages/eve/src/cli/dev/tui/traces/trace-view.ts b/packages/eve/src/cli/dev/tui/traces/trace-view.tsindex a4fd76a5f..3e392e70c 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-view.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-view.ts@@ -261,8 +261,8 @@ function renderHeader(state: TraceViewerState, options: RenderTraceViewerOptionsif (state.trace?.agentName !== undefined) {segments.push(stripTerminalControls(state.trace.agentName));}- if (state.trace?.sessionId !== undefined) {- segments.push(`session ${stripTerminalControls(state.trace.sessionId)}`);+ if (state.trace?.conversationId !== undefined) {+ segments.push(`conversation ${stripTerminalControls(state.trace.conversationId)}`);}if (state.trace !== undefined) {const end =diff --git a/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.test.tsindex a829cd53a..5aaf7e000 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.test.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.test.ts@@ -12,7 +12,7 @@ function span(spanId: string, sessionId: string): LocalTraceSpan {return {attributes: {"agent.name": "tester",- "agent.session.id": sessionId,+ "gen_ai.conversation.id": sessionId,"agent.turn.id": "turn_0",},endTimeNs: 1_000_000n,@@ -25,14 +25,14 @@ function span(spanId: string, sessionId: string): LocalTraceSpan {};}-function trace(traceId: string, sessionIds: readonly string[]): LocalTrace {- const spans = sessionIds.map((sessionId, index) =>- span(String(index).padStart(16, "0"), sessionId),+function trace(traceId: string, conversationIds: readonly string[]): LocalTrace {+ const spans = conversationIds.map((conversationId, index) =>+ span(String(index).padStart(16, "0"), conversationId),);return {endTimeNs: 1_000_000n,- sessionId: sessionIds[0],- sessionIds,+ conversationId: conversationIds[0],+ conversationIds,spans,startTimeNs: 1_000_000n,traceId,@@ -99,8 +99,8 @@ describe("TraceViewerSession drag copy", () => {const store = stubStore([{endTimeNs: 1_000_000n,- sessionId: "session-mine",- sessionIds: ["session-mine"],+ conversationId: "session-mine",+ conversationIds: ["session-mine"],spans: [turn, delivery, model],startTimeNs: 1_000_000n,traceId: "c".repeat(32),@@ -158,8 +158,8 @@ describe("TraceViewerSession background surfaces", () => {const store = stubStore([{endTimeNs: 1_000_000n,- sessionId: "session-mine",- sessionIds: ["session-mine"],+ conversationId: "session-mine",+ conversationIds: ["session-mine"],spans: [turn, model],startTimeNs: 1_000_000n,traceId: "c".repeat(32),@@ -201,8 +201,8 @@ describe("TraceViewerSession background surfaces", () => {const store = stubStore([{endTimeNs: 1_000_000n,- sessionId: "session-mine",- sessionIds: ["session-mine"],+ conversationId: "session-mine",+ conversationIds: ["session-mine"],spans: [turn, model],startTimeNs: 1_000_000n,traceId: "c".repeat(32),@@ -230,13 +230,13 @@ describe("TraceViewerSession background surfaces", () => {describe("TraceViewerSession session preference", () => {it("opens on the trace containing the current session, not the newest", async () => {// Trace ids are provider-generated, so the session's trace is found by- // matching sessionIds — here the session lives in the older trace.+ // matching conversation IDs — here the session lives in the older trace.const store = stubStore([trace("a".repeat(32), ["session-other"]),trace("b".repeat(32), ["session-mine"]),]);const frame = await openViewer(store, "session-mine");- expect(frame).toContain("session session-mine");+ expect(frame).toContain("conversation session-mine");expect(frame).toContain("[2/2]");});@@ -246,7 +246,7 @@ describe("TraceViewerSession session preference", () => {trace("b".repeat(32), ["session-other"]),]);const frame = await openViewer(store, "session-child");- expect(frame).toContain("session session-root");+ expect(frame).toContain("conversation session-root");expect(frame).toContain("[1/2]");});@@ -256,7 +256,7 @@ describe("TraceViewerSession session preference", () => {trace("b".repeat(32), ["session-older"]),]);const frame = await openViewer(store, "session-unseen");- expect(frame).toContain("session session-other");+ expect(frame).toContain("conversation session-other");expect(frame).toContain("[1/2]");});});diff --git a/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.ts b/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.tsindex 8864ecaea..6be2fce3b 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-viewer-session.ts@@ -70,9 +70,7 @@ export class TraceViewerSession {/*** Pending one-time preference for the current chat session. Trace ids are* provider-generated, so the session's trace is found by reading candidates- * newest-first and matching `sessionIds` — a windowed session resolves to- * its most recently active window, and a subagent session to the parent- * trace it recorded into.+ * newest-first and matching its root conversation ID.*/#preferSessionId?: string;#pollTimer?: ReturnType<typeof setInterval>;@@ -288,7 +286,7 @@ export class TraceViewerSession {for (const entry of entries) {const trace = await this.#store.read(entry.traceId);if (this.#disposed) return undefined;- if (trace?.sessionIds.includes(sessionId)) return entry.traceId;+ if (trace?.conversationIds.includes(sessionId)) return entry.traceId;}return undefined;}diff --git a/packages/eve/src/cli/dev/tui/traces/trace-viewer-state.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-viewer-state.test.tsindex 52d41cb6d..674ee2403 100644--- a/packages/eve/src/cli/dev/tui/traces/trace-viewer-state.test.ts+++ b/packages/eve/src/cli/dev/tui/traces/trace-viewer-state.test.ts@@ -34,7 +34,7 @@ function trace(spans: readonly LocalTraceSpan[], traceId = "t".repeat(32)): Locaconst ends = spans.map((s) => s.endTimeNs);return {endTimeNs: ends.reduce((a, b) => (b > a ? b : a), 0n),- sessionIds: [],+ conversationIds: [],spans,startTimeNs: starts.reduce((a, b) => (b < a ? b : a), starts[0] ?? 0n),traceId,diff --git a/packages/eve/src/execution/eve-workflow-attributes.ts b/packages/eve/src/execution/eve-workflow-attributes.tsindex 4b9d23120..7d241a329 100644--- a/packages/eve/src/execution/eve-workflow-attributes.ts+++ b/packages/eve/src/execution/eve-workflow-attributes.ts@@ -28,9 +28,9 @@* - `$eve.invocation_owner` — SHA-256 fingerprint of the invocation's initiating principal* - `$eve.is_trace_content_visible` — whether observability may read content-bearing workflow data* - `$eve.is_otel_trace_enabled` — whether hosted Agent Runs OTEL is enabled for the run- * - `$eve.trace_id` — trace id of the `agent.session` span, read from the pre-allocated- * trace seed in the serialized context. Present only when the trace is sampled;- * absence means no exported OTEL trace exists.+ * - `$eve.trace_id` — initial activation trace id from the pre-allocated seed.+ * Later activations have independent traces; query `gen_ai.conversation.id`+ * to find the conversation's exported traces.*/import { CHANNEL_CONTEXT_KEY_NAME } from "#context/key-names.js";diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts b/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.tsindex b14b02940..f76e3ca09 100644--- a/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts+++ b/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts@@ -194,6 +194,7 @@ describe("blocking workflow agent continuation", () => {]);const settled = await settleTaskAgentInvocationStep({+ serializedContext: dispatched.serializedContext ?? {},ownerId: "workflow-run-1",result: {callId,diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.test.ts b/packages/eve/src/execution/tools/subagent/invoke-step.test.tsindex 52dc0e63f..4630d5faf 100644--- a/packages/eve/src/execution/tools/subagent/invoke-step.test.ts+++ b/packages/eve/src/execution/tools/subagent/invoke-step.test.ts@@ -252,6 +252,7 @@ describe("task-owned agent settlement", () => {} as never);const settled = await settleTaskAgentInvocationStep({+ serializedContext: {},result: {callId: "call-1",kind: "subagent-result",@@ -344,6 +345,7 @@ describe("task-owned agent settlement", () => {} as never);const settled = await settleTaskAgentInvocationStep({+ serializedContext: {},ownerId: "workflow-run-1",result: {callId: "call-1",diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.ts b/packages/eve/src/execution/tools/subagent/invoke-step.tsindex 2deec3cbf..e4bf66a1b 100644--- a/packages/eve/src/execution/tools/subagent/invoke-step.ts+++ b/packages/eve/src/execution/tools/subagent/invoke-step.ts@@ -50,17 +50,24 @@ import {getTurnUsageState,setTurnUsageState,} from "#harness/turn-tag-state.js";+import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";+import {+ flushAgentInvocationTraces,+ settleAgentInvocationTrace,+} from "#tracing/agent-invocation-terminal.js";export type AgentInvocationDispatchResult =| {readonly kind: "dispatched";readonly agentId: string;readonly event: SubagentCalledStreamEvent;+ readonly serializedContext?: Record<string, unknown>;readonly sessionState: DurableSessionState;}| {readonly kind: "failed";readonly result: RuntimeSubagentResult;+ readonly serializedContext?: Record<string, unknown>;readonly sessionState: DurableSessionState;};@@ -72,7 +79,6 @@ export type TaskAgentInvocationDispatchResult =export async function dispatchAgentInvocation(input: {readonly callbackBaseUrl: string;readonly emit?: HandleEventFn;- readonly instrumentationCallId?: string;readonly replyTo: string;readonly request: AgentInvocationRequest;readonly serializedContext: Record<string, unknown>;@@ -93,6 +99,26 @@ export async function dispatchAgentInvocation(input: {if (entry === undefined) {throw new Error("Agent invocation produced no executable plan entry.");}+ if (entry.kind === "reject") {+ return {+ kind: "failed",+ result: entry.result,+ serializedContext: prepared.serializedContext,+ sessionState: input.sessionState,+ };+ }+ const invocationAction = entry.kind === "start" ? entry.target.action : entry.action;+ const tracing = prepareAgentInvocationTrace({+ channelMetadata: prepared.channelMetadata,+ invocation: invocationAction,+ ownerId: input.ownerId,+ serializedContext: prepared.serializedContext,+ sessionId: prepared.session.sessionId,+ sessionState: durableSession.state,+ startTimeMs: Date.now(),+ taskId: input.taskId,+ turnId: prepared.batch.event.turnId,+ });let session = prepared.session;const currentAgentHandles = (): readonly AgentHandle[] =>getAgentHandleStore(session.state)?.handles ?? [];@@ -101,15 +127,17 @@ export async function dispatchAgentInvocation(input: {session: projectToDurableSession(session),state: input.sessionState,});+ const failed = async (result: RuntimeSubagentResult): Promise<AgentInvocationDispatchResult> => ({+ kind: "failed",+ result,+ serializedContext: await flushAgentInvocationTraces(tracing.fail(result)),+ sessionState: sessionState(),+ });const applyHandleCommand = (command: AgentHandleStoreCommand): AgentHandleStoreCommandResult => {const applied = applyTaskAgentHandleCommand(session, command);session = applied.session;return applied.result;};- if (entry.kind === "reject") {- return { kind: "failed", result: entry.result, sessionState: sessionState() };- }-let outcome: DispatchOutcome;let agentId: string;if (entry.kind === "resume") {@@ -146,11 +174,7 @@ export async function dispatchAgentInvocation(input: {});claimed = readClaimedHandle(claim);if (claimed === undefined) {- return {- kind: "failed",- result: createTaskClaimError(entry.action, entry.agentId, claim),- sessionState: sessionState(),- };+ return failed(createTaskClaimError(entry.action, entry.agentId, claim));}}const bundle = createAgentContinuationBundle({@@ -218,16 +242,15 @@ export async function dispatchAgentInvocation(input: {channelMetadata: prepared.channelMetadata,currentSession: session,fanoutSize: prepared.fanoutSize,- instrumentationCallId: input.instrumentationCallId,initiatorAuth: prepared.initiatorAuth,localDevRequest: prepared.localDevRequest,parentContinuationToken: input.replyTo,activityObserver: prepared.activityObserver,sandboxSessionId: prepared.sandboxSessionId,- serializedContext: prepared.serializedContext,session,taskId: input.taskId,target: entry.target,+ trace: tracing.dispatch,});agentId = start.identity.id;if (outcome.kind === "error") {@@ -248,7 +271,7 @@ export async function dispatchAgentInvocation(input: {}if (outcome.kind === "error") {- return { kind: "failed", result: outcome.result, sessionState: sessionState() };+ return failed(outcome.result);}const action = entry.kind === "resume" ? entry.action : entry.target.action;@@ -282,7 +305,13 @@ export async function dispatchAgentInvocation(input: {if (input.emit !== undefined) {await input.emit(event);}- return { kind: "dispatched", agentId, event, sessionState: sessionState() };+ return {+ kind: "dispatched",+ agentId,+ event,+ serializedContext: tracing.serializedContext,+ sessionState: sessionState(),+ };}/**@@ -315,12 +344,24 @@ export async function settleTaskAgentInvocationStep(input: {readonly accumulateUsage?: boolean;readonly ownerId: string;readonly result: RuntimeSubagentChildResult;+ readonly serializedContext: Record<string, unknown>;readonly sessionState: DurableSessionState;readonly taskId?: string | undefined;-}): Promise<{ readonly sessionState: DurableSessionState }> {+}): Promise<{+ readonly serializedContext: Record<string, unknown>;+ readonly sessionState: DurableSessionState;+}> {"use step";const durable = await readDurableSession(input.sessionState);+ const serializedContext = await flushAgentInvocationTraces(+ settleAgentInvocationTrace({+ acceptedAtMs: Date.now(),+ result: input.result,+ serializedContext: input.serializedContext,+ sessionId: durable.sessionId,+ }),+ );const handles = getAgentHandleStore(durable.state)?.handles ?? [];const candidates = handles.filter((candidate) => candidate.phase === "claimed" && candidate.ownerId === input.ownerId,@@ -329,7 +370,9 @@ export async function settleTaskAgentInvocationStep(input: {candidates.find((candidate) => candidate.phase === "claimed" && candidate.callId === input.result.callId,) ?? (candidates.length === 1 ? candidates[0] : undefined);- if (handle?.phase !== "claimed") return { sessionState: input.sessionState };+ if (handle?.phase !== "claimed") {+ return { serializedContext, sessionState: input.sessionState };+ }const nextHandles =input.result.outcome.kind === "terminal"@@ -372,6 +415,7 @@ export async function settleTaskAgentInvocationStep(input: {);}return {+ serializedContext,sessionState: replaceDurableSessionSnapshot({ session, state: input.sessionState }),};}diff --git a/packages/eve/src/execution/tools/subagent/start.test.ts b/packages/eve/src/execution/tools/subagent/start.test.tsindex e3a23b4d7..442197951 100644--- a/packages/eve/src/execution/tools/subagent/start.test.ts+++ b/packages/eve/src/execution/tools/subagent/start.test.ts@@ -3,57 +3,25 @@ import { beforeEach, describe, expect, it, vi } from "vitest";import { startSubagent } from "#execution/tools/subagent/start.js";import { startLocalSubagent } from "#subagents/start-local.js";import { startRemoteSubagent } from "#subagents/start-remote.js";-import {- readActionTraceContext,- readSessionTraceContext,-} from "#tracing/agent-trace-context-store.js";vi.mock("#subagents/start-local.js", () => ({ startLocalSubagent: vi.fn() }));vi.mock("#subagents/start-remote.js", () => ({ startRemoteSubagent: vi.fn() }));-vi.mock("#tracing/agent-trace-context-store.js", () => ({- readActionTraceContext: vi.fn(),- readSessionTraceContext: vi.fn(),-}));beforeEach(() => {- vi.resetAllMocks();+ vi.clearAllMocks();vi.mocked(startLocalSubagent).mockResolvedValue({ kind: "error" } as never);vi.mocked(startRemoteSubagent).mockResolvedValue({ kind: "error" } as never);});-describe.each(["local", "remote"] as const)("startSubagent (%s)", (kind) => {- it.each(["caller", "action", "session", "direct"] as const)(- "applies the live audience ceiling to the selected %s context",- async (source) => {+describe("startSubagent", () => {+ it.each(["local", "remote"] as const)(+ "passes one parent context to the %s child with the exact caller span",+ async (kind) => {const caller = {- decision: { action: "record", recordInputs: true, recordOutputs: true } as const,- forwardedTracePolicy: {- originAudience: "public",- ceiling: { recordInputs: true, recordOutputs: true },- } as const,spanId: "2".repeat(16),traceFlags: 1,traceId: "1".repeat(32),};- const childAction = {- ...caller,- spanId: "4".repeat(16),- traceFlags: 1,- traceId: "3".repeat(32),- };- const session = { ...caller, spanId: "6".repeat(16), traceId: "5".repeat(32) };- vi.mocked(readActionTraceContext).mockImplementation(- (_serializedContext, _sessionId, _turnId, callId) =>- callId === "workflow-call"- ? source === "caller"- ? caller- : undefined- : source === "session"- ? undefined- : childAction,- );- vi.mocked(readSessionTraceContext).mockReturnValue(session);- const selected = source === "caller" ? caller : source === "session" ? session : childAction;await startSubagent({auth: null,@@ -61,15 +29,14 @@ describe.each(["local", "remote"] as const)("startSubagent (%s)", (kind) => {bundle: {} as never,callbackBaseUrl: "https://parent.example",capabilities: undefined,- channelMetadata: { kind: "http", metadata: { audience: "private" } },+ channelMetadata: undefined,currentSession: {} as never,fanoutSize: 1,- instrumentationCallId: source === "direct" ? undefined : "workflow-call",initiatorAuth: null,parentContinuationToken: "parent-token",sandboxSessionId: "parent-session",- serializedContext: {},session: { rootSessionId: "root-session", sessionId: "parent-session" } as never,+ trace: { originAudience: "private", parentTraceContext: caller },target:kind === "local"? {@@ -80,12 +47,6 @@ describe.each(["local", "remote"] as const)("startSubagent (%s)", (kind) => {: { action: { callId: "child-action" } as never, kind },});- expect(readActionTraceContext).toHaveBeenCalledWith(- {},- "parent-session",- "turn-1",- source === "direct" ? "child-action" : "workflow-call",- );const start = kind === "local" ? startLocalSubagent : startRemoteSubagent;const other = kind === "local" ? startRemoteSubagent : startLocalSubagent;expect(start).toHaveBeenCalledWith(@@ -99,14 +60,11 @@ describe.each(["local", "remote"] as const)("startSubagent (%s)", (kind) => {sessionId: "parent-session",turn: { id: "turn-1", sequence: 1 },},- traceContext: {- ...selected,- decision: { action: "record", recordInputs: false, recordOutputs: false },- },+ traceContext: caller,+ originAudience: "private",},}),);- expect(selected.decision.recordInputs).toBe(true);expect(other).not.toHaveBeenCalled();},);diff --git a/packages/eve/src/execution/tools/subagent/start.ts b/packages/eve/src/execution/tools/subagent/start.tsindex 32466064d..9b80d1f8a 100644--- a/packages/eve/src/execution/tools/subagent/start.ts+++ b/packages/eve/src/execution/tools/subagent/start.ts@@ -1,5 +1,5 @@import type { ActivityObserverConfig } from "#channel/types.js";-import { ConversationIdKey, type LocalDevRequestProvenance } from "#context/keys.js";+import type { LocalDevRequestProvenance } from "#context/keys.js";import type { ActivityWorkIdentityV1 } from "#protocol/activity.js";import type { DynamicSubagentAgentConfig } from "#runtime/subagents/dynamic-agent-config.js";import type { DynamicRemoteAgentConfig } from "#runtime/subagents/dynamic-remote-agent-config.js";@@ -8,21 +8,13 @@ import type {RuntimeRemoteAgentDispatchRequest,RuntimeSubagentDispatchRequest,} from "#shared/action-types.js";-import { normalizeChannelAudience } from "#shared/channel-audience.js";-import {- applyLiveDeliveryAudienceCeiling,- readForwardedTraceAssertion,-} from "#shared/forwarded-trace-policy.js";import type { DispatchOutcome, RuntimeSession } from "#subagents/handle-dispatch.js";import { startLocalSubagent } from "#subagents/start-local.js";import { startRemoteSubagent } from "#subagents/start-remote.js";import { buildSubagentRunInput, type SubagentInputSource } from "#subagents/tool.js";-import {- readActionTraceContext,- readSessionTraceContext,-} from "#tracing/agent-trace-context-store.js";import type { SubagentParentContext } from "#subagents/invocation.js";-import { readConversationId } from "#tracing/conversation-context.js";+import type { AgentChildTraceDispatch } from "#tracing/agent-invocation-coordinator.js";+import { resolveConversationId } from "#tracing/conversation-context.js";export type SubagentStartTarget =| {@@ -47,7 +39,6 @@ export async function startSubagent(input: {readonly channelMetadata: Parameters<typeof buildSubagentRunInput>[0]["channelMetadata"];readonly currentSession: RuntimeSession;readonly fanoutSize: number;- readonly instrumentationCallId?: string;readonly initiatorAuth: Parameters<typeof buildSubagentRunInput>[0]["initiatorAuth"];readonly localDevRequest?: LocalDevRequestProvenance;readonly parentContinuationToken: string | undefined;@@ -55,48 +46,16 @@ export async function startSubagent(input: {readonly workIdentity: ActivityWorkIdentityV1;};readonly sandboxSessionId: string;- readonly serializedContext: Record<string, unknown>;readonly session: RuntimeSession;readonly taskId?: string;readonly target: SubagentStartTarget;+ readonly trace: AgentChildTraceDispatch;}): Promise<DispatchOutcome> {- const actionCallId = input.instrumentationCallId;- const storedParentTraceContext =- (actionCallId === undefined- ? undefined- : readActionTraceContext(- input.serializedContext,- input.session.sessionId,- input.batchEvent.turnId,- actionCallId,- )) ??- readActionTraceContext(- input.serializedContext,- input.session.sessionId,- input.batchEvent.turnId,- input.target.action.callId,- ) ??- readSessionTraceContext(input.serializedContext, input.session.sessionId);- const forwardedTracePolicy = readForwardedTraceAssertion(- storedParentTraceContext?.forwardedTracePolicy,- );- const liveAudience = normalizeChannelAudience(input.channelMetadata?.metadata.audience);- const parentTraceContext =- storedParentTraceContext?.decision === undefined- ? storedParentTraceContext- : {- ...storedParentTraceContext,- decision: applyLiveDeliveryAudienceCeiling(- storedParentTraceContext.decision,- liveAudience,- forwardedTracePolicy,- ),- };+ const { trace } = input;const parent: SubagentParentContext = {conversationId:- readConversationId(input.serializedContext[ConversationIdKey.name]) ??- input.session.rootSessionId ??- input.session.sessionId,+ trace.conversationId ??+ resolveConversationId(input.session.rootSessionId ?? input.session.sessionId),lineage: {callId: input.target.action.callId,rootSessionId: input.session.rootSessionId ?? input.session.sessionId,@@ -104,7 +63,8 @@ export async function startSubagent(input: {turn: { id: input.batchEvent.turnId, sequence: input.batchEvent.sequence },},continuationToken: input.parentContinuationToken,- traceContext: parentTraceContext,+ traceContext: trace.parentTraceContext,+ originAudience: trace.originAudience,};switch (input.target.kind) {@@ -133,7 +93,6 @@ export async function startSubagent(input: {auth: input.auth,bundle: input.bundle,callbackBaseUrl: input.callbackBaseUrl,- originAudience: forwardedTracePolicy?.originAudience ?? liveAudience,currentSession: input.currentSession,dynamicRemoteAgent: input.target.dynamicRemoteAgent,initiatorAuth: input.initiatorAuth,diff --git a/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts b/packages/eve/src/execution/tools/subagent/task-agent-requests.test.tsnew file mode 100644index 000000000..c8dded0b3--- /dev/null+++ b/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts@@ -0,0 +1,98 @@+import { beforeEach, describe, expect, it, vi } from "vitest";++import { applyTaskAgentRequest } from "#execution/tools/subagent/task-agent-requests.js";+import {+ dispatchTaskAgentInvocationStep,+ settleTaskAgentInvocationStep,+} from "#execution/tools/subagent/invoke-step.js";+import { emitTaskSubagentCalledStep } from "#execution/tools/subagent/emit-called-step.js";+import type { RuntimeSubagentChildResult } from "#shared/action-types.js";++vi.mock("#execution/tools/subagent/invoke-step.js", () => ({+ dispatchTaskAgentInvocationStep: vi.fn(),+ settleTaskAgentInvocationStep: vi.fn(),+}));+vi.mock("#execution/tools/subagent/emit-called-step.js", () => ({+ emitTaskSubagentCalledStep: vi.fn(),+}));+vi.mock("#execution/tools/workflow/resume-hook-step.js", () => ({+ resumeHookStep: vi.fn(),+}));++const sessionState = { sessionId: "parent" } as never;+const result: RuntimeSubagentChildResult = {+ callId: "nested",+ kind: "subagent-result",+ origin: "child",+ outcome: {+ kind: "parked",+ result: { kind: "succeeded", output: "done" },+ usageDelta: { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 2, outputTokens: 3 },+ },+ output: "done",+ subagentName: "research",+};++beforeEach(() => vi.clearAllMocks());++describe("workflow-owned agent requests", () => {+ it("reuses the settlement step's flushed trace context during workflow replay", async () => {+ const serializedContext = { "eve.test": "before" };+ const flushedContext = { "eve.test": "after" };+ vi.mocked(settleTaskAgentInvocationStep).mockResolvedValue({+ serializedContext: flushedContext,+ sessionState,+ });+ const delivery = {+ ownerId: "workflow-run",+ replyTo: "reply",+ request: { kind: "agent-settled" as const, result },+ };+ const context = { parentWritable: {} as never, serializedContext, sessionState };+ const settled = await applyTaskAgentRequest(delivery, context);++ expect(settleTaskAgentInvocationStep).toHaveBeenCalledWith({+ accumulateUsage: undefined,+ ownerId: "workflow-run",+ result,+ serializedContext,+ sessionState,+ taskId: undefined,+ });+ expect(settled.serializedContext).toBe(flushedContext);+ const replay = await applyTaskAgentRequest(delivery, context);+ expect(replay).toEqual(settled);+ });++ it("retains existing context when replaying a dispatch result without tracing state", async () => {+ const serializedContext = { "eve.test": "preserved" };+ const event = { type: "subagent.called" } as never;+ vi.mocked(dispatchTaskAgentInvocationStep).mockResolvedValue({+ agentId: "child",+ event,+ kind: "dispatched",+ sessionState,+ });+ vi.mocked(emitTaskSubagentCalledStep).mockResolvedValue({ serializedContext });++ const applied = await applyTaskAgentRequest(+ {+ ownerId: "workflow-run",+ replyTo: "reply",+ request: {+ input: { message: "Find it", target: "research" },+ invocationId: "nested",+ kind: "agent-invoke",+ },+ },+ { parentWritable: {} as never, serializedContext, sessionState },+ );++ expect(emitTaskSubagentCalledStep).toHaveBeenCalledWith({+ event,+ parentWritable: {},+ serializedContext,+ });+ expect(applied.serializedContext).toBe(serializedContext);+ });+});diff --git a/packages/eve/src/execution/tools/subagent/task-agent-requests.ts b/packages/eve/src/execution/tools/subagent/task-agent-requests.tsindex 31134214b..80aa0e0c7 100644--- a/packages/eve/src/execution/tools/subagent/task-agent-requests.ts+++ b/packages/eve/src/execution/tools/subagent/task-agent-requests.ts@@ -9,7 +9,6 @@ import type { TaskAgentRequestDelivery } from "#tasks/types.js";export interface AgentRequestDelivery {readonly accumulateUsage?: boolean;- readonly actionCallId?: string;readonly ownerId: string;readonly replyTo: TaskAgentRequestDelivery["replyTo"];readonly request: TaskAgentRequestDelivery["request"];@@ -43,14 +42,17 @@ export async function applyTaskAgentRequest(accumulateUsage: delivery.accumulateUsage,ownerId: delivery.ownerId,result: request.result,+ serializedContext: ctx.serializedContext,sessionState: ctx.sessionState,taskId: delivery.taskId,});- return { serializedContext: ctx.serializedContext, sessionState: settled.sessionState };+ return {+ serializedContext: settled.serializedContext,+ sessionState: settled.sessionState,+ };}case "agent-invoke": {const dispatched = await dispatchTaskAgentInvocationStep({- instrumentationCallId: delivery.actionCallId,ownerId: delivery.ownerId,replyTo: delivery.replyTo,request,@@ -63,7 +65,7 @@ export async function applyTaskAgentRequest(const emitted = await emitTaskSubagentCalledStep({event: dispatched.event,parentWritable: ctx.parentWritable,- serializedContext: ctx.serializedContext,+ serializedContext: dispatched.serializedContext ?? ctx.serializedContext,});return {serializedContext: emitted.serializedContext,@@ -76,7 +78,7 @@ export async function applyTaskAgentRequest(results: [dispatched.result],});return {- serializedContext: ctx.serializedContext,+ serializedContext: dispatched.serializedContext ?? ctx.serializedContext,sessionState: dispatched.sessionState,};case "not-admitted":diff --git a/packages/eve/src/execution/turn-workflow-tool-run.ts b/packages/eve/src/execution/turn-workflow-tool-run.tsindex 2776eaf88..84fe44823 100644--- a/packages/eve/src/execution/turn-workflow-tool-run.ts+++ b/packages/eve/src/execution/turn-workflow-tool-run.ts@@ -154,7 +154,6 @@ async function handleWorkflowToolRunRequest(await applyTaskAgentRequest({accumulateUsage: message.from.resultKind !== "subagent",- actionCallId: message.from.resultKind === "subagent" ? message.from.callId : undefined,ownerId: message.from.runId,replyTo: message.replyTo,request: message.request,diff --git a/packages/eve/src/execution/turn-workflow.test.ts b/packages/eve/src/execution/turn-workflow.test.tsindex 748dc3ddb..9e26ee219 100644--- a/packages/eve/src/execution/turn-workflow.test.ts+++ b/packages/eve/src/execution/turn-workflow.test.ts@@ -1105,7 +1105,11 @@ describe("turnWorkflow", () => {expect(vi.mocked(applyTaskAgentRequest).mock.calls.map(([delivery]) => delivery)).toEqual([expect.objectContaining({ownerId: "run-1",- request: { kind: "agent-invoke", invocationId: "call-1", input: expect.any(Object) },+ request: {+ kind: "agent-invoke",+ invocationId: "call-1",+ input: expect.any(Object),+ },}),expect.objectContaining({ownerId: "run-1",diff --git a/packages/eve/src/execution/workflow-runtime.test.ts b/packages/eve/src/execution/workflow-runtime.test.tsindex d642c6fe9..7d802a3be 100644--- a/packages/eve/src/execution/workflow-runtime.test.ts+++ b/packages/eve/src/execution/workflow-runtime.test.ts@@ -1089,7 +1089,7 @@ describe("createWorkflowRuntime#createSession trace seed allocation", () => {expect(seed!.traceFlags).toBe(0);});- it("inherits the parent trace context for delegated subagents", async () => {+ it("inherits parent policy while allocating an independent delegated trace", async () => {const tracePolicy = vi.fn(() => false);installAgentOtelRuntime(new AgentSpanIdGenerator(), tracePolicy);mockBundleAndRun();@@ -1121,7 +1121,13 @@ describe("createWorkflowRuntime#createSession trace seed allocation", () => {const seed = serialized["eve.sessionTraceSeed"] as| { traceId: string; spanId: string; traceFlags: number }| undefined;- expect(seed).toEqual(parentTrace);+ expect(seed).toMatchObject({+ decision: parentTrace.decision,+ traceFlags: parentTrace.traceFlags,+ });+ expect(seed!.traceId).not.toBe(parentTrace.traceId);+ expect(seed!.spanId).not.toBe(parentTrace.spanId);+ expect(serialized["eve.parentTraceContext"]).toEqual(parentTrace);expect(serialized["eve.otelTraceEnabled"]).toBe(true);expect(startMock.mock.calls[0]?.[2].attributes["$eve.is_otel_trace_enabled"]).toBe("true");expect(tracePolicy).not.toHaveBeenCalled();diff --git a/packages/eve/src/execution/workflow-steps.ts b/packages/eve/src/execution/workflow-steps.tsindex 1dbb8163c..802b012dc 100644--- a/packages/eve/src/execution/workflow-steps.ts+++ b/packages/eve/src/execution/workflow-steps.ts@@ -41,6 +41,7 @@ import {} from "#harness/emission.js";import { bindSessionInstrumentation } from "#instrumentation/runtime.js";import { RuntimeActionSettlementTimesKey } from "#harness/runtime-action-settlement-state.js";+import * as agentTraceState from "#tracing/agent-trace-context-store.js";import { matchAuthorizationCallbacks } from "#execution/authorization-callback-match.js";import { isTurnCancellation, throwIfTurnAborted } from "#harness/turn-cancellation.js";import { setChannelContext } from "#execution/channel-context.js";@@ -590,6 +591,7 @@ export async function turnStep(rawInput: TurnStepInput): Promise<DurableStepResu// Re-stamp if a handler called `session.continuation.rekey(...)` (eg. Slack auto-anchor).const rekeyed = reconcileSessionContinuationToken(ctx, stepResult.session);+ agentTraceState.pruneAgentTraceState(ctx, rekeyed.sessionId, rekeyed.state);const nextSerializedContext = serializeContext(ctx);stepResult = { ...stepResult, session: rekeyed };diff --git a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.tsindex f34712196..6d3223902 100644--- a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts+++ b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts@@ -514,6 +514,7 @@ describe("createAiSdkHookBridge", () => {],finishReason: "tool-calls",idempotencyKey: modelCallIdempotencyKey(scope, 0),+ responseId: "response-1",scope,type: "model.call.completed",usage: {diff --git a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.ts b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.tsindex dc07bcd97..e27059045 100644--- a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.ts+++ b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.ts@@ -197,6 +197,7 @@ function toModelCallCompleted(content: state.capturesOutputs ? toContentParts(source.content) : undefined,finishReason: source.finishReason,idempotencyKey,+ responseId: source.responseId,scope: state.scope,type: "model.call.completed",usage: toUsage(source.usage),diff --git a/packages/eve/src/instrumentation/channel-delivery.ts b/packages/eve/src/instrumentation/channel-delivery.tsindex 19963e90c..5fa4a1b0d 100644--- a/packages/eve/src/instrumentation/channel-delivery.ts+++ b/packages/eve/src/instrumentation/channel-delivery.ts@@ -113,8 +113,10 @@ export async function instrumentChannelDelivery(input: deliveryInput,parentTraceContext: input.ctx.get(ParentTraceContextKey),rootSessionId: input.rootSessionId,+ sequence: input.sequence,sessionId: input.sessionId,traceSeed: input.ctx.get(SessionTraceSeedKey),+ turnId: input.turnId,type: "channel.delivery.started",});}diff --git a/packages/eve/src/instrumentation/lifecycle.ts b/packages/eve/src/instrumentation/lifecycle.tsindex ca9348196..ed67126c2 100644--- a/packages/eve/src/instrumentation/lifecycle.ts+++ b/packages/eve/src/instrumentation/lifecycle.ts@@ -414,6 +414,7 @@ export interface InstrumentationModelCallCompletedEvent {readonly content?: readonly InstrumentationContentPart[];readonly finishReason: string;readonly idempotencyKey: string;+ readonly responseId?: string;readonly scope: InstrumentationAttemptScope;readonly usage: InstrumentationUsage;}@@ -471,6 +472,8 @@ export interface InstrumentationActionStartedEvent {readonly idempotencyKey: string;/** Content. Absent unless this provider's trace policy records this direction. */readonly input?: unknown;+ /** Whether this action owns a workflow body that may invoke nested agents. */+ readonly isWorkflowTool?: boolean;readonly kind: InstrumentationActionKind;readonly name: string;readonly scope: InstrumentationAttemptScope;diff --git a/packages/eve/src/instrumentation/native-events.test.ts b/packages/eve/src/instrumentation/native-events.test.tsindex d317c83f0..9f391d7b7 100644--- a/packages/eve/src/instrumentation/native-events.test.ts+++ b/packages/eve/src/instrumentation/native-events.test.ts@@ -401,6 +401,7 @@ describe("createInstrumentationHandleEvent", () => {callId: "workflow-1",idempotencyKey: actionIdempotencyKey("session-1", "turn-1", "workflow-1"),input: { report: "weekly" },+ isWorkflowTool: true,kind: "tool-call",name: "publish",scope,diff --git a/packages/eve/src/instrumentation/native-events.ts b/packages/eve/src/instrumentation/native-events.tsindex b8a8f96d8..5deb31ca9 100644--- a/packages/eve/src/instrumentation/native-events.ts+++ b/packages/eve/src/instrumentation/native-events.ts@@ -84,7 +84,8 @@ export function createInstrumentationHandleEvent(includeTurn:event.type === "turn.cancelled" ||event.type === "turn.completed" ||- event.type === "turn.failed",+ event.type === "turn.failed" ||+ event.type === "session.failed",outcome:event.type === "turn.failed" || event.type === "session.failed"? "failed"@@ -203,6 +204,7 @@ async function publishActionStarts(callId: action.callId,idempotencyKey,input: capturesInputs ? action.input : undefined,+ ...(action.kind === "workflow-tool-call" ? { isWorkflowTool: true } : undefined),kind: action.kind === "workflow-tool-call" ? "tool-call" : action.kind,name: actionName(action),scope,diff --git a/packages/eve/src/instrumentation/parent-lineage.test.ts b/packages/eve/src/instrumentation/parent-lineage.test.tsindex bd353eb76..6415ea145 100644--- a/packages/eve/src/instrumentation/parent-lineage.test.ts+++ b/packages/eve/src/instrumentation/parent-lineage.test.ts@@ -3,37 +3,49 @@ import { describe, expect, it } from "vitest";import { resolveParentLineage } from "#instrumentation/parent-lineage.js";const parent = {- callId: "call-1",- rootSessionId: "root-1",- sessionId: "session-1",+ callId: "initial-call",+ rootSessionId: "root",+ sessionId: "parent",turn: { id: "turn-1", sequence: 0 },};+const adapter = {+ state: {+ callId: "continued-local-call",+ parentContinuationToken: "parent-token",+ parentSessionId: "parent",+ subagentName: "research",+ },+};-describe("resolveParentLineage", () => {- it("reads the call and turn from the parent and the name from the adapter", () => {- expect(- resolveParentLineage(parent, {- state: {- callId: "call-1",- parentContinuationToken: "token-1",- parentSessionId: "session-1",- subagentName: "researcher",- },- }),- ).toEqual({- callId: "call-1",- sessionId: "session-1",- subagentName: "researcher",+describe("parent trace lineage", () => {+ it("uses the existing local adapter's current caller", () => {+ expect(resolveParentLineage(parent, adapter)).toEqual({+ callId: "continued-local-call",+ sessionId: "parent",+ subagentName: "research",turnId: "turn-1",});});- it("returns undefined for a top-level session", () => {- expect(resolveParentLineage(undefined, undefined)).toBeUndefined();+ it("uses the current callback for remote continuations", () => {+ expect(+ resolveParentLineage(parent, undefined, {+ callId: "continued-remote-call",+ subagentName: "research",+ token: "callback-token",+ url: "https://parent.example/callback",+ })?.callId,+ ).toBe("continued-remote-call");+ });++ it("does not invent delegated lineage from callback metadata", () => {+ expect(resolveParentLineage(undefined, adapter)).toBeUndefined();+ expect(resolveParentLineage(parent, undefined)?.callId).toBe("initial-call");});it("omits the name when the child did not come through the subagent adapter", () => {expect(resolveParentLineage(parent, { state: { kind: "http" } })?.subagentName).toBeUndefined();expect(resolveParentLineage(parent, undefined)?.subagentName).toBeUndefined();+ expect(resolveParentLineage(undefined, undefined)).toBeUndefined();});});diff --git a/packages/eve/src/instrumentation/parent-lineage.ts b/packages/eve/src/instrumentation/parent-lineage.tsindex d8cdbea89..334d68928 100644--- a/packages/eve/src/instrumentation/parent-lineage.ts+++ b/packages/eve/src/instrumentation/parent-lineage.ts@@ -1,4 +1,4 @@-import type { SessionParent } from "#channel/types.js";+import type { SessionCallback, SessionParent } from "#channel/types.js";import { isSubagentAdapterState } from "#subagents/adapter-state.js";import type { InstrumentationParentLineage } from "#instrumentation/lifecycle.js";@@ -11,13 +11,15 @@ import type { InstrumentationParentLineage } from "#instrumentation/lifecycle.jsexport function resolveParentLineage(parent: SessionParent | undefined,adapter: { readonly state?: unknown } | undefined,+ callback?: SessionCallback,): InstrumentationParentLineage | undefined {if (parent === undefined) return undefined;const state = adapter?.state;+ const subagent = isSubagentAdapterState(state) ? state : undefined;return {- callId: parent.callId,+ callId: callback?.callId ?? subagent?.callId ?? parent.callId,sessionId: parent.sessionId,- subagentName: isSubagentAdapterState(state) ? state.subagentName : undefined,+ subagentName: subagent?.subagentName,turnId: parent.turn.id,};}diff --git a/packages/eve/src/instrumentation/runtime.test.ts b/packages/eve/src/instrumentation/runtime.test.tsindex 3c1b78da2..17fd3ba50 100644--- a/packages/eve/src/instrumentation/runtime.test.ts+++ b/packages/eve/src/instrumentation/runtime.test.ts@@ -124,6 +124,12 @@ describe("initializeSessionInstrumentation", () => {traceFlags: 1,});expect(ctx.get(ParentTraceContextKey)).toMatchObject({ traceFlags: 1 });+ expect(ctx.get(ParentTraceContextKey)).toMatchObject({+ spanId: "c".repeat(16),+ traceId: "d".repeat(32),+ });+ expect(ctx.get(SessionTraceSeedKey)?.spanId).not.toBe("c".repeat(16));+ expect(ctx.get(SessionTraceSeedKey)?.traceId).not.toBe("d".repeat(32));expect(ctx.get(ParentTraceContextKey)).not.toHaveProperty("forwardedTracePolicy");expect(ctx.get(SessionTraceSeedKey)?.forwardedTracePolicy).toEqual({ceiling: { recordInputs: true, recordOutputs: true },@@ -302,7 +308,7 @@ describe("initializeSessionInstrumentation", () => {expect(ceiling).toEqual({ recordInputs: false, recordOutputs: false });});- it("preserves sampled flags for a non-forwarded parent decision", () => {+ it("drops the independent trace when a local parent decision drops", () => {const ctx = createContext("public");ctx.set(ParentTraceContextKey, {decision: { action: "drop" },@@ -322,20 +328,36 @@ describe("initializeSessionInstrumentation", () => {expect(ctx.get(SessionTraceSeedKey)).toMatchObject({decision: { action: "drop" },- traceFlags: 1,+ traceFlags: 0,});expect(ctx.get(ParentTraceContextKey)).toMatchObject({ traceFlags: 1 });});++ it("does not widen an unsampled parent with a record decision", () => {+ const ctx = createContext("public");+ ctx.set(ParentTraceContextKey, {+ decision: { action: "record", recordInputs: true, recordOutputs: true },+ spanId: "c".repeat(16),+ traceFlags: 0,+ traceId: "d".repeat(32),+ });+ initializeSessionInstrumentation({ agentName: "local-subagent", ctx });+ expect(ctx.get(SessionTraceSeedKey)).toMatchObject({+ decision: { action: "drop" },+ traceFlags: 0,+ });+ expect(ctx.get(SessionTraceSeedKey)?.traceId).not.toBe("d".repeat(32));+ });});describe("bindInstrumentationRuntime", () => {- it("returns no worker controls when no runtime is loaded", () => {+ it("initializes conversation identity without worker controls when no runtime is loaded", () => {const ctx = new ContextContainer();expect(bindInstrumentationRuntime(undefined, ctx, boundSession)).toBeUndefined();- expect(ctx.get(ConversationIdKey)).toBeUndefined();+ expect(ctx.get(ConversationIdKey)).toBe(boundSession.rootSessionId);});- it("uses the effective parent root to initialize correlation without replacing it", () => {+ it.each([true, false])("preserves effective correlation (runtime installed: %s)", (installed) => {const ctx = createContext();ctx.set(ParentSessionKey, {callId: "call",@@ -343,7 +365,9 @@ describe("bindInstrumentationRuntime", () => {rootSessionId: "effective-root",turn: { id: "turn", sequence: 0 },});- const runtime = createRuntime({ capturesContent: true, publish: vi.fn() });+ const runtime = installed+ ? createRuntime({ capturesContent: true, publish: vi.fn() })+ : undefined;bindInstrumentationRuntime(runtime, ctx, boundSession);expect(ctx.get(ConversationIdKey)).toBe("effective-root");ctx.set(ConversationIdKey, "original-conversation");@@ -741,7 +765,7 @@ describe("initializeSessionInstrumentation", () => {});}- it("marks the seed unsampled when the installed sampler drops the trace", () => {+ it("defers sampler admission until activation metadata is available", () => {const samplesTrace = vi.fn(() => false);registerSeedRuntime({ samplesTrace });const ctx = createContext();@@ -749,9 +773,9 @@ describe("initializeSessionInstrumentation", () => {initializeSessionInstrumentation({ agentName: "test-agent", ctx });const seed = ctx.get(SessionTraceSeedKey);- expect(seed?.traceFlags).toBe(0);+ expect(seed?.traceFlags).toBe(1);expect(seed?.decision).toMatchObject({ action: "record" });- expect(samplesTrace).toHaveBeenCalledExactlyOnceWith(seed?.traceId);+ expect(samplesTrace).not.toHaveBeenCalled();});it("keeps the seed sampled when the sampler admits the trace", () => {diff --git a/packages/eve/src/instrumentation/runtime.ts b/packages/eve/src/instrumentation/runtime.tsindex a432f0f7a..8e7fc74b4 100644--- a/packages/eve/src/instrumentation/runtime.ts+++ b/packages/eve/src/instrumentation/runtime.ts@@ -42,7 +42,7 @@ import {type PrepareTurnTraceContextInput,} from "#instrumentation/prepare-trace-context.js";import type { RuntimeTraceContext } from "#protocol/message.js";-import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";+import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import type { OtelHarnessSettings, RuntimeContextResolver } from "#tracing/otel-declaration.js";import type { SessionTraceSeed } from "#context/keys.js";import { contextStorage, type ContextContainer } from "#context/container.js";@@ -52,10 +52,13 @@ import {OtelTraceEnabledKey,ParentSessionKey,ParentTraceContextKey,+ SessionCallbackKey,SessionTraceSeedKey,} from "#context/keys.js";import { ChannelKey } from "#runtime/sessions/runtime-context-keys.js";import { normalizeChannelAudience } from "#shared/channel-audience.js";+import { withErrorContent } from "#tracing/error-content-context.js";+import type { AgentSamplingOperation } from "#tracing/agent-span-contract.js";import {isSampledTrace,resolveTracePolicy,@@ -144,6 +147,8 @@ export interface BoundInstrumentationSession {/** Process-wide runtime consumed by every harness execution surface. */export interface InstrumentationRuntime {+ /** Materializes settled invocation spans without draining the exporter pipeline. */+ readonly flushSettledInvocations?: () => Promise<void>;readonly forceFlush: () => Promise<void>;readonly hooks: InstrumentationHooks;readonly idGenerator?: AgentSpanIdGenerator;@@ -159,7 +164,7 @@ export interface InstrumentationRuntime {readonly runtimeContextResolvers?: readonly RuntimeContextResolver[];readonly runInContext: InstrumentationContextRunner;/** Whether the installed OTel sampler would record a trace with this id. */- readonly samplesTrace?: (traceId: string) => boolean;+ readonly samplesTrace?: (traceId: string, operation?: AgentSamplingOperation) => boolean;readonly shutdown: () => Promise<void>;stepStartedRuntimeContextResolver?: InstrumentationEvents["step.started"];}@@ -197,13 +202,13 @@ export function bindInstrumentationRuntime(ctx: ContextContainer,boundSession: BoundInstrumentationSession,): ExecutionInstrumentation | undefined {- if (runtime === undefined) return undefined;if (readConversationId(ctx.get(ConversationIdKey)) === undefined) {ctx.set(ConversationIdKey,ctx.get(ParentSessionKey)?.rootSessionId ?? boundSession.rootSessionId,);}+ if (runtime === undefined) return undefined;const baseHooks = runtime.hooks;const readSessionContext = () => {const context = contextStorage.getStore() ?? ctx;@@ -214,12 +219,15 @@ export function bindInstrumentationRuntime(? undefined: { ...storedTraceSeed, ...resolvedTraceState };const parentTraceContext = context.get(ParentTraceContextKey);+ const parent = context.get(ParentSessionKey),+ channel = context.get(ChannelKey);return {- channel: context.get(ChannelKey),+ channel,context,instrumentation: context.get(ChannelInstrumentationKey),forwardedTracePolicy: readForwardedTraceAssertion(traceSeed?.forwardedTracePolicy),- parent: context.get(ParentSessionKey),+ parent,+ parentLineage: resolveParentLineage(parent, channel, context.get(SessionCallbackKey)),parentTraceContext,traceSeed,};@@ -259,7 +267,7 @@ export function bindInstrumentationRuntime(agentName: boundSession.agentName,channelAudience: audience,channelType: channel?.channelType,- parentLineage: resolveParentLineage(sessionContext.parent, sessionContext.channel),+ parentLineage: sessionContext.parentLineage,parentTraceContext: sessionContext.parentTraceContext,rootSessionId: sessionContext.parent?.rootSessionId ?? boundSession.rootSessionId,sessionId: boundSession.sessionId,@@ -402,7 +410,7 @@ export function bindInstrumentationRuntime(channelAudience: audience,channelKind: channel?.kind,hooks,- parentLineage: resolveParentLineage(sessionContext.parent, sessionContext.channel),+ parentLineage: sessionContext.parentLineage,parentTraceContext: sessionContext.parentTraceContext,rootSessionId: sessionContext.parent?.rootSessionId,sessionId: boundSession.sessionId,@@ -473,9 +481,10 @@ export function bindInstrumentationRuntime(},});try {- return parentContext === undefined- ? await run()- : await otelContext.with(parentContext, run);+ return await otelContext.with(+ withErrorContent(parentContext ?? otelContext.active(), content.recordOutputs),+ run,+ );} finally {turnSpan?.end();turnSpan = undefined;@@ -553,7 +562,11 @@ export function initializeSessionInstrumentation(input: {});}if (forwardedTracePolicy !== undefined && parentTraceContext !== undefined) {- const resolvedParent = { ...parentTraceContext, ...traceSeed };+ const resolvedParent = {+ ...parentTraceContext,+ decision: traceSeed.decision,+ traceFlags: traceSeed.traceFlags,+ };delete resolvedParent.forwardedTracePolicy;input.ctx.set(ParentTraceContextKey, resolvedParent);}@@ -580,31 +593,33 @@ function allocateSessionTraceSeed(input: {? traceContentCeilingToDecision(input.forwardedTracePolicy.ceiling): undefined;const inheritedDecision = readInstrumentationDecision(input.parentTraceContext.decision);- const parentDecision = forwardedCeiling- ? inheritedDecision === undefined- ? forwardedCeiling- : intersectInstrumentationDecisions(forwardedCeiling, inheritedDecision)- : (inheritedDecision ??- resolveTracePolicyDecision(isSampledTrace(input.parentTraceContext), input.audience));+ const parentDecision = !isSampledTrace(input.parentTraceContext)+ ? { action: "drop" as const }+ : forwardedCeiling+ ? inheritedDecision === undefined+ ? forwardedCeiling+ : intersectInstrumentationDecisions(forwardedCeiling, inheritedDecision)+ : (inheritedDecision ??+ resolveTracePolicyDecision(isSampledTrace(input.parentTraceContext), input.audience));const decision = input.forwardedTracePolicy? intersectInstrumentationDecisions(parentDecision, localDecision()): parentDecision;+ const idGenerator = input.runtime?.idGenerator ?? new AgentSpanIdGenerator();return {decision,- forwardedTracePolicy: input.forwardedTracePolicy,- spanId: input.parentTraceContext.spanId,- traceFlags:- input.forwardedTracePolicy !== undefined && decision.action === "drop"- ? input.parentTraceContext.traceFlags & ~1- : input.parentTraceContext.traceFlags,- traceId: input.parentTraceContext.traceId,+ ...(input.forwardedTracePolicy === undefined+ ? undefined+ : { forwardedTracePolicy: input.forwardedTracePolicy }),+ spanId: idGenerator.allocateSpanId(),+ traceFlags: decision.action === "drop" ? 0 : input.parentTraceContext.traceFlags,+ traceId: idGenerator.generateTraceId(),};}if (input.runtime?.prepareSessionTrace === undefined || input.runtime.idGenerator === undefined)return undefined;const decision = localDecision();const traceId = input.runtime.idGenerator.generateTraceId();- const sampled = decision.action === "record" && (input.runtime.samplesTrace?.(traceId) ?? true);+ const sampled = decision.action === "record";return {decision,spanId: input.runtime.idGenerator.allocateSpanId(),@@ -666,7 +681,6 @@ export function registerInstrumentationRuntime(return runtime;}-/** Returns the process instrumentation runtime, when one was installed. */export function getInstrumentationRuntime(): InstrumentationRuntime | undefined {return globalRuntime[INSTRUMENTATION_RUNTIME_KEY];}diff --git a/packages/eve/src/internal/logging.test.ts b/packages/eve/src/internal/logging.test.tsindex 8b1871580..d6298b308 100644--- a/packages/eve/src/internal/logging.test.ts+++ b/packages/eve/src/internal/logging.test.ts@@ -1,10 +1,19 @@import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import {+ ROOT_CONTEXT,+ SpanStatusCode,+ context,+ trace,+ type Span,+} from "#compiled/@opentelemetry/api/index.js";+import { withErrorContent } from "#tracing/error-content-context.js";import {createErrorId,createLogger,formatError,logError,+ recordErrorOnSpan,setLogRecordSubscriber,type LogRecord,} from "#internal/logging.js";@@ -22,6 +31,61 @@ describe("createErrorId", () => {});});+describe("span error content policy", () => {+ afterEach(() => vi.restoreAllMocks());++ function mockSpan() {+ return {+ addEvent: vi.fn(),+ end: vi.fn(),+ recordException: vi.fn(),+ setAttribute: vi.fn(),+ setStatus: vi.fn(),+ spanContext: vi.fn<Span["spanContext"]>(),+ };+ }++ it.each([undefined, true, false])("records error content with policy %s", (allowed) => {+ const span = mockSpan();+ vi.spyOn(trace, "getActiveSpan").mockReturnValue(span);+ vi.spyOn(context, "active").mockReturnValue(+ allowed === undefined ? ROOT_CONTEXT : withErrorContent(ROOT_CONTEXT, allowed),+ );+ vi.spyOn(console, "error").mockImplementation(() => {});+ const error = new Error("error content");+ recordErrorOnSpan(span, error);+ createLogger("test").error("failed", { error });+ logError(createLogger("test"), "failed", error);++ expect(span.setStatus).toHaveBeenCalledTimes(3);+ if (allowed === false) {+ expect(span.recordException).not.toHaveBeenCalled();+ expect(span.setStatus.mock.calls).toEqual(+ Array.from({ length: 3 }, () => [{ code: SpanStatusCode.ERROR }]),+ );+ } else {+ expect(span.recordException).toHaveBeenCalledTimes(3);+ expect(span.setStatus).toHaveBeenCalledWith({+ code: SpanStatusCode.ERROR,+ message: "error content",+ });+ }+ });++ it.each([undefined, false])("does not mark a log-only event as a failure (%s)", (allowed) => {+ const span = mockSpan();+ vi.spyOn(trace, "getActiveSpan").mockReturnValue(span);+ vi.spyOn(context, "active").mockReturnValue(+ allowed === undefined ? ROOT_CONTEXT : withErrorContent(ROOT_CONTEXT, allowed),+ );+ vi.spyOn(console, "error").mockImplementation(() => {});+ createLogger("test").error("log content", { reason: "detail" });+ expect(span.setStatus).not.toHaveBeenCalled();+ expect(span.recordException).not.toHaveBeenCalled();+ expect(span.addEvent).toHaveBeenCalledTimes(allowed === false ? 0 : 1);+ });+});+// ---------------------------------------------------------------------------// formatError// ---------------------------------------------------------------------------diff --git a/packages/eve/src/internal/logging.ts b/packages/eve/src/internal/logging.tsindex bda9fa52e..2f8dc4fd9 100644--- a/packages/eve/src/internal/logging.ts+++ b/packages/eve/src/internal/logging.ts@@ -10,6 +10,8 @@ import { inspect } from "node:util";import { isNonEmptyString, isObject } from "#shared/guards.js";import type { JsonObject, JsonValue } from "#shared/json.js";+import { capturesErrorContent } from "#tracing/error-content-context.js";+import { truncateTelemetryText } from "#tracing/telemetry-budget.js";const MAX_INSPECT_STRING_LENGTH = 8 * 1024;const MAX_DETAIL_BYTES = 16 * 1024;@@ -228,8 +230,15 @@ function truncateForDisplay(value: string, maxChars = 160): string {* non-active span should be annotated.*/export function recordErrorOnSpan(span: Span, error: unknown): void {- const message = error instanceof Error ? error.message : getErrorMessage(error);- const name = error instanceof Error ? error.name : "Error";+ if (!capturesErrorContent()) {+ span.setStatus({ code: SpanStatusCode.ERROR });+ return;+ }+ const message = truncateTelemetryText(+ error instanceof Error ? error.message : getErrorMessage(error),+ 4096,+ );+ const name = error instanceof Error ? truncateTelemetryText(error.name, 128) : "Error";span.setStatus({ code: SpanStatusCode.ERROR, message });span.recordException({ message, name, stack: inspectError(error) });@@ -356,6 +365,12 @@ function recordOnActiveSpan(message: string, fields?: LogFields): void {}const error = fields?.error;+ if (!capturesErrorContent()) {+ if (error instanceof Error || isFormattedError(error)) {+ span.setStatus({ code: SpanStatusCode.ERROR });+ }+ return;+ }if (error instanceof Error) {recordErrorOnSpan(span, error);return;@@ -364,11 +379,13 @@ function recordOnActiveSpan(message: string, fields?: LogFields): void {// `logError` passes a pre-formatted object, not a live `Error`; still// record it as a span exception so the trace keeps the `detail`.if (isFormattedError(error)) {- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ const message = truncateTelemetryText(error.message, 4096);+ span.setStatus({ code: SpanStatusCode.ERROR, message });span.recordException({- message: error.message,- name: typeof error.name === "string" ? error.name : "Error",- stack: typeof error.detail === "string" ? error.detail : undefined,+ message,+ name: typeof error.name === "string" ? truncateTelemetryText(error.name, 128) : "Error",+ stack:+ typeof error.detail === "string" ? truncateTelemetryText(error.detail, 8192) : undefined,});return;}diff --git a/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts b/packages/eve/src/internal/nitro/routes/channel-dispatch.test.tsindex bcd717975..586a49919 100644--- a/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts+++ b/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts@@ -749,6 +749,26 @@ function slackChannel(} satisfies ResolvedChannelDefinition;}+function mockRouteSendRuntime(): Runtime {+ const tracedRuntime = createRuntime({+ dispatchContinuation: vi.fn().mockResolvedValue({ sessionId: "session-1", status: "accepted" }),+ });+ mockedResolveNitroChannelRuntimeBundle.mockResolvedValue({+ agentName: "test-agent",+ channels: [+ slackChannel(async () => new Response("unused"), {+ adapter: { kind: "channel:slack" },+ handler: async (_request, args) => {+ await args.from("thread-1").send("hello", { auth: null });+ return new Response("ok");+ },+ }),+ ],+ runtime: tracedRuntime,+ });+ return tracedRuntime;+}+describe("dispatchChannelRequest tracing", () => {let exporter: InMemorySpanExporter;let provider: BasicTracerProvider;@@ -901,24 +921,7 @@ describe("dispatchChannelRequest tracing", () => {});it("captures the created SERVER span context on channel delivery metadata", async () => {- const tracedRuntime = createRuntime({- dispatchContinuation: vi- .fn()- .mockResolvedValue({ sessionId: "session-1", status: "accepted" }),- });- mockedResolveNitroChannelRuntimeBundle.mockResolvedValue({- agentName: "test-agent",- channels: [- slackChannel(async () => new Response("unused"), {- adapter: { kind: "channel:slack" },- handler: async (_request, args) => {- await args.from("thread-1").send("hello", { auth: null });- return new Response("ok");- },- }),- ],- runtime: tracedRuntime,- });+ const tracedRuntime = mockRouteSendRuntime();await dispatchChannelRequest(createEvent({@@ -972,7 +975,7 @@ describe("dispatchChannelRequest tracing", () => {expect(parentSpanId(span!)).toBe(parentId);});- it("marks the span an error and records the exception once for a handler that throws", async () => {+ it("marks failed request spans without exporting exception content", async () => {const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});mockedResolveNitroChannelRuntimeBundle.mockResolvedValue({agentName: "test-agent",@@ -1000,11 +1003,12 @@ describe("dispatchChannelRequest tracing", () => {expect(span!.attributes["http.response.status_code"]).toBe(500);expect(span!.status.code).toBe(2 /* SpanStatusCode.ERROR */);const exceptions = span!.events.filter((event) => event.name === "exception");- expect(exceptions).toHaveLength(1);+ expect(exceptions).toHaveLength(0);+ expect(span!.status.message).toBeUndefined();errorSpy.mockRestore();});- it("records the exception and rethrows when runtime-bundle resolution fails", async () => {+ it("records failure status and rethrows when runtime-bundle resolution fails", async () => {mockedResolveNitroChannelRuntimeBundle.mockRejectedValue(new Error("bundle boom"));await expect(@@ -1014,7 +1018,8 @@ describe("dispatchChannelRequest tracing", () => {const [span] = await finishedSpans();expect(span!.name).toBe("POST /slack");expect(span!.status.code).toBe(2 /* SpanStatusCode.ERROR */);- expect(span!.events.filter((event) => event.name === "exception")).toHaveLength(1);+ expect(span!.events.filter((event) => event.name === "exception")).toHaveLength(0);+ expect(span!.status.message).toBeUndefined();// No channel was resolved, so no channel identity is attached.expect(span!.attributes["eve.channel.name"]).toBeUndefined();});@@ -1171,6 +1176,30 @@ describe("dispatchChannelRequest with request tracing not enabled", () => {expect(exporter.getFinishedSpans()).toHaveLength(0);}+ it("carries the active platform context without emitting an eve request span", async () => {+ const tracedRuntime = mockRouteSendRuntime();+ const platformSpan = apiTrace.getTracer("vercel").startSpan("vercel.function");++ await apiContext.with(apiTrace.setSpan(apiContext.active(), platformSpan), () =>+ dispatchChannelRequest(+ createEvent({ method: "POST", waitUntil: vi.fn() }),+ "POST /slack",+ {} as never,+ ),+ );+ platformSpan.end();++ const command = vi.mocked(tracedRuntime.dispatchContinuation).mock.calls[0]?.[0].command;+ const delivery = command?.kind === "send" ? command.delivery : undefined;+ expect(delivery?.requestTraceContext).toEqual({+ spanId: platformSpan.spanContext().spanId,+ traceFlags: platformSpan.spanContext().traceFlags,+ traceId: platformSpan.spanContext().traceId,+ });+ await provider.forceFlush();+ expect(exporter.getFinishedSpans().map((span) => span.name)).toEqual(["vercel.function"]);+ });+it("emits no request span by default when the flag is unset", async () => {registerInstrumentationConfig({}, { agentName: "test" });await expectNoSpan();diff --git a/packages/eve/src/internal/nitro/routes/channel-dispatch.ts b/packages/eve/src/internal/nitro/routes/channel-dispatch.tsindex 712f4b74f..3fb015a3b 100644--- a/packages/eve/src/internal/nitro/routes/channel-dispatch.ts+++ b/packages/eve/src/internal/nitro/routes/channel-dispatch.ts@@ -1,5 +1,9 @@import type { H3Event } from "nitro";-import type { Span } from "#compiled/@opentelemetry/api/index.js";+import {+ context as otelContext,+ trace as otelTrace,+ type SpanContext,+} from "#compiled/@opentelemetry/api/index.js";import type { RouteContext } from "#public/definitions/channel.js";import { getChannelInstrumentationKind } from "#channel/compiled-channel.js";import { createCrossChannelToFn, toCrossChannelTargets } from "#channel/cross-channel-receive.js";@@ -51,6 +55,10 @@ export async function dispatchChannelRequest(config: NitroArtifactsConfig,): Promise<Response> {return await traceChannelRequest({ request: event.req, routeKey }, async (span) => {+ // Correlation does not require an eve-owned request span. Preserve any+ // active platform request or function span before route resolution.+ const requestTraceContext =+ span?.spanContext() ?? otelTrace.getSpan(otelContext.active())?.spanContext();const bundle = await resolveNitroChannelRuntimeBundle(config);const matchedChannel = bundle.channels.find(@@ -83,7 +91,7 @@ export async function dispatchChannelRequest(matchedChannel.name,channelKind ?? "channel",config,- span,+ requestTraceContext,);let response: Response;@@ -212,7 +220,7 @@ function buildRouteArgs(channelName: string,channelKind: string,config: NitroArtifactsConfig,- requestSpan: Span | undefined,+ requestTraceContext: SpanContext | undefined,): BuiltRouteArgs {const requestId = readVercelRequestId(event.req.headers);const requestIp = extractRequestIp(event, config);@@ -228,7 +236,6 @@ function buildRouteArgs(};const channel = bundle.channels.find((candidate) => candidate.name === channelName);const adapter = channel?.adapter ?? { kind: "channel" };- const requestSpanContext = requestSpan?.spanContext();const acceptedDeploymentId = process.env.VERCEL_DEPLOYMENT_ID?.trim() || undefined;const deliverySource = {acceptedDeploymentId,@@ -236,12 +243,12 @@ function buildRouteArgs(channelName,requestId,requestTraceContext:- requestSpanContext === undefined+ requestTraceContext === undefined? undefined: {- spanId: requestSpanContext.spanId,- traceFlags: requestSpanContext.traceFlags,- traceId: requestSpanContext.traceId,+ spanId: requestTraceContext.spanId,+ traceFlags: requestTraceContext.traceFlags,+ traceId: requestTraceContext.traceId,},};const channelOperations = createChannelOperations({diff --git a/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts b/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.tsindex 4f770094d..3d10e025a 100644--- a/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts+++ b/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts@@ -9,9 +9,10 @@ import {trace,} from "#compiled/@opentelemetry/api/index.js";import { getInstrumentationRuntime } from "#instrumentation/runtime.js";-import { recordErrorOnSpan } from "#internal/logging.js";import { markAgentTraceContext } from "#tracing/agent-trace-context.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";+import { AGENT_SPAN_NAMES } from "#tracing/agent-span-contract.js";+import { withErrorContent } from "#tracing/error-content-context.js";/*** Stable tracer name for every inbound eve channel HTTP request. Kept@@ -51,15 +52,14 @@ export interface TraceChannelRequestInput {** The handler runs inside the span's active context. When it returns, the* response status is recorded and a `>= 500` status marks the span as an- * error. A thrown handler is recorded once and rethrown — handler- * exceptions that eve already converts to a JSON 500 return normally and- * are recorded by `logError` against this active span, so they are not- * recorded a second time here. The span always ends in `finally`, without+ * error. A thrown handler marks failure status and is rethrown; request+ * spans never record exception content. The span always ends in `finally`, without* waiting for `event.waitUntil()` work or streamed response bodies.** Emitting these spans is opt-in: unless authored instrumentation enables it* via `traceChannelRequests: true`, the handler runs with no span (`undefined`)- * and no context extraction — a true bypass, not a non-recording span.+ * and performs no header extraction. The channel dispatcher can still use an+ * already-active platform span for the activation's cross-trace link.** This is observability-only: it never changes the response and performs no* synchronous span export in the request path, adding only minimal in-process@@ -77,7 +77,7 @@ export async function traceChannelRequest<T extends Response>(const parentContext = propagation.extract(context.active(), request.headers, headersGetter);const spanName =getInstrumentationRuntime()?.instrumentationProviders === true- ? "agent.channel.request"+ ? AGENT_SPAN_NAMES.channelRequest: routeKey;const span = trace.getTracer(TRACER_NAME).startSpan(spanName,@@ -90,7 +90,9 @@ export async function traceChannelRequest<T extends Response>(},parentContext,);- const activeContext = markAgentTraceContext(trace.setSpan(parentContext, span));+ const activeContext = markAgentTraceContext(+ withErrorContent(trace.setSpan(parentContext, span), false),+ );try {const response = await context.with(activeContext, () => handler(span));@@ -100,7 +102,7 @@ export async function traceChannelRequest<T extends Response>(}return response;} catch (error) {- recordErrorOnSpan(span, error);+ span.setStatus({ code: SpanStatusCode.ERROR });throw error;} finally {span.end();diff --git a/packages/eve/src/protocol/baggage.test.ts b/packages/eve/src/protocol/baggage.test.tsindex f9a037098..bb9c5f8ba 100644--- a/packages/eve/src/protocol/baggage.test.ts+++ b/packages/eve/src/protocol/baggage.test.ts@@ -92,6 +92,20 @@ describe("readForwardedAudienceBaggage", () => {});describe("writeForwardedAudienceBaggage", () => {+ it("preserves the assertion at the byte limit and rejects overflow", () => {+ const assertion = "eve.audience=public;ceiling=i0o1";+ const baggage = `vendor=${"a".repeat(8192 - assertion.length - "vendor=,".length)}`;+ const result = writeForwardedAudienceBaggage(baggage, PUBLIC_OUTPUTS);+ expect(new TextEncoder().encode(result).byteLength).toBe(8192);+ expect(readForwardedAudienceBaggage(result!)).toEqual(PUBLIC_OUTPUTS);+ expect(() => writeForwardedAudienceBaggage(`${baggage}a`, PUBLIC_OUTPUTS)).toThrow(+ "Cannot forward baggage: header exceeds 8192 bytes",+ );+ expect(() => writeForwardedAudienceBaggage(`${baggage}\u00e9`, PUBLIC_OUTPUTS)).toThrow(+ "Cannot forward baggage: header exceeds 8192 bytes",+ );+ });+it("preserves unrelated entries and replaces authored Eve assertions", () => {expect(writeForwardedAudienceBaggage(diff --git a/packages/eve/src/protocol/baggage.ts b/packages/eve/src/protocol/baggage.tsindex 3482294b1..0f68f7718 100644--- a/packages/eve/src/protocol/baggage.ts+++ b/packages/eve/src/protocol/baggage.ts@@ -83,7 +83,12 @@ export function replaceBaggageMember(const retained = baggageMembers(value ?? "").filter((entry) => baggageKey(entry) !== key);if (member !== undefined) {const result = [...retained, `${key}=${member}`].join(",");- if (encoder.encode(result).byteLength <= MAX_BAGGAGE_BYTES) return result;+ if (encoder.encode(result).byteLength > MAX_BAGGAGE_BYTES) {+ throw new Error(+ "Cannot forward baggage: header exceeds 8192 bytes. Reduce remote.headers.baggage.",+ );+ }+ return result;}return retained.join(",") || undefined;}diff --git a/packages/eve/src/public/instrumentation/index.ts b/packages/eve/src/public/instrumentation/index.tsindex 77d604538..b8692999b 100644--- a/packages/eve/src/public/instrumentation/index.ts+++ b/packages/eve/src/public/instrumentation/index.ts@@ -167,10 +167,11 @@ export interface InstrumentationDefinition {*/readonly recordOutputs?: boolean;/**- * Whether to emit the inbound HTTP `SERVER` span that wraps each channel- * request (the parent of the turn trace and any `hook.resume`/outgoing- * HTTP spans). Defaults to `false`. Set `true` to emit these request spans- * alongside the rest of the trace.+ * Whether to emit an eve-owned HTTP `SERVER` span around each channel+ * request. In the authored hierarchy, this span parents the turn trace. In+ * the provider layout, a one-to-one activation remains a separate trace+ * root and links to this span; when disabled, it links to any already-active+ * upstream request or function span instead. Defaults to `false`.*/readonly traceChannelRequests?: boolean;/**diff --git a/packages/eve/src/subagents/invocation.ts b/packages/eve/src/subagents/invocation.tsindex fa9fe8187..10fc3b694 100644--- a/packages/eve/src/subagents/invocation.ts+++ b/packages/eve/src/subagents/invocation.ts@@ -1,5 +1,6 @@import type { StepInput } from "#harness/types.js";import type { SessionParent, SessionTraceContext } from "#channel/types.js";+import type { ChannelAudience } from "#shared/channel-audience.js";import { isJsonObjectValue, type JsonObject, type JsonValue } from "#shared/json.js";export interface SubagentParentContext {@@ -7,6 +8,7 @@ export interface SubagentParentContext {readonly lineage: SessionParent;readonly continuationToken?: string;readonly traceContext?: SessionTraceContext;+ readonly originAudience?: ChannelAudience;}/**diff --git a/packages/eve/src/subagents/remote-dispatch.test.ts b/packages/eve/src/subagents/remote-dispatch.test.tsindex 71e609abf..82a96215d 100644--- a/packages/eve/src/subagents/remote-dispatch.test.ts+++ b/packages/eve/src/subagents/remote-dispatch.test.ts@@ -732,6 +732,43 @@ describe("startRemoteAgentSession — forwarded principal", () => {});});+ it.each([false, true])(+ "rejects baggage overflow before dispatch (conversation addition: %s)",+ async (withConversation) => {+ const fetchMock = vi.fn();+ vi.stubGlobal("fetch", fetchMock);+ const assertion = "eve.audience=private;ceiling=i0o0";+ const baggage = `vendor=${"a".repeat(+ 8192 - "vendor=,".length - assertion.length + (withConversation ? 0 : 1),+ )}`;++ await expect(+ startRemoteAgentSession({+ action: createAction(),+ auth: CURRENT_AUTH,+ callbackBaseUrl: "https://caller.example.com",+ originAudience: "private",+ remote: {+ ...createRemoteAgent(),+ forwardPrincipal: true,+ headers: { baggage },+ },+ session: createSession(),+ parent: {+ conversationId: withConversation ? "logical-conversation" : undefined,+ traceContext: {+ decision: { action: "record", recordInputs: false, recordOutputs: false },+ spanId: "2".repeat(16),+ traceFlags: 1,+ traceId: "1".repeat(32),+ },+ },+ }),+ ).rejects.toThrow("Cannot forward baggage: header exceeds 8192 bytes");+ expect(fetchMock).not.toHaveBeenCalled();+ },+ );+it("uses unsampled trace flags as the only propagated drop signal", async () => {const fetchMock = vi.fn().mockResolvedValue(createSessionResponse());vi.stubGlobal("fetch", fetchMock);diff --git a/packages/eve/src/subagents/start-local.test.ts b/packages/eve/src/subagents/start-local.test.tsindex 853b5b9e4..eae616fe1 100644--- a/packages/eve/src/subagents/start-local.test.ts+++ b/packages/eve/src/subagents/start-local.test.ts@@ -45,6 +45,7 @@ describe("startLocalSubagent", () => {initiatorAuth: null,parent: {continuationToken: "parent-token",+ originAudience: "private",lineage: {callId: "call-1",rootSessionId: "parent-session",diff --git a/packages/eve/src/subagents/start-remote.ts b/packages/eve/src/subagents/start-remote.tsindex b466eccb1..689cf29e7 100644--- a/packages/eve/src/subagents/start-remote.ts+++ b/packages/eve/src/subagents/start-remote.ts@@ -9,7 +9,6 @@ import {import { createLogger, logError } from "#internal/logging.js";import type { RuntimeRemoteAgentDispatchRequest } from "#shared/action-types.js";import type { CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";-import type { ChannelAudience } from "#shared/channel-audience.js";import type { SubagentParentContext } from "#subagents/invocation.js";const log = createLogger("execution.subagent-start-remote");@@ -20,7 +19,6 @@ export async function startRemoteSubagent(input: {readonly auth: Parameters<typeof startRemoteAgentSession>[0]["auth"];readonly bundle: CompiledBundle;readonly callbackBaseUrl: string | undefined;- readonly originAudience: ChannelAudience;readonly currentSession: RuntimeSession;readonly dynamicRemoteAgent?: NonNullable<Parameters<typeof resolveRemoteAgentForAction>[0]["dynamicRemoteAgent"]@@ -87,7 +85,7 @@ export async function startRemoteSubagent(input: {action,auth: input.auth,callbackBaseUrl,- originAudience: input.originAudience,+ originAudience: input.parent.originAudience,initiatorAuth: input.initiatorAuth,operationId: operation.id,parent: input.parent,diff --git a/packages/eve/src/tracing/agent-action-instrumentation.ts b/packages/eve/src/tracing/agent-action-instrumentation.tsindex 15cbfacd0..48fecace8 100644--- a/packages/eve/src/tracing/agent-action-instrumentation.ts+++ b/packages/eve/src/tracing/agent-action-instrumentation.ts@@ -1,6 +1,6 @@import {ROOT_CONTEXT,- SpanStatusCode,+ SpanKind,type Context,type Span,type SpanContext,@@ -9,20 +9,30 @@ import {} from "#compiled/@opentelemetry/api/index.js";import type {+ InstrumentationActionKind,InstrumentationActionStartedEvent,InstrumentationActionTerminalEvent,InstrumentationAttemptScope,InstrumentationProviderDefinition,+ InstrumentationSessionTransitionEvent,} from "#instrumentation/lifecycle.js";import { actionIdempotencyKey, attemptIdempotencyKey } from "#instrumentation/lifecycle.js";+import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";import { contentAttribute } from "#tracing/agent-otel-content.js";import { setAgentUsage } from "#tracing/agent-otel-usage.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";-import type { AgentActionTraceState, AgentTraceStateStore } from "#tracing/agent-trace-state.js";+import type {+ AgentActionTraceState,+ AgentActionTraceTerminalState,+ AgentInvocationTraceState,+ AgentTraceStateStore,+} from "#tracing/agent-trace-state.js";import { normalizeChannelAudience } from "#shared/channel-audience.js";import { isSampledTrace } from "#tracing/sampled-trace.js";import { withChannelAudience } from "#tracing/channel-audience-context.js";+import { AGENT_SPAN_NAMES } from "#tracing/agent-span-contract.js";+import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";export interface AgentActionInstrumentation {readonly events: Pick<@@ -32,6 +42,8 @@ export interface AgentActionInstrumentation {deleteForSession(sessionId: string): void | PromiseLike<void>;deleteForTurn(sessionId: string, turnId: string): void | PromiseLike<void>;failForAttempt(scope: InstrumentationAttemptScope, error: unknown): Promise<void>;+ flushSettledInvocations(): Promise<void>;+ flushForSessionTransition(event: InstrumentationSessionTransitionEvent): Promise<void>;contextFor(sessionId: string,turnId: string,@@ -83,6 +95,9 @@ export function createAgentActionInstrumentation(input: {turnId: event.scope.turnId,};await input.stateStore.setAction(event.idempotencyKey, state);+ if (event.isWorkflowTool === true) {+ await input.stateStore.setActionAnchor(event.idempotencyKey, state);+ }const keys = byAttempt.get(event.scope.attemptId) ?? new Set<string>();keys.add(event.idempotencyKey);byAttempt.set(event.scope.attemptId, keys);@@ -92,26 +107,7 @@ export function createAgentActionInstrumentation(input: {const state = await input.stateStore.getAction(event.idempotencyKey);if (state === undefined) return;try {- const span = startSpan(state);- span.setAttribute("agent.action.outcome", event.outcome);- if (event.type === "action.failed") {- if (event.errorCode !== undefined) {- span.setAttribute("agent.action.error.code", event.errorCode);- span.setAttribute("error.type", event.errorCode);- }- recordError(span, event.error);- } else if (event.output.type === "error") {- recordError(span, event.output.error);- } else {- if (event.usage !== undefined) {- setAgentUsage(span, event.usage, { includeGenAiDetails: false });- }- if (input.recordOutputs) {- const result = contentAttribute(event.output.output, false);- if (result !== undefined) span.setAttribute("gen_ai.tool.call.result", result);- }- }- span.end(event.acceptedAtMs);+ finishActionSpan(state, event);} finally {await input.stateStore.deleteAction(event.idempotencyKey);forget(event.idempotencyKey);@@ -119,9 +115,10 @@ export function createAgentActionInstrumentation(input: {};const startSpan = (state: AgentActionTraceState): Span => {+ const invocation = isAgentInvocation(state.kind);const span = input.idGenerator.withSpanId(state.spanId, () =>input.tracer.startSpan(- "agent.action",+ AGENT_SPAN_NAMES.action,{attributes: {"agent.action.call_id": state.callId,@@ -129,18 +126,28 @@ export function createAgentActionInstrumentation(input: {"agent.action.name": state.name,"agent.framework.name": "eve","agent.framework.version": input.frameworkVersion,- "agent.session.id": state.sessionId,"agent.step.attempt": state.attemptIndex,"agent.step.index": state.stepIndex,"agent.turn.id": state.turnId,...agentSpanNamingAttributes("agent.action"),+ ...agentTraceIdentityAttributes({+ rootSessionId: state.rootSessionId,+ sessionId: state.sessionId,+ }),+ ...(invocation+ ? {+ "gen_ai.agent.name": state.name,+ "agent.invocation.role": "caller",+ }+ : undefined),},+ kind: state.kind === "remote-agent-call" ? SpanKind.CLIENT : SpanKind.INTERNAL,startTime: state.startTimeMs,},contextFromActionState(state),),);- if (state.inputAttribute !== undefined) {+ if (!invocation && state.inputAttribute !== undefined) {span.setAttribute("gen_ai.tool.call.arguments", state.inputAttribute);}return span;@@ -154,7 +161,11 @@ export function createAgentActionInstrumentation(input: {const state = await input.stateStore.findAction(sessionId, callId);return state === undefined ? undefined : actionContext(state);},- deleteForSession: (sessionId) => input.stateStore.deleteActions(sessionId),+ async deleteForSession(sessionId) {+ await input.stateStore.deleteActions(sessionId);+ await input.stateStore.deleteActionAnchors(sessionId);+ await input.stateStore.deleteInvocations(sessionId);+ },deleteForTurn: (sessionId, turnId) => input.stateStore.deleteActions(sessionId, turnId),async failForAttempt(scope, error) {const keys = byAttempt.get(scope.attemptId);@@ -169,6 +180,16 @@ export function createAgentActionInstrumentation(input: {await input.stateStore.deleteAction(key);}},+ flushSettledInvocations: () => flushInvocations(),+ async flushForSessionTransition(event) {+ const terminal =+ event.type === "session.completed"+ ? { outcome: "abandoned" as const }+ : event.type === "session.failed"+ ? { error: event.error, outcome: "failed" as const }+ : undefined;+ await flushInvocations(event.sessionId, terminal);+ },events: {"action.completed": onTerminal,"action.failed": onTerminal,@@ -176,12 +197,69 @@ export function createAgentActionInstrumentation(input: {},};+ async function flushInvocations(+ sessionId?: string,+ fallback?: AgentActionTraceTerminalState,+ ): Promise<void> {+ for (const invocation of await input.stateStore.findInvocations(sessionId)) {+ const terminal = invocation.terminal ?? fallback;+ if (terminal === undefined) continue;+ finishInvocationSpan(invocation, terminal);+ await input.stateStore.deleteInvocation(+ actionIdempotencyKey(invocation.sessionId, invocation.turnId, invocation.callId),+ );+ }+ }+function forget(idempotencyKey: string): void {for (const [attemptId, keys] of byAttempt) {keys.delete(idempotencyKey);if (keys.size === 0) byAttempt.delete(attemptId);}}++ function finishActionSpan(+ state: AgentActionTraceState,+ event: InstrumentationActionTerminalEvent,+ ): void {+ const span = startSpan(state);+ span.setAttribute("agent.action.outcome", event.outcome);+ if (event.type === "action.failed") {+ if (event.errorCode !== undefined) {+ span.setAttribute("agent.action.error.code", event.errorCode);+ }+ recordError(span, event.error, event.errorCode);+ } else if (event.output.type === "error") {+ recordError(span, event.output.error);+ } else {+ if (event.usage !== undefined) {+ setAgentUsage(span, event.usage);+ }+ if (input.recordOutputs && !isAgentInvocation(state.kind)) {+ const result = contentAttribute(event.output.output, false);+ if (result !== undefined) span.setAttribute("gen_ai.tool.call.result", result);+ }+ }+ span.end(event.acceptedAtMs);+ }++ function finishInvocationSpan(+ state: AgentInvocationTraceState,+ terminal: AgentActionTraceTerminalState,+ ): void {+ const span = startSpan(state);+ span.setAttribute("agent.action.outcome", terminal.outcome);+ if (terminal.usage !== undefined) {+ setAgentUsage(span, terminal.usage);+ }+ if (terminal.outcome === "failed") {+ recordError(+ span,+ input.recordOutputs && state.recordOutputs === true ? terminal.error : undefined,+ );+ }+ span.end(terminal.acceptedAtMs);+ }}function actionContext(state: AgentActionTraceState): AgentActionContext {@@ -207,9 +285,6 @@ function contextFromActionState(state: AgentActionTraceState): Context {);}-function recordError(span: Span, error: unknown): void {- if (error instanceof Error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- } else span.setStatus({ code: SpanStatusCode.ERROR });+function isAgentInvocation(kind: InstrumentationActionKind): boolean {+ return kind === "subagent-call" || kind === "remote-agent-call";}diff --git a/packages/eve/src/tracing/agent-approval-instrumentation.ts b/packages/eve/src/tracing/agent-approval-instrumentation.tsindex 205825b10..0544aee25 100644--- a/packages/eve/src/tracing/agent-approval-instrumentation.ts+++ b/packages/eve/src/tracing/agent-approval-instrumentation.ts@@ -1,7 +1,5 @@import {ROOT_CONTEXT,- SpanStatusCode,- type Span,type SpanContext,type Tracer,trace,@@ -16,6 +14,9 @@ import type {import type { JsonValue } from "#shared/json.js";import { contentAttribute } from "#tracing/agent-otel-content.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";+import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";+import { AGENT_SPAN_NAMES } from "#tracing/agent-span-contract.js";+import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";import { withChannelAudience } from "#tracing/channel-audience-context.js";import type { AgentActionContext } from "#tracing/agent-action-instrumentation.js";import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";@@ -93,7 +94,7 @@ export function createAgentApprovalInstrumentation(input: {input.idGenerator.deriveSpanId(`approval:${event.idempotencyKey}`),() =>input.tracer.startSpan(- "agent.approval",+ AGENT_SPAN_NAMES.approval,{attributes: {"agent.action.call_id": state.actionCallId,@@ -103,11 +104,14 @@ export function createAgentApprovalInstrumentation(input: {"agent.approval.request_id": state.requestId,"agent.framework.name": "eve","agent.framework.version": input.frameworkVersion,- "agent.session.id": state.sessionId,"agent.step.attempt": state.attemptIndex,"agent.step.index": state.stepIndex,"agent.turn.id": state.turnId,...agentSpanNamingAttributes("agent.approval"),+ ...agentTraceIdentityAttributes({+ rootSessionId: state.rootSessionId,+ sessionId: state.sessionId,+ }),},startTime: state.startTimeMs,},@@ -181,10 +185,3 @@ function readState(value: unknown): AgentApprovalSpanState | undefined {turnId: state["turnId"],};}--function recordError(span: Span, error: unknown): void {- if (error instanceof Error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- } else span.setStatus({ code: SpanStatusCode.ERROR });-}diff --git a/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts b/packages/eve/src/tracing/agent-channel-delivery-instrumentation.tsindex 44eb3e280..e77e6cdf8 100644--- a/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts+++ b/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts@@ -1,12 +1,4 @@-import {- ROOT_CONTEXT,- SpanKind,- SpanStatusCode,- type Span,- type SpanContext,- type Tracer,- trace,-} from "#compiled/@opentelemetry/api/index.js";+import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";import type {InstrumentationChannelDeliveryStartedEvent,@@ -15,35 +7,26 @@ import type {InstrumentationProviderDefinition,InstrumentationSessionStartedEvent,} from "#instrumentation/lifecycle.js";+import { contextStorage } from "#context/container.js";+import { ActiveChannelDeliveriesKey } from "#context/keys.js";import { sessionIdempotencyKey } from "#instrumentation/lifecycle.js";import type { JsonValue } from "#shared/json.js";-import { normalizeChannelAudience, type ChannelAudience } from "#shared/channel-audience.js";import { contentAttribute } from "#tracing/agent-otel-content.js";-import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";-import { withChannelAudience } from "#tracing/channel-audience-context.js";-import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import type { AgentSessionTraceState, AgentTraceStateStore } from "#tracing/agent-trace-state.js";import { isSampledTrace } from "#tracing/sampled-trace.js";-interface ChannelDeliverySpanState {- readonly channelAudience: ChannelAudience;+interface ChannelDeliveryState {readonly inputAttribute?: string;- readonly parent: SpanContext;readonly requestTraceContext?: SpanContext;- readonly spanId: string;- readonly startTimeMs: number;}-/** Builds durable channel delivery spans around the turn that consumes each request. */+/** Captures one-to-one channel delivery metadata on the activation that consumes it. */export function createAgentChannelDeliveryInstrumentation(input: {readonly ensureSessionContext: (event: InstrumentationSessionStartedEvent,) => Promise<AgentSessionTraceState>;- readonly frameworkVersion: string;- readonly idGenerator: AgentSpanIdGenerator;readonly recordInputs: boolean;readonly stateStore: AgentTraceStateStore;- readonly tracer: Tracer;}): Pick<NonNullable<InstrumentationProviderDefinition["events"]>,| "channel.delivery.cancelled"@@ -66,19 +49,13 @@ export function createAgentChannelDeliveryInstrumentation(input: {traceSeed: event.traceSeed,type: "session.started",});- if (!isSampledTrace(session.context)) return;+ const turn =+ event.turnId === undefined || event.sequence === undefined+ ? undefined+ : await input.stateStore.getTurn(event.sessionId, event.turnId);+ if (!isSampledTrace(turn?.context ?? session.context)) return;const inputAttribute = input.recordInputs ? contentAttribute(event.input, false) : undefined;- const state: Record<string, JsonValue> = {- channelAudience: normalizeChannelAudience(event.delivery.channelAudience),- parent: {- isRemote: session.context.isRemote ?? false,- spanId: session.context.spanId,- traceFlags: session.context.traceFlags,- traceId: session.context.traceId,- },- spanId: input.idGenerator.deriveSpanId(`channel-delivery:${event.idempotencyKey}`),- startTimeMs: Date.now(),- };+ const state: Record<string, JsonValue> = {};if (inputAttribute !== undefined) state.inputAttribute = inputAttribute;if (event.delivery.requestTraceContext !== undefined) {state.requestTraceContext = {@@ -95,75 +72,32 @@ export function createAgentChannelDeliveryInstrumentation(input: {ctx: InstrumentationHandlerContext,): Promise<void> => {const state = readState(ctx.state.get());- if (state === undefined) return;- const turn =- event.turnId === undefined- ? undefined- : await input.stateStore.getTurn(event.sessionId, event.turnId);- const startTimeMs =- turn === undefined ? state.startTimeMs : Math.max(state.startTimeMs, turn.startTimeMs);- const requestLink = state.requestTraceContext;- const span = input.idGenerator.withSpanId(state.spanId, () =>- input.tracer.startSpan(- "agent.channel.delivery",- {- attributes: {- "agent.channel.delivery.id": event.delivery.deliveryId,- "agent.channel.delivery.outcome": event.outcome,- "agent.channel.kind": event.delivery.channelKind,- "agent.channel.name": event.delivery.channelName,- "agent.channel.request.id": event.delivery.requestId,- "agent.framework.name": "eve",- "agent.framework.version": input.frameworkVersion,- "agent.name": event.agentName,- "agent.session.id": event.sessionId,- "agent.turn.id": event.turnId,- "agent.turn.sequence": event.sequence,- ...agentSpanNamingAttributes("agent.channel.delivery"),- },- kind: SpanKind.CONSUMER,- links:- requestLink === undefined- ? undefined- : [- {- attributes: { "eve.link.type": "channel.request" },- context: requestLink,- },- ],- startTime: startTimeMs,- },- withChannelAudience(contextFromSpanContext(state.parent), state.channelAudience),- ),- );- span.addEvent("channel.delivery.started", undefined, startTimeMs);- span.addEvent(event.type);- if (state.inputAttribute !== undefined) {- span.setAttribute("agent.channel.delivery.input", state.inputAttribute);- }- if (event.outcome === "failed") {- const errorType =- event.errorCode ?? (event.error instanceof Error ? event.error.name : "Error");- span.setAttribute("error.type", errorType);- if (event.errorCode !== undefined) {- span.setAttribute("agent.channel.delivery.error.code", event.errorCode);- }- recordError(span, event.error);- }- span.end();+ if (state === undefined || event.turnId === undefined) return;+ const turn = await input.stateStore.getTurn(event.sessionId, event.turnId);if (- turn !== undefined &&- event.turnId !== undefined &&- turn.parentSpanId === state.parent.spanId &&- turn.context.traceId === state.parent.traceId+ turn === undefined ||+ !isSampledTrace(turn.context) ||+ !isOnlyActiveDelivery(event.sessionId, event.turnId, event.delivery.deliveryId)) {- await input.stateStore.updateTurn(event.sessionId, event.turnId, (current) =>- current.parentSpanId === state.parent.spanId &&- current.context.traceId === state.parent.traceId- ? { ...current, parentIsRemote: false, parentSpanId: state.spanId }- : current,- );+ return;}+ await input.stateStore.updateTurn(event.sessionId, event.turnId, (current) => ({+ ...current,+ channelDelivery: {+ channelKind: event.delivery.channelKind,+ channelName: event.delivery.channelName,+ deliveryId: event.delivery.deliveryId,+ ...(state.inputAttribute === undefined+ ? undefined+ : { inputAttribute: state.inputAttribute }),+ ...(event.delivery.requestId === undefined+ ? undefined+ : { requestId: event.delivery.requestId }),+ ...(state.requestTraceContext === undefined+ ? undefined+ : { requestTraceContext: state.requestTraceContext }),+ },+ }));};return {@@ -174,21 +108,25 @@ export function createAgentChannelDeliveryInstrumentation(input: {};}-function readState(value: unknown): ChannelDeliverySpanState | undefined {- if (!isRecord(value) || !isSpanContext(value.parent)) return undefined;- if (typeof value.spanId !== "string" || typeof value.startTimeMs !== "number") {- return undefined;- }+function isOnlyActiveDelivery(sessionId: string, turnId: string, deliveryId: string): boolean {+ const active = contextStorage.getStore()?.get(ActiveChannelDeliveriesKey);+ const delivery = active?.[0];+ return (+ active?.length === 1 &&+ delivery?.sessionId === sessionId &&+ delivery.turnId === turnId &&+ delivery.delivery.deliveryId === deliveryId+ );+}++function readState(value: unknown): ChannelDeliveryState | undefined {+ if (!isRecord(value)) return undefined;const requestTraceContext = isSpanContext(value.requestTraceContext)? value.requestTraceContext: undefined;return {- channelAudience: normalizeChannelAudience(value.channelAudience),inputAttribute: typeof value.inputAttribute === "string" ? value.inputAttribute : undefined,- parent: value.parent,requestTraceContext,- spanId: value.spanId,- startTimeMs: value.startTimeMs,};}@@ -204,14 +142,3 @@ function isSpanContext(value: unknown): value is SpanContext {function isRecord(value: unknown): value is Record<string, unknown> {return typeof value === "object" && value !== null && !Array.isArray(value);}--function contextFromSpanContext(spanContext: SpanContext) {- return trace.setSpan(ROOT_CONTEXT, trace.wrapSpanContext(spanContext));-}--function recordError(span: Span, error: unknown): void {- if (error instanceof Error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- } else span.setStatus({ code: SpanStatusCode.ERROR });-}diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.test.ts b/packages/eve/src/tracing/agent-invocation-coordinator.test.tsnew file mode 100644index 000000000..9dc076689--- /dev/null+++ b/packages/eve/src/tracing/agent-invocation-coordinator.test.ts@@ -0,0 +1,303 @@+import { describe, expect, it } from "vitest";++import { ContextContainer, contextStorage } from "#context/container.js";+import { SessionTraceSeedKey } from "#context/keys.js";+import { deserializeContext, serializeContext } from "#context/serialize.js";+import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";+import { settleAgentInvocationTrace } from "#tracing/agent-invocation-terminal.js";+import { deriveAgentActionSpanId } from "#tracing/agent-span-id-generator.js";+import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";+import { deriveTaskId } from "#tasks/task-id.js";+import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";++const outerKey = actionIdempotencyKey("session-1", "turn-1", "workflow");++const sessionState = {+ "eve.runtime.workflowToolRuns": [+ {+ callId: "workflow",+ hookToken: "workflow-hook",+ runId: "workflow-run",+ toolName: "coordinate",+ },+ ],+};++describe("agent invocation trace coordinator", () => {+ it("keeps parallel invocation identities and caller contexts independent", async () => {+ let serializedContext = await contextWithActionAnchor();++ const first = prepare(serializedContext, "workflow:first");+ serializedContext = first.serializedContext;+ const second = prepare(serializedContext, "workflow:second");+ serializedContext = second.serializedContext;++ expect(first.dispatch.parentTraceContext?.spanId).toBe(+ deriveAgentActionSpanId("session-1", "turn-1", "workflow:first"),+ );+ expect(second.dispatch.parentTraceContext?.spanId).toBe(+ deriveAgentActionSpanId("session-1", "turn-1", "workflow:second"),+ );++ const invocations = await readInvocations(serializedContext);+ expect(invocations).toEqual([+ expect.objectContaining({+ callId: "workflow:first",+ parentActionCallId: "workflow",+ }),+ expect.objectContaining({+ callId: "workflow:second",+ parentActionCallId: "workflow",+ }),+ ]);+ });++ it("replays one invocation with the same caller coordinates", async () => {+ const serializedContext = await contextWithActionAnchor();++ const first = prepare(serializedContext, "workflow:first");+ const replay = prepare(first.serializedContext, "workflow:first");++ expect(replay.dispatch).toEqual(first.dispatch);+ await expect(readInvocations(replay.serializedContext)).resolves.toHaveLength(1);+ expect(prepare(serializedContext, "workflow:first").serializedContext).toEqual(+ first.serializedContext,+ );+ });++ it.each([0, 1])(+ "falls back to the turn when a nested caller anchor is missing (%s)",+ async (traceFlags) => {+ const context = new ContextContainer();+ const parent = { ...outerAction().parent, traceFlags };+ contextStorage.run(context, () => {+ new ContextAgentTraceStateStore().setTurn("session-1", "turn-1", {+ context: parent,+ rootSessionId: "session-1",+ sequence: 0,+ startTimeMs: 1,+ });+ });+ const prepared = prepare(serializeContext(context), "workflow:nested");+ expect(prepared.dispatch.parentTraceContext).toMatchObject(parent);+ await expect(readInvocations(prepared.serializedContext)).resolves.toEqual([]);+ },+ );++ it.each([+ ["subagent-call", true],+ ["subagent-call", false],+ ["remote-agent-call", true],+ ["remote-agent-call", false],+ ] as const)("caps %s context with or without a recorded caller (%s)", async (kind, recorded) => {+ const context = new ContextContainer();+ const action = outerAction();+ context.set(SessionTraceSeedKey, {+ ...action.parent,+ decision: { action: "record", recordInputs: true, recordOutputs: true },+ forwardedTracePolicy: {+ originAudience: "public",+ ceiling: { recordInputs: true, recordOutputs: true },+ },+ });+ await contextStorage.run(context, () => {+ const store = new ContextAgentTraceStateStore();+ store.setTurn("session-1", "turn-1", {+ context: action.parent,+ rootSessionId: "session-1",+ sequence: 0,+ startTimeMs: 1,+ });+ if (recorded) {+ store.setAction(outerKey, action);+ store.setActionAnchor(outerKey, action);+ }+ });+ const serializedContext = await serializeContext(context);++ const prepared = prepareAgentInvocationTrace({+ channelMetadata: { kind: "http", metadata: { audience: "private" } },+ invocation: {+ callId: "workflow",+ kind,+ name: "research",+ },+ ownerId: "workflow-run",+ startTimeMs: 2,+ serializedContext,+ sessionId: "session-1",+ sessionState,+ turnId: "turn-1",+ });+ expect(prepared.dispatch.parentTraceContext).toMatchObject({+ ...action.parent,+ spanId: recorded ? action.spanId : action.parent.spanId,+ decision: { action: "record", recordInputs: false, recordOutputs: false },+ });+ expect(prepared.dispatch.originAudience).toBe("public");+ const restored = await deserializeContext(prepared.serializedContext);+ await contextStorage.run(restored, () => {+ const store = new ContextAgentTraceStateStore();+ expect(store.getAction(outerKey)?.kind).toBe(recorded ? kind : undefined);+ expect(store.findActionAnchor("session-1", "turn-1", "workflow")?.kind).toBe(+ recorded ? kind : undefined,+ );+ expect(store.findInvocations("session-1")).toHaveLength(0);+ });+ });++ it("uses the persisted action anchor after the live action is gone", async () => {+ const context = new ContextContainer();+ await contextStorage.run(context, () => {+ const store = new ContextAgentTraceStateStore();+ const action = outerAction();+ store.setAction(outerKey, action);+ store.setActionAnchor(outerKey, action);+ store.deleteAction(outerKey);+ });++ const prepared = prepareAgentInvocationTrace({+ invocation: { callId: "workflow:background", kind: "subagent-call", name: "research" },+ ownerId: "background-task",+ startTimeMs: 2,+ serializedContext: serializeContext(context),+ sessionId: "session-1",+ taskId: deriveTaskId({+ callId: "workflow",+ parentSessionId: "session-1",+ parentTurnId: "turn-1",+ }),+ turnId: "later-turn",+ });++ expect(prepared.dispatch.parentTraceContext?.spanId).toBe(+ deriveAgentActionSpanId("session-1", "turn-1", "workflow:background"),+ );+ await expect(readInvocations(prepared.serializedContext)).resolves.toEqual([+ expect.objectContaining({+ callId: "workflow:background",+ parentActionCallId: "workflow",+ }),+ ]);+ const failed = prepared.fail({+ callId: "workflow:background",+ isError: true,+ kind: "subagent-result",+ origin: "dispatch",+ output: "unreachable",+ subagentName: "research",+ });+ await expect(readInvocations(failed)).resolves.toEqual([+ expect.objectContaining({+ turnId: "turn-1",+ terminal: expect.objectContaining({ outcome: "failed" }),+ }),+ ]);+ });++ it("does not infer an outer call from caller-chosen invocation IDs", async () => {+ const prepared = prepareAgentInvocationTrace({+ invocation: { callId: "workflow:unowned", kind: "subagent-call", name: "research" },+ ownerId: "other-run",+ startTimeMs: 2,+ serializedContext: await contextWithActionAnchor(),+ sessionId: "session-1",+ sessionState,+ turnId: "turn-1",+ });+ expect(prepared.dispatch.parentTraceContext).toBeUndefined();+ await expect(readInvocations(prepared.serializedContext)).resolves.toEqual([]);+ });++ it("records settlement on the matching invocation only", async () => {+ const first = prepare(await contextWithActionAnchor(), "workflow:first");+ const second = prepare(first.serializedContext, "workflow:second");+ const serializedContext = settleAgentInvocationTrace({+ acceptedAtMs: 3,+ result: {+ callId: "workflow:first",+ kind: "subagent-result",+ origin: "child",+ outcome: {+ kind: "parked",+ result: { kind: "succeeded", output: "done" },+ usageDelta: {+ cacheReadTokens: 1,+ cacheWriteTokens: 2,+ inputTokens: 3,+ outputTokens: 4,+ },+ },+ output: "done",+ subagentName: "research",+ },+ serializedContext: second.serializedContext,+ sessionId: "session-1",+ });++ const invocations = await readInvocations(serializedContext);+ expect(invocations.find((invocation) => invocation.callId === "workflow:first")).toMatchObject({+ terminal: {+ acceptedAtMs: 3,+ outcome: "completed",+ usage: { inputTokens: 3, outputTokens: 4 },+ },+ });+ expect(+ invocations.find((invocation) => invocation.callId === "workflow:second")?.terminal,+ ).toBeUndefined();+ });+});++function prepare(serializedContext: Record<string, unknown>, callId: string) {+ return prepareAgentInvocationTrace({+ channelMetadata: { kind: "http", metadata: { audience: "private" } },+ invocation: {+ callId,+ kind: "subagent-call",+ name: "research",+ },+ ownerId: "workflow-run",+ startTimeMs: 2,+ serializedContext,+ sessionId: "session-1",+ sessionState,+ turnId: "turn-1",+ });+}++async function contextWithActionAnchor(): Promise<Record<string, unknown>> {+ const context = new ContextContainer();+ await contextStorage.run(context, () => {+ new ContextAgentTraceStateStore().setActionAnchor(outerKey, outerAction());+ });+ return await serializeContext(context);+}++async function readInvocations(serializedContext: Record<string, unknown>) {+ const context = await deserializeContext(serializedContext);+ return contextStorage.run(context, () =>+ new ContextAgentTraceStateStore().findInvocations("session-1", "turn-1", "workflow"),+ );+}++function outerAction() {+ return {+ attemptIndex: 0,+ callId: "workflow",+ channelAudience: "private" as const,+ kind: "tool-call" as const,+ name: "coordinate",+ parent: {+ spanId: "2".repeat(16),+ traceFlags: 1,+ traceId: "1".repeat(32),+ },+ rootSessionId: "session-1",+ sessionId: "session-1",+ spanId: "3".repeat(16),+ startTimeMs: 1,+ stepIndex: 0,+ turnId: "turn-1",+ };+}diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.ts b/packages/eve/src/tracing/agent-invocation-coordinator.tsnew file mode 100644index 000000000..b44b7bf5b--- /dev/null+++ b/packages/eve/src/tracing/agent-invocation-coordinator.ts@@ -0,0 +1,143 @@+import type { ChannelInstrumentationProjection, SessionTraceContext } from "#channel/types.js";+import { ConversationIdKey } from "#context/keys.js";+import { readConversationId } from "#tracing/conversation-context.js";+import type { RuntimeSubagentResult } from "#shared/action-types.js";+import type { SessionStateMap } from "#harness/types.js";+import { getWorkflowToolRuns } from "#harness/workflow-tool-runs.js";+import { normalizeChannelAudience, type ChannelAudience } from "#shared/channel-audience.js";+import {+ applyLiveDeliveryAudienceCeiling,+ readForwardedTraceAssertion,+} from "#shared/forwarded-trace-policy.js";+import { deriveAgentActionSpanId } from "#tracing/agent-span-id-generator.js";+import {+ readActionTraceContext,+ readTaskActionTrace,+ readTurnTraceContext,+ recordActionInvocationKind,+ recordNestedAgentInvocation,+} from "#tracing/agent-trace-context-store.js";+import {+ invocationError,+ recordNestedAgentInvocationTerminal,+} from "#tracing/agent-invocation-terminal.js";++export interface AgentChildTraceDispatch {+ readonly conversationId?: string;+ readonly originAudience: ChannelAudience;+ readonly parentTraceContext?: SessionTraceContext;+}++export function prepareAgentInvocationTrace(input: {+ readonly channelMetadata?: ChannelInstrumentationProjection;+ readonly invocation: {+ readonly callId: string;+ readonly kind: "remote-agent-call" | "subagent-call";+ readonly name: string;+ };+ readonly ownerId: string;+ readonly serializedContext: Record<string, unknown>;+ readonly sessionId: string;+ readonly sessionState?: SessionStateMap;+ readonly startTimeMs: number;+ readonly taskId?: string;+ readonly turnId: string;+}): {+ readonly dispatch: AgentChildTraceDispatch;+ fail(result: RuntimeSubagentResult): Record<string, unknown>;+ readonly serializedContext: Record<string, unknown>;+} {+ const conversationId = readConversationId(input.serializedContext[ConversationIdKey.name]);+ const taskAction =+ input.taskId === undefined+ ? undefined+ : readTaskActionTrace(input.serializedContext, input.sessionId, input.taskId);+ const parentActionCallId =+ input.taskId === undefined+ ? getWorkflowToolRuns(input.sessionState).find((run) => run.runId === input.ownerId)?.callId+ : taskAction?.callId;+ const turnId = taskAction?.turnId ?? input.turnId;+ const parentTurnContext = readTurnTraceContext(input.serializedContext, input.sessionId, turnId);+ const liveAudience = normalizeChannelAudience(input.channelMetadata?.metadata.audience);+ const outerTrace =+ (parentActionCallId === undefined+ ? undefined+ : readActionTraceContext(+ input.serializedContext,+ input.sessionId,+ turnId,+ parentActionCallId,+ )) ?? parentTurnContext;+ const outputDecision =+ outerTrace?.decision === undefined+ ? undefined+ : applyLiveDeliveryAudienceCeiling(+ outerTrace.decision,+ liveAudience,+ outerTrace.forwardedTracePolicy,+ );+ const serializedContext =+ parentActionCallId === undefined+ ? input.serializedContext+ : parentActionCallId === input.invocation.callId+ ? recordActionInvocationKind({+ callId: input.invocation.callId,+ kind: input.invocation.kind,+ serializedContext: input.serializedContext,+ sessionId: input.sessionId,+ turnId,+ })+ : recordNestedAgentInvocation({+ callId: input.invocation.callId,+ kind: input.invocation.kind,+ name: input.invocation.name,+ outerCallId: parentActionCallId,+ recordOutputs: outputDecision?.action === "record" && outputDecision.recordOutputs,+ serializedContext: input.serializedContext,+ sessionId: input.sessionId,+ spanId: deriveAgentActionSpanId(input.sessionId, turnId, input.invocation.callId),+ startTimeMs: input.startTimeMs,+ turnId,+ });+ const callerTraceContext = readActionTraceContext(+ serializedContext,+ input.sessionId,+ turnId,+ input.invocation.callId,+ );+ const storedParentTraceContext = callerTraceContext ?? outerTrace;+ const forwardedTracePolicy = readForwardedTraceAssertion(+ storedParentTraceContext?.forwardedTracePolicy,+ );+ const parentTraceContext =+ storedParentTraceContext?.decision === undefined+ ? storedParentTraceContext+ : {+ ...storedParentTraceContext,+ decision: applyLiveDeliveryAudienceCeiling(+ storedParentTraceContext.decision,+ liveAudience,+ forwardedTracePolicy,+ ),+ };+ return {+ dispatch: {+ conversationId,+ originAudience: forwardedTracePolicy?.originAudience ?? liveAudience,+ parentTraceContext,+ },+ fail: (result) =>+ recordNestedAgentInvocationTerminal({+ callId: input.invocation.callId,+ serializedContext,+ sessionId: input.sessionId,+ terminal: {+ acceptedAtMs: Date.now(),+ error: invocationError(result.output),+ outcome: "failed",+ },+ turnId,+ }),+ serializedContext,+ };+}diff --git a/packages/eve/src/tracing/agent-invocation-terminal.ts b/packages/eve/src/tracing/agent-invocation-terminal.tsnew file mode 100644index 000000000..25312792e--- /dev/null+++ b/packages/eve/src/tracing/agent-invocation-terminal.ts@@ -0,0 +1,120 @@+import type { RuntimeSubagentChildResult } from "#shared/action-types.js";+import type { AgentActionTraceTerminalState } from "#tracing/agent-trace-state.js";+import {+ AGENT_TRACE_CONTEXT_KEY,+ deserializeAgentTraceContextState,+ serializeAgentTraceContextState,+} from "#tracing/agent-trace-context-codec.js";+import { truncateTelemetryText } from "#tracing/telemetry-budget.js";+import { boundedTraceError } from "#tracing/bounded-error.js";+import { contextStorage } from "#context/container.js";+import { deserializeContext, serializeContext } from "#context/serialize.js";+import { getInstrumentationRuntime } from "#instrumentation/runtime.js";+import { createLogger, formatError } from "#internal/logging.js";++const log = createLogger("tracing.agent-invocation-terminal");++export function recordNestedAgentInvocationTerminal(input: {+ readonly callId: string;+ readonly serializedContext: Record<string, unknown>;+ readonly sessionId: string;+ readonly terminal: AgentActionTraceTerminalState;+ readonly turnId?: string;+}): Record<string, unknown> {+ const raw = input.serializedContext[AGENT_TRACE_CONTEXT_KEY];+ if (raw === undefined) return input.serializedContext;+ const state = deserializeAgentTraceContextState(raw);+ const entry = Object.entries(state.invocations).find(+ ([, invocation]) =>+ invocation.sessionId === input.sessionId &&+ (input.turnId === undefined || invocation.turnId === input.turnId) &&+ invocation.callId === input.callId,+ );+ if (entry === undefined) return input.serializedContext;+ const [key, invocation] = entry;+ if (invocation.terminal !== undefined) return input.serializedContext;+ return {+ ...input.serializedContext,+ [AGENT_TRACE_CONTEXT_KEY]: serializeAgentTraceContextState({+ ...state,+ invocations: {+ ...state.invocations,+ [key]: {+ ...invocation,+ terminal: {+ ...input.terminal,+ error:+ invocation.recordOutputs === true && input.terminal.error !== undefined+ ? invocationError(input.terminal.error)+ : undefined,+ },+ },+ },+ }),+ };+}++export function settleAgentInvocationTrace(input: {+ readonly acceptedAtMs: number;+ readonly result: RuntimeSubagentChildResult;+ readonly serializedContext: Record<string, unknown>;+ readonly sessionId: string;+}): Record<string, unknown> {+ const turnResult = input.result.outcome.result;+ const usage = input.result.usage ?? input.result.outcome.usageDelta;+ return recordNestedAgentInvocationTerminal({+ callId: input.result.callId,+ serializedContext: input.serializedContext,+ sessionId: input.sessionId,+ terminal: {+ acceptedAtMs: input.acceptedAtMs,+ error: turnResult.kind === "failed" ? invocationError(turnResult.error) : undefined,+ outcome:+ turnResult.kind === "succeeded"+ ? "completed"+ : turnResult.kind === "cancelled"+ ? "cancelled"+ : "failed",+ usage: {+ inputTokenDetails: {+ cacheReadTokens: usage.cacheReadTokens,+ cacheWriteTokens: usage.cacheWriteTokens,+ },+ inputTokens: usage.inputTokens,+ outputTokens: usage.outputTokens,+ },+ },+ });+}++/** Called inside dispatch/settlement steps, so replay reuses the flushed context. */+export async function flushAgentInvocationTraces(+ serializedContext: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ if (serializedContext[AGENT_TRACE_CONTEXT_KEY] === undefined) return serializedContext;+ const runtime = getInstrumentationRuntime();+ if (runtime?.flushSettledInvocations === undefined) return serializedContext;+ try {+ const ctx = await deserializeContext(serializedContext);+ await contextStorage.run(ctx, runtime.flushSettledInvocations);+ return {+ ...serializedContext,+ [AGENT_TRACE_CONTEXT_KEY]: serializeContext(ctx)[AGENT_TRACE_CONTEXT_KEY],+ };+ } catch (error) {+ log.warn("could not materialize settled invocation traces", { error: formatError(error) });+ return serializedContext;+ }+}++export function invocationError(value: unknown): Error {+ if (value instanceof Error) {+ return boundedTraceError(value);+ }+ if (typeof value === "object" && value !== null && "message" in value) {+ return new Error(truncateTelemetryText(String(value.message), 4096));+ }+ return new Error(+ typeof value === "string" ? truncateTelemetryText(value, 4096) : "Agent invocation failed.",+ );+}diff --git a/packages/eve/src/tracing/agent-otel-attributes.test.ts b/packages/eve/src/tracing/agent-otel-attributes.test.tsnew file mode 100644index 000000000..f067940b3--- /dev/null+++ b/packages/eve/src/tracing/agent-otel-attributes.test.ts@@ -0,0 +1,30 @@+import { afterEach, describe, expect, it, vi } from "vitest";++import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";++afterEach(() => {+ vi.unstubAllEnvs();+});++describe("agentTraceIdentityAttributes", () => {+ const input = { rootSessionId: "conversation-1", sessionId: "session-1" };++ it("uses the GenAI conversation identifier outside Vercel", () => {+ vi.stubEnv("VERCEL_ENV", undefined);++ expect(agentTraceIdentityAttributes(input)).toEqual({+ "agent.trace.schema.version": 4,+ "gen_ai.conversation.id": "conversation-1",+ });+ });++ it("adds the Vercel session identifier on Vercel", () => {+ vi.stubEnv("VERCEL_ENV", "preview");++ expect(agentTraceIdentityAttributes(input)).toEqual({+ "agent.trace.schema.version": 4,+ "gen_ai.conversation.id": "conversation-1",+ "vercel.session_id": "session-1",+ });+ });+});diff --git a/packages/eve/src/tracing/agent-otel-attributes.ts b/packages/eve/src/tracing/agent-otel-attributes.tsnew file mode 100644index 000000000..8444862f1--- /dev/null+++ b/packages/eve/src/tracing/agent-otel-attributes.ts@@ -0,0 +1,15 @@+import { resolveConversationId } from "#tracing/conversation-context.js";+import { AGENT_TRACE_SCHEMA_VERSION } from "#tracing/agent-span-contract.js";++export function agentTraceIdentityAttributes(input: {+ readonly rootSessionId: string;+ readonly sessionId: string;+}): Record<string, string | number> {+ return {+ "agent.trace.schema.version": AGENT_TRACE_SCHEMA_VERSION,+ "gen_ai.conversation.id": resolveConversationId(input.rootSessionId),+ ...(process.env.VERCEL_ENV === undefined+ ? undefined+ : { "vercel.session_id": input.sessionId }),+ };+}diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.tsindex bf699a0da..18cfe632b 100644--- a/packages/eve/src/tracing/agent-otel-provider.test.ts+++ b/packages/eve/src/tracing/agent-otel-provider.test.ts@@ -15,7 +15,7 @@ import {trace as runtimeTrace,} from "#compiled/@opentelemetry/api/index.js";import { ContextContainer, contextStorage } from "#context/container.js";-import { SessionTraceSeedKey } from "#context/keys.js";+import { ActiveChannelDeliveriesKey, SessionTraceSeedKey } from "#context/keys.js";import { deserializeContext, serializeContext } from "#context/serialize.js";import { createAiSdkHookBridge } from "#instrumentation/ai-sdk-hook-bridge.js";import {@@ -316,6 +316,7 @@ async function publishTurnStarted(input: {readonly parentTraceContext?: InstrumentationTraceContext;readonly rootSessionId?: string;readonly sessionId: string;+ readonly traceSeed?: InstrumentationTraceContext;readonly turnId: string;readonly turnSequence: number;}): Promise<void> {@@ -329,6 +330,7 @@ async function publishTurnStarted(input: {parentTraceContext: input.parentTraceContext,rootSessionId,sessionId: input.sessionId,+ traceSeed: input.traceSeed,type: "session.started",});await input.hooks.publish({@@ -373,7 +375,7 @@ function nanos(hrTime: readonly [number, number]): bigint {describe("createAgentOtelInstrumentation", () => {it.each(["public", "private"] as const)(- "exports explicit names for %s spans without changing session grouping",+ "exports explicit names for %s spans across bounded activation traces",async (channelAudience) => {const metadata = new InMemorySpanExporter();const runtime = createRuntime(undefined, undefined, [@@ -394,25 +396,24 @@ describe("createAgentOtelInstrumentation", () => {}await runtime.provider.forceFlush();const exported = runtime.exporter.getFinishedSpans();- const traceId = exported[0]!.spanContext().traceId;- expect(new Set(exported.map((span) => span.spanContext().traceId)).size).toBe(1);+ const traceIds = [...new Set(exported.map((span) => span.spanContext().traceId))];+ expect(traceIds).toHaveLength(2);for (const spans of [exported, metadata.getFinishedSpans()]) {- const parsed = parseLocalTraceSegment(- new TextDecoder().decode(JsonTraceSerializer.serializeRequest(spans)!),- traceId,- );+ const serialized = new TextDecoder().decode(JsonTraceSerializer.serializeRequest(spans)!);+ const parsed = traceIds.flatMap((traceId) => parseLocalTraceSegment(serialized, traceId));expect(parsed).toHaveLength(exported.length);for (const span of parsed) {expect(span.attributes["resource.name"]).toBe(span.name);expect(span.attributes["operation.name"]).toBe(span.attributes["gen_ai.operation.name"] ?? span.name,);+ expect(span.attributes["agent.trace.schema.version"]).toBe(4);+ expect(span.attributes["gen_ai.conversation.id"]).toBe("session-1");}- const session = parsed.find((span) => span.name === "agent.session")!;- expect(session.attributes["agent.trace.schema.version"]).toBe(3);+ expect(parsed.some((span) => span.name === "agent.session")).toBe(false);const turns = parsed.filter((span) => span.name === "invoke_agent weather");expect(turns).toHaveLength(2);- expect(turns.every((span) => span.parentSpanId === session.spanId)).toBe(true);+ expect(turns.every((span) => span.parentSpanId === undefined)).toBe(true);expect(turns.map((span) => span.attributes["agent.turn.outcome"])).toEqual(["completed","completed",@@ -500,18 +501,17 @@ describe("createAgentOtelInstrumentation", () => {expect(turnTrace).toEqual(sessionTrace);expect(replayedTrace).toEqual(turnTrace);const spans = runtime.exporter.getFinishedSpans();- const session = byName(spans, "agent.session")[0]!;const turn = byName(spans, "invoke_agent weather")[0]!;- expect(session.spanContext()).toMatchObject({+ expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(turn.spanContext()).toMatchObject({spanId: turnTrace.spanId,traceFlags: turnTrace.traceFlags,traceId: turnTrace.traceId,});- expect(turn.spanContext().traceId).toBe(turnTrace.traceId);- expect(turn.parentSpanContext?.spanId).toBe(turnTrace.spanId);+ expect(turn.parentSpanContext).toBeUndefined();});- it("uses the pre-allocated trace seed's trace id for agent.session", async () => {+ it("uses the pre-allocated trace seed for the first activation root", async () => {const runtime = createRuntime();const seed: InstrumentationTraceContext = {spanId: "a".repeat(16),@@ -526,21 +526,87 @@ describe("createAgentOtelInstrumentation", () => {traceSeed: seed,type: "session.started" as const,};+ const turnEvent = {+ idempotencyKey: turnIdempotencyKey("session-seed", "turn-1"),+ rootSessionId: "session-seed",+ sequence: 0,+ sessionId: "session-seed",+ turnId: "turn-1",+ type: "turn.started" as const,+ };await runtime.prepareSessionTrace(sessionEvent);+ await runtime.prepareTurnTrace(turnEvent);await runtime.hooks.publish(sessionEvent);+ await runtime.hooks.publish(turnEvent);+ await completeTurn(runtime.hooks, "session-seed", "turn-1");await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- const session = byName(spans, "agent.session")[0]!;- expect(session.spanContext().traceId).toBe(seed.traceId);- expect(session.spanContext().spanId).toBe(seed.spanId);- expect(session.attributes).toMatchObject({- "operation.name": "agent.session",- "resource.name": "agent.session",+ const turn = byName(spans, "invoke_agent weather")[0]!;+ expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(turn.spanContext()).toMatchObject(seed);+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.attributes).toMatchObject({+ "operation.name": "invoke_agent",+ "resource.name": "invoke_agent weather",});});+ it("links a delegated activation to the exact caller from its own trace", async () => {+ const runtime = createRuntime();+ const parent = {+ spanId: "c".repeat(16),+ traceFlags: 1,+ traceId: "d".repeat(32),+ };+ const seed = {+ spanId: "a".repeat(16),+ traceFlags: 1,+ traceId: "b".repeat(32),+ };+ const parentLineage = {+ callId: "call-child",+ sessionId: "parent-session",+ subagentName: "researcher",+ turnId: "parent-turn",+ };++ await publishTurnStarted({+ hooks: runtime.hooks,+ parentLineage,+ parentTraceContext: parent,+ rootSessionId: "root-session",+ sessionId: "child-session",+ traceSeed: seed,+ turnId: "child-turn",+ turnSequence: 0,+ });+ await completeTurn(runtime.hooks, "child-session", "child-turn");+ await runtime.provider.forceFlush();++ const spans = runtime.exporter.getFinishedSpans();+ const invocation = byName(spans, "invoke_agent weather")[0]!;+ expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(invocation.spanContext().traceId).toBe(seed.traceId);+ expect(invocation.parentSpanContext).toBeUndefined();+ expect(invocation.links).toEqual([+ {+ context: { ...parent, isRemote: false },+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);+ expect(invocation.attributes).toMatchObject({+ "gen_ai.conversation.id": "root-session",+ "gen_ai.operation.name": "invoke_agent",+ });+ expect(invocation.attributes).not.toHaveProperty("agent.parent_call.id");+ expect(invocation.attributes).not.toHaveProperty("agent.parent_run.id");+ expect(invocation.attributes).not.toHaveProperty("agent.root_run.id");+ expect(invocation.attributes).not.toHaveProperty("agent.session.id");+ expect(invocation.attributes).not.toHaveProperty("vercel.session_id");+ });+it("falls back to fresh ids when no trace seed is present", async () => {const runtime = createRuntime();const sessionEvent = {@@ -550,16 +616,31 @@ describe("createAgentOtelInstrumentation", () => {sessionId: "session-noseed",type: "session.started" as const,};+ const turnEvent = {+ idempotencyKey: turnIdempotencyKey("session-noseed", "turn-1"),+ rootSessionId: "session-noseed",+ sequence: 0,+ sessionId: "session-noseed",+ turnId: "turn-1",+ type: "turn.started" as const,+ };const trace = await runtime.prepareSessionTrace(sessionEvent);+ const turnTrace = await runtime.prepareTurnTrace(turnEvent);await runtime.hooks.publish(sessionEvent);+ await runtime.hooks.publish(turnEvent);+ await completeTurn(runtime.hooks, "session-noseed", "turn-1");await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- const session = byName(spans, "agent.session")[0]!;- expect(session.spanContext().traceId).toBe(trace.traceId);- // Fresh span id is random, not derived — just verify it matches the returned context.- expect(session.spanContext().spanId).toBe(trace.spanId);+ const turn = byName(spans, "invoke_agent weather")[0]!;+ expect(turnTrace).toEqual(trace);+ expect(turn.spanContext()).toMatchObject({+ spanId: trace.spanId,+ traceFlags: trace.traceFlags,+ traceId: trace.traceId,+ });+ expect(turn.parentSpanContext).toBeUndefined();});it("passes channelType to the policy on the seedless fallback path", async () => {@@ -582,7 +663,7 @@ describe("createAgentOtelInstrumentation", () => {expect(captured?.channelType).toBe("slack");});- it("inherits parent trace context for delegated agents", async () => {+ it("allocates independent trace context for delegated agents", async () => {const runtime = createRuntime();const parentTrace: InstrumentationTraceContext = {spanId: "c".repeat(16),@@ -599,8 +680,9 @@ describe("createAgentOtelInstrumentation", () => {};const trace = await runtime.prepareSessionTrace(sessionEvent);- expect(trace.traceId).toBe(parentTrace.traceId);- expect(trace.spanId).toBe(parentTrace.spanId);+ expect(trace.traceId).not.toBe(parentTrace.traceId);+ expect(trace.spanId).not.toBe(parentTrace.spanId);+ expect(trace.traceFlags).toBe(parentTrace.traceFlags);});it("treats the seed as authoritative when late policy would reject", async () => {@@ -618,15 +700,26 @@ describe("createAgentOtelInstrumentation", () => {traceSeed: seed,type: "session.started" as const,};+ const turnEvent = {+ idempotencyKey: turnIdempotencyKey("session-seed-authoritative", "turn-1"),+ rootSessionId: "session-seed-authoritative",+ sequence: 0,+ sessionId: "session-seed-authoritative",+ turnId: "turn-1",+ type: "turn.started" as const,+ };await runtime.prepareSessionTrace(sessionEvent);+ await runtime.prepareTurnTrace(turnEvent);await runtime.hooks.publish(sessionEvent);+ await runtime.hooks.publish(turnEvent);+ await completeTurn(runtime.hooks, "session-seed-authoritative", "turn-1");await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- const session = byName(spans, "agent.session")[0]!;- expect(session.spanContext().traceId).toBe(seed.traceId);- expect(session.spanContext().traceFlags).toBe(1);+ const turn = byName(spans, "invoke_agent weather")[0]!;+ expect(turn.spanContext().traceId).toBe(seed.traceId);+ expect(turn.spanContext().traceFlags).toBe(1);});it("treats an unsampled seed as authoritative when late policy would accept", async () => {@@ -644,59 +737,63 @@ describe("createAgentOtelInstrumentation", () => {traceSeed: seed,type: "session.started" as const,};+ const turnEvent = {+ idempotencyKey: turnIdempotencyKey("session-seed-unsampled", "turn-1"),+ rootSessionId: "session-seed-unsampled",+ sequence: 0,+ sessionId: "session-seed-unsampled",+ turnId: "turn-1",+ type: "turn.started" as const,+ };await runtime.prepareSessionTrace(sessionEvent);+ await runtime.prepareTurnTrace(turnEvent);await runtime.hooks.publish(sessionEvent);+ await runtime.hooks.publish(turnEvent);+ await completeTurn(runtime.hooks, "session-seed-unsampled", "turn-1");await runtime.provider.forceFlush();- const spans = runtime.exporter.getFinishedSpans();- expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(runtime.exporter.getFinishedSpans()).toEqual([]);});- it.each([- ["cancelled", SpanStatusCode.UNSET],- ["failed", SpanStatusCode.ERROR],- ] as const)("records a %s channel delivery with the expected status", async (outcome, status) => {- const runtime = createRuntime();- const ctx = new ContextContainer();- const delivery = {- channelKind: "channel:slack",- channelName: "slack",- deliveryId: `delivery-${outcome}`,- };- const idempotencyKey = `channel-delivery:session-1:${delivery.deliveryId}`;+ it.each(["cancelled", "failed"] as const)(+ "does not emit a standalone span for a %s delivery without a turn",+ async (outcome) => {+ const runtime = createRuntime();+ const ctx = new ContextContainer();+ const delivery = {+ channelKind: "channel:slack",+ channelName: "slack",+ deliveryId: `delivery-${outcome}`,+ };+ const idempotencyKey = `channel-delivery:session-1:${delivery.deliveryId}`;- await contextStorage.run(ctx, async () => {- await runtime.hooks.publish({- agentName: "weather",- delivery,- idempotencyKey,- rootSessionId: "session-1",- sessionId: "session-1",- type: "channel.delivery.started",- });- await runtime.hooks.publish({- delivery,- error: outcome === "failed" ? new Error("failed") : undefined,- errorCode: outcome === "failed" ? "CHANNEL_DELIVERY_FAILED" : undefined,- idempotencyKey,- outcome,- rootSessionId: "session-1",- sessionId: "session-1",- type: `channel.delivery.${outcome}`,+ await contextStorage.run(ctx, async () => {+ await runtime.hooks.publish({+ agentName: "weather",+ delivery,+ idempotencyKey,+ rootSessionId: "session-1",+ sessionId: "session-1",+ type: "channel.delivery.started",+ });+ await runtime.hooks.publish({+ delivery,+ error: outcome === "failed" ? new Error("failed") : undefined,+ errorCode: outcome === "failed" ? "CHANNEL_DELIVERY_FAILED" : undefined,+ idempotencyKey,+ outcome,+ rootSessionId: "session-1",+ sessionId: "session-1",+ type: `channel.delivery.${outcome}`,+ });});- });- const span = runtime.exporter- .getFinishedSpans()- .find((candidate) => candidate.name === "agent.channel.delivery")!;- expect(span.status.code).toBe(status);- expect(span.attributes["error.type"]).toBe(- outcome === "failed" ? "CHANNEL_DELIVERY_FAILED" : undefined,- );- });+ expect(runtime.exporter.getFinishedSpans()).toEqual([]);+ },+ );- it("maps channel delivery under the session trace with an HTTP request link", async () => {+ it("maps one channel delivery onto its activation with an HTTP request link", async () => {const runtime = createRuntime();const ctx = new ContextContainer();const requestTraceContext = {@@ -717,7 +814,9 @@ describe("createAgentOtelInstrumentation", () => {idempotencyKey: "channel-delivery:session-1:delivery-1",input: { message: "private" },rootSessionId: "session-1",+ sequence: 0,sessionId: "session-1",+ turnId: "turn_0",type: "channel.delivery.started" as const,};const completed = {@@ -743,48 +842,52 @@ describe("createAgentOtelInstrumentation", () => {type: "session.started",});await runtime.hooks.publish(started);+ await runtime.hooks.publish({+ idempotencyKey: turnIdempotencyKey("session-1", "turn_0"),+ rootSessionId: "session-1",+ sequence: 0,+ sessionId: "session-1",+ turnId: "turn_0",+ type: "turn.started",+ });+ ctx.set(ActiveChannelDeliveriesKey, [+ {+ agentName: started.agentName,+ delivery: started.delivery,+ policyAgentName: started.agentName,+ rootSessionId: started.rootSessionId,+ sequence: 0,+ sessionId: started.sessionId,+ turnId: "turn_0",+ },+ ]);await runtime.hooks.publish(completed);+ await completeTurn(runtime.hooks, "session-1", "turn_0");});const spans = runtime.exporter.getFinishedSpans();- const session = spans.find((span) => span.name === "agent.session")!;- const delivery = spans.find((span) => span.name === "agent.channel.delivery")!;- expect(delivery.attributes).toMatchObject({- "operation.name": "agent.channel.delivery",- "resource.name": "agent.channel.delivery",- });- expect(delivery.kind).toBe(SpanKind.CONSUMER);- expect(delivery.parentSpanContext?.spanId).toBe(session.spanContext().spanId);- expect(delivery.links).toEqual([+ const turn = spans.find((span) => span.name === "invoke_agent support")!;+ expect(byName(spans, "agent.channel.delivery")).toHaveLength(0);+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.links).toEqual([expect.objectContaining({attributes: { "eve.link.type": "channel.request" },context: expect.objectContaining(requestTraceContext),}),]);- expect(delivery.attributes).toMatchObject({+ expect(turn.attributes).toMatchObject({"agent.channel.delivery.id": "delivery-1","agent.channel.delivery.input": JSON.stringify({ message: "private" }),- "agent.channel.delivery.outcome": "completed","agent.channel.kind": "channel:slack","agent.channel.name": "slack","agent.channel.request.id": "iad1::request-1",- "agent.session.id": "session-1","agent.turn.id": "turn_0","agent.turn.sequence": 0,+ "gen_ai.conversation.id": "session-1",});-- await contextStorage.run(ctx, async () => {- await runtime.hooks.publish(started);- await runtime.hooks.publish(completed);- });- const replayed = runtime.exporter- .getFinishedSpans()- .filter((span) => span.name === "agent.channel.delivery");- expect(replayed).toHaveLength(2);- expect(replayed[1]!.spanContext().spanId).toBe(replayed[0]!.spanContext().spanId);});- it("preserves a remote parent when delivery starts before the session event", async () => {+ it("does not emit delivery spans when remote deliveries fan in before the session event", async () => {const runtime = createRuntime();const ctx = new ContextContainer();const parentTraceContext = {@@ -806,7 +909,9 @@ describe("createAgentOtelInstrumentation", () => {idempotencyKey: `channel-delivery:remote-session:${delivery.deliveryId}`,parentTraceContext,rootSessionId: "parent-session",+ sequence: 0,sessionId: "remote-session",+ turnId: "turn-remote",type: "channel.delivery.started",});}@@ -828,6 +933,18 @@ describe("createAgentOtelInstrumentation", () => {turnId: "turn-remote",type: "turn.started",});+ ctx.set(+ ActiveChannelDeliveriesKey,+ deliveries.map((delivery) => ({+ agentName: "weather",+ delivery,+ policyAgentName: "weather",+ rootSessionId: "parent-session",+ sequence: 0,+ sessionId: "remote-session",+ turnId: "turn-remote",+ })),+ );for (const delivery of deliveries) {await runtime.hooks.publish({delivery,@@ -845,26 +962,26 @@ describe("createAgentOtelInstrumentation", () => {await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- const channelDeliveries = byName(spans, "agent.channel.delivery");- const firstDelivery = channelDeliveries.find(- (span) => span.attributes["agent.channel.delivery.id"] === "delivery-remote-first",- )!;const turn = byName(spans, "invoke_agent")[0]!;expect(byName(spans, "agent.session")).toHaveLength(0);- expect(channelDeliveries).toHaveLength(2);- for (const delivery of channelDeliveries) {- expect(delivery.parentSpanContext).toMatchObject(parentTraceContext);- }- expect(turn.parentSpanContext?.spanId).toBe(firstDelivery.spanContext().spanId);- expect(turn.spanContext().traceId).toBe(parentTraceContext.traceId);+ expect(byName(spans, "agent.channel.delivery")).toHaveLength(0);+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.links).toEqual([+ {+ context: parentTraceContext,+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);+ expect(turn.spanContext().traceId).not.toBe(parentTraceContext.traceId);expect(turn.attributes).toMatchObject({- "gen_ai.conversation.id": "remote-session",+ "gen_ai.conversation.id": "parent-session","gen_ai.operation.name": "invoke_agent",});+ expect(turn.attributes).not.toHaveProperty("agent.channel.delivery.id");expect(turn.attributes).not.toHaveProperty("gen_ai.agent.name");});- it("emits the agent hierarchy in one session trace", async () => {+ it("emits one bounded activation trace rooted at invoke_agent", async () => {const runtime = createRuntime();const delivery = runtime.tracer.startSpan("workflow.delivery");const activeContext = runtimeTrace.setSpan(ROOT_CONTEXT, delivery);@@ -893,15 +1010,8 @@ describe("createAgentOtelInstrumentation", () => {const action = byName(spans, "agent.action")[0]!;const tool = byName(spans, "execute_tool weather")[0]!;- const session = byName(spans, "agent.session")[0]!;- expect(session.parentSpanContext).toBeUndefined();- expect(session.events.map((event) => event.name)).toEqual(["session.started"]);- expect(session.attributes).toMatchObject({- "agent.channel.audience": "public",- "agent.session.id": "session-1",- "agent.trace.schema.version": 3,- });- expect(turn.parentSpanContext?.spanId).toBe(session.spanContext().spanId);+ expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(turn.parentSpanContext).toBeUndefined();expect(step.parentSpanContext?.spanId).toBe(turn.spanContext().spanId);expect(step.links).toEqual([expect.objectContaining({@@ -910,6 +1020,7 @@ describe("createAgentOtelInstrumentation", () => {}),]);expect(model.parentSpanContext?.spanId).toBe(step.spanContext().spanId);+ expect(model.kind).toBe(SpanKind.CLIENT);expect(executionParents.some((parent) =>@@ -920,6 +1031,9 @@ describe("createAgentOtelInstrumentation", () => {expect(tool.parentSpanContext?.spanId).toBe(action.spanContext().spanId);for (const span of [turn, step, model, action, tool]) {expect(span.attributes).not.toHaveProperty("agent.channel.audience");+ expect(span.attributes).toMatchObject({+ "gen_ai.conversation.id": "session-1",+ });}expect(byName(spans, "ai.streamText")).toHaveLength(0);expect(@@ -935,11 +1049,11 @@ describe("createAgentOtelInstrumentation", () => {expect(turn.kind).toBe(SpanKind.INTERNAL);expect(turn.attributes).toMatchObject({"agent.name": "weather",+ "agent.usage.input_tokens": 10,+ "agent.usage.output_tokens": 5,"gen_ai.agent.name": "weather","gen_ai.conversation.id": "session-1","gen_ai.operation.name": "invoke_agent",- "gen_ai.usage.input_tokens": 10,- "gen_ai.usage.output_tokens": 5,});expect(nanos(turn.startTime)).toBeLessThanOrEqual(nanos(step.startTime));expect(nanos(turn.endTime)).toBeGreaterThanOrEqual(nanos(step.endTime) - 1_000_000n);@@ -947,10 +1061,17 @@ describe("createAgentOtelInstrumentation", () => {"agent.framework.name": "eve","agent.model.id": "claude-test","agent.model.provider": "anthropic",+ "agent.usage.cache_read_tokens": 4,+ "agent.usage.cache_write_tokens": 2,"agent.usage.input_tokens": 10,"agent.usage.output_tokens": 5,+ });+ expect(model.attributes).toMatchObject({+ "gen_ai.response.id": "response-1","gen_ai.usage.cache_creation.input_tokens": 2,"gen_ai.usage.cache_read.input_tokens": 4,+ "gen_ai.usage.input_tokens": 10,+ "gen_ai.usage.output_tokens": 5,});expect(action.attributes).toMatchObject({"agent.action.kind": "tool-call",@@ -967,6 +1088,117 @@ describe("createAgentOtelInstrumentation", () => {expect(tool.attributes).not.toHaveProperty("gen_ai.agent.name");});+ it("keeps a workflow tool intact while emitting independent nested callers", async () => {+ const stateStore = new InMemoryAgentTraceStateStore();+ const runtime = createRuntime(stateStore);+ const scope: InstrumentationAttemptScope = {+ attemptId: "session-nested:turn-1:0:0",+ attemptIndex: 0,+ functionId: "weather",+ sessionId: "session-nested",+ stepIndex: 0,+ turnId: "turn-1",+ };+ const actionKey = actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow");+ const toolKey = `tool:${scope.attemptId}:workflow:0`;++ await publishTurnStarted({+ hooks: runtime.hooks,+ sessionId: scope.sessionId,+ turnId: scope.turnId,+ turnSequence: 0,+ });+ await runtime.hooks.publish({+ idempotencyKey: attemptIdempotencyKey(scope),+ operation: { modelId: "model", operationId: "ai.streamText", provider: "test" },+ scope,+ type: "step.attempt.started",+ });+ await runtime.hooks.publish({+ callId: "workflow",+ idempotencyKey: actionKey,+ input: {},+ isWorkflowTool: true,+ kind: "tool-call",+ name: "coordinate",+ scope,+ type: "action.started",+ });+ await runtime.hooks.publish({+ callId: "workflow",+ idempotencyKey: toolKey,+ input: {},+ scope,+ toolName: "coordinate",+ type: "tool.call.started",+ });+ await runtime.hooks.publish({+ idempotencyKey: toolKey,+ output: { output: "done", type: "result" },+ scope,+ type: "tool.call.completed",+ });+ await runtime.hooks.publish({+ idempotencyKey: actionKey,+ outcome: "completed",+ output: { output: "done", type: "result" },+ scope,+ type: "action.completed",+ });++ const anchor = stateStore.findActionAnchor(scope.sessionId, scope.turnId, "workflow")!;+ stateStore.setInvocation(+ actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow:first"),+ {+ ...anchor,+ callId: "workflow:first",+ kind: "subagent-call",+ name: "research",+ parent: { ...anchor.parent, spanId: anchor.spanId },+ parentActionCallId: "workflow",+ spanId: "b".repeat(16),+ terminal: { outcome: "completed" },+ },+ );+ stateStore.setInvocation(+ actionIdempotencyKey(scope.sessionId, scope.turnId, "workflow:second"),+ {+ ...anchor,+ callId: "workflow:second",+ kind: "remote-agent-call",+ name: "review",+ parent: { ...anchor.parent, spanId: anchor.spanId },+ parentActionCallId: "workflow",+ spanId: "d".repeat(16),+ terminal: { error: new Error("review failed"), outcome: "failed" },+ },+ );+ await runtime.hooks.publish({+ idempotencyKey: attemptIdempotencyKey(scope),+ scope,+ type: "step.attempt.completed",+ });+ await completeTurn(runtime.hooks, scope.sessionId, scope.turnId);+ await runtime.provider.forceFlush();++ const spans = runtime.exporter.getFinishedSpans();+ const outer = byName(spans, "agent.action")[0]!;+ const tool = byName(spans, "execute_tool coordinate")[0]!;+ const first = spans.find(+ (span) => span.attributes["agent.action.call_id"] === "workflow:first",+ )!;+ const second = spans.find(+ (span) => span.attributes["agent.action.call_id"] === "workflow:second",+ )!;+ expect(byName(spans, "agent.action")).toHaveLength(3);+ expect(first.attributes).not.toHaveProperty("gen_ai.operation.name");+ expect(second.attributes).not.toHaveProperty("gen_ai.operation.name");+ expect(tool.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);+ expect(first.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);+ expect(second.parentSpanContext?.spanId).toBe(outer.spanContext().spanId);+ expect(second.status.code).toBe(SpanStatusCode.ERROR);+ });+it.each(["private", "unknown"] as const)("records %s conversation traces by default",async (audience) => {@@ -982,7 +1214,8 @@ describe("createAgentOtelInstrumentation", () => {});await runtime.provider.forceFlush();- expect(byName(runtime.exporter.getFinishedSpans(), "agent.session")).toHaveLength(1);+ const turn = byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;+ expect(turn.attributes).not.toHaveProperty("vercel.session_id");},);@@ -1089,7 +1322,7 @@ describe("createAgentOtelInstrumentation", () => {).resolves.toMatchObject({ input: undefined });});- it("normalizes an event-carried drop before opening its session span", async () => {+ it("normalizes an event-carried drop before preparing its activation trace", async () => {const runtime = createRuntime(new InMemoryAgentTraceStateStore(), null);const ctx = new ContextContainer();const traceSeed = {@@ -1119,7 +1352,7 @@ describe("createAgentOtelInstrumentation", () => {}),);await runtime.provider.forceFlush();- expect(byName(runtime.exporter.getFinishedSpans(), "agent.session")).toHaveLength(0);+ expect(runtime.exporter.getFinishedSpans()).toEqual([]);});it("reconstructs a sampled public session decision when persisted policy is absent", async () => {@@ -1184,7 +1417,7 @@ describe("createAgentOtelInstrumentation", () => {},);- it("preserves the parent's sampling decision for adopted traces", async () => {+ it("preserves recording eligibility without adopting the parent's trace", async () => {const runtime = createRuntime(new InMemoryAgentTraceStateStore(), null);await emitAttempt({@@ -1202,11 +1435,10 @@ describe("createAgentOtelInstrumentation", () => {});await runtime.provider.forceFlush();- // The parent's traceFlags are authoritative — the child does not re-evaluate.const spans = runtime.exporter.getFinishedSpans();expect(spans.length).toBeGreaterThan(0);for (const span of spans) {- expect(span.spanContext().traceId).toBe("b".repeat(32));+ expect(span.spanContext().traceId).not.toBe("b".repeat(32));}});@@ -1223,7 +1455,7 @@ describe("createAgentOtelInstrumentation", () => {});await runtime.provider.forceFlush();- expect(byName(runtime.exporter.getFinishedSpans(), "agent.session")).toHaveLength(1);+ expect(byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")).toHaveLength(1);});it("writes merged runtime context onto the step and chat spans", async () => {@@ -1311,6 +1543,9 @@ describe("createAgentOtelInstrumentation", () => {expect(byName(redeliverySpans, name)[0]!.spanContext().spanId).toBe(byName(firstSpans, name)[0]!.spanContext().spanId,);+ expect(byName(redeliverySpans, name)[0]!.spanContext().traceId).toBe(+ byName(firstSpans, name)[0]!.spanContext().traceId,+ );}expect(byName(redeliverySpans, "chat claude-test")[0]!.spanContext().spanId).not.toBe(byName(firstSpans, "chat claude-test")[0]!.spanContext().spanId,@@ -1579,7 +1814,7 @@ describe("createAgentOtelInstrumentation", () => {"agent.approval.response": expect.stringContaining("approve"),"operation.name": "agent.approval","resource.name": "agent.approval",- "agent.session.id": "session-1",+ "gen_ai.conversation.id": "session-1","agent.step.index": 0,"agent.turn.id": "turn-1",});@@ -1685,7 +1920,7 @@ describe("createAgentOtelInstrumentation", () => {});expect(audiences.get("agent.approval")).toBe("private");- expect(audiences.get("agent.channel.delivery")).toBe("private");+ expect(audiences.has("agent.channel.delivery")).toBe(false);expect(audiences.get("invoke_agent weather")).toBe("private");});@@ -1804,7 +2039,7 @@ describe("createAgentOtelInstrumentation", () => {for (const name of ["chat claude-test", "agent.action", "execute_tool weather"]) {const span = byName(spans, name)[0]!;expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: error.message });- if (name !== "agent.action") expect(span.attributes["error.type"]).toBe("Error");+ expect(span.attributes["error.type"]).toBe("Error");expect(span.events).toContainEqual(expect.objectContaining({attributes: expect.objectContaining({ "exception.message": error.message }),@@ -1831,7 +2066,8 @@ describe("createAgentOtelInstrumentation", () => {});await runtime.provider.forceFlush();- const action = runtime.exporter.getFinishedSpans().find((span) => span.name === "agent.action");+ const spans = runtime.exporter.getFinishedSpans();+ const action = byName(spans, "agent.action")[0];expect(action?.attributes).toMatchObject({"agent.action.kind": "subagent-call","agent.action.name": "weather",@@ -1844,6 +2080,11 @@ describe("createAgentOtelInstrumentation", () => {expect(Object.keys(action?.attributes ?? {}).some((key) => key.startsWith("gen_ai.usage.")),).toBe(false);+ expect(action?.attributes).not.toHaveProperty("gen_ai.tool.call.arguments");+ expect(action?.attributes).not.toHaveProperty("gen_ai.tool.call.result");+ expect(action?.attributes).not.toHaveProperty("gen_ai.operation.name");+ expect(byName(spans, "invoke_agent weather")).toHaveLength(1);+ expect(byName(spans, "execute_tool weather")).toHaveLength(1);});it("captures model and tool inputs/outputs on the operation spans", async () => {@@ -2134,7 +2375,7 @@ describe("createAgentOtelInstrumentation", () => {}});- it("reuses one trace id across turns", async () => {+ it("opens a fresh trace for each resumed turn", async () => {const stateStore = new InMemoryAgentTraceStateStore();const firstRuntime = createRuntime(stateStore);await emitAttempt({@@ -2162,7 +2403,13 @@ describe("createAgentOtelInstrumentation", () => {...byName(secondRuntime.exporter.getFinishedSpans(), "invoke_agent weather"),];expect(turns).toHaveLength(2);- expect(turns[0]!.spanContext().traceId).toBe(turns[1]!.spanContext().traceId);+ expect(turns[0]!.spanContext().traceId).not.toBe(turns[1]!.spanContext().traceId);+ for (const turn of turns) {+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.attributes).toMatchObject({+ "gen_ai.conversation.id": "session-1",+ });+ }const firstModel = byName(firstRuntime.exporter.getFinishedSpans(), "chat claude-test")[0]!;const secondModel = byName(secondRuntime.exporter.getFinishedSpans(), "chat claude-test")[0]!;expect(firstModel.attributes["ai.prompt.system"]).toBe(@@ -2181,7 +2428,7 @@ describe("createAgentOtelInstrumentation", () => {expect([...byName(firstRuntime.exporter.getFinishedSpans(), "agent.session"),...byName(secondRuntime.exporter.getFinishedSpans(), "agent.session"),- ]).toHaveLength(1);+ ]).toHaveLength(0);});it("restores durable turn context when a replacement worker runs the attempt", async () => {@@ -2221,7 +2468,7 @@ describe("createAgentOtelInstrumentation", () => {expect(step.spanContext().traceId).toBe(turn.spanContext().traceId);});- it("opens a fresh trace when a durable attempt replays before its state checkpoints", async () => {+ it.each([0, 1])("reuses activation identity before turn %s checkpoints", async (turnSequence) => {const firstRuntime = createRuntime();const replayRuntime = createRuntime();await emitAttempt({@@ -2229,14 +2476,14 @@ describe("createAgentOtelInstrumentation", () => {runInContext: firstRuntime.runInContext,sessionId: "session-1",turnId: "turn-1",- turnSequence: 0,+ turnSequence,});await emitAttempt({hooks: replayRuntime.hooks,runInContext: replayRuntime.runInContext,sessionId: "session-1",turnId: "turn-1",- turnSequence: 0,+ turnSequence,});const firstTurn = byName(firstRuntime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;@@ -2244,16 +2491,12 @@ describe("createAgentOtelInstrumentation", () => {replayRuntime.exporter.getFinishedSpans(),"invoke_agent weather",)[0]!;- // The abandoned attempt keeps its own trace rather than interleaving with- // the retry. Both carry `agent.session.id`, so the session view still- // resolves to every trace the session produced.- expect(replayTurn.spanContext().traceId).not.toBe(firstTurn.spanContext().traceId);- expect(replayTurn.attributes["agent.session.id"]).toBe(- firstTurn.attributes["agent.session.id"],- );+ expect(replayTurn.spanContext()).toEqual(firstTurn.spanContext());+ expect(replayTurn.attributes).not.toHaveProperty("vercel.session_id");+ expect(firstTurn.attributes).not.toHaveProperty("vercel.session_id");});- it("keeps a long session on one persisted trace", async () => {+ it("bounds a long session to one trace per turn", async () => {const runtime = createRuntime();const turnCount = 201;for (let sequence = 0; sequence < turnCount; sequence += 1) {@@ -2268,26 +2511,20 @@ describe("createAgentOtelInstrumentation", () => {await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- const sessions = byName(spans, "agent.session");const turns = byName(spans, "invoke_agent weather");- expect(sessions).toHaveLength(1);+ expect(byName(spans, "agent.session")).toHaveLength(0);expect(turns).toHaveLength(turnCount);- expect(- turns.every((turn) => turn.spanContext().traceId === sessions[0]!.spanContext().traceId),- ).toBe(true);+ expect(new Set(turns.map((turn) => turn.spanContext().traceId))).toHaveLength(turnCount);+ for (const turn of turns) {+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.attributes).not.toHaveProperty("vercel.session_id");+ }});- it("records a subagent child into its parent's trace", async () => {+ it("records a subagent child in a separate trace linked to its caller", async () => {const runtime = createRuntime();- await publishTurnStarted({- hooks: runtime.hooks,- sessionId: "session-1",- turnId: "turn-1",- turnSequence: 0,- });- await runtime.provider.forceFlush();- const parentSession = byName(runtime.exporter.getFinishedSpans(), "agent.session")[0]!;+ const caller = spanContext("1", "2");await publishTurnStarted({hooks: runtime.hooks,@@ -2297,7 +2534,7 @@ describe("createAgentOtelInstrumentation", () => {subagentName: "researcher",turnId: "turn-1",},- parentTraceContext: parentSession.spanContext(),+ parentTraceContext: caller,rootSessionId: "session-1",sessionId: "child-1",turnId: "child-turn-1",@@ -2308,19 +2545,28 @@ describe("createAgentOtelInstrumentation", () => {const spans = runtime.exporter.getFinishedSpans();const childTurn = byName(spans, "invoke_agent weather").find(- (span) => span.attributes["agent.session.id"] === "child-1",+ (span) => span.attributes["agent.subagent.name"] === "researcher",)!;- expect(childTurn.spanContext().traceId).toBe(parentSession.spanContext().traceId);- expect(childTurn.parentSpanContext?.spanId).toBe(parentSession.spanContext().spanId);+ expect(childTurn.spanContext().traceId).not.toBe(caller.traceId);+ expect(childTurn.parentSpanContext).toBeUndefined();+ expect(childTurn.links).toEqual([+ {+ context: { ...caller, isRemote: false },+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);expect(childTurn.attributes["agent.subagent.name"]).toBe("researcher");- expect(childTurn.attributes).not.toHaveProperty("agent.parent.call_id");- expect(childTurn.attributes).not.toHaveProperty("agent.parent.session.id");- expect(childTurn.attributes).not.toHaveProperty("agent.parent.turn.id");- expect(byName(spans, "agent.session")).toHaveLength(1);+ expect(childTurn.attributes["gen_ai.conversation.id"]).toBe("session-1");+ expect(childTurn.attributes).not.toHaveProperty("agent.parent_call.id");+ expect(childTurn.attributes).not.toHaveProperty("agent.parent_run.id");+ expect(childTurn.attributes).not.toHaveProperty("agent.root_run.id");+ expect(childTurn.attributes).not.toHaveProperty("agent.session.id");+ expect(childTurn.attributes).not.toHaveProperty("vercel.session_id");+ expect(byName(spans, "agent.session")).toHaveLength(0);});- it("preserves a remote action as the parent of a remote child turn", async () => {+ it("links a remote action to a separately rooted remote child turn", async () => {const runtime = createRuntime();const parentTraceContext: InstrumentationTraceContext & { readonly isRemote: true } = {isRemote: true,@@ -2340,8 +2586,14 @@ describe("createAgentOtelInstrumentation", () => {await runtime.provider.forceFlush();const childTurn = byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;- expect(childTurn.spanContext().traceId).toBe(parentTraceContext.traceId);- expect(childTurn.parentSpanContext).toMatchObject(parentTraceContext);+ expect(childTurn.spanContext().traceId).not.toBe(parentTraceContext.traceId);+ expect(childTurn.parentSpanContext).toBeUndefined();+ expect(childTurn.links).toEqual([+ {+ context: parentTraceContext,+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);});it("opens its own root when a child session is handed no parent trace", async () => {@@ -2353,29 +2605,27 @@ describe("createAgentOtelInstrumentation", () => {turnId: "child-turn-1",turnSequence: 0,});+ await completeTurn(runtime.hooks, "child-1", "child-turn-1");await runtime.provider.forceFlush();- const sessions = byName(runtime.exporter.getFinishedSpans(), "agent.session");- expect(sessions).toHaveLength(1);- expect(sessions[0]!.attributes["agent.session.id"]).toBe("child-1");- expect(sessions[0]!.parentSpanContext).toBeUndefined();+ const turn = byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;+ expect(byName(runtime.exporter.getFinishedSpans(), "agent.session")).toHaveLength(0);+ expect(turn.attributes).toMatchObject({+ "gen_ai.conversation.id": "session-1",+ });+ expect(turn.attributes).not.toHaveProperty("agent.session.id");+ expect(turn.attributes).not.toHaveProperty("vercel.session_id");+ expect(turn.parentSpanContext).toBeUndefined();});- it("keeps a long subagent child in its adopted trace", async () => {+ it("links only the first subagent turn to the original caller", async () => {const runtime = createRuntime();- await publishTurnStarted({- hooks: runtime.hooks,- sessionId: "session-1",- turnId: "turn-1",- turnSequence: 0,- });- await runtime.provider.forceFlush();- const parentSession = byName(runtime.exporter.getFinishedSpans(), "agent.session")[0]!;+ const caller = spanContext("1", "2");for (let sequence = 0; sequence < 201; sequence += 1) {await publishTurnStarted({hooks: runtime.hooks,- parentTraceContext: parentSession.spanContext(),+ parentTraceContext: caller,rootSessionId: "session-1",sessionId: "child-1",turnId: `child-turn-${sequence}`,@@ -2386,13 +2636,29 @@ describe("createAgentOtelInstrumentation", () => {await runtime.provider.forceFlush();const spans = runtime.exporter.getFinishedSpans();- expect(byName(spans, "agent.session")).toHaveLength(1);- expect(byName(spans, "invoke_agent weather")).toHaveLength(201);- expect(- byName(spans, "invoke_agent weather").every(- (turn) => turn.spanContext().traceId === parentSession.spanContext().traceId,- ),- ).toBe(true);+ const turns = byName(spans, "invoke_agent weather");+ expect(byName(spans, "agent.session")).toHaveLength(0);+ expect(turns).toHaveLength(201);+ expect(turns[0]!.spanContext().traceId).not.toBe(caller.traceId);+ expect(turns[0]!.parentSpanContext).toBeUndefined();+ expect(turns[0]!.links).toEqual([+ {+ context: { ...caller, isRemote: false },+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);+ expect(new Set(turns.map((turn) => turn.spanContext().traceId))).toHaveLength(201);+ for (const turn of turns.slice(1)) {+ expect(turn.spanContext().traceId).not.toBe(caller.traceId);+ expect(turn.parentSpanContext).toBeUndefined();+ expect(turn.links).toEqual([]);+ }+ for (const turn of turns) {+ expect(turn.attributes).toMatchObject({+ "gen_ai.conversation.id": "session-1",+ });+ expect(turn.attributes).not.toHaveProperty("vercel.session_id");+ }});it("marks a failed action without failing its turn", async () => {diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.tsindex 9f16ed7c8..45bdf858d 100644--- a/packages/eve/src/tracing/agent-otel-provider.ts+++ b/packages/eve/src/tracing/agent-otel-provider.ts@@ -1,9 +1,9 @@import {ROOT_CONTEXT,SpanKind,- SpanStatusCode,context,type Context,+ type Link,type Span,type SpanContext,type Tracer,@@ -14,7 +14,7 @@ import { contextStorage } from "#context/container.js";import { SessionTraceSeedKey } from "#context/keys.js";import { withoutInstrumentationContent } from "#instrumentation/content.js";import { instrumentationEventForTraceDecision } from "#instrumentation/content-policy.js";-import type { AgentTraceStateStore } from "#tracing/agent-trace-state.js";+import type { AgentTraceStateStore, AgentTurnTraceState } from "#tracing/agent-trace-state.js";import {contentAttribute,genAiInputMessagesAttribute,@@ -31,13 +31,14 @@ import { createAgentApprovalInstrumentation } from "#tracing/agent-approval-instimport { createAgentChannelDeliveryInstrumentation } from "#tracing/agent-channel-delivery-instrumentation.js";import { createAgentToolInstrumentation } from "#tracing/agent-tool-instrumentation.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";-import { resolveConversationId } from "#tracing/conversation-context.js";import { markAgentTraceContext } from "#tracing/agent-trace-context.js";+import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";import * as runtimeAttributes from "#tracing/agent-otel-runtime-context.js";import {readGatewayCost,setAgentInvocationUsage,setAgentUsage,+ setGenAiUsage,} from "#tracing/agent-otel-usage.js";import { createAgentOtelSessionContext } from "#tracing/agent-otel-session-context.js";import type { TraceCapturePolicy } from "#tracing/otel-declaration.js";@@ -67,23 +68,24 @@ import type {InstrumentationTurnTerminalEvent,} from "#instrumentation/lifecycle.js";import { attemptIdempotencyKey } from "#instrumentation/lifecycle.js";+import {+ AGENT_SPAN_NAMES,+ agentInvocationSpanName,+ type AgentSamplingOperation,+} from "#tracing/agent-span-contract.js";+import { withErrorContent } from "#tracing/error-content-context.js";+import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";-interface SpanState {- readonly context: Context;- readonly span: Span;-}+type SpanState = { readonly context: Context; readonly span: Span };export interface AgentOtelInstrumentationInput {- /**- * Whether to write model prompts and tool call inputs onto spans at all.- * This is the union across destinations, not one destination's policy: a- * destination that declined drops these on its way out instead.- */+ /** Whether any destination records model and tool inputs. */readonly recordInputs?: boolean;- /** The same, for model responses and tool call outputs. */+ /** Whether any destination records model and tool outputs. */readonly recordOutputs?: boolean;readonly frameworkVersion: string;readonly idGenerator: AgentSpanIdGenerator;+ readonly samplesTrace?: (traceId: string, operation?: AgentSamplingOperation) => boolean;readonly stateStore: AgentTraceStateStore;readonly tracer: Tracer;readonly tracePolicy?: TraceCapturePolicy;@@ -109,9 +111,7 @@ export function createAgentOtelInstrumentation(const recordOutputs = input.recordOutputs ?? false;const executionContexts = new WeakMap<InstrumentationAttemptScope, Map<string, Context>>();const attemptScopes = new Map<string, InstrumentationAttemptScope>();- // A serverless turn runs inside one `turnStep` "use step" invocation. If- // that worker is lost, Workflow retries the whole step from entry rather- // than resuming this callback sequence in a replacement process.+ // A lost serverless worker retries the whole turn step from entry.const steps = new WeakMap<InstrumentationAttemptScope, SpanState>();const modelSpans = new WeakMap<InstrumentationAttemptScope, Map<string, SpanState>>();const actions = createAgentActionInstrumentation({@@ -220,10 +220,9 @@ export function createAgentOtelInstrumentation(input.idGenerator.deriveSpanId(attemptIdempotencyKey(event.scope)),() =>input.tracer.startSpan(- "agent.step",+ AGENT_SPAN_NAMES.step,{attributes: {- "agent.session.id": event.scope.sessionId,"agent.framework.name": "eve","agent.framework.version": input.frameworkVersion,"agent.step.attempt": event.scope.attemptIndex,@@ -231,6 +230,10 @@ export function createAgentOtelInstrumentation("agent.turn.id": event.scope.turnId,"agent.name": event.scope.functionId,...agentSpanNamingAttributes("agent.step"),+ ...agentTraceIdentityAttributes({+ rootSessionId: event.scope.rootSessionId ?? event.scope.sessionId,+ sessionId: event.scope.sessionId,+ }),...runtimeAttributes.runtimeContextAttributes(event.runtimeContext),},links:@@ -292,6 +295,7 @@ export function createAgentOtelInstrumentation(const onSessionTransition = async (event: InstrumentationSessionTransitionEvent,): Promise<void> => {+ await actions.flushForSessionTransition(event);if (event.type === "session.failed" && event.turnId !== undefined) {await input.stateStore.updateTurn(event.sessionId, event.turnId, (turn) => ({...turn,@@ -304,37 +308,27 @@ export function createAgentOtelInstrumentation(const session = await input.stateStore.getSession(event.sessionId);if (isSampledTrace(turn.context)) {const agentName = session?.agentName ?? turn.subagentName;- const span = input.idGenerator.withSpanId(turn.context.spanId, () =>+ const parentContext = withChannelAudience(ROOT_CONTEXT, session?.channelAudience);+ const startSpan = () =>input.tracer.startSpan(- agentSpanName(agentName),+ agentInvocationSpanName(agentName),{- attributes: {- "agent.framework.name": "eve",- "agent.framework.version": input.frameworkVersion,- "agent.name": agentName,- ...runtimeAttributes.agentLineageAttributes(turn),- "agent.session.id": event.sessionId,- "agent.subagent.name": turn.subagentName,- "agent.turn.id": event.turnId,- "agent.turn.sequence": turn.sequence,- "gen_ai.agent.name": agentName,- "gen_ai.conversation.id": resolveConversationId(event.sessionId),- "gen_ai.operation.name": "invoke_agent",- ...agentSpanNamingAttributes(agentSpanName(agentName), "invoke_agent"),- },+ attributes: runtimeAttributes.agentActivationAttributes({+ agentName,+ frameworkVersion: input.frameworkVersion,+ sessionId: event.sessionId,+ turnId: event.turnId!,+ turn,+ }),kind: SpanKind.INTERNAL,+ links: agentActivationLinks(turn),+ root: true,startTime: turn.startTimeMs,},- withChannelAudience(- contextFromSpanContext({- isRemote: turn.parentIsRemote ?? false,- spanId: turn.parentSpanId,- traceFlags: turn.context.traceFlags,- traceId: turn.context.traceId,- }),- session?.channelAudience,- ),- ),+ parentContext,+ );+ const span = input.idGenerator.withSpanId(turn.context.spanId, () =>+ input.idGenerator.withTraceId(turn.context.traceId, startSpan),);setAgentInvocationUsage(span, turn.modelUsage);span.addEvent("turn.started", undefined, turn.startTimeMs);@@ -380,8 +374,13 @@ export function createAgentOtelInstrumentation("gen_ai.provider.name": event.model.provider,"gen_ai.request.model": event.model.modelId,...agentSpanNamingAttributes(modelSpanName(event.model.modelId), "chat"),+ ...agentTraceIdentityAttributes({+ rootSessionId: event.scope.rootSessionId ?? event.scope.sessionId,+ sessionId: event.scope.sessionId,+ }),...runtimeAttributes.runtimeContextAttributes(event.runtimeContext),},+ kind: SpanKind.CLIENT,},attempt.context,);@@ -412,7 +411,10 @@ export function createAgentOtelInstrumentation(recordError(state.span, event.error);} else {await recordTurnUsage(event);- setAgentUsage(state.span, event.usage);+ setGenAiUsage(state.span, event.usage);+ if (event.responseId !== undefined) {+ state.span.setAttribute("gen_ai.response.id", event.responseId);+ }state.span.setAttribute("gen_ai.response.finish_reasons", [event.finishReason]);const attempt = steps.get(event.scope);if (attempt !== undefined) setAgentUsage(attempt.span, event.usage);@@ -505,11 +507,8 @@ export function createAgentOtelInstrumentation(const channelDeliveries = createAgentChannelDeliveryInstrumentation({ensureSessionContext,- frameworkVersion: input.frameworkVersion,- idGenerator: input.idGenerator,recordInputs,stateStore: input.stateStore,- tracer: input.tracer,});const onStepMetadata = (event: InstrumentationStepAttemptMetadataEvent): void => {@@ -527,6 +526,7 @@ export function createAgentOtelInstrumentation(return {hook: {+ flush: actions.flushSettledInvocations,events: {...channelDeliveries,"action.completed": actions.events["action.completed"],@@ -579,9 +579,28 @@ export function createAgentOtelInstrumentation(if (!isSampledTrace(turn.context)) parent = suppressTracing(parent);}}+ const session = await input.stateStore.getSession(operation.scope.sessionId);+ const seed = resolveForwardedTraceSeed(contextStorage.getStore()?.get(SessionTraceSeedKey));+ const decision = seed?.decision ?? session?.decision;+ const effective =+ decision === undefined+ ? undefined+ : applyLiveDeliveryAudienceCeiling(+ decision,+ normalizeChannelAudience(operation.scope.channelAudience),+ seed?.forwardedTracePolicy,+ );return parent === undefined? execute()- : context.with(markAgentTraceContext(parent), execute);+ : context.with(+ markAgentTraceContext(+ withErrorContent(+ parent,+ recordOutputs && effective?.action === "record" && effective.recordOutputs,+ ),+ ),+ execute,+ );},};@@ -603,6 +622,23 @@ export function createAgentOtelInstrumentation(}}+function agentActivationLinks(turn: AgentTurnTraceState): Link[] | undefined {+ const links: Link[] = [];+ if (turn.caller !== undefined) {+ links.push({+ context: turn.caller,+ attributes: { "eve.link.type": "agent.dispatch" },+ });+ }+ if (turn.channelDelivery?.requestTraceContext !== undefined) {+ links.push({+ context: turn.channelDelivery.requestTraceContext,+ attributes: { "eve.link.type": "channel.request" },+ });+ }+ return links.length === 0 ? undefined : links;+}+function sessionIdForEvent(event: InstrumentationEvent): string {return "scope" in event ? event.scope.sessionId : event.sessionId;}@@ -654,18 +690,6 @@ function modelSpanName(modelId: string): string {return `chat ${modelId}`;}-function agentSpanName(agentName: string | undefined): string {- return agentName === undefined ? "invoke_agent" : `invoke_agent ${agentName}`;-}-function errorText(error: unknown): unknown {return error instanceof Error ? error.message : error;}--function recordError(span: Span, error: unknown): void {- span.setAttribute("error.type", error instanceof Error ? error.name || "Error" : "_OTHER");- if (error instanceof Error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- } else span.setStatus({ code: SpanStatusCode.ERROR });-}diff --git a/packages/eve/src/tracing/agent-otel-runtime-context.ts b/packages/eve/src/tracing/agent-otel-runtime-context.tsindex 20a13d23d..f328bb6ff 100644--- a/packages/eve/src/tracing/agent-otel-runtime-context.ts+++ b/packages/eve/src/tracing/agent-otel-runtime-context.ts@@ -1,19 +1,38 @@import type { AgentTurnTraceState } from "#tracing/agent-trace-state.js";+import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";+import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";+import { agentInvocationSpanName } from "#tracing/agent-span-contract.js";type SpanAttributePrimitive = string | number | boolean;type SpanAttributeValue = SpanAttributePrimitive | SpanAttributePrimitive[];-export function agentLineageAttributes(- turn: AgentTurnTraceState,-): Record<string, SpanAttributeValue> {- const attributes: Record<string, SpanAttributeValue> = {- "agent.root_run.id": turn.rootSessionId,+export function agentActivationAttributes(input: {+ readonly agentName?: string;+ readonly frameworkVersion: string;+ readonly sessionId: string;+ readonly turnId: string;+ readonly turn: AgentTurnTraceState;+}): Record<string, string | number | boolean | undefined> {+ return {+ "agent.framework.name": "eve",+ "agent.framework.version": input.frameworkVersion,+ "agent.name": input.agentName,+ "agent.channel.delivery.id": input.turn.channelDelivery?.deliveryId,+ "agent.channel.delivery.input": input.turn.channelDelivery?.inputAttribute,+ "agent.channel.kind": input.turn.channelDelivery?.channelKind,+ "agent.channel.name": input.turn.channelDelivery?.channelName,+ "agent.channel.request.id": input.turn.channelDelivery?.requestId,+ "agent.subagent.name": input.turn.subagentName,+ "agent.turn.id": input.turnId,+ "agent.turn.sequence": input.turn.sequence,+ "gen_ai.agent.name": input.agentName,+ "gen_ai.operation.name": "invoke_agent",+ ...agentSpanNamingAttributes(agentInvocationSpanName(input.agentName), "invoke_agent"),+ ...agentTraceIdentityAttributes({+ rootSessionId: input.turn.rootSessionId,+ sessionId: input.sessionId,+ }),};- if (turn.parentLineage !== undefined) {- attributes["agent.parent_run.id"] = turn.parentLineage.sessionId;- attributes["agent.parent_call.id"] = turn.parentLineage.callId;- }- return attributes;}/** Flattens merged runtime context into AI SDK-compatible span attributes. */diff --git a/packages/eve/src/tracing/agent-otel-session-context.ts b/packages/eve/src/tracing/agent-otel-session-context.tsindex 19e0b3b24..c43661ff2 100644--- a/packages/eve/src/tracing/agent-otel-session-context.ts+++ b/packages/eve/src/tracing/agent-otel-session-context.ts@@ -1,4 +1,4 @@-import type { SpanContext, Tracer } from "#compiled/@opentelemetry/api/index.js";+import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";import {type InstrumentationSessionStartedEvent,@@ -17,13 +17,18 @@ import {} from "#tracing/sampled-trace.js";import type { AgentSessionTraceState, AgentTraceStateStore } from "#tracing/agent-trace-state.js";import { readInstrumentationDecision } from "#shared/instrumentation-decision.js";-import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";+import {+ agentInvocationSpanName,+ type AgentSamplingOperation,+} from "#tracing/agent-span-contract.js";+import { agentActivationAttributes } from "#tracing/agent-otel-runtime-context.js";+import type { AgentTurnTraceState } from "#tracing/agent-trace-state.js";interface AgentOtelSessionContextInput {readonly frameworkVersion: string;readonly idGenerator: AgentSpanIdGenerator;+ readonly samplesTrace?: (traceId: string, operation?: AgentSamplingOperation) => boolean;readonly stateStore: AgentTraceStateStore;- readonly tracer: Tracer;readonly tracePolicy?: TraceCapturePolicy;}@@ -42,66 +47,6 @@ interface AgentOtelSessionContext {export function createAgentOtelSessionContext(input: AgentOtelSessionContextInput,): AgentOtelSessionContext {- const openSessionTrace = (- session: SessionMetadata,- channelAudience: ChannelAudience,- traceDecision: ReturnType<typeof resolveTracePolicy>,- ): SpanContext => {- if (session.traceSeed !== undefined) {- if (traceDecision.action === "drop" || !isSampledTrace(session.traceSeed)) {- return {- isRemote: false,- spanId: session.traceSeed.spanId,- traceFlags: 0,- traceId: session.traceSeed.traceId,- };- }- const startSpan = () =>- input.tracer.startSpan("agent.session", {- attributes: {- "agent.framework.name": "eve",- "agent.framework.version": input.frameworkVersion,- "agent.channel.audience": channelAudience,- "agent.name": session.agentName,- "agent.session.id": session.sessionId,- ...agentSpanNamingAttributes("agent.session"),- "agent.trace.schema.version": 3,- },- root: true,- });- const span = input.idGenerator.withSpanId(session.traceSeed.spanId, () =>- input.idGenerator.withTraceId(session.traceSeed!.traceId, startSpan),- );- span.addEvent("session.started");- span.end();- return span.spanContext();- }- // Fallback for already-running workflows that predate the seed.- if (traceDecision.action === "drop") {- return {- isRemote: false,- spanId: input.idGenerator.deriveSpanId(`session:${session.sessionId}`),- traceFlags: 0,- traceId: input.idGenerator.generateTraceId(),- };- }- const span = input.tracer.startSpan("agent.session", {- attributes: {- "agent.framework.name": "eve",- "agent.framework.version": input.frameworkVersion,- "agent.channel.audience": channelAudience,- "agent.name": session.agentName,- "agent.session.id": session.sessionId,- ...agentSpanNamingAttributes("agent.session"),- "agent.trace.schema.version": 3,- },- root: true,- });- span.addEvent("session.started");- span.end();- return span.spanContext();- };-const ensureSessionContext = async (event: SessionMetadata): Promise<AgentSessionTraceState> => {let state = await input.stateStore.getSession(event.sessionId);if (state === undefined) {@@ -112,10 +57,7 @@ export function createAgentOtelSessionContext(channelAudience,channelKind: event.channelKind,decision,- context:- event.parentTraceContext === undefined- ? openSessionTrace(event, channelAudience, decision)- : adoptedSpanContext(event.parentTraceContext),+ context: initialSessionContext(input, event, decision),rootSessionId: event.rootSessionId,parentLineage: event.parentLineage,};@@ -137,35 +79,44 @@ export function createAgentOtelSessionContext(const prepared = await input.stateStore.getTurn(event.sessionId, event.turnId);if (prepared !== undefined) {const session = await input.stateStore.getSession(event.sessionId);- return {- decision: session?.decision,- spanId: prepared.parentSpanId,- traceFlags: prepared.context.traceFlags,- traceId: prepared.context.traceId,- };+ return portableSpanContext(prepared.context, session?.decision);}const session = await ensureSessionContext(event);- // The turn outlives this worker, so no live span can cover it: the span- // id is allocated now for descendants to parent through, and the span- // itself is emitted at the turn's session transition.- const turnContext: SpanContext = {- isRemote: false,- spanId: input.idGenerator.deriveSpanId(`turn:${event.idempotencyKey}`),- traceFlags: session.context.traceFlags,- traceId: session.context.traceId,- };- await input.stateStore.setTurn(event.sessionId, event.turnId, {+ const useInitialContext = event.sequence === 0;+ const caller = useInitialContext ? event.parentTraceContext : undefined;+ let turnContext = useInitialContext+ ? { ...session.context, isRemote: false }+ : freshTurnContext(input, event.idempotencyKey, session.decision);+ const turn: AgentTurnTraceState = {+ caller: caller === undefined ? undefined : adoptedSpanContext(caller),context: turnContext,- parentIsRemote: session.context.isRemote,parentLineage: event.parentLineage ?? session.parentLineage,- parentSpanId: session.context.spanId,rootSessionId: event.rootSessionId,sequence: event.sequence,startTimeMs: Date.now(),subagentName: (event.parentLineage ?? session.parentLineage)?.subagentName,+ };+ if (isSampledTrace(turn.context)) {+ const agentName = session.agentName ?? turn.subagentName;+ const sampled =+ input.samplesTrace?.(turn.context.traceId, {+ name: agentInvocationSpanName(agentName),+ attributes: agentActivationAttributes({+ agentName,+ frameworkVersion: input.frameworkVersion,+ sessionId: event.sessionId,+ turnId: event.turnId,+ turn,+ }),+ }) ?? true;+ turnContext = { ...turnContext, traceFlags: sampled ? 1 : 0 };+ }+ await input.stateStore.setTurn(event.sessionId, event.turnId, {+ ...turn,+ context: turnContext,});- return portableSpanContext(session.context, session.decision);+ return portableSpanContext(turnContext, session.decision);};return { ensureSessionContext, prepareSessionTrace, prepareTurnTrace };@@ -192,11 +143,51 @@ function adoptedSpanContext(handed: InstrumentationTraceContext): SpanContext {};}+function initialSessionContext(+ input: AgentOtelSessionContextInput,+ event: SessionMetadata,+ decision: ReturnType<typeof resolveTracePolicy>,+): SpanContext {+ const handed = event.traceSeed;+ if (handed !== undefined) {+ return {+ ...adoptedSpanContext(handed),+ traceFlags: decision.action === "drop" ? 0 : handed.traceFlags,+ };+ }+ const traceId = input.idGenerator.deriveTraceId(`session:${event.sessionId}`);+ const sampled = decision.action === "record";+ return {+ isRemote: false,+ spanId: input.idGenerator.deriveSpanId(`session:${event.sessionId}`),+ traceFlags: sampled ? 1 : 0,+ traceId,+ };+}++function freshTurnContext(+ input: AgentOtelSessionContextInput,+ idempotencyKey: string,+ decision: AgentSessionTraceState["decision"],+): SpanContext {+ const traceId = input.idGenerator.deriveTraceId(`turn:${idempotencyKey}`);+ const sampled = decision?.action === "record";+ return {+ isRemote: false,+ spanId: input.idGenerator.deriveSpanId(`turn:${idempotencyKey}`),+ traceFlags: sampled ? 1 : 0,+ traceId,+ };+}+function resolveSessionTraceDecision(event: SessionMetadata,audience: ChannelAudience,policy: TraceCapturePolicy | undefined,): ReturnType<typeof resolveTracePolicy> {+ if (event.parentTraceContext !== undefined && !isSampledTrace(event.parentTraceContext)) {+ return { action: "drop" };+ }if (event.traceSeed?.decision !== undefined) {return readInstrumentationDecision(event.traceSeed.decision) ?? { action: "drop" };}diff --git a/packages/eve/src/tracing/agent-otel-usage.ts b/packages/eve/src/tracing/agent-otel-usage.tsindex 0369c0902..102f50f56 100644--- a/packages/eve/src/tracing/agent-otel-usage.ts+++ b/packages/eve/src/tracing/agent-otel-usage.ts@@ -2,33 +2,40 @@ import type { Span } from "#compiled/@opentelemetry/api/index.js";import type { InstrumentationUsage } from "#instrumentation/lifecycle.js";import type { AgentTurnTraceState } from "#tracing/agent-trace-state.js";+import { AGENT_USAGE_ATTRIBUTES } from "#tracing/agent-span-contract.js";/** Applies eve's structural token usage attributes to an agent span. */-export function setAgentUsage(- span: Span,- usage: InstrumentationUsage,- options: { readonly includeGenAiDetails?: boolean } = {},-): void {+export function setAgentUsage(span: Span, usage: InstrumentationUsage): void {if (usage.inputTokens !== undefined) {- span.setAttribute("agent.usage.input_tokens", usage.inputTokens);+ span.setAttribute(AGENT_USAGE_ATTRIBUTES.inputTokens, usage.inputTokens);}if (usage.outputTokens !== undefined) {- span.setAttribute("agent.usage.output_tokens", usage.outputTokens);+ span.setAttribute(AGENT_USAGE_ATTRIBUTES.outputTokens, usage.outputTokens);}const details = usage.inputTokenDetails;if (details?.cacheReadTokens !== undefined) {- if (options.includeGenAiDetails === false) {- span.setAttribute("agent.usage.cache_read_tokens", details.cacheReadTokens);- } else {- span.setAttribute("gen_ai.usage.cache_read.input_tokens", details.cacheReadTokens);- }+ span.setAttribute(AGENT_USAGE_ATTRIBUTES.cacheReadTokens, details.cacheReadTokens);}if (details?.cacheWriteTokens !== undefined) {- if (options.includeGenAiDetails === false) {- span.setAttribute("agent.usage.cache_write_tokens", details.cacheWriteTokens);- } else {- span.setAttribute("gen_ai.usage.cache_creation.input_tokens", details.cacheWriteTokens);- }+ span.setAttribute(AGENT_USAGE_ATTRIBUTES.cacheWriteTokens, details.cacheWriteTokens);+ }+}++/** Applies standard GenAI token usage while retaining eve's compatibility attributes. */+export function setGenAiUsage(span: Span, usage: InstrumentationUsage): void {+ setAgentUsage(span, usage);+ if (usage.inputTokens !== undefined) {+ span.setAttribute("gen_ai.usage.input_tokens", usage.inputTokens);+ }+ if (usage.outputTokens !== undefined) {+ span.setAttribute("gen_ai.usage.output_tokens", usage.outputTokens);+ }+ const details = usage.inputTokenDetails;+ if (details?.cacheReadTokens !== undefined) {+ span.setAttribute("gen_ai.usage.cache_read.input_tokens", details.cacheReadTokens);+ }+ if (details?.cacheWriteTokens !== undefined) {+ span.setAttribute("gen_ai.usage.cache_creation.input_tokens", details.cacheWriteTokens);}}@@ -37,21 +44,10 @@ export function setAgentInvocationUsage(modelUsage: AgentTurnTraceState["modelUsage"],): void {if (modelUsage === undefined) return;- if (modelUsage.inputTokens !== undefined) {- span.setAttribute("gen_ai.usage.input_tokens", modelUsage.inputTokens);- }- if (modelUsage.outputTokens !== undefined) {- span.setAttribute("gen_ai.usage.output_tokens", modelUsage.outputTokens);- }+ setGenAiUsage(span, modelUsage);}-/**- * Extracts cost data from a step result's provider metadata. Only Vercel AI- * Gateway reports it (`providerMetadata.gateway`): raw inference cost, the- * gateway's surcharged total, the input/output split, and the generation id- * for dashboard reconciliation. Values arrive as USD strings; anything- * missing or non-numeric is skipped, so non-gateway providers get nothing.- */+/** Projects Vercel AI Gateway cost metadata onto GenAI span attributes. */export function readGatewayCost(providerMetadata: Readonly<Record<string, unknown>>,): Record<string, string | number> | undefined {diff --git a/packages/eve/src/tracing/agent-span-contract.ts b/packages/eve/src/tracing/agent-span-contract.tsnew file mode 100644index 000000000..11a78cdef--- /dev/null+++ b/packages/eve/src/tracing/agent-span-contract.ts@@ -0,0 +1,46 @@+export const AGENT_TRACE_SCHEMA_VERSION = 4;++export const AGENT_SPAN_NAMES = {+ action: "agent.action",+ approval: "agent.approval",+ channelRequest: "agent.channel.request",+ step: "agent.step",+} as const;++export const AGENT_USAGE_ATTRIBUTES = {+ inputTokens: "agent.usage.input_tokens",+ outputTokens: "agent.usage.output_tokens",+ cacheReadTokens: "agent.usage.cache_read_tokens",+ cacheWriteTokens: "agent.usage.cache_write_tokens",+} as const;++export function agentInvocationSpanName(agentName: string | undefined): string {+ return agentName === undefined ? "invoke_agent" : `invoke_agent ${agentName}`;+}++export interface AgentSamplingOperation {+ readonly name: string;+ readonly attributes?: Readonly<Record<string, string | number | boolean | undefined>>;+}++interface AgentSpanRecord {+ readonly name: string;+ readonly attributes: Readonly<Record<string, unknown>>;+}++export function isAgentActivationSpan(span: AgentSpanRecord): boolean {+ return (+ span.name === "agent.turn" ||+ (span.attributes["gen_ai.operation.name"] === "invoke_agent" &&+ span.attributes["agent.invocation.role"] !== "caller" &&+ typeof span.attributes["agent.turn.id"] === "string")+ );+}++export function agentTurnIdentity(span: AgentSpanRecord): string | undefined {+ const conversationId = span.attributes["gen_ai.conversation.id"];+ const turnId = span.attributes["agent.turn.id"];+ return typeof turnId === "string"+ ? `${typeof conversationId === "string" ? conversationId : ""}\0${turnId}`+ : undefined;+}diff --git a/packages/eve/src/tracing/agent-span-error.ts b/packages/eve/src/tracing/agent-span-error.tsnew file mode 100644index 000000000..e2b4dcf8a--- /dev/null+++ b/packages/eve/src/tracing/agent-span-error.ts@@ -0,0 +1,21 @@+import { SpanStatusCode, type Span } from "#compiled/@opentelemetry/api/index.js";+import { boundedTraceError } from "#tracing/bounded-error.js";+import { truncateTelemetryText } from "#tracing/telemetry-budget.js";++/** Call with policy-projected errors; undefined retains failure status without content. */+export function recordAgentSpanError(span: Span, error: unknown, errorType?: string): void {+ span.setAttribute(+ "error.type",+ truncateTelemetryText(+ errorType ?? (error instanceof Error ? error.name || "Error" : "_OTHER"),+ 128,+ ),+ );+ if (error instanceof Error) {+ const bounded = boundedTraceError(error);+ span.recordException(bounded);+ span.setStatus({ code: SpanStatusCode.ERROR, message: bounded.message });+ } else {+ span.setStatus({ code: SpanStatusCode.ERROR });+ }+}diff --git a/packages/eve/src/tracing/agent-span-id-generator.ts b/packages/eve/src/tracing/agent-span-id-generator.tsindex bcb1aae9e..0339c9b25 100644--- a/packages/eve/src/tracing/agent-span-id-generator.ts+++ b/packages/eve/src/tracing/agent-span-id-generator.ts@@ -1,5 +1,7 @@import { createHash } from "node:crypto";+import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";+/*** Id generator shared by `registerOTel` and the agent OTel provider. A span* whose lifetime crosses durable worker boundaries (`invoke_agent`) is emitted@@ -17,8 +19,12 @@ export class AgentSpanIdGenerator {/** Derives one span id for a replay-stable instrumentation event. */deriveSpanId(key: string): string {- const spanId = createHash("sha256").update(key).digest("hex").slice(0, 16);- return /^0+$/u.test(spanId) ? "0000000000000001" : spanId;+ return deriveAgentSpanId(key);+ }++ deriveTraceId(key: string): string {+ const traceId = createHash("sha256").update(key).digest("hex").slice(0, 32);+ return /^0+$/u.test(traceId) ? "00000000000000000000000000000001" : traceId;}generateSpanId(): string {@@ -59,6 +65,15 @@ export class AgentSpanIdGenerator {}}+export function deriveAgentSpanId(key: string): string {+ const spanId = createHash("sha256").update(key).digest("hex").slice(0, 16);+ return /^0+$/u.test(spanId) ? "0000000000000001" : spanId;+}++export function deriveAgentActionSpanId(sessionId: string, turnId: string, callId: string): string {+ return deriveAgentSpanId(`action:${actionIdempotencyKey(sessionId, turnId, callId)}`);+}+const HEX_DIGITS = "0123456789abcdef";function randomHexId(length: number): string {diff --git a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.tsnew file mode 100644index 000000000..6c7f352cf--- /dev/null+++ b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts@@ -0,0 +1,574 @@+import {+ BasicTracerProvider,+ InMemorySpanExporter,+ SimpleSpanProcessor,+} from "@opentelemetry/sdk-trace-base";+import { describe, expect, it, vi } from "vitest";+import { SpanStatusCode } from "#compiled/@opentelemetry/api/index.js";++import { JsonTraceSerializer } from "#compiled/@opentelemetry/otlp-transformer/index.js";+import { ContextContainer, contextStorage } from "#context/container.js";+import {+ ChannelInstrumentationKey,+ ConversationIdKey,+ ParentSessionKey,+ ParentTraceContextKey,+ SessionTraceSeedKey,+} from "#context/keys.js";+import { deserializeContext, serializeContext } from "#context/serialize.js";+import {+ actionIdempotencyKey,+ attemptIdempotencyKey,+ createInstrumentationHooks,+ inputIdempotencyKey,+ modelCallIdempotencyKey,+ sessionIdempotencyKey,+ toolCallIdempotencyKey,+ turnIdempotencyKey,+ type InstrumentationAttemptScope,+} from "#instrumentation/lifecycle.js";+import { bindInstrumentationRuntime } from "#instrumentation/runtime.js";+import { createAgentOtelInstrumentation } from "#tracing/agent-otel-provider.js";+import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";+import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";+import { AGENT_TRACE_CONTEXT_KEY } from "#tracing/agent-trace-context-codec.js";+import { prepareAgentInvocationTrace } from "#tracing/agent-invocation-coordinator.js";+import {+ flushAgentInvocationTraces,+ settleAgentInvocationTrace,+} from "#tracing/agent-invocation-terminal.js";+import * as instrumentation from "#instrumentation/runtime.js";+import {+ assembleLocalTrace,+ isAgentTurnSpan,+ parseLocalTraceSegment,+} from "#tracing/local-trace-reader.js";+import { summarizeLocalTrace } from "#cli/commands/trace-detail.js";+import { buildConversationItems } from "#cli/dev/tui/traces/trace-conversation.js";+import { contentFilteringProcessor } from "#tracing/content-span-processor.js";+import {+ composeSpanExportPolicies,+ redactSpanInputs,+ redactSpanOutputs,+} from "#tracing/span-export-policy.js";++function createRuntime() {+ const exporter = new InMemorySpanExporter();+ const metadata = new InMemorySpanExporter();+ const idGenerator = new AgentSpanIdGenerator();+ const provider = new BasicTracerProvider({+ idGenerator,+ spanProcessors: [+ new SimpleSpanProcessor(exporter),+ contentFilteringProcessor(+ new SimpleSpanProcessor(metadata),+ composeSpanExportPolicies(redactSpanInputs(), redactSpanOutputs()),+ ),+ ],+ });+ const agent = createAgentOtelInstrumentation({+ frameworkVersion: "test",+ idGenerator,+ recordInputs: true,+ recordOutputs: true,+ stateStore: new ContextAgentTraceStateStore(),+ tracer: provider.getTracer("eve.agent"),+ });+ const hooks = createInstrumentationHooks([agent.hook]);+ return {+ ...agent,+ hooks,+ exporter,+ metadata,+ provider,+ idGenerator,+ ownsAgentSpans: true,+ otelSettings: { recordInputs: true, recordOutputs: true, traceChannelRequests: false },+ flushSettledInvocations: async () => {+ await agent.hook.flush?.();+ },+ forceFlush: async () => {+ await agent.hook.flush?.();+ await provider.forceFlush();+ },+ shutdown: () => provider.shutdown(),+ };+}++function contextFor(audience: "public" | "private") {+ const ctx = new ContextContainer();+ ctx.set(ChannelInstrumentationKey, { kind: "http", metadata: { audience } });+ return ctx;+}++function scopeFor(sessionId: string, audience: "public" | "private"): InstrumentationAttemptScope {+ return {+ attemptId: `${sessionId}:turn_0:0:0`,+ attemptIndex: 0,+ channelAudience: audience,+ functionId: sessionId,+ sessionId,+ rootSessionId: "parent",+ stepIndex: 0,+ turnId: "turn_0",+ };+}++describe("exported agent telemetry contract", () => {+ it("flushes 1000 settled invocations without closing the session or retaining their errors", async () => {+ const runtime = createRuntime();+ const exporterFlush = vi.spyOn(runtime.provider, "forceFlush").mockImplementation(() => {+ throw new Error("Exporter unavailable");+ });+ const registered = vi+ .spyOn(instrumentation, "getInstrumentationRuntime")+ .mockReturnValue(runtime);+ let ctx = contextFor("public");+ const store = new ContextAgentTraceStateStore();+ const count = 1000;+ try {+ for (let index = 0; index < count; index++) {+ const callId = `nested-${index}`;+ contextStorage.run(ctx, () =>+ store.setInvocation(actionIdempotencyKey("parent", "turn_0", callId), {+ attemptIndex: 0,+ callId,+ channelAudience: "public",+ kind: "subagent-call",+ name: "child",+ parent: { spanId: "1".repeat(16), traceFlags: 1, traceId: "2".repeat(32) },+ parentActionCallId: "outer",+ recordOutputs: true,+ rootSessionId: "parent",+ sessionId: "parent",+ spanId: runtime.idGenerator.deriveSpanId(callId),+ startTimeMs: 1,+ stepIndex: 0,+ turnId: "turn_0",+ terminal: {+ acceptedAtMs: 2,+ error: new Error("private " + "x".repeat(16384)),+ outcome: index % 2 === 0 ? "failed" : "cancelled",+ },+ }),+ );+ const flushed = await flushAgentInvocationTraces(serializeContext(ctx));+ expect(JSON.stringify(flushed)).not.toContain("private ");+ ctx = await deserializeContext(flushed);+ contextStorage.run(ctx, () => expect(store.findInvocations("parent")).toEqual([]));+ }+ expect(runtime.exporter.getFinishedSpans()).toHaveLength(count);+ for (const span of runtime.exporter.getFinishedSpans()) {+ expect(span.status.code).toBe(+ span.attributes["agent.action.outcome"] === "failed"+ ? SpanStatusCode.ERROR+ : SpanStatusCode.UNSET,+ );+ }+ await flushAgentInvocationTraces(serializeContext(ctx));+ expect(runtime.exporter.getFinishedSpans()).toHaveLength(count);+ expect(exporterFlush).not.toHaveBeenCalled();+ } finally {+ exporterFlush.mockRestore();+ registered.mockRestore();+ await runtime.shutdown();+ }+ });++ it.each(["throw", "reject"] as const)(+ "preserves the settlement context when invocation materialization fails (%s)",+ async (failure) => {+ const runtime = createRuntime();+ const flush = vi.spyOn(runtime, "flushSettledInvocations").mockImplementation(() => {+ const error = new Error("Span processor unavailable");+ if (failure === "throw") throw error;+ return Promise.reject(error);+ });+ const registered = vi+ .spyOn(instrumentation, "getInstrumentationRuntime")+ .mockReturnValue(runtime);+ const serializedContext = { [AGENT_TRACE_CONTEXT_KEY]: {} };+ try {+ await expect(flushAgentInvocationTraces(serializedContext)).resolves.toBe(+ serializedContext,+ );+ expect(flush).toHaveBeenCalledOnce();+ } finally {+ registered.mockRestore();+ await runtime.shutdown();+ }+ },+ );++ it.each(["session.completed", "session.failed"] as const)(+ "only marks unfinished invocations as errors on %s",+ async (type) => {+ const runtime = createRuntime();+ const ctx = contextFor("public");+ const hooks = runtime.hooks.forTrace!({ agentName: "parent", audience: "public" });+ await contextStorage.run(ctx, async () => {+ const store = new ContextAgentTraceStateStore();+ store.setInvocation(actionIdempotencyKey("parent", "turn_0", "nested"), {+ attemptIndex: 0,+ callId: "nested",+ kind: "subagent-call",+ name: "child",+ parent: { spanId: "1".repeat(16), traceFlags: 1, traceId: "2".repeat(32) },+ parentActionCallId: "outer",+ rootSessionId: "parent",+ sessionId: "parent",+ spanId: "3".repeat(16),+ startTimeMs: 1,+ stepIndex: 0,+ turnId: "turn_0",+ });+ await hooks.publish({+ error: new Error("private failure"),+ idempotencyKey: sessionIdempotencyKey("parent"),+ sessionId: "parent",+ type,+ });+ expect(store.findInvocations("parent")).toEqual([]);+ });+ const [span] = runtime.exporter.getFinishedSpans();+ expect(span?.attributes["agent.action.outcome"]).toBe(+ type === "session.failed" ? "failed" : "abandoned",+ );+ expect(span?.status.code).toBe(+ type === "session.failed" ? SpanStatusCode.ERROR : SpanStatusCode.UNSET,+ );+ await runtime.shutdown();+ },+ );++ it.each(["public", "private"] as const)(+ "round-trips %s channel, approval, delegated error and usage spans through the readers",+ async (audience) => {+ const runtime = createRuntime();+ let parent = contextFor(audience);+ parent.set(ConversationIdKey, "original-conversation");+ const scope = scopeFor("parent", audience);+ const hooks = runtime.hooks.forTrace!({ agentName: "parent", audience });+ const actionKey = actionIdempotencyKey("parent", "turn_0", "workflow");+ const operation = { modelId: "test", operationId: "ai.streamText", provider: "test" };+ const binding = bindInstrumentationRuntime(runtime, parent, {+ agentName: "parent",+ rootSessionId: "parent",+ sessionId: "parent",+ })!;+ let dispatch: ReturnType<typeof prepareAgentInvocationTrace>;+ await contextStorage.run(parent, async () => {+ await binding.instrumentChannelDelivery({+ ctx: parent,+ agentName: "parent",+ rootSessionId: "parent",+ sequence: 0,+ sessionId: "parent",+ turnId: "turn_0",+ delivery: {+ kind: "deliver",+ payloads: [{ message: "private input" }],+ deliveryMetadata: [+ { channelKind: "http", channelName: "web", deliveryId: "delivery", payloadIndex: 0 },+ ],+ },+ });+ await binding.preparePreamble({ sequence: 0, sessionStarted: false, turnId: "turn_0" });+ await hooks.publish({+ idempotencyKey: attemptIdempotencyKey(scope),+ operation,+ scope,+ type: "step.attempt.started",+ });+ await hooks.publish({+ callId: "workflow",+ idempotencyKey: actionKey,+ input: { secret: "private input" },+ isWorkflowTool: true,+ kind: "tool-call",+ name: "coordinate",+ scope,+ type: "action.started",+ });+ await hooks.publish({+ callId: "workflow",+ idempotencyKey: toolCallIdempotencyKey(scope, "workflow", 0),+ input: { secret: "private input" },+ toolName: "coordinate",+ scope,+ type: "tool.call.started",+ });+ const approvalKey = inputIdempotencyKey("parent", "turn_0", "approval");+ await hooks.publish({+ action: { callId: "workflow", name: "coordinate" },+ idempotencyKey: approvalKey,+ kind: "tool-approval",+ requestId: "approval",+ scope,+ type: "input.requested",+ request: { prompt: "private approval" },+ });+ await hooks.publish({+ idempotencyKey: approvalKey,+ kind: "tool-approval",+ outcome: "approved",+ requestId: "approval",+ response: { text: "private approval answer" },+ scope,+ type: "input.resolved",+ });+ dispatch = prepareAgentInvocationTrace({+ channelMetadata: parent.get(ChannelInstrumentationKey),+ invocation: { callId: "nested", kind: "subagent-call", name: "child" },+ ownerId: "workflow-run",+ startTimeMs: Date.now(),+ serializedContext: serializeContext(parent),+ sessionId: "parent",+ turnId: "turn_0",+ sessionState: {+ "eve.runtime.workflowToolRuns": [+ {+ callId: "workflow",+ hookToken: "hook",+ runId: "workflow-run",+ toolName: "coordinate",+ },+ ],+ },+ });+ });+ parent = await deserializeContext(dispatch!.serializedContext);+ const child = contextFor(audience);+ child.set(ConversationIdKey, dispatch!.dispatch.conversationId!);+ child.set(ParentSessionKey, {+ callId: "nested",+ rootSessionId: "parent",+ sessionId: "parent",+ turn: { id: "turn_0", sequence: 0 },+ });+ if (dispatch!.dispatch.parentTraceContext !== undefined) {+ child.set(ParentTraceContextKey, dispatch!.dispatch.parentTraceContext);+ child.set(SessionTraceSeedKey, {+ ...dispatch!.dispatch.parentTraceContext,+ spanId: runtime.idGenerator.allocateSpanId(),+ traceId: runtime.idGenerator.generateTraceId(),+ });+ }+ const childScope = scopeFor("child", audience);+ const childHooks = runtime.hooks.forTrace!({ agentName: "child", audience });+ await contextStorage.run(child, async () => {+ const childBinding = bindInstrumentationRuntime(runtime, child, {+ agentName: "child",+ rootSessionId: "parent",+ sessionId: "child",+ })!;+ await childBinding.preparePreamble({+ sequence: 0,+ sessionStarted: false,+ turnId: "turn_0",+ });+ await childHooks.publish({+ idempotencyKey: attemptIdempotencyKey(childScope),+ operation,+ scope: childScope,+ type: "step.attempt.started",+ });+ const modelKey = modelCallIdempotencyKey(childScope, 0);+ await childHooks.publish({+ idempotencyKey: modelKey,+ model: { modelId: "test", provider: "test" },+ scope: childScope,+ type: "model.call.started",+ });+ await childHooks.publish({+ idempotencyKey: modelKey,+ content: [{ type: "text", text: "private reply" }],+ finishReason: "stop",+ scope: childScope,+ type: "model.call.completed",+ usage: {+ inputTokens: 10,+ outputTokens: 5,+ inputTokenDetails: { cacheReadTokens: 4, cacheWriteTokens: 2 },+ },+ });+ await childHooks.publish({+ idempotencyKey: attemptIdempotencyKey(childScope),+ scope: childScope,+ type: "step.attempt.completed",+ });+ await childHooks.publish({+ idempotencyKey: turnIdempotencyKey("child", "turn_0"),+ sessionId: "child",+ turnId: "turn_0",+ type: "turn.completed",+ });+ await childHooks.publish({+ idempotencyKey: sessionIdempotencyKey("child"),+ sessionId: "child",+ turnId: "turn_0",+ type: "session.waiting",+ });+ });+ const settled = settleAgentInvocationTrace({+ acceptedAtMs: Date.now(),+ serializedContext: serializeContext(parent),+ sessionId: "parent",+ result: {+ callId: "nested",+ kind: "subagent-result",+ origin: "child",+ subagentName: "child",+ output: "private failure",+ outcome: {+ kind: "terminal",+ result: { kind: "failed", error: { message: "private failure" } },+ usageDelta: {+ inputTokens: 10,+ outputTokens: 5,+ cacheReadTokens: 4,+ cacheWriteTokens: 2,+ },+ },+ },+ });+ if (audience === "private") expect(JSON.stringify(settled)).not.toContain("private failure");+ parent = await deserializeContext(settled);+ await contextStorage.run(parent, async () => {+ await runtime.forceFlush();+ expect(+ runtime.exporter+ .getFinishedSpans()+ .filter((span) => span.attributes["agent.invocation.role"] === "caller"),+ ).toHaveLength(1);+ expect(new ContextAgentTraceStateStore().findInvocations("parent")).toEqual([]);+ await hooks.publish({+ idempotencyKey: toolCallIdempotencyKey(scope, "workflow", 0),+ output: { type: "result", output: "private output" },+ scope,+ type: "tool.call.completed",+ });+ await hooks.publish({+ idempotencyKey: actionKey,+ outcome: "completed",+ output: { type: "result", output: "private output" },+ scope,+ type: "action.completed",+ });+ await hooks.publish({+ idempotencyKey: attemptIdempotencyKey(scope),+ scope,+ type: "step.attempt.completed",+ });+ await binding.instrumentChannelDelivery({+ ctx: parent,+ includeTurn: true,+ outcome: "completed",+ });+ await hooks.publish({+ idempotencyKey: turnIdempotencyKey("parent", "turn_0"),+ sessionId: "parent",+ turnId: "turn_0",+ type: "turn.completed",+ });+ await hooks.publish({+ idempotencyKey: sessionIdempotencyKey("parent"),+ sessionId: "parent",+ turnId: "turn_0",+ type: "session.waiting",+ });+ });+ await runtime.forceFlush();+ const exported = runtime.exporter.getFinishedSpans();+ const bytes = JsonTraceSerializer.serializeRequest(exported)!;+ const traceIds = [...new Set(exported.map((span) => span.spanContext().traceId))];+ expect(traceIds).toHaveLength(2);+ const traces = traceIds.map((traceId) =>+ assembleLocalTrace(+ traceId,+ parseLocalTraceSegment(new TextDecoder().decode(bytes), traceId),+ ),+ );+ const parsed = traceIds.flatMap((traceId) =>+ parseLocalTraceSegment(new TextDecoder().decode(bytes), traceId),+ );+ expect(parsed).toHaveLength(exported.length);+ expect(+ parsed.every(+ (span) => span.attributes["gen_ai.conversation.id"] === "original-conversation",+ ),+ ).toBe(true);+ expect(+ parsed.every((span) => Number(span.attributes["agent.trace.schema.version"]) === 4),+ ).toBe(true);+ for (const span of parsed) {+ expect(span.attributes).not.toHaveProperty("agent.session.id");+ expect(span.attributes).not.toHaveProperty("vercel.session_id");+ }+ expect(+ parsed+ .filter((span) => span.parentSpanId === undefined)+ .map((span) => span.name)+ .sort(),+ ).toEqual(["invoke_agent child", "invoke_agent parent"]);+ expect(parsed.filter(isAgentTurnSpan)).toHaveLength(2);+ const caller = parsed.find((span) => span.attributes["agent.invocation.role"] === "caller")!;+ const activation = parsed.find(+ (span) => span.name === "invoke_agent child" && isAgentTurnSpan(span),+ )!;+ const parentActivation = parsed.find(+ (span) => span.name === "invoke_agent parent" && isAgentTurnSpan(span),+ )!;+ expect(activation.parentSpanId).toBeUndefined();+ expect(activation.traceId).not.toBe(caller.traceId);+ expect(parentActivation.attributes).toMatchObject({+ "agent.channel.delivery.id": "delivery",+ "agent.channel.kind": "http",+ "agent.channel.name": "web",+ });+ const childSpan = exported.find((span) => span.spanContext().spanId === activation.spanId)!;+ expect(activation.attributes).toMatchObject({+ "gen_ai.usage.input_tokens": 10,+ "gen_ai.usage.output_tokens": 5,+ "agent.usage.input_tokens": 10,+ "agent.usage.output_tokens": 5,+ });+ expect(caller.attributes["gen_ai.usage.input_tokens"]).toBeUndefined();+ expect(caller.attributes["gen_ai.usage.output_tokens"]).toBeUndefined();+ expect(childSpan.links).toEqual([+ {+ context: expect.objectContaining({ spanId: caller.spanId, traceId: caller.traceId }),+ attributes: { "eve.link.type": "agent.dispatch" },+ },+ ]);+ expect(parsed.map((span) => span.name).sort()).toEqual(+ [+ "agent.action",+ "agent.action",+ "agent.approval",+ "agent.step",+ "agent.step",+ "chat test",+ "execute_tool coordinate",+ "invoke_agent child",+ "invoke_agent parent",+ ].sort(),+ );+ expect(summarizeLocalTrace(parsed)).toMatchObject({+ inputTokens: 10,+ outputTokens: 5,+ cacheReadTokens: 4,+ cacheWriteTokens: 2,+ });+ const items = traces.flatMap(buildConversationItems);+ expect(items.find((item) => item.kind === "assistant")).toBeDefined();+ const metadata = new TextDecoder().decode(+ JsonTraceSerializer.serializeRequest(runtime.metadata.getFinishedSpans())!,+ );+ expect(metadata).not.toContain("private ");+ if (audience === "private") expect(new TextDecoder().decode(bytes)).not.toContain("private ");+ else expect(new TextDecoder().decode(bytes)).toContain("private failure");+ await runtime.shutdown();+ },+ );+});diff --git a/packages/eve/src/tracing/agent-tool-instrumentation.ts b/packages/eve/src/tracing/agent-tool-instrumentation.tsindex 73929cb00..65d434deb 100644--- a/packages/eve/src/tracing/agent-tool-instrumentation.ts+++ b/packages/eve/src/tracing/agent-tool-instrumentation.ts@@ -1,7 +1,6 @@import {ROOT_CONTEXT,SpanKind,- SpanStatusCode,type Context,type Span,type SpanContext,@@ -17,9 +16,11 @@ import type {import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";import { contentAttribute } from "#tracing/agent-otel-content.js";import { agentSpanNamingAttributes } from "#tracing/agent-span-naming.js";+import { agentTraceIdentityAttributes } from "#tracing/agent-otel-attributes.js";import { withChannelAudience } from "#tracing/channel-audience-context.js";import type { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import type { AgentActionContext } from "#tracing/agent-action-instrumentation.js";+import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";interface ToolSpanState {readonly actionKey: string;@@ -213,24 +214,22 @@ export function createAgentToolInstrumentation(input: {}}-function toolAttributes(event: InstrumentationToolCallStartedEvent): Record<string, string> {+function toolAttributes(+ event: InstrumentationToolCallStartedEvent,+): Record<string, string | number> {return {"gen_ai.operation.name": "execute_tool","gen_ai.tool.call.id": event.callId,"gen_ai.tool.name": event.toolName,"gen_ai.tool.type": "function",...agentSpanNamingAttributes(`execute_tool ${event.toolName}`, "execute_tool"),+ ...agentTraceIdentityAttributes({+ rootSessionId: event.scope.rootSessionId ?? event.scope.sessionId,+ sessionId: event.scope.sessionId,+ }),};}function contextFromSpanContext(spanContext: SpanContext): Context {return trace.setSpan(ROOT_CONTEXT, trace.wrapSpanContext(spanContext));}--function recordError(span: Span, error: unknown): void {- span.setAttribute("error.type", error instanceof Error ? error.name || "Error" : "_OTHER");- if (error instanceof Error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- } else span.setStatus({ code: SpanStatusCode.ERROR });-}diff --git a/packages/eve/src/tracing/agent-trace-context-codec.ts b/packages/eve/src/tracing/agent-trace-context-codec.tsnew file mode 100644index 000000000..3aec9db5e--- /dev/null+++ b/packages/eve/src/tracing/agent-trace-context-codec.ts@@ -0,0 +1,326 @@+import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";++import type {+ AgentActionTraceState,+ AgentActionTraceTerminalState,+ AgentInvocationTraceState,+ AgentSessionTraceState,+ AgentTurnTraceState,+} from "#tracing/agent-trace-state.js";+import { normalizeChannelAudience } from "#shared/channel-audience.js";+import { readInstrumentationDecision } from "#shared/instrumentation-decision.js";+import { boundedTraceError } from "#tracing/bounded-error.js";++export const AGENT_TRACE_CONTEXT_KEY = "eve.harness.agentTrace";++export interface AgentTraceContextState {+ readonly actionAnchors: Readonly<Record<string, AgentActionTraceState>>;+ readonly actions: Readonly<Record<string, AgentActionTraceState>>;+ readonly invocations: Readonly<Record<string, AgentInvocationTraceState>>;+ readonly sessions: Readonly<Record<string, AgentSessionTraceState>>;+ readonly turns: Readonly<Record<string, AgentTurnTraceState>>;+}++export function emptyAgentTraceContextState(): AgentTraceContextState {+ return { actionAnchors: {}, actions: {}, invocations: {}, sessions: {}, turns: {} };+}++export function serializeAgentTraceContextState(state: AgentTraceContextState): unknown {+ return {+ actionAnchors: state.actionAnchors,+ actions: state.actions,+ invocations: Object.fromEntries(+ Object.entries(state.invocations).map(([id, value]) => [+ id,+ {+ ...value,+ terminal:+ value.terminal === undefined+ ? undefined+ : {+ ...value.terminal,+ error:+ value.recordOutputs === true ? serializeError(value.terminal.error) : undefined,+ },+ },+ ]),+ ),+ sessions: Object.fromEntries(+ Object.entries(state.sessions).map(([id, value]) => [+ id,+ { ...value, context: serializeSpanContext(value.context) },+ ]),+ ),+ turns: Object.fromEntries(+ Object.entries(state.turns).map(([id, value]) => [+ id,+ {+ ...value,+ caller: value.caller === undefined ? undefined : serializeSpanContext(value.caller),+ channelDelivery:+ value.channelDelivery === undefined+ ? undefined+ : {+ ...value.channelDelivery,+ requestTraceContext:+ value.channelDelivery.requestTraceContext === undefined+ ? undefined+ : serializeSpanContext(value.channelDelivery.requestTraceContext),+ },+ context: serializeSpanContext(value.context),+ terminal:+ value.terminal === undefined+ ? undefined+ : value.terminal.type === "turn.failed"+ ? { error: serializeError(value.terminal.error), type: value.terminal.type }+ : { type: value.terminal.type },+ },+ ]),+ ),+ };+}++export function deserializeAgentTraceContextState(data: unknown): AgentTraceContextState {+ if (!isRecord(data)) return emptyAgentTraceContextState();+ return {+ actionAnchors: deserializeRecord(data.actionAnchors, deserializeAction),+ actions: deserializeRecord(data.actions, deserializeAction),+ invocations: deserializeRecord(data.invocations, deserializeInvocation),+ sessions: deserializeRecord(data.sessions, deserializeSession),+ turns: deserializeRecord(data.turns, deserializeTurn),+ };+}++function deserializeSession(value: unknown): AgentSessionTraceState | undefined {+ if (!isRecord(value) || !isSpanContext(value.context)) return undefined;+ return {+ agentName: typeof value.agentName === "string" ? value.agentName : undefined,+ channelAudience: normalizeChannelAudience(value.channelAudience),+ channelKind: typeof value.channelKind === "string" ? value.channelKind : undefined,+ context: value.context,+ decision: readInstrumentationDecision(value.decision),+ parentLineage: deserializeParentLineage(value.parentLineage),+ rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",+ };+}++function deserializeTurn(value: unknown): AgentTurnTraceState | undefined {+ if (!isRecord(value) || !isSpanContext(value.context) || typeof value.startTimeMs !== "number") {+ return undefined;+ }+ return {+ caller: isSpanContext(value.caller) ? value.caller : undefined,+ channelDelivery: deserializeTurnChannelDelivery(value.channelDelivery),+ context: value.context,+ modelUsage: deserializeModelUsage(value.modelUsage),+ parentLineage: deserializeParentLineage(value.parentLineage),+ rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",+ sequence: typeof value.sequence === "number" ? value.sequence : 0,+ startTimeMs: value.startTimeMs,+ subagentName: typeof value.subagentName === "string" ? value.subagentName : undefined,+ terminal: deserializeTurnTerminal(value.terminal),+ };+}++function deserializeTurnChannelDelivery(value: unknown): AgentTurnTraceState["channelDelivery"] {+ if (+ !isRecord(value) ||+ typeof value.channelKind !== "string" ||+ typeof value.channelName !== "string" ||+ typeof value.deliveryId !== "string"+ ) {+ return undefined;+ }+ return {+ channelKind: value.channelKind,+ channelName: value.channelName,+ deliveryId: value.deliveryId,+ inputAttribute: typeof value.inputAttribute === "string" ? value.inputAttribute : undefined,+ requestId: typeof value.requestId === "string" ? value.requestId : undefined,+ requestTraceContext: isSpanContext(value.requestTraceContext)+ ? value.requestTraceContext+ : undefined,+ };+}++function deserializeAction(value: unknown): AgentActionTraceState | undefined {+ if (+ !isRecord(value) ||+ typeof value.attemptIndex !== "number" ||+ typeof value.callId !== "string" ||+ !isActionKind(value.kind) ||+ typeof value.name !== "string" ||+ !isSpanContext(value.parent) ||+ typeof value.rootSessionId !== "string" ||+ typeof value.sessionId !== "string" ||+ typeof value.spanId !== "string" ||+ typeof value.startTimeMs !== "number" ||+ typeof value.stepIndex !== "number" ||+ typeof value.turnId !== "string"+ ) {+ return undefined;+ }+ return {+ attemptIndex: value.attemptIndex,+ callId: value.callId,+ channelAudience: normalizeChannelAudience(value.channelAudience),+ inputAttribute: typeof value.inputAttribute === "string" ? value.inputAttribute : undefined,+ kind: value.kind,+ name: value.name,+ parent: value.parent,+ rootSessionId: value.rootSessionId,+ sessionId: value.sessionId,+ spanId: value.spanId,+ startTimeMs: value.startTimeMs,+ stepIndex: value.stepIndex,+ turnId: value.turnId,+ };+}++function deserializeInvocation(value: unknown): AgentInvocationTraceState | undefined {+ const action = deserializeAction(value);+ if (+ action === undefined ||+ (action.kind !== "subagent-call" && action.kind !== "remote-agent-call") ||+ !isRecord(value) ||+ typeof value.parentActionCallId !== "string"+ ) {+ return undefined;+ }+ return {+ attemptIndex: action.attemptIndex,+ callId: action.callId,+ channelAudience: action.channelAudience,+ kind: action.kind,+ name: action.name,+ parent: action.parent,+ parentActionCallId: value.parentActionCallId,+ recordOutputs: value.recordOutputs === true,+ rootSessionId: action.rootSessionId,+ sessionId: action.sessionId,+ spanId: action.spanId,+ startTimeMs: action.startTimeMs,+ stepIndex: action.stepIndex,+ terminal: deserializeActionTerminal(value.terminal),+ turnId: action.turnId,+ };+}++function deserializeActionTerminal(value: unknown): AgentActionTraceTerminalState | undefined {+ if (!isRecord(value) || !isActionOutcome(value.outcome)) return undefined;+ return {+ acceptedAtMs: typeof value.acceptedAtMs === "number" ? value.acceptedAtMs : undefined,+ error: deserializeError(value.error),+ outcome: value.outcome,+ usage: isRecord(value.usage) ? value.usage : undefined,+ } as AgentActionTraceTerminalState;+}++function deserializeModelUsage(+ value: unknown,+): NonNullable<AgentTurnTraceState["modelUsage"]> | undefined {+ if (!isRecord(value)) return undefined;+ const inputTokens = typeof value.inputTokens === "number" ? value.inputTokens : undefined;+ const outputTokens = typeof value.outputTokens === "number" ? value.outputTokens : undefined;+ return inputTokens === undefined && outputTokens === undefined+ ? undefined+ : { inputTokens, outputTokens };+}++function deserializeParentLineage(value: unknown): AgentSessionTraceState["parentLineage"] {+ if (+ !isRecord(value) ||+ typeof value.callId !== "string" ||+ typeof value.sessionId !== "string" ||+ typeof value.turnId !== "string"+ ) {+ return undefined;+ }+ return {+ callId: value.callId,+ sessionId: value.sessionId,+ subagentName: typeof value.subagentName === "string" ? value.subagentName : undefined,+ turnId: value.turnId,+ };+}++function deserializeTurnTerminal(value: unknown): AgentTurnTraceState["terminal"] {+ if (!isRecord(value) || !isTurnTerminalType(value.type)) return undefined;+ return value.type === "turn.failed"+ ? { error: deserializeError(value.error), type: value.type }+ : { type: value.type };+}++function deserializeRecord<T>(+ value: unknown,+ deserialize: (entry: unknown) => T | undefined,+): Record<string, T> {+ if (!isRecord(value)) return {};+ const result: Record<string, T> = {};+ for (const [key, entry] of Object.entries(value)) {+ const parsed = deserialize(entry);+ if (parsed !== undefined) result[key] = parsed;+ }+ return result;+}++function serializeSpanContext(context: SpanContext): Record<string, unknown> {+ return {+ isRemote: context.isRemote,+ spanId: context.spanId,+ traceFlags: context.traceFlags,+ traceId: context.traceId,+ };+}++function serializeError(error: unknown): unknown {+ if (!(error instanceof Error)) return undefined;+ const bounded = boundedTraceError(error);+ return { message: bounded.message, name: bounded.name, stack: bounded.stack };+}++function deserializeError(value: unknown): Error | undefined {+ if (!isRecord(value) || typeof value.message !== "string") return undefined;+ const error = new Error(value.message);+ if (typeof value.name === "string") error.name = value.name;+ if (typeof value.stack === "string") error.stack = value.stack;+ return error;+}++function isActionKind(value: unknown): value is AgentActionTraceState["kind"] {+ return (+ value === "load-skill" ||+ value === "remote-agent-call" ||+ value === "subagent-call" ||+ value === "tool-call"+ );+}++function isActionOutcome(value: unknown): value is AgentActionTraceTerminalState["outcome"] {+ return (+ value === "abandoned" ||+ value === "cancelled" ||+ value === "completed" ||+ value === "failed" ||+ value === "rejected"+ );+}++function isTurnTerminalType(+ value: unknown,+): value is "turn.cancelled" | "turn.completed" | "turn.failed" {+ return value === "turn.cancelled" || value === "turn.completed" || value === "turn.failed";+}++function isSpanContext(value: unknown): value is SpanContext {+ return (+ isRecord(value) &&+ typeof value.spanId === "string" &&+ typeof value.traceFlags === "number" &&+ typeof value.traceId === "string"+ );+}++function isRecord(value: unknown): value is Record<string, unknown> {+ return typeof value === "object" && value !== null && !Array.isArray(value);+}diff --git a/packages/eve/src/tracing/agent-trace-context-store.test.ts b/packages/eve/src/tracing/agent-trace-context-store.test.tsindex b52f7014a..356543f59 100644--- a/packages/eve/src/tracing/agent-trace-context-store.test.ts+++ b/packages/eve/src/tracing/agent-trace-context-store.test.ts@@ -8,7 +8,7 @@ import {preserveSerializedAgentTraceState,readActionTraceContext,readCurrentSessionTraceDecision,- readSessionTraceContext,+ readTurnTraceContext,} from "#tracing/agent-trace-context-store.js";describe("ContextAgentTraceStateStore", () => {@@ -22,10 +22,17 @@ describe("ContextAgentTraceStateStore", () => {rootSessionId: "session-1",});store.setTurn("session-1", "turn-1", {+ channelDelivery: {+ channelKind: "channel:slack",+ channelName: "slack",+ deliveryId: "delivery-1",+ inputAttribute: '{"message":"hello"}',+ requestId: "request-1",+ requestTraceContext: { ...spanContext("5", "6"), isRemote: true },+ },context: spanContext("1", "3"),modelUsage: { inputTokens: 12, outputTokens: 4 },- parentIsRemote: true,- parentSpanId: "2".repeat(16),+ caller: { ...spanContext("4", "2"), isRemote: true },rootSessionId: "session-1",sequence: 0,startTimeMs: 1_700_000_000_000,@@ -41,9 +48,16 @@ describe("ContextAgentTraceStateStore", () => {expect(store.getSession("session-1")?.context).toEqual(spanContext("1", "2"));expect(store.getTurn("session-1", "turn-1")?.context).toEqual(spanContext("1", "3"));expect(store.getTurn("session-1", "turn-1")).toMatchObject({+ channelDelivery: {+ channelKind: "channel:slack",+ channelName: "slack",+ deliveryId: "delivery-1",+ inputAttribute: '{"message":"hello"}',+ requestId: "request-1",+ requestTraceContext: { ...spanContext("5", "6"), isRemote: true },+ },modelUsage: { inputTokens: 12, outputTokens: 4 },- parentIsRemote: true,- parentSpanId: "2".repeat(16),+ caller: { ...spanContext("4", "2"), isRemote: true },startTimeMs: 1_700_000_000_000,subagentName: "researcher",});@@ -64,7 +78,7 @@ describe("ContextAgentTraceStateStore", () => {});store.setTurn("session-1", "turn-1", {context: spanContext("1", "3"),- parentSpanId: "2".repeat(16),+ caller: spanContext("4", "2"),rootSessionId: "session-1",sequence: 0,startTimeMs: 1_700_000_000_000,@@ -83,7 +97,7 @@ describe("ContextAgentTraceStateStore", () => {const store = new ContextAgentTraceStateStore();store.setTurn("session-1", "turn-1", {context: spanContext("1", "3"),- parentSpanId: "2".repeat(16),+ caller: spanContext("4", "2"),rootSessionId: "session-1",sequence: 0,startTimeMs: 1_700_000_000_000,@@ -128,8 +142,8 @@ describe("ContextAgentTraceStateStore", () => {});});-describe("readSessionTraceContext", () => {- it("reads one session's trace context out of a serialized context", async () => {+describe("readTurnTraceContext", () => {+ it("reads one active turn's trace context out of a serialized context", async () => {const context = new ContextContainer();await contextStorage.run(context, () => {context.set(SessionTraceSeedKey, {@@ -144,35 +158,49 @@ describe("readSessionTraceContext", () => {});new ContextAgentTraceStateStore().setSession("session-1", {context: spanContext("1", "2"),+ decision: { action: "record", recordInputs: true, recordOutputs: false },rootSessionId: "session-1",});+ new ContextAgentTraceStateStore().setTurn("session-1", "turn-1", {+ context: spanContext("3", "4"),+ rootSessionId: "session-1",+ sequence: 0,+ startTimeMs: 1,+ });});const serialized = await serializeContext(context);- expect(readSessionTraceContext(serialized, "session-1")).toEqual({- ...spanContext("1", "2"),+ expect(readTurnTraceContext(serialized, "session-1", "turn-1")).toEqual({+ ...spanContext("3", "4"),decision: { action: "record", recordInputs: true, recordOutputs: false },forwardedTracePolicy: {ceiling: { recordInputs: true, recordOutputs: false },originAudience: "private",},});- expect(readSessionTraceContext(serialized, "session-2")).toBeUndefined();- expect(readSessionTraceContext({}, "session-1")).toBeUndefined();+ expect(readTurnTraceContext(serialized, "session-1", "turn-2")).toBeUndefined();+ expect(readTurnTraceContext({}, "session-1", "turn-1")).toBeUndefined();});- it("preserves a stored decision when the legacy context has no trace seed", async () => {+ it("preserves a stored decision when the context has no trace seed", async () => {const context = new ContextContainer();await contextStorage.run(context, () => {- new ContextAgentTraceStateStore().setSession("session-1", {+ const store = new ContextAgentTraceStateStore();+ store.setSession("session-1", {context: spanContext("1", "2"),decision: { action: "record", recordInputs: false, recordOutputs: true },rootSessionId: "session-1",});+ store.setTurn("session-1", "turn-1", {+ context: spanContext("3", "4"),+ rootSessionId: "session-1",+ sequence: 0,+ startTimeMs: 1,+ });});- expect(readSessionTraceContext(await serializeContext(context), "session-1")).toEqual({- ...spanContext("1", "2"),+ expect(readTurnTraceContext(await serializeContext(context), "session-1", "turn-1")).toEqual({+ ...spanContext("3", "4"),decision: { action: "record", recordInputs: false, recordOutputs: true },});});@@ -207,7 +235,7 @@ describe("readActionTraceContext", () => {traceFlags: 1,traceId: "1".repeat(32),});- new ContextAgentTraceStateStore().setAction("action-1", {+ new ContextAgentTraceStateStore().setAction("action:session-1:turn-1:call-1", {attemptIndex: 0,callId: "call-1",kind: "subagent-call",@@ -246,7 +274,7 @@ describe("readActionTraceContext", () => {decision: { action: "record", recordInputs: true, recordOutputs: false },rootSessionId: "session-1",});- store.setAction("action-1", {+ store.setAction("action:session-1:turn-1:call-1", {attemptIndex: 0,callId: "call-1",kind: "subagent-call",diff --git a/packages/eve/src/tracing/agent-trace-context-store.ts b/packages/eve/src/tracing/agent-trace-context-store.tsindex ae8a0134c..b54ba0b2d 100644--- a/packages/eve/src/tracing/agent-trace-context-store.ts+++ b/packages/eve/src/tracing/agent-trace-context-store.ts@@ -7,33 +7,97 @@ import type { ContextAccessor } from "#context/key.js";import { SessionTraceSeedKey, type SessionTraceSeed } from "#context/keys.js";import type {AgentActionTraceState,+ AgentInvocationTraceState,AgentSessionTraceState,AgentTraceStateStore,AgentTurnTraceState,} from "#tracing/agent-trace-state.js";-import { normalizeChannelAudience } from "#shared/channel-audience.js";-import {- type InstrumentationDecision,- readInstrumentationDecision,-} from "#shared/instrumentation-decision.js";+import { actionIdempotencyKey } from "#instrumentation/lifecycle.js";+import { deriveTaskId } from "#tasks/task-id.js";+import type { SessionStateMap } from "#harness/types.js";+import { getWorkflowToolRuns } from "#harness/workflow-tool-runs.js";+import { getSessionTaskIndex } from "#tasks/session-index.js";+import { createLogger } from "#internal/logging.js";+import type { InstrumentationDecision } from "#shared/instrumentation-decision.js";import {decisionToTraceContentCeiling,resolveForwardedTraceSeed,} from "#shared/forwarded-trace-policy.js";--interface AgentTraceContextState {- readonly actions: Readonly<Record<string, AgentActionTraceState>>;- readonly sessions: Readonly<Record<string, AgentSessionTraceState>>;- readonly turns: Readonly<Record<string, AgentTurnTraceState>>;-}--const AgentTraceContextKey = new ContextKey<AgentTraceContextState>("eve.harness.agentTrace", {+import {+ deserializeAgentTraceContextState,+ AGENT_TRACE_CONTEXT_KEY,+ emptyAgentTraceContextState,+ serializeAgentTraceContextState,+ type AgentTraceContextState,+} from "#tracing/agent-trace-context-codec.js";++const AgentTraceContextKey = new ContextKey<AgentTraceContextState>(AGENT_TRACE_CONTEXT_KEY, {codec: {- deserialize: deserializeState,- serialize: serializeState,+ deserialize: deserializeAgentTraceContextState,+ serialize: serializeAgentTraceContextState,},});+/** Run after task-provider commits, so a returned background receipt keeps its anchor. */+export function pruneAgentTraceState(+ context: ContextAccessor,+ sessionId: string,+ sessionState: SessionStateMap | undefined,+): void {+ try {+ pruneTraceOwnership(context, sessionId, sessionState);+ } catch (error) {+ createLogger("tracing.retention").warn(+ "could not reconcile trace ownership; preserving trace state",+ { error },+ );+ }+}++function pruneTraceOwnership(+ context: ContextAccessor,+ sessionId: string,+ sessionState: SessionStateMap | undefined,+): void {+ const state = context.get(AgentTraceContextKey);+ if (state === undefined) return;+ const calls = new Set(getWorkflowToolRuns(sessionState).map((run) => run.callId));+ const tasks = new Set(+ getSessionTaskIndex(sessionState)+ .filter((task) => task.terminalView === undefined)+ .map((task) => task.taskId),+ );+ const actionAnchors = Object.fromEntries(+ Object.entries(state.actionAnchors).filter(+ ([key, action]) =>+ action.sessionId !== sessionId ||+ state.actions[key] !== undefined ||+ calls.has(action.callId) ||+ tasks.has(+ deriveTaskId({+ callId: action.callId,+ parentSessionId: sessionId,+ parentTurnId: action.turnId,+ }),+ ),+ ),+ );+ const retainedCalls = new Set(+ Object.values(actionAnchors)+ .filter((action) => action.sessionId === sessionId)+ .map((action) => action.callId),+ );+ const invocations = Object.fromEntries(+ Object.entries(state.invocations).filter(+ ([, invocation]) =>+ invocation.sessionId !== sessionId ||+ invocation.terminal !== undefined ||+ retainedCalls.has(invocation.parentActionCallId),+ ),+ );+ context.set(AgentTraceContextKey, { ...state, actionAnchors, invocations });+}+/** Reads the decision already bound to a session in the current worker context. */export function readCurrentSessionTraceDecision(sessionId: string,@@ -60,21 +124,19 @@ export function preserveSerializedAgentTraceState(: { ...original, [AgentTraceContextKey.name]: traceState };}-/**- * Reads a named session's trace context straight out of a serialized context,- * which {@link ContextAgentTraceStateStore} cannot do — its reads are scoped- * to the ambient session.- */-export function readSessionTraceContext(+/** Reads the active turn context straight out of a serialized Workflow context. */+export function readTurnTraceContext(serializedContext: Readonly<Record<string, unknown>>,sessionId: string,+ turnId: string,): SessionTraceContext | undefined {const raw = serializedContext[AgentTraceContextKey.name];if (raw === undefined) return undefined;- const session = deserializeState(raw).sessions[sessionId];- return session === undefined+ const state = deserializeAgentTraceContextState(raw);+ const turn = state.turns[turnKey(sessionId, turnId)];+ return turn === undefined? undefined- : withTraceDecision(serializedContext, session.context, session.decision);+ : withTraceDecision(serializedContext, turn.context, state.sessions[sessionId]?.decision);}/** Reads the durable action span that should parent a dispatched child agent. */@@ -86,10 +148,9 @@ export function readActionTraceContext(): SessionTraceContext | undefined {const raw = serializedContext[AgentTraceContextKey.name];if (raw === undefined) return undefined;- const state = deserializeState(raw);- const action = Object.values(state.actions).find(- (state) => state.sessionId === sessionId && state.turnId === turnId && state.callId === callId,- );+ const state = deserializeAgentTraceContextState(raw);+ const key = actionIdempotencyKey(sessionId, turnId, callId);+ const action = state.invocations[key] ?? state.actions[key] ?? state.actionAnchors[key];if (action === undefined) return undefined;return withTraceDecision(serializedContext,@@ -103,6 +164,102 @@ export function readActionTraceContext();}+/** Task IDs already bind the originating call and turn, including after that turn ends. */+export function readTaskActionTrace(+ serializedContext: Readonly<Record<string, unknown>>,+ sessionId: string,+ taskId: string,+): AgentActionTraceState | undefined {+ const state = deserializeAgentTraceContextState(serializedContext[AgentTraceContextKey.name]);+ return Object.values(state.actionAnchors).find(+ (action) =>+ action.sessionId === sessionId &&+ deriveTaskId({+ callId: action.callId,+ parentSessionId: sessionId,+ parentTurnId: action.turnId,+ }) === taskId,+ );+}++export function recordNestedAgentInvocation(input: {+ readonly callId: string;+ readonly kind: "remote-agent-call" | "subagent-call";+ readonly name: string;+ readonly outerCallId: string;+ readonly recordOutputs?: boolean;+ readonly serializedContext: Record<string, unknown>;+ readonly sessionId: string;+ readonly spanId: string;+ readonly startTimeMs: number;+ readonly turnId: string;+}): Record<string, unknown> {+ const raw = input.serializedContext[AgentTraceContextKey.name];+ if (raw === undefined) return input.serializedContext;+ const state = deserializeAgentTraceContextState(raw);+ const outerKey = actionIdempotencyKey(input.sessionId, input.turnId, input.outerCallId);+ const outer = state.actions[outerKey] ?? state.actionAnchors[outerKey];+ if (outer === undefined) return input.serializedContext;+ const key = actionIdempotencyKey(input.sessionId, input.turnId, input.callId);+ if (state.invocations[key] !== undefined) return input.serializedContext;+ const invocation: AgentInvocationTraceState = {+ attemptIndex: outer.attemptIndex,+ callId: input.callId,+ channelAudience: outer.channelAudience,+ kind: input.kind,+ name: input.name,+ parent: {+ spanId: outer.spanId,+ traceFlags: outer.parent.traceFlags,+ traceId: outer.parent.traceId,+ },+ parentActionCallId: input.outerCallId,+ recordOutputs: input.recordOutputs === true,+ rootSessionId: outer.rootSessionId,+ sessionId: outer.sessionId,+ spanId: input.spanId,+ startTimeMs: input.startTimeMs,+ stepIndex: outer.stepIndex,+ turnId: outer.turnId,+ };+ return {+ ...input.serializedContext,+ [AgentTraceContextKey.name]: serializeAgentTraceContextState({+ ...state,+ invocations: { ...state.invocations, [key]: invocation },+ }),+ };+}++export function recordActionInvocationKind(input: {+ readonly callId: string;+ readonly kind: "remote-agent-call" | "subagent-call";+ readonly serializedContext: Record<string, unknown>;+ readonly sessionId: string;+ readonly turnId: string;+}): Record<string, unknown> {+ const raw = input.serializedContext[AgentTraceContextKey.name];+ if (raw === undefined) return input.serializedContext;+ const state = deserializeAgentTraceContextState(raw);+ const update = (action: AgentActionTraceState): AgentActionTraceState =>+ action.sessionId === input.sessionId &&+ action.turnId === input.turnId &&+ action.callId === input.callId+ ? { ...action, kind: input.kind }+ : action;+ return {+ ...input.serializedContext,+ [AgentTraceContextKey.name]: serializeAgentTraceContextState({+ ...state,+ actionAnchors: Object.fromEntries(+ Object.entries(state.actionAnchors).map(([key, action]) => [key, update(action)]),+ ),+ actions: Object.fromEntries(+ Object.entries(state.actions).map(([key, action]) => [key, update(action)]),+ ),+ }),+ };+}function withTraceDecision(serializedContext: Readonly<Record<string, unknown>>,context: SpanContext,@@ -143,6 +300,15 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {});}+ deleteActionAnchors(sessionId: string): void {+ updateState((state) => ({+ ...state,+ actionAnchors: Object.fromEntries(+ Object.entries(state.actionAnchors).filter(([, anchor]) => anchor.sessionId !== sessionId),+ ),+ }));+ }+deleteActions(sessionId: string, turnId?: string): void {updateState((state) => {const actions = { ...state.actions };@@ -155,6 +321,27 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {});}+ deleteInvocation(idempotencyKey: string): void {+ updateState((state) => {+ const invocations = { ...state.invocations };+ delete invocations[idempotencyKey];+ return { ...state, invocations };+ });+ }++ deleteInvocations(sessionId: string, turnId?: string): void {+ updateState((state) => ({+ ...state,+ invocations: Object.fromEntries(+ Object.entries(state.invocations).filter(+ ([, invocation]) =>+ invocation.sessionId !== sessionId ||+ (turnId !== undefined && invocation.turnId !== turnId),+ ),+ ),+ }));+ }+deleteSession(sessionId: string): void {updateState((state) => {const sessions = { ...state.sessions };@@ -177,6 +364,34 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {);}+ findActionAnchor(+ sessionId: string,+ turnId: string,+ callId: string,+ ): AgentActionTraceState | undefined {+ return Object.values(+ contextStorage.getStore()?.get(AgentTraceContextKey)?.actionAnchors ?? {},+ ).find(+ (anchor) =>+ anchor.sessionId === sessionId && anchor.turnId === turnId && anchor.callId === callId,+ );+ }++ findInvocations(+ sessionId?: string,+ turnId?: string,+ parentActionCallId?: string,+ ): readonly AgentInvocationTraceState[] {+ return Object.values(+ contextStorage.getStore()?.get(AgentTraceContextKey)?.invocations ?? {},+ ).filter(+ (invocation) =>+ (sessionId === undefined || invocation.sessionId === sessionId) &&+ (turnId === undefined || invocation.turnId === turnId) &&+ (parentActionCallId === undefined || invocation.parentActionCallId === parentActionCallId),+ );+ }+getAction(idempotencyKey: string): AgentActionTraceState | undefined {return contextStorage.getStore()?.get(AgentTraceContextKey)?.actions[idempotencyKey];}@@ -196,6 +411,20 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {}));}+ setActionAnchor(idempotencyKey: string, value: AgentActionTraceState): void {+ updateState((state) => ({+ ...state,+ actionAnchors: { ...state.actionAnchors, [idempotencyKey]: value },+ }));+ }++ setInvocation(idempotencyKey: string, value: AgentInvocationTraceState): void {+ updateState((state) => ({+ ...state,+ invocations: { ...state.invocations, [idempotencyKey]: value },+ }));+ }+setSession(sessionId: string, value: AgentSessionTraceState): void {updateState((state) => ({...state,@@ -227,206 +456,10 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {function updateState(update: (state: AgentTraceContextState) => AgentTraceContextState): void {loadContext().set(AgentTraceContextKey, (state) =>- update(state ?? { actions: {}, sessions: {}, turns: {} }),+ update(state ?? emptyAgentTraceContextState()),);}function turnKey(sessionId: string, turnId: string): string {return `${sessionId}\0${turnId}`;}--function serializeState(state: AgentTraceContextState): unknown {- return {- actions: state.actions,- sessions: Object.fromEntries(- Object.entries(state.sessions).map(([id, value]) => [- id,- { ...value, context: serializeSpanContext(value.context) },- ]),- ),- turns: Object.fromEntries(- Object.entries(state.turns).map(([id, value]) => [- id,- {- ...value,- context: serializeSpanContext(value.context),- terminal:- value.terminal === undefined- ? undefined- : value.terminal.type === "turn.failed"- ? { error: serializeError(value.terminal.error), type: value.terminal.type }- : { type: value.terminal.type },- },- ]),- ),- };-}--function deserializeState(data: unknown): AgentTraceContextState {- if (!isRecord(data)) return { actions: {}, sessions: {}, turns: {} };- const actions = deserializeRecord(data.actions, deserializeAction);- const sessions = deserializeRecord(data.sessions, (value) => {- if (!isRecord(value) || !isSpanContext(value.context)) return undefined;- return {- agentName: typeof value.agentName === "string" ? value.agentName : undefined,- channelAudience: normalizeChannelAudience(value.channelAudience),- channelKind: typeof value.channelKind === "string" ? value.channelKind : undefined,- context: value.context,- decision: readInstrumentationDecision(value.decision),- parentLineage: deserializeParentLineage(value.parentLineage),- rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",- } satisfies AgentSessionTraceState;- });- const turns = deserializeRecord(data.turns, (value) => {- if (!isRecord(value) || !isSpanContext(value.context)) return undefined;- if (typeof value.parentSpanId !== "string" || typeof value.startTimeMs !== "number") {- return undefined;- }- return {- context: value.context,- modelUsage: deserializeModelUsage(value.modelUsage),- parentIsRemote: typeof value.parentIsRemote === "boolean" ? value.parentIsRemote : undefined,- parentLineage: deserializeParentLineage(value.parentLineage),- parentSpanId: value.parentSpanId,- rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",- sequence: typeof value.sequence === "number" ? value.sequence : 0,- startTimeMs: value.startTimeMs,- subagentName: typeof value.subagentName === "string" ? value.subagentName : undefined,- terminal: deserializeTerminal(value.terminal),- } satisfies AgentTurnTraceState;- });- return { actions, sessions, turns };-}--function deserializeModelUsage(- value: unknown,-): NonNullable<AgentTurnTraceState["modelUsage"]> | undefined {- if (!isRecord(value)) return undefined;- const inputTokens = typeof value.inputTokens === "number" ? value.inputTokens : undefined;- const outputTokens = typeof value.outputTokens === "number" ? value.outputTokens : undefined;- return inputTokens === undefined && outputTokens === undefined- ? undefined- : { inputTokens, outputTokens };-}--function deserializeAction(value: unknown): AgentActionTraceState | undefined {- if (- !isRecord(value) ||- typeof value.attemptIndex !== "number" ||- typeof value.callId !== "string" ||- !isActionKind(value.kind) ||- typeof value.name !== "string" ||- !isSpanContext(value.parent) ||- typeof value.rootSessionId !== "string" ||- typeof value.sessionId !== "string" ||- typeof value.spanId !== "string" ||- typeof value.startTimeMs !== "number" ||- typeof value.stepIndex !== "number" ||- typeof value.turnId !== "string"- ) {- return undefined;- }- return {- attemptIndex: value.attemptIndex,- callId: value.callId,- channelAudience: normalizeChannelAudience(value.channelAudience),- inputAttribute: typeof value.inputAttribute === "string" ? value.inputAttribute : undefined,- kind: value.kind,- name: value.name,- parent: value.parent,- rootSessionId: value.rootSessionId,- sessionId: value.sessionId,- spanId: value.spanId,- startTimeMs: value.startTimeMs,- stepIndex: value.stepIndex,- turnId: value.turnId,- };-}--function deserializeParentLineage(value: unknown): AgentSessionTraceState["parentLineage"] {- if (- !isRecord(value) ||- typeof value.callId !== "string" ||- typeof value.sessionId !== "string" ||- typeof value.turnId !== "string"- ) {- return undefined;- }- return {- callId: value.callId,- sessionId: value.sessionId,- subagentName: typeof value.subagentName === "string" ? value.subagentName : undefined,- turnId: value.turnId,- };-}--function isActionKind(value: unknown): value is AgentActionTraceState["kind"] {- return (- value === "load-skill" ||- value === "remote-agent-call" ||- value === "subagent-call" ||- value === "tool-call"- );-}--function deserializeRecord<T>(- value: unknown,- deserialize: (entry: unknown) => T | undefined,-): Record<string, T> {- if (!isRecord(value)) return {};- const result: Record<string, T> = {};- for (const [key, entry] of Object.entries(value)) {- const parsed = deserialize(entry);- if (parsed !== undefined) result[key] = parsed;- }- return result;-}--function deserializeTerminal(value: unknown): AgentTurnTraceState["terminal"] {- if (!isRecord(value) || typeof value.type !== "string") return undefined;- const type = value.type;- if (!isTurnTerminalType(type)) return undefined;- return type === "turn.failed" ? { error: deserializeError(value.error), type } : { type };-}--function serializeSpanContext(context: SpanContext): Record<string, unknown> {- return {- isRemote: context.isRemote,- spanId: context.spanId,- traceFlags: context.traceFlags,- traceId: context.traceId,- };-}--function isSpanContext(value: unknown): value is SpanContext {- return (- isRecord(value) &&- typeof value.spanId === "string" &&- typeof value.traceFlags === "number" &&- typeof value.traceId === "string"- );-}--function serializeError(error: unknown): unknown {- return error instanceof Error- ? { message: error.message, name: error.name, stack: error.stack }- : undefined;-}--function deserializeError(value: unknown): Error | undefined {- if (!isRecord(value) || typeof value.message !== "string") return undefined;- const error = new Error(value.message);- if (typeof value.name === "string") error.name = value.name;- if (typeof value.stack === "string") error.stack = value.stack;- return error;-}--function isTurnTerminalType(- value: string,-): value is "turn.cancelled" | "turn.completed" | "turn.failed" {- return value === "turn.cancelled" || value === "turn.completed" || value === "turn.failed";-}--function isRecord(value: unknown): value is Record<string, unknown> {- return typeof value === "object" && value !== null && !Array.isArray(value);-}diff --git a/packages/eve/src/tracing/agent-trace-retention.test.ts b/packages/eve/src/tracing/agent-trace-retention.test.tsnew file mode 100644index 000000000..002635554--- /dev/null+++ b/packages/eve/src/tracing/agent-trace-retention.test.ts@@ -0,0 +1,116 @@+import { describe, expect, it, vi } from "vitest";+import { ContextContainer, contextStorage } from "#context/container.js";+import { serializeContext } from "#context/serialize.js";+import {+ ContextAgentTraceStateStore,+ pruneAgentTraceState,+} from "#tracing/agent-trace-context-store.js";+import { AGENT_TRACE_CONTEXT_KEY } from "#tracing/agent-trace-context-codec.js";+import { deriveTaskId } from "#tasks/task-id.js";+import { AgentTraceSpanProcessor } from "#tracing/agent-trace-span-processor.js";++const anchor = {+ attemptIndex: 0,+ callId: "call",+ kind: "tool-call" as const,+ name: "workflow",+ parent: { spanId: "1".repeat(16), traceFlags: 1, traceId: "2".repeat(32) },+ rootSessionId: "session",+ sessionId: "session",+ spanId: "3".repeat(16),+ startTimeMs: 1,+ stepIndex: 0,+ turnId: "turn",+};++describe("trace retention by live work", () => {+ it("does not turn a task-index compatibility problem into an execution failure", () => {+ const warn = vi.spyOn(console, "warn").mockImplementation(() => {});+ const context = new ContextContainer();+ contextStorage.run(context, () =>+ new ContextAgentTraceStateStore().setActionAnchor("key", anchor),+ );+ const before = serializeContext(context);+ expect(() =>+ pruneAgentTraceState(context, "session", { "eve.tasks": { version: 1, tasks: [] } }),+ ).not.toThrow();+ expect(serializeContext(context)).toEqual(before);+ expect(warn).toHaveBeenCalledWith(+ expect.stringContaining("could not reconcile trace ownership"),+ { error: expect.objectContaining({ message: expect.any(String) }) },+ );+ warn.mockRestore();+ });++ it("does not accumulate anchors across 1000 completed turns", () => {+ const context = new ContextContainer();+ contextStorage.run(context, () => {+ const store = new ContextAgentTraceStateStore();+ for (let index = 0; index < 1000; index++) {+ store.setActionAnchor(`key-${index}`, {+ ...anchor,+ callId: `call-${index}`,+ turnId: `turn-${index}`,+ });+ pruneAgentTraceState(context, "session", undefined);+ }+ });+ expect(serializeContext(context)[AGENT_TRACE_CONTEXT_KEY]).toMatchObject({ actionAnchors: {} });+ });++ it("keeps background anchors until their recorded task finishes", () => {+ const context = new ContextContainer();+ contextStorage.run(context, () =>+ new ContextAgentTraceStateStore().setActionAnchor("key", anchor),+ );+ const task = {+ taskId: deriveTaskId({ callId: "call", parentSessionId: "session", parentTurnId: "turn" }),+ taskRunId: "task-run",+ taskInboxToken: "task-token",+ createdByTurnId: "turn",+ metadata: { kind: "tool", name: "workflow" },+ };+ pruneAgentTraceState(context, "session", { "eve.tasks": { version: 2, tasks: [task] } });+ expect(serializeContext(context)[AGENT_TRACE_CONTEXT_KEY]).toMatchObject({+ actionAnchors: { key: anchor },+ });+ pruneAgentTraceState(context, "session", {+ "eve.tasks": {+ version: 2,+ tasks: [+ {+ ...task,+ terminalView: {+ taskId: task.taskId,+ metadata: task.metadata,+ status: "completed",+ lastOutput: { type: "result", data: "done" },+ },+ },+ ],+ },+ });+ expect(serializeContext(context)[AGENT_TRACE_CONTEXT_KEY]).toMatchObject({ actionAnchors: {} });+ });++ it("unpins completed activation traces only after the writer flushes", async () => {+ const processor = new AgentTraceSpanProcessor([]);+ for (let index = 0; index < 1000; index++) {+ const span = {+ name: "invoke_agent test",+ attributes: {+ "gen_ai.conversation.id": "session",+ "agent.turn.id": `turn-${index}`,+ "gen_ai.operation.name": "invoke_agent",+ },+ spanContext: () => ({ traceId: `trace-${index}` }),+ };+ processor.onStart(span, {});+ processor.onEnd(span);+ expect(processor.activeTraceIds().has(`trace-${index}`)).toBe(true);+ await processor.forceFlush();+ processor.releaseCompletedTraces();+ expect(processor.activeTraceIds().size).toBe(0);+ }+ });+});diff --git a/packages/eve/src/tracing/agent-trace-span-processor.test.ts b/packages/eve/src/tracing/agent-trace-span-processor.test.tsindex c4abcb93e..bc30efc81 100644--- a/packages/eve/src/tracing/agent-trace-span-processor.test.ts+++ b/packages/eve/src/tracing/agent-trace-span-processor.test.ts@@ -16,7 +16,7 @@ describe("AgentTraceSpanProcessor", () => {processor.onEnd(unrelated);expect(child.onStart).not.toHaveBeenCalled();- const turn = span("trace-1", { "agent.session.id": "session-1" });+ const turn = span("trace-1", { "gen_ai.conversation.id": "session-1" });const user = span("trace-1");processor.onStart(turn, {});processor.onStart(user, {});@@ -25,7 +25,7 @@ describe("AgentTraceSpanProcessor", () => {expect(child.onStart).toHaveBeenCalledTimes(2);expect(child.onEnd).toHaveBeenCalledTimes(2);- processor.releaseSession("session-1");+ processor.releaseConversation("session-1");processor.onEnd(span("trace-1"));expect(child.onEnd).toHaveBeenCalledTimes(2);});@@ -34,28 +34,28 @@ describe("AgentTraceSpanProcessor", () => {const processor = new AgentTraceSpanProcessor([]);expect([...processor.activeTraceIds()]).toEqual([]);- processor.onStart(span("trace-1", { "agent.session.id": "session-1" }), {});- processor.onStart(span("trace-2", { "agent.session.id": "session-2" }), {});+ processor.onStart(span("trace-1", { "gen_ai.conversation.id": "session-1" }), {});+ processor.onStart(span("trace-2", { "gen_ai.conversation.id": "session-2" }), {});expect([...processor.activeTraceIds()].sort()).toEqual(["trace-1", "trace-2"]);- expect(processor.releaseSession("session-1")).toBe(true);+ expect(processor.releaseConversation("session-1")).toBe(true);expect([...processor.activeTraceIds()]).toEqual(["trace-2"]);});it("releases every trace a session owns", () => {const processor = new AgentTraceSpanProcessor([]);- processor.onStart(span("trace-1", { "agent.session.id": "session-1" }), {});- processor.onStart(span("trace-2", { "agent.session.id": "session-1" }), {});+ processor.onStart(span("trace-1", { "gen_ai.conversation.id": "session-1" }), {});+ processor.onStart(span("trace-2", { "gen_ai.conversation.id": "session-1" }), {});expect([...processor.activeTraceIds()].sort()).toEqual(["trace-1", "trace-2"]);- expect(processor.releaseSession("session-1")).toBe(true);+ expect(processor.releaseConversation("session-1")).toBe(true);expect([...processor.activeTraceIds()]).toEqual([]);});it("reports no release for a session it never owned", () => {const processor = new AgentTraceSpanProcessor([]);- expect(processor.releaseSession("session-unknown")).toBe(false);+ expect(processor.releaseConversation("session-unknown")).toBe(false);});it("keeps a shared trace pinned when a subagent child finishes first", () => {@@ -66,12 +66,12 @@ describe("AgentTraceSpanProcessor", () => {shutdown: vi.fn(async () => {}),};const processor = new AgentTraceSpanProcessor([child]);- const owned = { "agent.session.id": "session-1" };- const delegated = { "agent.session.id": "child-1" };+ const owned = { "gen_ai.conversation.id": "session-1" };+ const delegated = { "gen_ai.conversation.id": "child-1" };processor.onStart(span("trace-1", owned), {});processor.onStart(span("trace-1", delegated), {});- expect(processor.releaseSession("child-1")).toBe(false);+ expect(processor.releaseConversation("child-1")).toBe(false);expect([...processor.activeTraceIds()]).toEqual(["trace-1"]);// The parent is still writing to the trace the child recorded into.@@ -79,7 +79,7 @@ describe("AgentTraceSpanProcessor", () => {processor.onEnd(later);expect(child.onEnd).toHaveBeenCalledWith(later);- expect(processor.releaseSession("session-1")).toBe(true);+ expect(processor.releaseConversation("session-1")).toBe(true);expect([...processor.activeTraceIds()]).toEqual([]);});@@ -91,7 +91,7 @@ describe("AgentTraceSpanProcessor", () => {shutdown: vi.fn(async () => {}),};const processor = new AgentTraceSpanProcessor([child]);- processor.onStart(span("trace-1", { "agent.session.id": "session-1" }), {});+ processor.onStart(span("trace-1", { "gen_ai.conversation.id": "session-1" }), {});const workflow = span("trace-1", {}, "workflow");processor.onStart(workflow, {});diff --git a/packages/eve/src/tracing/agent-trace-span-processor.ts b/packages/eve/src/tracing/agent-trace-span-processor.tsindex 1b27d922c..861de1a97 100644--- a/packages/eve/src/tracing/agent-trace-span-processor.ts+++ b/packages/eve/src/tracing/agent-trace-span-processor.ts@@ -1,6 +1,10 @@import type { SpanProcessor } from "#compiled/@vercel/otel/index.js";+import { isAgentActivationSpan } from "#tracing/agent-span-contract.js";++const REMEMBERED_TRACE_LIMIT = 2048;interface SpanLike {+ readonly name?: string;readonly attributes: Readonly<Record<string, unknown>>;readonly instrumentationScope?: { readonly name?: string };readonly spanContext: () => { readonly traceId: string };@@ -10,8 +14,9 @@ interface SpanLike {export class AgentTraceSpanProcessor implements SpanProcessor {readonly #children: readonly SpanProcessor[];readonly #ownedTraceIds = new Set<string>();- readonly #sessionTraceIds = new Map<string, Set<string>>();- readonly #traceOwners = new Map<string, string>();+ readonly #completedTraceIds = new Set<string>();+ readonly #rememberedTraceIds = new Set<string>();+ readonly #traceConversations = new Map<string, string>();constructor(children: readonly SpanProcessor[]) {this.#children = children;}@@ -22,15 +27,13 @@ export class AgentTraceSpanProcessor implements SpanProcessor {onStart(span: unknown, parentContext: unknown): void {if (!isSpanLike(span)) return;- const sessionId = span.attributes["agent.session.id"];- if (typeof sessionId === "string") {+ const conversationId = span.attributes["gen_ai.conversation.id"];+ if (typeof conversationId === "string") {const traceId = span.spanContext().traceId;- this.#ownedTraceIds.add(traceId);- if (!this.#traceOwners.has(traceId)) {- this.#traceOwners.set(traceId, sessionId);- const owned = this.#sessionTraceIds.get(sessionId) ?? new Set<string>();- owned.add(traceId);- this.#sessionTraceIds.set(sessionId, owned);+ const known = this.#ownedTraceIds.has(traceId) || this.#rememberedTraceIds.has(traceId);+ if (!known) {+ this.#ownedTraceIds.add(traceId);+ this.#traceConversations.set(traceId, conversationId);}}if (!this.#accepts(span)) return;@@ -40,29 +43,49 @@ export class AgentTraceSpanProcessor implements SpanProcessor {onEnd(span: unknown): void {if (!isSpanLike(span) || !this.#accepts(span)) return;for (const child of this.#children) child.onEnd(span);+ const traceId = span.spanContext().traceId;+ if (+ isAgentActivationSpan({ name: span.name ?? "", attributes: span.attributes }) &&+ this.#traceConversations.get(traceId) === span.attributes["gen_ai.conversation.id"]+ ) {+ this.#completedTraceIds.add(traceId);+ }}- /** Trace ids whose session is still open, so retention never evicts them. */+ /** Trace IDs protected by an unfinished activation or pending final writes. */activeTraceIds(): ReadonlySet<string> {return this.#ownedTraceIds;}- /**- * Forgets every trace one root session owned, reporting whether it owned any.- * A subagent child owns none, so releasing one reports `false` and leaves the- * shared trace pinned until its root finishes.- */- releaseSession(sessionId: string): boolean {- const owned = this.#sessionTraceIds.get(sessionId);- if (owned === undefined) return false;- for (const traceId of owned) {+ /** Called only after writes drain; recently completed IDs still accept late descendants. */+ releaseCompletedTraces(): boolean {+ if (this.#completedTraceIds.size === 0) return false;+ for (const traceId of this.#completedTraceIds) {this.#ownedTraceIds.delete(traceId);- this.#traceOwners.delete(traceId);+ this.#traceConversations.delete(traceId);+ this.#rememberedTraceIds.add(traceId);+ }+ this.#completedTraceIds.clear();+ while (this.#rememberedTraceIds.size > REMEMBERED_TRACE_LIMIT) {+ const oldest = this.#rememberedTraceIds.values().next().value!;+ this.#rememberedTraceIds.delete(oldest);}- this.#sessionTraceIds.delete(sessionId);return true;}+ /** Releases only traces owned by this conversation. */+ releaseConversation(conversationId: string): boolean {+ let released = false;+ for (const [traceId, traceConversationId] of this.#traceConversations) {+ if (traceConversationId !== conversationId) continue;+ released = true;+ this.#ownedTraceIds.delete(traceId);+ this.#completedTraceIds.delete(traceId);+ this.#traceConversations.delete(traceId);+ }+ return released;+ }+async shutdown(): Promise<void> {await Promise.all(this.#children.map((child) => child.shutdown()));}@@ -70,7 +93,8 @@ export class AgentTraceSpanProcessor implements SpanProcessor {#accepts(span: SpanLike): boolean {return (span.instrumentationScope?.name !== "workflow" &&- this.#ownedTraceIds.has(span.spanContext().traceId)+ (this.#ownedTraceIds.has(span.spanContext().traceId) ||+ this.#rememberedTraceIds.has(span.spanContext().traceId)));}}diff --git a/packages/eve/src/tracing/agent-trace-state.ts b/packages/eve/src/tracing/agent-trace-state.tsindex ba1f9742a..da53a8bcb 100644--- a/packages/eve/src/tracing/agent-trace-state.ts+++ b/packages/eve/src/tracing/agent-trace-state.ts@@ -2,10 +2,12 @@ import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";import type {InstrumentationActionKind,+ InstrumentationActionOutcome,InstrumentationParentLineage,InstrumentationTraceContext,InstrumentationTurnFailedEvent,InstrumentationTurnSettledEvent,+ InstrumentationUsage,} from "#instrumentation/lifecycle.js";import type { ChannelAudience } from "#shared/channel-audience.js";import type { InstrumentationDecision } from "#shared/instrumentation-decision.js";@@ -21,11 +23,11 @@ export interface AgentSessionTraceState {}export interface AgentTurnTraceState {+ readonly caller?: SpanContext;+ readonly channelDelivery?: AgentTurnChannelDeliveryTraceState;readonly context: SpanContext;readonly parentLineage?: InstrumentationParentLineage;readonly modelUsage?: { readonly inputTokens?: number; readonly outputTokens?: number };- readonly parentIsRemote?: boolean;- readonly parentSpanId: string;readonly rootSessionId: string;readonly sequence: number;readonly startTimeMs: number;@@ -35,6 +37,15 @@ export interface AgentTurnTraceState {| { readonly type: InstrumentationTurnSettledEvent["type"] };}+export interface AgentTurnChannelDeliveryTraceState {+ readonly channelKind: string;+ readonly channelName: string;+ readonly deliveryId: string;+ readonly inputAttribute?: string;+ readonly requestId?: string;+ readonly requestTraceContext?: SpanContext;+}+export interface AgentActionTraceState {readonly attemptIndex: number;readonly callId: string;@@ -51,16 +62,46 @@ export interface AgentActionTraceState {readonly turnId: string;}+export interface AgentInvocationTraceState extends Omit<+ AgentActionTraceState,+ "inputAttribute" | "kind"+> {+ readonly kind: "remote-agent-call" | "subagent-call";+ readonly parentActionCallId: string;+ readonly recordOutputs?: boolean;+ readonly terminal?: AgentActionTraceTerminalState;+}++export interface AgentActionTraceTerminalState {+ readonly acceptedAtMs?: number;+ readonly error?: unknown;+ readonly outcome: InstrumentationActionOutcome;+ readonly usage?: InstrumentationUsage;+}+/** Provider-owned serializable storage for durable agent trace state. */export interface AgentTraceStateStore {deleteAction(idempotencyKey: string): void | PromiseLike<void>;+ deleteActionAnchors(sessionId: string): void | PromiseLike<void>;deleteActions(sessionId: string, turnId?: string): void | PromiseLike<void>;+ deleteInvocation(idempotencyKey: string): void | PromiseLike<void>;+ deleteInvocations(sessionId: string, turnId?: string): void | PromiseLike<void>;deleteSession(sessionId: string): void | PromiseLike<void>;deleteTurn(sessionId: string, turnId: string): void | PromiseLike<void>;findAction(sessionId: string,callId: string,): AgentActionTraceState | undefined | PromiseLike<AgentActionTraceState | undefined>;+ findActionAnchor(+ sessionId: string,+ turnId: string,+ callId: string,+ ): AgentActionTraceState | undefined | PromiseLike<AgentActionTraceState | undefined>;+ findInvocations(+ sessionId?: string,+ turnId?: string,+ parentActionCallId?: string,+ ): readonly AgentInvocationTraceState[] | PromiseLike<readonly AgentInvocationTraceState[]>;getAction(idempotencyKey: string,): AgentActionTraceState | undefined | PromiseLike<AgentActionTraceState | undefined>;@@ -72,6 +113,8 @@ export interface AgentTraceStateStore {turnId: string,): AgentTurnTraceState | undefined | PromiseLike<AgentTurnTraceState | undefined>;setAction(idempotencyKey: string, state: AgentActionTraceState): void | PromiseLike<void>;+ setActionAnchor(idempotencyKey: string, state: AgentActionTraceState): void | PromiseLike<void>;+ setInvocation(idempotencyKey: string, state: AgentInvocationTraceState): void | PromiseLike<void>;setSession(sessionId: string, state: AgentSessionTraceState): void | PromiseLike<void>;setTurn(sessionId: string, turnId: string, state: AgentTurnTraceState): void | PromiseLike<void>;/** Atomically updates an existing turn and does nothing after that turn is deleted. */@@ -84,7 +127,9 @@ export interface AgentTraceStateStore {/** In-memory trace state used by tests and non-durable runtimes. */export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {+ readonly #actionAnchors = new Map<string, AgentActionTraceState>();readonly #actions = new Map<string, AgentActionTraceState>();+ readonly #invocations = new Map<string, AgentInvocationTraceState>();readonly #sessions = new Map<string, AgentSessionTraceState>();readonly #turns = new Map<string, AgentTurnTraceState>();@@ -92,6 +137,12 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {this.#actions.delete(idempotencyKey);}+ deleteActionAnchors(sessionId: string): void {+ for (const [key, state] of this.#actionAnchors) {+ if (state.sessionId === sessionId) this.#actionAnchors.delete(key);+ }+ }+deleteActions(sessionId: string, turnId?: string): void {for (const [key, state] of this.#actions) {if (state.sessionId === sessionId && (turnId === undefined || state.turnId === turnId)) {@@ -100,6 +151,18 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {}}+ deleteInvocation(idempotencyKey: string): void {+ this.#invocations.delete(idempotencyKey);+ }++ deleteInvocations(sessionId: string, turnId?: string): void {+ for (const [key, state] of this.#invocations) {+ if (state.sessionId === sessionId && (turnId === undefined || state.turnId === turnId)) {+ this.#invocations.delete(key);+ }+ }+ }+deleteSession(sessionId: string): void {this.#sessions.delete(sessionId);}@@ -114,6 +177,30 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {);}+ findActionAnchor(+ sessionId: string,+ turnId: string,+ callId: string,+ ): AgentActionTraceState | undefined {+ return [...this.#actionAnchors.values()].find(+ (state) =>+ state.sessionId === sessionId && state.turnId === turnId && state.callId === callId,+ );+ }++ findInvocations(+ sessionId?: string,+ turnId?: string,+ parentActionCallId?: string,+ ): readonly AgentInvocationTraceState[] {+ return [...this.#invocations.values()].filter(+ (state) =>+ (sessionId === undefined || state.sessionId === sessionId) &&+ (turnId === undefined || state.turnId === turnId) &&+ (parentActionCallId === undefined || state.parentActionCallId === parentActionCallId),+ );+ }+getAction(idempotencyKey: string): AgentActionTraceState | undefined {return this.#actions.get(idempotencyKey);}@@ -130,6 +217,14 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {this.#actions.set(idempotencyKey, state);}+ setActionAnchor(idempotencyKey: string, state: AgentActionTraceState): void {+ this.#actionAnchors.set(idempotencyKey, state);+ }++ setInvocation(idempotencyKey: string, state: AgentInvocationTraceState): void {+ this.#invocations.set(idempotencyKey, state);+ }+setSession(sessionId: string, state: AgentSessionTraceState): void {this.#sessions.set(sessionId, state);}diff --git a/packages/eve/src/tracing/bounded-error.ts b/packages/eve/src/tracing/bounded-error.tsnew file mode 100644index 000000000..c457c476b--- /dev/null+++ b/packages/eve/src/tracing/bounded-error.ts@@ -0,0 +1,8 @@+import { truncateTelemetryText } from "#tracing/telemetry-budget.js";++export function boundedTraceError(value: Error): Error {+ const error = new Error(truncateTelemetryText(value.message, 4096));+ error.name = truncateTelemetryText(value.name, 128);+ error.stack = value.stack === undefined ? undefined : truncateTelemetryText(value.stack, 8192);+ return error;+}diff --git a/packages/eve/src/tracing/content-span-processor.test.ts b/packages/eve/src/tracing/content-span-processor.test.tsindex 26dd66535..c4942f2ac 100644--- a/packages/eve/src/tracing/content-span-processor.test.ts+++ b/packages/eve/src/tracing/content-span-processor.test.ts@@ -319,16 +319,16 @@ describe("contentFilteringProcessor", () => {});it("preserves local trace session release through the wrapper", async () => {- const releaseSession = vi.fn(async () => true);+ const releaseConversation = vi.fn(async () => true);const downstream: SpanProcessor & {- releaseSession(sessionId: string): Promise<boolean>;- } = { ...recordingProcessor(), releaseSession };+ releaseConversation(sessionId: string): Promise<boolean>;+ } = { ...recordingProcessor(), releaseConversation };const processor = contentFilteringProcessor(downstream,composeSpanExportPolicies(redactSpanInputs(), redactSpanOutputs()),- ) as SpanProcessor & { releaseSession(sessionId: string): Promise<boolean> };+ ) as SpanProcessor & { releaseConversation(sessionId: string): Promise<boolean> };- await expect(processor.releaseSession("session-1")).resolves.toBe(true);- expect(releaseSession).toHaveBeenCalledExactlyOnceWith("session-1");+ await expect(processor.releaseConversation("session-1")).resolves.toBe(true);+ expect(releaseConversation).toHaveBeenCalledExactlyOnceWith("session-1");});});diff --git a/packages/eve/src/tracing/content-span-processor.ts b/packages/eve/src/tracing/content-span-processor.tsindex 563162c8c..bb0e207df 100644--- a/packages/eve/src/tracing/content-span-processor.ts+++ b/packages/eve/src/tracing/content-span-processor.ts@@ -4,7 +4,7 @@ import {withoutDeclinedContent,type ResolvedContentOptions,} from "#tracing/content-attributes.js";-import { hasSessionRelease, type LocalTracesProcessor } from "#tracing/local-traces.js";+import { hasConversationRelease, type LocalTracesProcessor } from "#tracing/local-traces.js";import { normalizeChannelAudience } from "#shared/channel-audience.js";import type { ChannelAudience } from "#shared/channel-audience.js";import {@@ -89,10 +89,10 @@ function policyFilteringProcessor(shutdown: () => downstream.shutdown(),};- if (!hasSessionRelease(downstream)) return filtering;+ if (!hasConversationRelease(downstream)) return filtering;const releasing: LocalTracesProcessor = {...filtering,- releaseSession: (sessionId) => downstream.releaseSession(sessionId),+ releaseConversation: (conversationId) => downstream.releaseConversation(conversationId),};return releasing;}diff --git a/packages/eve/src/tracing/error-content-context.ts b/packages/eve/src/tracing/error-content-context.tsnew file mode 100644index 000000000..d32415030--- /dev/null+++ b/packages/eve/src/tracing/error-content-context.ts@@ -0,0 +1,11 @@+import { context, createContextKey, type Context } from "#compiled/@opentelemetry/api/index.js";++const ERROR_CONTENT_KEY = createContextKey("eve.trace.error-content");++export function withErrorContent(context: Context, allowed: boolean): Context {+ return context.setValue(ERROR_CONTENT_KEY, allowed);+}++export function capturesErrorContent(): boolean {+ return context.active().getValue(ERROR_CONTENT_KEY) !== false;+}diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts b/packages/eve/src/tracing/install-instrumentation-runtime.test.tsindex 0c17a6e7e..b867f6ace 100644--- a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts+++ b/packages/eve/src/tracing/install-instrumentation-runtime.test.ts@@ -1,12 +1,14 @@import { beforeEach, describe, expect, it, vi } from "vitest";import { ContextContainer, contextStorage } from "#context/container.js";+import { ConversationIdKey } from "#context/keys.js";import { sessionIdempotencyKey, turnIdempotencyKey } from "#instrumentation/lifecycle.js";import { installInstrumentationRuntime } from "#tracing/install-instrumentation-runtime.js";import { otelIntegration, collectOtelPipeline } from "#tracing/otel-declaration.js";-const { forceFlush, internalTerminalState, shutdown } = vi.hoisted(() => ({+const { forceFlush, invocationFlush, internalTerminalState, shutdown } = vi.hoisted(() => ({forceFlush: vi.fn(async () => undefined),+ invocationFlush: vi.fn(async () => undefined),internalTerminalState: vi.fn(),shutdown: vi.fn(async () => undefined),}));@@ -29,6 +31,7 @@ vi.mock("#tracing/otel-registration.js", async (importOriginal) => {vi.mock("#tracing/agent-otel-provider.js", () => ({createAgentOtelInstrumentation: () => ({hook: {+ flush: invocationFlush,events: {"turn.completed": (_event: unknown, ctx: { state: { get(): unknown } }) => {internalTerminalState(ctx.state.get());@@ -48,6 +51,7 @@ const RUNTIME_GLOBAL_KEY = Symbol.for("eve.instrumentation-runtime");describe("installInstrumentationRuntime", () => {beforeEach(() => {forceFlush.mockClear();+ invocationFlush.mockClear();internalTerminalState.mockClear();shutdown.mockClear();delete (globalThis as Record<symbol, unknown>)[RUNTIME_GLOBAL_KEY];@@ -82,15 +86,36 @@ describe("installInstrumentationRuntime", () => {expect(providerShutdown).toHaveBeenCalledOnce();});- it.each(["session.completed", "session.failed"] as const)(- "releases local trace liveness after %s",- async (type) => {- const releaseSession = vi.fn(async () => true);+ it("materializes settled invocations without draining authored providers or the exporter", async () => {+ const providerFlush = vi.fn();+ const runtime = installInstrumentationRuntime({+ collected: collectOtelPipeline([otelIntegration()]),+ frameworkVersion: "test",+ providers: [{ flush: providerFlush, name: "test" }],+ serviceName: "weather",+ });++ await runtime.flushSettledInvocations!();++ expect(invocationFlush).toHaveBeenCalledOnce();+ expect(providerFlush).not.toHaveBeenCalled();+ expect(forceFlush).not.toHaveBeenCalled();+ });++ it.each([+ ["session.completed", "conversation-1", true],+ ["session.failed", "conversation-1", true],+ ["session.completed", "child-1", false],+ ["session.failed", "child-1", false],+ ] as const)(+ "releases conversation traces after %s for %s",+ async (type, sessionId, expectedRelease) => {+ const releaseConversation = vi.fn(async () => true);const processor = {forceFlush: vi.fn(async () => undefined),onEnd: vi.fn(),onStart: vi.fn(),- releaseSession,+ releaseConversation,shutdown: vi.fn(async () => undefined),};const runtime = installInstrumentationRuntime({@@ -101,15 +126,17 @@ describe("installInstrumentationRuntime", () => {});const hooks = runtime.hooks.forTrace!({ agentName: "weather", audience: "unknown" });const event = {- idempotencyKey: sessionIdempotencyKey("session-1"),- sessionId: "session-1",+ idempotencyKey: sessionIdempotencyKey(sessionId),+ sessionId,type,...(type === "session.failed" ? { error: new Error("failed") } : undefined),};- await contextStorage.run(new ContextContainer(), () => hooks.publish(event));+ const context = new ContextContainer();+ context.set(ConversationIdKey, "conversation-1");+ await contextStorage.run(context, () => hooks.publish(event));- expect(releaseSession).toHaveBeenCalledExactlyOnceWith("session-1");+ expect(releaseConversation.mock.calls).toEqual(expectedRelease ? [["conversation-1"]] : []);},);diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.ts b/packages/eve/src/tracing/install-instrumentation-runtime.tsindex 4be229db1..30aee239a 100644--- a/packages/eve/src/tracing/install-instrumentation-runtime.ts+++ b/packages/eve/src/tracing/install-instrumentation-runtime.ts@@ -1,6 +1,8 @@import { trace } from "#compiled/@opentelemetry/api/index.js";import type { SpanProcessor } from "#compiled/@vercel/otel/index.js";+import { contextStorage } from "#context/container.js";+import { ConversationIdKey } from "#context/keys.js";import {createInstrumentationHooks,type InstrumentationProviderDefinition,@@ -13,9 +15,10 @@ import { createLogger, formatError } from "#internal/logging.js";import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import { ContextAgentTraceStateStore } from "#tracing/agent-trace-context-store.js";import { createAgentOtelInstrumentation } from "#tracing/agent-otel-provider.js";-import { hasSessionRelease, type LocalTracesProcessor } from "#tracing/local-traces.js";+import { hasConversationRelease, type LocalTracesProcessor } from "#tracing/local-traces.js";import type { CollectedOtel, RuntimeContextResolver } from "#tracing/otel-declaration.js";import { registerOtelPipeline, type RegisteredOtelPipeline } from "#tracing/otel-registration.js";+import { readConversationId } from "#tracing/conversation-context.js";const log = createLogger("tracing.install-instrumentation-runtime");@@ -40,6 +43,7 @@ export function installInstrumentationRuntime(input: {const serialBefore: InstrumentationProviderDefinition[] = [];const serialAfter: InstrumentationProviderDefinition[] = [];let otelRuntime: RegisteredOtelPipeline | undefined;+ let flushSettledInvocations: InstrumentationRuntime["flushSettledInvocations"];let prepareSessionTrace: InstrumentationRuntime["prepareSessionTrace"];let prepareTurnTrace: InstrumentationRuntime["prepareTurnTrace"];let runInContext: InstrumentationRuntime["runInContext"] = (_operation, execute) => execute();@@ -54,30 +58,34 @@ export function installInstrumentationRuntime(input: {idGenerator: otelRuntime.idGenerator,recordInputs: input.collected.settings.recordInputs,recordOutputs: input.collected.settings.recordOutputs,+ samplesTrace: otelRuntime.samplesTrace,stateStore: new ContextAgentTraceStateStore(),tracer: trace.getTracer("eve.agent", input.frameworkVersion),tracePolicy: input.collected.settings.tracePolicy,});// The span must exist before authored providers observe the lifecycle event.serialBefore.push({ ...agentOtel.hook, stateNamespace: "internal:otel" });+ flushSettledInvocations = async () => {+ await agentOtel.hook.flush?.();+ };prepareSessionTrace = agentOtel.prepareSessionTrace;prepareTurnTrace = agentOtel.prepareTurnTrace;runInContext = agentOtel.runInContext;const releasable = input.collected.pipeline.spanProcessors.filter(isSpanProcessor)- .filter(hasSessionRelease);+ .filter(hasConversationRelease);if (releasable.length > 0) serialAfter.push(sessionReleaseProvider(releasable));}const allProviders = [...serialBefore, ...input.providers, ...serialAfter];let shutdown: Promise<void> | undefined;return registerInstrumentationRuntime({- forceFlush: () =>- settleAll([- ...(otelRuntime === undefined ? [] : [otelRuntime.forceFlush]),- ...allProviders.map((provider) => () => provider.flush?.()),- ]),+ flushSettledInvocations,+ forceFlush: async () => {+ await settleAll(allProviders.map((provider) => () => provider.flush?.()));+ await settleAll(otelRuntime === undefined ? [] : [otelRuntime.forceFlush]);+ },hooks: createInstrumentationHooks({parallel: input.providers,serialAfter,@@ -93,10 +101,10 @@ export function installInstrumentationRuntime(input: {runInContext,samplesTrace: otelRuntime?.samplesTrace,shutdown: () => {- shutdown ??= settleAll([- ...(otelRuntime === undefined ? [] : [otelRuntime.shutdown]),- ...allProviders.map((provider) => () => provider.shutdown?.()),- ]);+ shutdown ??= (async () => {+ await settleAll(allProviders.map((provider) => () => provider.shutdown?.()));+ await settleAll(otelRuntime === undefined ? [] : [otelRuntime.shutdown]);+ })();return shutdown;},});@@ -110,12 +118,15 @@ function sessionReleaseProvider(processors: readonly LocalTracesProcessor[],): InstrumentationProviderDefinition {const release = async (event: { readonly sessionId: string }): Promise<void> => {- await Promise.all(processors.map((processor) => processor.releaseSession(event.sessionId)));+ const conversationId =+ readConversationId(contextStorage.getStore()?.get(ConversationIdKey)) ?? event.sessionId;+ if (conversationId !== event.sessionId) return;+ await Promise.all(processors.map((processor) => processor.releaseConversation(conversationId)));};return {events: { "session.completed": release, "session.failed": release },- name: "eve.session-release",- stateNamespace: "internal:session-release",+ name: "eve.conversation-release",+ stateNamespace: "internal:conversation-release",};}diff --git a/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts b/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.tsindex 0457fcdc5..f3e0f8dd5 100644--- a/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts+++ b/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts@@ -197,14 +197,13 @@ describe("local instrumentation runtime", () => {);const spans = spanGroups.flat();expect(formatTraceTree(spans)).toEqual([- "agent.session",- " invoke_agent weather",- " agent.step",- " agent.action",- " execute_tool weather",- " user.tool-work",- " chat model-1",- " user.model-work",+ "invoke_agent weather",+ " agent.step",+ " agent.action",+ " execute_tool weather",+ " user.tool-work",+ " chat model-1",+ " user.model-work",]);for (const exported of spans.filter((span) => !span.name.startsWith("user."))) {expect(exported.attributes).toEqual(@@ -239,7 +238,7 @@ describe("local instrumentation runtime", () => {]);const listed = await listLocalTraces(appRoot);expect(listed).toHaveLength(1);- expect(listed[0]).toMatchObject({ sessionId: "session-1", traceId });+ expect(listed[0]).toMatchObject({ conversationId: "session-1", traceId });});it("keeps segments from overlapping worker writers", async () => {diff --git a/packages/eve/src/tracing/local-trace-reader.ts b/packages/eve/src/tracing/local-trace-reader.tsindex bd5fe983e..1c2883fea 100644--- a/packages/eve/src/tracing/local-trace-reader.ts+++ b/packages/eve/src/tracing/local-trace-reader.ts@@ -15,6 +15,7 @@ import {resolveLocalTraceSchemaDirectory,resolveLocalTraceSegmentsDirectory,} from "#tracing/local-trace-span-processor.js";+import { isAgentActivationSpan } from "#tracing/agent-span-contract.js";const TRACE_ID_PATTERN = /^[0-9a-f]{32}$/u;const SPAN_FILE_PATTERN = /^[0-9a-f]{16}\.otlp\.json$/u;@@ -47,15 +48,15 @@ export interface LocalTraceSpan {export interface LocalTrace {readonly agentName?: string;readonly endTimeNs: bigint;- /** Session that opened the trace, which is the one the list shows. */- readonly sessionId?: string;+ /** Conversation that owns the trace, which is the one the list shows. */+ readonly conversationId?: string;/**- * Every session recorded in the trace, opener first.+ * Every conversation identifier recorded in the trace.*- * A subagent child records into the trace its parent opened, so one- * trace can hold several sessions and any of their ids resolves to it.+ * A logical conversation can contain separate root traces for the parent+ * and child activations. Any matching conversation ID resolves to them.*/- readonly sessionIds: readonly string[];+ readonly conversationIds: readonly string[];readonly spans: readonly LocalTraceSpan[];readonly startTimeNs: bigint;readonly traceId: string;@@ -183,15 +184,15 @@ export function compareLocalTraceSpans(left: LocalTraceSpan, right: LocalTraceSpexport function assembleLocalTrace(traceId: string, spans: readonly LocalTraceSpan[]): LocalTrace {const ordered = [...spans].sort(compareLocalTraceSpans);const attributes = ordered.map((span) => span.attributes);- const sessionIds = distinctAttributes(attributes, "agent.session.id");+ const conversationIds = distinctAttributes(attributes, "gen_ai.conversation.id");return {agentName: firstAttribute(attributes, "agent.name"),endTimeNs: ordered.reduce((value, span) => (span.endTimeNs > value ? span.endTimeNs : value),0n,),- sessionId: sessionIds[0],- sessionIds,+ conversationId: conversationIds[0],+ conversationIds,spans: ordered,startTimeNs: ordered.reduce((value, span) => (span.startTimeNs < value ? span.startTimeNs : value),@@ -234,11 +235,7 @@ export function describeLocalTraceSpan(span: LocalTraceSpan): string[] {}export function isAgentTurnSpan(span: LocalTraceSpan): boolean {- return (- span.name === "agent.turn" ||- (stringSpanAttribute(span, "gen_ai.operation.name") === "invoke_agent" &&- stringSpanAttribute(span, "agent.turn.id") !== undefined)- );+ return isAgentActivationSpan(span);}/** Parses one OTLP/JSON segment file into spans belonging to `expectedTraceId`. */diff --git a/packages/eve/src/tracing/local-traces.test.ts b/packages/eve/src/tracing/local-traces.test.tsindex 22f7e3cce..562b094c3 100644--- a/packages/eve/src/tracing/local-traces.test.ts+++ b/packages/eve/src/tracing/local-traces.test.ts@@ -25,7 +25,7 @@ vi.mock("#tracing/local-trace-retention.js", () => ({function agentSpan(sessionId: string, traceId: string): unknown {return {- attributes: { "agent.session.id": sessionId },+ attributes: { "gen_ai.conversation.id": sessionId },spanContext: () => ({ traceId }),};}@@ -40,10 +40,10 @@ describe("createLocalTracesProcessor", () => {spool.onStart(agentSpan("session-one", "a".repeat(32)), undefined);// A subagent child owns none, so releasing it leaves the trace pinned.- await expect(spool.releaseSession("child-one")).resolves.toBe(false);- await expect(spool.releaseSession("session-one")).resolves.toBe(true);+ await expect(spool.releaseConversation("child-one")).resolves.toBe(false);+ await expect(spool.releaseConversation("session-one")).resolves.toBe(true);// Releasing twice is not an error, it just owns nothing the second time.- await expect(spool.releaseSession("session-one")).resolves.toBe(false);+ await expect(spool.releaseConversation("session-one")).resolves.toBe(false);});it("is a span processor, so it composes wherever one goes", () => {diff --git a/packages/eve/src/tracing/local-traces.ts b/packages/eve/src/tracing/local-traces.tsindex b9310bb93..c0465adfa 100644--- a/packages/eve/src/tracing/local-traces.ts+++ b/packages/eve/src/tracing/local-traces.ts@@ -16,24 +16,24 @@ import {*/export interface LocalTracesProcessor extends SpanProcessor {/**- * Settles pending writes and drops one root session's liveness, then bounds- * the store. A subagent child owns no traces, so releasing one is a no-op- * and leaves the shared trace pinned until its root finishes.+ * Settles pending writes and releases this conversation's active traces.*/- releaseSession(sessionId: string): Promise<boolean>;+ releaseConversation(conversationId: string): Promise<boolean>;}/**- * Reports whether a processor tracks which session owns which trace, so eve can- * tell it when that session is done.+ * Reports whether a processor tracks which conversation owns which trace, so+ * eve can tell it when that conversation is done.** Anything standing between eve and the spool has to answer for the spool, so* this is the check a wrapper uses to decide whether it must forward the call.** @internal*/-export function hasSessionRelease(processor: SpanProcessor): processor is LocalTracesProcessor {- return typeof (processor as Partial<LocalTracesProcessor>).releaseSession === "function";+export function hasConversationRelease(+ processor: SpanProcessor,+): processor is LocalTracesProcessor {+ return typeof (processor as Partial<LocalTracesProcessor>).releaseConversation === "function";}/**@@ -42,7 +42,7 @@ export function hasSessionRelease(processor: SpanProcessor): processor is LocalT* `EVE_TRACES=off` removes the writer but keeps the processor: eve still has* to observe spans to track which session owns which trace.*- * Internal because of `releaseSession`, which eve's runtime drives off session+ * Internal because of `releaseConversation`, which eve's runtime drives off session* lifecycle. The authored surface is `localTraces()`, which wraps this in an* `OtelIntegration`.*/@@ -72,16 +72,20 @@ export function createLocalTracesProcessor(// before this worker adds to it.requestPrune();+ const forceFlush = async (): Promise<void> => {+ await processor.forceFlush();+ if (processor.releaseCompletedTraces()) requestPrune();+ };return {- forceFlush: () => processor.forceFlush(),+ forceFlush,onEnd: (span) => processor.onEnd(span),onStart: (span, parentContext) => processor.onStart(span, parentContext),- async releaseSession(sessionId) {+ async releaseConversation(conversationId) {// Settle pending segment writes before dropping liveness: a sweep already// running reads the same live set, so releasing first would expose the// trace to eviction while it is still being written.- await processor.forceFlush();- if (!processor.releaseSession(sessionId)) return false;+ await forceFlush();+ if (!processor.releaseConversation(conversationId)) return false;requestPrune();return true;},@@ -111,7 +115,7 @@ function inertLocalTracesProcessor(): LocalTracesProcessor {forceFlush: async () => undefined,onEnd: () => undefined,onStart: () => undefined,- releaseSession: async () => false,+ releaseConversation: async () => false,shutdown: async () => undefined,};}diff --git a/packages/eve/src/tracing/otel-declaration.ts b/packages/eve/src/tracing/otel-declaration.tsindex cd778464b..5089c67b2 100644--- a/packages/eve/src/tracing/otel-declaration.ts+++ b/packages/eve/src/tracing/otel-declaration.ts@@ -60,9 +60,11 @@ export interface OtelOptions {*/readonly functionId?: string;/**- * Whether to emit the inbound HTTP `SERVER` span that wraps each channel- * request — the parent of the turn trace and of any `hook.resume` or- * outgoing HTTP spans. Defaults to `false`.+ * Whether to emit an eve-owned HTTP `SERVER` span around each channel+ * request. For a one-to-one delivery, the activation remains a separate+ * trace root and links to this span; when disabled, it links to any+ * already-active upstream request or function span instead. Defaults to+ * `false`.*/readonly traceChannelRequests?: boolean;/**diff --git a/packages/eve/src/tracing/otel-registration.scenario.test.ts b/packages/eve/src/tracing/otel-registration.scenario.test.tsindex 735be1b45..dcae7e81e 100644--- a/packages/eve/src/tracing/otel-registration.scenario.test.ts+++ b/packages/eve/src/tracing/otel-registration.scenario.test.ts@@ -15,11 +15,15 @@ import {trace as runtimeTrace,} from "#compiled/@opentelemetry/api/index.js";import { registerOtelPipeline } from "#tracing/otel-registration.js";+import { withErrorContent } from "#tracing/error-content-context.js";+import { createLogger, logError } from "#internal/logging.js";const require = createRequire(import.meta.url);const authoredApi = require("@opentelemetry/api") as typeof import("@opentelemetry/api");afterEach(() => {+ vi.unstubAllEnvs();+ vi.restoreAllMocks();authoredApi.context.disable();authoredApi.metrics.disable();authoredApi.propagation.disable();@@ -33,6 +37,85 @@ afterEach(() => {});describe("registerOtelPipeline", () => {+ it("samples using the real activation name and attributes without exporting probes", async () => {+ const exporter = new InMemorySpanExporter();+ const sampler = {+ shouldSample: (+ _context: unknown,+ _traceId: string,+ name: string,+ _kind: unknown,+ attributes: Record<string, unknown>,+ ) => ({+ decision:+ name === "invoke_agent researcher" && attributes["gen_ai.conversation.id"] === "session-1"+ ? 2+ : 0,+ }),+ toString: () => "activation-sampler",+ };+ const runtime = registerOtelPipeline({+ pipeline: { sampler, spanProcessors: [new SimpleSpanProcessor(exporter)] },+ serviceName: "researcher",+ });+ const operation = {+ name: "invoke_agent researcher",+ attributes: { "gen_ai.conversation.id": "session-1" },+ };+ expect(runtime.samplesTrace("a".repeat(32), operation)).toBe(true);+ expect(+ runtime.samplesTrace("b".repeat(32), {+ ...operation,+ attributes: { "gen_ai.conversation.id": "other" },+ }),+ ).toBe(false);+ runtimeTrace+ .getTracer("eve.agent")+ .startSpan(operation.name, { attributes: operation.attributes, root: true })+ .end();+ await runtime.forceFlush();+ expect(exporter.getFinishedSpans().map((span) => span.name)).toEqual([operation.name]);+ await runtime.shutdown();+ });++ it.each(["traceidratio", "parentbased_traceidratio"] as const)(+ "honors the configured %s root ratio",+ async (sampler) => {+ vi.stubEnv("OTEL_TRACES_SAMPLER_ARG", "0");+ const runtime = registerOtelPipeline({+ pipeline: { sampler, spanProcessors: [] },+ serviceName: "test",+ });+ expect(runtime.samplesTrace("a".repeat(32), { name: "invoke_agent test" })).toBe(false);+ await runtime.shutdown();+ },+ );++ it("keeps logger-to-span error details behind the active output policy", async () => {+ vi.spyOn(console, "error").mockImplementation(() => {});+ const exporter = new InMemorySpanExporter();+ const runtime = registerOtelPipeline({+ pipeline: { spanProcessors: [new SimpleSpanProcessor(exporter)] },+ serviceName: "test",+ });+ for (const allowed of [false, true]) {+ const span = runtimeTrace.getTracer("eve.agent").startSpan(`logger-${allowed}`);+ const active = withErrorContent(runtimeTrace.setSpan(COMPILED_ROOT_CONTEXT, span), allowed);+ await runtimeContext.with(active, async () => {+ logError(createLogger("test"), "operation failed", new Error("sensitive payload"));+ });+ span.end();+ }+ await runtime.forceFlush();+ const [privateSpan, publicSpan] = exporter.getFinishedSpans();+ expect(privateSpan?.status.code).toBe(2);+ expect(privateSpan?.status.message).toBeUndefined();+ expect(privateSpan?.events).toEqual([]);+ expect(publicSpan?.status.message).toContain("sensitive payload");+ expect(publicSpan?.events[0]?.attributes?.["exception.message"]).toContain("sensitive payload");+ await runtime.shutdown();+ });+it("delegates an authored tracer cached before registration", async () => {const authoredTracer = authoredApi.trace.getTracer("authored");expect(authoredTracer.startSpan("before-registration").isRecording()).toBe(false);diff --git a/packages/eve/src/tracing/otel-registration.test.ts b/packages/eve/src/tracing/otel-registration.test.tsindex b786dfb74..22364f68f 100644--- a/packages/eve/src/tracing/otel-registration.test.ts+++ b/packages/eve/src/tracing/otel-registration.test.ts@@ -1,4 +1,4 @@-import { describe, expect, it, vi } from "vitest";+import { beforeEach, describe, expect, it, vi } from "vitest";import { registerOtelPipeline } from "#tracing/otel-registration.js";@@ -7,6 +7,10 @@ const { registerOTel } = vi.hoisted(() => ({ registerOTel: vi.fn() }));vi.mock("#compiled/@vercel/otel/index.js", () => ({ registerOTel }));describe("registerOtelPipeline", () => {+ beforeEach(() => {+ delete (globalThis as Record<symbol, unknown>)[Symbol.for("eve.otel.replay-deduplication")];+ });+it("maps eve's option names onto the ones @vercel/otel accepts", () => {registerOTel.mockImplementation(() => undefined);@@ -92,8 +96,14 @@ describe("registerOtelPipeline", () => {expect(configuration.spanProcessors[0]).toBe("auto");});- it("exports only the first terminal for a replay-stable span", () => {- const downstream = {+ it("exports a unique span to every concrete destination", () => {+ const first = {+ forceFlush: vi.fn(async () => {}),+ onEnd: vi.fn(),+ onStart: vi.fn(),+ shutdown: vi.fn(async () => {}),+ };+ const second = {forceFlush: vi.fn(async () => {}),onEnd: vi.fn(),onStart: vi.fn(),@@ -103,22 +113,34 @@ describe("registerOtelPipeline", () => {expect(() =>registerOtelPipeline({- pipeline: { spanProcessors: [downstream] },+ pipeline: { spanProcessors: [first, second] },serviceName: "weather",}),).toThrow();- const processor = (- registerOTel.mock.calls.at(-1)![0] as {- spanProcessors: { onEnd(span: unknown): void }[];- }- ).spanProcessors[0]!;- const first = replaySpan("first");+ const configuration = registerOTel.mock.calls.at(-1)![0] as {+ spanProcessors: { onEnd(span: unknown): void }[];+ };+ const processor = configuration.spanProcessors[0]!;+ const span = replaySpan("first");+ processor.onEnd(span);++ expect(configuration.spanProcessors).toHaveLength(1);+ expect(first.onEnd).toHaveBeenCalledExactlyOnceWith(span);+ expect(second.onEnd).toHaveBeenCalledExactlyOnceWith(span);+ });++ it("exports only the first terminal across replayed runtime registrations", () => {+ const first = filteringProcessor();+ const second = filteringProcessor();+ const winner = replaySpan("first");const losingReplay = replaySpan("losing replay");- processor.onEnd(first);- processor.onEnd(losingReplay);- expect(downstream.onEnd).toHaveBeenCalledExactlyOnceWith(first);+ first.processor.onEnd(winner);+ second.processor.onEnd(losingReplay);++ expect(first.downstream.onEnd).toHaveBeenCalledExactlyOnceWith(winner);+ expect(second.downstream.onEnd).not.toHaveBeenCalled();});it("holds a physical child until its started parent completes", () => {diff --git a/packages/eve/src/tracing/otel-registration.ts b/packages/eve/src/tracing/otel-registration.tsindex 3c8cc811b..a2346acd7 100644--- a/packages/eve/src/tracing/otel-registration.ts+++ b/packages/eve/src/tracing/otel-registration.ts@@ -5,6 +5,7 @@ import {metrics,propagation,trace,+ SpanKind,type Context,} from "#compiled/@opentelemetry/api/index.js";import {@@ -16,12 +17,23 @@ import {import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";import type { OtelPipeline } from "#tracing/otel-declaration.js";+import {+ agentInvocationSpanName,+ type AgentSamplingOperation,+} from "#tracing/agent-span-contract.js";const REGISTRATION_SPAN_NAME = "eve.otel.registration";const REPLAY_DEDUPLICATION_LIMIT = 100_000;const PENDING_CHILD_SPAN_LIMIT = 10_000;+const REPLAY_DEDUPLICATION_KEY = Symbol.for("eve.otel.replay-deduplication");const require = createRequire(import.meta.url);+interface ReplayDeduplicationGlobal {+ [REPLAY_DEDUPLICATION_KEY]?: Set<string>;+}++const replayDeduplicationGlobal = globalThis as typeof globalThis & ReplayDeduplicationGlobal;+class RegistrationMarkerPropagator {#injected = false;@@ -46,15 +58,16 @@ class RegistrationMarkerPropagator {/** Keeps eve's ownership check out of every authored destination. */class PrivateSpanFilteringProcessor implements SpanProcessor {- private readonly endedSpans = new Set<string>();+ private readonly endedSpans: Set<string>;private readonly forwardedSpans = new Set<string>();private readonly pendingByParent = new Map<string, unknown[]>();private pendingSpanCount = 0;private readonly processors: readonly SpanProcessor[];private readonly startedSpans = new Set<string>();- constructor(processors: readonly SpanProcessor[]) {+ constructor(processors: readonly SpanProcessor[], endedSpans: Set<string>) {this.processors = processors;+ this.endedSpans = endedSpans;}async forceFlush(): Promise<void> {@@ -147,6 +160,7 @@ export function registerOtelPipeline(input: {const optionalPeerTracerProxy = captureOptionalPeerTracerProxy();const idGenerator = new AgentSpanIdGenerator();const markerPropagator = new RegistrationMarkerPropagator();+ const spanProcessors = privateSpanProcessors(pipeline.spanProcessors);const configuration: Configuration = {attributes: pipeline.resource,autoDetectResources: false,@@ -155,9 +169,7 @@ export function registerOtelPipeline(input: {metricReaders: pipeline.metricReaders,propagators: [...(pipeline.propagators ?? ["auto"]), markerPropagator],serviceName: input.serviceName,- spanProcessors: pipeline.spanProcessors.map((processor) =>- isSpanProcessor(processor) ? new PrivateSpanFilteringProcessor([processor]) : processor,- ),+ spanProcessors,};registerOTel(// Absent means "let `@vercel/otel` decide", which is not the same as@@ -198,7 +210,7 @@ export function registerOtelPipeline(input: {await Promise.all([provider.forceFlush!(), meterProvider.forceFlush?.()]);},idGenerator,- samplesTrace: (traceId) => samplerAdmitsTrace(idGenerator, traceId),+ samplesTrace: (traceId, operation) => samplerAdmitsTrace(idGenerator, traceId, operation),shutdown: async () => {// Stop auto-instrumentations first so nothing records into providers// that are about to shut down.@@ -208,18 +220,52 @@ export function registerOtelPipeline(input: {};}+function privateSpanProcessors(processors: readonly SpanProcessorOrName[]): SpanProcessorOrName[] {+ const concrete = processors.filter(isSpanProcessor);+ if (concrete.length === 0) return [...processors];+ const filtering = new PrivateSpanFilteringProcessor(concrete, replayDeduplicationRegistry());+ const result: SpanProcessorOrName[] = [];+ let inserted = false;+ for (const processor of processors) {+ if (!isSpanProcessor(processor)) {+ result.push(processor);+ continue;+ }+ if (inserted) continue;+ inserted = true;+ result.push(filtering);+ }+ return result;+}++function replayDeduplicationRegistry(): Set<string> {+ const existing = replayDeduplicationGlobal[REPLAY_DEDUPLICATION_KEY];+ if (existing !== undefined) return existing;+ const created = new Set<string>();+ replayDeduplicationGlobal[REPLAY_DEDUPLICATION_KEY] = created;+ return created;+}+/** Lifecycle retained from the providers that own every destination. */export interface RegisteredOtelPipeline {readonly forceFlush: () => Promise<void>;readonly idGenerator: AgentSpanIdGenerator;/** Whether the installed sampler would record a trace with this id. */- readonly samplesTrace: (traceId: string) => boolean;+ readonly samplesTrace: (traceId: string, operation?: AgentSamplingOperation) => boolean;readonly shutdown: () => Promise<void>;}-function samplerAdmitsTrace(idGenerator: AgentSpanIdGenerator, traceId: string): boolean {+function samplerAdmitsTrace(+ idGenerator: AgentSpanIdGenerator,+ traceId: string,+ operation: AgentSamplingOperation = { name: agentInvocationSpanName(undefined) },+): boolean {const probe = idGenerator.withTraceId(traceId, () =>- trace.getTracer("eve.registration").startSpan(REGISTRATION_SPAN_NAME, { root: true }),+ trace.getTracer("eve.registration").startSpan(operation.name, {+ attributes: operation.attributes,+ kind: SpanKind.INTERNAL,+ root: true,+ }),);return (probe.spanContext().traceFlags & 1) === 1;}@@ -315,8 +361,9 @@ function isRegistrationSpan(span: unknown): boolean {return (typeof span === "object" &&span !== null &&- "name" in span &&- span.name === REGISTRATION_SPAN_NAME+ (("name" in span && span.name === REGISTRATION_SPAN_NAME) ||+ ("instrumentationScope" in span &&+ (span.instrumentationScope as { name?: string } | undefined)?.name === "eve.registration")));}diff --git a/packages/eve/src/tracing/telemetry-budget.test.ts b/packages/eve/src/tracing/telemetry-budget.test.tsnew file mode 100644index 000000000..9e4232b9b--- /dev/null+++ b/packages/eve/src/tracing/telemetry-budget.test.ts@@ -0,0 +1,30 @@+import { describe, expect, it } from "vitest";+import { telemetryByteLength, truncateTelemetryText } from "#tracing/telemetry-budget.js";++describe("trace text bounds", () => {+ it("counts UTF-8 bytes without splitting characters", () => {+ const text = String.fromCodePoint(0x1f600).repeat(1000);+ const truncated = truncateTelemetryText(text, 100);+ expect(telemetryByteLength(truncated)).toBeLessThanOrEqual(100);+ expect(truncated).not.toContain("\ufffd");+ });++ it("keeps surrogate pairs intact at every prefix and byte boundary", () => {+ const text = `ab${String.fromCodePoint(0x1f600)}cd`.repeat(20);+ for (let maxBytes = 0; maxBytes <= telemetryByteLength(text) + 1; maxBytes++) {+ const truncated = truncateTelemetryText(text, maxBytes);+ expect(telemetryByteLength(truncated)).toBeLessThanOrEqual(maxBytes);+ expect(truncated).not.toContain("\ufffd");+ }+ });++ it.each([16, 17, 32, 4096])(+ "does not emit a surrogate split at the %i-byte prefix",+ (maxBytes) => {+ const text = "a".repeat(maxBytes) + String.fromCodePoint(0x1f600) + "tail";+ expect(truncateTelemetryText(text, maxBytes)).toBe(+ "a".repeat(maxBytes - "... [truncated]".length) + "... [truncated]",+ );+ },+ );+});diff --git a/packages/eve/src/tracing/telemetry-budget.ts b/packages/eve/src/tracing/telemetry-budget.tsnew file mode 100644index 000000000..9256c0005--- /dev/null+++ b/packages/eve/src/tracing/telemetry-budget.ts@@ -0,0 +1,17 @@+const encoder = new TextEncoder();+const decoder = new TextDecoder();+const MARKER = "... [truncated]";++export function telemetryByteLength(text: string): number {+ return encoder.encode(text).length;+}++export function truncateTelemetryText(text: string, maxBytes: number): string {+ const prefix = text.slice(0, maxBytes + 1);+ const bytes = encoder.encode(prefix);+ if (prefix.length === text.length && bytes.length <= maxBytes) return text;+ const marker = MARKER.slice(0, maxBytes);+ let end = Math.max(0, maxBytes - marker.length);+ while (end > 0 && (bytes[end]! & 0xc0) === 0x80) end -= 1;+ return decoder.decode(bytes.subarray(0, end)) + marker;+}diff --git a/packages/eve/test/tui-client/tui-traces.ts b/packages/eve/test/tui-client/tui-traces.tsindex bb4d815bf..72d666312 100644--- a/packages/eve/test/tui-client/tui-traces.ts+++ b/packages/eve/test/tui-client/tui-traces.ts@@ -72,7 +72,7 @@ void (async () => {start: 900,end: 900,attributes: {- "agent.session.id": "session-smoke",+ "gen_ai.conversation.id": "session-smoke","agent.name": "smoke-agent",},});@@ -83,7 +83,7 @@ void (async () => {end: 1_000,parentSpanId: sessionRoot,attributes: {- "agent.session.id": "session-smoke",+ "gen_ai.conversation.id": "session-smoke","agent.name": "smoke-agent","agent.turn.id": "turn_0",},@@ -155,7 +155,7 @@ void (async () => {end: 7_000,parentSpanId: subagentAction,attributes: {- "agent.session.id": "child-session",+ "gen_ai.conversation.id": "child-session","agent.turn.id": "turn_child",},});@@ -194,7 +194,7 @@ void (async () => {name: "agent.turn",start: 500,end: 900,- attributes: { "agent.session.id": "older-session" },+ attributes: { "gen_ai.conversation.id": "older-session" },});// The viewer lists traces by segments-dir mtime (last span activity);// same-millisecond writes would race, so pin the ordering explicitly.