Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(eve): nest delegated invocations in caller traces (#4106)

Chad Hietala9553b511ceb0
diff --git a/.changeset/nested-agent-traces.md b/.changeset/nested-agent-traces.md
new file mode 100644
index 000000000..540583682
--- /dev/null
+++ b/.changeset/nested-agent-traces.md
@@ -0,0 +1,5 @@
+---
+"eve": minor
+---
+
+The first local subagent turn now uses its caller's trace; remote turns remain separate roots with dispatch links, and later child turns start new traces. Queries that followed local `eve.link.type=agent.dispatch` links must use `parentSpanId` or `agent.parent_run.id` and `agent.parent_call.id`; third-party spans receive the conversation ID only when they do not set one.
diff --git a/docs/guides/instrumentation/otel.mdx b/docs/guides/instrumentation/otel.mdx
index 9a4f3a3e3..90fa5e10b 100644
--- a/docs/guides/instrumentation/otel.mdx
+++ b/docs/guides/instrumentation/otel.mdx
@@ -139,12 +139,30 @@ invoke_agent <agent>
└── execute_tool <tool>
```
-Each turn starts a new trace. The first subagent trace links to its caller with
-`eve.link.type=agent.dispatch`; the link is not an authorization grant. For a
-remote agent, `trustedForwarders` lets the receiver accept the sender's session
-lineage and trace-content policy. See [Remote agents](../guides/remote-agents#preserving-trace-content).
-Use `gen_ai.conversation.id` to find the traces for one conversation in your
-destination's span-search surface.
+Each root agent turn starts a new trace. If eve receives a local caller's trace
+context, the first local subagent turn uses that trace. Its span is a
+child of the dispatch span and has a different span ID. Without caller trace
+context, the first subagent turn starts a new trace. Later turns in the child
+session start new traces. They do not use the original dispatch span as their
+parent.
+
+A remote child starts a separate trace with a root `invoke_agent` span. Its first
+turn links to the caller with `eve.link.type=agent.dispatch`. The receiving
+deployment applies its sampler. Queries for local dispatches must use
+`parentSpanId` or `agent.parent_run.id` and `agent.parent_call.id` instead of a
+dispatch link.
+
+Use `gen_ai.conversation.id` to find the root session and its child sessions
+across traces. eve adds this ID to spans created during model, tool, and memory
+execution. This includes spans from third-party instrumentation. `agent.run.id`
+identifies one agent session, not the full conversation.
+
+The shared conversation ID does not give access to message history, agent state,
+or credentials. It does not give authorization or require the same worker for
+each turn. Remote requests carry the conversation ID in W3C Baggage.
+`trustedForwarders` controls whether the receiver accepts parent session data
+and the trace-content policy. See
+[Remote agents](../guides/remote-agents#preserving-trace-content).
## Local traces
diff --git a/docs/guides/remote-agents.md b/docs/guides/remote-agents.md
index 72d131871..53885498a 100644
--- a/docs/guides/remote-agents.md
+++ b/docs/guides/remote-agents.md
@@ -148,11 +148,16 @@ Forwarding identity is explicit on both sides. The receiver names which deployme
## Trace propagation
-Each remote turn starts a new trace. eve links the child trace to the
-dispatching turn and carries `gen_ai.conversation.id` so you can find the
-traces for one conversation. Trace context is observability metadata, not an
-authorization grant. See [OpenTelemetry](../observability/otel#trace-topology)
-for the trace topology.
+A remote child starts its own trace with a root `invoke_agent` span. When caller
+trace context is available, the first turn links to the dispatch span with
+`eve.link.type=agent.dispatch`. The receiving deployment applies its sampler.
+Later turns start new traces without the original dispatch link. The first turn
+of a local child can use the remote agent's trace as a nested span.
+
+Use `gen_ai.conversation.id` to find all turns and child sessions for one
+conversation. The child has its own session ID, message history, and agent
+state. Trace context identifies related work. It does not give authorization.
+See [OpenTelemetry](../observability/otel#trace-topology) for the trace structure.
eve carries parent session lineage separately. The receiver accepts it only
when `trustedForwarders` approves the authenticated caller; otherwise, trace
diff --git a/docs/reference/cli.md b/docs/reference/cli.md
index 11114138b..f03b87c15 100644
--- a/docs/reference/cli.md
+++ b/docs/reference/cli.md
@@ -391,13 +391,13 @@ Reads the immutable OTLP/JSON segments under `.eve/traces/v1`, so `eve dev` need
Span rows carry inline metrics when the span recorded them — `↑input`/`↓output` token counts, gateway cost, and the tool name for `execute_tool` spans. The header lists models across the trace, sums token usage and cost from step spans, and counts all error-bearing spans. `--verbose` expands each span under its tree row: status (with the error message on failures), timing, ids, every attribute (prompts, responses, and tool payloads as transcripts or pretty-printed JSON), and every span event with its offset from span start. `--json` prints the same records as JSON, one object per selected trace.
-Every subagent activation starts its own trace. The first child's `invoke_agent` root links to the dispatching caller with `eve.link.type=agent.dispatch`; remote agents preserve that caller in W3C `tracestate` even when HTTP `traceparent` advances through platform ingress. Later turns also start fresh traces without repeating the initial caller link. All related sessions retain the same `gen_ai.conversation.id`, and `agent.subagent.name` labels the child invocation.
+With local caller trace context, the first local subagent turn uses the caller's trace. Its span is a child of the dispatch span. A remote child starts a separate root trace and links its first turn to the dispatch with `eve.link.type=agent.dispatch`. Remote requests use W3C `tracestate` to identify that dispatch span if platform HTTP handling changes `traceparent`. Later child turns start new traces. All related sessions have the same `gen_ai.conversation.id`. `agent.subagent.name` identifies the subagent.
-Each workflow tool call has one `agent.action` span and keeps its `execute_tool <tool>` span. Sessions the tool opens with `ctx.agent` link to that call's `agent.action` span as their dispatching caller; individual `ctx.agent` calls get no span of their own. Only agent execution uses `invoke_agent`.
+Each workflow tool call has an `agent.action` span and an `execute_tool <tool>` span. The first turn of each local session it opens with `ctx.agent` is a child of the call's `agent.action` span. A remote session links to that action from its own trace. An individual `ctx.agent` call does not create a separate dispatch span. Only agent execution uses `invoke_agent`.
Outbound MCP `tools/call` requests add MCP semantic attributes to the matching `execute_tool` span. When eve has no tool span to enrich, it creates a `CLIENT` `tools/call <tool>` span. Each `tools/list` discovery also has a `CLIENT` span. Configured OpenTelemetry propagation fields are injected into MCP `params._meta` for JSON-RPC bodies up to 1 MiB and propagation metadata up to 8 KiB; larger requests are sent unchanged. eve removes its audience and session-lineage baggage before forwarding, and keeps the input/output content-capture policy local.
-A durable conversation produces one bounded trace per turn. Worker replacements reuse the prepared context for the same turn, while a later turn or an independently replayed attempt starts a fresh trace. Passing the conversation ID shows every trace it produced, oldest first.
+Each root agent turn starts a new trace. With local caller trace context, that trace includes the first turn of each local subagent it starts. Remote child turns and later local child turns start separate traces. If a worker is replaced, the new worker uses the prepared trace context for the same turn. Supply the conversation ID to show all related traces, oldest first.
Every span carries a real duration. A turn's root `invoke_agent` span is written when the turn settles, so a running turn shows only its steps.
diff --git a/e2e/fixtures/agent-subagents/agent/instrumentation/otel.ts b/e2e/fixtures/agent-subagents/agent/instrumentation/otel.ts
new file mode 100644
index 000000000..25a6bb8b0
--- /dev/null
+++ b/e2e/fixtures/agent-subagents/agent/instrumentation/otel.ts
@@ -0,0 +1,5 @@
+import { otel } from "eve/instrumentation/otel";
+
+// The Postgres suite runs a production Node server without automatic tracing.
+// Enable OTel so the remote child-stream eval can check trace identity.
+export default otel({});
diff --git a/e2e/fixtures/agent-subagents/evals/remote-subagent-stream.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-subagent-stream.eval.ts
index ae462da32..6e6494a19 100644
--- a/e2e/fixtures/agent-subagents/evals/remote-subagent-stream.eval.ts
+++ b/e2e/fixtures/agent-subagents/evals/remote-subagent-stream.eval.ts
@@ -44,6 +44,16 @@ export default defineEval({
"the proxied child stream carries the child's completed workspace lookup",
),
);
+ const parentTrace = turn.events.find((event) => event.type === "turn.started")?.data.trace;
+ const childTrace = childEvents.find((event) => event.type === "turn.started")?.data.trace;
+ turn.eventsSatisfy(
+ "remote dispatch starts a distinct child trace",
+ () =>
+ parentTrace !== undefined &&
+ childTrace !== undefined &&
+ childTrace.traceId !== parentTrace.traceId &&
+ childTrace.spanId !== parentTrace.spanId,
+ );
t.succeeded();
},
});
diff --git a/e2e/fixtures/agent-workflow-tools/agent/instrumentation/otel.ts b/e2e/fixtures/agent-workflow-tools/agent/instrumentation/otel.ts
new file mode 100644
index 000000000..e41127c71
--- /dev/null
+++ b/e2e/fixtures/agent-workflow-tools/agent/instrumentation/otel.ts
@@ -0,0 +1,5 @@
+import { otel } from "eve/instrumentation/otel";
+
+// The Postgres suite runs a production Node server without automatic tracing.
+// Enable OTel so the local fan-out eval can check trace identity.
+export default otel({});
diff --git a/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts
index 04fe1859f..c40ec56c1 100644
--- a/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts
+++ b/e2e/fixtures/agent-workflow-tools/evals/agent-fanout.wait.eval.ts
@@ -1,4 +1,5 @@
import { defineEval } from "eve/evals";
+import { isCurrentTurnBoundaryEvent } from "eve/client";
import { readHookAudit, recordsEveryAgentStart } from "./subagent-hook-audit.shared";
@@ -25,6 +26,31 @@ export default defineEval({
});
turn.messageIncludes("api:replica-0");
turn.messageIncludes("api:replica-1");
+ const parentTrace = turn.events.find((event) => event.type === "turn.started")?.data.trace;
+ const childTraces = await Promise.all(
+ turn.events
+ .filter((event) => event.type === "agent.started")
+ .map(async (started) => {
+ for await (const event of turn.session.agent(started).stream()) {
+ if (event.type === "turn.started") return event.data.trace;
+ if (isCurrentTurnBoundaryEvent(event)) break;
+ }
+ return undefined;
+ }),
+ );
+ turn.eventsSatisfy(
+ "parallel local children share the caller trace but not span identity",
+ () =>
+ parentTrace !== undefined &&
+ childTraces.length === 2 &&
+ childTraces.every(
+ (child) =>
+ child !== undefined &&
+ child.traceId === parentTrace.traceId &&
+ child.spanId !== parentTrace.spanId,
+ ) &&
+ new Set(childTraces.map((child) => child?.spanId)).size === 2,
+ );
turn.eventsSatisfy("both children start before the waiting tool resolves", (events) => {
const called = events.flatMap((event, index) =>
event.type === "agent.started" && event.data.name === "workflow-marker" ? [index] : [],
diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
index 634e5f336..cbfe66238 100644
--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
@@ -149,7 +149,14 @@ describe("buildConversationItems", () => {
});
it("marks turns dispatched by another turn as subagent items", () => {
- const parent = weatherTurn();
+ const parent = weatherTurn().map((item) => ({
+ ...item,
+ attributes: {
+ ...item.attributes,
+ "agent.run.id": "parent-session",
+ "gen_ai.conversation.id": "conversation",
+ },
+ }));
const parentAction = span("e".repeat(16), "agent.action", 5900, 6000, parent[0]!.spanId, {
"agent.action.call_id": "call-7",
"agent.action.kind": "subagent-call",
@@ -164,7 +171,9 @@ describe("buildConversationItems", () => {
parentAction.spanId,
{
"agent.channel.delivery.input": JSON.stringify({ message: "delegated task" }),
- "agent.turn.id": "turn_child",
+ "agent.turn.id": "turn_0",
+ "agent.run.id": "child-session",
+ "gen_ai.conversation.id": "conversation",
"gen_ai.operation.name": "invoke_agent",
},
);
diff --git a/packages/eve/src/execution/workflow-runtime.test.ts b/packages/eve/src/execution/workflow-runtime.test.ts
index 6c3ffe5c4..e696eb0d6 100644
--- a/packages/eve/src/execution/workflow-runtime.test.ts
+++ b/packages/eve/src/execution/workflow-runtime.test.ts
@@ -1065,7 +1065,7 @@ describe("createWorkflowRuntime#createSession trace seed allocation", () => {
expect(seed!.traceFlags).toBe(0);
});
- it("inherits parent policy while allocating an independent delegated trace", async () => {
+ it("inherits parent policy and trace while allocating a distinct child span", async () => {
const tracePolicy = vi.fn(() => false);
installAgentOtelRuntime(new AgentSpanIdGenerator(), tracePolicy);
mockBundleAndRun();
@@ -1100,7 +1100,7 @@ describe("createWorkflowRuntime#createSession trace seed allocation", () => {
decision: parentTrace.decision,
traceFlags: parentTrace.traceFlags,
});
- expect(seed!.traceId).not.toBe(parentTrace.traceId);
+ expect(seed!.traceId).toBe(parentTrace.traceId);
expect(seed!.spanId).not.toBe(parentTrace.spanId);
expect(serialized["eve.parentTraceContext"]).toEqual(parentTrace);
expect(serialized["eve.otelTraceEnabled"]).toBe(true);
diff --git a/packages/eve/src/instrumentation/runtime.test.ts b/packages/eve/src/instrumentation/runtime.test.ts
index 69b1087bb..bad8b25c1 100644
--- a/packages/eve/src/instrumentation/runtime.test.ts
+++ b/packages/eve/src/instrumentation/runtime.test.ts
@@ -147,7 +147,7 @@ describe("initializeSessionInstrumentation", () => {
traceId: "d".repeat(32),
});
expect(ctx.get(SessionTraceSeedKey)?.spanId).not.toBe("c".repeat(16));
- expect(ctx.get(SessionTraceSeedKey)?.traceId).not.toBe("d".repeat(32));
+ expect(ctx.get(SessionTraceSeedKey)?.traceId).toBe("d".repeat(32));
expect(ctx.get(ParentTraceContextKey)).not.toHaveProperty("forwardedTracePolicy");
expect(ctx.get(SessionTraceSeedKey)?.forwardedTracePolicy).toEqual({
ceiling: { recordInputs: true, recordOutputs: true },
@@ -367,7 +367,7 @@ describe("initializeSessionInstrumentation", () => {
decision: { action: "drop" },
traceFlags: 0,
});
- expect(ctx.get(SessionTraceSeedKey)?.traceId).not.toBe("d".repeat(32));
+ expect(ctx.get(SessionTraceSeedKey)?.traceId).toBe("d".repeat(32));
});
});
diff --git a/packages/eve/src/instrumentation/session-init.test.ts b/packages/eve/src/instrumentation/session-init.test.ts
index 5861c18c8..60d7fa8aa 100644
--- a/packages/eve/src/instrumentation/session-init.test.ts
+++ b/packages/eve/src/instrumentation/session-init.test.ts
@@ -1,7 +1,12 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { ContextContainer } from "#context/container.js";
-import { AuthKey, ChannelInstrumentationKey, SessionTraceSeedKey } from "#context/keys.js";
+import {
+ AuthKey,
+ ChannelInstrumentationKey,
+ ParentTraceContextKey,
+ SessionTraceSeedKey,
+} from "#context/keys.js";
import { initializeSessionInstrumentation } from "#instrumentation/session-init.js";
import { registerInstrumentationRuntime } from "#instrumentation/runtime-global.js";
import type { InstrumentationRuntime } from "#instrumentation/runtime.js";
@@ -29,6 +34,39 @@ function createRuntime(tracePolicy: TraceCapturePolicy): InstrumentationRuntime
}
describe("initializeSessionInstrumentation", () => {
+ it("applies the receiver capture policy and allocates a separate remote trace", () => {
+ registerInstrumentationRuntime({
+ ...createRuntime(() => false),
+ idGenerator: new AgentSpanIdGenerator(),
+ });
+ const ctx = new ContextContainer();
+ ctx.set(ParentTraceContextKey, {
+ isRemote: true,
+ spanId: "a".repeat(16),
+ traceFlags: 1,
+ traceId: "b".repeat(32),
+ } as never);
+ initializeSessionInstrumentation({ agentName: "remote", ctx });
+ expect(ctx.get(SessionTraceSeedKey)?.traceId).not.toBe("b".repeat(32));
+ expect(ctx.get(SessionTraceSeedKey)?.decision).toEqual({ action: "drop" });
+ expect(ctx.get(SessionTraceSeedKey)?.traceFlags).toBe(0);
+ });
+ it.each([0, 1])(
+ "preserves the caller's trace and sampling flag %s at child creation",
+ (traceFlags) => {
+ const ctx = new ContextContainer();
+ const parent = { spanId: "a".repeat(16), traceFlags, traceId: "b".repeat(32) };
+ ctx.set(ParentTraceContextKey, parent);
+
+ initializeSessionInstrumentation({ agentName: "child", ctx });
+
+ const seed = ctx.get(SessionTraceSeedKey)!;
+ expect(seed.traceId).toBe(parent.traceId);
+ expect(seed.spanId).not.toBe(parent.spanId);
+ expect(seed.traceFlags).toBe(traceFlags);
+ },
+ );
+
it("reconstructs a legacy conversation from session context", () => {
vi.stubEnv("EVE_DEV", "1");
registerInstrumentationRuntime({
diff --git a/packages/eve/src/instrumentation/session-init.ts b/packages/eve/src/instrumentation/session-init.ts
index 83634f2ae..6ec447ba7 100644
--- a/packages/eve/src/instrumentation/session-init.ts
+++ b/packages/eve/src/instrumentation/session-init.ts
@@ -115,9 +115,12 @@ function allocateSessionTraceSeed(input: {
: intersectInstrumentationDecisions(forwardedCeiling, inheritedDecision)
: (inheritedDecision ??
resolveTracePolicyDecision(isSampledTrace(input.parentTraceContext), input.conversation));
- const decision = input.forwardedTracePolicy
- ? intersectInstrumentationDecisions(parentDecision, localDecision())
- : parentDecision;
+ const isRemote =
+ "isRemote" in input.parentTraceContext && input.parentTraceContext.isRemote === true;
+ const decision =
+ input.forwardedTracePolicy || isRemote
+ ? intersectInstrumentationDecisions(parentDecision, localDecision())
+ : parentDecision;
const idGenerator = input.runtime?.idGenerator ?? new AgentSpanIdGenerator();
return {
decision,
@@ -126,7 +129,7 @@ function allocateSessionTraceSeed(input: {
: { forwardedTracePolicy: input.forwardedTracePolicy }),
spanId: idGenerator.allocateSpanId(),
traceFlags: decision.action === "drop" ? 0 : input.parentTraceContext.traceFlags,
- traceId: idGenerator.generateTraceId(),
+ traceId: isRemote ? idGenerator.generateTraceId() : input.parentTraceContext.traceId,
};
}
if (input.runtime?.prepareSessionTrace === undefined || input.runtime.idGenerator === undefined)
diff --git a/packages/eve/src/tracing/agent-memory-instrumentation.ts b/packages/eve/src/tracing/agent-memory-instrumentation.ts
index 63066d51e..7caa1387e 100644
--- a/packages/eve/src/tracing/agent-memory-instrumentation.ts
+++ b/packages/eve/src/tracing/agent-memory-instrumentation.ts
@@ -32,6 +32,7 @@ import { withChannelAudience } from "#tracing/channel-audience-context.js";
import { withErrorContent } from "#tracing/error-content-context.js";
import { isSampledTrace } from "#tracing/sampled-trace.js";
import { suppressTracing } from "#tracing/suppress-tracing.js";
+import { withOperationConversation } from "#tracing/conversation-context.js";
import type { ConversationEnvironment } from "#shared/conversation-context.js";
type SpanState = { readonly context: Context; readonly span: Span };
@@ -143,7 +144,10 @@ export function createAgentMemoryInstrumentation(input: {
: await context.with(
markAgentTraceContext(
withErrorContent(
- parent,
+ withOperationConversation(parent, {
+ rootSessionId: session?.rootSessionId,
+ sessionId: operation.sessionId,
+ }),
recordOutputs && effective?.action === "record" && effective.recordOutputs,
),
),
diff --git a/packages/eve/src/tracing/agent-otel-attributes.ts b/packages/eve/src/tracing/agent-otel-attributes.ts
index 9eb30cc37..f7c5dafb7 100644
--- a/packages/eve/src/tracing/agent-otel-attributes.ts
+++ b/packages/eve/src/tracing/agent-otel-attributes.ts
@@ -1,4 +1,4 @@
-import { resolveConversationId } from "#shared/conversation-identity.js";
+import { operationConversationId } from "#tracing/conversation-context.js";
import { AGENT_TRACE_SCHEMA_VERSION } from "#tracing/agent-span-contract.js";
export function traceSessionIdOf(scope: {
@@ -14,7 +14,7 @@ export function agentTraceIdentityAttributes(input: {
readonly sessionId: string;
readonly traceSessionId: string;
}): Record<string, string | number> {
- const conversationId = resolveConversationId(input.rootSessionId);
+ const conversationId = operationConversationId(input);
const attributes: Record<string, string | number> = {
"agent.run.id": input.sessionId,
"agent.trace.schema.version": AGENT_TRACE_SCHEMA_VERSION,
diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.ts
index e0e6fe7cd..90ebb1ce4 100644
--- a/packages/eve/src/tracing/agent-otel-provider.test.ts
+++ b/packages/eve/src/tracing/agent-otel-provider.test.ts
@@ -746,7 +746,7 @@ describe("createAgentOtelInstrumentation", () => {
});
});
- it("links a delegated activation to the exact caller from its own trace", async () => {
+ it("parents a delegated activation to the exact caller while retaining its span seed", async () => {
const runtime = createRuntime();
const parent = {
spanId: "c".repeat(16),
@@ -804,14 +804,10 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const invocation = byName(spans, "invoke_agent weather")[0]!;
expect(byName(spans, "agent.session")).toHaveLength(0);
- expect(invocation.spanContext().traceId).toBe(seed.traceId);
- expect(invocation.parentSpanContext).toBeUndefined();
- expect(invocation.links).toEqual([
- {
- context: { ...parent, isRemote: false },
- attributes: { "eve.link.type": "agent.dispatch" },
- },
- ]);
+ expect(invocation.spanContext().traceId).toBe(parent.traceId);
+ expect(invocation.spanContext().spanId).toBe(seed.spanId);
+ expect(invocation.parentSpanContext).toEqual({ ...parent, isRemote: false });
+ expect(invocation.links).toEqual([]);
expect(invocation.attributes).toMatchObject({
"agent.parent_call.id": "call-child",
"agent.parent_run.id": "parent-session",
@@ -959,7 +955,7 @@ describe("createAgentOtelInstrumentation", () => {
expect(captured?.channel.kind).toBe("channel:slack");
});
- it("allocates independent trace context for delegated agents", async () => {
+ it("allocates a distinct span in the caller's trace for delegated agents", async () => {
const runtime = createRuntime();
const parentTrace: InstrumentationTraceContext = {
spanId: "c".repeat(16),
@@ -976,7 +972,7 @@ describe("createAgentOtelInstrumentation", () => {
};
const trace = await runtime.prepareSessionTrace(sessionEvent);
- expect(trace.traceId).not.toBe(parentTrace.traceId);
+ expect(trace.traceId).toBe(parentTrace.traceId);
expect(trace.spanId).not.toBe(parentTrace.spanId);
expect(trace.traceFlags).toBe(parentTrace.traceFlags);
});
@@ -1274,10 +1270,7 @@ describe("createAgentOtelInstrumentation", () => {
expect(byName(spans, "agent.channel.delivery")).toHaveLength(0);
expect(turn.parentSpanContext).toBeUndefined();
expect(turn.links).toEqual([
- {
- context: parentTraceContext,
- attributes: { "eve.link.type": "agent.dispatch" },
- },
+ { context: parentTraceContext, attributes: { "eve.link.type": "agent.dispatch" } },
]);
expect(turn.spanContext().traceId).not.toBe(parentTraceContext.traceId);
expect(turn.attributes).toMatchObject({
@@ -1861,7 +1854,7 @@ describe("createAgentOtelInstrumentation", () => {
},
);
- it("preserves recording eligibility without adopting the parent's trace", async () => {
+ it("inherits recording eligibility and the parent's trace", async () => {
const runtime = createRuntime(new InMemoryAgentTraceStateStore(), null);
await emitAttempt({
@@ -1882,7 +1875,7 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
expect(spans.length).toBeGreaterThan(0);
for (const span of spans) {
- expect(span.spanContext().traceId).not.toBe("b".repeat(32));
+ expect(span.spanContext().traceId).toBe("b".repeat(32));
}
});
@@ -3149,7 +3142,7 @@ describe("createAgentOtelInstrumentation", () => {
}
});
- it("records a subagent child in a separate trace linked to its caller", async () => {
+ it("nests a subagent invocation beneath its caller in the same trace", async () => {
const runtime = createRuntime();
const caller = spanContext("1", "2");
@@ -3175,14 +3168,9 @@ describe("createAgentOtelInstrumentation", () => {
(span) => span.attributes["agent.subagent.name"] === "researcher",
)!;
- expect(childTurn.spanContext().traceId).not.toBe(caller.traceId);
- expect(childTurn.parentSpanContext).toBeUndefined();
- expect(childTurn.links).toEqual([
- {
- context: { ...caller, isRemote: false },
- attributes: { "eve.link.type": "agent.dispatch" },
- },
- ]);
+ expect(childTurn.spanContext().traceId).toBe(caller.traceId);
+ expect(childTurn.parentSpanContext).toEqual({ ...caller, isRemote: false });
+ expect(childTurn.links).toEqual([]);
expect(childTurn.attributes["agent.subagent.name"]).toBe("researcher");
expect(childTurn.attributes["agent.parent_call.id"]).toBe("call-1");
expect(childTurn.attributes["agent.parent_run.id"]).toBe("session-1");
@@ -3194,8 +3182,52 @@ describe("createAgentOtelInstrumentation", () => {
expect(byName(spans, "agent.session")).toHaveLength(0);
});
- it("links a remote action to a separately rooted remote child turn", async () => {
- const runtime = createRuntime();
+ it("exports a local child when its durable dispatch never terminates", async () => {
+ const store = new InMemoryAgentTraceStateStore();
+ const runtime = createRuntime(store);
+ await publishTurnStarted({
+ hooks: runtime.hooks,
+ sessionId: "parent",
+ turnId: "turn_0",
+ turnSequence: 0,
+ });
+ const scope: InstrumentationAttemptScope = {
+ attemptId: "parent:turn_0:0:0",
+ attemptIndex: 0,
+ sessionId: "parent",
+ turnId: "turn_0",
+ stepIndex: 0,
+ };
+ const key = actionIdempotencyKey("parent", "turn_0", "dispatch");
+ await runtime.hooks.publish({
+ type: "action.started",
+ idempotencyKey: key,
+ scope,
+ callId: "dispatch",
+ name: "child",
+ kind: "subagent-call",
+ });
+ const action = (await store.getAction(key))!;
+ await publishTurnStarted({
+ hooks: runtime.hooks,
+ sessionId: "child",
+ rootSessionId: "parent",
+ turnId: "turn_0",
+ turnSequence: 0,
+ parentTraceContext: { ...action.parent, spanId: action.spanId },
+ });
+ await completeTurn(runtime.hooks, "child", "turn_0");
+ await runtime.provider.forceFlush();
+ const spans = runtime.exporter.getFinishedSpans();
+ expect(byName(spans, "agent.action")).toHaveLength(0);
+ const child = byName(spans, "invoke_agent weather")[0]!;
+ expect(child.parentSpanContext?.spanId).toBe(action.spanId);
+ expect(child.spanContext().traceId).toBe(action.parent.traceId);
+ });
+
+ it("starts a remote root with a dispatch link and receiver sampling", async () => {
+ const samplesTrace = vi.fn(() => true);
+ const runtime = createRuntime(undefined, undefined, [], samplesTrace);
const parentTraceContext: InstrumentationTraceContext & { readonly isRemote: true } = {
isRemote: true,
spanId: "2".repeat(16),
@@ -3217,11 +3249,9 @@ describe("createAgentOtelInstrumentation", () => {
expect(childTurn.spanContext().traceId).not.toBe(parentTraceContext.traceId);
expect(childTurn.parentSpanContext).toBeUndefined();
expect(childTurn.links).toEqual([
- {
- context: parentTraceContext,
- attributes: { "eve.link.type": "agent.dispatch" },
- },
+ { context: parentTraceContext, attributes: { "eve.link.type": "agent.dispatch" } },
]);
+ expect(samplesTrace).toHaveBeenCalled();
});
it("opens its own root when a child session is handed no parent trace", async () => {
@@ -3247,7 +3277,7 @@ describe("createAgentOtelInstrumentation", () => {
expect(turn.parentSpanContext).toBeUndefined();
});
- it("links only the first subagent turn to the original caller", async () => {
+ it("nests the first child turn and starts fresh traces for persistent follow-ups", async () => {
const runtime = createRuntime();
const caller = spanContext("1", "2");
@@ -3268,15 +3298,11 @@ describe("createAgentOtelInstrumentation", () => {
const turns = byName(spans, "invoke_agent weather");
expect(byName(spans, "agent.session")).toHaveLength(0);
expect(turns).toHaveLength(201);
- expect(turns[0]!.spanContext().traceId).not.toBe(caller.traceId);
- expect(turns[0]!.parentSpanContext).toBeUndefined();
- expect(turns[0]!.links).toEqual([
- {
- context: { ...caller, isRemote: false },
- attributes: { "eve.link.type": "agent.dispatch" },
- },
- ]);
+ expect(turns[0]!.spanContext().traceId).toBe(caller.traceId);
+ expect(turns[0]!.parentSpanContext).toEqual({ ...caller, isRemote: false });
+ expect(turns[0]!.links).toEqual([]);
expect(new Set(turns.map((turn) => turn.spanContext().traceId))).toHaveLength(201);
+ expect(new Set(turns.map((turn) => turn.spanContext().spanId))).toHaveLength(201);
for (const turn of turns.slice(1)) {
expect(turn.spanContext().traceId).not.toBe(caller.traceId);
expect(turn.parentSpanContext).toBeUndefined();
diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.ts
index 3d2ef98be..94aaf70cb 100644
--- a/packages/eve/src/tracing/agent-otel-provider.ts
+++ b/packages/eve/src/tracing/agent-otel-provider.ts
@@ -75,6 +75,7 @@ import {
} from "#tracing/agent-span-contract.js";
import { withErrorContent } from "#tracing/error-content-context.js";
import { withAgentToolContentPolicy } from "#tracing/agent-tool-span-context.js";
+import { withOperationConversation } from "#tracing/conversation-context.js";
import { recordAgentSpanError as recordError } from "#tracing/agent-span-error.js";
import { resolveInstrumentationEnvironment } from "#internal/application/dev-environment.js";
import type { ConversationEnvironment } from "#shared/conversation-context.js";
@@ -306,7 +307,12 @@ export function createAgentOtelInstrumentation(
const session = await input.stateStore.getSession(event.sessionId);
if (isSampledTrace(turn.context)) {
const agentName = session?.agentName ?? turn.subagentName;
- const parentContext = withChannelAudience(ROOT_CONTEXT, session?.channelAudience);
+ const parentContext = withChannelAudience(
+ turn.caller === undefined || turn.caller.isRemote === true
+ ? ROOT_CONTEXT
+ : contextFromSpanContext(turn.caller),
+ session?.channelAudience,
+ );
const startSpan = () =>
input.tracer.startSpan(
agentInvocationSpanName(agentName),
@@ -321,7 +327,7 @@ export function createAgentOtelInstrumentation(
}),
kind: SpanKind.INTERNAL,
links: agentActivationLinks(turn),
- root: true,
+ root: turn.caller === undefined || turn.caller.isRemote === true,
startTime: turn.startTimeMs,
},
parentContext,
@@ -600,7 +606,10 @@ export function createAgentOtelInstrumentation(
operation.type === "tool.call"
? withAgentToolContentPolicy(withErrorPolicy, toolContentPolicy)
: withErrorPolicy;
- return context.with(markAgentTraceContext(operationContext), execute);
+ return context.with(
+ markAgentTraceContext(withOperationConversation(operationContext, operation.scope)),
+ execute,
+ );
},
};
@@ -624,11 +633,8 @@ export function createAgentOtelInstrumentation(
function agentActivationLinks(turn: AgentTurnTraceState): Link[] | undefined {
const links: Link[] = [];
- if (turn.caller !== undefined) {
- links.push({
- context: turn.caller,
- attributes: { "eve.link.type": "agent.dispatch" },
- });
+ if (turn.caller?.isRemote === true) {
+ links.push({ context: turn.caller, attributes: { "eve.link.type": "agent.dispatch" } });
}
if (turn.channelDelivery?.requestTraceContext !== undefined) {
links.push({
diff --git a/packages/eve/src/tracing/agent-otel-session-context.ts b/packages/eve/src/tracing/agent-otel-session-context.ts
index da02a5048..b3c8d0513 100644
--- a/packages/eve/src/tracing/agent-otel-session-context.ts
+++ b/packages/eve/src/tracing/agent-otel-session-context.ts
@@ -102,6 +102,8 @@ export function createAgentOtelSessionContext(
let turnContext = useInitialContext
? { ...session.context, isRemote: false }
: freshTurnContext(input, event.idempotencyKey, session.decision);
+ if (caller !== undefined && !("isRemote" in caller && caller.isRemote === true))
+ turnContext = { ...turnContext, traceId: caller.traceId };
const turn: AgentTurnTraceState = {
caller: caller === undefined ? undefined : adoptedSpanContext(caller),
context: turnContext,
@@ -114,7 +116,10 @@ export function createAgentOtelSessionContext(
startTimeMs: Date.now(),
subagentName: (event.parentLineage ?? session.parentLineage)?.subagentName,
};
- if (isSampledTrace(turn.context)) {
+ if (
+ isSampledTrace(turn.context) &&
+ (caller === undefined || ("isRemote" in caller && caller.isRemote === true))
+ ) {
const agentName = session.agentName ?? turn.subagentName;
const sampled =
input.samplesTrace?.(turn.context.traceId, {
@@ -173,7 +178,13 @@ function initialSessionContext(
traceFlags: decision.action === "drop" ? 0 : handed.traceFlags,
};
}
- const traceId = input.idGenerator.deriveTraceId(`session:${event.sessionId}`);
+ const traceId =
+ (event.parentTraceContext !== undefined &&
+ "isRemote" in event.parentTraceContext &&
+ event.parentTraceContext.isRemote === true
+ ? undefined
+ : event.parentTraceContext?.traceId) ??
+ input.idGenerator.deriveTraceId(`session:${event.sessionId}`);
const sampled = decision.action === "record";
return {
isRemote: false,
diff --git a/packages/eve/src/tracing/agent-span-contract.ts b/packages/eve/src/tracing/agent-span-contract.ts
index 2e0d6c914..ab7defec1 100644
--- a/packages/eve/src/tracing/agent-span-contract.ts
+++ b/packages/eve/src/tracing/agent-span-contract.ts
@@ -43,9 +43,9 @@ export function isAgentActivationSpan(span: AgentSpanRecord): boolean {
}
export function agentTurnIdentity(span: AgentSpanRecord): string | undefined {
- const conversationId = span.attributes["gen_ai.conversation.id"];
+ const sessionId = span.attributes["agent.run.id"];
const turnId = span.attributes["agent.turn.id"];
return typeof turnId === "string"
- ? `${typeof conversationId === "string" ? conversationId : ""}\0${turnId}`
+ ? `${typeof sessionId === "string" ? sessionId : ""}\0${turnId}`
: undefined;
}
diff --git a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts
index 258ba9fa2..3c09215fd 100644
--- a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts
+++ b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts
@@ -298,6 +298,129 @@ describe("exported agent telemetry contract", () => {
}
});
+ it("exports remote roots and a nested local child with the remote project's session grouping", async () => {
+ vi.stubEnv("VERCEL_ENV", "preview");
+ const runtime = createRuntime();
+ const caller = {
+ isRemote: true,
+ spanId: "a".repeat(16),
+ traceFlags: 1,
+ traceId: "b".repeat(32),
+ };
+ const hooks = runtime.hooks.forTrace!(traceContext("remote", "public"));
+ const remote = contextFor("public");
+ remote.set(ConversationIdKey, "caller-root");
+ remote.set(TraceRootKey, { kind: "own" });
+ remote.set(ParentSessionKey, {
+ callId: "remote-call",
+ rootSessionId: "caller-root",
+ sessionId: "caller-root",
+ turn: { id: "turn_0", sequence: 0 },
+ });
+ remote.set(ParentTraceContextKey, caller);
+ try {
+ for (const sequence of [0, 1]) {
+ const turnId = `turn_${sequence}`;
+ await contextStorage.run(remote, async () => {
+ const binding = bindInstrumentationRuntime(runtime, remote, {
+ agentName: "remote",
+ rootSessionId: "caller-root",
+ sessionId: "remote-session",
+ })!;
+ await binding.preparePreamble({ sequence, sessionStarted: sequence > 0, turnId });
+ await hooks.publish({
+ idempotencyKey: turnIdempotencyKey("remote-session", turnId),
+ sessionId: "remote-session",
+ turnId,
+ type: "turn.completed",
+ });
+ await hooks.publish({
+ idempotencyKey: sessionIdempotencyKey("remote-session"),
+ sessionId: "remote-session",
+ turnId,
+ type: "session.waiting",
+ });
+ });
+ }
+ const remoteSpan = runtime.exporter.getFinishedSpans()[0]!;
+ const child = contextFor("public");
+ child.set(ConversationIdKey, "caller-root");
+ child.set(TraceRootKey, { kind: "inherited", sessionId: "remote-session" });
+ child.set(ParentSessionKey, {
+ callId: "local-call",
+ rootSessionId: "caller-root",
+ sessionId: "remote-session",
+ turn: { id: "turn_0", sequence: 0 },
+ });
+ child.set(ParentTraceContextKey, remoteSpan.spanContext());
+ await contextStorage.run(child, async () => {
+ const binding = bindInstrumentationRuntime(runtime, child, {
+ agentName: "local",
+ rootSessionId: "caller-root",
+ sessionId: "local-session",
+ })!;
+ await binding.preparePreamble({ sequence: 0, sessionStarted: false, turnId: "turn_0" });
+ await hooks.publish({
+ idempotencyKey: sessionIdempotencyKey("local-session"),
+ sessionId: "local-session",
+ turnId: "turn_0",
+ type: "session.waiting",
+ });
+ });
+ await runtime.forceFlush();
+ const exported = runtime.exporter.getFinishedSpans();
+ const serialized = new TextDecoder().decode(JsonTraceSerializer.serializeRequest(exported)!);
+ const traceIds = [...new Set(exported.map((span) => span.spanContext().traceId))];
+ expect(traceIds).toHaveLength(2);
+ const parsed = traceIds.flatMap((traceId) => parseLocalTraceSegment(serialized, traceId));
+ expect(parsed).toHaveLength(3);
+ expect(new Set(parsed.map((span) => span.spanId)).size).toBe(3);
+ for (const span of parsed) {
+ expect(span.attributes["gen_ai.conversation.id"]).toBe("caller-root");
+ expect(span.attributes["vercel.session_id"]).toBe("remote-session");
+ }
+ const remoteTurns = parsed.filter(
+ (span) => span.attributes["agent.run.id"] === "remote-session",
+ );
+ expect(remoteTurns).toHaveLength(2);
+ expect(remoteTurns[0]!.parentSpanId).toBeUndefined();
+ expect(remoteTurns[0]!.traceId).not.toBe(caller.traceId);
+ expect(remoteSpan.links).toEqual([
+ { context: caller, attributes: { "eve.link.type": "agent.dispatch" } },
+ ]);
+ expect(remoteTurns[1]!.parentSpanId).toBeUndefined();
+ expect(remoteTurns[1]!.traceId).not.toBe(caller.traceId);
+ const local = parsed.find((span) => span.attributes["agent.run.id"] === "local-session")!;
+ expect(local.parentSpanId).toBe(remoteSpan.spanContext().spanId);
+ expect(local.attributes["agent.parent_run.id"]).toBe("remote-session");
+ // Downstream readers use this export; normalize host metadata and clock values for CI.
+ const fixture = JSON.parse(serialized) as unknown;
+ const normalizeTimes = (value: unknown): void => {
+ if (Array.isArray(value)) {
+ for (const child of value) normalizeTimes(child);
+ return;
+ }
+ if (value === null || typeof value !== "object") return;
+ if (Reflect.get(value, "key") === "service.name") {
+ Reflect.set(value, "value", { stringValue: "eve-trace-contract" });
+ }
+ for (const [key, child] of Object.entries(value)) {
+ if (key === "startTimeUnixNano" || key === "timeUnixNano")
+ Reflect.set(value, key, "1700000000000000000");
+ else if (key === "endTimeUnixNano") Reflect.set(value, key, "1700000000001000000");
+ else normalizeTimes(child);
+ }
+ };
+ normalizeTimes(fixture);
+ await expect(`${JSON.stringify(fixture, null, 2)}\n`).toMatchFileSnapshot(
+ "./test-data/remote-local-agent-trace.otlp.json",
+ );
+ } finally {
+ await runtime.shutdown();
+ vi.unstubAllEnvs();
+ }
+ });
+
it.each(["public", "private"] as const)(
"round-trips the normalized %s v4 trace forest through OTLP",
async (audience) => {
@@ -407,7 +530,7 @@ describe("exported agent telemetry contract", () => {
child.set(SessionTraceSeedKey, {
...dispatch!.parentTraceContext,
spanId: runtime.idGenerator.allocateSpanId(),
- traceId: runtime.idGenerator.generateTraceId(),
+ traceId: dispatch!.parentTraceContext.traceId,
});
}
const childScope = scopeFor("child", audience);
@@ -564,7 +687,7 @@ describe("exported agent telemetry contract", () => {
const exported = runtime.exporter.getFinishedSpans();
const bytes = JsonTraceSerializer.serializeRequest(exported)!;
const traceIds = [...new Set(exported.map((span) => span.spanContext().traceId))];
- expect(traceIds).toHaveLength(4);
+ expect(traceIds).toHaveLength(3);
const traces = traceIds.map((traceId) =>
assembleLocalTrace(
traceId,
@@ -606,7 +729,8 @@ describe("exported agent telemetry contract", () => {
}
expect(new TextDecoder().decode(bytes)).not.toContain("auth-only-secret");
const workflow = parsed.find(
- (span) => span.attributes["agent.action.call_id"] === "workflow",
+ (span) =>
+ span.name === "agent.action" && span.attributes["agent.action.call_id"] === "workflow",
)!;
const activation = parsed.find(
(span) => span.name === "invoke_agent child" && isAgentTurnSpan(span),
@@ -614,8 +738,8 @@ describe("exported agent telemetry contract", () => {
const parentActivation = parsed.find(
(span) => span.name === "invoke_agent parent" && isAgentTurnSpan(span),
)!;
- expect(activation.parentSpanId).toBeUndefined();
- expect(activation.traceId).not.toBe(workflow.traceId);
+ expect(activation.parentSpanId).toBe(workflow.spanId);
+ expect(activation.traceId).toBe(workflow.traceId);
expect(parentActivation.attributes).toMatchObject({
"agent.channel.delivery.id": "delivery",
"agent.channel.kind": "http",
@@ -639,11 +763,9 @@ describe("exported agent telemetry contract", () => {
" agent.action coordinate",
" agent.approval approved",
" execute_tool coordinate",
- "trace child:turn_0 outcome=completed",
- " invoked from parent:turn_0/agent.action coordinate via agent.dispatch",
- " invoke_agent child",
- " agent.step",
- " chat test",
+ " invoke_agent child",
+ " agent.step",
+ " chat test",
"trace parent:turn_1 outcome=failed channel=http:web delivery=delivery-failed",
" invoke_agent parent",
"trace parent:turn_2 outcome=cancelled channel=http:web delivery=delivery-cancelled",
@@ -887,17 +1009,6 @@ function normalizeTraceForest(
)}`,
]),
);
- const causalParents = new Map(
- exported.flatMap((span) =>
- span.links
- .filter(
- (link) =>
- link.attributes?.["eve.link.type"] === "agent.dispatch" &&
- aliases.has(link.context.traceId),
- )
- .map((link) => [span.spanContext().traceId, link.context.traceId] as const),
- ),
- );
const byIdentity = new Map(spans.map((span) => [`${span.traceId}:${span.spanId}`, span]));
const children = Map.groupBy(
spans.filter((span) => span.parentSpanId !== undefined),
@@ -909,8 +1020,6 @@ function normalizeTraceForest(
const lines =
conversationIds.length === 1 ? [`conversation ${conversationIds[0]}`] : ["conversation mixed"];
for (const root of roots.toSorted((left, right) => {
- if (causalParents.get(left.traceId) === right.traceId) return 1;
- if (causalParents.get(right.traceId) === left.traceId) return -1;
const sequence =
Number(left.attributes["agent.turn.sequence"]) -
Number(right.attributes["agent.turn.sequence"]);
diff --git a/packages/eve/src/tracing/agent-trace-span-processor.test.ts b/packages/eve/src/tracing/agent-trace-span-processor.test.ts
index bc30efc81..efa542938 100644
--- a/packages/eve/src/tracing/agent-trace-span-processor.test.ts
+++ b/packages/eve/src/tracing/agent-trace-span-processor.test.ts
@@ -3,6 +3,24 @@ import { describe, expect, it, vi } from "vitest";
import { AgentTraceSpanProcessor } from "#tracing/agent-trace-span-processor.js";
describe("AgentTraceSpanProcessor", () => {
+ it("adopts the activation owner after a third-party span arrives first", () => {
+ const processor = new AgentTraceSpanProcessor([]);
+ const attributes = { "gen_ai.conversation.id": "conversation" };
+ processor.onStart(span("trace", attributes), {});
+ const activation = {
+ ...span("trace", {
+ ...attributes,
+ "agent.run.id": "run",
+ "agent.turn.id": "turn_0",
+ "gen_ai.operation.name": "invoke_agent",
+ }),
+ name: "invoke_agent root",
+ };
+ processor.onStart(activation, {});
+ processor.onEnd(activation);
+ expect(processor.releaseCompletedTraces()).toBe(true);
+ expect([...processor.activeTraceIds()]).toEqual([]);
+ });
it("routes an agent trace and releases it at session terminal", () => {
const child = {
forceFlush: vi.fn(async () => {}),
@@ -67,9 +85,25 @@ describe("AgentTraceSpanProcessor", () => {
};
const processor = new AgentTraceSpanProcessor([child]);
const owned = { "gen_ai.conversation.id": "session-1" };
- const delegated = { "gen_ai.conversation.id": "child-1" };
- processor.onStart(span("trace-1", owned), {});
- processor.onStart(span("trace-1", delegated), {});
+ const root = {
+ ...span("trace-1", {
+ ...owned,
+ "agent.run.id": "root-session",
+ "agent.turn.id": "turn_0",
+ "gen_ai.operation.name": "invoke_agent",
+ }),
+ name: "invoke_agent root",
+ };
+ const delegated = {
+ ...root,
+ attributes: { ...root.attributes, "agent.run.id": "child-session" },
+ name: "invoke_agent child",
+ parentSpanContext: { spanId: "caller" },
+ };
+ processor.onStart(root, {});
+ processor.onStart(delegated, {});
+ processor.onEnd(delegated);
+ processor.releaseCompletedTraces();
expect(processor.releaseConversation("child-1")).toBe(false);
expect([...processor.activeTraceIds()]).toEqual(["trace-1"]);
@@ -79,7 +113,8 @@ describe("AgentTraceSpanProcessor", () => {
processor.onEnd(later);
expect(child.onEnd).toHaveBeenCalledWith(later);
- expect(processor.releaseConversation("session-1")).toBe(true);
+ processor.onEnd(root);
+ expect(processor.releaseCompletedTraces()).toBe(true);
expect([...processor.activeTraceIds()]).toEqual([]);
});
diff --git a/packages/eve/src/tracing/agent-trace-span-processor.ts b/packages/eve/src/tracing/agent-trace-span-processor.ts
index 861de1a97..acd50272c 100644
--- a/packages/eve/src/tracing/agent-trace-span-processor.ts
+++ b/packages/eve/src/tracing/agent-trace-span-processor.ts
@@ -16,7 +16,10 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
readonly #ownedTraceIds = new Set<string>();
readonly #completedTraceIds = new Set<string>();
readonly #rememberedTraceIds = new Set<string>();
- readonly #traceConversations = new Map<string, string>();
+ readonly #traceOwnership = new Map<
+ string,
+ { readonly conversationId: string; readonly ownerRunId?: string }
+ >();
constructor(children: readonly SpanProcessor[]) {
this.#children = children;
}
@@ -28,12 +31,21 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
onStart(span: unknown, parentContext: unknown): void {
if (!isSpanLike(span)) return;
const conversationId = span.attributes["gen_ai.conversation.id"];
+ const runId = span.attributes["agent.run.id"];
if (typeof conversationId === "string") {
const traceId = span.spanContext().traceId;
const known = this.#ownedTraceIds.has(traceId) || this.#rememberedTraceIds.has(traceId);
if (!known) {
this.#ownedTraceIds.add(traceId);
- this.#traceConversations.set(traceId, conversationId);
+ this.#traceOwnership.set(traceId, {
+ conversationId,
+ ownerRunId: typeof runId === "string" ? runId : undefined,
+ });
+ } else if (
+ this.#traceOwnership.get(traceId)?.ownerRunId === undefined &&
+ typeof runId === "string"
+ ) {
+ this.#traceOwnership.set(traceId, { conversationId, ownerRunId: runId });
}
}
if (!this.#accepts(span)) return;
@@ -44,9 +56,11 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
if (!isSpanLike(span) || !this.#accepts(span)) return;
for (const child of this.#children) child.onEnd(span);
const traceId = span.spanContext().traceId;
+ const ownership = this.#traceOwnership.get(traceId);
if (
isAgentActivationSpan({ name: span.name ?? "", attributes: span.attributes }) &&
- this.#traceConversations.get(traceId) === span.attributes["gen_ai.conversation.id"]
+ ownership?.ownerRunId === span.attributes["agent.run.id"] &&
+ ownership?.conversationId === span.attributes["gen_ai.conversation.id"]
) {
this.#completedTraceIds.add(traceId);
}
@@ -62,7 +76,7 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
if (this.#completedTraceIds.size === 0) return false;
for (const traceId of this.#completedTraceIds) {
this.#ownedTraceIds.delete(traceId);
- this.#traceConversations.delete(traceId);
+ this.#traceOwnership.delete(traceId);
this.#rememberedTraceIds.add(traceId);
}
this.#completedTraceIds.clear();
@@ -76,12 +90,12 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
/** Releases only traces owned by this conversation. */
releaseConversation(conversationId: string): boolean {
let released = false;
- for (const [traceId, traceConversationId] of this.#traceConversations) {
- if (traceConversationId !== conversationId) continue;
+ for (const [traceId, ownership] of this.#traceOwnership) {
+ if (ownership.conversationId !== conversationId) continue;
released = true;
this.#ownedTraceIds.delete(traceId);
this.#completedTraceIds.delete(traceId);
- this.#traceConversations.delete(traceId);
+ this.#traceOwnership.delete(traceId);
}
return released;
}
diff --git a/packages/eve/src/tracing/conversation-context.ts b/packages/eve/src/tracing/conversation-context.ts
index 67c46ddf3..4f0be5061 100644
--- a/packages/eve/src/tracing/conversation-context.ts
+++ b/packages/eve/src/tracing/conversation-context.ts
@@ -1,7 +1,34 @@
+import { createContextKey, type Context } from "#compiled/@opentelemetry/api/index.js";
import { readBaggageMember, replaceBaggageMember } from "#protocol/baggage.js";
-import { readConversationId } from "#shared/conversation-identity.js";
+import { readConversationId, resolveConversationId } from "#shared/conversation-identity.js";
const BAGGAGE_KEY = "eve.conversation.id";
+const CONVERSATION_KEY = createContextKey("eve.trace.conversation.id");
+
+export function withConversationId(context: Context, conversationId: string): Context {
+ return context.setValue(CONVERSATION_KEY, conversationId);
+}
+
+interface OperationConversation {
+ readonly rootSessionId?: string;
+ readonly sessionId: string;
+}
+
+export function operationConversationId(input: OperationConversation): string {
+ return resolveConversationId(input.rootSessionId ?? input.sessionId);
+}
+
+export function withOperationConversation(context: Context, input: OperationConversation): Context {
+ return withConversationId(context, operationConversationId(input));
+}
+
+export function conversationIdFromContext(context: unknown): string | undefined {
+ if (typeof context !== "object" || context === null) return undefined;
+ const getValue = Reflect.get(context, "getValue");
+ return typeof getValue === "function"
+ ? readConversationId(Reflect.apply(getValue, context, [CONVERSATION_KEY]))
+ : undefined;
+}
export function readConversationBaggage(value: string | null): string | undefined {
const member = readBaggageMember(value, BAGGAGE_KEY);
diff --git a/packages/eve/src/tracing/otel-registration.scenario.test.ts b/packages/eve/src/tracing/otel-registration.scenario.test.ts
index dcae7e81e..9c517c1aa 100644
--- a/packages/eve/src/tracing/otel-registration.scenario.test.ts
+++ b/packages/eve/src/tracing/otel-registration.scenario.test.ts
@@ -16,6 +16,7 @@ import {
} from "#compiled/@opentelemetry/api/index.js";
import { registerOtelPipeline } from "#tracing/otel-registration.js";
import { withErrorContent } from "#tracing/error-content-context.js";
+import { withConversationId } from "#tracing/conversation-context.js";
import { createLogger, logError } from "#internal/logging.js";
const require = createRequire(import.meta.url);
@@ -128,11 +129,20 @@ describe("registerOtelPipeline", () => {
});
const parent = runtimeTrace.getTracer("eve").startSpan("eve.parent");
const parentContext = parent.spanContext();
- const activeContext = runtimeTrace.setSpan(COMPILED_ROOT_CONTEXT, parent);
+ const activeContext = withConversationId(
+ runtimeTrace.setSpan(COMPILED_ROOT_CONTEXT, parent),
+ "caller-conversation",
+ );
const child = await runtimeContext.with(activeContext, async () => {
await Promise.resolve();
return authoredTracer.startSpan("authored.child");
});
+ const authoredConversation = await runtimeContext.with(activeContext, () =>
+ authoredTracer.startSpan("authored.conversation", {
+ attributes: { "gen_ai.conversation.id": "authored-id" },
+ }),
+ );
+ authoredConversation.end();
expect(child.isRecording()).toBe(true);
expect(child.spanContext().traceId).toBe(parentContext.traceId);
@@ -144,6 +154,17 @@ describe("registerOtelPipeline", () => {
.getFinishedSpans()
.find((span) => span.name === "authored.child");
expect(exportedChild?.parentSpanContext?.spanId).toBe(parentContext.spanId);
+ expect(exportedChild?.attributes["gen_ai.conversation.id"]).toBe("caller-conversation");
+ expect(
+ exporter.getFinishedSpans().find((span) => span.name === "authored.conversation")?.attributes[
+ "gen_ai.conversation.id"
+ ],
+ ).toBe("authored-id");
+ expect(
+ exporter.getFinishedSpans().find((span) => span.name === "eve.parent")?.attributes[
+ "gen_ai.conversation.id"
+ ],
+ ).toBeUndefined();
await runtime.shutdown();
});
diff --git a/packages/eve/src/tracing/otel-registration.ts b/packages/eve/src/tracing/otel-registration.ts
index a2346acd7..c20dac492 100644
--- a/packages/eve/src/tracing/otel-registration.ts
+++ b/packages/eve/src/tracing/otel-registration.ts
@@ -16,6 +16,7 @@ import {
} from "#compiled/@vercel/otel/index.js";
import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";
+import { conversationIdFromContext } from "#tracing/conversation-context.js";
import type { OtelPipeline } from "#tracing/otel-declaration.js";
import {
agentInvocationSpanName,
@@ -100,6 +101,17 @@ class PrivateSpanFilteringProcessor implements SpanProcessor {
onStart(span: unknown, parentContext: unknown): void {
if (isRegistrationSpan(span)) return;
+ const conversationId = conversationIdFromContext(parentContext);
+ if (
+ conversationId !== undefined &&
+ (span as { attributes?: Record<string, unknown> }).attributes?.["gen_ai.conversation.id"] ===
+ undefined
+ ) {
+ (span as { setAttribute(key: string, value: string): void }).setAttribute(
+ "gen_ai.conversation.id",
+ conversationId,
+ );
+ }
const identity = spanIdentity(span);
if (identity !== undefined) addBounded(this.startedSpans, identity);
for (const processor of this.processors) processor.onStart(span, parentContext);
diff --git a/packages/eve/src/tracing/test-data/remote-local-agent-trace.otlp.json b/packages/eve/src/tracing/test-data/remote-local-agent-trace.otlp.json
new file mode 100644
index 000000000..d8c9a0766
--- /dev/null
+++ b/packages/eve/src/tracing/test-data/remote-local-agent-trace.otlp.json
@@ -0,0 +1,600 @@
+{
+ "resourceSpans": [
+ {
+ "resource": {
+ "attributes": [
+ {
+ "key": "service.name",
+ "value": {
+ "stringValue": "eve-trace-contract"
+ }
+ },
+ {
+ "key": "telemetry.sdk.language",
+ "value": {
+ "stringValue": "nodejs"
+ }
+ },
+ {
+ "key": "telemetry.sdk.name",
+ "value": {
+ "stringValue": "opentelemetry"
+ }
+ },
+ {
+ "key": "telemetry.sdk.version",
+ "value": {
+ "stringValue": "2.6.1"
+ }
+ }
+ ],
+ "droppedAttributesCount": 0
+ },
+ "scopeSpans": [
+ {
+ "scope": {
+ "name": "eve.agent"
+ },
+ "spans": [
+ {
+ "traceId": "b31314b8db6855d6662fc41a0ea318e2",
+ "spanId": "b31314b8db6855d6",
+ "name": "invoke_agent remote",
+ "kind": 1,
+ "startTimeUnixNano": "1700000000000000000",
+ "endTimeUnixNano": "1700000000001000000",
+ "attributes": [
+ {
+ "key": "agent.framework.name",
+ "value": {
+ "stringValue": "eve"
+ }
+ },
+ {
+ "key": "agent.framework.version",
+ "value": {
+ "stringValue": "test"
+ }
+ },
+ {
+ "key": "agent.name",
+ "value": {
+ "stringValue": "remote"
+ }
+ },
+ {
+ "key": "agent.channel.audience",
+ "value": {
+ "stringValue": "public"
+ }
+ },
+ {
+ "key": "agent.principal.current.id",
+ "value": {
+ "stringValue": "current-user"
+ }
+ },
+ {
+ "key": "agent.principal.current.type",
+ "value": {
+ "stringValue": "service"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.id",
+ "value": {
+ "stringValue": "initiator-user"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.type",
+ "value": {
+ "stringValue": "user"
+ }
+ },
+ {
+ "key": "agent.channel.kind",
+ "value": {
+ "stringValue": "http"
+ }
+ },
+ {
+ "key": "agent.session.origin",
+ "value": {
+ "stringValue": "channel"
+ }
+ },
+ {
+ "key": "agent.parent_call.id",
+ "value": {
+ "stringValue": "remote-call"
+ }
+ },
+ {
+ "key": "agent.parent_run.id",
+ "value": {
+ "stringValue": "caller-root"
+ }
+ },
+ {
+ "key": "agent.run.type",
+ "value": {
+ "stringValue": "subagent"
+ }
+ },
+ {
+ "key": "agent.trace.content.input",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.trace.content.output",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.turn.id",
+ "value": {
+ "stringValue": "turn_0"
+ }
+ },
+ {
+ "key": "agent.turn.sequence",
+ "value": {
+ "intValue": 0
+ }
+ },
+ {
+ "key": "gen_ai.agent.name",
+ "value": {
+ "stringValue": "remote"
+ }
+ },
+ {
+ "key": "gen_ai.operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "resource.name",
+ "value": {
+ "stringValue": "invoke_agent remote"
+ }
+ },
+ {
+ "key": "agent.run.id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ },
+ {
+ "key": "agent.trace.schema.version",
+ "value": {
+ "intValue": 4
+ }
+ },
+ {
+ "key": "gen_ai.conversation.id",
+ "value": {
+ "stringValue": "caller-root"
+ }
+ },
+ {
+ "key": "vercel.session_id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ },
+ {
+ "key": "agent.turn.outcome",
+ "value": {
+ "stringValue": "completed"
+ }
+ }
+ ],
+ "droppedAttributesCount": 0,
+ "events": [
+ {
+ "attributes": [],
+ "name": "turn.started",
+ "timeUnixNano": "1700000000000000000",
+ "droppedAttributesCount": 0
+ },
+ {
+ "attributes": [],
+ "name": "turn.completed",
+ "timeUnixNano": "1700000000000000000",
+ "droppedAttributesCount": 0
+ }
+ ],
+ "droppedEventsCount": 0,
+ "status": {
+ "code": 0
+ },
+ "links": [
+ {
+ "attributes": [
+ {
+ "key": "eve.link.type",
+ "value": {
+ "stringValue": "agent.dispatch"
+ }
+ }
+ ],
+ "spanId": "aaaaaaaaaaaaaaaa",
+ "traceId": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+ "droppedAttributesCount": 0,
+ "flags": 769
+ }
+ ],
+ "droppedLinksCount": 0,
+ "flags": 257
+ },
+ {
+ "traceId": "86d795248ef3bc2325f68a0cc80f27d7",
+ "spanId": "86d795248ef3bc23",
+ "name": "invoke_agent remote",
+ "kind": 1,
+ "startTimeUnixNano": "1700000000000000000",
+ "endTimeUnixNano": "1700000000001000000",
+ "attributes": [
+ {
+ "key": "agent.framework.name",
+ "value": {
+ "stringValue": "eve"
+ }
+ },
+ {
+ "key": "agent.framework.version",
+ "value": {
+ "stringValue": "test"
+ }
+ },
+ {
+ "key": "agent.name",
+ "value": {
+ "stringValue": "remote"
+ }
+ },
+ {
+ "key": "agent.channel.audience",
+ "value": {
+ "stringValue": "public"
+ }
+ },
+ {
+ "key": "agent.principal.current.id",
+ "value": {
+ "stringValue": "current-user"
+ }
+ },
+ {
+ "key": "agent.principal.current.type",
+ "value": {
+ "stringValue": "service"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.id",
+ "value": {
+ "stringValue": "initiator-user"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.type",
+ "value": {
+ "stringValue": "user"
+ }
+ },
+ {
+ "key": "agent.channel.kind",
+ "value": {
+ "stringValue": "http"
+ }
+ },
+ {
+ "key": "agent.session.origin",
+ "value": {
+ "stringValue": "channel"
+ }
+ },
+ {
+ "key": "agent.parent_call.id",
+ "value": {
+ "stringValue": "remote-call"
+ }
+ },
+ {
+ "key": "agent.parent_run.id",
+ "value": {
+ "stringValue": "caller-root"
+ }
+ },
+ {
+ "key": "agent.run.type",
+ "value": {
+ "stringValue": "subagent"
+ }
+ },
+ {
+ "key": "agent.trace.content.input",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.trace.content.output",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.turn.id",
+ "value": {
+ "stringValue": "turn_1"
+ }
+ },
+ {
+ "key": "agent.turn.sequence",
+ "value": {
+ "intValue": 1
+ }
+ },
+ {
+ "key": "gen_ai.agent.name",
+ "value": {
+ "stringValue": "remote"
+ }
+ },
+ {
+ "key": "gen_ai.operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "resource.name",
+ "value": {
+ "stringValue": "invoke_agent remote"
+ }
+ },
+ {
+ "key": "agent.run.id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ },
+ {
+ "key": "agent.trace.schema.version",
+ "value": {
+ "intValue": 4
+ }
+ },
+ {
+ "key": "gen_ai.conversation.id",
+ "value": {
+ "stringValue": "caller-root"
+ }
+ },
+ {
+ "key": "vercel.session_id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ },
+ {
+ "key": "agent.turn.outcome",
+ "value": {
+ "stringValue": "completed"
+ }
+ }
+ ],
+ "droppedAttributesCount": 0,
+ "events": [
+ {
+ "attributes": [],
+ "name": "turn.started",
+ "timeUnixNano": "1700000000000000000",
+ "droppedAttributesCount": 0
+ },
+ {
+ "attributes": [],
+ "name": "turn.completed",
+ "timeUnixNano": "1700000000000000000",
+ "droppedAttributesCount": 0
+ }
+ ],
+ "droppedEventsCount": 0,
+ "status": {
+ "code": 0
+ },
+ "links": [],
+ "droppedLinksCount": 0,
+ "flags": 257
+ },
+ {
+ "traceId": "b31314b8db6855d6662fc41a0ea318e2",
+ "spanId": "d8f3a7d23b288e65",
+ "parentSpanId": "b31314b8db6855d6",
+ "name": "invoke_agent local",
+ "kind": 1,
+ "startTimeUnixNano": "1700000000000000000",
+ "endTimeUnixNano": "1700000000001000000",
+ "attributes": [
+ {
+ "key": "agent.framework.name",
+ "value": {
+ "stringValue": "eve"
+ }
+ },
+ {
+ "key": "agent.framework.version",
+ "value": {
+ "stringValue": "test"
+ }
+ },
+ {
+ "key": "agent.name",
+ "value": {
+ "stringValue": "local"
+ }
+ },
+ {
+ "key": "agent.channel.audience",
+ "value": {
+ "stringValue": "public"
+ }
+ },
+ {
+ "key": "agent.principal.current.id",
+ "value": {
+ "stringValue": "current-user"
+ }
+ },
+ {
+ "key": "agent.principal.current.type",
+ "value": {
+ "stringValue": "service"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.id",
+ "value": {
+ "stringValue": "initiator-user"
+ }
+ },
+ {
+ "key": "agent.principal.initiator.type",
+ "value": {
+ "stringValue": "user"
+ }
+ },
+ {
+ "key": "agent.parent_call.id",
+ "value": {
+ "stringValue": "local-call"
+ }
+ },
+ {
+ "key": "agent.parent_run.id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ },
+ {
+ "key": "agent.run.type",
+ "value": {
+ "stringValue": "subagent"
+ }
+ },
+ {
+ "key": "agent.trace.content.input",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.trace.content.output",
+ "value": {
+ "boolValue": true
+ }
+ },
+ {
+ "key": "agent.turn.id",
+ "value": {
+ "stringValue": "turn_0"
+ }
+ },
+ {
+ "key": "agent.turn.sequence",
+ "value": {
+ "intValue": 0
+ }
+ },
+ {
+ "key": "gen_ai.agent.name",
+ "value": {
+ "stringValue": "local"
+ }
+ },
+ {
+ "key": "gen_ai.operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "operation.name",
+ "value": {
+ "stringValue": "invoke_agent"
+ }
+ },
+ {
+ "key": "resource.name",
+ "value": {
+ "stringValue": "invoke_agent local"
+ }
+ },
+ {
+ "key": "agent.run.id",
+ "value": {
+ "stringValue": "local-session"
+ }
+ },
+ {
+ "key": "agent.trace.schema.version",
+ "value": {
+ "intValue": 4
+ }
+ },
+ {
+ "key": "gen_ai.conversation.id",
+ "value": {
+ "stringValue": "caller-root"
+ }
+ },
+ {
+ "key": "vercel.session_id",
+ "value": {
+ "stringValue": "remote-session"
+ }
+ }
+ ],
+ "droppedAttributesCount": 0,
+ "events": [
+ {
+ "attributes": [],
+ "name": "turn.started",
+ "timeUnixNano": "1700000000000000000",
+ "droppedAttributesCount": 0
+ }
+ ],
+ "droppedEventsCount": 0,
+ "status": {
+ "code": 0
+ },
+ "links": [],
+ "droppedLinksCount": 0,
+ "flags": 257
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}