Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): adopt GenAI semantic conventions for agent traces (#2706)

Chad Hietala3bdcb88ffd0b
diff --git a/.changeset/standard-genai-agent-spans.md b/.changeset/standard-genai-agent-spans.md
new file mode 100644
index 000000000..52c6c9c2f
--- /dev/null
+++ b/.changeset/standard-genai-agent-spans.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Align local agent invocation and tool execution spans with the OpenTelemetry GenAI agent conventions. Agent turns now emit usage-bearing `invoke_agent` spans, model calls emit `chat` directly beneath `agent.step` without a redundant `ai.streamText` wrapper, tool calls emit `execute_tool` spans, and terminal failures include `error.type`. In experimental provider mode, channel requests use `agent.channel.request`; legacy instrumentation retains HTTP-semantic route names. The first terminal delivery for a turn parents its invocation, and Workflow SDK spans started from agent contexts remain on a separate trace. These naming changes advance the Agent Trace schema to version 3 while local readers retain version 2 compatibility.
diff --git a/docs/reference/cli.md b/docs/reference/cli.md
index fa9609d28..2f0c61dca 100644
--- a/docs/reference/cli.md
+++ b/docs/reference/cli.md
@@ -294,15 +294,15 @@ eve traces --json # dump the full trace as JSON
Reads the immutable OTLP/JSON segments under `.eve/traces/v1`, so `eve dev` need not be running. Accepts a full trace id, an `agent.session.id`, or an unambiguous prefix of either. Malformed segments are skipped without hiding valid spans from the same trace.
-Span rows carry inline metrics when the span recorded them — `↑input`/`↓output` token counts, gateway cost, and the tool name for `ai.toolCall` spans — and the header aggregates models, token totals, cost, and error count across the trace's step spans. `--verbose` expands each span under its tree row: status (with the error message on failures), timing, ids, every attribute (prompts, responses, and tool payloads as transcripts or pretty-printed JSON), and every span event with its offset from span start. `--json` prints the same records as JSON, one object per selected trace.
+Span rows carry inline metrics when the span recorded them — `↑input`/`↓output` token counts, gateway cost, and the tool name for `execute_tool` spans — and the header aggregates models, token totals, cost, and error count across the trace's step spans. `--verbose` expands each span under its tree row: status (with the error message on failures), timing, ids, every attribute (prompts, responses, and tool payloads as transcripts or pretty-printed JSON), and every span event with its offset from span start. `--json` prints the same records as JSON, one object per selected trace.
-A local subagent keeps its own session id but records into the parent trace. Its turn is parented to the `agent.action` span that dispatched it, so the span tree carries the relationship without duplicate lineage attributes; `agent.subagent.name` remains on the child turn as a standalone label. Either session id resolves to that trace. Remote agents propagate the parent trace context over `traceparent`.
+A local subagent keeps its own session id but records into the parent trace. Its `invoke_agent` span is parented to the `agent.action` span that dispatched it, so the span tree carries the relationship without duplicate lineage attributes; `agent.subagent.name` remains on the child invocation as a standalone label. Either session id resolves to that trace. Remote agents propagate the parent trace context over `traceparent`.
A durable session keeps one persisted trace context across turns and worker resumptions. Independently replayed attempts can still produce another trace; passing the session id shows every trace it produced, oldest first.
-Every span carries a real duration except `agent.session`: an idle session never closes, so it is recorded as a zero-duration marker and the span tree shows its descendant extent instead. A turn's span is written when the turn settles, so a running turn shows only its steps.
+Every span carries a real duration except `agent.session`: an idle session never closes, so it is recorded as a zero-duration marker and the span tree shows its descendant extent instead. A turn's `invoke_agent` span is written when the turn settles, so a running turn shows only its steps.
-Model and tool-call spans omit their inputs and outputs by default. Set `EVE_TRACES_CONTENT=on` to capture system prompts, prompt messages, and response text for models, plus call arguments and results for tools. Each captured value is capped at 32 KB.
+Model and `execute_tool` spans omit their inputs and outputs by default. Set `EVE_TRACES_CONTENT=on` to capture system prompts, prompt messages, and response text for models, plus call arguments and results for tools. Each captured value is capped at 32 KB.
Step spans carry token counts, and cost when Vercel AI Gateway served the call. Both follow the [OTel GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) (`gen_ai.usage.*`), so a third-party backend reads them without mapping.
diff --git a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
index f184cb17c..a97befb4a 100644
--- a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
+++ b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
@@ -87,6 +87,7 @@ export declare const propagation: {
};
export declare const trace: {
+ deleteSpan(context: Context): Context;
getActiveSpan(): Span | undefined;
getSpan(context: Context): Span | undefined;
getTracer(name: string, version?: string): Tracer;
diff --git a/packages/eve/src/cli/commands/trace.integration.test.ts b/packages/eve/src/cli/commands/trace.integration.test.ts
index d3d083a10..f1483241d 100644
--- a/packages/eve/src/cli/commands/trace.integration.test.ts
+++ b/packages/eve/src/cli/commands/trace.integration.test.ts
@@ -138,10 +138,11 @@ describe("eve traces", () => {
await writeSegment(
root,
TRACE_ONE,
- span(turn, "agent.turn", 10, 100, undefined, {
+ span(turn, "invoke_agent weather", 10, 100, undefined, {
"agent.name": "weather",
"agent.session.id": "session-one",
"agent.turn.id": "turn-1",
+ "gen_ai.operation.name": "invoke_agent",
}),
);
await writeSegment(
@@ -170,7 +171,7 @@ describe("eve traces", () => {
await runTraceShowCommand(output.logger, root, "session-one");
- expect(output.out[0]).toContain("agent.turn [turn-1]");
+ expect(output.out[0]).toContain("invoke_agent weather [turn-1]");
expect(output.out[0]).toContain("└─ agent.step [step 0, attempt 0]");
expect(output.out[0]).toContain("└─ agent.action [tool-call: weather]");
expect(output.out[0]).toContain("failed 10ms ERROR");
diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
index 7f8c6f4d4..8c7f9789e 100644
--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.test.ts
@@ -55,7 +55,10 @@ function delivery(
}
function weatherTurn(): LocalTraceSpan[] {
- const turn = span("a".repeat(16), "agent.turn", 0, 0, undefined, { "agent.turn.id": "turn_0" });
+ const turn = span("a".repeat(16), "invoke_agent weather", 0, 0, undefined, {
+ "agent.turn.id": "turn_0",
+ "gen_ai.operation.name": "invoke_agent",
+ });
const inbound = delivery("0".repeat(16), 0, "turn_0", "weather in sf?");
const step = span("b".repeat(16), "agent.step", 10, 5000, turn.spanId, {});
const messages = JSON.stringify([
@@ -110,7 +113,7 @@ describe("buildConversationItems", () => {
});
it("renders provider-executed tool results as tool cards after the assistant", () => {
- // Provider-executed tools (e.g. web_search) never get an ai.toolCall
+ // Provider-executed tools (e.g. web_search) never get an execute_tool
// span; their outcomes live on the model span's tool_results attribute.
const spans = weatherTurn().map((s) =>
s.name === "ai.streamText.doStream"
@@ -147,7 +150,9 @@ describe("buildConversationItems", () => {
it("finds turns parented to a session span", () => {
const session = span("0".repeat(16), "agent.session", 0, 0);
const spans = weatherTurn().map((s) =>
- s.name === "agent.turn" ? { ...s, parentSpanId: session.spanId } : s,
+ s.attributes["gen_ai.operation.name"] === "invoke_agent"
+ ? { ...s, parentSpanId: session.spanId }
+ : s,
);
const items = buildConversationItems(trace([session, ...spans]));
expect(items.map((item) => item.kind)).toEqual(["user", "assistant", "tool"]);
@@ -161,9 +166,17 @@ describe("buildConversationItems", () => {
"agent.action.name": "echo-marker",
"agent.turn.id": "turn_0",
});
- const childTurn = span("f".repeat(16), "agent.turn", 6000, 6000, parentAction.spanId, {
- "agent.turn.id": "turn_child",
- });
+ const childTurn = span(
+ "f".repeat(16),
+ "invoke_agent echo-marker",
+ 6000,
+ 6000,
+ parentAction.spanId,
+ {
+ "agent.turn.id": "turn_child",
+ "gen_ai.operation.name": "invoke_agent",
+ },
+ );
const childDelivery = delivery("3".repeat(16), 6000, "turn_child", "delegated task");
const childStep = span("1".repeat(16), "agent.step", 6010, 8000, childTurn.spanId, {});
const childModel = span(
diff --git a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts b/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts
index 148512fa5..53b5f60cc 100644
--- a/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts
+++ b/packages/eve/src/cli/dev/tui/traces/trace-conversation.ts
@@ -8,7 +8,7 @@
import { formatElapsed } from "#cli/format-elapsed.js";
import { clipVisible, stripTerminalControls, visibleLength } from "#cli/ui/terminal-text.js";
import type { LocalTrace, LocalTraceSpan } from "#tracing/local-trace-reader.js";
-import { compareLocalTraceSpans } from "#tracing/local-trace-reader.js";
+import { compareLocalTraceSpans, isAgentTurnSpan } from "#tracing/local-trace-reader.js";
import { formatCompactTokenCount } from "../stream-format.js";
import type { Theme } from "../theme.js";
@@ -61,7 +61,7 @@ export function buildConversationItems(trace: LocalTrace): ConversationItem[] {
const byId = new Map(trace.spans.map((span) => [span.spanId, span]));
const subagents = new Map<string, ConversationSubagent>();
for (const span of trace.spans) {
- if (span.name !== "agent.turn") continue;
+ if (!isAgentTurnSpan(span)) continue;
const turnId = stringAttribute(span, "agent.turn.id");
const subagent = turnSubagent(span, byId);
if (turnId !== undefined && subagent !== undefined) subagents.set(turnId, subagent);
diff --git a/packages/eve/src/execution/subagent-usage-span.test.ts b/packages/eve/src/execution/subagent-usage-span.test.ts
deleted file mode 100644
index dd221e00f..000000000
--- a/packages/eve/src/execution/subagent-usage-span.test.ts
+++ /dev/null
@@ -1,92 +0,0 @@
-import { beforeEach, describe, expect, it, vi } from "vitest";
-
-import { recordSubagentUsageSpans } from "#execution/subagent-usage-span.js";
-import type { RuntimeActionResult, RuntimeSubagentChildResult } from "#shared/action-types.js";
-
-const startSpanMock = vi.fn();
-const endSpanMock = vi.fn();
-
-vi.mock("#compiled/@opentelemetry/api/index.js", async (importOriginal) => {
- const actual = await importOriginal<typeof import("#compiled/@opentelemetry/api/index.js")>();
- return {
- ...actual,
- trace: {
- getTracer: () => ({
- startSpan: (name: string, options: unknown) => {
- startSpanMock(name, options);
- return { end: endSpanMock };
- },
- }),
- },
- };
-});
-
-const USAGE = { cacheReadTokens: 10, cacheWriteTokens: 5, inputTokens: 100, outputTokens: 50 };
-
-function subagentResult(overrides: Partial<RuntimeSubagentChildResult> = {}): RuntimeActionResult {
- return {
- callId: "call-1",
- kind: "subagent-result",
- origin: "child",
- outcome: {
- kind: "terminal",
- result: { kind: "succeeded", output: "done" },
- usageDelta: { cacheReadTokens: 0, cacheWriteTokens: 0, inputTokens: 0, outputTokens: 0 },
- },
- output: "done",
- subagentName: "research",
- ...overrides,
- };
-}
-
-describe("recordSubagentUsageSpans", () => {
- beforeEach(() => {
- startSpanMock.mockReset();
- endSpanMock.mockReset();
- });
-
- it("emits one invoke_agent span per successful result with usage", () => {
- recordSubagentUsageSpans([
- subagentResult({ usage: USAGE }),
- subagentResult({ callId: "call-2", subagentName: "writer", usage: USAGE }),
- ]);
-
- expect(startSpanMock).toHaveBeenCalledTimes(2);
- expect(startSpanMock).toHaveBeenCalledWith("invoke_agent research", {
- attributes: {
- "gen_ai.operation.name": "invoke_agent",
- "gen_ai.agent.name": "research",
- "gen_ai.usage.input_tokens": 100,
- "gen_ai.usage.output_tokens": 50,
- "gen_ai.usage.cache_read.input_tokens": 10,
- "gen_ai.usage.cache_creation.input_tokens": 5,
- },
- });
- expect(endSpanMock).toHaveBeenCalledTimes(2);
- });
-
- it("skips results without usage, error results, and non-subagent kinds", () => {
- recordSubagentUsageSpans([
- subagentResult(),
- subagentResult({ isError: true, usage: USAGE }),
- {
- callId: "call-3",
- kind: "tool-result",
- output: "ok",
- toolName: "search",
- },
- ]);
-
- expect(startSpanMock).not.toHaveBeenCalled();
- });
-
- it("does not throw when the tracer fails", () => {
- startSpanMock.mockImplementation(() => {
- throw new Error("tracer unavailable");
- });
-
- expect(() => {
- recordSubagentUsageSpans([subagentResult({ usage: USAGE })]);
- }).not.toThrow();
- });
-});
diff --git a/packages/eve/src/execution/subagent-usage-span.ts b/packages/eve/src/execution/subagent-usage-span.ts
deleted file mode 100644
index 8f5280f47..000000000
--- a/packages/eve/src/execution/subagent-usage-span.ts
+++ /dev/null
@@ -1,40 +0,0 @@
-import { trace } from "#compiled/@opentelemetry/api/index.js";
-
-import { createLogger } from "#internal/logging.js";
-import type { RuntimeActionResult } from "#shared/action-types.js";
-
-const log = createLogger("execution.subagent-usage-span");
-
-/**
- * Emits one caller-side `invoke_agent` span per successful delegated
- * subagent result that reports token usage, so the parent session's
- * observability can attribute a child agent's tokens. Runs in the
- * parent's turn step, so the span lands in the caller's trace.
- * Best-effort: a tracer failure never blocks the turn.
- */
-export function recordSubagentUsageSpans(results: readonly RuntimeActionResult[]): void {
- for (const result of results) {
- if (result.kind !== "subagent-result" || result.isError === true) {
- continue;
- }
- const usage = result.usage;
- if (usage === undefined) {
- continue;
- }
- try {
- const span = trace.getTracer("eve").startSpan(`invoke_agent ${result.subagentName}`, {
- attributes: {
- "gen_ai.operation.name": "invoke_agent",
- "gen_ai.agent.name": result.subagentName,
- "gen_ai.usage.input_tokens": usage.inputTokens,
- "gen_ai.usage.output_tokens": usage.outputTokens,
- "gen_ai.usage.cache_read.input_tokens": usage.cacheReadTokens,
- "gen_ai.usage.cache_creation.input_tokens": usage.cacheWriteTokens,
- },
- });
- span.end();
- } catch (error) {
- log.warn("failed to emit subagent usage span", { error });
- }
- }
-}
diff --git a/packages/eve/src/execution/workflow-runtime.test.ts b/packages/eve/src/execution/workflow-runtime.test.ts
index 484a455b2..cde919e79 100644
--- a/packages/eve/src/execution/workflow-runtime.test.ts
+++ b/packages/eve/src/execution/workflow-runtime.test.ts
@@ -1,5 +1,12 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { context as apiContext } from "@opentelemetry/api";
+import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
+import {
+ ROOT_CONTEXT,
+ context as otelContext,
+ trace as otelTrace,
+} from "#compiled/@opentelemetry/api/index.js";
import type { ChannelAdapter } from "#channel/adapter.js";
import { attachChannelActivityPresentation } from "#channel/activity-renderer.js";
import { ChannelRequestIdKey, ActivityObserverKey } from "#context/keys.js";
@@ -9,6 +16,7 @@ import {
LATEST_DEPLOYMENT_UNSUPPORTED_MESSAGE,
activityCollectorWorkflowReference,
sessionTimeoutWorkflowReference,
+ startWorkflowPreferLatest,
turnWorkflowReference,
workflowEntryReference,
} from "#execution/workflow-runtime.js";
@@ -18,6 +26,7 @@ import { registerInstrumentationRuntime } from "#instrumentation/runtime.js";
import { AgentSpanIdGenerator } from "#tracing/agent-span-id-generator.js";
import type { RuntimeCompiledArtifactsSource } from "#runtime/compiled-artifacts-source.js";
import { getCompiledRuntimeAgentBundle } from "#runtime/sessions/compiled-agent-cache.js";
+import { markAgentTraceContext } from "#tracing/agent-trace-context.js";
const getHookByTokenMock = vi.fn();
const getRunMock = vi.fn();
@@ -81,6 +90,42 @@ describe("workflowEntryReference", () => {
});
});
+describe("startWorkflowPreferLatest", () => {
+ it("detaches Workflow telemetry only from marked agent contexts", async () => {
+ const contextManager = new AsyncLocalStorageContextManager().enable();
+ apiContext.setGlobalContextManager(contextManager);
+ const caller = otelTrace.wrapSpanContext({
+ isRemote: false,
+ spanId: "2".repeat(16),
+ traceFlags: 1,
+ traceId: "1".repeat(32),
+ });
+ const callerContext = otelTrace.setSpan(ROOT_CONTEXT, caller);
+ const observedParents: unknown[] = [];
+ startMock.mockImplementation(async () => {
+ observedParents.push(otelTrace.getSpan(otelContext.active()));
+ return { runId: "run-1" };
+ });
+
+ try {
+ await otelContext.with(callerContext, async () => {
+ expect(otelTrace.getSpan(otelContext.active())).toBe(caller);
+ await startWorkflowPreferLatest(workflowEntryReference, []);
+ expect(otelTrace.getSpan(otelContext.active())).toBe(caller);
+ });
+ await otelContext.with(markAgentTraceContext(callerContext), async () => {
+ expect(otelTrace.getSpan(otelContext.active())).toBe(caller);
+ await startWorkflowPreferLatest(workflowEntryReference, []);
+ expect(otelTrace.getSpan(otelContext.active())).toBe(caller);
+ });
+ } finally {
+ apiContext.disable();
+ contextManager.disable();
+ }
+ expect(observedParents).toEqual([caller, undefined]);
+ });
+});
+
describe("createWorkflowRuntime command dispatch", () => {
const NOT_FOUND_TOKEN = "test:no-such-hook";
diff --git a/packages/eve/src/execution/workflow-runtime.ts b/packages/eve/src/execution/workflow-runtime.ts
index 764e2f3d9..cd795d399 100644
--- a/packages/eve/src/execution/workflow-runtime.ts
+++ b/packages/eve/src/execution/workflow-runtime.ts
@@ -1,5 +1,6 @@
import { randomBytes } from "node:crypto";
+import { context, trace } from "#compiled/@opentelemetry/api/index.js";
import {
EntityConflictError,
HookNotFoundError,
@@ -59,6 +60,7 @@ import {
} from "#execution/workflow-callback-url.js";
import { walkCauseChain } from "#shared/errors.js";
import { buildInvocationAttributes } from "#internal/invocation/metadata.js";
+import { isAgentTraceContext } from "#tracing/agent-trace-context.js";
import { sessionCommandHookToken } from "#execution/session-command-token.js";
import { resumeSessionInbox } from "#execution/wire/session-inbox-resume.js";
import type { DynamicSubagentAgentConfig } from "#runtime/subagents/dynamic-agent-config.js";
@@ -471,23 +473,32 @@ export async function startWorkflowPreferLatest<TArgs extends unknown[], TResult
args: TArgs,
options?: StartOptionsWithoutDeploymentId,
): Promise<Run<unknown> | Run<TResult>> {
- if (!shouldRouteToLatestDeployment()) {
- return options === undefined
- ? await start(workflow, args)
- : await start(workflow, args, options);
- }
-
- try {
- return await start(workflow, args, { ...options, deploymentId: "latest" });
- } catch (error) {
- if (!isLatestDeploymentUnsupportedError(error)) {
- throw error;
+ // Agent parentage is reconstructed from Eve's serialized trace context. Only
+ // remove the ambient span marked by an agent boundary; the marker is not
+ // propagated into Workflow runs, so Workflow-to-Workflow traces stay intact.
+ const activeContext = context.active();
+ const workflowContext = isAgentTraceContext(activeContext)
+ ? trace.deleteSpan(activeContext)
+ : activeContext;
+ return await context.with(workflowContext, async () => {
+ if (!shouldRouteToLatestDeployment()) {
+ return options === undefined
+ ? await start(workflow, args)
+ : await start(workflow, args, options);
}
- return options === undefined
- ? await start(workflow, args)
- : await start(workflow, args, options);
- }
+ try {
+ return await start(workflow, args, { ...options, deploymentId: "latest" });
+ } catch (error) {
+ if (!isLatestDeploymentUnsupportedError(error)) {
+ throw error;
+ }
+
+ return options === undefined
+ ? await start(workflow, args)
+ : await start(workflow, args, options);
+ }
+ });
}
/**
diff --git a/packages/eve/src/execution/workflow-steps.ts b/packages/eve/src/execution/workflow-steps.ts
index 78d7c987b..ad406ad54 100644
--- a/packages/eve/src/execution/workflow-steps.ts
+++ b/packages/eve/src/execution/workflow-steps.ts
@@ -97,7 +97,6 @@ import {
} from "#execution/tasks/child/instructions.js";
import { prepareWorkflowPreambleTrace } from "#execution/workflow-trace-context.js";
import { resolveEffectiveAgentRuntime } from "#execution/effective-agent-config.js";
-import { recordSubagentUsageSpans } from "#execution/subagent-usage-span.js";
import { reconcileSessionContinuationToken } from "#execution/reconcile-session-continuation-token.js";
import { hydrateDurableSession, refreshSessionFromTurnAgent } from "#execution/session.js";
import { createExecutionHistoryView } from "#execution/history-view.js";
@@ -255,7 +254,6 @@ export async function turnStep(rawInput: TurnStepInput): Promise<DurableStepResu
}
resolved = results.length === 0 ? undefined : results.reduce(coalesceTurnInputs);
} else if (input.input?.kind === "runtime-action-result") {
- recordSubagentUsageSpans(input.input.results);
if (input.input.acceptedAtMsByCallId !== undefined) {
ctx.set(RuntimeActionSettlementTimesKey, input.input.acceptedAtMsByCallId);
}
diff --git a/packages/eve/src/harness/turn-tag-state.ts b/packages/eve/src/harness/turn-tag-state.ts
index 9d8051daf..47b10948b 100644
--- a/packages/eve/src/harness/turn-tag-state.ts
+++ b/packages/eve/src/harness/turn-tag-state.ts
@@ -244,7 +244,7 @@ export function accumulateTurnUsage(input: {
* Folds a delegated child session's reported totals into the parent's
* session totals without touching the in-flight turn totals. Turn tags
* attribute only the parent's own model calls (child spend is attributed by
- * the caller-side `invoke_agent` span); session totals feed the session
+ * the caller's durable `agent.action` span); session totals feed the session
* token limits and the remaining-quota budget granted to later delegations.
*/
export function accumulateSessionUsage(input: {
diff --git a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts
index ec0b7466d..eed73210c 100644
--- a/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts
+++ b/packages/eve/src/instrumentation/ai-sdk-hook-bridge.test.ts
@@ -69,6 +69,43 @@ describe("createAiSdkHookBridge", () => {
]);
});
+ it("publishes usage once when the SDK repeats a model terminal callback", async () => {
+ let inputTokens = 0;
+ const completed = vi.fn((event: InstrumentationModelCallTerminalEvent) => {
+ if (event.type === "model.call.completed") {
+ inputTokens += event.usage.inputTokens ?? 0;
+ }
+ });
+ const bridge = createAiSdkHookBridge(
+ scope,
+ createInstrumentationHooks([
+ { events: { "model.call.completed": completed }, name: "usage" },
+ ]),
+ );
+ const execute = vi.fn(async () => "result");
+ await Reflect.apply(bridge.onLanguageModelCallStart!, bridge, [
+ { callId: "call-1", messages: [], modelId: "model", provider: "test", tools: undefined },
+ ]);
+ await bridge.executeLanguageModelCall!({ callId: "call-1", execute });
+ const terminal = {
+ callId: "call-1",
+ content: [],
+ finishReason: "stop",
+ performance: { responseTimeMs: 1 },
+ responseId: "response-1",
+ usage: { inputTokens: 10, outputTokens: 5 },
+ };
+
+ await Promise.all([
+ Reflect.apply(bridge.onLanguageModelCallEnd!, bridge, [terminal]),
+ Reflect.apply(bridge.onLanguageModelCallEnd!, bridge, [terminal]),
+ ]);
+
+ expect(execute).toHaveBeenCalledOnce();
+ expect(completed).toHaveBeenCalledOnce();
+ expect(inputTokens).toBe(10);
+ });
+
it("uses an eve-owned context runner while executing the model exactly once", async () => {
const order: string[] = [];
const hooks = createInstrumentationHooks([]);
diff --git a/packages/eve/src/instrumentation/providers.test.ts b/packages/eve/src/instrumentation/providers.test.ts
index 946b97f75..de9f93140 100644
--- a/packages/eve/src/instrumentation/providers.test.ts
+++ b/packages/eve/src/instrumentation/providers.test.ts
@@ -210,6 +210,7 @@ describe("finalizeInstrumentationProviders", () => {
const runtime = finalizeInstrumentationProviders({ serviceName: "weather-agent" });
await runtime.hooks.publish(turnStarted);
+ expect(runtime.instrumentationProviders).toBe(true);
expect(started).toHaveBeenCalledOnce();
expect(started.mock.calls[0]?.[0]).toMatchObject({ turnId: "turn-1" });
});
diff --git a/packages/eve/src/instrumentation/providers.ts b/packages/eve/src/instrumentation/providers.ts
index 64f752079..320ef3e9f 100644
--- a/packages/eve/src/instrumentation/providers.ts
+++ b/packages/eve/src/instrumentation/providers.ts
@@ -120,6 +120,7 @@ export function finalizeInstrumentationProviders(input: {
return installInstrumentationRuntime({
collected,
frameworkVersion: resolveInstalledPackageInfo().version,
+ instrumentationProviders: true,
providers: providerDefinitions,
runtimeContextResolvers: collected.runtimeContextResolvers,
serviceName: input.serviceName,
diff --git a/packages/eve/src/instrumentation/runtime.ts b/packages/eve/src/instrumentation/runtime.ts
index cf3a44275..aa86b554e 100644
--- a/packages/eve/src/instrumentation/runtime.ts
+++ b/packages/eve/src/instrumentation/runtime.ts
@@ -146,6 +146,7 @@ export interface InstrumentationRuntime {
readonly forceFlush: () => Promise<void>;
readonly hooks: InstrumentationHooks;
readonly idGenerator?: AgentSpanIdGenerator;
+ readonly instrumentationProviders?: boolean;
readonly prepareSessionTrace?: (
event: InstrumentationSessionStartedEvent,
) => Promise<InstrumentationTraceSeed>;
diff --git a/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts b/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts
index 40c05be95..a8effb8bd 100644
--- a/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts
+++ b/packages/eve/src/internal/nitro/routes/channel-dispatch.test.ts
@@ -13,7 +13,10 @@ import {
} from "@opentelemetry/sdk-trace-base";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
-import { trace as vendoredTrace } from "#compiled/@opentelemetry/api/index.js";
+import {
+ context as vendoredContext,
+ trace as vendoredTrace,
+} from "#compiled/@opentelemetry/api/index.js";
import {
CHANNEL_SENTINEL,
@@ -22,10 +25,12 @@ import {
} from "#channel/compiled-channel.js";
import type { RouteHandlerArgs } from "#channel/routes.js";
import { registerInstrumentationConfig } from "#instrumentation/config.js";
+import { getInstrumentationRuntime } from "#instrumentation/runtime.js";
import type { Runtime } from "#channel/types.js";
import { readVercelProjectLink } from "#internal/vercel/project-link.js";
import { resolveVercelOidcCurrentProject } from "#channel/auth/vercel-oidc-project.js";
import type { ResolvedChannelDefinition } from "#runtime/types.js";
+import { isAgentTraceContext } from "#tracing/agent-trace-context.js";
import {
dispatchChannelRequest,
dispatchChannelWebSocketRequest,
@@ -729,13 +734,40 @@ describe("dispatchChannelRequest tracing", () => {
return exporter.getFinishedSpans();
}
+ it("uses the semantic request name for the experimental provider layout", async () => {
+ const instrumentationRuntime = getInstrumentationRuntime();
+ if (instrumentationRuntime === undefined)
+ throw new Error("Expected an instrumentation runtime.");
+ Object.defineProperty(instrumentationRuntime, "instrumentationProviders", {
+ configurable: true,
+ value: true,
+ });
+ try {
+ mockedResolveNitroChannelRuntimeBundle.mockResolvedValue({
+ agentName: "test-agent",
+ channels: [slackChannel(async () => new Response("ok"))],
+ runtime,
+ });
+
+ await dispatchChannelRequest(createEvent({ waitUntil: vi.fn() }), "POST /slack", {} as never);
+
+ const [span] = await finishedSpans();
+ expect(span!.name).toBe("agent.channel.request");
+ expect(span!.attributes["http.route"]).toBe("/slack");
+ } finally {
+ Reflect.deleteProperty(instrumentationRuntime, "instrumentationProviders");
+ }
+ });
+
it("emits one SERVER span named for the route with method, channel, and status attributes", async () => {
+ let agentContextDuringHandler = false;
let spansDuringHandler = -1;
mockedResolveNitroChannelRuntimeBundle.mockResolvedValue({
agentName: "test-agent",
channels: [
slackChannel(
async () => {
+ agentContextDuringHandler = isAgentTraceContext(vendoredContext.active());
spansDuringHandler = exporter.getFinishedSpans().length;
return new Response("ok");
},
@@ -752,6 +784,7 @@ describe("dispatchChannelRequest tracing", () => {
);
expect(response.status).toBe(200);
+ expect(agentContextDuringHandler).toBe(true);
// The span had not ended while the handler was still running.
expect(spansDuringHandler).toBe(0);
diff --git a/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts b/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts
index 7ddb9454a..91247e872 100644
--- a/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts
+++ b/packages/eve/src/internal/nitro/routes/channel-request-instrumentation.ts
@@ -10,6 +10,7 @@ import {
} from "#compiled/@opentelemetry/api/index.js";
import { getInstrumentationRuntime } from "#instrumentation/runtime.js";
import { recordErrorOnSpan } from "#internal/logging.js";
+import { markAgentTraceContext } from "#tracing/agent-trace-context.js";
/**
* Stable tracer name for every inbound eve channel HTTP request. Kept
@@ -42,12 +43,10 @@ export interface TraceChannelRequestInput {
*
* The span is named for the low-cardinality registered `routeKey`
* (`"POST /eve/v1/session/:sessionId"`), never the concrete URL, while the
- * `http.route` attribute carries only the path template — the OTel HTTP
- * semantic convention reserves `http.route` for the route template alone,
- * with the method in `http.request.method`. The span is started from the
- * context extracted off the incoming request headers so an upstream
- * `traceparent` becomes its parent and nested channel and Workflow spans
- * (`hook.resume` and outgoing HTTP) become its descendants.
+ * `http.route` attribute carries only the path template and
+ * `http.request.method` carries the method. It starts from context extracted
+ * off the incoming request headers so an upstream `traceparent` becomes its
+ * parent and nested channel operations become its descendants.
*
* The handler runs inside the span's active context. When it returns, the
* response status is recorded and a `>= 500` status marks the span as an
@@ -75,14 +74,18 @@ export async function traceChannelRequest<T extends Response>(
const { request, routeKey } = input;
const parentContext = propagation.extract(context.active(), request.headers, headersGetter);
+ const spanName =
+ getInstrumentationRuntime()?.instrumentationProviders === true
+ ? "agent.channel.request"
+ : routeKey;
const span = trace
.getTracer(TRACER_NAME)
.startSpan(
- routeKey,
+ spanName,
{ attributes: baseAttributes(request, routeKey), kind: SpanKind.SERVER },
parentContext,
);
- const activeContext = trace.setSpan(parentContext, span);
+ const activeContext = markAgentTraceContext(trace.setSpan(parentContext, span));
try {
const response = await context.with(activeContext, () => handler(span));
diff --git a/packages/eve/src/tracing/agent-action-instrumentation.ts b/packages/eve/src/tracing/agent-action-instrumentation.ts
index 1be39377c..1f2b0e9a6 100644
--- a/packages/eve/src/tracing/agent-action-instrumentation.ts
+++ b/packages/eve/src/tracing/agent-action-instrumentation.ts
@@ -102,7 +102,9 @@ export function createAgentActionInstrumentation(input: {
} else if (event.output.type === "error") {
recordError(span, event.output.error);
} else {
- if (event.usage !== undefined) setAgentUsage(span, event.usage);
+ if (event.usage !== undefined) {
+ setAgentUsage(span, event.usage, { includeGenAiDetails: false });
+ }
if (input.recordOutputs) {
const result = contentAttribute(event.output.output, false);
if (result !== undefined) span.setAttribute("gen_ai.tool.call.result", result);
diff --git a/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts b/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts
index 331472372..7a1188e20 100644
--- a/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts
+++ b/packages/eve/src/tracing/agent-channel-delivery-instrumentation.ts
@@ -98,15 +98,6 @@ export function createAgentChannelDeliveryInstrumentation(input: {
event.turnId === undefined
? undefined
: await input.stateStore.getTurn(event.sessionId, event.turnId);
- const parent =
- turn === undefined
- ? state.parent
- : {
- isRemote: turn.parentIsRemote ?? false,
- spanId: turn.parentSpanId,
- traceFlags: turn.context.traceFlags,
- traceId: turn.context.traceId,
- };
const startTimeMs =
turn === undefined ? state.startTimeMs : Math.max(state.startTimeMs, turn.startTimeMs);
const requestLink = state.requestTraceContext;
@@ -139,7 +130,7 @@ export function createAgentChannelDeliveryInstrumentation(input: {
],
startTime: startTimeMs,
},
- contextFromSpanContext(parent),
+ contextFromSpanContext(state.parent),
),
);
span.addEvent("channel.delivery.started", undefined, startTimeMs);
@@ -157,6 +148,19 @@ export function createAgentChannelDeliveryInstrumentation(input: {
recordError(span, event.error);
}
span.end();
+ if (
+ turn !== undefined &&
+ event.turnId !== undefined &&
+ turn.parentSpanId === state.parent.spanId &&
+ turn.context.traceId === state.parent.traceId
+ ) {
+ await input.stateStore.updateTurn(event.sessionId, event.turnId, (current) =>
+ current.parentSpanId === state.parent.spanId &&
+ current.context.traceId === state.parent.traceId
+ ? { ...current, parentIsRemote: false, parentSpanId: state.spanId }
+ : current,
+ );
+ }
};
return {
diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.ts
index 3996dda3a..8e732986a 100644
--- a/packages/eve/src/tracing/agent-otel-provider.test.ts
+++ b/packages/eve/src/tracing/agent-otel-provider.test.ts
@@ -386,7 +386,7 @@ describe("createAgentOtelInstrumentation", () => {
expect(replayedTrace).toEqual(turnTrace);
const spans = runtime.exporter.getFinishedSpans();
const session = byName(spans, "agent.session")[0]!;
- const turn = byName(spans, "agent.turn")[0]!;
+ const turn = byName(spans, "invoke_agent weather")[0]!;
expect(session.spanContext()).toMatchObject({
spanId: turnTrace.spanId,
traceFlags: turnTrace.traceFlags,
@@ -669,22 +669,23 @@ describe("createAgentOtelInstrumentation", () => {
traceFlags: 1,
traceId: "11111111111111111111111111111111",
};
- const delivery = {
+ const deliveries = ["first", "second"].map((suffix) => ({
channelKind: "http",
channelName: "eve",
- deliveryId: "delivery-remote",
- };
- const idempotencyKey = `channel-delivery:remote-session:${delivery.deliveryId}`;
+ deliveryId: `delivery-remote-${suffix}`,
+ }));
await contextStorage.run(ctx, async () => {
- await runtime.hooks.publish({
- delivery,
- idempotencyKey,
- parentTraceContext,
- rootSessionId: "parent-session",
- sessionId: "remote-session",
- type: "channel.delivery.started",
- });
+ for (const delivery of deliveries) {
+ await runtime.hooks.publish({
+ delivery,
+ idempotencyKey: `channel-delivery:remote-session:${delivery.deliveryId}`,
+ parentTraceContext,
+ rootSessionId: "parent-session",
+ sessionId: "remote-session",
+ type: "channel.delivery.started",
+ });
+ }
await runtime.hooks.publish({
agentName: "weather",
channelKind: "http",
@@ -703,27 +704,40 @@ describe("createAgentOtelInstrumentation", () => {
turnId: "turn-remote",
type: "turn.started",
});
- await runtime.hooks.publish({
- delivery,
- idempotencyKey,
- outcome: "completed",
- rootSessionId: "parent-session",
- sequence: 0,
- sessionId: "remote-session",
- turnId: "turn-remote",
- type: "channel.delivery.completed",
- });
+ for (const delivery of deliveries) {
+ await runtime.hooks.publish({
+ delivery,
+ idempotencyKey: `channel-delivery:remote-session:${delivery.deliveryId}`,
+ outcome: "completed",
+ rootSessionId: "parent-session",
+ sequence: 0,
+ sessionId: "remote-session",
+ turnId: "turn-remote",
+ type: "channel.delivery.completed",
+ });
+ }
await completeTurn(runtime.hooks, "remote-session", "turn-remote");
});
await runtime.provider.forceFlush();
const spans = runtime.exporter.getFinishedSpans();
- const channelDelivery = byName(spans, "agent.channel.delivery")[0]!;
- const turn = byName(spans, "agent.turn")[0]!;
+ const channelDeliveries = byName(spans, "agent.channel.delivery");
+ const firstDelivery = channelDeliveries.find(
+ (span) => span.attributes["agent.channel.delivery.id"] === "delivery-remote-first",
+ )!;
+ const turn = byName(spans, "invoke_agent")[0]!;
expect(byName(spans, "agent.session")).toHaveLength(0);
- expect(channelDelivery.parentSpanContext).toMatchObject(parentTraceContext);
- expect(turn.parentSpanContext).toMatchObject(parentTraceContext);
+ expect(channelDeliveries).toHaveLength(2);
+ for (const delivery of channelDeliveries) {
+ expect(delivery.parentSpanContext).toMatchObject(parentTraceContext);
+ }
+ expect(turn.parentSpanContext?.spanId).toBe(firstDelivery.spanContext().spanId);
expect(turn.spanContext().traceId).toBe(parentTraceContext.traceId);
+ expect(turn.attributes).toMatchObject({
+ "gen_ai.conversation.id": "remote-session",
+ "gen_ai.operation.name": "invoke_agent",
+ });
+ expect(turn.attributes).not.toHaveProperty("gen_ai.agent.name");
});
it("emits the agent hierarchy in one session trace", async () => {
@@ -749,12 +763,11 @@ describe("createAgentOtelInstrumentation", () => {
await runtime.provider.forceFlush();
const spans = runtime.exporter.getFinishedSpans();
- const turn = byName(spans, "agent.turn")[0]!;
+ const turn = byName(spans, "invoke_agent weather")[0]!;
const step = byName(spans, "agent.step")[0]!;
- const operation = byName(spans, "ai.streamText")[0]!;
const model = byName(spans, "chat claude-test")[0]!;
const action = byName(spans, "agent.action")[0]!;
- const tool = byName(spans, "ai.toolCall")[0]!;
+ const tool = byName(spans, "execute_tool weather")[0]!;
const session = byName(spans, "agent.session")[0]!;
expect(session.parentSpanContext).toBeUndefined();
@@ -762,18 +775,17 @@ describe("createAgentOtelInstrumentation", () => {
expect(session.attributes).toMatchObject({
"agent.channel.audience": "public",
"agent.session.id": "session-1",
- "agent.trace.schema.version": 2,
+ "agent.trace.schema.version": 3,
});
expect(turn.parentSpanContext?.spanId).toBe(session.spanContext().spanId);
expect(step.parentSpanContext?.spanId).toBe(turn.spanContext().spanId);
- expect(operation.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
expect(step.links).toEqual([
expect.objectContaining({
attributes: { "eve.link.type": "workflow.delivery" },
context: delivery.spanContext(),
}),
]);
- expect(model.parentSpanContext?.spanId).toBe(operation.spanContext().spanId);
+ expect(model.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
expect(
executionParents.some(
(parent) =>
@@ -782,9 +794,10 @@ describe("createAgentOtelInstrumentation", () => {
).toBe(true);
expect(action.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
expect(tool.parentSpanContext?.spanId).toBe(action.spanContext().spanId);
- for (const span of [turn, step, operation, model, action, tool]) {
+ for (const span of [turn, step, model, action, tool]) {
expect(span.attributes).not.toHaveProperty("agent.channel.audience");
}
+ expect(byName(spans, "ai.streamText")).toHaveLength(0);
expect(
new Set(
spans
@@ -795,7 +808,15 @@ describe("createAgentOtelInstrumentation", () => {
expect(turn.events.map((event) => event.name)).toEqual(["turn.started", "turn.completed"]);
// Turn timestamps are millisecond-quantized (`Date.now`), so the end
// comparison against the step's sub-millisecond clock gets 1ms of slack.
- expect(turn.attributes).toMatchObject({ "agent.name": "weather" });
+ expect(turn.kind).toBe(SpanKind.INTERNAL);
+ expect(turn.attributes).toMatchObject({
+ "agent.name": "weather",
+ "gen_ai.agent.name": "weather",
+ "gen_ai.conversation.id": "session-1",
+ "gen_ai.operation.name": "invoke_agent",
+ "gen_ai.usage.input_tokens": 10,
+ "gen_ai.usage.output_tokens": 5,
+ });
expect(nanos(turn.startTime)).toBeLessThanOrEqual(nanos(step.startTime));
expect(nanos(turn.endTime)).toBeGreaterThanOrEqual(nanos(step.endTime) - 1_000_000n);
expect(step.attributes).toMatchObject({
@@ -812,6 +833,14 @@ describe("createAgentOtelInstrumentation", () => {
"agent.action.name": "weather",
"agent.framework.name": "eve",
});
+ expect(tool.kind).toBe(SpanKind.INTERNAL);
+ expect(tool.attributes).toMatchObject({
+ "gen_ai.operation.name": "execute_tool",
+ "gen_ai.tool.call.id": "tool-1",
+ "gen_ai.tool.name": "weather",
+ "gen_ai.tool.type": "function",
+ });
+ expect(tool.attributes).not.toHaveProperty("gen_ai.agent.name");
});
it.each(["private", "unknown"] as const)(
@@ -952,7 +981,7 @@ describe("createAgentOtelInstrumentation", () => {
expect(byName(runtime.exporter.getFinishedSpans(), "agent.session")).toHaveLength(1);
});
- it("writes merged runtime context onto the step, operation, and chat spans", async () => {
+ it("writes merged runtime context onto the step and chat spans", async () => {
const runtime = createRuntime();
await emitAttempt({
@@ -972,9 +1001,8 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const step = byName(spans, "agent.step")[0]!;
- const operation = byName(spans, "ai.streamText")[0]!;
const model = byName(spans, "chat claude-test")[0]!;
- for (const span of [step, operation, model]) {
+ for (const span of [step, model]) {
expect(span.attributes).toMatchObject({
"ai.settings.context.eve.session.id": "session-1",
"ai.settings.context.posthog.distinct_id": "user-123",
@@ -998,9 +1026,8 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const step = byName(spans, "agent.step")[0]!;
- const operation = byName(spans, "ai.streamText")[0]!;
const model = byName(spans, "chat claude-test")[0]!;
- for (const span of [step, operation, model]) {
+ for (const span of [step, model]) {
expect(
Object.keys(span.attributes).some((key) => key.startsWith("ai.settings.context.")),
).toBe(false);
@@ -1030,16 +1057,19 @@ describe("createAgentOtelInstrumentation", () => {
const firstSpans = first.exporter.getFinishedSpans();
const redeliverySpans = redelivery.exporter.getFinishedSpans();
- for (const name of ["agent.turn", "agent.step", "agent.action", "ai.toolCall"]) {
+ for (const name of [
+ "invoke_agent weather",
+ "agent.step",
+ "agent.action",
+ "execute_tool weather",
+ ]) {
expect(byName(redeliverySpans, name)[0]!.spanContext().spanId).toBe(
byName(firstSpans, name)[0]!.spanContext().spanId,
);
}
- for (const name of ["ai.streamText", "chat claude-test"]) {
- expect(byName(redeliverySpans, name)[0]!.spanContext().spanId).not.toBe(
- byName(firstSpans, name)[0]!.spanContext().spanId,
- );
- }
+ expect(byName(redeliverySpans, "chat claude-test")[0]!.spanContext().spanId).not.toBe(
+ byName(firstSpans, "chat claude-test")[0]!.spanContext().spanId,
+ );
});
it("parents a tool to its action when SDK telemetry arrives first", async () => {
@@ -1121,9 +1151,10 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const action = byName(spans, "agent.action")[0]!;
- const [tool, uncorrelatedTool] = byName(spans, "ai.toolCall");
- expect(tool!.parentSpanContext?.spanId).toBe(action.spanContext().spanId);
- expect(uncorrelatedTool!.parentSpanContext?.spanId).toBe(
+ const tool = byName(spans, "execute_tool weather")[0]!;
+ const uncorrelatedTool = byName(spans, "execute_tool final_output")[0]!;
+ expect(tool.parentSpanContext?.spanId).toBe(action.spanContext().spanId);
+ expect(uncorrelatedTool.parentSpanContext?.spanId).toBe(
byName(spans, "agent.step")[0]!.spanContext().spanId,
);
});
@@ -1212,7 +1243,7 @@ describe("createAgentOtelInstrumentation", () => {
const replacementSpans = replacement.exporter.getFinishedSpans();
const action = byName(replacementSpans, "agent.action")[0]!;
- const tool = byName(replacementSpans, "ai.toolCall")[0]!;
+ const tool = byName(replacementSpans, "execute_tool weather")[0]!;
expect(action.spanContext().spanId).toBe(tool.parentSpanContext?.spanId);
expect(action.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
expect(action.attributes).toMatchObject({
@@ -1356,7 +1387,7 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
expect(byName(spans, "chat claude-test")).toHaveLength(1);
expect(byName(spans, "agent.action")).toHaveLength(0);
- expect(byName(spans, "ai.toolCall")).toHaveLength(1);
+ expect(byName(spans, "execute_tool weather")).toHaveLength(1);
expect(byName(spans, "agent.step")).toHaveLength(1);
});
@@ -1376,9 +1407,10 @@ describe("createAgentOtelInstrumentation", () => {
await runtime.provider.forceFlush();
const spans = runtime.exporter.getFinishedSpans();
- for (const name of ["chat claude-test", "agent.action", "ai.toolCall"]) {
+ for (const name of ["chat claude-test", "agent.action", "execute_tool weather"]) {
const span = byName(spans, name)[0]!;
expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: error.message });
+ if (name !== "agent.action") expect(span.attributes["error.type"]).toBe("Error");
expect(span.events).toContainEqual(
expect.objectContaining({
attributes: expect.objectContaining({ "exception.message": error.message }),
@@ -1410,11 +1442,14 @@ describe("createAgentOtelInstrumentation", () => {
"agent.action.kind": "subagent-call",
"agent.action.name": "weather",
"agent.action.outcome": "completed",
+ "agent.usage.cache_read_tokens": 3,
+ "agent.usage.cache_write_tokens": 4,
"agent.usage.input_tokens": 10,
"agent.usage.output_tokens": 5,
- "gen_ai.usage.cache_creation.input_tokens": 4,
- "gen_ai.usage.cache_read.input_tokens": 3,
});
+ expect(
+ Object.keys(action?.attributes ?? {}).some((key) => key.startsWith("gen_ai.usage.")),
+ ).toBe(false);
});
it("captures model and tool inputs/outputs on the operation spans", async () => {
@@ -1430,7 +1465,7 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const model = byName(spans, "chat claude-test")[0]!;
- const tool = byName(spans, "ai.toolCall")[0]!;
+ const tool = byName(spans, "execute_tool weather")[0]!;
// Provider transport noise (signatures et al.) is stripped at capture time.
expect(model.attributes["ai.prompt.messages"]).toBe(
'[{"content":"real user text","role":"user"}]',
@@ -1491,10 +1526,10 @@ describe("createAgentOtelInstrumentation", () => {
expect(byName(spans, "chat claude-test")[0]?.attributes).not.toHaveProperty(
"gen_ai.output.messages",
);
- expect(byName(spans, "ai.toolCall")[0]?.attributes).not.toHaveProperty(
+ expect(byName(spans, "execute_tool weather")[0]?.attributes).not.toHaveProperty(
"gen_ai.tool.call.arguments",
);
- expect(byName(spans, "ai.toolCall")[0]?.attributes).not.toHaveProperty(
+ expect(byName(spans, "execute_tool weather")[0]?.attributes).not.toHaveProperty(
"gen_ai.tool.call.result",
);
});
@@ -1725,8 +1760,8 @@ describe("createAgentOtelInstrumentation", () => {
await secondRuntime.provider.forceFlush();
const turns = [
- ...byName(firstRuntime.exporter.getFinishedSpans(), "agent.turn"),
- ...byName(secondRuntime.exporter.getFinishedSpans(), "agent.turn"),
+ ...byName(firstRuntime.exporter.getFinishedSpans(), "invoke_agent weather"),
+ ...byName(secondRuntime.exporter.getFinishedSpans(), "invoke_agent weather"),
];
expect(turns).toHaveLength(2);
expect(turns[0]!.spanContext().traceId).toBe(turns[1]!.spanContext().traceId);
@@ -1775,9 +1810,11 @@ describe("createAgentOtelInstrumentation", () => {
// The replacement's session transition emits the turn span with the span
// id the first worker allocated, so descendants from both workers attach.
- expect(byName(firstRuntime.exporter.getFinishedSpans(), "agent.turn")).toHaveLength(0);
+ expect(byName(firstRuntime.exporter.getFinishedSpans(), "invoke_agent weather")).toHaveLength(
+ 0,
+ );
const replacementSpans = replacementRuntime.exporter.getFinishedSpans();
- const turn = byName(replacementSpans, "agent.turn")[0]!;
+ const turn = byName(replacementSpans, "invoke_agent weather")[0]!;
const step = byName(replacementSpans, "agent.step")[0]!;
const action = byName(replacementSpans, "agent.action")[0]!;
@@ -1804,8 +1841,11 @@ describe("createAgentOtelInstrumentation", () => {
turnSequence: 0,
});
- const firstTurn = byName(firstRuntime.exporter.getFinishedSpans(), "agent.turn")[0]!;
- const replayTurn = byName(replayRuntime.exporter.getFinishedSpans(), "agent.turn")[0]!;
+ const firstTurn = byName(firstRuntime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;
+ const replayTurn = byName(
+ replayRuntime.exporter.getFinishedSpans(),
+ "invoke_agent weather",
+ )[0]!;
// The abandoned attempt keeps its own trace rather than interleaving with
// the retry. Both carry `agent.session.id`, so the session view still
// resolves to every trace the session produced.
@@ -1831,7 +1871,7 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
const sessions = byName(spans, "agent.session");
- const turns = byName(spans, "agent.turn");
+ const turns = byName(spans, "invoke_agent weather");
expect(sessions).toHaveLength(1);
expect(turns).toHaveLength(turnCount);
@@ -1869,7 +1909,7 @@ describe("createAgentOtelInstrumentation", () => {
await runtime.provider.forceFlush();
const spans = runtime.exporter.getFinishedSpans();
- const childTurn = byName(spans, "agent.turn").find(
+ const childTurn = byName(spans, "invoke_agent weather").find(
(span) => span.attributes["agent.session.id"] === "child-1",
)!;
@@ -1901,7 +1941,7 @@ describe("createAgentOtelInstrumentation", () => {
await completeTurn(runtime.hooks, "remote-child", "child-turn-1");
await runtime.provider.forceFlush();
- const childTurn = byName(runtime.exporter.getFinishedSpans(), "agent.turn")[0]!;
+ const childTurn = byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;
expect(childTurn.spanContext().traceId).toBe(parentTraceContext.traceId);
expect(childTurn.parentSpanContext).toMatchObject(parentTraceContext);
});
@@ -1949,9 +1989,9 @@ describe("createAgentOtelInstrumentation", () => {
const spans = runtime.exporter.getFinishedSpans();
expect(byName(spans, "agent.session")).toHaveLength(1);
- expect(byName(spans, "agent.turn")).toHaveLength(201);
+ expect(byName(spans, "invoke_agent weather")).toHaveLength(201);
expect(
- byName(spans, "agent.turn").every(
+ byName(spans, "invoke_agent weather").every(
(turn) => turn.spanContext().traceId === parentSession.spanContext().traceId,
),
).toBe(true);
@@ -1977,7 +2017,37 @@ describe("createAgentOtelInstrumentation", () => {
"agent.action.outcome": "failed",
"error.type": "TOOL_CALL_FAILED",
});
- expect(byName(spans, "agent.turn")[0]!.status.code).toBe(SpanStatusCode.UNSET);
+ expect(byName(spans, "invoke_agent weather")[0]!.status.code).toBe(SpanStatusCode.UNSET);
+ });
+
+ it("records error.type on a failed agent invocation", async () => {
+ const runtime = createRuntime();
+ const error = new TypeError("turn failed");
+ await publishTurnStarted({
+ hooks: runtime.hooks,
+ sessionId: "session-1",
+ turnId: "turn-1",
+ turnSequence: 0,
+ });
+ await runtime.hooks.publish({
+ error,
+ idempotencyKey: turnIdempotencyKey("session-1", "turn-1"),
+ sessionId: "session-1",
+ turnId: "turn-1",
+ type: "turn.failed",
+ });
+ await runtime.hooks.publish({
+ error,
+ idempotencyKey: sessionIdempotencyKey("session-1"),
+ sessionId: "session-1",
+ turnId: "turn-1",
+ type: "session.failed",
+ });
+ await runtime.provider.forceFlush();
+
+ const turn = byName(runtime.exporter.getFinishedSpans(), "invoke_agent weather")[0]!;
+ expect(turn.status).toEqual({ code: SpanStatusCode.ERROR, message: error.message });
+ expect(turn.attributes["error.type"]).toBe("TypeError");
});
});
diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.ts
index aa4765cf5..1b65fab01 100644
--- a/packages/eve/src/tracing/agent-otel-provider.ts
+++ b/packages/eve/src/tracing/agent-otel-provider.ts
@@ -1,5 +1,6 @@
import {
ROOT_CONTEXT,
+ SpanKind,
SpanStatusCode,
context,
type Context,
@@ -13,7 +14,7 @@ import { contextStorage } from "#context/container.js";
import { SessionTraceSeedKey } from "#context/keys.js";
import { withoutInstrumentationContent } from "#instrumentation/content.js";
import { instrumentationEventForTraceDecision } from "#instrumentation/content-policy.js";
-import type { AgentTraceStateStore } from "#tracing/agent-trace-state.js";
+import type { AgentTraceStateStore, AgentTurnTraceState } from "#tracing/agent-trace-state.js";
import {
contentAttribute,
genAiInputMessagesAttribute,
@@ -29,6 +30,8 @@ import { createAgentActionInstrumentation } from "#tracing/agent-action-instrume
import { createAgentApprovalInstrumentation } from "#tracing/agent-approval-instrumentation.js";
import { createAgentChannelDeliveryInstrumentation } from "#tracing/agent-channel-delivery-instrumentation.js";
import { createAgentToolInstrumentation } from "#tracing/agent-tool-instrumentation.js";
+import { markAgentTraceContext } from "#tracing/agent-trace-context.js";
+import { runtimeContextAttributes } from "#tracing/agent-otel-runtime-context.js";
import { setAgentUsage } from "#tracing/agent-otel-usage.js";
import { createAgentOtelSessionContext } from "#tracing/agent-otel-session-context.js";
import type { TraceCapturePolicy } from "#tracing/otel-declaration.js";
@@ -59,11 +62,6 @@ interface SpanState {
readonly span: Span;
}
-interface AttemptSpanState {
- readonly operation: SpanState & { readonly name: string };
- readonly step: SpanState;
-}
-
export interface AgentOtelInstrumentationInput {
/**
* Whether to write model prompts and tool call inputs onto spans at all.
@@ -92,7 +90,7 @@ export interface AgentOtelInstrumentation {
readonly runInContext: InstrumentationContextRunner;
}
-/** Creates OTel instrumentation for eve's structural `agent.*` convention. */
+/** Creates OTel instrumentation for eve's structural and GenAI spans. */
export function createAgentOtelInstrumentation(
input: AgentOtelInstrumentationInput,
): AgentOtelInstrumentation {
@@ -103,7 +101,7 @@ export function createAgentOtelInstrumentation(
// A serverless turn runs inside one `turnStep` "use step" invocation. If
// that worker is lost, Workflow retries the whole step from entry rather
// than resuming this callback sequence in a replacement process.
- const steps = new WeakMap<InstrumentationAttemptScope, AttemptSpanState>();
+ const steps = new WeakMap<InstrumentationAttemptScope, SpanState>();
const modelSpans = new WeakMap<InstrumentationAttemptScope, Map<string, SpanState>>();
const actions = createAgentActionInstrumentation({
frameworkVersion: input.frameworkVersion,
@@ -130,7 +128,7 @@ export function createAgentOtelInstrumentation(
recordOutputs,
resolveFallback: (event) => {
const scope = attemptScopes.get(event.scope.attemptId) ?? event.scope;
- const step = steps.get(scope)?.step;
+ const step = steps.get(scope);
return step === undefined
? undefined
: { context: step.context, spanContext: step.span.spanContext() };
@@ -217,27 +215,7 @@ export function createAgentOtelInstrumentation(
);
stepSpan.addEvent("step.started");
const stepContext = trace.setSpan(turnContext, stepSpan);
- const operationName = event.operation.operationId;
- const operationSpan = input.tracer.startSpan(
- operationName,
- {
- attributes: {
- "ai.operation.name": operationName,
- "ai.provider.name": event.operation.provider,
- "ai.request.model": event.operation.modelId,
- ...runtimeContextAttributes(event.runtimeContext),
- },
- },
- stepContext,
- );
- steps.set(event.scope, {
- operation: {
- context: trace.setSpan(stepContext, operationSpan),
- name: operationName,
- span: operationSpan,
- },
- step: { context: stepContext, span: stepSpan },
- });
+ steps.set(event.scope, { context: stepContext, span: stepSpan });
attemptScopes.set(event.scope.attemptId, event.scope);
};
@@ -257,15 +235,13 @@ export function createAgentOtelInstrumentation(
if (attempt === undefined) return;
// The span event drops the `attempt` segment: this span *is* one attempt,
// and `agent.step.attempt` on it already says which.
- attempt.step.span.addEvent(
+ attempt.span.addEvent(
event.type === "step.attempt.completed" ? "step.completed" : "step.failed",
);
if (event.type === "step.attempt.failed") {
- recordError(attempt.operation.span, event.error);
- recordError(attempt.step.span, event.error);
+ recordError(attempt.span, event.error);
}
- attempt.operation.span.end();
- attempt.step.span.end();
+ attempt.span.end();
steps.delete(scope);
};
@@ -273,15 +249,13 @@ export function createAgentOtelInstrumentation(
if (event.type === "turn.cancelled" || event.type === "turn.failed") {
await actions.deleteForTurn(event.sessionId, event.turnId);
}
- const turn = await input.stateStore.getTurn(event.sessionId, event.turnId);
- if (turn === undefined) return;
- await input.stateStore.setTurn(event.sessionId, event.turnId, {
+ await input.stateStore.updateTurn(event.sessionId, event.turnId, (turn) => ({
...turn,
terminal:
event.type === "turn.failed"
? { error: event.error, type: event.type }
: { type: event.type },
- });
+ }));
};
const onSessionTransition = async (
@@ -292,19 +266,24 @@ export function createAgentOtelInstrumentation(
if (turn !== undefined) {
const session = await input.stateStore.getSession(event.sessionId);
if (isSampledTrace(turn.context)) {
+ const agentName = session?.agentName ?? turn.subagentName;
const span = input.idGenerator.withSpanId(turn.context.spanId, () =>
input.tracer.startSpan(
- "agent.turn",
+ agentSpanName(agentName),
{
attributes: {
"agent.framework.name": "eve",
"agent.framework.version": input.frameworkVersion,
- "agent.name": session?.agentName,
+ "agent.name": agentName,
"agent.session.id": event.sessionId,
"agent.subagent.name": turn.subagentName,
"agent.turn.id": event.turnId,
"agent.turn.sequence": turn.sequence,
+ "gen_ai.agent.name": agentName,
+ "gen_ai.conversation.id": event.sessionId,
+ "gen_ai.operation.name": "invoke_agent",
},
+ kind: SpanKind.INTERNAL,
startTime: turn.startTimeMs,
},
contextFromSpanContext({
@@ -315,6 +294,7 @@ export function createAgentOtelInstrumentation(
}),
),
);
+ setAgentInvocationUsage(span, turn.modelUsage);
span.addEvent("turn.started", undefined, turn.startTimeMs);
if (turn.terminal !== undefined) {
span.addEvent(turn.terminal.type);
@@ -339,8 +319,8 @@ export function createAgentOtelInstrumentation(
const onModelCallStarted = (event: InstrumentationModelCallStartedEvent): void => {
const attempt = steps.get(event.scope);
if (attempt === undefined) return;
- attempt.step.span.setAttribute("agent.model.id", event.model.modelId);
- attempt.step.span.setAttribute("agent.model.provider", event.model.provider);
+ attempt.span.setAttribute("agent.model.id", event.model.modelId);
+ attempt.span.setAttribute("agent.model.provider", event.model.provider);
const span = input.tracer.startSpan(
modelSpanName(event.model.modelId),
{
@@ -352,7 +332,7 @@ export function createAgentOtelInstrumentation(
...runtimeContextAttributes(event.runtimeContext),
},
},
- attempt.operation.context,
+ attempt.context,
);
if (recordInputs && event.input !== undefined) {
const messages = messagesContentAttribute(event.input.messages);
@@ -366,22 +346,25 @@ export function createAgentOtelInstrumentation(
span.setAttribute("gen_ai.system_instructions", genAiSystem);
}
}
- const state = { context: trace.setSpan(attempt.operation.context, span), span };
+ const state = { context: trace.setSpan(attempt.context, span), span };
getExecutionContexts(event.scope).set(event.idempotencyKey, state.context);
getSpanStates(modelSpans, event.scope).set(event.idempotencyKey, state);
};
- const onModelCallTerminal = (event: InstrumentationModelCallTerminalEvent): void => {
+ const onModelCallTerminal = async (
+ event: InstrumentationModelCallTerminalEvent,
+ ): Promise<void> => {
executionContexts.get(event.scope)?.delete(event.idempotencyKey);
const state = takeSpanState(modelSpans, event.scope, event.idempotencyKey);
if (state === undefined) return;
if (event.type === "model.call.failed") {
recordError(state.span, event.error);
} else {
+ await recordTurnUsage(event);
setAgentUsage(state.span, event.usage);
state.span.setAttribute("gen_ai.response.finish_reasons", [event.finishReason]);
const attempt = steps.get(event.scope);
- if (attempt !== undefined) setAgentUsage(attempt.step.span, event.usage);
+ if (attempt !== undefined) setAgentUsage(attempt.span, event.usage);
if (recordOutputs) {
state.span.setAttribute("ai.response.finish_reason", event.finishReason);
const content = event.content;
@@ -416,7 +399,7 @@ export function createAgentOtelInstrumentation(
if (json !== undefined) state.span.setAttribute("ai.response.tool_calls", json);
}
// Provider-executed tools (e.g. web_search) run inside the model call,
- // never reach eve's tool loop, and so never get an ai.toolCall span.
+ // never reach eve's tool loop, and so never get an execute_tool span.
// Their results only exist as content parts on the model response.
const toolResults = content
.filter((part) => part.type === "tool-result" || part.type === "tool-error")
@@ -444,6 +427,31 @@ export function createAgentOtelInstrumentation(
state.span.end();
};
+ const recordTurnUsage = async (
+ event: Extract<
+ InstrumentationModelCallTerminalEvent,
+ { readonly type: "model.call.completed" }
+ >,
+ ): Promise<void> => {
+ if (event.usage.inputTokens === undefined && event.usage.outputTokens === undefined) return;
+ // The bridge publishes at most one completion per physical execution.
+ // Workflow retries restart from pre-step state, so abandoned additions are
+ // not merged; distinct completed retries consumed tokens and count here.
+ await input.stateStore.updateTurn(event.scope.sessionId, event.scope.turnId, (turn) => ({
+ ...turn,
+ modelUsage: {
+ inputTokens:
+ event.usage.inputTokens === undefined
+ ? turn.modelUsage?.inputTokens
+ : (turn.modelUsage?.inputTokens ?? 0) + event.usage.inputTokens,
+ outputTokens:
+ event.usage.outputTokens === undefined
+ ? turn.modelUsage?.outputTokens
+ : (turn.modelUsage?.outputTokens ?? 0) + event.usage.outputTokens,
+ },
+ }));
+ };
+
const channelDeliveries = createAgentChannelDeliveryInstrumentation({
ensureSessionContext,
frameworkVersion: input.frameworkVersion,
@@ -462,7 +470,7 @@ export function createAgentOtelInstrumentation(
if (costAttributes === undefined) return;
// The vendored OTel Span surface only has singular setAttribute.
for (const [key, value] of Object.entries(costAttributes)) {
- attempt.step.span.setAttribute(key, value);
+ attempt.span.setAttribute(key, value);
}
};
@@ -523,7 +531,9 @@ export function createAgentOtelInstrumentation(
if (!isSampledTrace(turn.context)) parent = suppressTracing(parent);
}
}
- return parent === undefined ? execute() : context.with(parent, execute);
+ return parent === undefined
+ ? execute()
+ : context.with(markAgentTraceContext(parent), execute);
},
};
@@ -633,51 +643,17 @@ function modelSpanName(modelId: string): string {
return `chat ${modelId}`;
}
-type SpanAttributePrimitive = string | number | boolean;
-type SpanAttributeValue = SpanAttributePrimitive | SpanAttributePrimitive[];
-
-/**
- * Flattens merged runtime context into `ai.settings.context.*` span
- * attributes, the same key convention the AI SDK's own OTel integration used
- * for the legacy `step.started` return. Records recurse with dotted keys;
- * arrays pass through only when homogeneous primitives, matching OTel's
- * attribute value contract.
- */
-function runtimeContextAttributes(
- runtimeContext: Readonly<Record<string, unknown>> | undefined,
-): Record<string, SpanAttributeValue> {
- const attributes: Record<string, SpanAttributeValue> = {};
- if (runtimeContext === undefined) return attributes;
- for (const [key, value] of Object.entries(runtimeContext)) {
- flattenContextAttribute(attributes, `ai.settings.context.${key}`, value);
- }
- return attributes;
+function agentSpanName(agentName: string | undefined): string {
+ return agentName === undefined ? "invoke_agent" : `invoke_agent ${agentName}`;
}
-function flattenContextAttribute(
- attributes: Record<string, SpanAttributeValue>,
- key: string,
- value: unknown,
-): void {
- if (value == null) return;
- if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") {
- attributes[key] = value;
- return;
+function setAgentInvocationUsage(span: Span, modelUsage: AgentTurnTraceState["modelUsage"]): void {
+ if (modelUsage === undefined) return;
+ if (modelUsage.inputTokens !== undefined) {
+ span.setAttribute("gen_ai.usage.input_tokens", modelUsage.inputTokens);
}
- if (Array.isArray(value)) {
- const primitives = value.filter(
- (entry): entry is SpanAttributePrimitive =>
- typeof entry === "string" || typeof entry === "number" || typeof entry === "boolean",
- );
- if (primitives.length !== value.length) return;
- if (new Set(primitives.map((entry) => typeof entry)).size !== 1) return;
- attributes[key] = primitives;
- return;
- }
- if (typeof value === "object") {
- for (const [nestedKey, nestedValue] of Object.entries(value)) {
- flattenContextAttribute(attributes, `${key}.${nestedKey}`, nestedValue);
- }
+ if (modelUsage.outputTokens !== undefined) {
+ span.setAttribute("gen_ai.usage.output_tokens", modelUsage.outputTokens);
}
}
@@ -686,6 +662,7 @@ function errorText(error: unknown): unknown {
}
function recordError(span: Span, error: unknown): void {
+ span.setAttribute("error.type", error instanceof Error ? error.name || "Error" : "_OTHER");
if (error instanceof Error) {
span.recordException(error);
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
diff --git a/packages/eve/src/tracing/agent-otel-runtime-context.ts b/packages/eve/src/tracing/agent-otel-runtime-context.ts
new file mode 100644
index 000000000..7f4f06d8f
--- /dev/null
+++ b/packages/eve/src/tracing/agent-otel-runtime-context.ts
@@ -0,0 +1,41 @@
+type SpanAttributePrimitive = string | number | boolean;
+type SpanAttributeValue = SpanAttributePrimitive | SpanAttributePrimitive[];
+
+/** Flattens merged runtime context into AI SDK-compatible span attributes. */
+export function runtimeContextAttributes(
+ runtimeContext: Readonly<Record<string, unknown>> | undefined,
+): Record<string, SpanAttributeValue> {
+ const attributes: Record<string, SpanAttributeValue> = {};
+ if (runtimeContext === undefined) return attributes;
+ for (const [key, value] of Object.entries(runtimeContext)) {
+ flattenContextAttribute(attributes, `ai.settings.context.${key}`, value);
+ }
+ return attributes;
+}
+
+function flattenContextAttribute(
+ attributes: Record<string, SpanAttributeValue>,
+ key: string,
+ value: unknown,
+): void {
+ if (value == null) return;
+ if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") {
+ attributes[key] = value;
+ return;
+ }
+ if (Array.isArray(value)) {
+ const primitives = value.filter(
+ (entry): entry is SpanAttributePrimitive =>
+ typeof entry === "string" || typeof entry === "number" || typeof entry === "boolean",
+ );
+ if (primitives.length !== value.length) return;
+ if (new Set(primitives.map((entry) => typeof entry)).size !== 1) return;
+ attributes[key] = primitives;
+ return;
+ }
+ if (typeof value === "object") {
+ for (const [nestedKey, nestedValue] of Object.entries(value)) {
+ flattenContextAttribute(attributes, `${key}.${nestedKey}`, nestedValue);
+ }
+ }
+}
diff --git a/packages/eve/src/tracing/agent-otel-session-context.ts b/packages/eve/src/tracing/agent-otel-session-context.ts
index e3b4198ce..f7eeb0841 100644
--- a/packages/eve/src/tracing/agent-otel-session-context.ts
+++ b/packages/eve/src/tracing/agent-otel-session-context.ts
@@ -67,7 +67,7 @@ export function createAgentOtelSessionContext(
"agent.channel.audience": session.channelAudience,
"agent.name": session.agentName,
"agent.session.id": session.sessionId,
- "agent.trace.schema.version": 2,
+ "agent.trace.schema.version": 3,
},
root: true,
});
@@ -94,7 +94,7 @@ export function createAgentOtelSessionContext(
"agent.channel.audience": session.channelAudience,
"agent.name": session.agentName,
"agent.session.id": session.sessionId,
- "agent.trace.schema.version": 2,
+ "agent.trace.schema.version": 3,
},
root: true,
});
diff --git a/packages/eve/src/tracing/agent-otel-usage.ts b/packages/eve/src/tracing/agent-otel-usage.ts
index 3bb81a133..3cdbbba91 100644
--- a/packages/eve/src/tracing/agent-otel-usage.ts
+++ b/packages/eve/src/tracing/agent-otel-usage.ts
@@ -3,7 +3,11 @@ import type { Span } from "#compiled/@opentelemetry/api/index.js";
import type { InstrumentationUsage } from "#instrumentation/lifecycle.js";
/** Applies eve's structural token usage attributes to an agent span. */
-export function setAgentUsage(span: Span, usage: InstrumentationUsage): void {
+export function setAgentUsage(
+ span: Span,
+ usage: InstrumentationUsage,
+ options: { readonly includeGenAiDetails?: boolean } = {},
+): void {
if (usage.inputTokens !== undefined) {
span.setAttribute("agent.usage.input_tokens", usage.inputTokens);
}
@@ -12,9 +16,17 @@ export function setAgentUsage(span: Span, usage: InstrumentationUsage): void {
}
const details = usage.inputTokenDetails;
if (details?.cacheReadTokens !== undefined) {
- span.setAttribute("gen_ai.usage.cache_read.input_tokens", details.cacheReadTokens);
+ if (options.includeGenAiDetails === false) {
+ span.setAttribute("agent.usage.cache_read_tokens", details.cacheReadTokens);
+ } else {
+ span.setAttribute("gen_ai.usage.cache_read.input_tokens", details.cacheReadTokens);
+ }
}
if (details?.cacheWriteTokens !== undefined) {
- span.setAttribute("gen_ai.usage.cache_creation.input_tokens", details.cacheWriteTokens);
+ if (options.includeGenAiDetails === false) {
+ span.setAttribute("agent.usage.cache_write_tokens", details.cacheWriteTokens);
+ } else {
+ span.setAttribute("gen_ai.usage.cache_creation.input_tokens", details.cacheWriteTokens);
+ }
}
}
diff --git a/packages/eve/src/tracing/agent-span-id-generator.ts b/packages/eve/src/tracing/agent-span-id-generator.ts
index 5646d3663..bcb1aae9e 100644
--- a/packages/eve/src/tracing/agent-span-id-generator.ts
+++ b/packages/eve/src/tracing/agent-span-id-generator.ts
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
/**
* Id generator shared by `registerOTel` and the agent OTel provider. A span
- * whose lifetime crosses durable worker boundaries (`agent.turn`) is emitted
+ * whose lifetime crosses durable worker boundaries (`invoke_agent`) is emitted
* at its terminal; priming the next span id lets that span carry the
* pre-allocated id its descendants already parented to.
*/
diff --git a/packages/eve/src/tracing/agent-tool-instrumentation.ts b/packages/eve/src/tracing/agent-tool-instrumentation.ts
index 21d0568df..b467caa10 100644
--- a/packages/eve/src/tracing/agent-tool-instrumentation.ts
+++ b/packages/eve/src/tracing/agent-tool-instrumentation.ts
@@ -1,5 +1,6 @@
import {
ROOT_CONTEXT,
+ SpanKind,
SpanStatusCode,
type Context,
type Span,
@@ -164,9 +165,10 @@ export function createAgentToolInstrumentation(input: {
if (state.span !== undefined || state.finished === true) return;
state.span = input.idGenerator.withSpanId(state.spanId, () =>
input.tracer.startSpan(
- "ai.toolCall",
+ `execute_tool ${state.event.toolName}`,
{
attributes: toolAttributes(state.event),
+ kind: SpanKind.INTERNAL,
startTime: state.startTimeMs,
},
parent,
@@ -211,6 +213,7 @@ function toolAttributes(event: InstrumentationToolCallStartedEvent): Record<stri
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.call.id": event.callId,
"gen_ai.tool.name": event.toolName,
+ "gen_ai.tool.type": "function",
};
}
@@ -219,6 +222,7 @@ function contextFromSpanContext(spanContext: SpanContext): Context {
}
function recordError(span: Span, error: unknown): void {
+ span.setAttribute("error.type", error instanceof Error ? error.name || "Error" : "_OTHER");
if (error instanceof Error) {
span.recordException(error);
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
diff --git a/packages/eve/src/tracing/agent-trace-context-store.test.ts b/packages/eve/src/tracing/agent-trace-context-store.test.ts
index 3a0ce1fae..4069519a0 100644
--- a/packages/eve/src/tracing/agent-trace-context-store.test.ts
+++ b/packages/eve/src/tracing/agent-trace-context-store.test.ts
@@ -23,6 +23,7 @@ describe("ContextAgentTraceStateStore", () => {
});
store.setTurn("session-1", "turn-1", {
context: spanContext("1", "3"),
+ modelUsage: { inputTokens: 12, outputTokens: 4 },
parentIsRemote: true,
parentSpanId: "2".repeat(16),
rootSessionId: "session-1",
@@ -40,6 +41,7 @@ describe("ContextAgentTraceStateStore", () => {
expect(store.getSession("session-1")?.context).toEqual(spanContext("1", "2"));
expect(store.getTurn("session-1", "turn-1")?.context).toEqual(spanContext("1", "3"));
expect(store.getTurn("session-1", "turn-1")).toMatchObject({
+ modelUsage: { inputTokens: 12, outputTokens: 4 },
parentIsRemote: true,
parentSpanId: "2".repeat(16),
startTimeMs: 1_700_000_000_000,
@@ -76,6 +78,39 @@ describe("ContextAgentTraceStateStore", () => {
});
});
+ it("composes atomic turn updates without recreating deleted turns", () => {
+ contextStorage.run(new ContextContainer(), () => {
+ const store = new ContextAgentTraceStateStore();
+ store.setTurn("session-1", "turn-1", {
+ context: spanContext("1", "3"),
+ parentSpanId: "2".repeat(16),
+ rootSessionId: "session-1",
+ sequence: 0,
+ startTimeMs: 1_700_000_000_000,
+ });
+
+ store.updateTurn("session-1", "turn-1", (turn) => ({
+ ...turn,
+ modelUsage: { inputTokens: 12, outputTokens: 4 },
+ }));
+ store.updateTurn("session-1", "turn-1", (turn) => ({
+ ...turn,
+ terminal: { type: "turn.completed" },
+ }));
+
+ expect(store.getTurn("session-1", "turn-1")).toMatchObject({
+ modelUsage: { inputTokens: 12, outputTokens: 4 },
+ terminal: { type: "turn.completed" },
+ });
+ store.deleteTurn("session-1", "turn-1");
+ store.updateTurn("session-1", "turn-1", (turn) => ({
+ ...turn,
+ modelUsage: { inputTokens: 99 },
+ }));
+ expect(store.getTurn("session-1", "turn-1")).toBeUndefined();
+ });
+ });
+
it("preserves only trace state from an interrupted context", async () => {
const context = new ContextContainer();
await contextStorage.run(context, () => {
diff --git a/packages/eve/src/tracing/agent-trace-context-store.ts b/packages/eve/src/tracing/agent-trace-context-store.ts
index 7b15fce31..13773b2ac 100644
--- a/packages/eve/src/tracing/agent-trace-context-store.ts
+++ b/packages/eve/src/tracing/agent-trace-context-store.ts
@@ -182,6 +182,20 @@ export class ContextAgentTraceStateStore implements AgentTraceStateStore {
turns: { ...state.turns, [turnKey(sessionId, turnId)]: value },
}));
}
+
+ updateTurn(
+ sessionId: string,
+ turnId: string,
+ update: (state: AgentTurnTraceState) => AgentTurnTraceState,
+ ): void {
+ updateState((state) => {
+ const key = turnKey(sessionId, turnId);
+ const current = state.turns[key];
+ return current === undefined
+ ? state
+ : { ...state, turns: { ...state.turns, [key]: update(current) } };
+ });
+ }
}
function updateState(update: (state: AgentTraceContextState) => AgentTraceContextState): void {
@@ -242,6 +256,7 @@ function deserializeState(data: unknown): AgentTraceContextState {
}
return {
context: value.context,
+ modelUsage: deserializeModelUsage(value.modelUsage),
parentIsRemote: typeof value.parentIsRemote === "boolean" ? value.parentIsRemote : undefined,
parentSpanId: value.parentSpanId,
rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",
@@ -254,6 +269,17 @@ function deserializeState(data: unknown): AgentTraceContextState {
return { actions, sessions, turns };
}
+function deserializeModelUsage(
+ value: unknown,
+): NonNullable<AgentTurnTraceState["modelUsage"]> | undefined {
+ if (!isRecord(value)) return undefined;
+ const inputTokens = typeof value.inputTokens === "number" ? value.inputTokens : undefined;
+ const outputTokens = typeof value.outputTokens === "number" ? value.outputTokens : undefined;
+ return inputTokens === undefined && outputTokens === undefined
+ ? undefined
+ : { inputTokens, outputTokens };
+}
+
function deserializeAction(value: unknown): AgentActionTraceState | undefined {
if (
!isRecord(value) ||
diff --git a/packages/eve/src/tracing/agent-trace-context.ts b/packages/eve/src/tracing/agent-trace-context.ts
new file mode 100644
index 000000000..6f5b76364
--- /dev/null
+++ b/packages/eve/src/tracing/agent-trace-context.ts
@@ -0,0 +1,17 @@
+import { createContextKey, type Context } from "#compiled/@opentelemetry/api/index.js";
+
+// OTel context keys use Symbol.for(description), so this marker survives duplicate
+// module evaluations while remaining local to the process and async context.
+const AGENT_TRACE_KEY = createContextKey("eve.agent.trace");
+
+export function isAgentTraceContext(context: unknown): boolean {
+ if (typeof context !== "object" || context === null) return false;
+ const getValue = Reflect.get(context, "getValue");
+ return (
+ typeof getValue === "function" && Reflect.apply(getValue, context, [AGENT_TRACE_KEY]) === true
+ );
+}
+
+export function markAgentTraceContext(context: Context): Context {
+ return context.setValue(AGENT_TRACE_KEY, true);
+}
diff --git a/packages/eve/src/tracing/agent-trace-state.ts b/packages/eve/src/tracing/agent-trace-state.ts
index f66171903..cc4280e95 100644
--- a/packages/eve/src/tracing/agent-trace-state.ts
+++ b/packages/eve/src/tracing/agent-trace-state.ts
@@ -20,6 +20,7 @@ export interface AgentSessionTraceState {
export interface AgentTurnTraceState {
readonly context: SpanContext;
+ readonly modelUsage?: { readonly inputTokens?: number; readonly outputTokens?: number };
readonly parentIsRemote?: boolean;
readonly parentSpanId: string;
readonly rootSessionId: string;
@@ -70,6 +71,12 @@ export interface AgentTraceStateStore {
setAction(idempotencyKey: string, state: AgentActionTraceState): void | PromiseLike<void>;
setSession(sessionId: string, state: AgentSessionTraceState): void | PromiseLike<void>;
setTurn(sessionId: string, turnId: string, state: AgentTurnTraceState): void | PromiseLike<void>;
+ /** Atomically updates an existing turn and does nothing after that turn is deleted. */
+ updateTurn(
+ sessionId: string,
+ turnId: string,
+ update: (state: AgentTurnTraceState) => AgentTurnTraceState,
+ ): void | PromiseLike<void>;
}
/** In-memory trace state used by tests and non-durable runtimes. */
@@ -127,6 +134,16 @@ export class InMemoryAgentTraceStateStore implements AgentTraceStateStore {
setTurn(sessionId: string, turnId: string, state: AgentTurnTraceState): void {
this.#turns.set(turnKey(sessionId, turnId), state);
}
+
+ updateTurn(
+ sessionId: string,
+ turnId: string,
+ update: (state: AgentTurnTraceState) => AgentTurnTraceState,
+ ): void {
+ const key = turnKey(sessionId, turnId);
+ const state = this.#turns.get(key);
+ if (state !== undefined) this.#turns.set(key, update(state));
+ }
}
export function turnKey(sessionId: string, turnId: string): string {
diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts b/packages/eve/src/tracing/install-instrumentation-runtime.test.ts
index 3e51681e0..08fb7495c 100644
--- a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts
+++ b/packages/eve/src/tracing/install-instrumentation-runtime.test.ts
@@ -59,6 +59,7 @@ describe("installInstrumentationRuntime", () => {
const runtime = installInstrumentationRuntime({
collected: collectOtelPipeline([otelIntegration()]),
frameworkVersion: "test",
+ instrumentationProviders: true,
providers: [{ flush: providerFlush, name: "test", shutdown: providerShutdown }],
serviceName: "weather",
});
@@ -69,6 +70,7 @@ describe("installInstrumentationRuntime", () => {
expect(forceFlush).toHaveBeenCalledOnce();
expect(providerFlush).toHaveBeenCalledOnce();
+ expect(runtime.instrumentationProviders).toBe(true);
expect(runtime.otelSettings).toEqual({
functionId: undefined,
recordInputs: true,
diff --git a/packages/eve/src/tracing/install-instrumentation-runtime.ts b/packages/eve/src/tracing/install-instrumentation-runtime.ts
index 39adb343a..da23d7a2b 100644
--- a/packages/eve/src/tracing/install-instrumentation-runtime.ts
+++ b/packages/eve/src/tracing/install-instrumentation-runtime.ts
@@ -32,6 +32,7 @@ const log = createLogger("tracing.install-instrumentation-runtime");
export function installInstrumentationRuntime(input: {
readonly collected: CollectedOtel;
readonly frameworkVersion: string;
+ readonly instrumentationProviders?: boolean;
readonly providers: readonly InstrumentationProviderDefinition[];
readonly runtimeContextResolvers?: readonly RuntimeContextResolver[];
readonly serviceName: string;
@@ -83,6 +84,7 @@ export function installInstrumentationRuntime(input: {
serialBefore,
}),
idGenerator: otelRuntime?.idGenerator ?? new AgentSpanIdGenerator(),
+ instrumentationProviders: input.instrumentationProviders,
otelSettings: input.collected.declared ? input.collected.settings : undefined,
prepareSessionTrace,
prepareTurnTrace,
diff --git a/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts b/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts
index 8a0088451..becefc65e 100644
--- a/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts
+++ b/packages/eve/src/tracing/local-instrumentation-runtime.scenario.test.ts
@@ -197,14 +197,13 @@ describe("local instrumentation runtime", () => {
const spans = spanGroups.flat();
expect(formatTraceTree(spans)).toEqual([
"agent.session",
- " agent.turn",
+ " invoke_agent weather",
" agent.step",
" agent.action",
- " ai.toolCall",
+ " execute_tool weather",
" user.tool-work",
- " ai.streamText",
- " chat model-1",
- " user.model-work",
+ " chat model-1",
+ " user.model-work",
]);
expect(span(spans, "agent.step").links).toEqual([
expect.objectContaining({
diff --git a/packages/eve/src/tracing/local-trace-reader.test.ts b/packages/eve/src/tracing/local-trace-reader.test.ts
index 90f974707..7fda102ce 100644
--- a/packages/eve/src/tracing/local-trace-reader.test.ts
+++ b/packages/eve/src/tracing/local-trace-reader.test.ts
@@ -1,6 +1,10 @@
import { describe, expect, it } from "vitest";
-import { parseLocalTraceSegment } from "./local-trace-reader.js";
+import {
+ describeLocalTraceSpan,
+ isAgentTurnSpan,
+ parseLocalTraceSegment,
+} from "./local-trace-reader.js";
const TRACE_ID = "1".repeat(32);
const SPAN_ID = "a".repeat(16);
@@ -30,6 +34,37 @@ function span(overrides: Record<string, unknown> = {}): Record<string, unknown>
}
describe("parseLocalTraceSegment", () => {
+ it("describes semantic agent spans while retaining legacy turn support", () => {
+ const spans = parseLocalTraceSegment(
+ segment([
+ span({
+ attributes: [
+ { key: "agent.turn.id", value: { stringValue: "turn-1" } },
+ { key: "gen_ai.operation.name", value: { stringValue: "invoke_agent" } },
+ ],
+ name: "invoke_agent weather",
+ }),
+ span({
+ attributes: [{ key: "agent.turn.id", value: { stringValue: "turn-legacy" } }],
+ name: "agent.turn",
+ spanId: "b".repeat(16),
+ }),
+ span({
+ attributes: [
+ { key: "gen_ai.operation.name", value: { stringValue: "invoke_agent" } },
+ { key: "gen_ai.usage.input_tokens", value: { intValue: 10 } },
+ ],
+ name: "invoke_agent researcher",
+ spanId: "c".repeat(16),
+ }),
+ ]),
+ TRACE_ID,
+ );
+
+ expect(spans.map(isAgentTurnSpan)).toEqual([true, true, false]);
+ expect(spans.map(describeLocalTraceSpan)).toEqual([["turn-1"], ["turn-legacy"], []]);
+ });
+
it("parses span events with attributes, sorted by time", () => {
const spans = parseLocalTraceSegment(
segment([
diff --git a/packages/eve/src/tracing/local-trace-reader.ts b/packages/eve/src/tracing/local-trace-reader.ts
index 3d316cab2..bd5fe983e 100644
--- a/packages/eve/src/tracing/local-trace-reader.ts
+++ b/packages/eve/src/tracing/local-trace-reader.ts
@@ -216,7 +216,7 @@ export function describeLocalTraceSpan(span: LocalTraceSpan): string[] {
const model =
stringSpanAttribute(span, "agent.model.id") ??
stringSpanAttribute(span, "gen_ai.request.model");
- if (span.name === "agent.turn" && turnId !== undefined) {
+ if (isAgentTurnSpan(span) && turnId !== undefined) {
details.push(turnId);
}
if (span.name === "agent.step" && stepIndex !== undefined) {
@@ -233,6 +233,14 @@ export function describeLocalTraceSpan(span: LocalTraceSpan): string[] {
return details;
}
+export function isAgentTurnSpan(span: LocalTraceSpan): boolean {
+ return (
+ span.name === "agent.turn" ||
+ (stringSpanAttribute(span, "gen_ai.operation.name") === "invoke_agent" &&
+ stringSpanAttribute(span, "agent.turn.id") !== undefined)
+ );
+}
+
/** Parses one OTLP/JSON segment file into spans belonging to `expectedTraceId`. */
export function parseLocalTraceSegment(content: string, expectedTraceId: string): LocalTraceSpan[] {
try {
diff --git a/research/provider-neutral-local-observability.md b/research/provider-neutral-local-observability.md
index e951fdcfb..cb43aed50 100644
--- a/research/provider-neutral-local-observability.md
+++ b/research/provider-neutral-local-observability.md
@@ -1,7 +1,7 @@
---
issue: TBD
status: in-progress
-last_updated: "2026-07-30"
+last_updated: "2026-08-28"
---
# Provider-neutral local observability
@@ -75,20 +75,23 @@ or tool more than once.
```text
agent.session {agent.session.id}
- +-- agent.turn {agent.turn.id, agent.turn.sequence}
+ +-- invoke_agent {agent.name} {agent.turn.id, agent.turn.sequence}
+-- agent.step {agent.step.index, agent.step.attempt}
+-- model-provider spans
+-- agent.action {agent.action.kind, name, call_id}
- +-- tool-provider spans
+ +-- execute_tool {tool.name}
+ +-- tool-provider spans
```
-GenAI spans stay nested implementation detail; they do not define the agent run. An `agent.step`
-covers one model call and the actions it requests through their resolution. `agent.action.kind` is
-an open discriminator — `tool` today, with `skill`, `subagent`, and `remote-agent` reserved.
+The durable turn and tool-execution boundaries use the OpenTelemetry GenAI agent conventions so
+backends can recognize agent invocations and tool calls without eve-specific mapping. An
+`agent.step` covers one model call and the actions it requests through their resolution.
+`agent.action.kind` remains an open discriminator for eve's broader durable dispatch boundary.
The `agent.*` namespace carries cheap structural attributes only: session/turn/step/attempt
-identity, action identity, and agent and framework identity. Message content, model output, and tool payloads are optional capture, off by
-default. The OTel provider owns this mapping; the bridge and hooks contain no span names,
+identity, action identity, and agent and framework identity. Standard `gen_ai.*` attributes describe
+the operation, agent, conversation, and tool. Message content, model output, and tool payloads are
+optional capture. The OTel provider owns this mapping; the bridge and hooks contain no span names,
attributes, or parent contexts.
### The session root is a real span
@@ -107,7 +110,7 @@ author can key a sampler on it.
### Marker spans
-The session root is recorded and ended immediately, as `agent.turn` already is. A session outlives the
+The session root is recorded and ended immediately. A session outlives the
worker that opened it and a span object cannot cross that boundary, so eve records the root,
persists its span context, and parents later spans through it.
@@ -119,8 +122,8 @@ A real duration would mean emitting the span once at session close, with explici
span id chosen before the span exists. The ids are reachable — `registerOTel` accepts an
`idGenerator` — but the close is not: a session that goes idle and never resumes closes on nothing,
so its root would never be emitted and every span in the trace would reference a parent that never
-arrives. A marker is the only representation that is always emitted. `agent.turn` does have a
-guaranteed close and could carry a real duration; that is tracked separately.
+arrives. A marker is the only representation that is always emitted. A turn does have a guaranteed
+close, so its `invoke_agent` span carries its real duration.
### Subagents