The Open Framework for Building Agents
test(eve): define the v4 exported trace contract (#3142)
Chad Hietalaa9efcd01974c
diff --git a/docs/guides/instrumentation-providers.md b/docs/guides/instrumentation-providers.mdindex d5237eee3..5219fb00d 100644--- a/docs/guides/instrumentation-providers.md+++ b/docs/guides/instrumentation-providers.md@@ -153,6 +153,8 @@ export default otelIntegration({});```+Custom destinations receive the schema v4 agent trace contract. Follow [Query exported traces](./instrumentation#query-exported-traces) to find activations and join sessions across traces.+Add `agent/instrumentation/otel.ts` when you need process-wide settings or want to control which content eve writes to OpenTelemetry spans:```ts title="agent/instrumentation/otel.ts"diff --git a/docs/guides/instrumentation.md b/docs/guides/instrumentation.mdindex 51c7487f3..98e707d01 100644--- a/docs/guides/instrumentation.md+++ b/docs/guides/instrumentation.md@@ -56,7 +56,7 @@ Three more fields control what the AI SDK records inside those spans (see the AIeve records metadata without model or tool inputs and outputs by default. Enable either content category only after reviewing the exporter and its data-retention path.-In the provider layout, eve stamps each span with `gen_ai.conversation.id`, which stays fixed across local and remote activations, including independent remote root workflows. Query this attribute to find the conversation's exported, retained traces; it does not grant access or control trace parenting. On Vercel, `vercel.session_id` additionally identifies the current Workflow run.+In the provider layout, eve stamps each span with `gen_ai.conversation.id`, which stays fixed across local and remote activations, including independent remote root workflows. Query this attribute to find the conversation's exported, retained traces; it does not grant access or control trace parenting. On Vercel, `vercel.session_id` additionally identifies the current Workflow run. See [Query exported traces](#query-exported-traces) for the cross-backend workflow.You are responsible for ensuring any observability or eval provider is approved for the data exported to it.@@ -146,6 +146,7 @@ The legacy `instrumentation.ts` layout still uses its authored OTel setup.| ----------------------- | -------------------------------------------------------- || `invoke_agent <agent>` | One agent activation in its own trace || `agent.step` | One model attempt |+| `chat <model>` | One model call beneath its step || `agent.action` | Durable action lifecycle, including dispatch and waiting || `execute_tool <tool>` | In-process tool execution beneath its action || `agent.approval` | Approval waiting beneath its action |@@ -157,6 +158,8 @@ Only activations use the `invoke_agent` operation. Dispatch lifecycle spans useexecutes under `execute_tool agent`. Turn IDs such as `turn_0` are local to a session,so correlate turns by session ID and turn ID together.+A conversation remains durable state; `gen_ai.conversation.id` joins the activations that operate on it.+Each activation owns a fresh trace identity, including local and remotesubagents. The first child activation links to its caller with`eve.link.type=agent.dispatch`; incoming `traceparent` provides that link, not@@ -170,10 +173,12 @@ see the activation name and attributes after its session coordinates areavailable. Sampling must be deterministic for the same trace and operationbecause durable reconstruction can evaluate it again.+Use `agent.turn.outcome` to distinguish completed, failed, and cancelled activations. A failed action can be handled by the agent without failing its activation; see [Exported span names and outcomes](#exported-span-names-and-outcomes).+Activation spans retain `gen_ai.usage.input_tokens` and`gen_ai.usage.output_tokens` alongside `agent.usage.*` totals for that activation.Model spans carry `gen_ai.usage.*`; step and dispatch spans report-`agent.usage.*`. Do not sum usage across these levels.+`agent.usage.*`. Activation totals summarize that activation's own model calls, while caller spans can summarize delegated usage. Do not sum usage across these levels.Nested dispatch spans are materialized when the child settles and handed tospan processors; successful materialization removes their completed records@@ -196,6 +201,62 @@ Types are limited to `user`, `service`, `runtime`, `app`, `anonymous`, `local-dePrincipal IDs require a content-visible audience and a resolved trace decision that allows both `recordInputs` and `recordOutputs`. Public turns and unknown turns under `eve dev` can include IDs; private and hosted-unknown turns omit them. A user-configured trace policy or forwarded content ceiling that denies either direction omits both principal IDs before turn state is stored or sampled. Empty IDs and IDs larger than 1 KiB of UTF-8 data are omitted, not truncated; authentication records are unchanged. eve does not copy other authentication fields, such as claims, email attributes, issuers, or subjects, into these summaries.+## Query exported traces++Use a trace to investigate one activation's work. Use session and conversation attributes to find the other activations. Exporting to an OpenTelemetry backend does not turn a conversation into one continuous waterfall.++This workflow applies to schema v4 from the [instrumentation provider layout](./instrumentation-providers). Configure third-party exporters through `otelIntegration()` so they receive the same framework spans as local tracing and Agent Runs. The legacy `instrumentation.ts` setup emits the [authored trace hierarchy](#authored-trace-hierarchy), not this contract.++### Find an activation or conversation++| Investigation | Filter |+| ------------------------ | ----------------------------------------------------------------------- |+| Agent activations | `agent.trace.schema.version=4` and `gen_ai.operation.name=invoke_agent` |+| One logical conversation | `gen_ai.conversation.id=<conversation ID>` |+| One turn | Both `gen_ai.conversation.id` and `agent.turn.id` |+| One Vercel Workflow run | `vercel.session_id=<session ID>` |+| Delegated dispatches | `agent.invocation.role=caller` |++Start with an activation filter for latency, failure, or turn-count dashboards. Every activation is a trace root. Within a conversation, group by trace ID, order activations by `agent.turn.sequence` or start time, and open the selected trace ID. Child activations link to their caller with `eve.link.type=agent.dispatch`; remote workflows keep their own execution lineage, so use conversation IDs and caller links for cross-deployment correlation.++For example, the built-in `agent` tool dispatches research into a new workflow and trace, and the next user turn starts another trace:++```text+Trace A+invoke_agent support session=S, turn=turn_0, conversation=S+ agent.step+ chat <model>+ agent.action role=caller, action.call_id=C+ execute_tool agent++Trace B+invoke_agent research session=R, turn=turn_0, conversation=S+ link: agent.dispatch -> Trace A's caller action C+ agent.step+ chat <model>++Trace C+invoke_agent support session=S, turn=turn_1, conversation=S+ agent.step+ chat <model>+```++The conversation filter finds all three retained traces. eve initializes the conversation ID once and carries it through the existing parent context for local dispatch and `eve.conversation.id` baggage for remote dispatch. Remote session creation does not carry execution lineage or add authorization requirements for tracing. Existing authentication, principal forwarding, and root-session limits remain unchanged.++`traceparent` carries the caller's trace and span IDs for the causal link; adopting it as the child's parent would instead keep both in the same trace. Resolved trace-policy decisions and trusted remote content ceilings continue across this boundary independently of correlation, and an unsampled caller cannot be widened into a sampled child. No synthetic session span is needed.++Activation duration is elapsed time for that activation, including waits inside it, not the lifetime of the conversation or CPU time. Idle time between ended activations is not part of their durations. For token totals, choose one level: filter to `chat` model spans and sum `gen_ai.usage.*`, or filter to activations and sum their `agent.usage.input_tokens` and `agent.usage.output_tokens`. Exclude caller summaries from the activation total. Query cache usage on model spans. Do not add model, step, activation, and caller counters together.++### Search traces++Use your destination's span-search surface to filter by the attributes in the table, group activations by session or trace ID, and open one activation's trace waterfall. Use the OTel span name and `gen_ai.operation.name` to identify the operation. Preserve eve's ownership of the OTel provider; do not register a second provider merely to add a destination.++### Sampling and completeness++A session can contain both retained and missing activation traces. Head sampling, backend retention, destination filtering, and exporter failures all affect what a conversation query returns. Preserve the schema and correlation attributes in Collector transforms and destination policies, and configure indexing and retention for the queries above.++Do not interpret a missing activation as a missing execution, or sampled trace counts and token sums as an exact session ledger. A backend may receive descendants before the activation span finishes and exports; a temporarily missing root is not necessarily a broken parent ID. After deployment, verify a two-turn conversation and a delegation in the actual destination, including names, conversation grouping across services, separate trace roots, caller links, and outcome attributes.+## Runtime context_Runtime context_ is an [AI SDK concept](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text): a user-defined object that flows through a generation lifecycle. eve exposes it through `events["step.started"]`, a callback that runs once eve has assembled the model input for an attempt and returns `{ runtimeContext }`. Because eve registers the AI SDK's OpenTelemetry integration with runtime context enabled, those returned values ride onto the model-call span and its children. The field is named `runtimeContext`, not `metadata`, because AI SDK v7 carries per-call attributes on runtime context rather than a dedicated metadata field.@@ -283,7 +344,7 @@ Structural tags describe each run's place in the tree:- `$eve.trigger`: the channel kind that started the run- `$eve.schedule`: the authored schedule that created the session, including sessions started through a target channel- `$eve.title`: truncated title derived from the first user message-- `$eve.trace_id`: trace id of the sampled agent trace containing the run, written on session, subagent, and turn rows so a dashboard run can be joined to its OpenTelemetry trace. Present only when the trace is sampled; absence means no exported OTEL trace exists.+- `$eve.trace_id`: sampled trace seed available when tagging a session, subagent, or turn run. Use it as a trace link, not a conversation-wide trace identity. Later activations can use other trace IDs; query `gen_ai.conversation.id` for the history. The tag does not confirm that a destination retained the trace.Per-turn usage tags are written on each step of a turn, accumulating cumulative totals (last write wins):diff --git a/packages/eve/src/execution/eve-workflow-attributes.ts b/packages/eve/src/execution/eve-workflow-attributes.tsindex 7d241a329..6ba727a5e 100644--- a/packages/eve/src/execution/eve-workflow-attributes.ts+++ b/packages/eve/src/execution/eve-workflow-attributes.ts@@ -28,9 +28,9 @@* - `$eve.invocation_owner` — SHA-256 fingerprint of the invocation's initiating principal* - `$eve.is_trace_content_visible` — whether observability may read content-bearing workflow data* - `$eve.is_otel_trace_enabled` — whether hosted Agent Runs OTEL is enabled for the run- * - `$eve.trace_id` — initial activation trace id from the pre-allocated seed.- * Later activations have independent traces; query `gen_ai.conversation.id`- * to find the conversation's exported traces.+ * - `$eve.trace_id` — sampled trace seed available in the serialized context when+ * tagging the run. This is a trace link, not a session-wide trace identity or+ * confirmation that a destination retained the trace.*/import { CHANNEL_CONTEXT_KEY_NAME } from "#context/key-names.js";diff --git a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.tsindex d32429af3..eae743578 100644--- a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts+++ b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts@@ -2,6 +2,7 @@ import {BasicTracerProvider,InMemorySpanExporter,SimpleSpanProcessor,+ type ReadableSpan,} from "@opentelemetry/sdk-trace-base";import { describe, expect, it, vi } from "vitest";import { SpanStatusCode } from "#compiled/@opentelemetry/api/index.js";@@ -44,6 +45,7 @@ import {assembleLocalTrace,isAgentTurnSpan,parseLocalTraceSegment,+ type LocalTraceSpan,} from "#tracing/local-trace-reader.js";import { summarizeLocalTrace } from "#cli/commands/trace-detail.js";import { buildConversationItems } from "#cli/dev/tui/traces/trace-conversation.js";@@ -256,7 +258,7 @@ describe("exported agent telemetry contract", () => {);it.each(["public", "private"] as const)(- "round-trips %s channel, approval, delegated error and usage spans through the readers",+ "round-trips the normalized %s v4 trace forest through OTLP",async (audience) => {const runtime = createRuntime();let parent = contextFor(audience);@@ -283,7 +285,17 @@ describe("exported agent telemetry contract", () => {kind: "deliver",payloads: [{ message: "private input" }],deliveryMetadata: [- { channelKind: "http", channelName: "web", deliveryId: "delivery", payloadIndex: 0 },+ {+ channelKind: "http",+ channelName: "web",+ deliveryId: "delivery",+ payloadIndex: 0,+ requestTraceContext: {+ spanId: "e".repeat(16),+ traceFlags: 1,+ traceId: "f".repeat(32),+ },+ },],},});@@ -433,10 +445,10 @@ describe("exported agent telemetry contract", () => {kind: "subagent-result",origin: "child",subagentName: "child",- output: "private failure",+ output: "private reply",outcome: {kind: "terminal",- result: { kind: "failed", error: { message: "private failure" } },+ result: { kind: "succeeded", output: "private reply" },usageDelta: {inputTokens: 10,outputTokens: 5,@@ -446,7 +458,7 @@ describe("exported agent telemetry contract", () => {},},});- if (audience === "private") expect(JSON.stringify(settled)).not.toContain("private failure");+ if (audience === "private") expect(JSON.stringify(settled)).not.toContain("private reply");parent = await deserializeContext(settled);await contextStorage.run(parent, async () => {await runtime.forceFlush();@@ -492,11 +504,68 @@ describe("exported agent telemetry contract", () => {type: "session.waiting",});});+ for (const [sequence, outcome] of [+ [1, "failed"],+ [2, "cancelled"],+ ] as const) {+ const turnId = `turn_${String(sequence)}`;+ await contextStorage.run(parent, async () => {+ await binding.instrumentChannelDelivery({+ ctx: parent,+ agentName: "parent",+ rootSessionId: "parent",+ sequence,+ sessionId: "parent",+ turnId,+ delivery: {+ kind: "deliver",+ payloads: [{ message: outcome }],+ deliveryMetadata: [+ {+ channelKind: "http",+ channelName: "web",+ deliveryId: `delivery-${outcome}`,+ payloadIndex: 0,+ },+ ],+ },+ });+ await binding.preparePreamble({ sequence, sessionStarted: true, turnId });+ await binding.instrumentChannelDelivery({+ ctx: parent,+ error: outcome === "failed" ? new Error("expected failure") : undefined,+ includeTurn: true,+ outcome,+ });+ await hooks.publish(+ outcome === "failed"+ ? {+ error: new Error("expected failure"),+ idempotencyKey: turnIdempotencyKey("parent", turnId),+ sessionId: "parent",+ turnId,+ type: "turn.failed",+ }+ : {+ idempotencyKey: turnIdempotencyKey("parent", turnId),+ sessionId: "parent",+ turnId,+ type: "turn.cancelled",+ },+ );+ await hooks.publish({+ idempotencyKey: sessionIdempotencyKey("parent"),+ sessionId: "parent",+ turnId,+ type: "session.waiting",+ });+ });+ }await runtime.forceFlush();const exported = runtime.exporter.getFinishedSpans();const bytes = JsonTraceSerializer.serializeRequest(exported)!;const traceIds = [...new Set(exported.map((span) => span.spanContext().traceId))];- expect(traceIds).toHaveLength(2);+ expect(traceIds).toHaveLength(4);const traces = traceIds.map((traceId) =>assembleLocalTrace(traceId,@@ -516,16 +585,21 @@ describe("exported agent telemetry contract", () => {parsed.every((span) => Number(span.attributes["agent.trace.schema.version"]) === 4),).toBe(true);for (const span of parsed) {- expect(span.attributes).not.toHaveProperty("agent.session.id");+ expect(span.attributes["resource.name"]).toBe(span.name);+ expect(span.attributes["operation.name"]).toBe(+ span.attributes["gen_ai.operation.name"] ?? span.name,+ );+ for (const legacy of [+ "agent.parent_call.id",+ "agent.parent_run.id",+ "agent.root_run.id",+ "agent.session.id",+ ]) {+ expect(span.attributes).not.toHaveProperty(legacy);+ }expect(span.attributes).not.toHaveProperty("vercel.session_id");}- expect(- parsed- .filter((span) => span.parentSpanId === undefined)- .map((span) => span.name)- .sort(),- ).toEqual(["invoke_agent child", "invoke_agent parent"]);- expect(parsed.filter(isAgentTurnSpan)).toHaveLength(2);+ expect(parsed.filter(isAgentTurnSpan)).toHaveLength(4);for (const activation of parsed.filter(isAgentTurnSpan)) {expect(activation.attributes["agent.principal.current.type"]).toBe("service");expect(activation.attributes["agent.principal.initiator.type"]).toBe("user");@@ -551,7 +625,6 @@ describe("exported agent telemetry contract", () => {"agent.channel.kind": "http","agent.channel.name": "web",});- const childSpan = exported.find((span) => span.spanContext().spanId === activation.spanId)!;expect(activation.attributes).toMatchObject({"gen_ai.usage.input_tokens": 10,"gen_ai.usage.output_tokens": 5,@@ -560,25 +633,26 @@ describe("exported agent telemetry contract", () => {});expect(caller.attributes["gen_ai.usage.input_tokens"]).toBeUndefined();expect(caller.attributes["gen_ai.usage.output_tokens"]).toBeUndefined();- expect(childSpan.links).toEqual([- {- context: expect.objectContaining({ spanId: caller.spanId, traceId: caller.traceId }),- attributes: { "eve.link.type": "agent.dispatch" },- },+ expect(normalizeTraceForest(parsed, exported)).toEqual([+ "conversation original-conversation",+ "trace parent:turn_0 outcome=completed channel=http:web delivery=delivery",+ " invoked from external via channel.request",+ " invoke_agent parent",+ " agent.step",+ " agent.action coordinate",+ " agent.action child role=caller",+ " agent.approval approved",+ " execute_tool coordinate",+ "trace child:turn_0 outcome=completed",+ " invoked from parent:turn_0/agent.action child role=caller via agent.dispatch",+ " invoke_agent child",+ " agent.step",+ " chat test",+ "trace parent:turn_1 outcome=failed channel=http:web delivery=delivery-failed",+ " invoke_agent parent",+ "trace parent:turn_2 outcome=cancelled channel=http:web delivery=delivery-cancelled",+ " invoke_agent parent",]);- expect(parsed.map((span) => span.name).sort()).toEqual(- [- "agent.action",- "agent.action",- "agent.approval",- "agent.step",- "agent.step",- "chat test",- "execute_tool coordinate",- "invoke_agent child",- "invoke_agent parent",- ].sort(),- );expect(summarizeLocalTrace(parsed)).toMatchObject({inputTokens: 10,outputTokens: 5,@@ -590,9 +664,15 @@ describe("exported agent telemetry contract", () => {const metadata = new TextDecoder().decode(JsonTraceSerializer.serializeRequest(runtime.metadata.getFinishedSpans())!,);+ const metadataSpans = traceIds.flatMap((traceId) =>+ parseLocalTraceSegment(metadata, traceId),+ );+ expect(metadataSpans.map((span) => [span.name, span.attributes["resource.name"]])).toEqual(+ parsed.map((span) => [span.name, span.name]),+ );expect(metadata).not.toContain("private ");if (audience === "private") expect(new TextDecoder().decode(bytes)).not.toContain("private ");- else expect(new TextDecoder().decode(bytes)).toContain("private failure");+ else expect(new TextDecoder().decode(bytes)).toContain("private reply");await runtime.shutdown();},);@@ -641,6 +721,15 @@ describe("exported agent telemetry contract", () => {"initiator-user",]);expect(new Set(spans.map((span) => span.spanContext().traceId)).size).toBe(2);+ for (const span of spans) {+ expect(span.parentSpanContext).toBeUndefined();+ expect(span.attributes).toMatchObject({+ "agent.trace.schema.version": 4,+ "gen_ai.conversation.id": "parent",+ "operation.name": "invoke_agent",+ "resource.name": "invoke_agent parent",+ });+ }await runtime.shutdown();});@@ -733,4 +822,147 @@ describe("exported agent telemetry contract", () => {}},);++ it.each(["completed", "failed", "cancelled"] as const)(+ "exports a queryable %s activation outcome without relying on span events",+ async (outcome) => {+ const runtime = createRuntime();+ const ctx = contextFor("private");+ const hooks = runtime.hooks.forTrace!({ agentName: "parent", audience: "private" });+ const binding = bindInstrumentationRuntime(runtime, ctx, {+ agentName: "parent",+ rootSessionId: "parent",+ sessionId: "parent",+ })!;+ await contextStorage.run(ctx, async () => {+ await binding.preparePreamble({ sequence: 0, sessionStarted: false, turnId: "turn_0" });+ const identity = {+ idempotencyKey: turnIdempotencyKey("parent", "turn_0"),+ sessionId: "parent",+ turnId: "turn_0",+ };+ await hooks.publish(+ outcome === "failed"+ ? { ...identity, type: "turn.failed", error: new Error("private failure") }+ : { ...identity, type: outcome === "completed" ? "turn.completed" : "turn.cancelled" },+ );+ await hooks.publish({+ idempotencyKey: sessionIdempotencyKey("parent"),+ sessionId: "parent",+ turnId: "turn_0",+ type: "session.waiting",+ });+ });+ await runtime.forceFlush();+ const spans = runtime.metadata.getFinishedSpans();+ const bytes = JsonTraceSerializer.serializeRequest(+ spans.map((span) => ({+ ...span,+ events: [],+ spanContext: () => span.spanContext(),+ })),+ )!;+ const serialized = new TextDecoder().decode(bytes);+ const [activation] = parseLocalTraceSegment(serialized, spans[0]!.spanContext().traceId);+ expect(activation?.attributes["agent.turn.outcome"]).toBe(outcome);+ expect(activation?.statusCode).toBe(outcome === "failed" ? 2 : 0);+ expect(serialized).not.toContain("private failure");+ await runtime.shutdown();+ },+ );});++function normalizeTraceForest(+ spans: readonly LocalTraceSpan[],+ exported: readonly ReadableSpan[],+): string[] {+ const roots = spans.filter(+ (span) => span.parentSpanId === undefined && span.name.startsWith("invoke_agent "),+ );+ const aliases = new Map(+ roots.map((root) => [+ root.traceId,+ `${String(root.attributes["agent.name"] ?? root.name.slice("invoke_agent ".length))}:${String(+ root.attributes["agent.turn.id"] ?? "unknown",+ )}`,+ ]),+ );+ const causalParents = new Map(+ exported.flatMap((span) =>+ span.links+ .filter(+ (link) =>+ link.attributes?.["eve.link.type"] === "agent.dispatch" &&+ aliases.has(link.context.traceId),+ )+ .map((link) => [span.spanContext().traceId, link.context.traceId] as const),+ ),+ );+ const byIdentity = new Map(spans.map((span) => [`${span.traceId}:${span.spanId}`, span]));+ const children = Map.groupBy(+ spans.filter((span) => span.parentSpanId !== undefined),+ (span) => `${span.traceId}:${span.parentSpanId!}`,+ );+ const conversationIds = [+ ...new Set(roots.map((root) => String(root.attributes["gen_ai.conversation.id"]))),+ ];+ const lines =+ conversationIds.length === 1 ? [`conversation ${conversationIds[0]}`] : ["conversation mixed"];+ for (const root of roots.toSorted((left, right) => {+ if (causalParents.get(left.traceId) === right.traceId) return 1;+ if (causalParents.get(right.traceId) === left.traceId) return -1;+ const sequence =+ Number(left.attributes["agent.turn.sequence"]) -+ Number(right.attributes["agent.turn.sequence"]);+ return sequence || aliases.get(left.traceId)!.localeCompare(aliases.get(right.traceId)!);+ })) {+ const alias = aliases.get(root.traceId)!;+ const channelKind = root.attributes["agent.channel.kind"];+ const channelName = root.attributes["agent.channel.name"];+ const deliveryId = root.attributes["agent.channel.delivery.id"];+ lines.push(+ `trace ${alias} outcome=${String(root.attributes["agent.turn.outcome"] ?? "unknown")}${+ channelKind === undefined+ ? ""+ : ` channel=${String(channelKind)}:${String(channelName)} delivery=${String(deliveryId)}`+ }`,+ );+ const links = exported+ .filter((span) => span.spanContext().traceId === root.traceId)+ .flatMap((span) =>+ span.links.map((link) => {+ const target = byIdentity.get(`${link.context.traceId}:${link.context.spanId}`);+ const type = String(link.attributes?.["eve.link.type"] ?? "unknown");+ return ` invoked from ${+ target === undefined+ ? "external"+ : `${aliases.get(target.traceId)}/${spanLabel(target)}`+ } via ${type}`;+ }),+ )+ .sort();+ lines.push(...links);+ appendTree(root, 1);+ }+ return lines;++ function appendTree(span: LocalTraceSpan, depth: number): void {+ lines.push(`${" ".repeat(depth)}${spanLabel(span)}`);+ for (const child of (children.get(`${span.traceId}:${span.spanId}`) ?? []).toSorted(+ (left, right) => spanLabel(left).localeCompare(spanLabel(right)),+ )) {+ appendTree(child, depth + 1);+ }+ }+}++function spanLabel(span: LocalTraceSpan): string {+ if (span.name === "agent.action") {+ const role = span.attributes["agent.invocation.role"] === "caller" ? " role=caller" : "";+ return `agent.action ${String(span.attributes["agent.action.name"])}${role}`;+ }+ if (span.name === "agent.approval") {+ return `agent.approval ${String(span.attributes["agent.approval.outcome"])}`;+ }+ return span.name;+}