Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): persist zero-config local traces (#1244)

Chad Hietala0d2d0cdcc2bc
diff --git a/.changeset/calm-traces-climb.md b/.changeset/calm-traces-climb.md
new file mode 100644
index 000000000..b96c1bd49
--- /dev/null
+++ b/.changeset/calm-traces-climb.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Persist zero-config agent, AI SDK, and user-created OpenTelemetry spans under `.eve/traces` during `eve dev` when authored instrumentation is absent.
diff --git a/docs/guides/instrumentation.md b/docs/guides/instrumentation.md
index f9d9df9c8..ab0c4f386 100644
--- a/docs/guides/instrumentation.md
+++ b/docs/guides/instrumentation.md
@@ -7,15 +7,23 @@ description: "Trace an agent with OpenTelemetry in instrumentation.ts, read the
If you intend to export telemetry, review the exporter destination, data categories, and required legal approvals before enabling telemetry.
+## Zero-config local traces
+
+When no authored `instrumentation.ts` exists, `eve dev` records agent, AI SDK, and user-created OpenTelemetry spans under `.eve/traces/v1`. Each session has one trace, rooted independently from Workflow telemetry, with turns, model steps, and tool actions represented explicitly. Spans created by application code while a model or tool is executing inherit that active agent context.
+
+The directory is an immutable OTLP/JSON spool and remains available after `eve dev` exits. Inspection tools may build a query index from these segments, but the index is derived and can be rebuilt without changing the captured trace data.
+
+The local writer is an internal development default, not a second provider layered over authored instrumentation. When `instrumentation.ts` exists, its setup retains control and the zero-config writer is not installed.
+
## Three observability surfaces
eve observes an agent through three distinct surfaces. They do not all live in this file, and they write to different places:
-| Surface | Configured in `instrumentation.ts`? | What it is |
-| -------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Workflow run tags** (`$eve.*`) | No (automatic) | Framework-owned attributes on each Vercel Workflow run. Let dashboards stitch session, turn, and subagent runs into a tree and surface model and token usage. |
-| **OpenTelemetry export** | Yes: `setup`, `recordInputs`, `recordOutputs`, `functionId` | Where AI SDK spans are exported and what they record. |
-| **Runtime context events** | Yes: `events["step.started"]` | Per-model-call values written into the AI SDK's runtime context, which the AI SDK carries onto its spans. |
+| Surface | Configured in `instrumentation.ts`? | What it is |
+| -------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Workflow run tags** (`$eve.*`) | No (automatic) | Framework-owned attributes on each Vercel Workflow run. Let dashboards stitch session, turn, and subagent runs into a tree and surface model and token usage. |
+| **OpenTelemetry export** | Local: automatic. Authored: `setup` and capture settings | Where agent and AI spans are exported and what they record. |
+| **Runtime context events** | Yes: `events["step.started"]` | Per-model-call values written into the AI SDK's runtime context, which the AI SDK carries onto its spans. |
The two configurable surfaces send AI SDK spans to your OpenTelemetry backend. Workflow run tags are a separate system, queryable in the Workflow dashboard rather than on your OTel spans. The sections below cover what you configure here; [Workflow run tags](#workflow-run-tags) documents what eve emits on its own.
diff --git a/packages/eve/package.json b/packages/eve/package.json
index 39cc9873f..27b6397a0 100644
--- a/packages/eve/package.json
+++ b/packages/eve/package.json
@@ -315,6 +315,7 @@
"@chat-adapter/twilio": "4.34.0",
"@clack/core": "1.3.1",
"@nuxt/kit": "^4.0.0",
+ "@opentelemetry/otlp-transformer": "0.214.0",
"@opentelemetry/sdk-trace-base": "^2.6.1",
"@standard-schema/spec": "1.1.0",
"@sveltejs/kit": "^2.0.0",
@@ -324,6 +325,7 @@
"@vercel/detect-agent": "1.2.3",
"@vercel/eve-catalog": "workspace:*",
"@vercel/oidc": "3.8.0",
+ "@vercel/otel": "2.1.3",
"@vercel/sandbox": "catalog:",
"@vercel/sdk": "1.28.8",
"@workflow/core": "5.0.0-beta.36",
diff --git a/packages/eve/scripts/vendor-compiled/@opentelemetry/otlp-transformer.mjs b/packages/eve/scripts/vendor-compiled/@opentelemetry/otlp-transformer.mjs
new file mode 100644
index 000000000..fa121904a
--- /dev/null
+++ b/packages/eve/scripts/vendor-compiled/@opentelemetry/otlp-transformer.mjs
@@ -0,0 +1,8 @@
+import { loadDeclaration } from "../_shared.mjs";
+
+export default {
+ packageName: "@opentelemetry/otlp-transformer",
+ compiledPath: "@opentelemetry/otlp-transformer",
+ chunkGroup: "workflow",
+ declaration: await loadDeclaration("@opentelemetry/otlp-transformer.d.ts"),
+};
diff --git a/packages/eve/scripts/vendor-compiled/@vercel/otel.mjs b/packages/eve/scripts/vendor-compiled/@vercel/otel.mjs
new file mode 100644
index 000000000..2e76aada7
--- /dev/null
+++ b/packages/eve/scripts/vendor-compiled/@vercel/otel.mjs
@@ -0,0 +1,9 @@
+import { loadDeclaration } from "../_shared.mjs";
+
+export default {
+ packageName: "@vercel/otel",
+ compiledPath: "@vercel/otel",
+ chunkGroup: "workflow",
+ entry: "dist/node/index.js",
+ declaration: await loadDeclaration("@vercel/otel.d.ts"),
+};
diff --git a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
index 0e34fca17..7803d8add 100644
--- a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
+++ b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts
@@ -1,4 +1,5 @@
export interface SpanContext {
+ isRemote?: boolean;
spanId: string;
traceFlags: number;
traceId: string;
@@ -6,20 +7,28 @@ export interface SpanContext {
}
export interface Span {
- addEvent(name: string, attributes?: Record<string, unknown>): this;
+ addEvent(name: string, attributes?: Attributes): this;
end(): void;
- recordException(exception: unknown): void;
- setAttribute(key: string, value: unknown): this;
+ recordException(
+ exception: Error | string | { message?: string; name?: string; stack?: string },
+ ): void;
+ setAttribute(key: string, value: AttributeValue): this;
setStatus(status: { code: SpanStatusCode; message?: string | undefined }): this;
spanContext(): SpanContext;
}
export interface Tracer {
- startSpan(name: string, options?: { attributes?: Record<string, unknown> | undefined }): Span;
+ startSpan(
+ name: string,
+ options?: { attributes?: Attributes | undefined; root?: boolean | undefined },
+ context?: Context,
+ ): Span;
}
export interface Context {}
+export declare const ROOT_CONTEXT: Context;
+
export declare enum SpanStatusCode {
UNSET = 0,
OK = 1,
@@ -33,7 +42,7 @@ export declare const context: {
export declare const trace: {
getActiveSpan(): Span | undefined;
- getTracer(name: string): Tracer;
+ getTracer(name: string, version?: string): Tracer;
setSpan(context: Context, span: Span): Context;
wrapSpanContext(spanContext: SpanContext): Span;
};
@@ -45,3 +54,12 @@ export declare enum SpanKind {
PRODUCER = 3,
CONSUMER = 4,
}
+export type AttributeValue =
+ | string
+ | number
+ | boolean
+ | Array<string | null | undefined>
+ | Array<number | null | undefined>
+ | Array<boolean | null | undefined>;
+
+export type Attributes = Record<string, AttributeValue | undefined>;
diff --git a/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/otlp-transformer.d.ts b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/otlp-transformer.d.ts
new file mode 100644
index 000000000..51f6be89e
--- /dev/null
+++ b/packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/otlp-transformer.d.ts
@@ -0,0 +1,3 @@
+export declare const JsonTraceSerializer: {
+ serializeRequest(spans: unknown[]): Uint8Array | undefined;
+};
diff --git a/packages/eve/scripts/vendor-compiled/declarations/@vercel/otel.d.ts b/packages/eve/scripts/vendor-compiled/declarations/@vercel/otel.d.ts
new file mode 100644
index 000000000..b1b1e3145
--- /dev/null
+++ b/packages/eve/scripts/vendor-compiled/declarations/@vercel/otel.d.ts
@@ -0,0 +1,16 @@
+export interface SpanProcessor {
+ forceFlush(): Promise<void>;
+ onEnd(span: unknown): void;
+ onStart(span: unknown, parentContext: unknown): void;
+ shutdown(): Promise<void>;
+}
+
+export interface Configuration {
+ readonly autoDetectResources?: boolean;
+ readonly instrumentations?: readonly unknown[];
+ readonly propagators?: readonly ["none"];
+ readonly serviceName?: string;
+ readonly spanProcessors?: readonly SpanProcessor[];
+}
+
+export declare function registerOTel(configuration?: Configuration | string): void;
diff --git a/packages/eve/scripts/vendor-compiled/index.mjs b/packages/eve/scripts/vendor-compiled/index.mjs
index 2357b31bc..2b74bfce7 100644
--- a/packages/eve/scripts/vendor-compiled/index.mjs
+++ b/packages/eve/scripts/vendor-compiled/index.mjs
@@ -16,9 +16,11 @@ import chatAdapterStateMemory from "./@chat-adapter/state-memory.mjs";
import chatAdapterTwilio from "./@chat-adapter/twilio.mjs";
import opentelemetryApi from "./@opentelemetry/api.mjs";
+import opentelemetryOtlpTransformer from "./@opentelemetry/otlp-transformer.mjs";
import standardSchemaSpec from "./@standard-schema/spec.mjs";
import vercelDetectAgent from "./@vercel/detect-agent.mjs";
import vercelOidc from "./@vercel/oidc.mjs";
+import vercelOtel from "./@vercel/otel.mjs";
import vercelSandbox from "./@vercel/sandbox.mjs";
import workflowCore from "./@workflow/core.mjs";
import workflowErrors from "./@workflow/errors.mjs";
@@ -65,6 +67,7 @@ export const MODULES = [
mcp,
openai,
opentelemetryApi,
+ opentelemetryOtlpTransformer,
otel,
picocolors,
provider,
@@ -75,6 +78,7 @@ export const MODULES = [
turndown,
vercelDetectAgent,
vercelOidc,
+ vercelOtel,
vercelSandbox,
workflowCore,
workflowErrors,
diff --git a/packages/eve/src/execution/node-step.ts b/packages/eve/src/execution/node-step.ts
index 4e0a7ddd8..c43d4da5c 100644
--- a/packages/eve/src/execution/node-step.ts
+++ b/packages/eve/src/execution/node-step.ts
@@ -6,6 +6,7 @@ import type { HarnessToolDefinition } from "#harness/execute-tool.js";
import { createToolLoopHarness } from "#harness/tool-loop.js";
import type { HandleEventFn, HarnessToolMap, StepFn } from "#harness/types.js";
import { resolveInstalledPackageInfo } from "#internal/application/package.js";
+import { getInstrumentationRuntime } from "#harness/instrumentation-runtime.js";
import { createLogger } from "#internal/logging.js";
import type { RuntimeIdentity } from "#protocol/message.js";
import { UNSPECIFIED_INPUT_SCHEMA, toInputSchema, toOutputSchema } from "#shared/tool-schema.js";
@@ -81,12 +82,14 @@ export function createExecutionNodeStep(input: CreateExecutionNodeStepInput): St
input.node.turnAgent.dynamicModel,
);
const tools = createNodeHarnessTools({ node: input.node });
- return createToolLoopHarness({
+ const instrumentation = getInstrumentationRuntime();
+ const step = createToolLoopHarness({
abortSignal: input.abortSignal,
capabilities: input.capabilities,
workflow: input.node.agent.workflowTool !== undefined,
workflowMaxSubagents: input.workflowMaxSubagents,
handleEvent: input.handleEvent,
+ instrumentation,
mode: input.mode,
onCompaction: preserveFrameworkStateOnCompaction,
dispatchDynamicModelEvent: dispatchModelEvent,
@@ -94,6 +97,14 @@ export function createExecutionNodeStep(input: CreateExecutionNodeStepInput): St
runtimeIdentity: buildRuntimeIdentity(input.node),
tools,
});
+ if (instrumentation === undefined) return step;
+ return async (session, stepInput) => {
+ try {
+ return await step(session, stepInput);
+ } finally {
+ await instrumentation.forceFlush();
+ }
+ };
}
/**
diff --git a/packages/eve/src/execution/settle-cancelled-turn-step.ts b/packages/eve/src/execution/settle-cancelled-turn-step.ts
index eb6544458..35e81bbb4 100644
--- a/packages/eve/src/execution/settle-cancelled-turn-step.ts
+++ b/packages/eve/src/execution/settle-cancelled-turn-step.ts
@@ -11,6 +11,7 @@ import {
} from "#execution/durable-session-store.js";
import { hydrateDurableSession } from "#execution/session.js";
import { reconcileSessionContinuationToken } from "#execution/reconcile-session-continuation-token.js";
+import { activeTurnId } from "#harness/active-turn-id.js";
import { emitCancelledTurn } from "#harness/cancelled-turn-emission.js";
import {
getHarnessEmissionState,
@@ -22,6 +23,8 @@ import {
hasProxyInputRequests,
} from "#harness/proxy-input-requests.js";
import { clearPendingRuntimeActionBatch } from "#harness/runtime-actions.js";
+import { createInstrumentationHandleEvent } from "#harness/instrumentation-native-events.js";
+import { getInstrumentationRuntime } from "#harness/instrumentation-runtime.js";
import { clearPendingWorkflowInterrupt } from "#harness/workflow-interrupt-state.js";
import {
encodeMessageStreamEvent,
@@ -53,6 +56,7 @@ export async function settleCancelledTurnStep(input: {
const adapter = ctx.require(ChannelKey);
const adapterCtx = buildAdapterContext(adapter, ctx);
const bundle = ctx.require(BundleKey);
+ const instrumentation = getInstrumentationRuntime();
let session = hydrateDurableSession({
compactionOverrides: {
@@ -73,7 +77,7 @@ export async function settleCancelledTurnStep(input: {
const writer = input.parentWritable.getWriter();
try {
const scoped = await withContextScope(ctx, session, async (enrichedSession) => {
- const emit = async (event: HandleMessageStreamEvent): Promise<void> => {
+ const baseEmit = async (event: HandleMessageStreamEvent): Promise<void> => {
const transformed = await callAdapterEventHandler(adapter, event, adapterCtx);
setChannelContext(ctx, { ...adapter, state: { ...adapterCtx.state } });
await writer.write(
@@ -85,6 +89,14 @@ export async function settleCancelledTurnStep(input: {
registry: bundle.hookRegistry,
});
};
+ const emit =
+ createInstrumentationHandleEvent({
+ agentName: bundle.resolvedAgent.config.name,
+ handleEvent: baseEmit,
+ hooks: instrumentation?.hooks,
+ sessionId: session.sessionId,
+ turnId: activeTurnId(emissionState),
+ }) ?? baseEmit;
return {
result: await emitCancelledTurn(emit, emissionState, enrichedSession.continuationToken),
session: enrichedSession,
@@ -93,6 +105,7 @@ export async function settleCancelledTurnStep(input: {
emissionState = scoped.result;
session = scoped.session;
} finally {
+ await instrumentation?.forceFlush();
writer.releaseLock();
}
}
diff --git a/packages/eve/src/execution/workflow-steps.ts b/packages/eve/src/execution/workflow-steps.ts
index c4b98ff26..f3aade630 100644
--- a/packages/eve/src/execution/workflow-steps.ts
+++ b/packages/eve/src/execution/workflow-steps.ts
@@ -19,6 +19,7 @@ import { BundleKey, ChannelKey } from "#runtime/sessions/runtime-context-keys.js
import { runStep } from "#context/run-step.js";
import { deserializeContext, serializeContext } from "#context/serialize.js";
import { getHarnessEmissionState } from "#harness/emission.js";
+import { preserveSerializedAgentTraceState } from "#harness/agent-trace-context-store.js";
import {
isSessionLimitDecline,
isTurnCancellation,
@@ -417,7 +418,10 @@ export async function turnStep(rawInput: TurnStepInput): Promise<DurableStepResu
}
return {
action: "cancelled",
- serializedContext: input.serializedContext,
+ serializedContext: preserveSerializedAgentTraceState(
+ input.serializedContext,
+ serializeContext(ctx),
+ ),
sessionState: input.sessionState,
};
}
diff --git a/packages/eve/src/harness/agent-otel-provider.test.ts b/packages/eve/src/harness/agent-otel-provider.test.ts
index 87c42a3a2..b37786e9b 100644
--- a/packages/eve/src/harness/agent-otel-provider.test.ts
+++ b/packages/eve/src/harness/agent-otel-provider.test.ts
@@ -164,17 +164,22 @@ describe("createAgentOtelInstrumentation", () => {
await runtime.provider.forceFlush();
const spans = runtime.exporter.getFinishedSpans();
- const session = byName(spans, "agent.session")[0]!;
const turn = byName(spans, "agent.turn")[0]!;
const turnTerminal = byName(spans, "agent.turn.terminal")[0]!;
const step = byName(spans, "agent.step")[0]!;
+ const operation = byName(spans, "ai.streamText")[0]!;
+ const model = byName(spans, "ai.streamText.doStream")[0]!;
const action = byName(spans, "agent.action")[0]!;
+ const tool = byName(spans, "ai.toolCall")[0]!;
- expect(session.parentSpanContext).toBeUndefined();
- expect(turn.parentSpanContext?.spanId).toBe(session.spanContext().spanId);
+ expect(byName(spans, "agent.session")).toHaveLength(0);
+ expect(turn.parentSpanContext?.spanId).toBeDefined();
expect(turnTerminal.parentSpanContext?.spanId).toBe(turn.spanContext().spanId);
expect(step.parentSpanContext?.spanId).toBe(turn.spanContext().spanId);
+ expect(operation.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
+ expect(model.parentSpanContext?.spanId).toBe(operation.spanContext().spanId);
expect(action.parentSpanContext?.spanId).toBe(step.spanContext().spanId);
+ expect(tool.parentSpanContext?.spanId).toBe(action.spanContext().spanId);
expect(new Set(spans.map((span) => span.spanContext().traceId))).toHaveLength(1);
expect(turn.events.map((event) => event.name)).toEqual(["turn.started", "session.started"]);
expect(turnTerminal.events.map((event) => event.name)).toEqual([
@@ -264,6 +269,29 @@ describe("createAgentOtelInstrumentation", () => {
expect(step.spanContext().traceId).toBe(turn.spanContext().traceId);
});
+ it("derives the same session trace id when a durable attempt replays before checkpoint", async () => {
+ const firstRuntime = createRuntime();
+ const replayRuntime = createRuntime();
+ await emitAttempt({
+ hooks: firstRuntime.hooks,
+ runInContext: firstRuntime.runInContext,
+ sessionId: "session-1",
+ turnId: "turn-1",
+ turnSequence: 0,
+ });
+ await emitAttempt({
+ hooks: replayRuntime.hooks,
+ runInContext: replayRuntime.runInContext,
+ sessionId: "session-1",
+ turnId: "turn-1",
+ turnSequence: 0,
+ });
+
+ const firstTurn = byName(firstRuntime.exporter.getFinishedSpans(), "agent.turn")[0]!;
+ const replayTurn = byName(replayRuntime.exporter.getFinishedSpans(), "agent.turn")[0]!;
+ expect(replayTurn.spanContext().traceId).toBe(firstTurn.spanContext().traceId);
+ });
+
it("marks a failed action without failing its turn", async () => {
const runtime = createRuntime();
await emitAttempt({
diff --git a/packages/eve/src/harness/agent-otel-provider.ts b/packages/eve/src/harness/agent-otel-provider.ts
index 7d0a8edd6..8df93fc67 100644
--- a/packages/eve/src/harness/agent-otel-provider.ts
+++ b/packages/eve/src/harness/agent-otel-provider.ts
@@ -1,3 +1,5 @@
+import { createHash } from "node:crypto";
+
import {
ROOT_CONTEXT,
SpanStatusCode,
@@ -7,7 +9,7 @@ import {
type SpanContext,
type Tracer,
trace,
-} from "@opentelemetry/api";
+} from "#compiled/@opentelemetry/api/index.js";
import type {
InstrumentationAttemptScope,
@@ -30,6 +32,15 @@ interface SpanState {
readonly span: Span;
}
+interface AttemptSpanState {
+ readonly operation: SpanState & { readonly name: string };
+ readonly step: SpanState;
+}
+
+interface ToolSpanState extends SpanState {
+ readonly toolSpan: Span;
+}
+
export interface AgentSessionTraceState {
readonly agentName?: string;
readonly channelKind?: string;
@@ -116,7 +127,7 @@ export function createAgentOtelInstrumentation(
// A serverless turn runs inside one `turnStep` "use step" invocation. If
// that worker is lost, Workflow retries the whole step from entry rather
// than resuming this callback sequence in a replacement process.
- const steps = new WeakMap<InstrumentationAttemptScope, SpanState>();
+ const steps = new WeakMap<InstrumentationAttemptScope, AttemptSpanState>();
const onSessionStarted = async (event: InstrumentationSessionStartedEvent): Promise<void> => {
await ensureSessionContext(event);
@@ -165,7 +176,7 @@ export function createAgentOtelInstrumentation(
const turn = await input.stateStore.getTurn(event.scope.sessionId, event.scope.turnId);
if (turn === undefined) return;
const turnContext = contextFromSpanContext(turn.context);
- const span = input.tracer.startSpan(
+ const stepSpan = input.tracer.startSpan(
"agent.step",
{
attributes: {
@@ -181,20 +192,43 @@ export function createAgentOtelInstrumentation(
},
turnContext,
);
- span.addEvent("step.started");
+ stepSpan.addEvent("step.started");
+ const stepContext = trace.setSpan(turnContext, stepSpan);
+ const operationName = event.operation.operationId;
+ const operationSpan = input.tracer.startSpan(
+ operationName,
+ {
+ attributes: {
+ "gen_ai.operation.name": operationName,
+ "gen_ai.provider.name": event.operation.provider,
+ "gen_ai.request.model": event.operation.modelId,
+ },
+ },
+ stepContext,
+ );
steps.set(event.scope, {
- context: trace.setSpan(turnContext, span),
- span,
+ operation: {
+ context: trace.setSpan(stepContext, operationSpan),
+ name: operationName,
+ span: operationSpan,
+ },
+ step: { context: stepContext, span: stepSpan },
});
};
const onAttemptTerminal = (event: InstrumentationAttemptTerminalEvent): void => {
executionContexts.delete(event.scope);
- const step = steps.get(event.scope);
- if (step === undefined) return;
- step.span.addEvent(event.type === "attempt.completed" ? "step.completed" : "step.failed");
- if (event.type === "attempt.failed") recordError(step.span, event.error);
- step.span.end();
+ const attempt = steps.get(event.scope);
+ if (attempt === undefined) return;
+ attempt.step.span.addEvent(
+ event.type === "attempt.completed" ? "step.completed" : "step.failed",
+ );
+ if (event.type === "attempt.failed") {
+ recordError(attempt.operation.span, event.error);
+ recordError(attempt.step.span, event.error);
+ }
+ attempt.operation.span.end();
+ attempt.step.span.end();
steps.delete(event.scope);
};
@@ -245,12 +279,24 @@ export function createAgentOtelInstrumentation(
};
const beforeModelCall = (event: InstrumentationModelCallStartedEvent): SpanState | undefined => {
- const step = steps.get(event.scope);
- if (step === undefined) return undefined;
- step.span.setAttribute("agent.model.id", event.source.modelId);
- step.span.setAttribute("agent.model.provider", event.source.provider);
- getExecutionContexts(event.scope).models.set(event.id, step.context);
- return step;
+ const attempt = steps.get(event.scope);
+ if (attempt === undefined) return undefined;
+ attempt.step.span.setAttribute("agent.model.id", event.source.modelId);
+ attempt.step.span.setAttribute("agent.model.provider", event.source.provider);
+ const span = input.tracer.startSpan(
+ modelSpanName(attempt.operation.name),
+ {
+ attributes: {
+ "gen_ai.operation.name": attempt.operation.name,
+ "gen_ai.provider.name": event.source.provider,
+ "gen_ai.request.model": event.source.modelId,
+ },
+ },
+ attempt.operation.context,
+ );
+ const state = { context: trace.setSpan(attempt.operation.context, span), span };
+ getExecutionContexts(event.scope).models.set(event.id, state.context);
+ return state;
};
const afterModelCall = (event: InstrumentationModelCallTerminalEvent, state: unknown): void => {
@@ -258,15 +304,20 @@ export function createAgentOtelInstrumentation(
if (!isSpanState(state)) return;
if (event.type === "model.call.failed") {
recordError(state.span, event.error);
- return;
+ } else {
+ setUsage(state.span, event.source.usage);
+ const attempt = steps.get(event.scope);
+ if (attempt !== undefined) setUsage(attempt.step.span, event.source.usage);
}
- setUsage(state.span, event.source.usage);
+ state.span.end();
};
- const beforeToolCall = (event: InstrumentationToolCallStartedEvent): SpanState | undefined => {
- const step = steps.get(event.scope);
- if (step === undefined) return undefined;
- const span = input.tracer.startSpan(
+ const beforeToolCall = (
+ event: InstrumentationToolCallStartedEvent,
+ ): ToolSpanState | undefined => {
+ const attempt = steps.get(event.scope);
+ if (attempt === undefined) return undefined;
+ const actionSpan = input.tracer.startSpan(
"agent.action",
{
attributes: {
@@ -282,21 +333,40 @@ export function createAgentOtelInstrumentation(
"agent.turn.id": event.scope.turnId,
},
},
- step.context,
+ attempt.step.context,
+ );
+ const actionContext = trace.setSpan(attempt.step.context, actionSpan);
+ const toolSpan = input.tracer.startSpan(
+ "ai.toolCall",
+ {
+ attributes: {
+ "gen_ai.operation.name": "execute_tool",
+ "gen_ai.tool.call.id": event.source.toolCall.toolCallId,
+ "gen_ai.tool.name": event.source.toolCall.toolName,
+ },
+ },
+ actionContext,
);
- const state = { context: trace.setSpan(step.context, span), span };
+ const state: ToolSpanState = {
+ context: trace.setSpan(actionContext, toolSpan),
+ span: actionSpan,
+ toolSpan,
+ };
getExecutionContexts(event.scope).tools.set(event.id, state.context);
return state;
};
const afterToolCall = (event: InstrumentationToolCallTerminalEvent, state: unknown): void => {
executionContexts.get(event.scope)?.tools.delete(event.id);
- if (!isSpanState(state)) return;
+ if (!isToolSpanState(state)) return;
if (event.type === "tool.call.failed") {
+ recordError(state.toolSpan, event.error);
recordError(state.span, event.error);
} else if (event.source.toolOutput.type !== "tool-result") {
+ recordError(state.toolSpan, event.source.toolOutput.error);
recordError(state.span, event.source.toolOutput.error);
}
+ state.toolSpan.end();
state.span.end();
};
@@ -305,30 +375,14 @@ export function createAgentOtelInstrumentation(
): Promise<AgentSessionTraceState> => {
let state = await input.stateStore.getSession(event.sessionId);
if (state === undefined) {
- const marker = input.tracer.startSpan(
- "agent.session",
- {
- attributes: {
- "agent.channel.kind": event.channelKind,
- "agent.framework.name": "eve",
- "agent.framework.version": input.frameworkVersion,
- "agent.name": event.agentName,
- "agent.root.session.id": event.rootSessionId,
- "agent.session.id": event.sessionId,
- },
- root: true,
- },
- ROOT_CONTEXT,
- );
state = {
agentName: event.agentName,
channelKind: event.channelKind,
- context: marker.spanContext(),
+ context: sessionTraceContext(event.sessionId),
pendingStarted: true,
rootSessionId: event.rootSessionId,
};
await input.stateStore.setSession(event.sessionId, state);
- marker.end();
}
return state;
};
@@ -378,6 +432,18 @@ function turnKey(sessionId: string, turnId: string): string {
return `${sessionId}:${turnId}`;
}
+function sessionTraceContext(sessionId: string): SpanContext {
+ return {
+ spanId: digestId(`eve:session-parent:${sessionId}`, 16),
+ traceFlags: 1,
+ traceId: digestId(`eve:session:${sessionId}`, 32),
+ };
+}
+
+function digestId(value: string, length: number): string {
+ return createHash("sha256").update(value).digest("hex").slice(0, length);
+}
+
function contextFromSpanContext(spanContext: SpanContext): Context {
return trace.setSpan(ROOT_CONTEXT, trace.wrapSpanContext(spanContext));
}
@@ -386,6 +452,16 @@ function isSpanState(value: unknown): value is SpanState {
return typeof value === "object" && value !== null && "context" in value && "span" in value;
}
+function isToolSpanState(value: unknown): value is ToolSpanState {
+ return isSpanState(value) && "toolSpan" in value;
+}
+
+function modelSpanName(operationName: string): string {
+ return operationName === "ai.generateText"
+ ? "ai.generateText.doGenerate"
+ : "ai.streamText.doStream";
+}
+
function setUsage(
span: Span,
usage: { readonly inputTokens?: number; readonly outputTokens?: number },
diff --git a/packages/eve/src/harness/agent-trace-context-store.test.ts b/packages/eve/src/harness/agent-trace-context-store.test.ts
new file mode 100644
index 000000000..e2155d9ee
--- /dev/null
+++ b/packages/eve/src/harness/agent-trace-context-store.test.ts
@@ -0,0 +1,83 @@
+import { describe, expect, it } from "vitest";
+
+import { ContextContainer, contextStorage } from "#context/container.js";
+import { deserializeContext, serializeContext } from "#context/serialize.js";
+import {
+ ContextAgentTraceStateStore,
+ preserveSerializedAgentTraceState,
+} from "#harness/agent-trace-context-store.js";
+
+describe("ContextAgentTraceStateStore", () => {
+ it("restores serializable session and turn context", async () => {
+ const context = new ContextContainer();
+ await contextStorage.run(context, () => {
+ const store = new ContextAgentTraceStateStore();
+ store.setSession("session-1", {
+ agentName: "weather",
+ context: spanContext("1", "2"),
+ pendingStarted: false,
+ rootSessionId: "session-1",
+ });
+ store.setTurn("session-1", "turn-1", {
+ context: spanContext("1", "3"),
+ rootSessionId: "session-1",
+ sequence: 0,
+ terminal: { error: new Error("failed"), type: "turn.failed" },
+ });
+ });
+
+ const serialized = await serializeContext(context);
+ const restored = await deserializeContext(serialized);
+ await contextStorage.run(restored, () => {
+ const store = new ContextAgentTraceStateStore();
+ expect(store.getSession("session-1")?.context).toEqual(spanContext("1", "2"));
+ expect(store.getTurn("session-1", "turn-1")?.context).toEqual(spanContext("1", "3"));
+ expect(store.getTurn("session-1", "turn-1")?.terminal?.error).toMatchObject({
+ message: "failed",
+ });
+ });
+ });
+
+ it("removes terminal state", () => {
+ contextStorage.run(new ContextContainer(), () => {
+ const store = new ContextAgentTraceStateStore();
+ store.setSession("session-1", {
+ context: spanContext("1", "2"),
+ pendingStarted: true,
+ rootSessionId: "session-1",
+ });
+ store.setTurn("session-1", "turn-1", {
+ context: spanContext("1", "3"),
+ rootSessionId: "session-1",
+ sequence: 0,
+ });
+
+ store.deleteTurn("session-1", "turn-1");
+ store.deleteSession("session-1");
+
+ expect(store.getTurn("session-1", "turn-1")).toBeUndefined();
+ expect(store.getSession("session-1")).toBeUndefined();
+ });
+ });
+
+ it("preserves only trace state from an interrupted context", async () => {
+ const context = new ContextContainer();
+ await contextStorage.run(context, () => {
+ new ContextAgentTraceStateStore().setSession("session-1", {
+ context: spanContext("1", "2"),
+ pendingStarted: true,
+ rootSessionId: "session-1",
+ });
+ });
+
+ const interrupted = await serializeContext(context);
+ const preserved = preserveSerializedAgentTraceState({ authored: "original" }, interrupted);
+
+ expect(preserved.authored).toBe("original");
+ expect(preserved["eve.harness.agentTrace"]).toBeDefined();
+ });
+});
+
+function spanContext(traceId: string, spanId: string) {
+ return { spanId: spanId.repeat(16), traceFlags: 1, traceId: traceId.repeat(32) };
+}
diff --git a/packages/eve/src/harness/agent-trace-context-store.ts b/packages/eve/src/harness/agent-trace-context-store.ts
new file mode 100644
index 000000000..c0decc981
--- /dev/null
+++ b/packages/eve/src/harness/agent-trace-context-store.ts
@@ -0,0 +1,194 @@
+import type { SpanContext } from "#compiled/@opentelemetry/api/index.js";
+
+import { contextStorage, loadContext } from "#context/container.js";
+import { ContextKey } from "#context/key.js";
+import type {
+ AgentSessionTraceState,
+ AgentTraceStateStore,
+ AgentTurnTraceState,
+} from "#harness/agent-otel-provider.js";
+
+interface AgentTraceContextState {
+ readonly sessions: Readonly<Record<string, AgentSessionTraceState>>;
+ readonly turns: Readonly<Record<string, AgentTurnTraceState>>;
+}
+
+const AgentTraceContextKey = new ContextKey<AgentTraceContextState>("eve.harness.agentTrace", {
+ codec: {
+ deserialize: deserializeState,
+ serialize: serializeState,
+ },
+});
+
+/** Keeps only framework trace state from an interrupted step's context changes. */
+export function preserveSerializedAgentTraceState(
+ original: Record<string, unknown>,
+ interrupted: Record<string, unknown>,
+): Record<string, unknown> {
+ const traceState = interrupted[AgentTraceContextKey.name];
+ return traceState === undefined
+ ? original
+ : { ...original, [AgentTraceContextKey.name]: traceState };
+}
+
+/** Durable trace state backed by eve's serialized Workflow context. */
+export class ContextAgentTraceStateStore implements AgentTraceStateStore {
+ deleteSession(sessionId: string): void {
+ updateState((state) => {
+ const sessions = { ...state.sessions };
+ delete sessions[sessionId];
+ return { ...state, sessions };
+ });
+ }
+
+ deleteTurn(sessionId: string, turnId: string): void {
+ updateState((state) => {
+ const turns = { ...state.turns };
+ delete turns[turnKey(sessionId, turnId)];
+ return { ...state, turns };
+ });
+ }
+
+ getSession(sessionId: string): AgentSessionTraceState | undefined {
+ return contextStorage.getStore()?.get(AgentTraceContextKey)?.sessions[sessionId];
+ }
+
+ getTurn(sessionId: string, turnId: string): AgentTurnTraceState | undefined {
+ return contextStorage.getStore()?.get(AgentTraceContextKey)?.turns[turnKey(sessionId, turnId)];
+ }
+
+ setSession(sessionId: string, value: AgentSessionTraceState): void {
+ updateState((state) => ({
+ ...state,
+ sessions: { ...state.sessions, [sessionId]: value },
+ }));
+ }
+
+ setTurn(sessionId: string, turnId: string, value: AgentTurnTraceState): void {
+ updateState((state) => ({
+ ...state,
+ turns: { ...state.turns, [turnKey(sessionId, turnId)]: value },
+ }));
+ }
+}
+
+function updateState(update: (state: AgentTraceContextState) => AgentTraceContextState): void {
+ loadContext().set(AgentTraceContextKey, (state) => update(state ?? { sessions: {}, turns: {} }));
+}
+
+function turnKey(sessionId: string, turnId: string): string {
+ return `${sessionId}\0${turnId}`;
+}
+
+function serializeState(state: AgentTraceContextState): unknown {
+ return {
+ sessions: Object.fromEntries(
+ Object.entries(state.sessions).map(([id, value]) => [
+ id,
+ { ...value, context: serializeSpanContext(value.context) },
+ ]),
+ ),
+ turns: Object.fromEntries(
+ Object.entries(state.turns).map(([id, value]) => [
+ id,
+ {
+ ...value,
+ context: serializeSpanContext(value.context),
+ terminal:
+ value.terminal === undefined
+ ? undefined
+ : {
+ error: serializeError(value.terminal.error),
+ type: value.terminal.type,
+ },
+ },
+ ]),
+ ),
+ };
+}
+
+function deserializeState(data: unknown): AgentTraceContextState {
+ if (!isRecord(data)) return { sessions: {}, turns: {} };
+ const sessions = deserializeRecord(data.sessions, (value) => {
+ if (!isRecord(value) || !isSpanContext(value.context)) return undefined;
+ return {
+ agentName: typeof value.agentName === "string" ? value.agentName : undefined,
+ channelKind: typeof value.channelKind === "string" ? value.channelKind : undefined,
+ context: value.context,
+ pendingStarted: value.pendingStarted === true,
+ rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",
+ } satisfies AgentSessionTraceState;
+ });
+ const turns = deserializeRecord(data.turns, (value) => {
+ if (!isRecord(value) || !isSpanContext(value.context)) return undefined;
+ return {
+ context: value.context,
+ rootSessionId: typeof value.rootSessionId === "string" ? value.rootSessionId : "",
+ sequence: typeof value.sequence === "number" ? value.sequence : 0,
+ terminal: deserializeTerminal(value.terminal),
+ } satisfies AgentTurnTraceState;
+ });
+ return { sessions, turns };
+}
+
+function deserializeRecord<T>(
+ value: unknown,
+ deserialize: (entry: unknown) => T | undefined,
+): Record<string, T> {
+ if (!isRecord(value)) return {};
+ const result: Record<string, T> = {};
+ for (const [key, entry] of Object.entries(value)) {
+ const parsed = deserialize(entry);
+ if (parsed !== undefined) result[key] = parsed;
+ }
+ return result;
+}
+
+function deserializeTerminal(value: unknown): AgentTurnTraceState["terminal"] {
+ if (!isRecord(value) || typeof value.type !== "string") return undefined;
+ const type = value.type;
+ if (!isTurnTerminalType(type)) return undefined;
+ return { error: deserializeError(value.error), type };
+}
+
+function serializeSpanContext(context: SpanContext): Record<string, unknown> {
+ return {
+ isRemote: context.isRemote,
+ spanId: context.spanId,
+ traceFlags: context.traceFlags,
+ traceId: context.traceId,
+ };
+}
+
+function isSpanContext(value: unknown): value is SpanContext {
+ return (
+ isRecord(value) &&
+ typeof value.spanId === "string" &&
+ typeof value.traceFlags === "number" &&
+ typeof value.traceId === "string"
+ );
+}
+
+function serializeError(error: unknown): unknown {
+ return error instanceof Error
+ ? { message: error.message, name: error.name, stack: error.stack }
+ : undefined;
+}
+
+function deserializeError(value: unknown): Error | undefined {
+ if (!isRecord(value) || typeof value.message !== "string") return undefined;
+ const error = new Error(value.message);
+ if (typeof value.name === "string") error.name = value.name;
+ if (typeof value.stack === "string") error.stack = value.stack;
+ return error;
+}
+
+function isTurnTerminalType(
+ value: string,
+): value is "turn.cancelled" | "turn.completed" | "turn.failed" {
+ return value === "turn.cancelled" || value === "turn.completed" || value === "turn.failed";
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+ return typeof value === "object" && value !== null && !Array.isArray(value);
+}
diff --git a/packages/eve/src/harness/agent-trace-span-processor.test.ts b/packages/eve/src/harness/agent-trace-span-processor.test.ts
new file mode 100644
index 000000000..ec830ab6e
--- /dev/null
+++ b/packages/eve/src/harness/agent-trace-span-processor.test.ts
@@ -0,0 +1,66 @@
+import { describe, expect, it, vi } from "vitest";
+
+import { AgentTraceSpanProcessor } from "#harness/agent-trace-span-processor.js";
+
+describe("AgentTraceSpanProcessor", () => {
+ it("routes an agent trace and releases it at session terminal", () => {
+ const child = {
+ forceFlush: vi.fn(async () => {}),
+ onEnd: vi.fn(),
+ onStart: vi.fn(),
+ shutdown: vi.fn(async () => {}),
+ };
+ const processor = new AgentTraceSpanProcessor([child]);
+ const unrelated = span("unrelated");
+ processor.onStart(unrelated, {});
+ processor.onEnd(unrelated);
+ expect(child.onStart).not.toHaveBeenCalled();
+
+ const turn = span("trace-1", { "agent.session.id": "session-1" });
+ const user = span("trace-1");
+ processor.onStart(turn, {});
+ processor.onStart(user, {});
+ processor.onEnd(user);
+ processor.onEnd(turn);
+ expect(child.onStart).toHaveBeenCalledTimes(2);
+ expect(child.onEnd).toHaveBeenCalledTimes(2);
+
+ processor.releaseSession("session-1");
+ processor.onEnd(span("trace-1"));
+ expect(child.onEnd).toHaveBeenCalledTimes(2);
+ });
+
+ it("excludes Workflow instrumentation from an agent trace", () => {
+ const child = {
+ forceFlush: vi.fn(async () => {}),
+ onEnd: vi.fn(),
+ onStart: vi.fn(),
+ shutdown: vi.fn(async () => {}),
+ };
+ const processor = new AgentTraceSpanProcessor([child]);
+ processor.onStart(span("trace-1", { "agent.session.id": "session-1" }), {});
+
+ const workflow = span("trace-1", {}, "workflow");
+ processor.onStart(workflow, {});
+ processor.onEnd(workflow);
+
+ expect(child.onStart).toHaveBeenCalledTimes(1);
+ expect(child.onEnd).not.toHaveBeenCalled();
+ });
+});
+
+function span(
+ traceId: string,
+ attributes: Record<string, unknown> = {},
+ scope = "test",
+): {
+ readonly attributes: Record<string, unknown>;
+ readonly instrumentationScope: { readonly name: string };
+ readonly spanContext: () => { readonly traceId: string };
+} {
+ return {
+ attributes,
+ instrumentationScope: { name: scope },
+ spanContext: () => ({ traceId }),
+ };
+}
diff --git a/packages/eve/src/harness/agent-trace-span-processor.ts b/packages/eve/src/harness/agent-trace-span-processor.ts
new file mode 100644
index 000000000..dd88a0892
--- /dev/null
+++ b/packages/eve/src/harness/agent-trace-span-processor.ts
@@ -0,0 +1,73 @@
+import type { SpanProcessor } from "#compiled/@vercel/otel/index.js";
+
+interface SpanLike {
+ readonly attributes: Readonly<Record<string, unknown>>;
+ readonly instrumentationScope?: { readonly name?: string };
+ readonly spanContext: () => { readonly traceId: string };
+}
+
+/** Routes spans from agent-owned traces to provider-neutral child processors. */
+export class AgentTraceSpanProcessor implements SpanProcessor {
+ readonly #children: readonly SpanProcessor[];
+ readonly #ownedTraceIds = new Set<string>();
+ readonly #sessionTraceIds = new Map<string, string>();
+ #attached = false;
+
+ constructor(children: readonly SpanProcessor[]) {
+ this.#children = children;
+ }
+
+ async forceFlush(): Promise<void> {
+ await Promise.all(this.#children.map((child) => child.forceFlush()));
+ }
+
+ isAttached(): boolean {
+ return this.#attached;
+ }
+
+ onStart(span: unknown, parentContext: unknown): void {
+ this.#attached = true;
+ if (!isSpanLike(span)) return;
+ const sessionId = span.attributes["agent.session.id"];
+ if (typeof sessionId === "string") {
+ const traceId = span.spanContext().traceId;
+ this.#ownedTraceIds.add(traceId);
+ this.#sessionTraceIds.set(sessionId, traceId);
+ }
+ if (!this.#accepts(span)) return;
+ for (const child of this.#children) child.onStart(span, parentContext);
+ }
+
+ onEnd(span: unknown): void {
+ if (!isSpanLike(span) || !this.#accepts(span)) return;
+ for (const child of this.#children) child.onEnd(span);
+ }
+
+ releaseSession(sessionId: string): void {
+ const traceId = this.#sessionTraceIds.get(sessionId);
+ if (traceId === undefined) return;
+ this.#ownedTraceIds.delete(traceId);
+ this.#sessionTraceIds.delete(sessionId);
+ }
+
+ async shutdown(): Promise<void> {
+ await Promise.all(this.#children.map((child) => child.shutdown()));
+ }
+
+ #accepts(span: SpanLike): boolean {
+ return (
+ span.instrumentationScope?.name !== "workflow" &&
+ this.#ownedTraceIds.has(span.spanContext().traceId)
+ );
+ }
+}
+
+function isSpanLike(value: unknown): value is SpanLike {
+ return (
+ typeof value === "object" &&
+ value !== null &&
+ "attributes" in value &&
+ "spanContext" in value &&
+ typeof value.spanContext === "function"
+ );
+}
diff --git a/packages/eve/src/harness/instrumentation-native-events.test.ts b/packages/eve/src/harness/instrumentation-native-events.test.ts
index 96ae284c8..80eda5a8c 100644
--- a/packages/eve/src/harness/instrumentation-native-events.test.ts
+++ b/packages/eve/src/harness/instrumentation-native-events.test.ts
@@ -70,4 +70,24 @@ describe("createInstrumentationHandleEvent", () => {
}),
).toBeUndefined();
});
+
+ it("uses the restored turn id when a continuation step emits a session transition", async () => {
+ const events: unknown[] = [];
+ const handleEvent = createInstrumentationHandleEvent({
+ handleEvent: async () => {},
+ hooks: {
+ after: async () => {},
+ before: async () => {},
+ publish: async (event) => {
+ events.push(event);
+ },
+ },
+ sessionId: "session-1",
+ turnId: "turn-1",
+ })!;
+
+ await handleEvent(createSessionWaitingEvent("continue-1"));
+
+ expect(events).toEqual([{ sessionId: "session-1", turnId: "turn-1", type: "session.waiting" }]);
+ });
});
diff --git a/packages/eve/src/harness/instrumentation-native-events.ts b/packages/eve/src/harness/instrumentation-native-events.ts
index a720c657d..f03fcc639 100644
--- a/packages/eve/src/harness/instrumentation-native-events.ts
+++ b/packages/eve/src/harness/instrumentation-native-events.ts
@@ -11,6 +11,7 @@ export interface CreateInstrumentationHandleEventInput {
readonly hooks?: InstrumentationHooks;
readonly rootSessionId?: string;
readonly sessionId: string;
+ readonly turnId?: string;
}
/** Publishes eve-native lifecycle transitions after durable event acceptance. */
@@ -22,7 +23,7 @@ export function createInstrumentationHandleEvent(
const handleEvent = input.handleEvent;
const hooks = input.hooks;
- let activeTurnId: string | undefined;
+ let activeTurnId = input.turnId;
return async (event, messages) => {
await handleEvent(event, messages);
const lifecycleEvent = toLifecycleEvent(event, input, activeTurnId);
diff --git a/packages/eve/src/harness/instrumentation-runtime.ts b/packages/eve/src/harness/instrumentation-runtime.ts
new file mode 100644
index 000000000..33c99efed
--- /dev/null
+++ b/packages/eve/src/harness/instrumentation-runtime.ts
@@ -0,0 +1,37 @@
+import type {
+ InstrumentationContextRunner,
+ InstrumentationHooks,
+} from "#harness/instrumentation-lifecycle.js";
+
+const INSTRUMENTATION_RUNTIME_KEY = Symbol.for("eve.instrumentation-runtime");
+
+/** Process-wide runtime consumed by every harness execution surface. */
+export interface InstrumentationRuntime {
+ readonly forceFlush: () => Promise<void>;
+ readonly hooks: InstrumentationHooks;
+ readonly runInContext: InstrumentationContextRunner;
+}
+
+/** Instrumentation capabilities consumed inside one harness execution. */
+export type HarnessInstrumentation = Pick<InstrumentationRuntime, "hooks" | "runInContext">;
+
+type InstrumentationGlobal = typeof globalThis & {
+ [INSTRUMENTATION_RUNTIME_KEY]?: InstrumentationRuntime;
+};
+
+const globalRuntime = globalThis as InstrumentationGlobal;
+
+/** Registers the process instrumentation runtime before agent execution begins. */
+export function registerInstrumentationRuntime(
+ runtime: InstrumentationRuntime,
+): InstrumentationRuntime {
+ const existing = globalRuntime[INSTRUMENTATION_RUNTIME_KEY];
+ if (existing !== undefined) return existing;
+ globalRuntime[INSTRUMENTATION_RUNTIME_KEY] = runtime;
+ return runtime;
+}
+
+/** Returns the process instrumentation runtime, when one was installed. */
+export function getInstrumentationRuntime(): InstrumentationRuntime | undefined {
+ return globalRuntime[INSTRUMENTATION_RUNTIME_KEY];
+}
diff --git a/packages/eve/src/harness/local-instrumentation-runtime-conflict.scenario.test.ts b/packages/eve/src/harness/local-instrumentation-runtime-conflict.scenario.test.ts
new file mode 100644
index 000000000..359100af2
--- /dev/null
+++ b/packages/eve/src/harness/local-instrumentation-runtime-conflict.scenario.test.ts
@@ -0,0 +1,30 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { trace } from "@opentelemetry/api";
+import { BasicTracerProvider } from "@opentelemetry/sdk-trace-base";
+import { afterEach, describe, expect, it } from "vitest";
+
+import { installLocalInstrumentationRuntime } from "#harness/local-instrumentation-runtime.js";
+
+let appRoot: string | undefined;
+
+afterEach(async () => {
+ if (appRoot !== undefined) await rm(appRoot, { force: true, recursive: true });
+});
+
+describe("local instrumentation runtime ownership", () => {
+ it("fails loudly when another global tracer provider already exists", async () => {
+ appRoot = await mkdtemp(join(tmpdir(), "eve-local-traces-conflict-"));
+ expect(trace.setGlobalTracerProvider(new BasicTracerProvider())).toBe(true);
+
+ expect(() =>
+ installLocalInstrumentationRuntime({
+ appRoot: appRoot!,
+ frameworkVersion: "test",
+ serviceName: "test-agent",
+ }),
+ ).toThrow(/another runtime already exists/u);
+ });
+});
diff --git a/packages/eve/src/harness/local-instrumentation-runtime.scenario.test.ts b/packages/eve/src/harness/local-instrumentation-runtime.scenario.test.ts
new file mode 100644
index 000000000..213f1bf98
--- /dev/null
+++ b/packages/eve/src/harness/local-instrumentation-runtime.scenario.test.ts
@@ -0,0 +1,196 @@
+import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { ROOT_CONTEXT, trace } from "@opentelemetry/api";
+import { BasicTracerProvider } from "@opentelemetry/sdk-trace-base";
+import { afterEach, describe, expect, it } from "vitest";
+
+import { ContextContainer, contextStorage } from "#context/container.js";
+import { createAiSdkHookBridge } from "#harness/ai-sdk-hook-bridge.js";
+import type { InstrumentationAttemptScope } from "#harness/instrumentation-lifecycle.js";
+import { installLocalInstrumentationRuntime } from "#harness/local-instrumentation-runtime.js";
+import { LocalTraceSpanProcessor } from "#harness/local-trace-span-processor.js";
+
+const temporaryDirectories: string[] = [];
+
+afterEach(async () => {
+ await Promise.all(
+ temporaryDirectories.splice(0).map((path) => rm(path, { force: true, recursive: true })),
+ );
+});
+
+describe("local instrumentation runtime", () => {
+ it("persists agent, AI, and user spans in one trace", async () => {
+ const appRoot = await mkdtemp(join(tmpdir(), "eve-local-traces-"));
+ temporaryDirectories.push(appRoot);
+ const runtime = installLocalInstrumentationRuntime({
+ appRoot,
+ frameworkVersion: "test",
+ serviceName: "test-agent",
+ });
+ const scope: InstrumentationAttemptScope = {
+ attemptId: "session-1:turn-1:0:0",
+ attemptIndex: 0,
+ functionId: "weather",
+ rootSessionId: "session-1",
+ sessionId: "session-1",
+ stepIndex: 0,
+ turnId: "turn-1",
+ };
+
+ await contextStorage.run(new ContextContainer(), async () => {
+ await runtime.hooks.publish({
+ agentName: "weather",
+ rootSessionId: "session-1",
+ sessionId: "session-1",
+ type: "session.started",
+ });
+ await runtime.hooks.publish({
+ rootSessionId: "session-1",
+ sequence: 0,
+ sessionId: "session-1",
+ turnId: "turn-1",
+ type: "turn.started",
+ });
+ const bridge = createAiSdkHookBridge(scope, runtime.hooks, runtime.runInContext);
+ Reflect.apply(bridge.onStart!, bridge, [
+ {
+ callId: "call-1",
+ modelId: "model-1",
+ operationId: "ai.streamText",
+ provider: "test",
+ },
+ ]);
+ await Reflect.apply(bridge.onStepStart!, bridge, [{ callId: "call-1", stepNumber: 0 }]);
+ await Reflect.apply(bridge.onLanguageModelCallStart!, bridge, [
+ { callId: "call-1", modelId: "model-1", provider: "test", tools: undefined },
+ ]);
+ await bridge.executeLanguageModelCall!({
+ callId: "call-1",
+ execute: async () => {
+ const span = trace.getTracer("test-user").startSpan("user.model-work");
+ await Promise.resolve();
+ span.end();
+ },
+ });
+ await Reflect.apply(bridge.onLanguageModelCallEnd!, bridge, [
+ {
+ callId: "call-1",
+ content: [],
+ finishReason: "stop",
+ performance: { responseTimeMs: 1 },
+ responseId: "response-1",
+ usage: { inputTokens: 1, outputTokens: 1 },
+ },
+ ]);
+ await Reflect.apply(bridge.onToolExecutionStart!, bridge, [
+ {
+ callId: "call-1",
+ toolCall: { input: {}, toolCallId: "tool-1", toolName: "weather" },
+ },
+ ]);
+ await bridge.executeTool!({
+ callId: "call-1",
+ execute: async () => {
+ const span = trace.getTracer("test-user").startSpan("user.tool-work");
+ await Promise.resolve();
+ span.end();
+ },
+ toolCallId: "tool-1",
+ });
+ await Reflect.apply(bridge.onToolExecutionEnd!, bridge, [
+ {
+ callId: "call-1",
+ messages: [],
+ toolCall: { input: {}, toolCallId: "tool-1", toolName: "weather" },
+ toolExecutionMs: 1,
+ toolOutput: { output: { temperature: 72 }, type: "tool-result" },
+ },
+ ]);
+ await runtime.hooks.publish({ scope, type: "attempt.completed" });
+ });
+ await runtime.forceFlush();
+
+ const traceRoot = join(appRoot, ".eve", "traces", "v1");
+ const [traceId] = await readdir(traceRoot);
+ expect(traceId).toMatch(/^[0-9a-f]{32}$/u);
+ const segmentRoot = join(traceRoot, traceId!, "segments");
+ const segments = await readdir(segmentRoot);
+ const spanGroups = await Promise.all(
+ segments.map(async (file) => {
+ const payload = JSON.parse(await readFile(join(segmentRoot, file), "utf8")) as OtlpRequest;
+ return payload.resourceSpans.flatMap((resource) =>
+ resource.scopeSpans.flatMap((scopeSpans) => scopeSpans.spans),
+ );
+ }),
+ );
+ const spans = spanGroups.flat();
+ expect(spans.map((span) => span.name)).toEqual(
+ expect.arrayContaining([
+ "agent.turn",
+ "agent.step",
+ "ai.streamText",
+ "ai.streamText.doStream",
+ "agent.action",
+ "ai.toolCall",
+ "user.model-work",
+ "user.tool-work",
+ ]),
+ );
+ expect(span(spans, "ai.streamText").parentSpanId).toBe(span(spans, "agent.step").spanId);
+ expect(span(spans, "ai.streamText.doStream").parentSpanId).toBe(
+ span(spans, "ai.streamText").spanId,
+ );
+ expect(span(spans, "user.model-work").parentSpanId).toBe(
+ span(spans, "ai.streamText.doStream").spanId,
+ );
+ expect(span(spans, "agent.action").parentSpanId).toBe(span(spans, "agent.step").spanId);
+ expect(span(spans, "ai.toolCall").parentSpanId).toBe(span(spans, "agent.action").spanId);
+ expect(span(spans, "user.tool-work").parentSpanId).toBe(span(spans, "ai.toolCall").spanId);
+ });
+
+ it("keeps segments from overlapping worker writers", async () => {
+ const appRoot = await mkdtemp(join(tmpdir(), "eve-local-traces-workers-"));
+ temporaryDirectories.push(appRoot);
+ const firstProcessor = new LocalTraceSpanProcessor(appRoot);
+ const secondProcessor = new LocalTraceSpanProcessor(appRoot);
+ const firstProvider = new BasicTracerProvider({ spanProcessors: [firstProcessor] });
+ const secondProvider = new BasicTracerProvider({ spanProcessors: [secondProcessor] });
+ const parent = trace.setSpan(
+ ROOT_CONTEXT,
+ trace.wrapSpanContext({
+ spanId: "b".repeat(16),
+ traceFlags: 1,
+ traceId: "a".repeat(32),
+ }),
+ );
+
+ firstProvider.getTracer("worker-1").startSpan("worker.one", {}, parent).end();
+ secondProvider.getTracer("worker-2").startSpan("worker.two", {}, parent).end();
+ await Promise.all([firstProcessor.forceFlush(), secondProcessor.forceFlush()]);
+
+ const segments = await readdir(
+ join(appRoot, ".eve", "traces", "v1", "a".repeat(32), "segments"),
+ );
+ expect(segments).toHaveLength(2);
+ });
+});
+
+interface OtlpSpan {
+ readonly name: string;
+ readonly parentSpanId?: string;
+ readonly spanId: string;
+}
+
+interface OtlpRequest {
+ readonly resourceSpans: ReadonlyArray<{
+ readonly scopeSpans: ReadonlyArray<{
+ readonly spans: ReadonlyArray<OtlpSpan>;
+ }>;
+ }>;
+}
+
+function span(spans: readonly OtlpSpan[], name: string): OtlpSpan {
+ return spans.find((candidate) => candidate.name === name)!;
+}
diff --git a/packages/eve/src/harness/local-instrumentation-runtime.ts b/packages/eve/src/harness/local-instrumentation-runtime.ts
new file mode 100644
index 000000000..4186ff10a
--- /dev/null
+++ b/packages/eve/src/harness/local-instrumentation-runtime.ts
@@ -0,0 +1,63 @@
+import { context, trace } from "#compiled/@opentelemetry/api/index.js";
+import { registerOTel } from "#compiled/@vercel/otel/index.js";
+
+import { ContextAgentTraceStateStore } from "#harness/agent-trace-context-store.js";
+import { createAgentOtelInstrumentation } from "#harness/agent-otel-provider.js";
+import { AgentTraceSpanProcessor } from "#harness/agent-trace-span-processor.js";
+import {
+ createInstrumentationHooks,
+ type InstrumentationProviderDefinition,
+} from "#harness/instrumentation-lifecycle.js";
+import {
+ getInstrumentationRuntime,
+ registerInstrumentationRuntime,
+ type InstrumentationRuntime,
+} from "#harness/instrumentation-runtime.js";
+import { LocalTraceSpanProcessor } from "#harness/local-trace-span-processor.js";
+
+/** Installs the zero-config local OTel runtime once in an `eve dev` worker. */
+export function installLocalInstrumentationRuntime(input: {
+ readonly appRoot: string;
+ readonly frameworkVersion: string;
+ readonly serviceName: string;
+}): InstrumentationRuntime {
+ const existing = getInstrumentationRuntime();
+ if (existing !== undefined) return existing;
+
+ const persistence = new LocalTraceSpanProcessor(input.appRoot);
+ const processor = new AgentTraceSpanProcessor([persistence]);
+ registerOTel({
+ autoDetectResources: false,
+ instrumentations: [],
+ propagators: ["none"],
+ serviceName: input.serviceName,
+ spanProcessors: [processor],
+ });
+ const probe = trace.getTracer("eve.registration").startSpan("eve.otel.registration");
+ const activeContext = trace.setSpan(context.active(), probe);
+ const contextAttached = context.with(activeContext, () => trace.getActiveSpan() === probe);
+ probe.end();
+ if (!processor.isAttached() || !contextAttached) {
+ throw new Error("eve could not register OpenTelemetry because another runtime already exists.");
+ }
+ const agentOtel = createAgentOtelInstrumentation({
+ frameworkVersion: input.frameworkVersion,
+ stateStore: new ContextAgentTraceStateStore(),
+ tracer: trace.getTracer("eve.agent", input.frameworkVersion),
+ });
+ const releaseTrace: InstrumentationProviderDefinition = {
+ events: {
+ "session.completed": releaseSessionTrace,
+ "session.failed": releaseSessionTrace,
+ },
+ };
+ return registerInstrumentationRuntime({
+ forceFlush: () => processor.forceFlush(),
+ hooks: createInstrumentationHooks([agentOtel.hook, releaseTrace]),
+ runInContext: agentOtel.runInContext,
+ });
+
+ function releaseSessionTrace(event: { readonly sessionId: string }): void {
+ processor.releaseSession(event.sessionId);
+ }
+}
diff --git a/packages/eve/src/harness/local-trace-span-processor.ts b/packages/eve/src/harness/local-trace-span-processor.ts
new file mode 100644
index 000000000..bcbf460d7
--- /dev/null
+++ b/packages/eve/src/harness/local-trace-span-processor.ts
@@ -0,0 +1,70 @@
+import { mkdir } from "node:fs/promises";
+import { join } from "node:path";
+
+import { JsonTraceSerializer } from "#compiled/@opentelemetry/otlp-transformer/index.js";
+import type { SpanProcessor } from "#compiled/@vercel/otel/index.js";
+
+import { createLogger, formatError } from "#internal/logging.js";
+import { atomicWriteFile } from "#shared/atomic-write-file.js";
+
+interface ReadableSpanLike {
+ readonly spanContext: () => { readonly spanId: string; readonly traceId: string };
+}
+
+const log = createLogger("harness.local-trace-span-processor");
+
+/** Persists spans from agent-owned traces as immutable OTLP/JSON segments. */
+export class LocalTraceSpanProcessor implements SpanProcessor {
+ readonly #appRoot: string;
+ #queue = Promise.resolve();
+ #reportedFailure = false;
+
+ constructor(appRoot: string) {
+ this.#appRoot = appRoot;
+ }
+
+ forceFlush(): Promise<void> {
+ return this.#queue;
+ }
+
+ onStart(): void {}
+
+ onEnd(span: unknown): void {
+ if (!isReadableSpan(span)) return;
+ const { spanId, traceId } = span.spanContext();
+ if (!isHexId(traceId, 32) || !isHexId(spanId, 16)) return;
+ const payload = JsonTraceSerializer.serializeRequest([span]);
+ if (payload === undefined) return;
+
+ this.#queue = this.#queue
+ .then(async () => {
+ const directory = join(this.#appRoot, ".eve", "traces", "v1", traceId, "segments");
+ await mkdir(directory, { recursive: true });
+ await atomicWriteFile(join(directory, `${spanId}.otlp.json`), payload);
+ })
+ .catch((error: unknown) => {
+ if (!this.#reportedFailure) {
+ this.#reportedFailure = true;
+ log.warn("local trace persistence failed", { error: formatError(error) });
+ }
+ });
+ }
+
+ shutdown(): Promise<void> {
+ return this.forceFlush();
+ }
+}
+
+function isReadableSpan(value: unknown): value is ReadableSpanLike {
+ return (
+ typeof value === "object" &&
+ value !== null &&
+ "attributes" in value &&
+ "spanContext" in value &&
+ typeof value.spanContext === "function"
+ );
+}
+
+function isHexId(value: string, length: number): boolean {
+ return value.length === length && /^[0-9a-f]+$/u.test(value);
+}
diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts
index b25ee2d52..970bc105c 100644
--- a/packages/eve/src/harness/tool-loop.test.ts
+++ b/packages/eve/src/harness/tool-loop.test.ts
@@ -51,7 +51,10 @@ import {
setTurnUsageState,
} from "#harness/turn-tag-state.js";
import type { HarnessEmitFn, HarnessSession, ToolLoopHarnessConfig } from "#harness/types.js";
-import { createInstrumentationHooks } from "#harness/instrumentation-lifecycle.js";
+import {
+ createInstrumentationHooks,
+ type InstrumentationContextRunner,
+} from "#harness/instrumentation-lifecycle.js";
import {
CONDITIONAL_DELIVERY_INSTRUCTION,
EMPTY_DELIVERY_SENTINEL,
@@ -3838,7 +3841,10 @@ describe("createToolLoopHarness", () => {
},
});
const config: ToolLoopHarnessConfig = {
- instrumentationHooks: createInstrumentationHooks([]),
+ instrumentation: {
+ hooks: createInstrumentationHooks([]),
+ runInContext: (_operation, execute) => execute(),
+ },
mode: "conversation",
resolveModel: vi.fn().mockResolvedValue("anthropic/claude-opus-4.7"),
tools: new Map([
@@ -8853,8 +8859,9 @@ describe("createToolLoopHarness", () => {
const hooks = createInstrumentationHooks([
{ events: { "attempt.completed": attemptCompleted } },
]);
+ const runInContext: InstrumentationContextRunner = (_operation, execute) => execute();
const config = createTestConfig("conversation", undefined, {
- instrumentationHooks: hooks,
+ instrumentation: { hooks, runInContext },
});
const runStep = createToolLoopHarness(config);
@@ -8869,6 +8876,7 @@ describe("createToolLoopHarness", () => {
turnId: "turn_0",
}),
hooks,
+ runInContext,
);
const bridge = mockCreateAiSdkHookBridge.mock.results[0]!.value;
const agentCall = vi.mocked(ToolLoopAgent).mock.calls[0]?.[0] as {
@@ -8903,7 +8911,10 @@ describe("createToolLoopHarness", () => {
const hooks = createInstrumentationHooks([]);
const runStep = createToolLoopHarness(
createTestConfig("conversation", undefined, {
- instrumentationHooks: hooks,
+ instrumentation: {
+ hooks,
+ runInContext: (_operation, execute) => execute(),
+ },
}),
);
diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts
index 45a95f741..c33e81e99 100644
--- a/packages/eve/src/harness/tool-loop.ts
+++ b/packages/eve/src/harness/tool-loop.ts
@@ -540,9 +540,10 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
const emit = createInstrumentationHandleEvent({
agentName: config.runtimeIdentity?.agentName,
handleEvent: baseEmit,
- hooks: config.instrumentationHooks,
+ hooks: config.instrumentation?.hooks,
rootSessionId: parent?.rootSessionId,
sessionId: session.sessionId,
+ turnId: activeTurnId(emissionState),
});
// Resolve deferred input, runtime actions, then HITL input; each stage
@@ -897,7 +898,7 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
const effectiveTools = marker ? applyLastToolCacheBreakpoint(modelTools, marker) : modelTools;
- const instrumentationHooks = config.instrumentationHooks;
+ const instrumentationHooks = config.instrumentation?.hooks;
const instrumentationTurnId = activeTurnId(emissionState);
const attemptScope: InstrumentationAttemptScope | undefined =
instrumentationHooks === undefined
@@ -914,7 +915,11 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {
const bridgeIntegration =
attemptScope === undefined || instrumentationHooks === undefined
? undefined
- : createAiSdkHookBridge(attemptScope, instrumentationHooks);
+ : createAiSdkHookBridge(
+ attemptScope,
+ instrumentationHooks,
+ config.instrumentation?.runInContext,
+ );
const hooks = buildStepHooks({
cachePath,
diff --git a/packages/eve/src/harness/types.ts b/packages/eve/src/harness/types.ts
index a08d05089..88a4c69d7 100644
--- a/packages/eve/src/harness/types.ts
+++ b/packages/eve/src/harness/types.ts
@@ -12,7 +12,7 @@ import type { JsonObject } from "#shared/json.js";
import type { InternalToolDefinition } from "#shared/tool-definition.js";
import type { AgentReasoningDefinition } from "#shared/agent-definition.js";
import type { HarnessToolDefinition } from "#harness/execute-tool.js";
-import type { InstrumentationHooks } from "#harness/instrumentation-lifecycle.js";
+import type { HarnessInstrumentation } from "#harness/instrumentation-runtime.js";
/**
* Serializable tool definition stored on the session.
@@ -245,7 +245,7 @@ export interface ToolLoopHarnessConfig {
* Internal lifecycle hooks injected into each actual model attempt.
* Omitted in production until an instrumentation runtime opts in.
*/
- readonly instrumentationHooks?: InstrumentationHooks;
+ readonly instrumentation?: HarnessInstrumentation;
/**
* Execution mode for the current harness.
*
diff --git a/packages/eve/src/internal/nitro/host/create-application-nitro.scenario.test.ts b/packages/eve/src/internal/nitro/host/create-application-nitro.scenario.test.ts
index 0aaaec38d..919138520 100644
--- a/packages/eve/src/internal/nitro/host/create-application-nitro.scenario.test.ts
+++ b/packages/eve/src/internal/nitro/host/create-application-nitro.scenario.test.ts
@@ -186,7 +186,7 @@ describe("application Nitro creation", () => {
delete process.env.VERCEL;
});
- it("installs compiled artifacts before constructing the Workflow world", async () => {
+ it("installs local tracing and compiled artifacts before constructing the Workflow world", async () => {
const nitroStub = createNitroStub();
createNitroMock.mockResolvedValueOnce(nitroStub.nitro);
@@ -196,12 +196,36 @@ describe("application Nitro creation", () => {
await createDevelopmentApplicationNitro(preparedHost);
const plugins = createNitroMock.mock.calls[0]?.[0].plugins as string[];
+ const localTracing = plugins.findIndex((plugin) =>
+ plugin.includes("local-tracing-runtime-plugin.ts"),
+ );
+ expect(localTracing).toBeGreaterThanOrEqual(0);
+ expect(localTracing).toBeLessThan(
+ plugins.indexOf(preparedHost.compiledArtifacts.bootstrapPath),
+ );
expect(plugins.indexOf(preparedHost.compiledArtifacts.bootstrapPath)).toBeLessThan(
plugins.indexOf(preparedHost.compiledArtifacts.workflowWorldPluginPath),
);
});
+ it("preserves authored instrumentation instead of installing local tracing", async () => {
+ const nitroStub = createNitroStub();
+ createNitroMock.mockResolvedValueOnce(nitroStub.nitro);
+ const { createDevelopmentApplicationNitro } =
+ await import("#internal/nitro/host/create-application-nitro.js");
+ const preparedHost = createPreparedHost();
+ preparedHost.compiledArtifacts.instrumentationPluginPath = "/app/instrumentation.mjs";
+
+ await createDevelopmentApplicationNitro(preparedHost);
+
+ const plugins = createNitroMock.mock.calls[0]?.[0].plugins as string[];
+ expect(plugins).toContain("/app/instrumentation.mjs");
+ expect(plugins).not.toEqual(
+ expect.arrayContaining([expect.stringContaining("local-tracing-runtime-plugin.ts")]),
+ );
+ });
+
it("preserves workflow bundle side effects and skips workflow transform for cached bundles", async () => {
const nitroStub = createNitroStub();
createNitroMock.mockResolvedValueOnce(nitroStub.nitro);
diff --git a/packages/eve/src/internal/nitro/host/create-application-nitro.ts b/packages/eve/src/internal/nitro/host/create-application-nitro.ts
index 08cae87b3..89e78ca19 100644
--- a/packages/eve/src/internal/nitro/host/create-application-nitro.ts
+++ b/packages/eve/src/internal/nitro/host/create-application-nitro.ts
@@ -739,6 +739,11 @@ export async function createDevelopmentApplicationNitro(
const nitroBuildDir = preparedHost.workspace.nitroBuildDir;
const bundler = createApplicationNitroBundlerConfiguration(preparedHost, undefined);
const plugins = createApplicationNitroPlugins(preparedHost);
+ if (preparedHost.compiledArtifacts.instrumentationPluginPath === undefined) {
+ plugins.unshift(
+ resolvePackageSourceFilePath("src/internal/nitro/host/local-tracing-runtime-plugin.ts"),
+ );
+ }
await prepareEveVersionedCacheDirectory(nitroBuildDir);
const nitro = await createNitro(
diff --git a/packages/eve/src/internal/nitro/host/local-tracing-runtime-plugin.ts b/packages/eve/src/internal/nitro/host/local-tracing-runtime-plugin.ts
new file mode 100644
index 000000000..af0ca6d37
--- /dev/null
+++ b/packages/eve/src/internal/nitro/host/local-tracing-runtime-plugin.ts
@@ -0,0 +1,18 @@
+import { basename } from "node:path";
+
+import { installLocalInstrumentationRuntime } from "#harness/local-instrumentation-runtime.js";
+import { resolveInstalledPackageInfo } from "#internal/application/package.js";
+import { DEVELOPMENT_WORKER_APP_ROOT_ENV } from "#internal/workflow/development-world-protocol.js";
+
+const appRoot = process.env[DEVELOPMENT_WORKER_APP_ROOT_ENV];
+if (appRoot === undefined) {
+ throw new Error(`${DEVELOPMENT_WORKER_APP_ROOT_ENV} is required for local tracing.`);
+}
+
+installLocalInstrumentationRuntime({
+ appRoot,
+ frameworkVersion: resolveInstalledPackageInfo().version,
+ serviceName: basename(appRoot),
+});
+
+export default function installLocalTracingRuntimePlugin(): void {}
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 53cd8500a..218d00d16 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -1173,6 +1173,9 @@ importers:
'@nuxt/kit':
specifier: ^4.0.0
version: 4.4.6([email protected])
+ '@opentelemetry/otlp-transformer':
+ specifier: 0.214.0
+ version: 0.214.0(@opentelemetry/[email protected])
'@opentelemetry/sdk-trace-base':
specifier: ^2.6.1
version: 2.6.1(@opentelemetry/[email protected])
@@ -1200,6 +1203,9 @@ importers:
'@vercel/oidc':
specifier: 3.8.0
version: 3.8.0
+ '@vercel/otel':
+ specifier: 2.1.3
+ version: 2.1.3(@opentelemetry/[email protected])(@opentelemetry/[email protected])(@opentelemetry/[email protected](@opentelemetry/[email protected])([email protected]))(@opentelemetry/[email protected](@opentelemetry/[email protected]))(@opentelemetry/[email protected](@opentelemetry/[email protected]))(@opentelemetry/[email protected](@opentelemetry/[email protected]))(@opentelemetry/[email protected](@opentelemetry/[email protected]))
'@vercel/sandbox':
specifier: 'catalog:'
version: 2.8.0
@@ -3641,6 +3647,12 @@ packages:
peerDependencies:
'@opentelemetry/api': ^1.3.0
+ '@opentelemetry/[email protected]':
+ resolution: {integrity: sha512-DSaYcuBRh6uozfsWN3R8HsN0yDhCuWP7tOFdkUOVaWD1KVJg8m4qiLUsg/tNhTLS9HUYUcwNpwL2eroLtsZZ/w==}
+ engines: {node: ^18.19.0 || >=20.6.0}
+ peerDependencies:
+ '@opentelemetry/api': ^1.3.0
+
'@opentelemetry/[email protected]':
resolution: {integrity: sha512-lg6lkOU08Az23jVcn/0Els9HP+V8PnR4Km6p0KgpTggS0n/WuhnmY64rSh83Of9iR9nD+dpWr6adlcX8KzAwjg==}
engines: {node: ^18.19.0 || >=20.6.0}
@@ -3659,6 +3671,12 @@ packages:
peerDependencies:
'@opentelemetry/api': '>=1.3.0 <1.10.0'
+ '@opentelemetry/[email protected]':
+ resolution: {integrity: sha512-zf6acnScjhsaBUU22zXZ/sLWim1dfhUAbGXdMmHmNG3LfBnQ3DKsOCITb2IZwoUsNNMTogqFKBnlIPPftUgGwA==}
+ engines: {node: ^18.19.0 || >=20.6.0}
+ peerDependencies:
+ '@opentelemetry/api': '>=1.4.0 <1.10.0'
+
'@opentelemetry/[email protected]':
resolution: {integrity: sha512-FaDcazjyMp7TZZZAsqbo4IkovP0UegoCu0EBkiNt+qCqvUf7FPAsfcrZ3+ZEkKgXZ/jHafop+JoGPDk3A0SmLg==}
engines: {node: ^18.19.0 || >=20.6.0}
@@ -3671,6 +3689,12 @@ packages:
peerDependencies:
'@opentelemetry/api': '>=1.9.0 <1.10.0'
+ '@opentelemetry/[email protected]':
+ resolution: {integrity: sha512-9t9hJHX15meBy2NmTJxL+NJfXmnausR2xUDvE19XQce0Qi/GBtDGamU8nS1RMbdgDmhgpm3VaOu2+fiS/SfTpQ==}
+ engines: {node: ^18.19.0 || >=20.6.0}
+ peerDependencies:
+ '@opentelemetry/api': '>=1.9.0 <1.10.0'
+
'@opentelemetry/[email protected]':
resolution: {integrity: sha512-r86ut4T1e8vNwB35CqCcKd45yzqH6/6Wzvpk2/cZB8PsPLlZFTvrh8yfOS3CYZYcUmAx4hHTZJ8AO8Dj8nrdhw==}
engines: {node: ^18.19.0 || >=20.6.0}
@@ -17637,6 +17661,17 @@ snapshots:
'@opentelemetry/core': 2.10.0(@opentelemetry/[email protected])
'@opentelemetry/otlp-transformer': 0.221.0(@opentelemetry/[email protected])
+ '@opentelemetry/[email protected](@opentelemetry/[email protected])':
+ dependencies:
+ '@opentelemetry/api': 1.9.1
+ '@opentelemetry/api-logs': 0.214.0
+ '@opentelemetry/core': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/resources': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/sdk-logs': 0.214.0(@opentelemetry/[email protected])
+ '@opentelemetry/sdk-metrics': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/sdk-trace-base': 2.6.1(@opentelemetry/[email protected])
+ protobufjs: 7.6.5
+
'@opentelemetry/[email protected](@opentelemetry/[email protected])':
dependencies:
'@opentelemetry/api': 1.9.1
@@ -17659,6 +17694,14 @@ snapshots:
'@opentelemetry/core': 2.6.1(@opentelemetry/[email protected])
'@opentelemetry/semantic-conventions': 1.43.0
+ '@opentelemetry/[email protected](@opentelemetry/[email protected])':
+ dependencies:
+ '@opentelemetry/api': 1.9.1
+ '@opentelemetry/api-logs': 0.214.0
+ '@opentelemetry/core': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/resources': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/semantic-conventions': 1.43.0
+
'@opentelemetry/[email protected](@opentelemetry/[email protected])':
dependencies:
'@opentelemetry/api': 1.9.1
@@ -17673,6 +17716,12 @@ snapshots:
'@opentelemetry/core': 2.10.0(@opentelemetry/[email protected])
'@opentelemetry/resources': 2.10.0(@opentelemetry/[email protected])
+ '@opentelemetry/[email protected](@opentelemetry/[email protected])':
+ dependencies:
+ '@opentelemetry/api': 1.9.1
+ '@opentelemetry/core': 2.6.1(@opentelemetry/[email protected])
+ '@opentelemetry/resources': 2.6.1(@opentelemetry/[email protected])
+
'@opentelemetry/[email protected](@opentelemetry/[email protected])':
dependencies:
'@opentelemetry/api': 1.9.1
diff --git a/research/provider-neutral-local-observability.md b/research/provider-neutral-local-observability.md
index c9852b2ed..ef35c5667 100644
--- a/research/provider-neutral-local-observability.md
+++ b/research/provider-neutral-local-observability.md
@@ -100,12 +100,17 @@ providers.
The local provider:
-- uses its own tracer provider;
+- is registered privately by eve while using the process OTel runtime so nested user spans share
+ agent context;
- creates one trace per eve session, independently from Workflow context;
-- never installs a global tracer provider or context manager;
+- explicitly roots session context instead of inheriting the ambient Workflow span;
- never captures Workflow spans;
- persists traces as OTLP/JSON for later inspection.
+Immutable OTLP/JSON segments are the canonical local store. A future DuckDB or dashboard index is a
+rebuildable consumer of that spool, not a second source of trace identity or Workflow continuation
+state.
+
Authored instrumentation and local capture may observe the same attempt concurrently without
executing the model or tool more than once.