Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

docs: design a framework-neutral agent tracing library (#4158)

Chad Hietala7342aee0527e
diff --git a/research/agent-tracing-library.md b/research/agent-tracing-library.md
new file mode 100644
index 000000000..2b316beac
--- /dev/null
+++ b/research/agent-tracing-library.md
@@ -0,0 +1,288 @@
+---
+issue: TBD
+status: in-progress
+last_updated: "2026-10-13"
+---
+
+# Agent tracing library
+
+## Summary
+
+Agents outside eve need the same trace topology, capture rules, and durable
+recovery that eve has, without adopting eve's runtime. This plan extracts a
+framework-neutral library, `@eve/tracing`, with one constructor:
+`createAgentTracing`. The host passes an `AgentTelemetry` object that writes
+spans and, for durable turns, a `TraceCheckpointer`. The library owns span
+topology, capture, completion, checkpoint keys, and hydration. A durable turn
+resumes when the host calls the same operations again, in any process.
+
+eve becomes the first consumer. Trace schema version 4, span names, topology,
+and durable span IDs stay unchanged.
+
+The work is split into a stack: this plan (#4158), the library (#4160), and the
+eve migration (#4164).
+
+## Entrypoints
+
+| Entrypoint | Contents |
+| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `@eve/tracing` | `createAgentTracing`, `otelTelemetry`, `AgentSpanIdGenerator`, operation and telemetry types, `currentCapture` |
+| `@eve/tracing/delegation` | Local handoff context and the remote delegation transport |
+| `@eve/tracing/ai-sdk` | `aiSdkTelemetry` and the AI SDK payload helpers `modelUsage` and `modelContent` |
+| `@eve/tracing/otel` | Plumbing for hosts that run their own spans beside agent tracing: active-operation lookup, capture context, content bounds, and MCP enrichment |
+
+The root depends only on `@opentelemetry/api`. Only `./ai-sdk` uses `ai`, as an
+optional peer dependency. Hosts cannot import private modules; the invariant
+guard (rule 52) enforces this.
+
+The library is a private workspace package, `packages/tracing`, named
+`@eve/tracing` for now. It is not published. eve vendors it the way it vendors
+`@eve/catalog`: `build:compiled` bundles the four entrypoints into
+`#compiled/@eve/tracing/`, in the same chunk group as eve's vendored
+`@opentelemetry/api`, so the published eve package carries one OpenTelemetry
+copy and no new runtime dependency. eve imports only those vendored
+entrypoints.
+
+## Construction
+
+```ts
+const tracing = createAgentTracing({
+ telemetry: otelTelemetry({ provider, idGenerator, samplesTrace, mapping }),
+ checkpointer, // optional: makes turns durable
+ serializer, // optional: defaults to the AI SDK content serializer
+ onError, // the only tracing error channel
+});
+```
+
+`AgentTelemetry` is a library-owned, OTel-shaped interface: `startSpan`,
+`active`, `run`, `suppressed`, `samples`, optional stable `ids`, `forceFlush`,
+and `shutdown`. `otelTelemetry` adapts an OpenTelemetry tracer provider and
+does not register global telemetry. Other backends implement the interface
+directly. Sampling and output mapping belong to the telemetry object.
+
+A checkpointer requires `telemetry.ids`. Without them, construction throws an
+error that names `AgentSpanIdGenerator`.
+
+## Operations
+
+```text
+turn ─┬─ attempt ─┬─ modelCall
+ │ └─ tool ─┬─ approval
+ │ └─ tool (a call made while the tool runs)
+ └─ memory (also under attempts and tools)
+```
+
+Each tool call is one `execute_tool` span. A host often learns about a call
+twice: the runtime records its dispatch, and the SDK reports its run. In a
+durable turn, calling `attempt.tool()` again with the same `callId` returns the
+same call and fills in what the first report lacked, such as `kind`,
+`arguments`, or an earlier start time. Tools with `kind: "subagent-call"` or
+`"remote-agent-call"` delegate to another agent.
+
+### Wrapped and handle forms
+
+Each operation has two forms. The wrapped form runs a callback inside the
+operation, returns its value or rethrows its error, and completes the operation
+either way. An optional `describe(value)` maps the value to a completion, such
+as an outcome or captured output:
+
+```ts
+await tracing.turn({ agentName: "support", identity, sequence: 0 }, (turn) =>
+ turn.attempt({ stepIndex: 0, attempt: 0 }, (attempt) =>
+ attempt.tool({ callId: "lookup", name: "lookup" }, lookup),
+ ),
+);
+```
+
+The handle form omits the callback and returns the operation. Hosts whose
+start and end arrive in different hooks, or different processes, hold the
+handle and complete it themselves:
+
+```ts
+const turn = await tracing.turn({ agentName: "support", identity, sequence: 0 });
+const attempt = await turn.attempt({ stepIndex: 0, attempt: 0 });
+const tool = await attempt.tool({ callId: "lookup", name: "lookup" });
+try {
+ const value = await tool.run(lookup);
+ await tool.complete({ outcome: "completed", output: value });
+} catch (error) {
+ await tool.fail(error);
+}
+await attempt.complete();
+await turn.complete();
+```
+
+### Handles
+
+Every handle shares one interface:
+
+```ts
+interface Operation {
+ readonly type: "activation" | "step" | "model" | "tool" | "approval" | "memory";
+ readonly reference: TraceReference; // trace and span ID, stable once started
+ readonly parent?: TraceReference;
+ readonly startTimeMs: number;
+ readonly finished: boolean;
+ readonly capture: CaptureDecision;
+ run<T>(execute: () => T, ceiling?: CaptureDecision): T;
+ attributes(attributes: Attributes): void;
+ complete(result?: Completion): Promise<void>;
+ fail(error: unknown): Promise<void>;
+ recordError(error: unknown, errorType?: string): void;
+}
+```
+
+- `run(execute, ceiling)` makes the operation active while `execute` runs, so
+ spans that other libraries start inside it become its children. The ceiling
+ limits captured content for work inside the run; it does not remove what the
+ operation already recorded, and it bounds content any operation receives
+ while the run is active.
+- `complete(result)` ends the operation and is idempotent. The completion can
+ carry `outcome`, `failed`, `error` and `errorType`, `output`, `usage`, a model
+ `result`, and `endTimeMs`. `outcomeUnknown: true` ends a turn without an
+ outcome when the host never learned one; a failure is still recorded.
+- `fail(error)` completes the operation as failed. `recordError(error)` marks
+ it failed without ending it, so a completion reported later, possibly by
+ another process, stays failed.
+- Completing a parent closes its unfinished children, except as described in
+ [Lifetimes](#lifetimes).
+
+Handles add the operations that may start beneath them:
+
+| Handle | Starts | Finds |
+| ----------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------- |
+| turn | `attempt`, `memory` | `findAttempt({ stepIndex, attempt })`, `findTool(callId)` |
+| attempt | `modelCall`, `modelStream`, `tool`, `callSubAgent`, `callRemoteAgent`, `memory` | |
+| tool | `approval`, nested `tool`, `memory` | `findApproval(requestId)` |
+| model, approval, memory | nothing | |
+
+`turn.links()` replaces the turn's links before its deferred span is exported.
+`modelStream` returns the stream unchanged and completes when its `completion`
+promise settles, not when the callback returns; parent completion waits for it. Find operations return `undefined` once the operation has
+finished, and are available only in durable turns.
+
+### Turn input
+
+`TurnInput` carries `identity` (`conversationId`, `runId`, `turnId`),
+`agentName`, `sequence`, framework metadata, a capture decision, and
+attributes. Durable hosts can also supply:
+
+- `reference`: a trace and span ID reserved before the turn started, such as a
+ seed already returned to a caller. The sampler still decides the trace flags.
+- `lineage` and `channel`: delegation and delivery metadata the host stores
+ itself, when no in-process handoff carries it.
+- `parent`: the caller's span, for a turn that nests in its caller's trace.
+- `links` and `startTimeMs`. `turn.links()` replaces links that the host learns
+ after the turn starts.
+- `context`: host execution context, such as OTel baggage, for spans this
+ process starts.
+
+`agentName` belongs to the turn, so one instance can trace several agents.
+
+## Durable turns
+
+```ts
+interface TraceCheckpointer {
+ get(key: string): unknown | PromiseLike<unknown>;
+ set(key: string, value: TraceSnapshot): void | PromiseLike<void>;
+ delete(key: string): void | PromiseLike<void>;
+}
+```
+
+- **Keys.** The library chooses keys. A turn is stored at `turn:<runId>:<turnId>`.
+ Inside the tree, attempts are keyed by step index and attempt, tool calls by
+ `callId`, and approvals by `requestId`. Model calls and memory are numbered
+ and get new span IDs when they run again, because a re-run is new work.
+- **Writes.** The library writes JSON after each change, in order. It deletes
+ the entry when the turn and all of its open work have completed. Each node is
+ capped at 64 KiB, and finished children leave the tree.
+- **Resume.** Calling `turn()` with the same identity continues the saved turn.
+ `tracing.resume({ identity })` returns a saved turn without starting one.
+ `turn.findAttempt()`, `turn.findTool(callId)`, and `tool.findApproval()`
+ return open operations, or `undefined`.
+- **Deferred spans.** Durable turns, tool calls, and approvals reserve their IDs
+ at start and export at completion, so a later process can complete them.
+ Attempts and model calls are live spans in the process that runs them.
+- **Process cache.** Concurrent hooks in one process share one tree. Each write
+ stores a revision token; when another process changed the checkpoint, the
+ next call reloads it.
+- **Failures.** An unreadable checkpoint is reported to `onError` with phase
+ `restore`, and a fresh turn starts. Write failures use phase `checkpoint`.
+
+### Lifetimes
+
+| Event | Open children |
+| ----------------------------------- | -------------------------------------------------------- |
+| Any operation fails | Closed as failed |
+| A non-durable operation completes | Closed as abandoned |
+| A durable attempt or turn completes | Tool calls and approvals stay open; other children close |
+| A tool call completes | Its nested calls and approvals close |
+
+Tool calls outlive their attempt and turn because human approval parks a turn:
+the approval resolves in a later turn, and its span still belongs to the
+original call.
+
+## AI SDK integration
+
+`aiSdkTelemetry(tracing, { turn })` returns an AI SDK `Telemetry` integration
+for `generateText`, `streamText`, and agents such as `HarnessAgent`. It keeps
+open operations keyed by the SDK `callId` and `toolCallId`, so hosts do not
+store handles between hooks. It runs model calls and tools inside their spans
+through `executeLanguageModelCall` and `executeTool`. `turn` can be a function
+that chooses the turn for each call.
+
+## Capture and delegation
+
+Capture defaults to metadata only. Children cannot increase capture, and
+declined content never enters a checkpoint. Content is checked when it arrives:
+an operation's own capture, narrowed by any run ceiling active at that moment,
+decides whether arguments, results, and errors are kept. Re-entering a turn with
+a narrower capture narrows it and its open operations and drops retained
+content the new decision declines; content recorded under earlier consent is
+kept. Failure classes survive redaction;
+exception messages do not. Serialized content is capped at 32 KiB.
+
+Local delegation carries lineage through async context. A local callee's first
+turn nests under the delegating tool call in the caller's trace; later turns
+start their own trace. Remote delegation uses a bounded `x-agent-tracing`
+header; the receiver validates it and asks the host to verify the authenticated
+caller before it adopts any lineage. A remote callee starts its own trace with
+an `agent.dispatch` link to the call.
+
+## eve as consumer
+
+eve uses one `AgentTracing` instance over `otelTelemetry`. Its checkpointer
+stores trees in the existing workflow-context trace state, so checkpoints commit
+with each workflow step. eve keeps these concerns:
+
+- session trace seeds, turn ID reservation, and trace policy;
+- channel delivery metadata and principals;
+- tool call locators and anchors that dispatched child agents read;
+- schema-4 attribute names, such as `agent.action.kind` for the library's
+ `agent.tool.kind`, through its output mapping;
+- direct `invokeTool` spans, which have no turn;
+- SDK registration, request and MCP spans, and local trace storage.
+
+eve's hook bridge maps AI SDK hooks to eve lifecycle events, not directly to
+spans, so eve does not use `aiSdkTelemetry`.
+
+## Validation
+
+- Library tests use only package entrypoints. They cover a real
+ `HarnessAgent`, `generateText`, an in-memory agent, delegation, streams, and
+ durable recovery across workers.
+- eve's span and telemetry contract suites pass without changes to their
+ assertions.
+- Live Agent Runs ingestion and fixture e2e run in CI only.
+
+## Open questions
+
+- Completions are one untyped object for every operation. Per-operation types,
+ such as `model.complete(result)` with no `outcome`, would remove ambiguous
+ fields.
+- `lineage` and the in-process handoff describe delegation in two ways.
+- Durable and non-durable modes close open tool calls differently.
+- A turn cannot record its outcome without ending, so eve keeps the outcome
+ until the session transition.
+- Each checkpoint write serializes the open tree. Write volume needs watching
+ on long tool-heavy turns.
diff --git a/research/provider-neutral-local-observability.md b/research/provider-neutral-local-observability.md
deleted file mode 100644
index f98376b3f..000000000
--- a/research/provider-neutral-local-observability.md
+++ /dev/null
@@ -1,195 +0,0 @@
----
-issue: TBD
-status: in-progress
-last_updated: "2026-08-28"
----
-
-# Provider-neutral local observability
-
-## Summary
-
-eve produces local traces without depending on Workflow tracing, changing production
-instrumentation, or coupling observability providers to the AI SDK. The first consumer is
-zero-config tracing in `eve dev`; the lifecycle contract must also support OTel, Vercel,
-Braintrust, and custom providers.
-
-```text
-eve lifecycle boundaries ------------------+
- +--> eve lifecycle hooks --> providers
-AI SDK callbacks -> per-attempt bridge -----+ (local OTel, Vercel, Braintrust, custom)
-AI SDK execute() -> eve-owned context runner
-```
-
-## Lifecycle contract
-
-```ts
-createAiSdkHookBridge(scope, hooks, runInContext): Telemetry;
-
-interface InstrumentationHooks {
- publish(event: InstrumentationPointEvent): Promise<void>;
- before(name: "model.call" | "tool.call", event: StartEvent): Promise<void>;
- after(name: "model.call" | "tool.call", event: TerminalEvent): Promise<void>;
-}
-```
-
-The bridge is an adapter, not a provider. It maps AI SDK callbacks into eve-owned lifecycle
-events, assigns stable attempt/model-call/tool-call identities, terminalizes open operations on
-error and abort, and invokes the trusted `runInContext` while calling execution exactly once. AI
-SDK `callId` values are bridge correlation details, not provider run identities.
-
-The bridge does not create spans, export records, apply provider policy, or publish durable
-eve-native events. Session, turn, compaction, suspension, and canonical step-terminal events are
-published by the harness.
-
-Attempt state is WeakMap-keyed by scope identity so abandoned attempts do not leak. `before` state
-is retained per provider and operation and supplied only to that provider's `after`.
-
-## Providers
-
-A provider is a hook definition: handlers for the eve-owned events it cares about. An OTel provider
-may return spans from `before` and close them in `after`; Braintrust may retain row handles; Vercel
-may enqueue records.
-
-Each provider owns an independent trace graph, rooted from its own serializable session state — not
-from ambient Workflow context or another provider's active context.
-
-Providers never receive an execution function. A built-in integration needing ambient context may
-register a trusted internal `runInContext` alongside its event definition; that is not part of the
-provider contract.
-
-## Local tracing
-
-In `eve dev` eve registers the process tracer provider (`registerOTel`) with a private span
-processor consuming the same lifecycle events as any other provider, so nested AI SDK and user
-spans share agent context. It creates one trace per session window, explicitly roots that window
-rather than inheriting the ambient Workflow span, drops spans from the `workflow` instrumentation
-scope, and persists OTLP/JSON segments under `.eve/traces/v1`.
-
-Those immutable segments are the canonical local store. A future index or dashboard is a
-rebuildable consumer of the spool, not a second source of trace identity.
-
-Authored instrumentation and local capture may observe the same attempt without executing the model
-or tool more than once.
-
-## Agent OTel semantics
-
-```text
-agent.session {agent.session.id}
- +-- invoke_agent {agent.name} {agent.turn.id, agent.turn.sequence}
- +-- agent.step {agent.step.index, agent.step.attempt}
- +-- model-provider spans
- +-- agent.action {agent.action.kind, name, call_id}
- +-- execute_tool {tool.name}
- +-- tool-provider spans
-```
-
-The durable turn and tool-execution boundaries use the OpenTelemetry GenAI agent conventions so
-backends can recognize agent invocations and tool calls without eve-specific mapping. An
-`agent.step` covers one model call and the actions it requests through their resolution.
-`agent.action.kind` remains an open discriminator for eve's broader durable dispatch boundary.
-
-The `agent.*` namespace carries cheap structural attributes only: session/turn/step/attempt
-identity, action identity, and agent and framework identity. Standard `gen_ai.*` attributes describe
-the operation, agent, conversation, and tool. Message content, model output, and tool payloads follow
-each provider's audience-aware `tracePolicy`. The OTel provider owns this mapping; the bridge and
-hooks contain no span names, attributes, or parent contexts.
-
-### The session root is a real span
-
-`agent.session` is a real root span opened with OTel's `root` option, not a synthesized parent
-context. eve persists its span context — including the sampling decision it actually received — and
-later turns parent to that stored context.
-
-This is what makes an authored sampler work. A synthesized parent asserting a sampled flag is a
-valid parent to `ParentBasedSampler`, which resolves through a parent branch and never consults the
-configured root rule. Asserting unsampled instead selects `localParentNotSampled`, which drops
-everything. A genuine root avoids both.
-
-Sampling is therefore per session trace. `agent.session.id` is a creation-time attribute, so an
-author can key a sampler on it.
-
-### Marker spans
-
-The session root is recorded and ended immediately. A session outlives the
-worker that opened it and a span object cannot cross that boundary, so eve records the root,
-persists its span context, and parents later spans through it.
-
-The cost is that a backend reports the root's duration rather than the window's. Turn and step
-durations, where the time actually is, are unaffected, and `eve traces` renders a marker's
-descendant extent in place of its zero duration.
-
-A real duration would mean emitting the span once at session close, with explicit timestamps and a
-span id chosen before the span exists. The ids are reachable — `registerOTel` accepts an
-`idGenerator` — but the close is not: a session that goes idle and never resumes closes on nothing,
-so its root would never be emitted and every span in the trace would reference a parent that never
-arrives. A marker is the only representation that is always emitted. A turn does have a guaranteed
-close, so its `invoke_agent` span carries its real duration.
-
-### Subagents
-
-A local subagent keeps its own session identity but adopts the parent action's `SpanContext`, so
-delegated work appears directly beneath the action that caused it. Remote dispatch sends the same
-context as `traceparent`; the receiver adopts it when available. A child with no propagated context
-opens its own root. `eve traces` resolves any session recorded in a trace, not only the one that
-opened it.
-
-## Runtime integration
-
-```text
-runModelCallWithRetries(attempt)
- -> create stable AttemptScope
- -> createAiSdkHookBridge(scope, hooks, runInContext)
- -> telemetry.integrations = [bridge, authoredOtel?]
-```
-
-The harness supplies session id, turn id, logical step index, and retry attempt index. The bridge
-derives model/tool identities at their start callbacks and keeps callback snapshots in
-invocation-local state; `onStepEnd` and `onEnd` only snapshot, and the harness publishes the
-canonical terminal after durable actions are emitted. On replay the harness recreates attempt state;
-only provider-owned serializable context crosses Workflow boundaries.
-
-AI SDK `onStepStart` is named `attempt.started` because it begins one concrete model attempt, so
-retries have distinct starts and terminals without overloading the stream's `step.started`.
-
-Per-call integrations replace the AI SDK global list; eve composes the bridge with `@ai-sdk/otel`
-when the documented `instrumentation.ts` path is active. Direct `registerTelemetry()` calls are not
-a supported customization surface. Braintrust's documented integration uses eve hooks and
-`defineInstrumentation`, so per-call injection does not disable it.
-
-## Compatibility
-
-Production keeps its current `defineInstrumentation` and `@ai-sdk/otel` behavior until provider
-migration ships. Without lifecycle hooks no bridge is installed and production follows the existing
-authored path unchanged. Stream hooks and `step.started` semantics are unchanged.
-
-## Delivery phases
-
-1. **Lifecycle bridge.** — _landed._ Attempt scope, WeakMap state, typed hooks, before/after
- correlation, trusted context runner, AI SDK callback adapter.
-2. **Per-attempt wiring.** — _landed._ One bridge per retry attempt through
- `telemetry.integrations`, composing authored OTel when active.
-3. **Local OTel provider.** — _landed._ Lifecycle events mapped to the `agent.*` convention.
-4. **Persistence.** — _landed._ OTLP/JSON segments, session context restored across dev worker
- restarts, retention bounds, capture policy.
-5. **Inspection.** — _landed._ `eve traces ls` and `eve traces [trace]`.
-6. **Session windows.** — _in review._ Real `agent.session` root with its recorded sampling
- decision, window rolling, subagent adoption, session-to-windows resolution in `eve traces`.
-7. **Public providers.** — _not started._ Promote the lifecycle contract into public hooks and
- migrate OTel, Vercel, Braintrust, and custom instrumentation onto it.
-
-## Acceptance criteria
-
-- The lifecycle layer imports no OTel types. It does derive AI SDK callback payload shapes from
- `Telemetry`, so `attempt`, `model.call`, and `tool.call` events currently expose AI SDK types to
- providers — a known deviation from provider neutrality, to close in phase 7.
-- Multiple providers observe one attempt while execution occurs exactly once.
-- No provider definition receives model or tool execution functions.
-- Documented authored instrumentation continues to observe eve calls.
-- Each provider builds its trace without inheriting Workflow or another provider's trace.
-- No eve span inherits a synthesized parent, so an authored sampler's root rule is consulted.
-- A session of ordinary length is exactly one trace; a session of any length has bounded traces.
-- `eve traces <session>` resolves every window of a session.
-- Parallel tool calls retain independent provider state.
-- Errors and aborts terminalize all started model and tool operations.
-- Local traces contain no Workflow spans.
-- Enabling local tracing does not change production instrumentation behavior.