The Open Framework for Building Agents
docs: design a framework-neutral agent tracing library (#4158)
Chad Hietala7342aee0527e
diff --git a/research/agent-tracing-library.md b/research/agent-tracing-library.mdnew file mode 100644index 000000000..2b316beac--- /dev/null+++ b/research/agent-tracing-library.md@@ -0,0 +1,288 @@+---+issue: TBD+status: in-progress+last_updated: "2026-10-13"+---++# Agent tracing library++## Summary++Agents outside eve need the same trace topology, capture rules, and durable+recovery that eve has, without adopting eve's runtime. This plan extracts a+framework-neutral library, `@eve/tracing`, with one constructor:+`createAgentTracing`. The host passes an `AgentTelemetry` object that writes+spans and, for durable turns, a `TraceCheckpointer`. The library owns span+topology, capture, completion, checkpoint keys, and hydration. A durable turn+resumes when the host calls the same operations again, in any process.++eve becomes the first consumer. Trace schema version 4, span names, topology,+and durable span IDs stay unchanged.++The work is split into a stack: this plan (#4158), the library (#4160), and the+eve migration (#4164).++## Entrypoints++| Entrypoint | Contents |+| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |+| `@eve/tracing` | `createAgentTracing`, `otelTelemetry`, `AgentSpanIdGenerator`, operation and telemetry types, `currentCapture` |+| `@eve/tracing/delegation` | Local handoff context and the remote delegation transport |+| `@eve/tracing/ai-sdk` | `aiSdkTelemetry` and the AI SDK payload helpers `modelUsage` and `modelContent` |+| `@eve/tracing/otel` | Plumbing for hosts that run their own spans beside agent tracing: active-operation lookup, capture context, content bounds, and MCP enrichment |++The root depends only on `@opentelemetry/api`. Only `./ai-sdk` uses `ai`, as an+optional peer dependency. Hosts cannot import private modules; the invariant+guard (rule 52) enforces this.++The library is a private workspace package, `packages/tracing`, named+`@eve/tracing` for now. It is not published. eve vendors it the way it vendors+`@eve/catalog`: `build:compiled` bundles the four entrypoints into+`#compiled/@eve/tracing/`, in the same chunk group as eve's vendored+`@opentelemetry/api`, so the published eve package carries one OpenTelemetry+copy and no new runtime dependency. eve imports only those vendored+entrypoints.++## Construction++```ts+const tracing = createAgentTracing({+ telemetry: otelTelemetry({ provider, idGenerator, samplesTrace, mapping }),+ checkpointer, // optional: makes turns durable+ serializer, // optional: defaults to the AI SDK content serializer+ onError, // the only tracing error channel+});+```++`AgentTelemetry` is a library-owned, OTel-shaped interface: `startSpan`,+`active`, `run`, `suppressed`, `samples`, optional stable `ids`, `forceFlush`,+and `shutdown`. `otelTelemetry` adapts an OpenTelemetry tracer provider and+does not register global telemetry. Other backends implement the interface+directly. Sampling and output mapping belong to the telemetry object.++A checkpointer requires `telemetry.ids`. Without them, construction throws an+error that names `AgentSpanIdGenerator`.++## Operations++```text+turn ─┬─ attempt ─┬─ modelCall+ │ └─ tool ─┬─ approval+ │ └─ tool (a call made while the tool runs)+ └─ memory (also under attempts and tools)+```++Each tool call is one `execute_tool` span. A host often learns about a call+twice: the runtime records its dispatch, and the SDK reports its run. In a+durable turn, calling `attempt.tool()` again with the same `callId` returns the+same call and fills in what the first report lacked, such as `kind`,+`arguments`, or an earlier start time. Tools with `kind: "subagent-call"` or+`"remote-agent-call"` delegate to another agent.++### Wrapped and handle forms++Each operation has two forms. The wrapped form runs a callback inside the+operation, returns its value or rethrows its error, and completes the operation+either way. An optional `describe(value)` maps the value to a completion, such+as an outcome or captured output:++```ts+await tracing.turn({ agentName: "support", identity, sequence: 0 }, (turn) =>+ turn.attempt({ stepIndex: 0, attempt: 0 }, (attempt) =>+ attempt.tool({ callId: "lookup", name: "lookup" }, lookup),+ ),+);+```++The handle form omits the callback and returns the operation. Hosts whose+start and end arrive in different hooks, or different processes, hold the+handle and complete it themselves:++```ts+const turn = await tracing.turn({ agentName: "support", identity, sequence: 0 });+const attempt = await turn.attempt({ stepIndex: 0, attempt: 0 });+const tool = await attempt.tool({ callId: "lookup", name: "lookup" });+try {+ const value = await tool.run(lookup);+ await tool.complete({ outcome: "completed", output: value });+} catch (error) {+ await tool.fail(error);+}+await attempt.complete();+await turn.complete();+```++### Handles++Every handle shares one interface:++```ts+interface Operation {+ readonly type: "activation" | "step" | "model" | "tool" | "approval" | "memory";+ readonly reference: TraceReference; // trace and span ID, stable once started+ readonly parent?: TraceReference;+ readonly startTimeMs: number;+ readonly finished: boolean;+ readonly capture: CaptureDecision;+ run<T>(execute: () => T, ceiling?: CaptureDecision): T;+ attributes(attributes: Attributes): void;+ complete(result?: Completion): Promise<void>;+ fail(error: unknown): Promise<void>;+ recordError(error: unknown, errorType?: string): void;+}+```++- `run(execute, ceiling)` makes the operation active while `execute` runs, so+ spans that other libraries start inside it become its children. The ceiling+ limits captured content for work inside the run; it does not remove what the+ operation already recorded, and it bounds content any operation receives+ while the run is active.+- `complete(result)` ends the operation and is idempotent. The completion can+ carry `outcome`, `failed`, `error` and `errorType`, `output`, `usage`, a model+ `result`, and `endTimeMs`. `outcomeUnknown: true` ends a turn without an+ outcome when the host never learned one; a failure is still recorded.+- `fail(error)` completes the operation as failed. `recordError(error)` marks+ it failed without ending it, so a completion reported later, possibly by+ another process, stays failed.+- Completing a parent closes its unfinished children, except as described in+ [Lifetimes](#lifetimes).++Handles add the operations that may start beneath them:++| Handle | Starts | Finds |+| ----------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------- |+| turn | `attempt`, `memory` | `findAttempt({ stepIndex, attempt })`, `findTool(callId)` |+| attempt | `modelCall`, `modelStream`, `tool`, `callSubAgent`, `callRemoteAgent`, `memory` | |+| tool | `approval`, nested `tool`, `memory` | `findApproval(requestId)` |+| model, approval, memory | nothing | |++`turn.links()` replaces the turn's links before its deferred span is exported.+`modelStream` returns the stream unchanged and completes when its `completion`+promise settles, not when the callback returns; parent completion waits for it. Find operations return `undefined` once the operation has+finished, and are available only in durable turns.++### Turn input++`TurnInput` carries `identity` (`conversationId`, `runId`, `turnId`),+`agentName`, `sequence`, framework metadata, a capture decision, and+attributes. Durable hosts can also supply:++- `reference`: a trace and span ID reserved before the turn started, such as a+ seed already returned to a caller. The sampler still decides the trace flags.+- `lineage` and `channel`: delegation and delivery metadata the host stores+ itself, when no in-process handoff carries it.+- `parent`: the caller's span, for a turn that nests in its caller's trace.+- `links` and `startTimeMs`. `turn.links()` replaces links that the host learns+ after the turn starts.+- `context`: host execution context, such as OTel baggage, for spans this+ process starts.++`agentName` belongs to the turn, so one instance can trace several agents.++## Durable turns++```ts+interface TraceCheckpointer {+ get(key: string): unknown | PromiseLike<unknown>;+ set(key: string, value: TraceSnapshot): void | PromiseLike<void>;+ delete(key: string): void | PromiseLike<void>;+}+```++- **Keys.** The library chooses keys. A turn is stored at `turn:<runId>:<turnId>`.+ Inside the tree, attempts are keyed by step index and attempt, tool calls by+ `callId`, and approvals by `requestId`. Model calls and memory are numbered+ and get new span IDs when they run again, because a re-run is new work.+- **Writes.** The library writes JSON after each change, in order. It deletes+ the entry when the turn and all of its open work have completed. Each node is+ capped at 64 KiB, and finished children leave the tree.+- **Resume.** Calling `turn()` with the same identity continues the saved turn.+ `tracing.resume({ identity })` returns a saved turn without starting one.+ `turn.findAttempt()`, `turn.findTool(callId)`, and `tool.findApproval()`+ return open operations, or `undefined`.+- **Deferred spans.** Durable turns, tool calls, and approvals reserve their IDs+ at start and export at completion, so a later process can complete them.+ Attempts and model calls are live spans in the process that runs them.+- **Process cache.** Concurrent hooks in one process share one tree. Each write+ stores a revision token; when another process changed the checkpoint, the+ next call reloads it.+- **Failures.** An unreadable checkpoint is reported to `onError` with phase+ `restore`, and a fresh turn starts. Write failures use phase `checkpoint`.++### Lifetimes++| Event | Open children |+| ----------------------------------- | -------------------------------------------------------- |+| Any operation fails | Closed as failed |+| A non-durable operation completes | Closed as abandoned |+| A durable attempt or turn completes | Tool calls and approvals stay open; other children close |+| A tool call completes | Its nested calls and approvals close |++Tool calls outlive their attempt and turn because human approval parks a turn:+the approval resolves in a later turn, and its span still belongs to the+original call.++## AI SDK integration++`aiSdkTelemetry(tracing, { turn })` returns an AI SDK `Telemetry` integration+for `generateText`, `streamText`, and agents such as `HarnessAgent`. It keeps+open operations keyed by the SDK `callId` and `toolCallId`, so hosts do not+store handles between hooks. It runs model calls and tools inside their spans+through `executeLanguageModelCall` and `executeTool`. `turn` can be a function+that chooses the turn for each call.++## Capture and delegation++Capture defaults to metadata only. Children cannot increase capture, and+declined content never enters a checkpoint. Content is checked when it arrives:+an operation's own capture, narrowed by any run ceiling active at that moment,+decides whether arguments, results, and errors are kept. Re-entering a turn with+a narrower capture narrows it and its open operations and drops retained+content the new decision declines; content recorded under earlier consent is+kept. Failure classes survive redaction;+exception messages do not. Serialized content is capped at 32 KiB.++Local delegation carries lineage through async context. A local callee's first+turn nests under the delegating tool call in the caller's trace; later turns+start their own trace. Remote delegation uses a bounded `x-agent-tracing`+header; the receiver validates it and asks the host to verify the authenticated+caller before it adopts any lineage. A remote callee starts its own trace with+an `agent.dispatch` link to the call.++## eve as consumer++eve uses one `AgentTracing` instance over `otelTelemetry`. Its checkpointer+stores trees in the existing workflow-context trace state, so checkpoints commit+with each workflow step. eve keeps these concerns:++- session trace seeds, turn ID reservation, and trace policy;+- channel delivery metadata and principals;+- tool call locators and anchors that dispatched child agents read;+- schema-4 attribute names, such as `agent.action.kind` for the library's+ `agent.tool.kind`, through its output mapping;+- direct `invokeTool` spans, which have no turn;+- SDK registration, request and MCP spans, and local trace storage.++eve's hook bridge maps AI SDK hooks to eve lifecycle events, not directly to+spans, so eve does not use `aiSdkTelemetry`.++## Validation++- Library tests use only package entrypoints. They cover a real+ `HarnessAgent`, `generateText`, an in-memory agent, delegation, streams, and+ durable recovery across workers.+- eve's span and telemetry contract suites pass without changes to their+ assertions.+- Live Agent Runs ingestion and fixture e2e run in CI only.++## Open questions++- Completions are one untyped object for every operation. Per-operation types,+ such as `model.complete(result)` with no `outcome`, would remove ambiguous+ fields.+- `lineage` and the in-process handoff describe delegation in two ways.+- Durable and non-durable modes close open tool calls differently.+- A turn cannot record its outcome without ending, so eve keeps the outcome+ until the session transition.+- Each checkpoint write serializes the open tree. Write volume needs watching+ on long tool-heavy turns.diff --git a/research/provider-neutral-local-observability.md b/research/provider-neutral-local-observability.mddeleted file mode 100644index f98376b3f..000000000--- a/research/provider-neutral-local-observability.md+++ /dev/null@@ -1,195 +0,0 @@-----issue: TBD-status: in-progress-last_updated: "2026-08-28"------# Provider-neutral local observability--## Summary--eve produces local traces without depending on Workflow tracing, changing production-instrumentation, or coupling observability providers to the AI SDK. The first consumer is-zero-config tracing in `eve dev`; the lifecycle contract must also support OTel, Vercel,-Braintrust, and custom providers.--```text-eve lifecycle boundaries ------------------+- +--> eve lifecycle hooks --> providers-AI SDK callbacks -> per-attempt bridge -----+ (local OTel, Vercel, Braintrust, custom)-AI SDK execute() -> eve-owned context runner-```--## Lifecycle contract--```ts-createAiSdkHookBridge(scope, hooks, runInContext): Telemetry;--interface InstrumentationHooks {- publish(event: InstrumentationPointEvent): Promise<void>;- before(name: "model.call" | "tool.call", event: StartEvent): Promise<void>;- after(name: "model.call" | "tool.call", event: TerminalEvent): Promise<void>;-}-```--The bridge is an adapter, not a provider. It maps AI SDK callbacks into eve-owned lifecycle-events, assigns stable attempt/model-call/tool-call identities, terminalizes open operations on-error and abort, and invokes the trusted `runInContext` while calling execution exactly once. AI-SDK `callId` values are bridge correlation details, not provider run identities.--The bridge does not create spans, export records, apply provider policy, or publish durable-eve-native events. Session, turn, compaction, suspension, and canonical step-terminal events are-published by the harness.--Attempt state is WeakMap-keyed by scope identity so abandoned attempts do not leak. `before` state-is retained per provider and operation and supplied only to that provider's `after`.--## Providers--A provider is a hook definition: handlers for the eve-owned events it cares about. An OTel provider-may return spans from `before` and close them in `after`; Braintrust may retain row handles; Vercel-may enqueue records.--Each provider owns an independent trace graph, rooted from its own serializable session state — not-from ambient Workflow context or another provider's active context.--Providers never receive an execution function. A built-in integration needing ambient context may-register a trusted internal `runInContext` alongside its event definition; that is not part of the-provider contract.--## Local tracing--In `eve dev` eve registers the process tracer provider (`registerOTel`) with a private span-processor consuming the same lifecycle events as any other provider, so nested AI SDK and user-spans share agent context. It creates one trace per session window, explicitly roots that window-rather than inheriting the ambient Workflow span, drops spans from the `workflow` instrumentation-scope, and persists OTLP/JSON segments under `.eve/traces/v1`.--Those immutable segments are the canonical local store. A future index or dashboard is a-rebuildable consumer of the spool, not a second source of trace identity.--Authored instrumentation and local capture may observe the same attempt without executing the model-or tool more than once.--## Agent OTel semantics--```text-agent.session {agent.session.id}- +-- invoke_agent {agent.name} {agent.turn.id, agent.turn.sequence}- +-- agent.step {agent.step.index, agent.step.attempt}- +-- model-provider spans- +-- agent.action {agent.action.kind, name, call_id}- +-- execute_tool {tool.name}- +-- tool-provider spans-```--The durable turn and tool-execution boundaries use the OpenTelemetry GenAI agent conventions so-backends can recognize agent invocations and tool calls without eve-specific mapping. An-`agent.step` covers one model call and the actions it requests through their resolution.-`agent.action.kind` remains an open discriminator for eve's broader durable dispatch boundary.--The `agent.*` namespace carries cheap structural attributes only: session/turn/step/attempt-identity, action identity, and agent and framework identity. Standard `gen_ai.*` attributes describe-the operation, agent, conversation, and tool. Message content, model output, and tool payloads follow-each provider's audience-aware `tracePolicy`. The OTel provider owns this mapping; the bridge and-hooks contain no span names, attributes, or parent contexts.--### The session root is a real span--`agent.session` is a real root span opened with OTel's `root` option, not a synthesized parent-context. eve persists its span context — including the sampling decision it actually received — and-later turns parent to that stored context.--This is what makes an authored sampler work. A synthesized parent asserting a sampled flag is a-valid parent to `ParentBasedSampler`, which resolves through a parent branch and never consults the-configured root rule. Asserting unsampled instead selects `localParentNotSampled`, which drops-everything. A genuine root avoids both.--Sampling is therefore per session trace. `agent.session.id` is a creation-time attribute, so an-author can key a sampler on it.--### Marker spans--The session root is recorded and ended immediately. A session outlives the-worker that opened it and a span object cannot cross that boundary, so eve records the root,-persists its span context, and parents later spans through it.--The cost is that a backend reports the root's duration rather than the window's. Turn and step-durations, where the time actually is, are unaffected, and `eve traces` renders a marker's-descendant extent in place of its zero duration.--A real duration would mean emitting the span once at session close, with explicit timestamps and a-span id chosen before the span exists. The ids are reachable — `registerOTel` accepts an-`idGenerator` — but the close is not: a session that goes idle and never resumes closes on nothing,-so its root would never be emitted and every span in the trace would reference a parent that never-arrives. A marker is the only representation that is always emitted. A turn does have a guaranteed-close, so its `invoke_agent` span carries its real duration.--### Subagents--A local subagent keeps its own session identity but adopts the parent action's `SpanContext`, so-delegated work appears directly beneath the action that caused it. Remote dispatch sends the same-context as `traceparent`; the receiver adopts it when available. A child with no propagated context-opens its own root. `eve traces` resolves any session recorded in a trace, not only the one that-opened it.--## Runtime integration--```text-runModelCallWithRetries(attempt)- -> create stable AttemptScope- -> createAiSdkHookBridge(scope, hooks, runInContext)- -> telemetry.integrations = [bridge, authoredOtel?]-```--The harness supplies session id, turn id, logical step index, and retry attempt index. The bridge-derives model/tool identities at their start callbacks and keeps callback snapshots in-invocation-local state; `onStepEnd` and `onEnd` only snapshot, and the harness publishes the-canonical terminal after durable actions are emitted. On replay the harness recreates attempt state;-only provider-owned serializable context crosses Workflow boundaries.--AI SDK `onStepStart` is named `attempt.started` because it begins one concrete model attempt, so-retries have distinct starts and terminals without overloading the stream's `step.started`.--Per-call integrations replace the AI SDK global list; eve composes the bridge with `@ai-sdk/otel`-when the documented `instrumentation.ts` path is active. Direct `registerTelemetry()` calls are not-a supported customization surface. Braintrust's documented integration uses eve hooks and-`defineInstrumentation`, so per-call injection does not disable it.--## Compatibility--Production keeps its current `defineInstrumentation` and `@ai-sdk/otel` behavior until provider-migration ships. Without lifecycle hooks no bridge is installed and production follows the existing-authored path unchanged. Stream hooks and `step.started` semantics are unchanged.--## Delivery phases--1. **Lifecycle bridge.** — _landed._ Attempt scope, WeakMap state, typed hooks, before/after- correlation, trusted context runner, AI SDK callback adapter.-2. **Per-attempt wiring.** — _landed._ One bridge per retry attempt through- `telemetry.integrations`, composing authored OTel when active.-3. **Local OTel provider.** — _landed._ Lifecycle events mapped to the `agent.*` convention.-4. **Persistence.** — _landed._ OTLP/JSON segments, session context restored across dev worker- restarts, retention bounds, capture policy.-5. **Inspection.** — _landed._ `eve traces ls` and `eve traces [trace]`.-6. **Session windows.** — _in review._ Real `agent.session` root with its recorded sampling- decision, window rolling, subagent adoption, session-to-windows resolution in `eve traces`.-7. **Public providers.** — _not started._ Promote the lifecycle contract into public hooks and- migrate OTel, Vercel, Braintrust, and custom instrumentation onto it.--## Acceptance criteria--- The lifecycle layer imports no OTel types. It does derive AI SDK callback payload shapes from- `Telemetry`, so `attempt`, `model.call`, and `tool.call` events currently expose AI SDK types to- providers — a known deviation from provider neutrality, to close in phase 7.-- Multiple providers observe one attempt while execution occurs exactly once.-- No provider definition receives model or tool execution functions.-- Documented authored instrumentation continues to observe eve calls.-- Each provider builds its trace without inheriting Workflow or another provider's trace.-- No eve span inherits a synthesized parent, so an authored sampler's root rule is consulted.-- A session of ordinary length is exactly one trace; a session of any length has bounded traces.-- `eve traces <session>` resolves every window of a session.-- Parallel tool calls retain independent provider state.-- Errors and aborts terminalize all started model and tool operations.-- Local traces contain no Workflow spans.-- Enabling local tracing does not change production instrumentation behavior.