The Open Framework for Building Agents
fix(eve)!: make instrumentation content opt-in (#2019)
Casey Gowrie3f92f7d9ca34
diff --git a/.changeset/instrumentation-content-opt-in.md b/.changeset/instrumentation-content-opt-in.mdnew file mode 100644index 000000000..f3f6ceb22--- /dev/null+++ b/.changeset/instrumentation-content-opt-in.md@@ -0,0 +1,5 @@+---+"eve": minor+---++Instrumentation now records trace metadata without model or tool inputs and outputs by default. Set `recordInputs` or `recordOutputs` to `true`, or use `EVE_TRACES_CONTENT=on` for the automatic local trace spool, to opt into content capture.diff --git a/docs/guides/instrumentation.md b/docs/guides/instrumentation.mdindex 20cdfd8ba..a14707fa0 100644--- a/docs/guides/instrumentation.md+++ b/docs/guides/instrumentation.md@@ -48,11 +48,11 @@ Any OTel-compatible backend works (Braintrust, PostHog, Raindrop, Arize, HoneycoThree more fields control what the AI SDK records inside those spans (see the AI SDK's [telemetry reference](https://ai-sdk.dev/docs/ai-sdk-core/telemetry)):-- `recordInputs` records full message history on each step span (defaults to `true`). Set it to `false` if inputs contain sensitive content or you want to reduce span payload size.-- `recordOutputs` records model outputs on spans (defaults to `true`). Set it to `false` to disable output recording.+- `recordInputs` records full message history on each step span. It defaults to `false`; set it to `true` to include input content.+- `recordOutputs` records model outputs on spans. It defaults to `false`; set it to `true` to include output content.- `functionId` overrides the function name on spans (defaults to the agent name).-For sensitive, regulated, or production data, set `recordInputs` and `recordOutputs` to `false` unless you have reviewed the exporter and its data-retention path.+eve records metadata without model or tool inputs and outputs by default. Enable either content category only after reviewing the exporter and its data-retention path.You are responsible for ensuring any observability or eval provider is approved for the data exported to it.@@ -173,15 +173,15 @@ Tag writes are best-effort: a failure is logged once per process and then swalloThese tags power the **Agent Runs** tab in the Vercel dashboard. When you deploy on Vercel, the platform auto-detects `eve` as the framework and surfaces an Agent Runs view under your project's **Observability** tab, where you can browse sessions and drill into each conversation's trace, with no `instrumentation.ts` required. The tab is currently gated per team. See [Deploy to Vercel](./deployment/vercel#inspect-agent-runs) for enablement. Agent Runs is separate from the OpenTelemetry export above. Use OTel when you want spans in Braintrust, PostHog, Datadog, or another third-party backend.-Note: By default, telemetry records full message history and model outputs You may need to disclose these data flows in your privacy materials if utilized.-## Local tracesWithout an `instrumentation.ts`, `eve dev` records spans to disk — one trace per session, with turns, model steps, and tool calls. Read them two ways:-- [`/traces`](dev-tui#logs-and-traces) in the dev TUI: a live viewer that replays the trace as a conversation.+- [`/traces`](dev-tui#logs-and-traces) in the dev TUI: a live trace viewer that replays captured content as a conversation.- [`eve traces`](../reference/cli#eve-traces): a span tree in the terminal, `eve traces ls` to list. Works after `eve dev` exits.+Local traces omit model and tool inputs and outputs by default. Set `EVE_TRACES_CONTENT=on` in `.env.local` to capture that content.+Writing `instrumentation.ts` replaces this: your `setup` takes over and nothing is recorded locally. For span attributes, retention, and the `EVE_TRACES*` variables, see [`eve traces`](../reference/cli#eve-traces).## Debuggingdiff --git a/docs/reference/cli.md b/docs/reference/cli.mdindex a355515a8..9a7a3ce1e 100644--- a/docs/reference/cli.md+++ b/docs/reference/cli.md@@ -300,7 +300,7 @@ One parent turn can dispatch several subagents into the same window, so a child'Every span carries a real duration except `agent.session`: an idle session never closes, so it is recorded as a zero-duration marker and the span tree shows its descendant extent instead. A turn's span is written when the turn settles, so a running turn shows only its steps.-Model and tool-call spans carry their inputs and outputs — system prompt, prompt messages, and response text for models; call arguments and results for tools — each capped at 32 KB. Set `EVE_TRACES_CONTENT=off` to keep payloads out of the spool.+Model and tool-call spans omit their inputs and outputs by default. Set `EVE_TRACES_CONTENT=on` to capture system prompts, prompt messages, and response text for models, plus call arguments and results for tools. Each captured value is capped at 32 KB.Step spans carry token counts, and cost when Vercel AI Gateway served the call. Both follow the [OTel GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) (`gen_ai.usage.*`), so a third-party backend reads them without mapping.@@ -308,13 +308,13 @@ Step spans carry token counts, and cost when Vercel AI Gateway served the call.eve sweeps the store when a session finishes and when the dev server starts, evicting oldest-first past the bounds below — except that the newest traces and anything written in the last five minutes are always kept, so a sweep will exceed the size budget rather than drop a trace you just recorded. Set the bounds in `.env.local`, which `eve dev` loads automatically; each accepts `off` to disable it individually.-| Variable | Default | Effect |-| ---------------------------- | -------------------- | ------------------------------------------------------------------------------------------- |-| `EVE_TRACES` | on | `off` stops writing traces and stops sweeping |-| `EVE_TRACES_CONTENT` | on | `off` stops capturing model prompt/response and tool input/output attributes on local spans |-| `EVE_TRACES_MAX_AGE_MS` | `604800000` (7d) | Age after which a trace may be evicted |-| `EVE_TRACES_MAX_TOTAL_BYTES` | `536870912` (512 MB) | Size budget for the whole store |-| `EVE_TRACES_RETAIN_COUNT` | `20` | Newest traces kept regardless of age or size |+| Variable | Default | Effect |+| ---------------------------- | -------------------- | ----------------------------------------------------------------------------------- |+| `EVE_TRACES` | on | `off` stops writing traces and stops sweeping |+| `EVE_TRACES_CONTENT` | off | `on` captures model prompt/response and tool input/output attributes on local spans |+| `EVE_TRACES_MAX_AGE_MS` | `604800000` (7d) | Age after which a trace may be evicted |+| `EVE_TRACES_MAX_TOTAL_BYTES` | `536870912` (512 MB) | Size budget for the whole store |+| `EVE_TRACES_RETAIN_COUNT` | `20` | Newest traces kept regardless of age or size |## `eve link`diff --git a/packages/eve/src/harness/instrumentation/config.test.ts b/packages/eve/src/harness/instrumentation/config.test.tsindex 8811f4e24..f8afd599e 100644--- a/packages/eve/src/harness/instrumentation/config.test.ts+++ b/packages/eve/src/harness/instrumentation/config.test.ts@@ -91,6 +91,20 @@ describe("instrumentation-config chunk-isolation regression", () => {});});+ it("disables input and output recording by default", async () => {+ const { registerInstrumentationConfig } = await import("#harness/instrumentation/config.js");+ const { getInstrumentationRuntime } = await import("#harness/instrumentation/runtime.js");++ await registerInstrumentationConfig({}, { agentName: "test-agent" });++ expect(getInstrumentationRuntime()?.otelSettings).toEqual({+ functionId: undefined,+ recordInputs: false,+ recordOutputs: false,+ traceChannelRequests: false,+ });+ });+it("awaits the setup callback with the resolved context", async () => {vi.resetModules();const { registerInstrumentationConfig } = await import("#harness/instrumentation/config.js");diff --git a/packages/eve/src/harness/instrumentation/config.ts b/packages/eve/src/harness/instrumentation/config.tsindex 4aaba953f..c8bf7d8d8 100644--- a/packages/eve/src/harness/instrumentation/config.ts+++ b/packages/eve/src/harness/instrumentation/config.ts@@ -51,8 +51,8 @@ export async function registerInstrumentationConfig(hooks: createInstrumentationHooks([]),otelSettings: {functionId: config.functionId,- recordInputs: config.recordInputs,- recordOutputs: config.recordOutputs,+ recordInputs: config.recordInputs === true,+ recordOutputs: config.recordOutputs === true,traceChannelRequests: config.traceChannelRequests === true,},runInContext: (_operation, execute) => execute(),diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.tsindex 3223568cc..eb944811b 100644--- a/packages/eve/src/harness/tool-loop.test.ts+++ b/packages/eve/src/harness/tool-loop.test.ts@@ -10397,8 +10397,8 @@ describe("createToolLoopHarness", () => {};expect(agentCall.telemetry).toMatchObject({integrations: [bridge],- recordInputs: true,- recordOutputs: true,+ recordInputs: false,+ recordOutputs: false,});expect(attemptCompleted).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.tsindex 5a2f7f4ce..711d453c9 100644--- a/packages/eve/src/harness/tool-loop.ts+++ b/packages/eve/src/harness/tool-loop.ts@@ -324,8 +324,8 @@ function enrichTelemetry(? undefined: [bridgeIntegration, ...getRegisteredTelemetryIntegrations()],isEnabled: true,- recordInputs: settings?.recordInputs ?? true,- recordOutputs: settings?.recordOutputs ?? true,+ recordInputs: settings?.recordInputs ?? false,+ recordOutputs: settings?.recordOutputs ?? false,};}diff --git a/packages/eve/src/public/instrumentation/index.ts b/packages/eve/src/public/instrumentation/index.tsindex 85be1b8da..44986d4b3 100644--- a/packages/eve/src/public/instrumentation/index.ts+++ b/packages/eve/src/public/instrumentation/index.ts@@ -149,13 +149,14 @@ export interface InstrumentationDefinition {readonly events?: InstrumentationEvents;/*** Whether to record full model inputs in telemetry spans. Defaults to- * `true` when `instrumentation.ts` is present. Set `false` for sensitive- * inputs or to reduce span payload size.+ * `false`. Set `true` only when the destination is approved to receive+ * input content.*/readonly recordInputs?: boolean;/*** Whether to record full model outputs in telemetry spans. Defaults to- * `true` when `instrumentation.ts` is present.+ * `false`. Set `true` only when the destination is approved to receive+ * output content.*/readonly recordOutputs?: boolean;/**diff --git a/packages/eve/src/public/instrumentation/otel.ts b/packages/eve/src/public/instrumentation/otel.tsindex f5beb31ed..0a988bec7 100644--- a/packages/eve/src/public/instrumentation/otel.ts+++ b/packages/eve/src/public/instrumentation/otel.ts@@ -34,8 +34,8 @@ export type { SpanExporter, SpanProcessor } from "#compiled/@vercel/otel/index.j/*** Vercel Agent Runs, enabled by default in production.*- * Export it from `agent/instrumentation/agent-runs.ts` to narrow content, or- * export `disableInstrumentation()` from that file to turn it off.+ * Export it from `agent/instrumentation/agent-runs.ts` to configure content+ * capture, or export `disableInstrumentation()` from that file to turn it off.*/export function agentRuns(options: ContentOptions = {}): OtelIntegration {return agentRunsIntegration(options);@@ -48,8 +48,9 @@ export function agentRuns(options: ContentOptions = {}): OtelIntegration {* backend, or export `disableInstrumentation()` from that file to turn it off.* Omitting the file leaves eve's default in place.*- * `EVE_TRACES_CONTENT=off` narrows this destination and no other, so declining- * content locally leaves what a hosted backend receives alone.+ * `EVE_TRACES_CONTENT=on` opts the default local spool into content capture.+ * `off` overrides this destination and no other, so declining content locally+ * leaves what a hosted backend receives alone.*/export function localTraces(options: ContentOptions = {}): OtelIntegration {return otelIntegration({diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.tsindex 66ace27fe..618c32600 100644--- a/packages/eve/src/tracing/agent-otel-provider.test.ts+++ b/packages/eve/src/tracing/agent-otel-provider.test.ts@@ -67,6 +67,8 @@ function createRuntime(const agentOtel = createAgentOtelInstrumentation({frameworkVersion: "test",idGenerator,+ recordInputs: true,+ recordOutputs: true,stateStore,tracer,});@@ -1233,7 +1235,7 @@ describe("createAgentOtelInstrumentation", () => {expect(model.attributes["agent.input.messages.delta"]).toBeUndefined();});- it("captures nothing when content capture is off", async () => {+ it("captures no content by default", async () => {const exporter = new InMemorySpanExporter();const idGenerator = new AgentSpanIdGenerator();const provider = new BasicTracerProvider({@@ -1243,8 +1245,6 @@ describe("createAgentOtelInstrumentation", () => {const agentOtel = createAgentOtelInstrumentation({frameworkVersion: "test",idGenerator,- recordInputs: false,- recordOutputs: false,stateStore: new InMemoryAgentTraceStateStore(),tracer: provider.getTracer("eve.agent"),});diff --git a/packages/eve/src/tracing/agent-otel-provider.ts b/packages/eve/src/tracing/agent-otel-provider.tsindex 1177c58b2..e71da5a47 100644--- a/packages/eve/src/tracing/agent-otel-provider.ts+++ b/packages/eve/src/tracing/agent-otel-provider.ts@@ -85,8 +85,8 @@ export interface AgentOtelInstrumentation {export function createAgentOtelInstrumentation(input: AgentOtelInstrumentationInput,): AgentOtelInstrumentation {- const recordInputs = input.recordInputs ?? true;- const recordOutputs = input.recordOutputs ?? true;+ const recordInputs = input.recordInputs ?? false;+ const recordOutputs = input.recordOutputs ?? false;const executionContexts = new WeakMap<InstrumentationAttemptScope, Map<string, Context>>();const attemptScopes = new Map<string, InstrumentationAttemptScope>();// A serverless turn runs inside one `turnStep` "use step" invocation. Ifdiff --git a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts b/packages/eve/src/tracing/install-instrumentation-runtime.test.tsindex fc4b03384..84d2edd9b 100644--- a/packages/eve/src/tracing/install-instrumentation-runtime.test.ts+++ b/packages/eve/src/tracing/install-instrumentation-runtime.test.ts@@ -70,8 +70,8 @@ describe("installInstrumentationRuntime", () => {expect(forceFlush).toHaveBeenCalledOnce();expect(providerFlush).toHaveBeenCalledOnce();expect(runtime.otelSettings).toEqual({- recordInputs: true,- recordOutputs: true,+ recordInputs: false,+ recordOutputs: false,traceChannelRequests: false,});expect(shutdown).toHaveBeenCalledOnce();diff --git a/packages/eve/src/tracing/local-traces.test.ts b/packages/eve/src/tracing/local-traces.test.tsindex fb8e1b50a..ac7e42240 100644--- a/packages/eve/src/tracing/local-traces.test.ts+++ b/packages/eve/src/tracing/local-traces.test.ts@@ -1,6 +1,6 @@import { afterEach, describe, expect, it, vi } from "vitest";-import { createLocalTracesProcessor } from "#tracing/local-traces.js";+import { createLocalTracesProcessor, resolveLocalTracesContent } from "#tracing/local-traces.js";import { localTraces } from "#public/instrumentation/otel.js";vi.mock("#tracing/local-trace-span-processor.js", () => ({@@ -29,11 +29,11 @@ function agentSpan(sessionId: string, traceId: string): unknown {};}-describe("createLocalTracesProcessor", () => {- afterEach(() => {- vi.unstubAllEnvs();- });+afterEach(() => {+ vi.unstubAllEnvs();+});+describe("createLocalTracesProcessor", () => {it("reports whether the released session owned any traces", async () => {const spool = createLocalTracesProcessor({ appRoot: "/tmp/eve-local-traces-test" });spool.onStart(agentSpan("session-one", "a".repeat(32)), undefined);@@ -63,3 +63,37 @@ describe("createLocalTracesProcessor", () => {await expect(processor.shutdown()).resolves.toBeUndefined();});});++describe("resolveLocalTracesContent", () => {+ it("records no content by default", () => {+ expect(resolveLocalTracesContent()).toEqual({+ recordInputs: false,+ recordOutputs: false,+ });+ });++ it("records content when explicitly enabled", () => {+ expect(resolveLocalTracesContent({ recordInputs: true, recordOutputs: true })).toEqual({+ recordInputs: true,+ recordOutputs: true,+ });+ });++ it("uses EVE_TRACES_CONTENT=on as the zero-config opt-in", () => {+ vi.stubEnv("EVE_TRACES_CONTENT", "on");++ expect(resolveLocalTracesContent()).toEqual({+ recordInputs: true,+ recordOutputs: true,+ });+ });++ it("lets EVE_TRACES_CONTENT=off override explicit local capture", () => {+ vi.stubEnv("EVE_TRACES_CONTENT", "off");++ expect(resolveLocalTracesContent({ recordInputs: true, recordOutputs: true })).toEqual({+ recordInputs: false,+ recordOutputs: false,+ });+ });+});diff --git a/packages/eve/src/tracing/local-traces.ts b/packages/eve/src/tracing/local-traces.tsindex 83ca4591f..37c51d50f 100644--- a/packages/eve/src/tracing/local-traces.ts+++ b/packages/eve/src/tracing/local-traces.ts@@ -93,9 +93,9 @@ export function createLocalTracesProcessor(* The local spool's content policy: its options, intersected with* `EVE_TRACES_CONTENT`.*- * The variable used to be the process-wide switch. It now applies to this one- * destination, and only ever narrows — `off` still wins where it applies, but- * it no longer decides what a hosted backend beside it receives.+ * The variable applies to this destination only. `on` opts the zero-config+ * spool into content, while `off` overrides authored local options without+ * changing what a hosted backend beside it receives.** @internal*/@@ -105,10 +105,11 @@ export function resolveLocalTracesContent(readonly recordOutputs?: boolean;} = {},): { readonly recordInputs: boolean; readonly recordOutputs: boolean } {+ const environmentDefault = process.env.EVE_TRACES_CONTENT === "on";const enabled = process.env.EVE_TRACES_CONTENT !== "off";return {- recordInputs: enabled && options.recordInputs !== false,- recordOutputs: enabled && options.recordOutputs !== false,+ recordInputs: enabled && (options.recordInputs ?? environmentDefault),+ recordOutputs: enabled && (options.recordOutputs ?? environmentDefault),};}diff --git a/packages/eve/src/tracing/otel-declaration.test.ts b/packages/eve/src/tracing/otel-declaration.test.tsindex 09a8eed6d..b0267e878 100644--- a/packages/eve/src/tracing/otel-declaration.test.ts+++ b/packages/eve/src/tracing/otel-declaration.test.ts@@ -40,7 +40,11 @@ describe("otel", () => {describe("otelIntegration", () => {it("passes declared processors through untouched", () => {const first = processor();- const integration = otelIntegration({ spanProcessors: [first] });+ const integration = otelIntegration({+ recordInputs: true,+ recordOutputs: true,+ spanProcessors: [first],+ });expect(isOtelIntegration(integration)).toBe(true);expect(integration.spanProcessors).toStrictEqual([first]);@@ -49,6 +53,8 @@ describe("otelIntegration", () => {it("wraps an exporter in a batching processor, after any declared ones", () => {const first = processor();const integration = otelIntegration({+ recordInputs: true,+ recordOutputs: true,spanProcessors: [first],traceExporter: exporter(),});@@ -57,15 +63,19 @@ describe("otelIntegration", () => {expect(integration.spanProcessors[0]).toBe(first);});- it("records everything unless told otherwise", () => {- expect(otelIntegration().content).toStrictEqual({ recordInputs: true, recordOutputs: true });+ it("records no content unless explicitly enabled", () => {+ expect(otelIntegration().content).toStrictEqual({ recordInputs: false, recordOutputs: false });});// An author's own processor is part of this destination, and the point of// declining is that nothing under it sees what was said.it("puts a declined policy in front of every processor, an author's included", () => {const first = processor();- const integration = otelIntegration({ recordOutputs: false, spanProcessors: [first] });+ const integration = otelIntegration({+ recordInputs: true,+ recordOutputs: false,+ spanProcessors: [first],+ });expect(integration.content).toStrictEqual({ recordInputs: true, recordOutputs: false });expect(integration.spanProcessors[0]).not.toBe(first);@@ -73,19 +83,19 @@ describe("otelIntegration", () => {});describe("agentRunsIntegration", () => {- it("uses Vercel's automatic processor by default", () => {+ it("records no content by default", () => {const integration = agentRunsIntegration();- expect(integration.content).toStrictEqual({ recordInputs: true, recordOutputs: true });- expect(integration.spanProcessors).toStrictEqual(["auto"]);+ expect(integration.content).toStrictEqual({ recordInputs: false, recordOutputs: false });+ expect(integration.spanProcessors).toHaveLength(1);+ expect(integration.spanProcessors[0]).not.toBe("auto");});- it("wraps the request-context transport when its content policy is narrowed", () => {- const integration = agentRunsIntegration({ recordInputs: false });+ it("uses Vercel's automatic processor when all content is enabled", () => {+ const integration = agentRunsIntegration({ recordInputs: true, recordOutputs: true });- expect(integration.content).toStrictEqual({ recordInputs: false, recordOutputs: true });- expect(integration.spanProcessors).toHaveLength(1);- expect(integration.spanProcessors[0]).not.toBe("auto");+ expect(integration.content).toStrictEqual({ recordInputs: true, recordOutputs: true });+ expect(integration.spanProcessors).toStrictEqual(["auto"]);});});@@ -98,8 +108,12 @@ describe("collectOtelPipeline", () => {it("concatenates destinations in declaration order", () => {const [first, second, third] = [processor(), processor(), processor()];const collected = collectOtelPipeline([- otelIntegration({ spanProcessors: [first, second] }),- otelIntegration({ spanProcessors: [third] }),+ otelIntegration({+ recordInputs: true,+ recordOutputs: true,+ spanProcessors: [first, second],+ }),+ otelIntegration({ recordInputs: true, recordOutputs: true, spanProcessors: [third] }),otel(),]);@@ -113,8 +127,8 @@ describe("collectOtelPipeline", () => {expect(collected.declared).toBe(true);expect(collected.settings).toStrictEqual({functionId: undefined,- recordInputs: true,- recordOutputs: true,+ recordInputs: false,+ recordOutputs: false,traceChannelRequests: false,});});diff --git a/packages/eve/src/tracing/otel-declaration.ts b/packages/eve/src/tracing/otel-declaration.tsindex abfd25614..bb1bdb6fe 100644--- a/packages/eve/src/tracing/otel-declaration.ts+++ b/packages/eve/src/tracing/otel-declaration.ts@@ -61,9 +61,9 @@ export interface OtelOptions {* longer have to agree.*/export interface ContentOptions {- /** Record model prompts and tool call inputs. Defaults to `true`. */+ /** Record model prompts and tool call inputs. Defaults to `false`. */readonly recordInputs?: boolean;- /** Record model responses and tool call outputs. Defaults to `true`. */+ /** Record model responses and tool call outputs. Defaults to `false`. */readonly recordOutputs?: boolean;}@@ -119,8 +119,8 @@ export function otel(options: OtelOptions = {}): OtelDeclaration {*/export function otelIntegration(options: OtelIntegrationOptions = {}): OtelIntegration {const content: ResolvedContentOptions = {- recordInputs: options.recordInputs !== false,- recordOutputs: options.recordOutputs !== false,+ recordInputs: options.recordInputs === true,+ recordOutputs: options.recordOutputs === true,};const declared = options.spanProcessors ?? [];const spanProcessors =@@ -155,8 +155,8 @@ export function agentRunsIntegration(options: ContentOptions = {}): OtelIntegratexport function resolveContentOptions(options: ContentOptions): ResolvedContentOptions {return {- recordInputs: options.recordInputs !== false,- recordOutputs: options.recordOutputs !== false,+ recordInputs: options.recordInputs === true,+ recordOutputs: options.recordOutputs === true,};}