Skip to content

Building AI agents, atomically

Document Phoenix and OpenTelemetry tracing

Eigenwise4801bcfcd92e
diff --git a/docs/guides/index.md b/docs/guides/index.md
index ae40e05..e1856e6 100644
--- a/docs/guides/index.md
+++ b/docs/guides/index.md
@@ -10,6 +10,7 @@ quickstart
memory
tools
hooks
+observability
orchestration
cookbook
error-handling
diff --git a/docs/guides/logging.md b/docs/guides/logging.md
index 53cd635..bdf58d8 100644
--- a/docs/guides/logging.md
+++ b/docs/guides/logging.md
@@ -386,7 +386,9 @@ async def prometheus_metrics():
### OpenTelemetry Integration
-Add distributed tracing for complex systems:
+For agent and tool spans that render in Arize Phoenix and other OpenInference-aware backends, use the built-in
+instrumentor described in the [Observability guide](observability.md). The snippet below shows the generic
+OpenTelemetry setup for everything else:
```python
from opentelemetry import trace
diff --git a/docs/guides/observability.md b/docs/guides/observability.md
new file mode 100644
index 0000000..7ba76d3
--- /dev/null
+++ b/docs/guides/observability.md
@@ -0,0 +1,150 @@
+# Observability with Arize Phoenix and OpenTelemetry
+
+Atomic Agents ships an optional [OpenInference](https://github.com/Arize-ai/openinference) instrumentor. Once it
+is installed, every `AtomicAgent` run and every `BaseTool` run becomes a span that
+[Arize Phoenix](https://github.com/Arize-ai/phoenix), Arize AX, Langfuse, or any other OpenTelemetry backend can
+display. Nothing changes for applications that do not opt in.
+
+## What you get
+
+| Call | Span | Attributes |
+|------|------|------------|
+| `agent.run`, `run_async`, `run_stream`, `run_async_stream` | `AGENT`, named `AtomicAgent[InputSchema, OutputSchema].run` (a subclass reports its own name) | `agent.name`, `input.value` / `output.value` as JSON, `atomic_agents.model`, `atomic_agents.mode`, `atomic_agents.input_schema`, `atomic_agents.output_schema`, `atomic_agents.history.message_count` (history size entering the turn), `atomic_agents.context_providers`, `atomic_agents.system_prompt.sha256` (hash of the static prompt sections: background, steps, output instructions) |
+| `tool.run`, `run_async`, `arun` on any `BaseTool` subclass, including MCP tools | `TOOL`, named after the tool class | `tool.name`, `tool.description`, `tool.parameters` (the input JSON schema), `input.value` / `output.value` as JSON |
+
+Spans carry `session.id`, `user.id`, `metadata` and tags when you wrap calls in the OpenInference context
+managers (`using_session`, `using_user`, `using_metadata`, `using_attributes`). Exceptions set the span status to
+`ERROR` and are recorded as span events. Streaming spans record the final response when the stream completes,
+or the last partial plus `atomic_agents.stream.closed_early = true` when the consumer stops early or the task
+is cancelled (a cancelled stream is not reported as an error).
+
+LLM spans are not produced by Atomic Agents. Install exactly one LLM instrumentor for the provider SDK you use
+(`openinference-instrumentation-openai`, `-anthropic`, `-litellm`, ...). Installing both a provider instrumentor and
+`openinference-instrumentation-instructor` gives you two layers of LLM spans. Because the agent span is the current
+span while the provider call runs, the LLM span (with prompts, tokens and the model name) nests under it. Instructor
+retries show up as several LLM spans under one agent span.
+
+## Install
+
+```bash
+pip install "atomic-agents[observability]" arize-phoenix-otel openinference-instrumentation-openai
+```
+
+The `observability` extra installs `openinference-instrumentation`, `openinference-semantic-conventions` and
+`opentelemetry-api`. The Phoenix path below works without the extra because `arize-phoenix-otel` already depends on
+those packages; the manual path needs the extra plus an OpenTelemetry SDK and exporter:
+
+```bash
+pip install "atomic-agents[observability]" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
+```
+
+## Setup with Phoenix (automatic)
+
+```python
+from phoenix.otel import register
+
+tracer_provider = register(project_name="my-app", auto_instrument=True)
+```
+
+`register(auto_instrument=True)` discovers the Atomic Agents instrumentor through the `openinference_instrumentor`
+entry point and activates it together with every other OpenInference instrumentor in the environment. Do this once,
+at import time or in your service's startup hook, before any agent runs. Phoenix reads `PHOENIX_COLLECTOR_ENDPOINT`,
+`PHOENIX_PROJECT` (alias `PHOENIX_PROJECT_NAME`) and `PHOENIX_API_KEY` from the environment. With no endpoint set,
+`register()` exports over gRPC to `localhost:4317`; set `PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006` to send
+HTTP to the port the Phoenix UI serves on.
+
+Two things to know about the automatic path:
+
+- Phoenix imports `atomic_agents` to load the entry point. That import is the same as any `import atomic_agents` in
+ your application, including the Instructor streaming patch the framework applies at import.
+- Existing `auto_instrument=True` users start exporting agent and tool payloads when they upgrade to a version with the
+ instrumentor. Use the privacy controls below if that is not wanted.
+
+## Setup with any OpenTelemetry backend (manual)
+
+```python
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from openinference.instrumentation import TraceConfig
+from openinference.instrumentation.openai import OpenAIInstrumentor
+
+from atomic_agents.observability import AtomicAgentsInstrumentor
+
+tracer_provider = TracerProvider()
+tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter("http://localhost:6006/v1/traces")))
+
+AtomicAgentsInstrumentor().instrument(tracer_provider=tracer_provider, config=TraceConfig())
+OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
+```
+
+`instrument()` is idempotent. Calling it again with a different provider or config logs a warning and keeps the first
+installation; call `uninstrument()` first to change it. `uninstrument()` restores the original methods on
+`AtomicAgent` and on every tool class.
+
+## Grouping agent and tool calls into one trace
+
+Atomic Agents has no tool-calling loop: your code calls `agent.run(...)`, then `tool.run(...)`, then another agent.
+Each of those is a root span, so without a parent they become separate traces. Open one parent span per unit of work:
+
+```python
+tracer = tracer_provider.get_tracer(__name__)
+
+with tracer.start_as_current_span("answer_question", openinference_span_kind="chain") as turn:
+ turn.set_input(question)
+ plan = planner.run(BasicChatInputSchema(chat_message=question))
+ calculation = calculator.run(CalculatorToolInputSchema(expression=plan.expression))
+ answer = answerer.run(CalculationResultSchema(question=question, expression=plan.expression, result=calculation.result))
+ turn.set_output(answer.chat_message)
+```
+
+The [phoenix-tracing example](https://github.com/eigenwise/atomic-agents/tree/main/atomic-examples/phoenix-tracing)
+is exactly this flow.
+
+## Sessions, users and metadata
+
+```python
+from openinference.instrumentation import using_session, using_user, using_metadata
+
+with using_session(session_id=conversation_id), using_user(user_id=user.id), using_metadata({"tenant": tenant}):
+ response = agent.run(user_message)
+```
+
+Phoenix groups spans that share a `session.id` into a conversation view. For a streaming endpoint, put the context
+manager inside the generator that yields the chunks, not around the endpoint function; the generator runs after the
+endpoint returned, so a context manager around the endpoint body no longer applies.
+
+## Privacy
+
+Agent and tool payloads are exported as `input.value` / `output.value`. To redact them:
+
+- On the automatic Phoenix path, set `OPENINFERENCE_HIDE_INPUTS=true` and/or `OPENINFERENCE_HIDE_OUTPUTS=true`.
+ `TraceConfig()` reads these variables.
+- On the manual path, pass `TraceConfig(hide_inputs=True, hide_outputs=True)` to `instrument()`.
+
+Redacted values are exported as `__REDACTED__`. `atomic_agents.system_prompt.sha256` stays available either way. It
+hashes the static sections of a `SystemPromptGenerator` (background, steps, output instructions) and ignores what
+context providers add per call, so it identifies the prompt version even when inputs are hidden. Multimodal inputs (images, audio, PDFs) are serialized into
+`input.value` in full; cap attribute size with `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` if that is too much.
+
+## Services and async
+
+- Call `register(...)` (or `instrument(...)`) at import or in the startup hook so every worker process is instrumented.
+- Use `register(batch=True)` (or a `BatchSpanProcessor`) in services so span export does not block request handling.
+- Concurrent `run_async` calls, including `asyncio.gather` over several agents, get independent spans.
+- Streaming token counts need `model_api_parameters={"stream_options": {"include_usage": True}}` for OpenAI.
+
+## Limitations
+
+- Methods bound before instrumentation stay untraced: `callback = agent.run` taken before `instrument()` keeps
+ calling the original.
+- An `AtomicAgent` subclass that overrides `run` (or a stream method) without calling `super()` bypasses the wrapper
+ and emits no span; add `@tracer.agent` to such overrides yourself.
+- Tool methods that are async generators are left untraced.
+- A plain `ThreadPoolExecutor` starts its worker with an empty context, so a tool that submits work to one loses the
+ current span and the re-entrancy guard inside the pool. Submit through a copied context instead, as the forge tools
+ do: `executor.submit(contextvars.copy_context().run, asyncio.run, self.run_async(params))`. `asyncio.to_thread` and
+ `asyncio.run` already copy the context.
+- The AGENT span records the final (or last partial) response of a stream, not every chunk.
+- The system prompt text is not on the AGENT span; it is the first message on the LLM span. The AGENT span carries its
+ SHA-256 instead.