Building AI agents, atomically
Document Phoenix and OpenTelemetry tracing
Eigenwise4801bcfcd92e
diff --git a/docs/guides/index.md b/docs/guides/index.mdindex ae40e05..e1856e6 100644--- a/docs/guides/index.md+++ b/docs/guides/index.md@@ -10,6 +10,7 @@ quickstartmemorytoolshooks+observabilityorchestrationcookbookerror-handlingdiff --git a/docs/guides/logging.md b/docs/guides/logging.mdindex 53cd635..bdf58d8 100644--- a/docs/guides/logging.md+++ b/docs/guides/logging.md@@ -386,7 +386,9 @@ async def prometheus_metrics():### OpenTelemetry Integration-Add distributed tracing for complex systems:+For agent and tool spans that render in Arize Phoenix and other OpenInference-aware backends, use the built-in+instrumentor described in the [Observability guide](observability.md). The snippet below shows the generic+OpenTelemetry setup for everything else:```pythonfrom opentelemetry import tracediff --git a/docs/guides/observability.md b/docs/guides/observability.mdnew file mode 100644index 0000000..7ba76d3--- /dev/null+++ b/docs/guides/observability.md@@ -0,0 +1,150 @@+# Observability with Arize Phoenix and OpenTelemetry++Atomic Agents ships an optional [OpenInference](https://github.com/Arize-ai/openinference) instrumentor. Once it+is installed, every `AtomicAgent` run and every `BaseTool` run becomes a span that+[Arize Phoenix](https://github.com/Arize-ai/phoenix), Arize AX, Langfuse, or any other OpenTelemetry backend can+display. Nothing changes for applications that do not opt in.++## What you get++| Call | Span | Attributes |+|------|------|------------|+| `agent.run`, `run_async`, `run_stream`, `run_async_stream` | `AGENT`, named `AtomicAgent[InputSchema, OutputSchema].run` (a subclass reports its own name) | `agent.name`, `input.value` / `output.value` as JSON, `atomic_agents.model`, `atomic_agents.mode`, `atomic_agents.input_schema`, `atomic_agents.output_schema`, `atomic_agents.history.message_count` (history size entering the turn), `atomic_agents.context_providers`, `atomic_agents.system_prompt.sha256` (hash of the static prompt sections: background, steps, output instructions) |+| `tool.run`, `run_async`, `arun` on any `BaseTool` subclass, including MCP tools | `TOOL`, named after the tool class | `tool.name`, `tool.description`, `tool.parameters` (the input JSON schema), `input.value` / `output.value` as JSON |++Spans carry `session.id`, `user.id`, `metadata` and tags when you wrap calls in the OpenInference context+managers (`using_session`, `using_user`, `using_metadata`, `using_attributes`). Exceptions set the span status to+`ERROR` and are recorded as span events. Streaming spans record the final response when the stream completes,+or the last partial plus `atomic_agents.stream.closed_early = true` when the consumer stops early or the task+is cancelled (a cancelled stream is not reported as an error).++LLM spans are not produced by Atomic Agents. Install exactly one LLM instrumentor for the provider SDK you use+(`openinference-instrumentation-openai`, `-anthropic`, `-litellm`, ...). Installing both a provider instrumentor and+`openinference-instrumentation-instructor` gives you two layers of LLM spans. Because the agent span is the current+span while the provider call runs, the LLM span (with prompts, tokens and the model name) nests under it. Instructor+retries show up as several LLM spans under one agent span.++## Install++```bash+pip install "atomic-agents[observability]" arize-phoenix-otel openinference-instrumentation-openai+```++The `observability` extra installs `openinference-instrumentation`, `openinference-semantic-conventions` and+`opentelemetry-api`. The Phoenix path below works without the extra because `arize-phoenix-otel` already depends on+those packages; the manual path needs the extra plus an OpenTelemetry SDK and exporter:++```bash+pip install "atomic-agents[observability]" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http+```++## Setup with Phoenix (automatic)++```python+from phoenix.otel import register++tracer_provider = register(project_name="my-app", auto_instrument=True)+```++`register(auto_instrument=True)` discovers the Atomic Agents instrumentor through the `openinference_instrumentor`+entry point and activates it together with every other OpenInference instrumentor in the environment. Do this once,+at import time or in your service's startup hook, before any agent runs. Phoenix reads `PHOENIX_COLLECTOR_ENDPOINT`,+`PHOENIX_PROJECT` (alias `PHOENIX_PROJECT_NAME`) and `PHOENIX_API_KEY` from the environment. With no endpoint set,+`register()` exports over gRPC to `localhost:4317`; set `PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006` to send+HTTP to the port the Phoenix UI serves on.++Two things to know about the automatic path:++- Phoenix imports `atomic_agents` to load the entry point. That import is the same as any `import atomic_agents` in+ your application, including the Instructor streaming patch the framework applies at import.+- Existing `auto_instrument=True` users start exporting agent and tool payloads when they upgrade to a version with the+ instrumentor. Use the privacy controls below if that is not wanted.++## Setup with any OpenTelemetry backend (manual)++```python+from opentelemetry.sdk.trace import TracerProvider+from opentelemetry.sdk.trace.export import BatchSpanProcessor+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter+from openinference.instrumentation import TraceConfig+from openinference.instrumentation.openai import OpenAIInstrumentor++from atomic_agents.observability import AtomicAgentsInstrumentor++tracer_provider = TracerProvider()+tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter("http://localhost:6006/v1/traces")))++AtomicAgentsInstrumentor().instrument(tracer_provider=tracer_provider, config=TraceConfig())+OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)+```++`instrument()` is idempotent. Calling it again with a different provider or config logs a warning and keeps the first+installation; call `uninstrument()` first to change it. `uninstrument()` restores the original methods on+`AtomicAgent` and on every tool class.++## Grouping agent and tool calls into one trace++Atomic Agents has no tool-calling loop: your code calls `agent.run(...)`, then `tool.run(...)`, then another agent.+Each of those is a root span, so without a parent they become separate traces. Open one parent span per unit of work:++```python+tracer = tracer_provider.get_tracer(__name__)++with tracer.start_as_current_span("answer_question", openinference_span_kind="chain") as turn:+ turn.set_input(question)+ plan = planner.run(BasicChatInputSchema(chat_message=question))+ calculation = calculator.run(CalculatorToolInputSchema(expression=plan.expression))+ answer = answerer.run(CalculationResultSchema(question=question, expression=plan.expression, result=calculation.result))+ turn.set_output(answer.chat_message)+```++The [phoenix-tracing example](https://github.com/eigenwise/atomic-agents/tree/main/atomic-examples/phoenix-tracing)+is exactly this flow.++## Sessions, users and metadata++```python+from openinference.instrumentation import using_session, using_user, using_metadata++with using_session(session_id=conversation_id), using_user(user_id=user.id), using_metadata({"tenant": tenant}):+ response = agent.run(user_message)+```++Phoenix groups spans that share a `session.id` into a conversation view. For a streaming endpoint, put the context+manager inside the generator that yields the chunks, not around the endpoint function; the generator runs after the+endpoint returned, so a context manager around the endpoint body no longer applies.++## Privacy++Agent and tool payloads are exported as `input.value` / `output.value`. To redact them:++- On the automatic Phoenix path, set `OPENINFERENCE_HIDE_INPUTS=true` and/or `OPENINFERENCE_HIDE_OUTPUTS=true`.+ `TraceConfig()` reads these variables.+- On the manual path, pass `TraceConfig(hide_inputs=True, hide_outputs=True)` to `instrument()`.++Redacted values are exported as `__REDACTED__`. `atomic_agents.system_prompt.sha256` stays available either way. It+hashes the static sections of a `SystemPromptGenerator` (background, steps, output instructions) and ignores what+context providers add per call, so it identifies the prompt version even when inputs are hidden. Multimodal inputs (images, audio, PDFs) are serialized into+`input.value` in full; cap attribute size with `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` if that is too much.++## Services and async++- Call `register(...)` (or `instrument(...)`) at import or in the startup hook so every worker process is instrumented.+- Use `register(batch=True)` (or a `BatchSpanProcessor`) in services so span export does not block request handling.+- Concurrent `run_async` calls, including `asyncio.gather` over several agents, get independent spans.+- Streaming token counts need `model_api_parameters={"stream_options": {"include_usage": True}}` for OpenAI.++## Limitations++- Methods bound before instrumentation stay untraced: `callback = agent.run` taken before `instrument()` keeps+ calling the original.+- An `AtomicAgent` subclass that overrides `run` (or a stream method) without calling `super()` bypasses the wrapper+ and emits no span; add `@tracer.agent` to such overrides yourself.+- Tool methods that are async generators are left untraced.+- A plain `ThreadPoolExecutor` starts its worker with an empty context, so a tool that submits work to one loses the+ current span and the re-entrancy guard inside the pool. Submit through a copied context instead, as the forge tools+ do: `executor.submit(contextvars.copy_context().run, asyncio.run, self.run_async(params))`. `asyncio.to_thread` and+ `asyncio.run` already copy the context.+- The AGENT span records the final (or last partial) response of a stream, not every chunk.+- The system prompt text is not on the AGENT span; it is the first message on the LLM span. The AGENT span carries its+ SHA-256 instead.