Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

feat(backend/copilot): send AutoPilot's tool calls and replies to PostHog as AI traces (SECRT-2979) (#15605)

Krzysztof Czerwinski6319f87b16f8
diff --git a/autogpt_platform/backend/backend/copilot/ai_spans.py b/autogpt_platform/backend/backend/copilot/ai_spans.py
new file mode 100644
index 0000000000..da14ae4ecb
--- /dev/null
+++ b/autogpt_platform/backend/backend/copilot/ai_spans.py
@@ -0,0 +1,341 @@
+"""AutoPilot's replies, tool calls and expert subtasks as PostHog LLM traces.
+
+OpenRouter Broadcast sends each model call of a reply as an ``$ai_generation``
+whose ``$ai_trace_id`` is the reply's trace (see ``llm_attribution``). The
+backend sends the rest of that trace itself (SECRT-2979):
+
+- ``$ai_trace``: one per reply, when its turn ends, with the turn's latency
+ and whether it failed.
+- ``$ai_span`` per tool call: the tool's name, latency and whether it failed,
+ under the turn that ran it (or under the tool that called it).
+- ``$ai_span`` per expert subtask: a turn another turn spawned joins the
+ trace of the reply that spawned it, under the tool call that started it,
+ and its own tool calls nest under it.
+
+No event carries a tool's input or output, or an error's message: those can
+hold Gmail or Drive content, which Google's Limited Use terms keep out of
+analytics. ``$ai_input_state`` and ``$ai_output_state`` are left out for
+that reason, and an error is named by its code or exception type only.
+
+Like the generations and ``$ai_metric``, these are sent server-side for every
+user, whatever their cookie consent, on the user's distinct id (the platform
+user id) and with ``impersonated`` set when an admin acting as the user ran
+the turn. ``posthog_client.capture`` queues each event for the client's
+background thread and swallows any failure, so a turn never waits on
+PostHog or fails because of it.
+"""
+
+import asyncio
+import logging
+import time
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import TYPE_CHECKING, Any
+from uuid import uuid4
+
+from pydantic import BaseModel
+
+from backend.copilot.feedback_db import RATEABLE_ROLES
+from backend.copilot.llm_attribution import is_turn_impersonated, turn_trace_id
+from backend.util import posthog_client
+from backend.util.posthog_events import PostHogEvent
+
+if TYPE_CHECKING:
+ from backend.copilot.executor.utils import CoPilotExecutionEntry
+ from backend.copilot.model import ChatMessage
+
+logger = logging.getLogger(__name__)
+
+SOURCE = "chat_copilot"
+TRACE_NAME = "copilot-turn"
+SUBTASK_SPAN_NAME = "copilot-subtask"
+
+# The span a tool call or subtask started now nests under: the running
+# tool's, or else the running turn's. The executor sets it for every turn.
+_current_span_id: ContextVar[str | None] = ContextVar("_current_span_id", default=None)
+# How the running tool call is ending, for the tool's own code to mark.
+_current_outcome: ContextVar["SpanOutcome | None"] = ContextVar(
+ "_current_outcome", default=None
+)
+
+
+def trace_id_for(entry: "CoPilotExecutionEntry") -> str:
+ """The trace a turn's events join.
+
+ A spawned turn (an expert subtask, a sub-session) joins the trace of the
+ turn that spawned it, which its envelope records when it is derived
+ (``TurnEnvelope.ai_trace_id``). That is not always its tree's root: an
+ approval wake is a reply of its own under the tree it was held in. A
+ queued child and a woken subtask start under their stored envelope, so
+ they keep that trace. An envelope stored before it kept the trace falls
+ back to its tree's root. Every other turn is a reply of its own.
+ """
+ envelope = entry.envelope
+ if envelope is not None and envelope.ai_trace_id:
+ return envelope.ai_trace_id
+ if envelope is not None and envelope.depth > 0 and envelope.tree_id:
+ return envelope.tree_id
+ return entry.turn_id
+
+
+def set_turn_span(span_id: str | None) -> None:
+ """Make *span_id* (the running turn's id) the parent of its tool calls."""
+ _current_span_id.set(span_id or None)
+
+
+def current_span_id() -> str | None:
+ """The span a tool call or subtask started now nests under."""
+ return _current_span_id.get()
+
+
+def mark_tool_failed() -> None:
+ """Mark the running tool call failed: it answered with an error rather
+ than raising one. Outside a tool span it does nothing."""
+ outcome = _current_outcome.get()
+ if outcome is not None:
+ outcome.is_error = True
+
+
+def mark_tool_raised(error: BaseException) -> None:
+ """Mark the running tool call failed by *error*, which the tool caught
+ and answered with instead of letting it reach ``tool_span``. Only the
+ exception's type is kept, never its message. Outside a tool span it does
+ nothing."""
+ outcome = _current_outcome.get()
+ if outcome is not None:
+ outcome.is_error = True
+ outcome.error = type(error).__name__
+
+
+class SpanOutcome(BaseModel):
+ """How a span ended; the code it wraps marks a failure."""
+
+ is_error: bool = False
+ error: str | None = None
+
+
+@contextmanager
+def tool_span(
+ tool_name: str,
+ *,
+ user_id: str | None,
+ session_id: str,
+ tool_call_id: str | None = None,
+) -> Iterator[SpanOutcome]:
+ """Time the block as one tool call of the running turn, sent as an
+ ``$ai_span`` when it ends. Outside a turn there is no trace, so nothing
+ is sent. Anything the block runs nests under this span."""
+ outcome = SpanOutcome()
+ trace_id = turn_trace_id()
+ if not trace_id:
+ yield outcome
+ return
+ span_id = str(uuid4())
+ enclosing = _current_span_id.get()
+ enclosing_outcome = _current_outcome.get()
+ parent_id = enclosing or trace_id
+ _current_span_id.set(span_id)
+ _current_outcome.set(outcome)
+ started_at = time.monotonic()
+ cancelled = False
+ try:
+ yield outcome
+ except asyncio.CancelledError:
+ # The turn was stopped; the tool itself did not fail.
+ cancelled = True
+ raise
+ except BaseException as error:
+ outcome.is_error = True
+ outcome.error = type(error).__name__
+ raise
+ finally:
+ _current_span_id.set(enclosing)
+ _current_outcome.set(enclosing_outcome)
+ _send_span(
+ user_id=user_id,
+ trace_id=trace_id,
+ span_id=span_id,
+ parent_id=parent_id,
+ name=tool_name,
+ session_id=session_id,
+ latency_seconds=time.monotonic() - started_at,
+ outcome=outcome,
+ extra={"tool_call_id": tool_call_id, "cancelled": cancelled},
+ )
+
+
+class _Started(BaseModel):
+ at: float
+ trace_id: str
+ parent_id: str
+
+
+# Built-in tools the CLI calls often and cheaply, sub-agents included: tens
+# a turn, each a lookup or a checklist update whose span says little. They
+# get no span, which keeps the span volume to the calls that act or spend.
+UNTRACED_BUILTIN_TOOLS: frozenset[str] = frozenset(
+ {"TodoWrite", "Read", "Glob", "Grep", "LS", "NotebookRead"}
+)
+
+
+class ToolSpanTimer:
+ """Times tool calls whose start and end arrive in separate callbacks: the
+ Claude Agent SDK's Pre/PostToolUse hooks for the CLI's built-in tools
+ (Bash, Task, WebSearch, ...), which never pass through ``BaseTool``.
+
+ One timer serves one turn's hooks and goes with them, so a call whose
+ end never arrives is dropped at the end of its turn.
+
+ *enabled* off (``ChatConfig.posthog_builtin_tool_spans``) sends no span
+ at all; on, every built-in but ``UNTRACED_BUILTIN_TOOLS`` gets one."""
+
+ def __init__(self, *, enabled: bool = True) -> None:
+ self._enabled = enabled
+ self._started: dict[str, _Started] = {}
+
+ def start(self, tool_use_id: str, tool_name: str) -> None:
+ if not self._enabled or tool_name in UNTRACED_BUILTIN_TOOLS:
+ return
+ trace_id = turn_trace_id()
+ if not trace_id:
+ return
+ self._started[tool_use_id] = _Started(
+ at=time.monotonic(),
+ trace_id=trace_id,
+ parent_id=_current_span_id.get() or trace_id,
+ )
+
+ def finish(
+ self,
+ tool_use_id: str,
+ tool_name: str,
+ *,
+ user_id: str | None,
+ session_id: str | None,
+ is_error: bool,
+ ) -> None:
+ """Send the call's span. A call that never started (refused, or
+ outside a turn) sends nothing."""
+ started = self._started.pop(tool_use_id, None)
+ if started is None:
+ return
+ _send_span(
+ user_id=user_id,
+ trace_id=started.trace_id,
+ span_id=str(uuid4()),
+ parent_id=started.parent_id,
+ name=tool_name,
+ session_id=session_id,
+ latency_seconds=time.monotonic() - started.at,
+ outcome=SpanOutcome(is_error=is_error),
+ extra={"tool_call_id": tool_use_id, "builtin": True},
+ )
+
+
+def capture_turn(
+ entry: "CoPilotExecutionEntry",
+ *,
+ latency_seconds: float,
+ outcome: str,
+ error_code: str | None,
+ engine: str,
+ origin: str,
+ tool_call_count: int,
+) -> None:
+ """Send the end of a turn: the ``$ai_trace`` of a reply, or the
+ ``$ai_span`` of a subtask in the trace of the reply it works for.
+
+ *outcome* is ``completed``, ``cancelled`` or ``failed``; only ``failed``
+ is an error, and *error_code* (a code, never a message) says which.
+
+ A redelivered turn ends twice; the turn id is the event's dedup key, so
+ PostHog keeps one root (or one subtask span) per turn."""
+ try:
+ trace_id = trace_id_for(entry)
+ failed = outcome == "failed"
+ properties: dict[str, Any] = {
+ "$ai_trace_id": trace_id,
+ "$ai_session_id": entry.session_id,
+ "$ai_latency": latency_seconds,
+ "$ai_is_error": failed,
+ "chat_session_id": entry.session_id,
+ "turn_id": entry.turn_id,
+ "outcome": outcome,
+ "engine": engine,
+ "turn_origin": origin,
+ "tool_call_count": tool_call_count,
+ "impersonated": entry.impersonated,
+ }
+ if failed and error_code:
+ properties["$ai_error"] = error_code
+ if trace_id == entry.turn_id:
+ properties["$ai_span_name"] = TRACE_NAME
+ event = PostHogEvent.AI_TRACE
+ else:
+ properties["$ai_span_name"] = SUBTASK_SPAN_NAME
+ properties["$ai_span_id"] = entry.turn_id
+ envelope = entry.envelope
+ properties["$ai_parent_id"] = (
+ envelope.ai_parent_span_id if envelope else None
+ ) or trace_id
+ properties["depth"] = envelope.depth if envelope else None
+ event = PostHogEvent.AI_SPAN
+ posthog_client.capture(
+ entry.user_id,
+ event,
+ properties,
+ source=SOURCE,
+ dedup_key=entry.turn_id,
+ )
+ except Exception:
+ logger.warning("Failed to send the turn's PostHog trace", exc_info=True)
+
+
+def stamp_turn_trace_id(messages: Sequence["ChatMessage"], start_index: int) -> None:
+ """Record the running turn's trace on the reply rows it wrote, so a
+ rating of the reply names the trace (``feedback._send_to_posthog``).
+
+ Bounded to the turn and never overwriting, like the other end-of-turn
+ stamps; a row already flushed mid-turn is flagged for back-fill."""
+ trace_id = turn_trace_id()
+ if not trace_id:
+ return
+ for message in messages[start_index:]:
+ if message.role in RATEABLE_ROLES and message.trace_id is None:
+ message.trace_id = trace_id
+ if message.sequence is not None:
+ message.stamps_pending_save = True
+
+
+def _send_span(
+ *,
+ user_id: str | None,
+ trace_id: str,
+ span_id: str,
+ parent_id: str,
+ name: str,
+ session_id: str | None,
+ latency_seconds: float,
+ outcome: SpanOutcome,
+ extra: dict[str, Any],
+) -> None:
+ try:
+ properties: dict[str, Any] = {
+ "$ai_trace_id": trace_id,
+ "$ai_span_id": span_id,
+ "$ai_parent_id": parent_id,
+ "$ai_span_name": name,
+ "$ai_latency": latency_seconds,
+ "$ai_is_error": outcome.is_error,
+ "impersonated": is_turn_impersonated(),
+ **{key: value for key, value in extra.items() if value is not None},
+ }
+ if session_id:
+ properties["$ai_session_id"] = session_id
+ properties["chat_session_id"] = session_id
+ if outcome.is_error and outcome.error:
+ properties["$ai_error"] = outcome.error
+ posthog_client.capture(user_id, PostHogEvent.AI_SPAN, properties, source=SOURCE)
+ except Exception:
+ logger.warning(f"Failed to send the PostHog span of {name}", exc_info=True)
diff --git a/autogpt_platform/backend/backend/copilot/ai_spans_test.py b/autogpt_platform/backend/backend/copilot/ai_spans_test.py
new file mode 100644
index 0000000000..b41107bbb2
--- /dev/null
+++ b/autogpt_platform/backend/backend/copilot/ai_spans_test.py
@@ -0,0 +1,614 @@
+"""Tests for ``copilot.ai_spans``: the PostHog trace of an AutoPilot reply,
+its tool calls and its expert subtasks."""
+
+import asyncio
+from unittest.mock import MagicMock
+
+import pytest
+import pytest_mock
+
+from backend.copilot import ai_spans
+from backend.copilot.ai_spans import (
+ ToolSpanTimer,
+ capture_turn,
+ current_span_id,
+ mark_tool_failed,
+ mark_tool_raised,
+ set_turn_span,
+ stamp_turn_trace_id,
+ tool_span,
+ trace_id_for,
+)
+from backend.copilot.executor.utils import CoPilotExecutionEntry
+from backend.copilot.llm_attribution import set_turn_impersonated, set_turn_trace_id
+from backend.copilot.model import ChatMessage
+from backend.copilot.tree import TurnEnvelope
+from backend.util import posthog_client
+from backend.util.posthog_events import PostHogEvent
+
+USER_ID = "user-1"
+SESSION_ID = "sess-1"
+TURN_ID = "turn-1"
+
+
[email protected](autouse=True)
+def _outside_a_turn():
+ set_turn_trace_id(None)
+ set_turn_span(None)
+ set_turn_impersonated(False)
+ yield
+ set_turn_trace_id(None)
+ set_turn_span(None)
+ set_turn_impersonated(False)
+
+
+def capture(mocker: pytest_mock.MockerFixture) -> MagicMock:
+ return mocker.patch.object(ai_spans.posthog_client, "capture")
+
+
+def _entry(**overrides) -> CoPilotExecutionEntry:
+ fields = {
+ "session_id": SESSION_ID,
+ "turn_id": TURN_ID,
+ "user_id": USER_ID,
+ "message": "hi",
+ "envelope": TurnEnvelope(tree_id=TURN_ID),
+ **overrides,
+ }
+ return CoPilotExecutionEntry(**fields)
+
+
+def _in_a_turn(turn_id: str = TURN_ID, trace_id: str = TURN_ID) -> None:
+ set_turn_trace_id(trace_id)
+ set_turn_span(turn_id)
+
+
+def _sent(capture: MagicMock) -> list[tuple[PostHogEvent, dict]]:
+ return [(call.args[1], call.args[2]) for call in capture.call_args_list]
+
+
+# -- trace_id_for -------------------------------------------------------------
+
+
+def test_a_reply_is_its_own_trace():
+ assert trace_id_for(_entry(envelope=None)) == TURN_ID
+ assert trace_id_for(_entry()) == TURN_ID
+
+
+def test_a_subtask_joins_the_trace_of_the_turn_that_spawned_it():
+ entry = _entry(
+ turn_id="child-turn",
+ envelope=TurnEnvelope(tree_id="root-turn", depth=1, ai_trace_id="root-turn"),
+ )
+
+ assert trace_id_for(entry) == "root-turn"
+
+
+def test_a_subtask_of_a_woken_reply_joins_that_reply_not_its_tree_root():
+ """An approval wake is a reply of its own (B) under the tree it was held
+ in (A), so its subtasks belong to B's trace."""
+ entry = _entry(
+ turn_id="child-turn",
+ envelope=TurnEnvelope(
+ tree_id="held-turn-a", depth=1, ai_trace_id="woken-turn-b"
+ ),
+ )
+
+ assert trace_id_for(entry) == "woken-turn-b"
+
+
+def test_a_woken_subtask_stays_in_the_trace_of_the_turn_that_held_the_call():
+ """Reply A is woken as reply B, B delegates to C, and C holds a call. The
+ wake C' starts under C's envelope, so it joins B's trace, not A's."""
+ held_by_c = TurnEnvelope(
+ tree_id="reply-a",
+ depth=1,
+ ai_trace_id="reply-b",
+ ai_parent_span_id="b-delegate-span",
+ )
+
+ assert trace_id_for(_entry(turn_id="woken-c", envelope=held_by_c)) == "reply-b"
+
+
+def test_a_subtask_from_before_the_trace_was_recorded_falls_back_to_its_tree_root():
+ """An envelope stored before it kept the trace (a queued child, a held
+ call), or one rebuilt from its tree id alone."""
+ entry = _entry(
+ turn_id="child-turn", envelope=TurnEnvelope(tree_id="root-turn", depth=1)
+ )
+
+ assert trace_id_for(entry) == "root-turn"
+
+
+def test_a_root_re_admitted_under_an_older_tree_is_still_its_own_trace():
+ """An approval wake re-admits the envelope its call was held under, so
+ a depth-0 tree id can name an earlier turn."""
+ assert trace_id_for(_entry(envelope=TurnEnvelope(tree_id="earlier"))) == TURN_ID
+
+
+# -- tool_span ----------------------------------------------------------------
+
+
+def test_a_tool_call_is_a_span_under_its_turn(capture: MagicMock):
+ _in_a_turn()
+
+ with tool_span(
+ "run_block", user_id=USER_ID, session_id=SESSION_ID, tool_call_id="call-1"
+ ):
+ pass
+
+ capture.assert_called_once()
+ distinct_id, event, properties = capture.call_args.args
+ assert (distinct_id, event) == (USER_ID, PostHogEvent.AI_SPAN)
+ assert capture.call_args.kwargs == {"source": "chat_copilot"}
+ span_id = properties.pop("$ai_span_id")
+ assert span_id and span_id != TURN_ID
+ assert properties.pop("$ai_latency") >= 0
+ assert properties == {
+ "$ai_trace_id": TURN_ID,
+ "$ai_parent_id": TURN_ID,
+ "$ai_span_name": "run_block",
+ "$ai_is_error": False,
+ "$ai_session_id": SESSION_ID,
+ "chat_session_id": SESSION_ID,
+ "tool_call_id": "call-1",
+ "cancelled": False,
+ "impersonated": False,
+ }
+
+
+def test_a_span_never_carries_the_tools_input_or_output(capture: MagicMock):
+ """Gmail and Drive content must stay out (Google Limited Use)."""
+ _in_a_turn()
+
+ with tool_span("gmail_read", user_id=USER_ID, session_id=SESSION_ID) as outcome:
+ outcome.is_error = True
+
+ properties = capture.call_args.args[2]
+ assert "$ai_input_state" not in properties
+ assert "$ai_output_state" not in properties
+ assert "$ai_error" not in properties
+ assert properties["$ai_is_error"] is True
+
+
+def test_a_tool_that_answers_with_an_error_is_marked_failed(capture: MagicMock):
+ _in_a_turn()
+
+ with (
+ tool_span("run_capability", user_id=USER_ID, session_id=SESSION_ID),
+ tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID),
+ ):
+ mark_tool_failed()
+
+ (_, inner), (_, outer) = _sent(capture)
+ assert inner["$ai_is_error"] is True
+ assert outer["$ai_is_error"] is False
+
+
+def test_marking_a_failure_outside_a_tool_does_nothing(capture: MagicMock):
+ mark_tool_failed()
+
+ capture.assert_not_called()
+
+
+def test_a_tool_that_raises_is_an_error_named_by_its_type_only(capture: MagicMock):
+ _in_a_turn()
+
+ with (
+ pytest.raises(ValueError),
+ tool_span("web_fetch", user_id=USER_ID, session_id=SESSION_ID),
+ ):
+ raise ValueError("the secret contents of an email")
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert properties["$ai_error"] == "ValueError"
+ assert "secret" not in repr(properties)
+
+
+def test_a_tool_that_caught_what_it_raised_is_named_by_its_type_only(
+ capture: MagicMock,
+):
+ """``BaseTool`` answers an exception with an error instead of letting it
+ reach the span, so it names the exception itself."""
+ _in_a_turn()
+
+ with tool_span("web_fetch", user_id=USER_ID, session_id=SESSION_ID):
+ try:
+ raise ValueError("the secret contents of an email")
+ except ValueError as error:
+ mark_tool_raised(error)
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert properties["$ai_error"] == "ValueError"
+ assert "secret" not in repr(properties)
+
+
+def test_marking_a_raise_outside_a_tool_does_nothing(capture: MagicMock):
+ mark_tool_raised(ValueError("x"))
+
+ capture.assert_not_called()
+
+
+def test_a_cancelled_tool_is_not_a_failure(capture: MagicMock):
+ _in_a_turn()
+
+ with (
+ pytest.raises(asyncio.CancelledError),
+ tool_span("bash_exec", user_id=USER_ID, session_id=SESSION_ID),
+ ):
+ raise asyncio.CancelledError()
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is False
+ assert properties["cancelled"] is True
+
+
+def test_a_tool_called_by_a_tool_nests_under_it(capture: MagicMock):
+ _in_a_turn()
+
+ with tool_span("run_capability", user_id=USER_ID, session_id=SESSION_ID):
+ outer = current_span_id()
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ pass
+
+ (_, inner), (_, outer_props) = _sent(capture)
+ assert inner["$ai_parent_id"] == outer == outer_props["$ai_span_id"]
+ assert outer_props["$ai_parent_id"] == TURN_ID
+ assert current_span_id() == TURN_ID
+
+
+def test_a_subtasks_tool_calls_nest_under_the_subtask(capture: MagicMock):
+ _in_a_turn(turn_id="child-turn", trace_id="root-turn")
+
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ pass
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_trace_id"] == "root-turn"
+ assert properties["$ai_parent_id"] == "child-turn"
+
+
+def test_an_impersonated_turns_spans_are_marked(capture: MagicMock):
+ _in_a_turn()
+ set_turn_impersonated(True)
+
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ pass
+
+ assert capture.call_args.args[2]["impersonated"] is True
+
+
+def test_outside_a_turn_no_span_is_sent(capture: MagicMock):
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ pass
+
+ capture.assert_not_called()
+
+
+async def test_spans_of_concurrent_tool_calls_do_not_cross(capture: MagicMock):
+ _in_a_turn()
+
+ async def call(name: str) -> str | None:
+ with tool_span(name, user_id=USER_ID, session_id=SESSION_ID):
+ await asyncio.sleep(0)
+ return current_span_id()
+
+ first, second = await asyncio.gather(
+ asyncio.create_task(call("a")), asyncio.create_task(call("b"))
+ )
+
+ by_name = {props["$ai_span_name"]: props for _, props in _sent(capture)}
+ assert by_name["a"]["$ai_span_id"] == first
+ assert by_name["b"]["$ai_span_id"] == second
+ assert by_name["a"]["$ai_parent_id"] == by_name["b"]["$ai_parent_id"] == TURN_ID
+
+
+def test_a_posthog_failure_never_reaches_the_tool(mocker: pytest_mock.MockerFixture):
+ """The real ``capture`` runs against a client that raises."""
+ client = MagicMock()
+ client.capture.side_effect = RuntimeError("posthog is down")
+ mocker.patch.object(posthog_client, "get_posthog_client", return_value=client)
+ _in_a_turn()
+
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ result = "ran"
+
+ assert result == "ran"
+ client.capture.assert_called_once()
+
+
+def test_a_broken_property_never_reaches_the_tool(capture: MagicMock):
+ capture.side_effect = RuntimeError("unexpected")
+ _in_a_turn()
+
+ with tool_span("run_block", user_id=USER_ID, session_id=SESSION_ID):
+ pass
+
+
+# -- ToolSpanTimer (SDK built-in tools) ---------------------------------------
+
+
+def test_a_builtin_tool_is_timed_from_its_hooks(capture: MagicMock):
+ _in_a_turn()
+ timer = ToolSpanTimer()
+
+ timer.start("tu-1", "WebSearch")
+ timer.finish(
+ "tu-1", "WebSearch", user_id=USER_ID, session_id=SESSION_ID, is_error=False
+ )
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_span_name"] == "WebSearch"
+ assert properties["$ai_trace_id"] == TURN_ID
+ assert properties["$ai_parent_id"] == TURN_ID
+ assert properties["$ai_is_error"] is False
+ assert properties["builtin"] is True
+ assert properties["tool_call_id"] == "tu-1"
+
+
+def test_a_failed_builtin_tool_is_an_error(capture: MagicMock):
+ _in_a_turn()
+ timer = ToolSpanTimer()
+
+ timer.start("tu-1", "Bash")
+ timer.finish("tu-1", "Bash", user_id=USER_ID, session_id=SESSION_ID, is_error=True)
+
+ assert capture.call_args.args[2]["$ai_is_error"] is True
+
+
+def test_a_builtin_tool_that_never_started_sends_nothing(capture: MagicMock):
+ _in_a_turn()
+ timer = ToolSpanTimer()
+
+ timer.finish("tu-1", "Bash", user_id=USER_ID, session_id=SESSION_ID, is_error=True)
+
+ capture.assert_not_called()
+
+
+def test_a_builtin_tool_outside_a_turn_sends_nothing(capture: MagicMock):
+ timer = ToolSpanTimer()
+
+ timer.start("tu-1", "Bash")
+ timer.finish("tu-1", "Bash", user_id=USER_ID, session_id=SESSION_ID, is_error=False)
+
+ capture.assert_not_called()
+
+
[email protected]("tool_name", ["TodoWrite", "Read", "Glob", "Grep", "LS"])
+def test_a_high_volume_builtin_tool_sends_no_span(capture: MagicMock, tool_name: str):
+ """The CLI calls these tens of times a turn, sub-agents included, and a
+ span of a file read or a checklist update says little."""
+ _in_a_turn()
+ timer = ToolSpanTimer()
+
+ timer.start("tu-1", tool_name)
+ timer.finish(
+ "tu-1", tool_name, user_id=USER_ID, session_id=SESSION_ID, is_error=False
+ )
+
+ capture.assert_not_called()
+
+
[email protected]("tool_name", ["Task", "Agent", "Bash", "Edit", "Write"])
+def test_a_builtin_tool_that_acts_keeps_its_span(capture: MagicMock, tool_name: str):
+ _in_a_turn()
+ timer = ToolSpanTimer()
+
+ timer.start("tu-1", tool_name)
+ timer.finish(
+ "tu-1", tool_name, user_id=USER_ID, session_id=SESSION_ID, is_error=False
+ )
+
+ assert capture.call_args.args[2]["$ai_span_name"] == tool_name
+
+
+def test_builtin_tool_spans_can_be_turned_off(capture: MagicMock):
+ _in_a_turn()
+ timer = ToolSpanTimer(enabled=False)
+
+ timer.start("tu-1", "Task")
+ timer.finish("tu-1", "Task", user_id=USER_ID, session_id=SESSION_ID, is_error=False)
+
+ capture.assert_not_called()
+
+
+# -- capture_turn -------------------------------------------------------------
+
+
+def _end(entry: CoPilotExecutionEntry, **overrides) -> None:
+ kwargs = {
+ "latency_seconds": 12.5,
+ "outcome": "completed",
+ "error_code": None,
+ "engine": "sdk",
+ "origin": "chat",
+ "tool_call_count": 3,
+ **overrides,
+ }
+ capture_turn(entry, **kwargs)
+
+
+def test_a_reply_ends_with_its_trace_root(capture: MagicMock):
+ _end(_entry())
+
+ capture.assert_called_once()
+ distinct_id, event, properties = capture.call_args.args
+ assert (distinct_id, event) == (USER_ID, PostHogEvent.AI_TRACE)
+ assert properties == {
+ "$ai_trace_id": TURN_ID,
+ "$ai_session_id": SESSION_ID,
+ "$ai_span_name": "copilot-turn",
+ "$ai_latency": 12.5,
+ "$ai_is_error": False,
+ "chat_session_id": SESSION_ID,
+ "turn_id": TURN_ID,
+ "outcome": "completed",
+ "engine": "sdk",
+ "turn_origin": "chat",
+ "tool_call_count": 3,
+ "impersonated": False,
+ }
+
+
+def test_a_failed_reply_names_its_error_code(capture: MagicMock):
+ _end(_entry(), outcome="failed", error_code="baseline_error")
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert properties["$ai_error"] == "baseline_error"
+
+
+def test_a_cancelled_reply_is_not_an_error(capture: MagicMock):
+ _end(_entry(), outcome="cancelled")
+
+ assert capture.call_args.args[2]["$ai_is_error"] is False
+
+
+def test_an_impersonated_reply_is_marked(capture: MagicMock):
+ _end(_entry(impersonated=True))
+
+ assert capture.call_args.args[2]["impersonated"] is True
+
+
+def test_a_subtask_is_a_span_under_the_tool_call_that_started_it(capture: MagicMock):
+ entry = _entry(
+ turn_id="child-turn",
+ session_id="expert-sess",
+ envelope=TurnEnvelope(
+ tree_id="root-turn",
+ depth=1,
+ ai_trace_id="root-turn",
+ ai_parent_span_id="spawn-tool-span",
+ ),
+ )
+
+ _end(entry)
+
+ _, event, properties = capture.call_args.args
+ assert event == PostHogEvent.AI_SPAN
+ assert properties["$ai_trace_id"] == "root-turn"
+ assert properties["$ai_span_id"] == "child-turn"
+ assert properties["$ai_parent_id"] == "spawn-tool-span"
+ assert properties["$ai_span_name"] == "copilot-subtask"
+ assert properties["$ai_session_id"] == "expert-sess"
+ assert properties["depth"] == 1
+
+
+def test_a_subtask_of_unknown_origin_hangs_off_the_trace_root(capture: MagicMock):
+ """An envelope stored before it kept the spawning tool's span."""
+ entry = _entry(
+ turn_id="child-turn", envelope=TurnEnvelope(tree_id="root-turn", depth=1)
+ )
+
+ _end(entry)
+
+ assert capture.call_args.args[2]["$ai_parent_id"] == "root-turn"
+
+
+def test_a_turn_end_failure_is_swallowed(capture: MagicMock):
+ capture.side_effect = RuntimeError("unexpected")
+
+ _end(_entry())
+
+
+def _redelivered_twice(
+ entry: CoPilotExecutionEntry, mocker: pytest_mock.MockerFixture
+) -> list[dict]:
+ """End the same turn twice, as a redelivered turn would, and return what
+ reached the PostHog client."""
+ client = MagicMock()
+ mocker.patch.object(posthog_client, "get_posthog_client", return_value=client)
+ _end(entry)
+ _end(entry)
+ return [call.kwargs for call in client.capture.call_args_list]
+
+
+def test_a_redelivered_reply_sends_one_trace_root(
+ mocker: pytest_mock.MockerFixture,
+):
+ first, second = _redelivered_twice(_entry(), mocker)
+
+ assert first["event"] == PostHogEvent.AI_TRACE.value
+ assert first["uuid"] is not None
+ assert first["uuid"] == second["uuid"]
+ assert first["properties"]["$insert_id"] == TURN_ID
+
+
+def test_a_redelivered_subtask_sends_one_span(mocker: pytest_mock.MockerFixture):
+ entry = _entry(
+ turn_id="child-turn",
+ envelope=TurnEnvelope(tree_id="root-turn", depth=1, ai_trace_id="root-turn"),
+ )
+
+ first, second = _redelivered_twice(entry, mocker)
+
+ assert first["event"] == PostHogEvent.AI_SPAN.value
+ assert first["uuid"] is not None
+ assert first["uuid"] == second["uuid"]
+
+
+def test_different_turns_are_not_deduplicated(mocker: pytest_mock.MockerFixture):
+ client = MagicMock()
+ mocker.patch.object(posthog_client, "get_posthog_client", return_value=client)
+
+ _end(_entry(turn_id="turn-a"))
+ _end(_entry(turn_id="turn-b"))
+
+ first, second = (call.kwargs for call in client.capture.call_args_list)
+ assert first["uuid"] != second["uuid"]
+
+
+# -- stamp_turn_trace_id ------------------------------------------------------
+
+
+def _messages() -> list[ChatMessage]:
+ return [
+ ChatMessage(role="assistant", content="old", sequence=0),
+ ChatMessage(role="user", content="q", sequence=1),
+ ChatMessage(role="assistant", content="working", sequence=2),
+ ChatMessage(role="tool", content="{}", tool_call_id="t1"),
+ ChatMessage(role="assistant", content="done"),
+ ChatMessage(role="reasoning", content="checking the totals"),
+ ]
+
+
+def test_stamps_the_turns_replies_with_its_trace():
+ set_turn_trace_id(TURN_ID)
+ messages = _messages()
+
+ stamp_turn_trace_id(messages, start_index=1)
+
+ assert [m.trace_id for m in messages] == [
+ None, # an earlier turn's reply
+ None, # user row
+ TURN_ID,
+ None, # tool row
+ TURN_ID,
+ TURN_ID, # a reply bubble can be named after its reasoning row
+ ]
+ assert messages[2].stamps_pending_save is True # flushed mid-turn
+ assert messages[4].stamps_pending_save is False # inserted by this save
+
+
+def test_never_overwrites_a_recorded_trace():
+ set_turn_trace_id(TURN_ID)
+ messages = _messages()
+ messages[2].trace_id = "earlier-turn"
+
+ stamp_turn_trace_id(messages, start_index=0)
+
+ assert messages[2].trace_id == "earlier-turn"
+ assert messages[2].stamps_pending_save is False
+
+
+def test_outside_a_turn_stamps_nothing():
+ messages = _messages()
+
+ stamp_turn_trace_id(messages, start_index=0)
+
+ assert all(m.trace_id is None for m in messages)
+ assert not any(m.stamps_pending_save for m in messages)
diff --git a/autogpt_platform/backend/backend/copilot/baseline/service.py b/autogpt_platform/backend/backend/copilot/baseline/service.py
index 96f08a031a..50a2f1f768 100644
--- a/autogpt_platform/backend/backend/copilot/baseline/service.py
+++ b/autogpt_platform/backend/backend/copilot/baseline/service.py
@@ -30,6 +30,7 @@ from opentelemetry import trace as otel_trace
from backend.blocks.desktop._common import workspace_volume_mounts
from backend.copilot import engine_switch
+from backend.copilot.ai_spans import stamp_turn_trace_id
from backend.copilot.anthropic_rate_card import (
compute_anthropic_cost_usd,
get_max_output_tokens,
@@ -2599,6 +2600,7 @@ async def stream_chat_completion_baseline(
# Persist the assistant/tool flush BEFORE the pending append
# so a later pending-persist failure can roll back the
# pending rows without also discarding LLM output.
+ stamp_turn_trace_id(current_session.messages, turn_start)
current_session = await persist_session_safe(
current_session, "[Baseline]"
)
@@ -2764,6 +2766,7 @@ async def stream_chat_completion_baseline(
retryable=failure.retryable if failure is not None else True,
failure=failure.as_part() if failure is not None else None,
):
+ stamp_turn_trace_id(_failed_session.messages, turn_start)
try:
await upsert_chat_session(_failed_session)
except Exception as marker_err:
@@ -2945,6 +2948,7 @@ async def stream_chat_completion_baseline(
llm_credential_id=state.llm_credential_id,
)
)
+ stamp_turn_trace_id(session.messages, turn_start)
try:
await upsert_chat_session(session)
final_checkpoint = turn_checkpoint(session.messages, turn_start)
diff --git a/autogpt_platform/backend/backend/copilot/baseline/service_unit_test.py b/autogpt_platform/backend/backend/copilot/baseline/service_unit_test.py
index 825fc04507..fea5074cb6 100644
--- a/autogpt_platform/backend/backend/copilot/baseline/service_unit_test.py
+++ b/autogpt_platform/backend/backend/copilot/baseline/service_unit_test.py
@@ -3199,3 +3199,32 @@ class TestBaselineToolExecutorForwardsDisabledGroups:
)
assert kwargs["disabled_tools"] == frozenset({"run_agent"})
+
+
+class TestStampPostHogTrace:
+ def test_every_save_of_the_turns_rows_stamps_its_posthog_trace_first(self):
+ """A rating of the reply names its PostHog trace from this stamp
+ (``ai_spans.stamp_turn_trace_id``): the mid-turn flush, the error
+ marker and the end-of-turn save each stamp before they persist."""
+ import inspect
+
+ from backend.copilot.baseline.service import stream_chat_completion_baseline
+
+ src = inspect.getsource(stream_chat_completion_baseline)
+ pairs = [
+ (
+ "stamp_turn_trace_id(current_session.messages, turn_start)",
+ "current_session = await persist_session_safe(",
+ ),
+ (
+ "stamp_turn_trace_id(_failed_session.messages, turn_start)",
+ "await upsert_chat_session(_failed_session)",
+ ),
+ (
+ "stamp_turn_trace_id(session.messages, turn_start)",
+ " await upsert_chat_session(session)\n",
+ ),
+ ]
+ for stamp, save in pairs:
+ stamped_at = src.index(stamp)
+ assert stamped_at < src.index(save, stamped_at)
diff --git a/autogpt_platform/backend/backend/copilot/config.py b/autogpt_platform/backend/backend/copilot/config.py
index 28a8e7bfdb..1ff8d2c5fa 100644
--- a/autogpt_platform/backend/backend/copilot/config.py
+++ b/autogpt_platform/backend/backend/copilot/config.py
@@ -394,6 +394,16 @@ class ChatConfig(BaseSettings):
"re-fetching it (0 to disable caching)",
)
+ # PostHog LLM analytics (copilot/ai_spans.py)
+ posthog_builtin_tool_spans: bool = Field(
+ default=True,
+ description="Send a PostHog ``$ai_span`` for each call of the SDK "
+ "CLI's built-in tools (Task/Agent, WebSearch, ...), sub-agents' calls "
+ "included. Off, only platform tools get spans. The high-volume lookups "
+ "in ``ai_spans.UNTRACED_BUILTIN_TOOLS`` (Read, Grep, TodoWrite, ...) "
+ "never get one.",
+ )
+
# Rate limiting — cost-based limits per day and per week, stored in
# microdollars (1 USD = 1_000_000). The counter tracks the real
# generation cost reported by the provider (OpenRouter ``usage.cost``
diff --git a/autogpt_platform/backend/backend/copilot/config_test.py b/autogpt_platform/backend/backend/copilot/config_test.py
index e27ddc7f88..8d55ad074c 100644
--- a/autogpt_platform/backend/backend/copilot/config_test.py
+++ b/autogpt_platform/backend/backend/copilot/config_test.py
@@ -1199,3 +1199,12 @@ class TestLangfusePromptCacheTTL:
# at once, which is the unbounded staleness this field exists to avoid.
with pytest.raises(ValidationError, match="langfuse_prompt_cache_ttl"):
ChatConfig(langfuse_prompt_cache_ttl=-1)
+
+
+class TestPosthogBuiltinToolSpans:
+ def test_on_by_default(self):
+ assert ChatConfig.model_fields["posthog_builtin_tool_spans"].default is True
+
+ def test_env_turns_it_off(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("CHAT_POSTHOG_BUILTIN_TOOL_SPANS", "false")
+ assert ChatConfig().posthog_builtin_tool_spans is False
diff --git a/autogpt_platform/backend/backend/copilot/db.py b/autogpt_platform/backend/backend/copilot/db.py
index a18fba0275..8f27debef8 100644
--- a/autogpt_platform/backend/backend/copilot/db.py
+++ b/autogpt_platform/backend/backend/copilot/db.py
@@ -735,6 +735,8 @@ async def add_chat_messages_batch(
data["llmCredentialId"] = msg["llm_credential_id"]
if msg.get("langfuse_trace_id") is not None:
data["langfuseTraceId"] = msg["langfuse_trace_id"]
+ if msg.get("trace_id") is not None:
+ data["traceId"] = msg["trace_id"]
# Per-row bag. The single-message path already persisted
# this; the batch path silently dropped it, so anything
@@ -1273,6 +1275,7 @@ async def update_chat_message_stamps(
llm_auth_provider: str | None = None,
llm_credential_id: str | None = None,
langfuse_trace_id: str | None = None,
+ trace_id: str | None = None,
) -> bool:
"""Back-fill the execution stamps on an already-persisted message row.
@@ -1296,6 +1299,8 @@ async def update_chat_message_stamps(
data["llmCredentialId"] = llm_credential_id
if langfuse_trace_id is not None:
data["langfuseTraceId"] = langfuse_trace_id
+ if trace_id is not None:
+ data["traceId"] = trace_id
result = await PrismaChatMessage.prisma().update(
where={"sessionId_sequence": {"sessionId": session_id, "sequence": sequence}},
data=data,
diff --git a/autogpt_platform/backend/backend/copilot/db_test.py b/autogpt_platform/backend/backend/copilot/db_test.py
index 18b413c265..914c9f8580 100644
--- a/autogpt_platform/backend/backend/copilot/db_test.py
+++ b/autogpt_platform/backend/backend/copilot/db_test.py
@@ -1106,6 +1106,7 @@ async def test_batch_persist_maps_stamp_columns_to_prisma_names():
"model": "claude-sonnet-4-6",
"routing_source": "catalog",
"langfuse_trace_id": "1edf31f11b1693cc6103f358c1481694",
+ "trace_id": "turn-1",
}
],
)
@@ -1116,6 +1117,8 @@ async def test_batch_persist_maps_stamp_columns_to_prisma_names():
assert "routing_source" not in row
assert row["langfuseTraceId"] == "1edf31f11b1693cc6103f358c1481694"
assert "langfuse_trace_id" not in row
+ assert row["traceId"] == "turn-1"
+ assert "trace_id" not in row
def test_from_db_restores_stamp_columns():
@@ -1131,6 +1134,7 @@ def test_from_db_restores_stamp_columns():
model="claude-sonnet-4-6",
routingSource="catalog",
langfuseTraceId="1edf31f11b1693cc6103f358c1481694",
+ traceId="turn-1",
createdAt=datetime.now(UTC),
)
restored = ChatMessage.from_db(prisma_msg)
@@ -1139,6 +1143,8 @@ def test_from_db_restores_stamp_columns():
assert restored.langfuse_trace_id == "1edf31f11b1693cc6103f358c1481694"
# Internal: the trace id never reaches the session payload clients get.
assert "langfuse_trace_id" not in restored.model_dump()
+ assert restored.trace_id == "turn-1"
+ assert "trace_id" not in restored.model_dump()
def test_from_db_null_stamps_stay_null():
diff --git a/autogpt_platform/backend/backend/copilot/executor/processor.py b/autogpt_platform/backend/backend/copilot/executor/processor.py
index c464c227f6..0cbbde77a4 100644
--- a/autogpt_platform/backend/backend/copilot/executor/processor.py
+++ b/autogpt_platform/backend/backend/copilot/executor/processor.py
@@ -18,6 +18,7 @@ from typing import TYPE_CHECKING, Callable, cast
from autogpt_libs.auth.dependencies import ACCOUNT_BANNED_DETAIL
from backend.copilot import stream_registry
+from backend.copilot.ai_spans import set_turn_span, trace_id_for
from backend.copilot.baseline import stream_chat_completion_baseline
from backend.copilot.budget_gate import record_refusal, refusal_for
from backend.copilot.config import ChatConfig
@@ -856,7 +857,9 @@ class CoPilotProcessor:
envelope = taint_for_source_platform(entry.envelope, session)
set_turn_unattended(session, scheduled=entry.unattended)
set_turn_impersonated(entry.impersonated)
- set_turn_trace_id(entry.turn_id)
+ # A subtask's calls join the trace of the reply it works for.
+ set_turn_trace_id(trace_id_for(entry))
+ set_turn_span(entry.turn_id)
set_turn_credential_pins(entry.credential_pins)
turn.engine_started(engine)
raw_stream = stream_fn(
@@ -951,7 +954,9 @@ class CoPilotProcessor:
log.error(f"Failed to release chat credential: {release_err}")
finally:
# After the release, which can still fail the turn.
- turn.set_outcome(error_msg, stream_registry.CANCELLED_MESSAGE)
+ turn.set_outcome(
+ error_msg, stream_registry.CANCELLED_MESSAGE, error_code
+ )
if scheduled_watch is not None:
scheduled_watch.report(error_msg)
try:
diff --git a/autogpt_platform/backend/backend/copilot/executor/processor_test.py b/autogpt_platform/backend/backend/copilot/executor/processor_test.py
index 9eb653bbfc..2550832a7f 100644
--- a/autogpt_platform/backend/backend/copilot/executor/processor_test.py
+++ b/autogpt_platform/backend/backend/copilot/executor/processor_test.py
@@ -19,6 +19,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from pydantic import SecretStr
+from backend.copilot.ai_spans import current_span_id
from backend.copilot.engine import resolve_use_sdk
from backend.copilot.executor.processor import (
_CODEX_CREDENTIAL_ACQUIRE_TIMEOUT_SECONDS,
@@ -46,6 +47,7 @@ from backend.copilot.response_model import (
StreamTextDelta,
StreamToolInputAvailable,
)
+from backend.copilot.tree import TurnEnvelope
from backend.data.model import OAuth2Credentials
from backend.integrations.codex.transport import (
CodexCredentialBusyError,
@@ -238,6 +240,7 @@ class TestExecuteAsyncScheduledTurnContext:
seen["pins"] = dict(turn_credential_pins())
seen["impersonated"] = is_turn_impersonated()
seen["trace_id"] = turn_trace_id()
+ seen["span_id"] = current_span_id()
return MagicMock()
with (
@@ -285,6 +288,7 @@ class TestExecuteAsyncScheduledTurnContext:
"pins": {"exa": pin},
"impersonated": False,
"trace_id": "turn-1",
+ "span_id": "turn-1",
}
@pytest.mark.asyncio
@@ -296,6 +300,7 @@ class TestExecuteAsyncScheduledTurnContext:
"pins": {},
"impersonated": False,
"trace_id": "turn-1",
+ "span_id": "turn-1",
}
@pytest.mark.asyncio
@@ -315,6 +320,26 @@ class TestExecuteAsyncScheduledTurnContext:
assert seen["trace_id"] == "turn-42"
+ @pytest.mark.asyncio
+ async def test_a_subtask_joins_the_trace_of_the_reply_it_works_for(
+ self,
+ ) -> None:
+ """Its model and tool calls group with the reply at its tree's root,
+ and its tool calls nest under the subtask's own span."""
+ entry = _make_entry().model_copy(
+ update={
+ "turn_id": "child-turn",
+ "envelope": TurnEnvelope(
+ tree_id="root-turn", depth=1, ai_trace_id="root-turn"
+ ),
+ }
+ )
+
+ seen = await self._seen_by_the_turn(entry)
+
+ assert seen["trace_id"] == "root-turn"
+ assert seen["span_id"] == "child-turn"
+
class TestExecuteAsyncAclose:
"""``_execute_async`` must call ``aclose`` on the published stream both
diff --git a/autogpt_platform/backend/backend/copilot/executor/utils.py b/autogpt_platform/backend/backend/copilot/executor/utils.py
index 43b69ce576..4a92fd18fb 100644
--- a/autogpt_platform/backend/backend/copilot/executor/utils.py
+++ b/autogpt_platform/backend/backend/copilot/executor/utils.py
@@ -22,9 +22,11 @@ from backend.copilot.active_turns import (
get_inflight_turn_limit,
inflight_turn_limit_message,
)
+from backend.copilot.ai_spans import current_span_id
from backend.copilot.config import CopilotLlmAuthProvider, CopilotLLMModel
from backend.copilot.context import get_current_envelope
from backend.copilot.credential_selection import CredentialPins
+from backend.copilot.llm_attribution import turn_trace_id
from backend.copilot.permissions import CopilotPermissions
from backend.copilot.tree import (
SpawnRequest,
@@ -806,8 +808,15 @@ async def _admitted_turn_envelope(
if spawner is None:
envelope = root_envelope(turn_id, session_id=session_id)
else:
+ # Same rule as the Sentry trace: a child derived from the caller's
+ # context joins the caller's PostHog trace, under its running tool.
envelope = derive_child_envelope(
spawner, spawn or SpawnRequest(), spawner_permissions=permissions
+ ).model_copy(
+ update={
+ "ai_trace_id": turn_trace_id(),
+ "ai_parent_span_id": current_span_id(),
+ }
)
await admit_turn(envelope, user_id=user_id)
return envelope
diff --git a/autogpt_platform/backend/backend/copilot/executor/utils_tracing_test.py b/autogpt_platform/backend/backend/copilot/executor/utils_tracing_test.py
index 92b43aef37..6827291541 100644
--- a/autogpt_platform/backend/backend/copilot/executor/utils_tracing_test.py
+++ b/autogpt_platform/backend/backend/copilot/executor/utils_tracing_test.py
@@ -7,8 +7,10 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from backend.copilot import usage_activation_fixtures
+from backend.copilot.ai_spans import current_span_id, set_turn_span, tool_span
from backend.copilot.executor import utils
from backend.copilot.executor.utils import CoPilotExecutionEntry
+from backend.copilot.llm_attribution import set_turn_trace_id
from backend.copilot.tree import root_envelope
from backend.util import tracing
@@ -113,3 +115,86 @@ async def test_dispatch_continues_only_the_trace_of_the_turns_cause(
trace_headers = enqueue.await_args.kwargs["trace_headers"]
assert trace_headers == (None if continues_caller else {})
+
+
+ "root, in_a_turn, nests_under_the_caller",
+ [
+ (False, True, True),
+ (True, True, False),
+ (False, False, False),
+ ],
+ ids=["derived-from-caller", "minted-root", "outside-a-turn"],
+)
+async def test_a_spawned_turns_envelope_records_the_tool_call_that_started_it(
+ root: bool, in_a_turn: bool, nests_under_the_caller: bool
+) -> None:
+ """Same rule as the Sentry trace: only a turn derived from the caller's
+ context is the caller's subtask in PostHog. The envelope keeps it, so a
+ queued child and an approval wake that start under it later keep it too."""
+ spawner = root_envelope("parent-turn") if in_a_turn else None
+ with (
+ patch.object(utils, "get_current_envelope", return_value=spawner),
+ patch.object(utils, "admit_turn", new=AsyncMock()),
+ ):
+ set_turn_trace_id("parent-turn")
+ set_turn_span("parent-turn")
+ try:
+ with tool_span("delegate_to_expert", user_id="u1", session_id="s0"):
+ spawning_tool = current_span_id()
+ envelope = await utils._admitted_turn_envelope(
+ "t1", "s1", "u1", None, None, root=root
+ )
+ finally:
+ set_turn_trace_id(None)
+ set_turn_span(None)
+
+ if nests_under_the_caller:
+ assert envelope.depth == 1
+ assert envelope.ai_trace_id == "parent-turn"
+ assert envelope.ai_parent_span_id == spawning_tool
+ else:
+ assert envelope.depth == 0
+ assert envelope.ai_trace_id is None
+ assert envelope.ai_parent_span_id is None
+
+
+async def test_a_re_admitted_turn_keeps_the_trace_its_envelope_recorded(
+ copilot_queue,
+) -> None:
+ """An approval wake starts under the envelope its call was held under, so
+ a woken subtask stays in the trace of the turn that held the call, not
+ the trace of whatever turn dispatches the wake."""
+ held = root_envelope("tree-a").model_copy(
+ update={
+ "depth": 1,
+ "ai_trace_id": "reply-b",
+ "ai_parent_span_id": "delegate-span",
+ }
+ )
+ with (
+ patch.object(utils, "readmit_turn", new=AsyncMock()),
+ patch("backend.copilot.stream_registry.create_session", new=AsyncMock()),
+ patch.object(utils, "capture_cost_context", new=AsyncMock(return_value=None)),
+ ):
+ set_turn_trace_id("unrelated-turn")
+ set_turn_span("unrelated-turn")
+ try:
+ await utils.dispatch_turn(
+ MagicMock(),
+ session_id="s1",
+ user_id="u1",
+ turn_id="woken-c",
+ message="hi",
+ envelope=held,
+ )
+ finally:
+ set_turn_trace_id(None)
+ set_turn_span(None)
+
+ envelope = _published_entry(copilot_queue.publish_message).envelope
+ assert envelope is not None
+ assert envelope.ai_trace_id == "reply-b"
+ assert envelope.ai_parent_span_id == "delegate-span"
diff --git a/autogpt_platform/backend/backend/copilot/feedback.py b/autogpt_platform/backend/backend/copilot/feedback.py
index e088473864..5084ff18d3 100644
--- a/autogpt_platform/backend/backend/copilot/feedback.py
+++ b/autogpt_platform/backend/backend/copilot/feedback.py
@@ -19,11 +19,12 @@ the user (distinct id = user id), for every user regardless of cookie
consent, like the generations OpenRouter Broadcast sends there.
``$ai_session_id`` is the chat session id, the same value the chat's
``$ai_generation`` events carry, which is what joins a rating to the
-generations of its chat; ``message_id`` names the reply. It carries no
-``$ai_trace_id`` yet: each generation's trace id is the turn that made it (see
-``llm_attribution``), but a reply row does not record which turn wrote it.
-Rating again sends another event; the latest per ``feedback_id`` is the
-rating that stands.
+generations of its chat; ``message_id`` names the reply. ``$ai_trace_id``
+is the trace of the turn that wrote the reply (``ChatMessage.traceId``, see
+``copilot.ai_spans``), the one its generations, tool spans and ``$ai_trace``
+carry, so the rating attaches to that reply's trace; a reply written before
+the column existed has none and attaches to the chat only. Rating again sends
+another event; the latest per ``feedback_id`` is the rating that stands.
"""
import logging
@@ -225,6 +226,8 @@ def _send_to_posthog(
"feedback_id": feedback_id,
"impersonated": impersonated,
}
+ if message.traceId:
+ properties["$ai_trace_id"] = message.traceId
if comment:
properties["$ai_feedback_text"] = comment
capture(user_id, PostHogEvent.AI_METRIC, properties)
diff --git a/autogpt_platform/backend/backend/copilot/feedback_test.py b/autogpt_platform/backend/backend/copilot/feedback_test.py
index 556363b4f8..4cf69de5be 100644
--- a/autogpt_platform/backend/backend/copilot/feedback_test.py
+++ b/autogpt_platform/backend/backend/copilot/feedback_test.py
@@ -128,6 +128,32 @@ async def test_impersonated_rating_is_marked(
assert capture.call_args.args[2]["impersonated"] is True
[email protected]("langfuse_score")
+async def test_a_rating_names_the_trace_of_the_turn_that_wrote_the_reply(
+ mocker: pytest_mock.MockerFixture, reply: PrismaChatMessage
+) -> None:
+ reply.traceId = "turn-1"
+ capture = mocker.patch.object(feedback, "capture")
+
+ await _rate()
+
+ assert capture.call_args.args[2]["$ai_trace_id"] == "turn-1"
+
+
[email protected]("reply", "langfuse_score")
+async def test_a_reply_with_no_recorded_trace_sends_no_trace_id(
+ mocker: pytest_mock.MockerFixture,
+) -> None:
+ """Replies written before the column existed attach to the chat only."""
+ capture = mocker.patch.object(feedback, "capture")
+
+ await _rate()
+
+ assert "$ai_trace_id" not in capture.call_args.args[2]
+
+
@pytest.mark.asyncio
@pytest.mark.usefixtures("reply")
async def test_posthog_failure_keeps_the_rating_and_the_langfuse_score(
diff --git a/autogpt_platform/backend/backend/copilot/llm_attribution.py b/autogpt_platform/backend/backend/copilot/llm_attribution.py
index 8503ecf6a7..20e1370c9f 100644
--- a/autogpt_platform/backend/backend/copilot/llm_attribution.py
+++ b/autogpt_platform/backend/backend/copilot/llm_attribution.py
@@ -8,9 +8,12 @@ OpenRouter Broadcast sends every generation to PostHog LLM analytics as an
the distinct id every other backend and browser event uses, so the
generation lands on the same person as their product events and replays.
- ``session_id`` becomes ``$ai_session_id``: the chat session id.
-- ``trace.trace_id`` becomes ``$ai_trace_id``: the id of the turn that made
- the call, so every model call behind one AutoPilot reply (the main loop,
- sub-agents, compaction, the chat title) groups into one trace.
+- ``trace.trace_id`` becomes ``$ai_trace_id``: the reply's trace, so every
+ model call behind one AutoPilot reply (the main loop, sub-agents,
+ compaction, the chat title, and the expert subtasks it spawns) groups into
+ one trace. It is the id of the turn that made the call, or for a subtask
+ the trace of the turn that spawned it (``ai_spans.trace_id_for``);
+ ``ai_spans`` sends the trace's tool spans and root from the backend.
- ``trace.generation_name`` becomes ``$ai_span_name``; every other ``trace``
key arrives as ``metadata_<key>`` (``metadata_environment``,
``metadata_impersonated``, and ``metadata_revision``: the deployed build's
@@ -43,7 +46,7 @@ GenerationName = Literal["copilot-title", "copilot-sdk", "copilot-baseline"]
# executor sets it for every turn, so a value never carries over.
_turn_impersonated: ContextVar[bool] = ContextVar("_turn_impersonated", default=False)
-# The running turn's id, sent as the trace id of every call it makes. The
+# The running turn's trace, sent as the trace id of every call it makes. The
# executor sets it for every turn, so a value never carries over; a call made
# outside a turn sends none.
_turn_trace_id: ContextVar[str | None] = ContextVar("_turn_trace_id", default=None)
@@ -85,9 +88,9 @@ def is_turn_impersonated() -> bool:
return _turn_impersonated.get()
-def set_turn_trace_id(turn_id: str | None) -> None:
- """Make *turn_id* the trace of every model call the running turn makes."""
- _turn_trace_id.set(turn_id or None)
+def set_turn_trace_id(trace_id: str | None) -> None:
+ """Make *trace_id* the trace of every model call the running turn makes."""
+ _turn_trace_id.set(trace_id or None)
def turn_trace_id() -> str | None:
diff --git a/autogpt_platform/backend/backend/copilot/model.py b/autogpt_platform/backend/backend/copilot/model.py
index 0f6d9561d4..f32e463c67 100644
--- a/autogpt_platform/backend/backend/copilot/model.py
+++ b/autogpt_platform/backend/backend/copilot/model.py
@@ -228,6 +228,11 @@ class ChatMessage(BaseModel):
# from payloads like routing_source, and persisted the same way.
langfuse_trace_id: str | None = Field(default=None, exclude=True)
+ # PostHog trace of the turn that wrote this assistant or reasoning row
+ # (``copilot.ai_spans``): the ``$ai_trace_id`` of its generations, which a
+ # rating of the reply carries. Internal and persisted like the above.
+ trace_id: str | None = Field(default=None, exclude=True)
+
stamps_pending_save: bool = Field(default=False, exclude=True)
"""True when model/routing_source were stamped after this row was already
persisted (mid-turn flush assigned its sequence before end-of-turn
@@ -296,6 +301,7 @@ class ChatMessage(BaseModel):
),
llm_credential_id=prisma_message.llmCredentialId,
langfuse_trace_id=prisma_message.langfuseTraceId,
+ trace_id=prisma_message.traceId,
)
@@ -1172,6 +1178,7 @@ async def _save_session_to_db(
"llm_auth_provider": msg.llm_auth_provider,
"llm_credential_id": msg.llm_credential_id,
"langfuse_trace_id": msg.langfuse_trace_id,
+ "trace_id": msg.trace_id,
"metadata": msg.metadata,
}
)
@@ -1226,6 +1233,7 @@ async def _save_session_to_db(
llm_auth_provider=msg.llm_auth_provider,
llm_credential_id=msg.llm_credential_id,
langfuse_trace_id=msg.langfuse_trace_id,
+ trace_id=msg.trace_id,
)
except Exception as e:
logger.error(
diff --git a/autogpt_platform/backend/backend/copilot/model_test.py b/autogpt_platform/backend/backend/copilot/model_test.py
index 4f16150bb7..7bbb37d5a9 100644
--- a/autogpt_platform/backend/backend/copilot/model_test.py
+++ b/autogpt_platform/backend/backend/copilot/model_test.py
@@ -2227,6 +2227,7 @@ async def test_save_session_to_db_backfills_stamps_on_flushed_rows(
llm_auth_provider=None,
llm_credential_id=None,
langfuse_trace_id=None,
+ trace_id=None,
)
assert flushed.stamps_pending_save is False
@@ -2235,8 +2236,9 @@ async def test_save_session_to_db_backfills_stamps_on_flushed_rows(
async def test_save_session_to_db_persists_the_turn_trace(
mocker: MockerFixture,
) -> None:
- """The Langfuse trace stamped at end of turn reaches the DB both on rows
- flushed mid-turn (back-fill) and on rows inserted by this save."""
+ """The Langfuse and PostHog traces stamped at end of turn reach the DB
+ both on rows flushed mid-turn (back-fill) and on rows inserted by this
+ save."""
trace_id = "1edf31f11b1693cc6103f358c1481694"
flushed = ChatMessage(
role="assistant",
@@ -2245,6 +2247,7 @@ async def test_save_session_to_db_persists_the_turn_trace(
model="claude-sonnet-4-6",
routing_source="env",
langfuse_trace_id=trace_id,
+ trace_id="turn-1",
stamps_pending_save=True,
)
unsaved = ChatMessage(
@@ -2253,6 +2256,7 @@ async def test_save_session_to_db_persists_the_turn_trace(
model="claude-sonnet-4-6",
routing_source="env",
langfuse_trace_id=trace_id,
+ trace_id="turn-1",
)
session = _make_session_with_messages(flushed, unsaved)
@@ -2269,8 +2273,10 @@ async def test_save_session_to_db_persists_the_turn_trace(
backfill = mock_db.update_chat_message_stamps.await_args
assert backfill.kwargs["sequence"] == 7
assert backfill.kwargs["langfuse_trace_id"] == trace_id
+ assert backfill.kwargs["trace_id"] == "turn-1"
(inserted,) = mock_db.add_chat_messages_batch.await_args.kwargs["messages"]
assert inserted["langfuse_trace_id"] == trace_id
+ assert inserted["trace_id"] == "turn-1"
@pytest.mark.asyncio(loop_scope="session")
diff --git a/autogpt_platform/backend/backend/copilot/sdk/security_hooks.py b/autogpt_platform/backend/backend/copilot/sdk/security_hooks.py
index 9365059aad..d5409f6268 100644
--- a/autogpt_platform/backend/backend/copilot/sdk/security_hooks.py
+++ b/autogpt_platform/backend/backend/copilot/sdk/security_hooks.py
@@ -12,6 +12,7 @@ from typing import Any, cast
from claude_agent_sdk.types import HookEvent, HookMatcher
+from backend.copilot.ai_spans import ToolSpanTimer
from backend.copilot.context import (
get_execution_context,
is_allowed_local_path,
@@ -212,6 +213,7 @@ def create_security_hooks(
on_compact: Callable[[str], None] | None = None,
tool_display_bridge: SDKToolDisplayBridge | None = None,
governor: TurnGovernor | None = None,
+ builtin_tool_spans: bool = True,
) -> dict[HookEvent, list[HookMatcher]]:
"""Create the security hooks configuration for Claude Agent SDK.
@@ -232,6 +234,8 @@ def create_security_hooks(
governor: The turn's spend and step governor. Once it is landing the
turn, every tool call -- the main agent's and its sub-agents' --
is refused with a reason that asks for a checkpoint.
+ builtin_tool_spans: Send a PostHog span for each call of the CLI's
+ built-in tools (``ChatConfig.posthog_builtin_tool_spans``).
Returns:
Hooks configuration dict for ClaudeAgentOptions
@@ -247,6 +251,24 @@ def create_security_hooks(
# as long as the sub-agent and PostToolUse releases the slot when it
# finishes.
subagent_tool_use_ids: set[str] = set()
+ # The CLI runs its built-in tools (Bash, Task, WebSearch, ...) itself,
+ # so they never pass through ``BaseTool``; their PostHog spans are
+ # timed from these hooks. Our MCP tools get theirs in ``BaseTool``.
+ builtin_spans = ToolSpanTimer(enabled=builtin_tool_spans)
+
+ def _finish_builtin_span(
+ tool_name: str, tool_use_id: str | None, *, is_error: bool
+ ) -> None:
+ if tool_use_id is None or tool_name.startswith(MCP_TOOL_PREFIX):
+ return
+ _, session = get_execution_context()
+ builtin_spans.finish(
+ tool_use_id,
+ tool_name,
+ user_id=user_id,
+ session_id=session.session_id if session is not None else None,
+ is_error=is_error,
+ )
async def pre_tool_use_hook(
input_data: HookInput,
@@ -304,6 +326,8 @@ def create_security_hooks(
# Reserve the sub-agent slot only after all validations pass
if tool_name in _SUBAGENT_TOOLS and tool_use_id is not None:
subagent_tool_use_ids.add(tool_use_id)
+ if not is_copilot_tool and tool_use_id is not None:
+ builtin_spans.start(tool_use_id, tool_name)
logger.debug(f"[SDK] Tool start: {tool_name}, user={user_id}")
if (
@@ -349,6 +373,7 @@ def create_security_hooks(
tool_name = cast(str, input_data.get("tool_name", ""))
_release_subagent_slot(tool_name, tool_use_id)
+ _finish_builtin_span(tool_name, tool_use_id, is_error=False)
is_builtin = not tool_name.startswith(MCP_TOOL_PREFIX)
safe_tool_use_id = _sanitize(str(tool_use_id or ""), max_len=12)
logger.info(
@@ -427,6 +452,7 @@ def create_security_hooks(
)
_release_subagent_slot(tool_name, tool_use_id)
+ _finish_builtin_span(tool_name, tool_use_id, is_error=True)
return cast(SyncHookJSONOutput, {})
diff --git a/autogpt_platform/backend/backend/copilot/sdk/security_hooks_test.py b/autogpt_platform/backend/backend/copilot/sdk/security_hooks_test.py
index 1290175c62..c04b744545 100644
--- a/autogpt_platform/backend/backend/copilot/sdk/security_hooks_test.py
+++ b/autogpt_platform/backend/backend/copilot/sdk/security_hooks_test.py
@@ -856,3 +856,143 @@ async def test_post_tool_use_no_session_skips_drain(monkeypatch):
assert drain_called is False
assert "hookSpecificOutput" not in result
+
+
+# -- PostHog spans for the CLI's built-in tools -------------------------------
+
+
+def _spans(monkeypatch):
+ """A turn in progress, with PostHog's capture recorded."""
+ from unittest.mock import MagicMock
+
+ from backend.copilot import ai_spans
+ from backend.copilot import context as ctx_mod
+ from backend.copilot.llm_attribution import set_turn_trace_id
+
+ async def no_drain(_session_id: str):
+ return []
+
+ monkeypatch.setattr(
+ "backend.copilot.pending_messages.drain_pending_messages", no_drain
+ )
+ session = MagicMock()
+ session.session_id = "sess-spans"
+ ctx_mod.set_execution_context(
+ user_id="u1", session=session, sandbox=None, sdk_cwd=SDK_CWD
+ )
+ set_turn_trace_id("turn-1")
+ ai_spans.set_turn_span("turn-1")
+ capture = MagicMock()
+ monkeypatch.setattr(ai_spans.posthog_client, "capture", capture)
+ yield capture
+ set_turn_trace_id(None)
+ ai_spans.set_turn_span(None)
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_a_builtin_tool_call_is_a_posthog_span(_hooks, _spans):
+ pre, post, _ = _hooks
+ task = {"prompt": "the user's private request"}
+
+ await pre({"tool_name": "Task", "tool_input": task}, "tu-1", {})
+ await post(
+ {"tool_name": "Task", "tool_input": task, "tool_response": "results"},
+ tool_use_id="tu-1",
+ context={},
+ )
+
+ _spans.assert_called_once()
+ distinct_id, event, properties = _spans.call_args.args
+ assert (distinct_id, event) == ("u1", "$ai_span")
+ assert properties["$ai_span_name"] == "Task"
+ assert properties["$ai_trace_id"] == "turn-1"
+ assert properties["$ai_session_id"] == "sess-spans"
+ assert properties["$ai_is_error"] is False
+ assert "private" not in repr(properties)
+ assert "results" not in repr(properties)
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_a_failed_builtin_tool_is_an_error_span(_hooks, _spans):
+ pre, _, post_failure = _hooks
+
+ await pre({"tool_name": "Task", "tool_input": {"prompt": "x"}}, "tu-1", {})
+ await post_failure(
+ {"tool_name": "Task", "error": "the page said something private"},
+ tool_use_id="tu-1",
+ context={},
+ )
+
+ properties = _spans.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert "private" not in repr(properties)
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_a_refused_builtin_tool_sends_no_span(_hooks, _spans):
+ pre, _, post_failure = _hooks
+
+ result = await pre(
+ {"tool_name": "Read", "tool_input": {"file_path": "/etc/passwd"}}, "tu-1", {}
+ )
+ await post_failure({"tool_name": "Read", "error": "denied"}, "tu-1", {})
+
+ assert _is_denied(result)
+ _spans.assert_not_called()
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_a_platform_tool_gets_its_span_from_basetool_not_the_hooks(
+ _hooks, _spans
+):
+ pre, post, _ = _hooks
+
+ await pre({"tool_name": "mcp__copilot__run_block", "tool_input": {}}, "tu-1", {})
+ await post(
+ {"tool_name": "mcp__copilot__run_block", "tool_response": "ok"}, "tu-1", {}
+ )
+
+ _spans.assert_not_called()
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_a_high_volume_builtin_tool_sends_no_span(_hooks, _spans):
+ """The CLI's checklist and file lookups run tens of times a turn."""
+ pre, post, _ = _hooks
+ todos = {"todos": [{"content": "x", "status": "pending", "activeForm": "x"}]}
+
+ result = await pre({"tool_name": "TodoWrite", "tool_input": todos}, "tu-1", {})
+ await post(
+ {"tool_name": "TodoWrite", "tool_input": todos, "tool_response": "ok"},
+ tool_use_id="tu-1",
+ context={},
+ )
+
+ assert not _is_denied(result)
+ _spans.assert_not_called()
+
+
[email protected](not _sdk_available(), reason="claude_agent_sdk not installed")
+async def test_builtin_tool_spans_turned_off_send_none(_spans):
+ hooks = create_security_hooks(
+ user_id="u1", sdk_cwd=SDK_CWD, max_subtasks=2, builtin_tool_spans=False
+ )
+ pre = hooks["PreToolUse"][0].hooks[0]
+ post = hooks["PostToolUse"][0].hooks[0]
+ task = {"prompt": "x"}
+
+ await pre({"tool_name": "Task", "tool_input": task}, "tu-1", {})
+ await post(
+ {"tool_name": "Task", "tool_input": task, "tool_response": "results"},
+ tool_use_id="tu-1",
+ context={},
+ )
+
+ _spans.assert_not_called()
diff --git a/autogpt_platform/backend/backend/copilot/sdk/service.py b/autogpt_platform/backend/backend/copilot/sdk/service.py
index 19347e86ee..0d1101fdc5 100644
--- a/autogpt_platform/backend/backend/copilot/sdk/service.py
+++ b/autogpt_platform/backend/backend/copilot/sdk/service.py
@@ -52,6 +52,7 @@ from backend.copilot.model_router import (
resolve_codex_model_route,
resolve_model_route,
)
+from backend.copilot.ai_spans import stamp_turn_trace_id
from backend.copilot.budget_gate import gate_auto_continue
from backend.copilot.budget_signal import build_turn_budget_block
from backend.copilot.continuation import as_model_prompt
@@ -5314,6 +5315,7 @@ async def stream_chat_completion_sdk( # pyright: ignore[reportGeneralTypeIssues
on_compact=compaction.on_compact,
tool_display_bridge=tool_display_bridge,
governor=governor,
+ builtin_tool_spans=config.posthog_builtin_tool_spans,
)
if permissions is not None:
@@ -6530,6 +6532,7 @@ async def stream_chat_completion_sdk( # pyright: ignore[reportGeneralTypeIssues
start_index=pre_turn_message_count,
trace_id=langfuse_trace_id,
)
+ stamp_turn_trace_id(session.messages, pre_turn_message_count)
# What this turn ran on, recorded on the turn rather than read
# back off the session later, so a route change cannot rewrite it.
stamp_segment(
diff --git a/autogpt_platform/backend/backend/copilot/sdk/service_helpers_test.py b/autogpt_platform/backend/backend/copilot/sdk/service_helpers_test.py
index e3f8e1e2a1..5481a04d90 100644
--- a/autogpt_platform/backend/backend/copilot/sdk/service_helpers_test.py
+++ b/autogpt_platform/backend/backend/copilot/sdk/service_helpers_test.py
@@ -2646,3 +2646,21 @@ class TestCompressionFailureIsReportedAsADrop:
assert stats.dropped is True
assert stats.messages_before == 3
assert stats.tokens_after is None
+
+
+class TestStampPostHogTrace:
+ def test_the_turns_posthog_trace_is_stamped_before_the_end_of_turn_save(self):
+ """A rating of the reply names its PostHog trace from this stamp
+ (``ai_spans.stamp_turn_trace_id``). The turn handler is too large to
+ drive in a unit test, so pin at the source that the stamp runs before
+ the save that persists it."""
+ import inspect
+
+ from backend.copilot.sdk.service import stream_chat_completion_sdk
+
+ src = inspect.getsource(stream_chat_completion_sdk)
+ stamped_at = src.index(
+ "stamp_turn_trace_id(session.messages, pre_turn_message_count)"
+ )
+ saved_at = src.index("await asyncio.shield(upsert_chat_session(session))")
+ assert stamped_at < saved_at
diff --git a/autogpt_platform/backend/backend/copilot/tools/base.py b/autogpt_platform/backend/backend/copilot/tools/base.py
index 69abda3ad8..616f71bbe5 100644
--- a/autogpt_platform/backend/backend/copilot/tools/base.py
+++ b/autogpt_platform/backend/backend/copilot/tools/base.py
@@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Any
from openai.types.chat import ChatCompletionToolParam
+from backend.copilot.ai_spans import mark_tool_failed, mark_tool_raised, tool_span
from backend.copilot.context import get_current_envelope
from backend.copilot.model import ChatSession
from backend.copilot.response_model import StreamToolOutputAvailable
@@ -22,6 +23,7 @@ from .models import (
ApprovalRequiredResponse,
ErrorResponse,
NeedLoginResponse,
+ ResponseType,
ToolResponseBase,
)
@@ -32,6 +34,12 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
+# Outputs that stop a call to wait on the user (an approval card, a sign-in),
+# which is not the tool failing.
+_AWAITING_THE_USER = frozenset(
+ {ResponseType.APPROVAL_REQUIRED.value, ResponseType.NEED_LOGIN.value}
+)
+
# Persist full tool output to workspace when it exceeds this threshold.
# Must be below _MAX_TOOL_OUTPUT_SIZE (100K) in response_model.py so we
# capture the data before model_post_init middle-out truncation discards it.
@@ -320,6 +328,16 @@ GATE_APPROVED = "_gate_approved"
GATE_DECIDED = "_gate_decided"
+def _awaits_the_user(output: StreamToolOutputAvailable) -> bool:
+ raw = output.output
+ if isinstance(raw, str):
+ try:
+ raw = json.loads(raw)
+ except ValueError:
+ return False
+ return isinstance(raw, dict) and raw.get("type") in _AWAITING_THE_USER
+
+
class BaseTool:
"""Base class for all chat tools."""
@@ -417,7 +435,9 @@ class BaseTool:
"""Execute the tool with authentication check.
Each call is its own Sentry ``copilot.tool`` transaction, named after
- the tool, in the trace of the turn that made it.
+ the tool, in the trace of the turn that made it, and a PostHog
+ ``$ai_span`` in the trace of the reply (``copilot.ai_spans``): its
+ name, latency and whether it failed, never its input or output.
Args:
user_id: User ID (None for anonymous users)
@@ -428,11 +448,21 @@ class BaseTool:
Pydantic response object
"""
- with tool_transaction(self.name, tool_call_id, session.session_id) as span:
+ with (
+ tool_transaction(self.name, tool_call_id, session.session_id) as span,
+ tool_span(
+ self.name,
+ user_id=user_id,
+ session_id=session.session_id,
+ tool_call_id=tool_call_id,
+ ) as outcome,
+ ):
output = await self._execute_checked(
user_id, session, tool_call_id, **kwargs
)
span.set_tag("copilot.tool.success", "true" if output.success else "false")
+ if not output.success and not _awaits_the_user(output):
+ outcome.is_error = True
return output
async def _execute_checked(
@@ -523,6 +553,8 @@ class BaseTool:
try:
result = await self._execute(user_id, session, **run_kwargs)
+ if result.type == ResponseType.ERROR:
+ mark_tool_failed()
if user_id:
await _record_activity(self, user_id, session, result, kwargs)
raw_output = full = result.model_dump_json(exclude_none=True)
@@ -546,6 +578,7 @@ class BaseTool:
except Exception as e:
logger.warning("Error in %s", self.name, exc_info=True)
tracing.set_status("internal_error")
+ mark_tool_raised(e)
return StreamToolOutputAvailable(
toolCallId=tool_call_id,
toolName=self.name,
diff --git a/autogpt_platform/backend/backend/copilot/tools/base_test.py b/autogpt_platform/backend/backend/copilot/tools/base_test.py
index f4e1e5354c..e2968e06a3 100644
--- a/autogpt_platform/backend/backend/copilot/tools/base_test.py
+++ b/autogpt_platform/backend/backend/copilot/tools/base_test.py
@@ -7,6 +7,8 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
+from backend.copilot import ai_spans
+from backend.copilot.llm_attribution import set_turn_trace_id
from backend.copilot.model import ChatSession
from backend.copilot.tools.base import (
_DIGEST_PREVIEW_CHARS,
@@ -709,3 +711,112 @@ class TestGateEnforcement:
return ErrorResponse(message="ran", session_id=session.session_id)
return _Recorder()
+
+
+class TestPostHogToolSpans:
+ """Every platform tool call, on either engine, is a PostHog ``$ai_span``
+ in the reply's trace, and none carries the tool's input or output."""
+
+ @staticmethod
+ def _tool(response: ToolResponseBase | None = None, error: bool = False):
+ class _Tool(BaseTool):
+ @property
+ def name(self) -> str:
+ return "gmail_read"
+
+ @property
+ def description(self) -> str:
+ return "reads mail"
+
+ @property
+ def parameters(self) -> dict:
+ return {"type": "object", "properties": {}}
+
+ async def _execute(self, user_id, session, **kwargs) -> ToolResponseBase:
+ if error:
+ raise RuntimeError("could not read the inbox")
+ assert response is not None
+ return response
+
+ return _Tool()
+
+ @pytest.fixture
+ def capture(self, mocker):
+ set_turn_trace_id("turn-1")
+ ai_spans.set_turn_span("turn-1")
+ yield mocker.patch.object(ai_spans.posthog_client, "capture")
+ set_turn_trace_id(None)
+ ai_spans.set_turn_span(None)
+
+ @pytest.mark.asyncio
+ async def test_a_tool_call_is_a_span_without_its_input_or_output(self, capture):
+ email_subject = "Subject: quarterly numbers"
+ tool = self._tool(
+ ToolResponseBase(type=ResponseType.NO_RESULTS, message=email_subject)
+ )
+
+ await tool.execute(
+ "u1", MagicMock(session_id="s1"), "call-1", query=email_subject
+ )
+
+ capture.assert_called_once()
+ _, event, properties = capture.call_args.args
+ assert event == "$ai_span"
+ assert properties["$ai_span_name"] == "gmail_read"
+ assert properties["$ai_trace_id"] == "turn-1"
+ assert properties["$ai_parent_id"] == "turn-1"
+ assert properties["$ai_session_id"] == "s1"
+ assert properties["$ai_is_error"] is False
+ assert email_subject not in repr(properties)
+
+ @pytest.mark.asyncio
+ async def test_a_tool_that_answers_with_an_error_failed(self, capture):
+ tool = self._tool(ErrorResponse(message="no such label", session_id="s1"))
+
+ await tool.execute("u1", MagicMock(session_id="s1"), "call-1")
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert "$ai_error" not in properties
+ assert "no such label" not in repr(properties)
+
+ @pytest.mark.asyncio
+ async def test_a_call_waiting_for_an_approval_did_not_fail(self, capture):
+ """An approval card is the user's decision to make, not the tool
+ failing, so it must not count against the tool."""
+ tool = self._tool(ToolResponseBase(type=ResponseType.NO_RESULTS, message=""))
+ session = MagicMock(session_id="s1")
+ refusal = tool._refusal("call-1", session, "Needs your approval")
+ with patch.object(tool, "_gate", new=AsyncMock(return_value=(refusal, False))):
+ result = await tool.execute("u1", session, "call-1")
+
+ assert result.success is False
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is False
+
+ @pytest.mark.asyncio
+ async def test_a_call_waiting_for_a_sign_in_did_not_fail(self, capture):
+ tool = self._tool(ToolResponseBase(type=ResponseType.NO_RESULTS, message=""))
+
+ with patch.object(
+ type(tool), "requires_auth", new_callable=lambda: property(lambda _: True)
+ ):
+ result = await tool.execute(None, MagicMock(session_id="s1"), "call-1")
+
+ assert result.success is False
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is False
+
+ @pytest.mark.asyncio
+ async def test_a_tool_that_raises_failed_with_its_type_not_its_message(
+ self, capture
+ ):
+ tool = self._tool(error=True)
+
+ result = await tool.execute("u1", MagicMock(session_id="s1"), "call-1")
+
+ assert result.success is False
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert properties["$ai_error"] == "RuntimeError"
+ assert "inbox" not in repr(properties)
diff --git a/autogpt_platform/backend/backend/copilot/tree.py b/autogpt_platform/backend/backend/copilot/tree.py
index 382d5e3a80..25720686f3 100644
--- a/autogpt_platform/backend/backend/copilot/tree.py
+++ b/autogpt_platform/backend/backend/copilot/tree.py
@@ -161,6 +161,12 @@ class TurnEnvelope(BaseModel):
deadline_at: datetime | None = None
# The chat whose spend ceiling this tree's paid calls count against.
spend_session_id: str | None = None
+ # Not limits: where a spawned turn sits in PostHog (``copilot.ai_spans``).
+ # The trace of the turn that spawned it and the span of the tool call that
+ # did, recorded when the child is derived. A queued child and an approval
+ # wake start under the stored envelope, so they keep both. None for a root.
+ ai_trace_id: str | None = None
+ ai_parent_span_id: str | None = None
def permits(self, tool_name: str) -> bool:
return self.tools is None or tool_name in self.tools
diff --git a/autogpt_platform/backend/backend/copilot/turn_tracing.py b/autogpt_platform/backend/backend/copilot/turn_tracing.py
index 59403413f8..4d6f4e79e9 100644
--- a/autogpt_platform/backend/backend/copilot/turn_tracing.py
+++ b/autogpt_platform/backend/backend/copilot/turn_tracing.py
@@ -5,6 +5,10 @@ continues the trace of whatever dispatched it: the chat request, a parent
turn's tool, a graph node. Each tool call the turn makes is a ``copilot.tool``
transaction of its own in that trace, so a long autopilot turn stays under the
SDK's span limit however many tools it runs.
+
+A turn that reached its engine also ends as an entry in PostHog's LLM
+analytics: the ``$ai_trace`` of a reply, or the ``$ai_span`` of a subtask in
+its reply's trace (see ``copilot.ai_spans``).
"""
import time
@@ -14,6 +18,7 @@ from typing import TYPE_CHECKING
from sentry_sdk.tracing import Span
+from backend.copilot import ai_spans
from backend.copilot.response_model import (
StreamBaseResponse,
StreamReasoningDelta,
@@ -42,9 +47,13 @@ class TurnTrace:
self._first_token_seen = False
self._chunks = 0
self._tool_calls = 0
+ self.engine: str | None = None
+ self.outcome = "completed"
+ self.error_code: str | None = None
def engine_started(self, engine: str) -> None:
"""Setup is done and the turn is handed to ``engine``."""
+ self.engine = engine
self.span.set_tag("copilot.engine", engine)
tracing.set_timing("copilot.turn.setup_ms", self._elapsed_ms(), span=self.span)
@@ -59,14 +68,22 @@ class TurnTrace:
"copilot.turn.ttft_ms", self._elapsed_ms(), span=self.span
)
- def set_outcome(self, error_message: str | None, cancelled_message: str) -> None:
- """Tag how the turn ended, given the error it reports to the user."""
+ def set_outcome(
+ self,
+ error_message: str | None,
+ cancelled_message: str,
+ error_code: str | None = None,
+ ) -> None:
+ """Tag how the turn ended, given the error it reports to the user
+ and the code it reports with it."""
if not error_message:
outcome, status = "completed", "ok"
elif error_message == cancelled_message:
outcome, status = "cancelled", "cancelled"
else:
outcome, status = "failed", "internal_error"
+ self.outcome = outcome
+ self.error_code = error_code
self.span.set_tag("copilot.turn.outcome", outcome)
tracing.set_status(status, span=self.span)
@@ -79,6 +96,21 @@ class TurnTrace:
"copilot.turn.tool_call_count", self._tool_calls, span=self.span
)
+ def report_to_posthog(self, entry: "CoPilotExecutionEntry") -> None:
+ """Send the turn's end to PostHog's LLM analytics. A turn that never
+ reached its engine made no model or tool calls, so it sends none."""
+ if self.engine is None:
+ return
+ ai_spans.capture_turn(
+ entry,
+ latency_seconds=self._elapsed_ms() / 1000,
+ outcome=self.outcome,
+ error_code=self.error_code,
+ engine=self.engine,
+ origin=_origin(entry),
+ tool_call_count=self._tool_calls,
+ )
+
def _elapsed_ms(self) -> float:
return (time.monotonic() - self._started_at) * 1000
@@ -117,6 +149,7 @@ def turn_transaction(entry: "CoPilotExecutionEntry") -> Iterator[TurnTrace]:
yield turn
finally:
turn.close()
+ turn.report_to_posthog(entry)
def _origin(entry: "CoPilotExecutionEntry") -> str:
diff --git a/autogpt_platform/backend/backend/copilot/turn_tracing_test.py b/autogpt_platform/backend/backend/copilot/turn_tracing_test.py
new file mode 100644
index 0000000000..07596fbe64
--- /dev/null
+++ b/autogpt_platform/backend/backend/copilot/turn_tracing_test.py
@@ -0,0 +1,77 @@
+"""Tests for ``copilot.turn_tracing``: how a turn's end reaches PostHog."""
+
+from unittest.mock import MagicMock
+
+import pytest
+import pytest_mock
+
+from backend.copilot import ai_spans
+from backend.copilot.executor.utils import CoPilotExecutionEntry
+from backend.copilot.response_model import StreamToolInputAvailable
+from backend.copilot.tree import TurnEnvelope
+from backend.copilot.turn_tracing import turn_transaction
+from backend.util.posthog_events import PostHogEvent
+
+CANCELLED = "Operation cancelled"
+
+
+def capture(mocker: pytest_mock.MockerFixture) -> MagicMock:
+ return mocker.patch.object(ai_spans.posthog_client, "capture")
+
+
+def _entry() -> CoPilotExecutionEntry:
+ return CoPilotExecutionEntry(
+ session_id="sess-1",
+ turn_id="turn-1",
+ user_id="user-1",
+ message="hi",
+ envelope=TurnEnvelope(tree_id="turn-1"),
+ )
+
+
+def test_a_turn_that_ran_ends_as_its_replys_trace(capture: MagicMock):
+ with turn_transaction(_entry()) as turn:
+ turn.engine_started("baseline")
+ turn.observe(
+ StreamToolInputAvailable(toolCallId="c1", toolName="run_block", input={})
+ )
+ turn.set_outcome(None, CANCELLED)
+
+ capture.assert_called_once()
+ distinct_id, event, properties = capture.call_args.args
+ assert (distinct_id, event) == ("user-1", PostHogEvent.AI_TRACE)
+ assert properties["$ai_trace_id"] == "turn-1"
+ assert properties["engine"] == "baseline"
+ assert properties["outcome"] == "completed"
+ assert properties["tool_call_count"] == 1
+ assert properties["turn_origin"] == "chat"
+ assert properties["$ai_latency"] >= 0
+
+
+def test_a_failed_turn_carries_its_error_code_not_its_message(capture: MagicMock):
+ with turn_transaction(_entry()) as turn:
+ turn.engine_started("sdk")
+ turn.set_outcome("the user's private text", CANCELLED, "sdk_stream_error")
+
+ properties = capture.call_args.args[2]
+ assert properties["$ai_is_error"] is True
+ assert properties["$ai_error"] == "sdk_stream_error"
+ assert "private" not in repr(properties)
+
+
+def test_a_turn_that_raised_still_ends_its_trace(capture: MagicMock):
+ with pytest.raises(RuntimeError), turn_transaction(_entry()) as turn:
+ turn.engine_started("sdk")
+ turn.set_outcome("boom", CANCELLED)
+ raise RuntimeError("boom")
+
+ assert capture.call_args.args[2]["outcome"] == "failed"
+
+
+def test_a_turn_that_never_reached_its_engine_sends_nothing(capture: MagicMock):
+ """A refused or misrouted turn made no model or tool calls."""
+ with turn_transaction(_entry()) as turn:
+ turn.set_outcome("copilot_session_not_found", CANCELLED)
+
+ capture.assert_not_called()
diff --git a/autogpt_platform/backend/backend/util/posthog_events.py b/autogpt_platform/backend/backend/util/posthog_events.py
index 663747ab2a..5900a20748 100644
--- a/autogpt_platform/backend/backend/util/posthog_events.py
+++ b/autogpt_platform/backend/backend/util/posthog_events.py
@@ -73,6 +73,10 @@ class PostHogEvent(StrEnum):
# event for a score on a generative AI trace, named by PostHog rather
# than the analytics plan; a thumbs up/down or copy of an AutoPilot reply.
AI_METRIC = "$ai_metric"
+ # backend/copilot/ai_spans.py (SECRT-2979): an AutoPilot reply's trace
+ # root, and a tool call or expert subtask within it.
+ AI_TRACE = "$ai_trace"
+ AI_SPAN = "$ai_span"
# Key moments (SECRT-2723): backend/util/product_analytics.py
SIGNUP_COMPLETED = "signup_completed"
diff --git a/autogpt_platform/backend/backend/util/posthog_events_test.py b/autogpt_platform/backend/backend/util/posthog_events_test.py
index 918fb0baab..5bcf395b40 100644
--- a/autogpt_platform/backend/backend/util/posthog_events_test.py
+++ b/autogpt_platform/backend/backend/util/posthog_events_test.py
@@ -43,6 +43,8 @@ LIVE_EVENT_NAMES = {
"payment_failed",
"$set",
"$ai_metric",
+ "$ai_trace",
+ "$ai_span",
"signup_completed",
"onboarding_completed",
"checkout_started",
diff --git a/autogpt_platform/backend/migrations/20261019120000_add_chat_message_trace_id/migration.sql b/autogpt_platform/backend/migrations/20261019120000_add_chat_message_trace_id/migration.sql
new file mode 100644
index 0000000000..d05538c41c
--- /dev/null
+++ b/autogpt_platform/backend/migrations/20261019120000_add_chat_message_trace_id/migration.sql
@@ -0,0 +1,6 @@
+-- The PostHog trace of the turn that wrote each assistant or reasoning row,
+-- so a rating of the reply can name the trace in PostHog LLM analytics.
+
+-- Nullable with no default: a catalog-only change, no table rewrite.
+-- AlterTable
+ALTER TABLE "ChatMessage" ADD COLUMN "traceId" TEXT;
diff --git a/autogpt_platform/backend/schema.prisma b/autogpt_platform/backend/schema.prisma
index 1b87a61bb0..ca3d7b5d31 100644
--- a/autogpt_platform/backend/schema.prisma
+++ b/autogpt_platform/backend/schema.prisma
@@ -825,6 +825,12 @@ model ChatMessage {
// predating this column.
langfuseTraceId String?
+ // PostHog trace of the turn that wrote this assistant or reasoning row:
+ // the `$ai_trace_id` its generations and spans carry, so a rating of the
+ // reply can name the trace. Null on user/tool rows, on rows written
+ // outside an executor turn and on rows predating this column.
+ traceId String?
+
// Per-row JSONB metadata bag. Currently holds the dispatcher's
// submit-time payload (file_ids, mode, model, permissions, context,
// request_arrival_at) on the user row that triggered a queued turn,
diff --git a/docs/engineering/tracking-plan.md b/docs/engineering/tracking-plan.md
index db992cee10..d636bfc150 100644
--- a/docs/engineering/tracking-plan.md
+++ b/docs/engineering/tracking-plan.md
@@ -67,9 +67,9 @@ last, so a caller cannot overwrite them. `revision` is a super property of
every backend PostHog client instead (`revision_properties()`), so the flag
client's `$feature_flag_called` carries it too. AutoPilot's `$ai_generation`
is the exception: OpenRouter builds it from the request rather than through
-our client, so it carries no `revision` until #15604 adds `trace.revision`,
-which arrives as `metadata_revision`. #15605's `$ai_trace` and `$ai_span` go
-through `capture()`, so they carry `revision` as soon as they ship. The
+our client, so it carries no `revision`; the build arrives as
+`metadata_revision` from `trace.revision` instead. AutoPilot's `$ai_trace` and
+`$ai_span` go through `capture()`, so they carry `revision`. The
browser registers all three once as PostHog super properties
(`src/services/analytics/posthog-base-properties.ts`), again after `reset()`
at logout; a Vercel preview reports `environment: preview`, while the dev
@@ -308,15 +308,22 @@ Return visits are `$pageview`; account-level retention is computed in
## LLM analytics
-AutoPilot's LLM calls and the ratings of its replies, in PostHog's LLM
-analytics (SECRT-2730). Both are sent server-side for every user, whatever
-their cookie consent, like every backend event. The event names are
-PostHog's own, not the analytics plan's.
+AutoPilot's LLM calls, tool calls and the ratings of its replies, in
+PostHog's LLM analytics (SECRT-2730, SECRT-2979). All are sent server-side
+for every user, whatever their cookie consent, like every backend event. The
+event names are PostHog's own, not the analytics plan's.
+
+Each AutoPilot reply is one trace, whose id is the turn id of the reply
+(`$ai_trace_id`). An expert subtask or sub-session the reply spawns joins the
+same trace. No event carries a tool's input or output, or an error message:
+they can hold Gmail or Drive content (Google Limited Use).
| Event | Sender | Status | Required properties | Fires when |
| --- | --- | --- | --- | --- |
-| `$ai_generation` | OpenRouter Broadcast | live | set by OpenRouter from the request (`backend/copilot/llm_attribution.py`): `distinct_id` = the request's `user` (the platform user id), `$ai_session_id` = its `session_id` (the chat session id), `$ai_span_name` = `trace.generation_name` (`copilot-title`, `copilot-sdk`, `copilot-baseline`), `$ai_trace_id` = `trace.trace_id` (the id of the executor turn that made the call, so all calls behind one reply share it; absent for a call made outside a turn), `metadata_environment` (`local`, `dev`, `prod`), `metadata_impersonated`; `metadata_revision` (the deployed build's git sha, `config.sentry_release`) when it is set; model, tokens, cost and latency from OpenRouter | An AutoPilot LLM call goes through OpenRouter: a session title, an SDK turn (main agent, sub-agents and the CLI's compaction, via `CLAUDE_CODE_EXTRA_BODY`) or a baseline turn. |
-| `$ai_metric` | backend | live | `$ai_metric_name` (`user-feedback`, `copy`), `$ai_metric_value` (`1` thumbs up or copy, `0` thumbs down), `$ai_session_id` and `chat_session_id` (the chat session id), `message_id` (the reply's row id), `message_sequence`, `feedback_id` (the stored rating, shared with its Langfuse score), `impersonated`; `$ai_feedback_text` when a thumbs-down has a comment. No `$ai_trace_id` (see below) | A reply is rated (`POST /api/chat/sessions/{id}/feedback`, `copilot/feedback.py`). Rating a reply again sends another event; the latest per `feedback_id` stands. |
+| `$ai_generation` | OpenRouter Broadcast | live | set by OpenRouter from the request (`backend/copilot/llm_attribution.py`): `distinct_id` = the request's `user` (the platform user id), `$ai_session_id` = its `session_id` (the chat session id), `$ai_span_name` = `trace.generation_name` (`copilot-title`, `copilot-sdk`, `copilot-baseline`), `$ai_trace_id` = `trace.trace_id` (the reply's trace: the id of the executor turn that made the call, or for a subtask the trace of the turn that spawned it, so all calls behind one reply share it; absent for a call made outside a turn), `metadata_environment` (`local`, `dev`, `prod`), `metadata_impersonated`; `metadata_revision` (the deployed build's git sha, `config.sentry_release`) when it is set; model, tokens, cost and latency from OpenRouter | An AutoPilot LLM call goes through OpenRouter: a session title, an SDK turn (main agent, sub-agents and the CLI's compaction, via `CLAUDE_CODE_EXTRA_BODY`) or a baseline turn. |
+| `$ai_metric` | backend | live | `$ai_metric_name` (`user-feedback`, `copy`), `$ai_metric_value` (`1` thumbs up or copy, `0` thumbs down), `$ai_session_id` and `chat_session_id` (the chat session id), `message_id` (the reply's row id), `message_sequence`, `feedback_id` (the stored rating, shared with its Langfuse score), `impersonated`; `$ai_trace_id` when the reply row records its trace (`ChatMessage.traceId`, replies written since SECRT-2979); `$ai_feedback_text` when a thumbs-down has a comment | A reply is rated (`POST /api/chat/sessions/{id}/feedback`, `copilot/feedback.py`). Rating a reply again sends another event; the latest per `feedback_id` stands. |
+| `$ai_trace` | backend | live | `$ai_trace_id` (the turn id), `$ai_span_name` = `copilot-turn`, `$ai_session_id` and `chat_session_id`, `$ai_latency` (seconds), `$ai_is_error`, `$ai_error` (the error code, when failed), `turn_id`, `outcome` (`completed`, `cancelled`, `failed`), `engine` (`sdk`, `baseline`, ...), `turn_origin` (`chat`, `scheduled`, `system`; `spawned` on a subtask's span), `tool_call_count`, `impersonated` | A reply's turn ends, if it reached its engine (`copilot/turn_tracing.py`, `copilot/ai_spans.py`). A redelivered turn sends it again under the same event uuid (dedup key = the turn id), so PostHog keeps one; likewise a subtask's span. |
+| `$ai_span` | backend | live | `$ai_trace_id`, `$ai_span_id`, `$ai_parent_id`, `$ai_span_name`, `$ai_session_id` and `chat_session_id`, `$ai_latency` (seconds), `$ai_is_error`, `impersonated`. A **tool call**: `$ai_span_name` = the tool's name, `tool_call_id`, `cancelled` (platform tools) or `builtin` (the SDK CLI's own tools, sub-agents' calls included; the high-volume lookups `TodoWrite`, `Read`, `Glob`, `Grep`, `LS` and `NotebookRead` get no span, and no built-in gets one with `CHAT_POSTHOG_BUILTIN_TOOL_SPANS=false`), `$ai_is_error` true when it raised, a gate refused it or it answered with an error, but not when it stopped for an approval card or a sign-in, `$ai_error` = the exception's type name when a platform tool raised (never its message); its parent is the turn (the trace root, or the subtask's span) or the tool that called it. A built-in tool's span is parented to the turn, not to the `Task` span of the sub-agent that called it. A **subtask**: `$ai_span_name` = `copilot-subtask`, `$ai_span_id` = its turn id, parent = the spawning tool's span, which its envelope keeps, so a child that waited in the turn queue or a subtask woken by an approval stays in the trace and under the span its spawner gave it (the trace root for one queued or held before this was recorded), `depth`, plus the `$ai_trace` turn properties, with `turn_origin` = `spawned` (or `scheduled` when the subtask's own turn runs unattended) | A platform tool call ends (`BaseTool.execute`, both engines), an SDK built-in tool call ends (Pre/PostToolUse hooks), or a spawned turn ends. |
What joins them:
@@ -325,20 +332,18 @@ What joins them:
both land on the person with their product events and replays.
- **Chat.** `$ai_session_id` is the chat session id on both, so a chat's
generations and its ratings group in the Sessions tab and join on it.
- `message_id` says which reply was rated. A rating carries no
- `$ai_trace_id` yet: a generation's trace id is the turn that made it, but a
- reply row does not record which turn wrote it, so a rating joins its
- reply's generations only through the chat.
-- **Reply.** All generations behind one reply (the main loop, sub-agents,
- compaction and, on a chat's first reply, the title) share `$ai_trace_id`,
- the executor turn id, so PostHog groups them into one trace. An expert
- subtask runs as a turn of its own and gets its own trace.
+ `message_id` says which reply was rated.
+- **Reply.** `$ai_trace_id` is the reply's turn id on its generations, its
+ spans, its `$ai_trace` and its ratings, so they form one trace.
+ Generations sit at the trace's top level: Broadcast has no field for a
+ generation's parent span, so a subtask's model calls are not nested under
+ its span.
- **Environment.** OpenRouter forwards our `trace` keys as `metadata_<key>`,
so a generation's environment is `metadata_environment`, not the
- `environment` base property; `$ai_metric` has both base properties as usual.
- Likewise the build is `metadata_revision` on a generation; `$ai_metric`
- sets no build property of its own and carries only the base properties our
- PostHog client adds to every event.
+ `environment` base property; `$ai_metric`, `$ai_trace` and `$ai_span` have
+ both base properties as usual. Likewise the build is `metadata_revision` on
+ a generation; the backend's events set no build property of their own and
+ carry only the base properties our PostHog client adds to every event.
- **Impersonation.** A turn or rating an admin made while impersonating the
user (`X-Act-As-User-Id`) is still sent, as the user, with
`metadata_impersonated` / `impersonated` true, so it can be filtered out.