CubeLoop — a Pythonic, async-native agent harness core designed for high performance, readability, and production-grade persistence.
Python agent-frameworkagent-harnessagent-runtimeagentic-aiai-agentsai-frameworkanthropicharnessharness-engineeringharness-frameworklanggraphlanggraph-alternativellmmcpopenaipi-agentpi-agent-corepythonpython-agent
fix(tracing): stamp cubepi.run_id from agent active_run_id
xf gong0c1ddb77ee50
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1cdc2d3..439f57d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Changed++- **`cubepi.run_id` now follows the agent business run id.** On+ `AgentStart`, the recorder stamps `agent.state.active_run_id` (the same+ string as `prompt(run_id=…)` / `Message.run_id`) onto every span in the+ activation. A tracer-private uuid is minted only as a fallback when+ `active_run_id` is unset (e.g. oneshot still mints its own session id).+ Hosts can filter traces with the same id they use for SSE, messages, and+ billing. OTel `trace_id` / `span_id` remain the tree identity; JSONL still+ shards by `trace_id`.+### Fixed- **Durable HITL pauses now finalize traces as suspended rather than aborted.**diff --git a/cubepi/tracing/recorder.py b/cubepi/tracing/recorder.pyindex 280213c..dfd964b 100644--- a/cubepi/tracing/recorder.py+++ b/cubepi/tracing/recorder.py@@ -190,7 +190,8 @@ class Recorder:"""Subscribe to agent + provider events and produce OTel spans.Lifetime: one :class:`Recorder` per :meth:`Tracer.attach` call. The- recorder maintains per-run state keyed by a generated run_id; one+ recorder maintains per-run state keyed by the business run_id+ (``agent.state.active_run_id`` when set, else a fallback uuid); oneagent can host many sequential runs over the recorder's lifetime."""@@ -572,7 +573,18 @@ class Recorder:self._close_open_spans(self._run)self._sweep_tool_span_tokens(self._run)- run_id = str(uuid.uuid4())+ # Prefer the agent's business run id (prompt/resume/respond set+ # active_run_id before AgentStartEvent). Same string as+ # Message.run_id / host SSE ids — not a second tracer-private uuid.+ # Fallback only if active_run_id is unset (defensive / odd attach).+ agent_run_id: str | None = None+ agent = self._agent+ if agent is not None:+ try:+ agent_run_id = getattr(agent.state, "active_run_id", None)+ except Exception: # pragma: no cover — never break tracing+ agent_run_id = None+ run_id = agent_run_id if agent_run_id else str(uuid.uuid4())# Open the root invoke_agent span. Caller-context propagation# (parent_trace_id / parent_span_id from a host service) lands# here in a future run_scope feature.diff --git a/dev/specs/2026-05-18-cubepi-tracing-design.md b/dev/specs/2026-05-18-cubepi-tracing-design.mdindex 2e9638e..8fa9ab6 100644--- a/dev/specs/2026-05-18-cubepi-tracing-design.md+++ b/dev/specs/2026-05-18-cubepi-tracing-design.md@@ -585,7 +585,7 @@ Notation: **R** = Required, **CR** = Conditionally Required, **Rec** = Recommend| `gen_ai.conversation.id` | string | CR | `thread_id` || `gen_ai.request.model` | string | CR | model id of first chat call (if pre-known) || `error.type` | string | CR (on error) | see §12.3 |-| `cubepi.run_id` | string | **R** | uuid generated at root open |+| `cubepi.run_id` | string | **R** | agent `active_run_id` when set (host/`prompt` business run id); else uuid fallback at root open || `cubepi.thread_id` | string | Rec | same as `gen_ai.conversation.id`, duplicated for `cubepi.*` namespace ergonomics || `cubepi.agent.tools` | string[] | Opt | tool names registered || `cubepi.agent.system_prompt.sha256` | string | Opt | first 16 hex of `sha256(system_prompt)` |diff --git a/dev/specs/2026-08-10-unify-trace-run-id.md b/dev/specs/2026-08-10-unify-trace-run-id.mdnew file mode 100644index 0000000..9d9c4a7--- /dev/null+++ b/dev/specs/2026-08-10-unify-trace-run-id.md@@ -0,0 +1,214 @@+# Unify Trace `cubepi.run_id` with Business Run ID++- Date: 2026-08-10+- Status: Implemented (pending release)+- Repos: cubepi (primary), cubeplex (consumer; almost no code change)+- Related: `dev/specs/2026-05-18-cubepi-tracing-design.md` §10.1,+ `dev/specs/2026-08-04-tracing-hitl-suspension.md` (HITL: one business run,+ multiple `invoke_agent` activations)++## Goal++One **run** concept end-to-end.++The string that hosts put on `agent.prompt(run_id=...)`, that lands on+`Message.run_id` / `cubepi_messages.run_id`, and that products use for SSE,+cancel, steer, and billing **is the same string** stamped on every span as+`cubepi.run_id`.++There is no second, tracer-private "trace run id".++## Context++Today two UUIDs coexist for one agent activation:++| Layer | Who mints | Example |+|---|---|---|+| Business / agent | host (`uuid7`) or `prompt` fallback (`uuid4().hex`) | cubeplex stream path, message ledger |+| Tracing | recorder always `str(uuid.uuid4())` at `AgentStart` | span attr `cubepi.run_id`, stream log name |++That split was never a deliberate product rule that "business run ≠ trace run".+History:++1. **2026-05-18** tracing Phase 1 defined `cubepi.run_id` as "uuid generated at+ root open" so JSONL could shard by run.+2. **2026-05-19** PR #92 protected that attribute from being *clobbered by+ user metadata* (namespace → `cubepi.metadata.*`) — not a ban on adopting+ the agent run id.+3. **2026-06-06** agent gained `prompt(run_id=...)` / `active_run_id` for the+ message ledger, fork, HITL. Tracing was never updated to read it.+4. JSONL later **re-sharded by OTel `trace_id`**, so the original reason for a+ tracer-minted run id (file path) is already gone. Comments on the exporter+ still describe parent + subagent as same `trace_id`, different+ `cubepi.run_id`.++Symptom hosts hit in the wild: UI/API `run_id` cannot find the matching+`cubepi trace` file; investigators rediscover the run via `conversation_id`.++## Concepts (normative)++```+Business run_id ──► messages / SSE / billing / cubepi.run_id (span attr)+OTel trace_id ──► span tree storage & CLI grouping (may hold multiple runs)+OTel span_id ──► one span node+```++- **Run** = one agent activation identity (`prompt` / `resume` / `respond`+ effective id, or oneshot session id). Product and tracing share it.+- **Trace** = OTel distributed tree. Not a run. Nesting may put several runs+ under one `trace_id`.+- **Span** = one node. Root `invoke_agent` starts a subtree; filtering "all+ spans of this run" uses the denormalized `cubepi.run_id` attribute (or a+ tree walk from the root `span_id`). The attribute must be the **business**+ id, not a second random UUID.++## Non-goals++- Continuing one OTel `trace_id` across HITL pause and resume (already out of+ scope per HITL suspension spec). Correlation across activations is the+ **shared business `run_id`** (and host metadata).+- Changing JSONL sharding (stays `trace_id`).+- Changing how hosts mint run ids (cubeplex keeps `uuid7()`).+- Building a UI; only the id contract.+- Removing the `cubepi.run_id` attribute name (keep it; change who fills it).++## Design++### 1. Recorder: prefer `agent.active_run_id`++On `AgentStartEvent` → `_on_agent_start`:++```text+if attached agent has active_run_id set:+ run_id = that value+else:+ run_id = str(uuid.uuid4()) # fallback only+stamp CUBEPI_RUN_ID = run_id on root and all children (unchanged)+```++Ordering already allows this: `Agent.prompt` / `resume` / `respond` set+`active_run_id` **before** the loop emits `AgentStartEvent`. Recorder holds+`self._agent` from `attach`.++No change to the PR #92 rule: user `tracing_context(metadata=...)` still lands+under `cubepi.metadata.*` and **must not** overwrite `cubepi.run_id`. Hosts+pass the business id through `agent.prompt(run_id=...)`, not through metadata.++### 2. Oneshot: keep minting, same semantic++`Tracer.oneshot` has no agent. It continues to mint a run id for that session.+That id **is** the oneshot's run id (only activation id that exists), not a+parallel "trace run" concept. Optional later: accept `run_id=` on oneshot for+host correlation.++### 3. Nested subagents++Unchanged structure:++- Parent and child may share one OTel `trace_id` (existing nesting).+- Each agent's activation has its own business `run_id` (child `prompt` with a+ fresh id, or host policy).+- Spans of each activation carry **that** activation's id.++Do not force child spans to inherit the parent's business run id.++### 4. HITL pause / resume++One business `run_id` across pause and resume (already true on the agent).+Each activation still opens a new `invoke_agent` root / may be a new+`trace_id`. After this change both roots' spans carry the **same**+`cubepi.run_id`, so `cubepi trace` filter-by-run-id finds both halves.++### 5. Stream recording path++`record_stream` writes `<run_id>.stream.jsonl`. After the change the filename+uses the business id. Sanitize as today (`_safe_filename`). Collision risk is+the same as "two concurrent activations with the same run_id", which hosts+already must not do for the ledger.++### 6. Spec text update++In the tracing design §10.1, change:++| Attribute | Source (old) | Source (new) |+|---|---|---|+| `cubepi.run_id` | uuid generated at root open | agent `active_run_id` when set; else generated fallback |++### 7. cubeplex++Already passes `run_id` into `agent.prompt`. No required change for alignment.++Optional follow-ups (separate PRs, not required for the contract):++- Message action **Info** chip: document that the id is the cubepi/trace run id+ (already true once this lands).+- Admin traces filter: accept the same id.++## Change list (implementation)++### cubepi++| Area | Change |+|---|---|+| `cubepi/tracing/recorder.py` | `_on_agent_start`: resolve run id from `self._agent` active run; fallback uuid |+| `cubepi/tracing/tracer.py` | oneshot: document that minted id *is* the run id; optional `run_id=` later |+| `dev/specs/2026-05-18-cubepi-tracing-design.md` | §10.1 source line for `cubepi.run_id` |+| Tests | See below |+| Changelog | breaking-ish note for anyone who stored old tracer-minted ids |++Public API surface: no new functions required. Behavior change only.++### Tests (invariants)++1. **`prompt(run_id="host-1")` → all spans `cubepi.run_id == "host-1"`**+ (root, turn, chat, tool if any).+2. **`prompt()` without run_id → spans share one non-empty id; equals returned+ run id from `prompt`.**+ (Today `prompt` returns `effective_run_id`; tracing must match that, not a+ third uuid — so fallback should use the same source as the agent, not a+ second `uuid4()` in the recorder.)++ **Important refinement:** when the agent already chose `effective_run_id`+ (including self-minted `uuid.uuid4().hex`), the recorder must **read that**,+ not mint again. Fallback self-mint only if `active_run_id` is unexpectedly+ unset (defensive).+3. **Metadata cannot clobber** (existing PR #92 test still passes).+4. **Subagent:** parent and child different business ids → different+ `cubepi.run_id` on their respective span subtrees; same `trace_id` when+ nested.+5. **Sequential runs** on one attached agent: run A then run B → no id leak.+6. **Oneshot:** still produces a non-empty `cubepi.run_id` on root + chat.++### Migration / compatibility++- Old JSONL files keep historical tracer-minted ids; no migration.+- Operators who bookmarked old `cubepi.run_id` values for in-flight runs will+ not match new spans; only new activations align.+- CLI filters (`cubepi trace ls` by run id attribute) start matching host ids+ after upgrade — that is the intended win.++## Success criteria++1. Given cubeplex (or any host) `run_id=R` on `prompt`, every span of that+ activation has `cubepi.run_id=R`.+2. `cubepi trace` lookup by the id shown in the product UI finds that run's+ spans (for agent activations; oneshot still uses its own id).+3. No second uuid is minted by the recorder when `active_run_id` is set.+4. JSONL layout, OTel parentage, and metadata namespacing remain unchanged.+5. Existing "metadata cannot clobber reserved attrs" tests stay green.++## Open points (non-blocking)++- Expose `run_id=` on `Tracer.oneshot` for host-supplied correlation of+ memory/background jobs.+- Whether `AgentStartEvent` should carry `run_id` explicitly (nice for pure+ observers; not required if recorder reads `active_run_id`).+- Admin UI deep-link from message Info chip → traces filtered by run id+ (cubeplex product; after this lands).++## Decision needed before implement++None for the core path: **one run concept; recorder adopts `active_run_id`.**++Approve this draft → implement in cubepi (tests first) → bump cubeplex's+cubepi pin when released.diff --git a/tests/tracing/test_recorder.py b/tests/tracing/test_recorder.pyindex 3fd0406..9b21ad3 100644--- a/tests/tracing/test_recorder.py+++ b/tests/tracing/test_recorder.py@@ -1478,6 +1478,63 @@ class TestTracingContext:assert attrs.get("cubepi.metadata.turn.index") == "x"+class TestBusinessRunIdAlignment:+ """cubepi.run_id must equal the agent/host business run id.++ See dev/specs/2026-08-10-unify-trace-run-id.md — no second tracer-+ private uuid when active_run_id is set.+ """++ async def test_host_run_id_stamped_on_all_spans(self):+ agent, provider, exporter, tracer = await _build()+ provider.append_responses([faux_assistant_message("ok")])++ returned = await agent.prompt("x", run_id="host-run-42")+ await agent.wait_for_idle()+ await tracer.shutdown()++ assert returned == "host-run-42"+ assert exporter.spans+ run_ids = {_attrs(s).get("cubepi.run_id") for s in exporter.spans}+ assert run_ids == {"host-run-42"}++ async def test_agent_minted_run_id_matches_prompt_return(self):+ agent, provider, exporter, tracer = await _build()+ provider.append_responses([faux_assistant_message("ok")])++ returned = await agent.prompt("x")+ await agent.wait_for_idle()+ await tracer.shutdown()++ assert returned+ run_ids = {_attrs(s).get("cubepi.run_id") for s in exporter.spans}+ assert run_ids == {returned}++ async def test_sequential_runs_do_not_leak_run_id(self):+ agent, provider, exporter, tracer = await _build()+ provider.append_responses(+ [+ faux_assistant_message("a"),+ faux_assistant_message("b"),+ ]+ )++ r1 = await agent.prompt("one", run_id="run-a")+ await agent.wait_for_idle()+ r2 = await agent.prompt("two", run_id="run-b")+ await agent.wait_for_idle()+ await tracer.shutdown()++ assert r1 == "run-a" and r2 == "run-b"+ by_run: dict[str, int] = {}+ for s in exporter.spans:+ rid = _attrs(s).get("cubepi.run_id")+ assert rid in ("run-a", "run-b")+ by_run[rid] = by_run.get(rid, 0) + 1+ assert by_run.get("run-a", 0) >= 1+ assert by_run.get("run-b", 0) >= 1++class TestLifecycle:async def test_shutdown_is_idempotent(self):agent, provider, exporter, tracer = await _build()diff --git a/website/docs/guides/tracing/getting-started.md b/website/docs/guides/tracing/getting-started.mdindex 457c9dc..77371b3 100644--- a/website/docs/guides/tracing/getting-started.md+++ b/website/docs/guides/tracing/getting-started.md@@ -85,9 +85,12 @@ The run produces one JSONL file per trace (sharded by `trace_id`):8e1c9a3f4b2d…d976a.jsonl ← one trace, one file, one span per line```-A trace is the whole run, including any nested subagent runs (they inherit the-parent's `trace_id`, so they land in the same file). Each span still carries-`cubepi.run_id` as an attribute if you want to filter by individual run.+A trace is the whole OTel tree, including any nested subagent runs (they+inherit the parent's `trace_id`, so they land in the same file). Each span+also carries `cubepi.run_id` — the **same** id as `agent.prompt(run_id=…)` /+`Message.run_id` (hosts that pass a run id will see it here; otherwise the+agent-minted id). Use it to filter by individual business run inside a+trace that may hold several nested activations.Open it with any tool that reads OTLP/JSON or with `jq` directly:@@ -164,9 +167,10 @@ than silently disappearing.Defaults (no opt-in needed):- `invoke_agent` (root) — `gen_ai.operation.name`, `gen_ai.provider.name`,- `gen_ai.agent.name`, `cubepi.run_id`, `cubepi.agent.system_prompt.sha256`,- `cubepi.agent.tools` (names list), `cubepi.input_messages.count`,- `cubepi.output_messages.count`+ `gen_ai.agent.name`, `cubepi.run_id` (business run id from+ `prompt`/`resume`/`respond`, not a separate tracer uuid),+ `cubepi.agent.system_prompt.sha256`, `cubepi.agent.tools` (names list),+ `cubepi.input_messages.count`, `cubepi.output_messages.count`- `cubepi.turn` — `cubepi.turn.index`, `cubepi.turn.stop_reason`,`cubepi.turn.tool_calls.count`, `cubepi.turn.terminated_by_tool`,`cubepi.run_id`