Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Run-scoped-telemetry-identity · L-260922-75299c — TelemetryIdentity, per-span distinct_id (#1231)
Louis Choquela955cb50b026
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 25f4de1c0..07b199b4c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,10 +4,19 @@### Added+- **Every span and event a run produces is attributed to that run's caller**: `PostHogSpanExporter` resolves the identity per span and `track_event` per event, both through one `TelemetryIdentity` value, instead of stamping the single `distinct_id` bound when the telemetry manager was built. A run's `user_id` becomes the capture's `distinct_id` and its `analytics_groups` ride PostHog's groups facet; the span sites write both onto the span as `pipelex.run.user_id` and `pipelex.run.analytics_groups`, so every OpenTelemetry exporter sees the same fact and Langfuse finally receives `langfuse.user.id` — withheld for those placeholders, since that field is the attribution itself and has no later resolution behind it. The configured `user_id` and the Gateway-key hash stay as the fallback for everything that names no caller of its own: an event outside any run, and a run whose `user_id` is one of the placeholders — `local` for a run on your own machine, `dry-run-no-user` for a dry-run sweep, `single-tenant` for every run on a pipelex-api deployment configured with no users — which are the same strings everywhere and so are declined as identities rather than merging every deployment onto one person. The groups ride a fallback capture too, so a host may group its runs without naming their callers. `mode = "anonymous"` identifies nobody, a run's own caller and the configured `user_id` included. Pipelex's own Gateway stream is one shared project across every deployment, so it receives neither: a run's caller reaches it only as a one-way digest taken inside the Gateway-key hash, which keeps two callers of one deployment apart while making a collision between two deployments impossible, and the host's `analytics_groups` are not forwarded to it at all. The Gateway-key hash itself rides that stream's group facet, because the digest consumes it and a shared project would otherwise have no way to count a deployment's usage under the key that produced it. Neither the user id nor the groups ever become an event property or a span name. A host that serves many callers from one process therefore sees each caller's generations on their own timeline rather than everything under one identity per deployment.- **A run carries opaque analytics groups the runtime never parses**: `RunMetadata` and the runtime-bridge payload gain `analytics_groups`, a host-supplied mapping of group type to group key — `{"organization": "org_acme"}` on a multi-tenant host, nothing at all on a single-user one — threaded through `pipeline_run_setup`, `prepare_pipe_job` and `PipelexKernel.make` beside `user_id` and `storage_scope`. No key is privileged and the runtime never reads one by name, the same boundary that keeps `storage_scope` opaque. It is validated where it enters, on the type and at the wire: a lowercase snake_case group type, a group key from `A-Za-z0-9_-`, and a small fixed number of group types per run, so a malformed mapping is a construction error naming the field rather than a failure inside a telemetry capture.+### Changed++- **`RunMetadata` is frozen**: a field can no longer be reassigned once the run exists, and an attempt raises a pydantic `ValidationError` at runtime rather than merely failing a type check. The span sites and both PostHog exporters read a run's identity off it while the run is in flight, so a value that changed underneath them would attribute one run's spans to two different callers. Construct a new `RunMetadata` wherever one used to be mutated.+### Fixed+- **A crash no longer sends a `PipelexError`'s message as it was written**: the privacy redaction replaced a `PipelexError`'s message only when the error was handed over bare, and the exception autocapture holds nothing but the `(type, value, traceback)` triple an interpreter excepthook is given — so the one capture nobody asks for was the one sending the message verbatim. Such a message may repeat whatever the caller passed in, a file path or a slice of the document that would not parse, and it travelled on the shipped default, where `custom_posthog.mode = "off"` still builds the Gateway client whose construction installs the hooks. Both forms are redacted now, to a same-class stand-in carrying the privacy notice, the class name and the traceback, and nothing else — and so is every `PipelexError` PostHog would reach from the error it is handed, since it sends the whole `__cause__` and `__context__` chain and every member of an exception group, not only the error at the top. The redaction works on a copy of that chain, so the exception the interpreter prints is left as it was. It goes by class: an error of another class keeps its own message, so one that copied a `PipelexError`'s text into it — `raise RuntimeError(f"failed: {error}") from error` — still sends that text.+- **Both telemetry streams record a crash, rather than whichever one went first**: PostHog stamps an error once a client has captured it and drops every later capture of that same object, a guard against one stream recording one error twice that read two destinations as a repeat. The stamp is cleared between the streams now, so Pipelex's stream — always the second — stops losing every unhandled exception on a machine that also reports to an operator's own project. An error the host had already captured itself before it escaped is still recognised as sent, redacted or not, and is not recorded a second time.+- **An unhandled exception is captured under the same identity as everything else on its stream**: the autocapture was built with the configured `user_id` and the Gateway-key hash and sent them raw, so `mode = "anonymous"` still named you on every crash, and a run's `$exception` landed on a different PostHog person from its own spans and events. Both streams now resolve the identity they use for a capture that belongs to no run — which is what an exception is, since the interpreter hooks are handed nothing but the exception itself.+- **An anonymous capture on your PostHog no longer marks the Gateway capture of the same event**: both streams were handed the same properties dictionary, and the anonymous branch stamped `$process_person_profile = false` onto it before the Gateway capture sent it with a `distinct_id`. Each capture now takes its own copy.- **A malformed `storage_scope` is refused before the run registers itself**: it was validated only inside `prepare_pipe_job`, which runs below the pipeline registration, the library acquire, the tracer open and the trace-start emission, so a scope carrying `..` announced a trace that cannot be unsent before anything looked at it. `pipeline_run_setup` now gates the caller's scope at its top, and gates the rebound value again when the local sentinel turns it into the caller's `pipeline_run_id`. The existing gate in `prepare_pipe_job` stays: it is the first one for a caller reaching that seam directly.## [v0.62.0] - 2026-09-21diff --git a/docs/configuration/config-practical/telemetry-config.md b/docs/configuration/config-practical/telemetry-config.mdindex 2827ab452..063ceb494 100644--- a/docs/configuration/config-practical/telemetry-config.md+++ b/docs/configuration/config-practical/telemetry-config.md@@ -101,7 +101,7 @@ The `[custom_posthog]` section configures event tracking and optional AI span tr| Setting | Type | Default | Description ||---------|------|---------|-------------|| `mode` | string | `"off"` | Telemetry mode: `"off"`, `"anonymous"`, or `"identified"` |-| `user_id` | string | (none) | Required when `mode = "identified"` |+| `user_id` | string | (none) | Required when `mode = "identified"`. The identity for events that have no run — a run reports under its own caller || `endpoint` | string | `"https://us.i.posthog.com"` | PostHog endpoint URL || `api_key` | string | (required) | Your PostHog project API key || `geoip` | boolean | `true` | Enable GeoIP lookup for location data |@@ -111,8 +111,10 @@ The `[custom_posthog]` section configures event tracking and optional AI span tr#### Mode Options- **`"off"`**: No events sent to your PostHog-- **`"anonymous"`**: Events sent without user identification-- **`"identified"`**: Events sent with your `user_id` for cross-session tracking+- **`"anonymous"`**: Events sent without user identification. This covers your users too, and yourself: a run's own caller is never identified, its analytics groups are never sent, and a `user_id` left in the file is not sent either+- **`"identified"`**: Events and spans produced during a run are attributed to that run's caller, with its analytics groups attached. Your `user_id` is what everything else reports under — an event outside any run, and a run that names no distinguishable caller, which is every run on your own machine. The groups ride either way: a run may carry them without naming a caller++See [Telemetry Setup](../../setup/telemetry.md) for how a run supplies its caller and its groups.### `[custom_posthog.tracing]` Settingsdiff --git a/docs/features/telemetry.md b/docs/features/telemetry.mdindex 0c1d3c0ec..c6daaf0c3 100644--- a/docs/features/telemetry.md+++ b/docs/features/telemetry.md@@ -23,6 +23,10 @@ Send execution spans to any OTLP-compatible backend for integration with your exEvent tracking and AI span tracing with fine-grained privacy controls. Choose between anonymous and identified modes. Configure what data to capture: content, pipe codes, output class names, and content length limits.+## Per-run attribution++A span or an event produced during a run is attributed to that run's own caller, not to the process that served it — which is what lets one deployment run many people's methods and still see each person's generations on their own timeline. The identity comes from the run: `user_id` is who started it, and the optional `analytics_groups` mapping says which of your entities it belongs to. Both are supplied where the run is, they ride the job through every nested pipe and across a process boundary, and they reach PostHog in the fields it reserves for identity — never as an event property or a span name. Anything that names no caller of its own reports under the `user_id` you configured: an event outside any run, and a run on your own machine, whose caller is the literal `local` rather than a person a backend could tell apart — though its groups still ride, because which entities a run belongs to is a separate question from who started it. All of that describes the destinations you control; Pipelex's own Gateway stream receives a one-way digest of the caller inside your key's hash and none of your groups. See [Telemetry Setup](../setup/telemetry.md) for how to supply the groups.+## Gateway TelemetryWhen using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration.diff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.mdindex 6a6bfcbf3..70e2e09a0 100644--- a/docs/setup/telemetry.md+++ b/docs/setup/telemetry.md@@ -12,6 +12,8 @@ Pipelex supports two independent telemetry streams that serve different purposesWhen you use **Pipelex Gateway** as your inference backend, identified telemetry is **automatically enabled**. This telemetry is tied to your Gateway API key (hashed for security) and operates independently from your `telemetry.toml` settings.+A run that names a caller of its own is still distinguished on this stream, but never by a value you supplied: the caller's `user_id` is folded one way into your key's hash, so the value you spelled never reaches us, and the same caller name at another deployment is a different person here. Your key's own hash travels beside the fold, so your usage stays countable as yours. Read the fold as a pseudonym rather than as a promise of anonymity: it cannot be reversed, but it is a digest and not a secret, so it is not proof against somebody who already holds a candidate id and wants to check it. Your `analytics_groups` are not forwarded to this stream at all — they are your own vocabulary about your own customers, and they stay on the destinations you control.+**What we collect:**- Model names used (e.g., `gpt-5.4`, `claude-4.5-sonnet`) and parameters@@ -25,6 +27,7 @@ When you use **Pipelex Gateway** as your inference backend, identified telemetry- Your prompts or completions- Your pipe codes or output class names- File contents or business data+- A run's `user_id` as you spell it, or its `analytics_groups` in any formThis telemetry allows us to:@@ -112,8 +115,14 @@ The run-level metadata every run carries (`RunMetadata`) has an `analytics_groupanalytics_groups = {"organization": "org_acme"}```-!!! note "What this field does today"- The runtime validates the mapping, carries it through every nested pipe, and keeps it on the run's metadata across a process boundary. It does **not** yet attach it to the events and spans the built-in PostHog and OpenTelemetry integrations emit — so setting it today will not group anything in your backend. Supply it if you want your runs to carry the labels from now on; wait if you want to see them in PostHog.+On **your own** PostHog stream, every span and every event the run produces is then captured under that run's `user_id`, with the groups attached through PostHog's own groups facet — so a generation made for one of your customers appears on that customer's timeline and inside their organization, instead of under one identity per deployment. The same values reach every OpenTelemetry exporter as the span attributes `pipelex.run.user_id` and `pipelex.run.analytics_groups`, and Langfuse receives the user id in the field it reserves for it. Pipelex's own Gateway stream is the one exception, and it is described above: it receives a one-way digest of the caller and none of your groups.++Anything that names no caller of its own keeps reporting under the `user_id` you configured in `telemetry.toml`, which is what that setting now means: the identity of everything that is not one caller's run. That covers an event outside any run — a CLI command listing your pipes — and a run whose caller is not a distinguishable person either: a run on your own machine is attributed to the literal `local`, the same string on every machine, so Pipelex declines it as an identity and uses your configured id instead. The same goes for `single-tenant`, which pipelex-api states for every run when it is deployed with no users. Per-run attribution is for a host that passes a real `user_id` per run.++The groups do not depend on that. A run that leaves `user_id` at its default still carries its `analytics_groups` onto every capture, under your configured id — knowing which entities a run belongs to and naming its caller are two separate decisions, and you may take one without the other.++!!! note "Anonymous mode covers your users too"+ With `mode = "anonymous"`, the runtime identifies nobody on your stream: no run's `user_id` is applied, no groups are sent, and the `user_id` you configured is not sent either — a mode that identifies nobody would not be one that still named you. Leaving a `user_id` in `telemetry.toml` while switching to `anonymous` therefore changes nothing. Per-run attribution needs `mode = "identified"`.The rules it follows:@@ -122,7 +131,7 @@ The rules it follows:- **It is optional.** Omitting it leaves the group facet empty, which misattributes nothing. Unlike `user_id` and `storage_scope`, which have no default because a missing one used to be invented, a missing group invents nothing.- **It travels with the run.** The mapping rides the job through every nested pipe, and the bridge payload that crosses a process boundary carries it, so a distributed worker is handed the same groups the entry point was given.-These are labels for grouping, not content: they belong in the reserved identity fields of whichever backend understands groups, never in an event property or a span name.+These are labels for grouping, not content: they land in the reserved identity fields of whichever backend understands groups, never in an event property or a span name.## Privacydiff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.pyindex ae95b74a6..44b1e46b1 100644--- a/pipelex/cogt/llm/llm_worker_abstract.py+++ b/pipelex/cogt/llm/llm_worker_abstract.py@@ -24,6 +24,7 @@ from pipelex.system.telemetry.otel_constants import (make_otel_gen_ai_output_type,)from pipelex.system.telemetry.otel_factory import OtelFactory+from pipelex.system.telemetry.telemetry_identity import make_run_identity_span_attributesfrom pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstractfrom pipelex.tools.misc.filetype_utils import UNKNOWN_FILE_TYPEfrom pipelex.tools.misc.package_utils import get_package_version@@ -118,7 +119,8 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):if not pipe_code:msg = "pipe_code is required for LLM span"raise JobMetadataError(msg)- pipeline_run_id = job_metadata.run_metadata.pipeline_run_id+ run_metadata = job_metadata.run_metadata+ pipeline_run_id = run_metadata.pipeline_run_id# Build output description for span name (full values - exporter handles redaction)output_desc: str@@ -158,7 +160,12 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):if output_class_name:span_attributes[PipelexSpanAttr.OUTPUT_CLASS_NAME] = output_class_name- if TelemetryManagerAbstract.get_langfuse_enabled():+ # The run's own identity, so a generation lands on the caller who asked for it+ # rather than on the process that served it.+ is_langfuse_enabled = TelemetryManagerAbstract.get_langfuse_enabled()+ span_attributes.update(make_run_identity_span_attributes(run_metadata=run_metadata, is_langfuse_enabled=is_langfuse_enabled))++ if is_langfuse_enabled:span_attributes[LangfuseSpanAttr.TRACE_NAME] = otel_context.trace_namespan_attributes[LangfuseSpanAttr.RELEASE] = get_package_version()if job_params.max_tokens:@@ -196,7 +203,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):messages_json = OtelFactory.stringify_json(json_conent=messages)span_attributes[GenAISpanAttr.PROMPT_CONTENT] = messages_json- if TelemetryManagerAbstract.get_langfuse_enabled():+ if is_langfuse_enabled:span_attributes[LangfuseSpanAttr.OBSERVATION_INPUT] = messages_json# Use trace_id and span_id from otel_context (precomputed)diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.pyindex d48b656a4..e6f0d23a9 100644--- a/pipelex/pipe_machinery/pipe_abstract.py+++ b/pipelex/pipe_machinery/pipe_abstract.py@@ -29,7 +29,7 @@ from pipelex.pipe_machinery.validation import is_variable_satisfied_by_inputsfrom pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_applyfrom pipelex.pipe_signature.exceptions import PipeSignatureNotExecutableErrorfrom pipelex.pipeline.pipeline_factory import PipelineFactory-from pipelex.system.job_metadata import JobMetadata, OtelContext+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.registries.class_registry_access import get_class_registryfrom pipelex.system.telemetry.otel_constants import (@@ -40,6 +40,7 @@ from pipelex.system.telemetry.otel_constants import (SpanOutcome,)from pipelex.system.telemetry.otel_factory import OtelFactory+from pipelex.system.telemetry.telemetry_identity import make_run_identity_span_attributesfrom pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstractfrom pipelex.tools.misc.package_utils import get_package_versionfrom pipelex.tools.misc.string_utils import is_snake_case@@ -932,7 +933,7 @@ class PipeAbstract(ABC, BaseModel):# Start OTel span firstspan, is_root_span = self._start_pipe_span(parent_otel_context=parent_otel_context,- pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,+ run_metadata=job_metadata.run_metadata,working_memory=working_memory,)# Get the actual span_id from OTel (OTel generates its own span_id)@@ -1034,7 +1035,7 @@ class PipeAbstract(ABC, BaseModel):self,*,parent_otel_context: OtelContext,- pipeline_run_id: str,+ run_metadata: RunMetadata,working_memory: WorkingMemory,) -> tuple[Span | None, bool]:"""Start an OTel span for this pipe execution.@@ -1044,7 +1045,8 @@ class PipeAbstract(ABC, BaseModel):Args:parent_otel_context: The parent's OTel context.- pipeline_run_id: The pipeline run ID for span attributes.+ run_metadata: The run half of the job metadata — the pipeline run ID for+ span attributes, and the identity every exporter attributes the span to.working_memory: The working memory containing input stuffs for telemetry capture.Returns:@@ -1056,6 +1058,8 @@ class PipeAbstract(ABC, BaseModel):log.verbose(f"[OTel] No tracer available for pipe '{self.code}'")return None, False+ pipeline_run_id = run_metadata.pipeline_run_id+# Always use full pipe code - redaction is handled by exportersspan_name = f"{self.pipe_type}: {self.code}"@@ -1080,8 +1084,13 @@ class PipeAbstract(ABC, BaseModel):PipelexSpanAttr.PIPE_CODE: self.code, # Full pipe code, exporter handles redaction}+ # The run's own identity, so every exporter attributes the span to the caller+ # rather than to the process that happens to be running it.+ is_langfuse_enabled = TelemetryManagerAbstract.get_langfuse_enabled()+ span_attributes.update(make_run_identity_span_attributes(run_metadata=run_metadata, is_langfuse_enabled=is_langfuse_enabled))+# Langfuse-specific attributes: always send full data- if TelemetryManagerAbstract.get_langfuse_enabled():+ if is_langfuse_enabled:span_attributes.update({LangfuseSpanAttr.TRACE_NAME: parent_otel_context.trace_name,diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex afc57903c..8edadae7f 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -176,11 +176,11 @@ async def pipeline_run_setup(# Validate the groups HERE, before this function causes anything observable.## `RunMetadata` validates them too, and `prepare_pipe_job` validates them at- # its own top — but BOTH run below `add_new_pipeline`, the open tracer and- # `handle_trace_start`, which emits a telemetry event that cannot be unsent.- # So a malformed mapping used to abort a run that had already registered- # itself and announced its own trace to the backend. The teardown below- # releases the local state; the emitted event it cannot take back.+ # its own top — but BOTH run below `add_new_pipeline` and the open tracer, so+ # a malformed mapping used to abort a run that had already registered itself+ # and opened a library and a tracer. The teardown below releases that state.+ # What it could not take back was the trace-start event, which this seam+ # emitted from up there too until the emission moved below the job build.## This is the same lesson `storage_scope` learned one seam lower, and the# same cure: ordering, not absence, was the defect.@@ -191,8 +191,8 @@ async def pipeline_run_setup(# `prepare_pipe_job` gates it too, and that gate stays — but it runs below# everything listed above, so until now the OPAQUE mapping was refused# earlier than the field that decides where a tenant's bytes land. A scope- # carrying `..` registered a pipeline, opened a library and a tracer and- # announced its trace before anything looked at it.+ # carrying `..` registered a pipeline and opened a library and a tracer+ # before anything looked at it.## ADDITIVE, never a move: the value validated here is the caller's raw one,# and the sentinel below rebinds it to something else that only the lower@@ -352,9 +352,6 @@ async def pipeline_run_setup(trace_name_redacted=trace_name_redacted,span_id=OTelConstants.OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID,)- # Emit trace start event immediately to establish trace name in PostHog- # This must happen before any pipe spans are created/exported- get_telemetry_manager().handle_trace_start(trace_name=trace_name, trace_name_redacted=trace_name_redacted, trace_id=trace_id)# Seam 2: build the pipe job (pure — no registration, telemetry, graph open, or library mutation).pipe_job = await prepare_pipe_job(@@ -378,11 +375,34 @@ async def pipeline_run_setup(inputs_base_dir=inputs_base_dir,)+ # Emit the trace start event to establish the trace name in the backend. It+ # must arrive before any pipe span, which is comfortably satisfied here: the+ # pipe has not run yet, and no span is created until it does.+ #+ # It sits BELOW `prepare_pipe_job` deliberately, although nothing between the+ # two emits anything. That seam is where the run's `RunMetadata` is built, and+ # the trace-start has to be attributed to the same person its spans will be —+ # so the event waits for the one object that says who that is, rather than a+ # second one assembled here from the same parameters. Waiting also means a+ # failure inside that seam no longer announces a trace for a run that never+ # started, which is the event that cannot be unsent.+ if otel_context is not None:+ get_telemetry_manager().handle_trace_start(+ trace_name=otel_context.trace_name,+ trace_name_redacted=otel_context.trace_name_redacted,+ trace_id=otel_context.trace_id,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )+properties = {EventProperty.PIPELINE_RUN_ID: pipeline_run_id,EventProperty.PIPE_TYPE: pipe.pipe_type,}- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_EXECUTE, properties=properties)+ get_telemetry_manager().track_event(+ event_name=EventName.PIPELINE_EXECUTE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )success = Truereturn pipe_job, pipeline_run_id, library_iddiff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex 5b9ee3fe1..da307e78c 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -248,7 +248,11 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,EventProperty.PIPELINE_OUTCOME: Outcome.FAILURE,}- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_COMPLETE, properties=properties)+ get_telemetry_manager().track_event(+ event_name=EventName.PIPELINE_COMPLETE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )raise PipelineExecutionError(message=exc.message,run_mode=pipe_job.pipe_run_params.run_mode,@@ -269,7 +273,11 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,EventProperty.PIPELINE_OUTCOME: Outcome.FAILURE,}- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_COMPLETE, properties=properties)+ get_telemetry_manager().track_event(+ event_name=EventName.PIPELINE_COMPLETE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )raise PipelineExecutionError(message=exc.message,run_mode=pipe_job.pipe_run_params.run_mode,@@ -326,7 +334,11 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,EventProperty.PIPELINE_OUTCOME: Outcome.SUCCESS,}- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_COMPLETE, properties=properties)+ get_telemetry_manager().track_event(+ event_name=EventName.PIPELINE_COMPLETE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )finished_at = datetime.now(UTC).isoformat()return PipelexRunResultExecute.from_pipe_output(diff --git a/pipelex/system/analytics_groups.py b/pipelex/system/analytics_groups.pyindex f3fabfcd5..51d5c6eb2 100644--- a/pipelex/system/analytics_groups.py+++ b/pipelex/system/analytics_groups.py@@ -8,9 +8,13 @@ tenant cares about, and Pipelex needs somewhere to put labels it will forwardwithout reading. `analytics_groups` is that somewhere — an opaque, host-suppliedmapping of group *type* to group *key* that the runtime carries on`RunMetadata`, unread, for a telemetry consumer that understands groups to-forward. Carrying it is all this module does: no exporter reads the mapping-yet, so do not take the field's presence as evidence that a span or an event is-already grouped by it.+forward. Carrying it unread is all this module does, but it is read downstream:+a run's mapping rides its spans as `pipelex.run.analytics_groups` and becomes+PostHog's own groups facet on the operator's stream, so whatever is put here is+a live group key in that project and a span attribute on every OpenTelemetry+export. Pipelex's own stream is the exception — one shared project across every+deployment, where a host's group vocabulary would collide with another's, so+the mapping never reaches it.The hosted Pipelex platform fills it with `{"organization": "<org_id>"}`, butnothing here knows or checks that: no key is privileged, and a deployment thatdiff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.pyindex fe8c181b6..356233122 100644--- a/pipelex/system/job_metadata.py+++ b/pipelex/system/job_metadata.py@@ -2,7 +2,7 @@ from datetime import datetimefrom enum import StrEnumfrom typing import Any-from pydantic import BaseModel, Field, field_validator+from pydantic import BaseModel, ConfigDict, Field, field_validatorfrom pipelex.system.analytics_groups import validate_analytics_groupsfrom pipelex.system.storage_scope import validate_storage_scope@@ -66,8 +66,30 @@ class RunMetadata(BaseModel):``JobMetadata`` it was produced under, so a transport that offloads anoversized result to storage can key it inside the run's own namespaceinstead of at the root of a bucket.++ **Frozen**, so the field validators below are the last word on every value+ here rather than a check at construction that a later assignment undoes.+ Nothing ever needed to reassign one — these are the facts that are constant+ for a whole run, which is the definition this class exists to draw — and+ since telemetry began forwarding ``analytics_groups`` to a backend and+ composing ``storage_scope`` into storage keys, "validated at construction"+ has to mean "validated, full stop".++ **What the freeze does not close**, named here so nobody mistakes any of it+ for closed. Freezing refuses a REASSIGNMENT and nothing else, so an in-place+ edit of the mapping (``run_metadata.analytics_groups[k] = v``) still reaches+ a backend unvalidated, and pydantic's two deliberate bypasses —+ ``model_copy(update=...)`` and ``model_construct`` — still build an instance+ without running a validator. The runtime does none of the three, and a+ future consumer must not start: ``copy_with_update`` only ever carries+ already-validated values forward, and a value that arrives from request data+ is constructed here or not at all. Closing them properly means re-validating+ at the point of use, which would put a raise inside a span site, where+ telemetry must never break the app."""+ model_config = ConfigDict(frozen=True)+user_id: strpipeline_run_id: str@@ -136,14 +158,13 @@ class RunMetadata(BaseModel):ARRIVED through construction is safe, which is the case that matters:every mapping crossing the wire or a call site is built here.- The guarantee stops at construction, and deliberately so. This model is- neither frozen nor `validate_assignment`, so mutating the mapping in- place — `run_metadata.analytics_groups[k] = v` — bypasses this- validator, exactly as reassigning `storage_scope` bypasses its own.- `model_copy` does not re-run validators either, which is fine:- `copy_with_update` only ever carries an already-validated mapping- forward. Nothing in the runtime mutates the mapping after construction;- a future consumer that forwards it must not start.+ The guarantee stops at construction, and deliberately so. The model is+ frozen, so reassigning the field raises — but freezing does not reach+ inside the mapping, and mutating it in place+ (`run_metadata.analytics_groups[k] = v`) still bypasses this validator.+ See the class docstring for that hole and for pydantic's two others.+ Nothing in the runtime mutates the mapping after construction; a future+ consumer that forwards it must not start."""return validate_analytics_groups(value=value)diff --git a/pipelex/system/storage_scope.py b/pipelex/system/storage_scope.pyindex 75a8ea15b..aae543aff 100644--- a/pipelex/system/storage_scope.py+++ b/pipelex/system/storage_scope.py@@ -80,11 +80,27 @@ DRY_RUN_USER_ID = "dry-run-no-user"# laptop separates nothing — there is exactly one tenant — while the run id is# the only thing that has to be distinct, because anything stored under a fixed# name (`results/main_stuff.json`) would otherwise be overwritten by the next-# run. `LOCAL_USER_ID` is not a sentinel: it is an identity, it is true, and it-# is reported as-is.+# run. `LOCAL_USER_ID` is not a sentinel either: it is an identity, it is true,+# and it is carried and stored as-is. Telemetry is the one place that declines+# it — it is the same string on every machine, so honouring it as a person would+# collapse every Pipelex user's local runs onto one, and both PostHog streams+# report under their own fallback instead. See+# `_NON_DISTINGUISHING_RUN_USER_IDS` in+# `pipelex.system.telemetry.telemetry_identity`.LOCAL_USER_ID = "local"LOCAL_STORAGE_SCOPE = "local"+# The caller of every run on a server that has declared it has no users.+#+# pipelex-api attributes a request to this when its own configuration says there+# is no user model — no authentication, or one shared static key — so there is+# exactly one tenant and it has nobody to name. Like `LOCAL_USER_ID` it is a true+# statement rather than an unknown caller, and it is carried and stored as-is;+# like `LOCAL_USER_ID` it is also the same string on every such deployment, so+# telemetry declines it as a person. It is defined here so a host imports the+# one string the runtime recognises instead of spelling its own.+SINGLE_TENANT_USER_ID = "single-tenant"+# The leaf generated bytes land under, composed onto the scope by# `GeneratedContentFactory`. It is a constant and not a config value on purpose:# an operator who can omit the leaf can put generated content back at the rootdiff --git a/pipelex/system/telemetry/exception_capture.py b/pipelex/system/telemetry/exception_capture.pyindex 82187eb6d..79e2769da 100644--- a/pipelex/system/telemetry/exception_capture.py+++ b/pipelex/system/telemetry/exception_capture.py@@ -4,10 +4,13 @@ from types import TracebackTypefrom typing import TYPE_CHECKINGfrom pipelex import log+from pipelex.system.telemetry.otel_constants import PostHogAttr+from pipelex.system.telemetry.telemetry_identity import TelemetryIdentityif TYPE_CHECKING:# Deferred import: avoid pulling heavy SDK at module-load timefrom posthog import Posthog+ from posthog.args import ExceptionArgclass DualClientExceptionCapture:@@ -15,19 +18,31 @@ class DualClientExceptionCapture:Unlike PostHog's built-in exception_autocapture which only uses default_client,this implementation sends to both custom and Pipelex PostHog clients.++ **An `$exception` is attributed like every other capture.** It is resolved+ through `TelemetryIdentity` rather than from a raw id, because the two+ interpreter hooks below are handed nothing but `(type, value, traceback)`:+ they cannot see the run that was in flight, so what they send is the runless+ identity of the stream, resolved once here. That resolution is what makes+ `anonymous` mode mean the same thing on this path as on every other — a mode+ that identifies nobody must not identify somebody when the process crashes."""+ # What PostHog stamps onto an error once a client has captured it. Every+ # later `capture_exception` of that same object returns without sending.+ POSTHOG_CAPTURE_MARKS = ("__posthog_exception_captured", "__posthog_exception_uuid")+def __init__(self,custom_posthog_client: "Posthog | None",- custom_distinct_id: str | None,+ custom_identity: TelemetryIdentity,pipelex_posthog_client: "Posthog | None",- pipelex_distinct_id: str | None,+ pipelex_identity: TelemetryIdentity,):self._custom_client = custom_posthog_client- self._custom_distinct_id = custom_distinct_id+ self._custom_identity = custom_identityself._pipelex_client = pipelex_posthog_client- self._pipelex_distinct_id = pipelex_distinct_id+ self._pipelex_identity = pipelex_identity# Save original hooksself._original_excepthook = sys.excepthook@@ -63,12 +78,19 @@ class DualClientExceptionCapture:self,exc_info: tuple[type[BaseException], BaseException | None, TracebackType | None],) -> None:- """Capture exception to both PostHog clients."""+ """Capture exception to both PostHog clients.++ An error already marked when the hook runs was captured by the host+ before it escaped, and is not sent again. Otherwise the mark the first+ stream leaves is cleared before the second, and only then.+ """exc_type, exc_value, exc_traceback = exc_info# Skip if no actual exception value (can happen with threading.excepthook)if exc_value is None:return+ if self.is_marked_as_captured(exception=exc_value):+ return# Create the properly typed tuple for PostHogposthog_exc_info: tuple[type[BaseException], BaseException, TracebackType | None] = (@@ -79,16 +101,85 @@ class DualClientExceptionCapture:# Send to custom PostHog clientif self._custom_client:- try:- self._custom_client.capture_exception(posthog_exc_info, distinct_id=self._custom_distinct_id)- except Exception as capture_exc: # ruff: ignore[blind-except]- # Telemetry must never break the app: a failed exception capture is logged at debug and swallowed.- log.debug(f"Failed to capture exception to custom PostHog: {capture_exc}")+ self._capture_to_client(+ client=self._custom_client,+ identity=self._custom_identity,+ posthog_exc_info=posthog_exc_info,+ stream_name="custom",+ )# Send to Pipelex PostHog clientif self._pipelex_client:- try:- self._pipelex_client.capture_exception(posthog_exc_info, distinct_id=self._pipelex_distinct_id)- except Exception as capture_exc: # ruff: ignore[blind-except]- # Telemetry must never break the app: a failed exception capture is logged at debug and swallowed.- log.debug(f"Failed to capture exception to Pipelex PostHog: {capture_exc}")+ self._forget_posthog_capture_marks(exc_value=exc_value)+ self._capture_to_client(+ client=self._pipelex_client,+ identity=self._pipelex_identity,+ posthog_exc_info=posthog_exc_info,+ stream_name="Pipelex",+ )++ def _capture_to_client(+ self,+ *,+ client: "Posthog",+ identity: TelemetryIdentity,+ posthog_exc_info: tuple[type[BaseException], BaseException, TracebackType | None],+ stream_name: str,+ ) -> None:+ """Send one `$exception` to one stream, under the identity that stream resolved.++ The identified and anonymous shapes are PostHog's own: an identified+ capture passes `distinct_id` and the groups it belongs to, an anonymous+ one passes no `distinct_id` at all and marks itself so, because PostHog+ rejects a null one and would otherwise mint a person for it.+ """+ try:+ if identity.distinct_id:+ client.capture_exception(posthog_exc_info, distinct_id=identity.distinct_id, groups=identity.groups or None)+ else:+ client.capture_exception(posthog_exc_info, properties={PostHogAttr.PROCESS_PERSON_PROFILE: False})+ except Exception as capture_exc: # ruff: ignore[blind-except]+ # Telemetry must never break the app: a failed exception capture is logged at debug and swallowed.+ log.debug(f"Failed to capture exception to {stream_name} PostHog: {capture_exc}")++ @classmethod+ def _forget_posthog_capture_marks(cls, *, exc_value: BaseException) -> None:+ """Let this stream judge the error for itself, whatever the other one did with it.++ Both streams are handed the same exception object, and PostHog drops a+ capture of an error it has already sent — a guard meant for one stream+ seeing one error twice. Two clients are two destinations rather than a+ repeat, so without this whichever stream goes second receives nothing at+ all, and the Gateway stream, which is always second, would never record+ a crash on a machine that also reports to an operator's own project.++ It runs only between the two streams, never before the first: a mark+ already present when the hook ran is the host's own capture, and+ `_capture_exception` has skipped that error. The marks left by the last+ stream to capture stay in place, so the guard still holds for anything+ downstream of this class.+ """+ for mark in cls.POSTHOG_CAPTURE_MARKS:+ if hasattr(exc_value, mark):+ delattr(exc_value, mark)++ @classmethod+ def is_marked_as_captured(cls, *, exception: "ExceptionArg") -> bool:+ """Whether PostHog has already sent this error, from any client, bare or as the triple."""+ exception_value = exception[1] if isinstance(exception, tuple) else exception+ return exception_value is not None and hasattr(exception_value, cls.POSTHOG_CAPTURE_MARKS[0])++ @classmethod+ def carry_capture_marks(cls, *, source: "ExceptionArg | None", target: "ExceptionArg | None") -> None:+ """Copy PostHog's capture marks from the error that was sent onto the error the caller holds.++ The two differ when the sent one is a redacted copy. Either argument may+ be bare or the `(type, value, traceback)` triple, as PostHog accepts both.+ """+ source_value = source[1] if isinstance(source, tuple) else source+ target_value = target[1] if isinstance(target, tuple) else target+ if source_value is None or target_value is None or source_value is target_value:+ return+ for mark in cls.POSTHOG_CAPTURE_MARKS:+ if hasattr(source_value, mark):+ setattr(target_value, mark, getattr(source_value, mark))diff --git a/pipelex/system/telemetry/observer_telemetry.py b/pipelex/system/telemetry/observer_telemetry.pyindex b358ec66f..d2713786c 100644--- a/pipelex/system/telemetry/observer_telemetry.py+++ b/pipelex/system/telemetry/observer_telemetry.py@@ -16,7 +16,11 @@ class ObserverTelemetry(ObserverProtocol):EventProperty.PIPELINE_RUN_ID: payload[PayloadKey.PIPELINE_RUN_ID],EventProperty.PIPE_TYPE: pipe_job.pipe_type,}- self.telemetry_manager.track_event(event_name=EventName.PIPE_RUN, properties=properties)+ self.telemetry_manager.track_event(+ event_name=EventName.PIPE_RUN,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )@overrideasync def observe_after_successful_run(self, payload: PayloadType) -> None:@@ -26,7 +30,11 @@ class ObserverTelemetry(ObserverProtocol):EventProperty.PIPE_TYPE: pipe_job.pipe_type,EventProperty.PIPE_RUN_OUTCOME: Outcome.SUCCESS,}- self.telemetry_manager.track_event(event_name=EventName.PIPE_COMPLETE, properties=properties)+ self.telemetry_manager.track_event(+ event_name=EventName.PIPE_COMPLETE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )@overrideasync def observe_after_failing_run(@@ -39,4 +47,8 @@ class ObserverTelemetry(ObserverProtocol):EventProperty.PIPE_TYPE: pipe_job.pipe_type,EventProperty.PIPE_RUN_OUTCOME: Outcome.FAILURE,}- self.telemetry_manager.track_event(event_name=EventName.PIPE_COMPLETE, properties=properties)+ self.telemetry_manager.track_event(+ event_name=EventName.PIPE_COMPLETE,+ properties=properties,+ run_metadata=pipe_job.job_metadata.run_metadata,+ )diff --git a/pipelex/system/telemetry/otel_constants.py b/pipelex/system/telemetry/otel_constants.pyindex 999f93b10..d52aaa01b 100644--- a/pipelex/system/telemetry/otel_constants.py+++ b/pipelex/system/telemetry/otel_constants.py@@ -121,6 +121,15 @@ class PipelexSpanAttr(StrEnum):OUTCOME = "pipelex.outcome" # "success" or "failure"OUTPUT_CLASS_NAME = "pipelex.output.class_name"+ # The run's identity, set at every span site from `RunMetadata` so the identity+ # rides the span object to every exporter instead of being correlated back to a+ # run through shared state. RUN_USER_ID is who started the run; the groups are+ # the host's opaque labels, serialized as a JSON string because an OTel attribute+ # is a scalar or a homogeneous list, never a mapping. The groups attribute is+ # omitted entirely when a run carries none.+ RUN_USER_ID = "pipelex.run.user_id"+ RUN_ANALYTICS_GROUPS = "pipelex.run.analytics_groups"+class LangfuseSpanAttr(StrEnum):"""Langfuse-specific span attribute keys for enhanced observability.diff --git a/pipelex/system/telemetry/otel_factory.py b/pipelex/system/telemetry/otel_factory.pyindex b4fae09ff..79cfbb8e9 100644--- a/pipelex/system/telemetry/otel_factory.py+++ b/pipelex/system/telemetry/otel_factory.py@@ -16,6 +16,7 @@ from pipelex.system.runtime import RunEnvironmentfrom pipelex.system.telemetry.exceptions import LangfuseCredentialsErrorfrom pipelex.system.telemetry.otel_constants import OTelConstantsfrom pipelex.system.telemetry.telemetry_config import LangfuseConfig, OtlpExporterConfig, TelemetryRedactionConfig+from pipelex.system.telemetry.telemetry_identity import RunIdentityPolicyfrom pipelex.tools.log.log import logfrom pipelex.tools.misc.hash_utils import hash_md5_to_intfrom pipelex.tools.misc.json_utils import JsonContent, pure_json_str@@ -214,7 +215,8 @@ class OtelFactory:def make_ai_tracer(cls,*,- user_id: str | None,+ custom_fallback_distinct_id: str | None,+ custom_run_identity_policy: RunIdentityPolicy,custom_posthog_client: "Posthog | None",custom_redaction_config: TelemetryRedactionConfig,pipelex_posthog_client: "Posthog | None",@@ -235,7 +237,13 @@ class OtelFactory:4. Langfuse Exporter: Sends OTLP traces to Langfuse for LLM observabilityArgs:- user_id: Optional User ID for event attribution (custom telemetry)+ custom_fallback_distinct_id: The operator's configured ``user_id``, used+ on their stream for a span that names no run (a run's own user is+ read from the span itself — see+ :mod:`pipelex.system.telemetry.telemetry_identity`)+ custom_run_identity_policy: What the operator's stream may do with a+ span's own user, which is what their ``posthog.mode`` amounts to —+ ``DIRECT`` under ``identified``, ``NONE`` otherwisecustom_posthog_client: Optional user's PostHog client for sending eventscustom_redaction_config: Redaction config for custom PostHog exporterpipelex_posthog_client: Optional Pipelex internal PostHog client (for gateway)@@ -276,7 +284,8 @@ class OtelFactory:if custom_posthog_client:custom_posthog_exporter = PostHogSpanExporter(posthog_client=custom_posthog_client,- distinct_id=user_id,+ fallback_distinct_id=custom_fallback_distinct_id,+ run_identity_policy=custom_run_identity_policy,redaction_config=custom_redaction_config,)provider.add_span_processor(OTelBatchSpanProcessor(custom_posthog_exporter))@@ -284,9 +293,13 @@ class OtelFactory:# Add Pipelex PostHog Exporter if client is provided (mandatory for gateway)if pipelex_posthog_client:+ # The Pipelex stream is one shared project for every deployment, so a+ # span's own user reaches it only as a digest taken inside the+ # gateway-key hash, and the run's groups do not reach it at all.pipelex_posthog_exporter = PostHogSpanExporter(posthog_client=pipelex_posthog_client,- distinct_id=pipelex_distinct_id,+ fallback_distinct_id=pipelex_distinct_id,+ run_identity_policy=RunIdentityPolicy.NAMESPACED,redaction_config=pipelex_gateway_redaction_config,)provider.add_span_processor(OTelBatchSpanProcessor(pipelex_posthog_exporter))diff --git a/pipelex/system/telemetry/posthog_span_exporter.py b/pipelex/system/telemetry/posthog_span_exporter.pyindex e6164d3c5..a03d43223 100644--- a/pipelex/system/telemetry/posthog_span_exporter.py+++ b/pipelex/system/telemetry/posthog_span_exporter.py@@ -22,36 +22,71 @@ from pipelex.system.telemetry.otel_constants import (SpanCategory,)from pipelex.system.telemetry.telemetry_config import TelemetryRedactionConfig+from pipelex.system.telemetry.telemetry_identity import RunIdentityPolicy, TelemetryIdentityclass PostHogSpanExporter(SpanExporter):"""Exports OTel spans to PostHog as $ai_generation or $ai_span events.- Applies redaction rules from TelemetryRedactionConfig before sending events.+ Applies redaction rules from TelemetryRedactionConfig before sending events,+ and attributes each span to the run that produced it — see+ :mod:`pipelex.system.telemetry.telemetry_identity`. One class serves both+ streams: the operator's exporter is built with their configured id and the+ policy their mode implies, Pipelex's with the gateway-key hash and the+ namespacing policy a shared project requires."""def __init__(self,+ *,posthog_client: Posthog,- distinct_id: str | None,+ fallback_distinct_id: str | None,+ run_identity_policy: RunIdentityPolicy,redaction_config: TelemetryRedactionConfig,):+ """Build an exporter for one PostHog stream.++ Args:+ posthog_client: The client this exporter captures through.+ fallback_distinct_id: The identity for a span that names no run —+ this used to be the identity for every span on the stream.+ run_identity_policy: What this stream may do with a span's own user.+ `NONE` on an operator's stream in `anonymous` mode, where the+ operator chose not to identify people at all; `NAMESPACED` on+ Pipelex's shared project.+ redaction_config: What this stream is allowed to see.+ """self.posthog_client = posthog_client- self.distinct_id = distinct_id+ self.fallback_distinct_id = fallback_distinct_id+ self.run_identity_policy = run_identity_policyself.redaction_config = redaction_config- def _capture_event(self, event: PostHogEvent, *, properties: dict[str, Any]) -> None:+ def _resolve_identity(self, *, attributes: Mapping[str, AttributeValue]) -> TelemetryIdentity:+ """Resolve who this span belongs to on this stream."""+ return TelemetryIdentity.make_from_span_attributes(+ attributes=attributes,+ fallback_distinct_id=self.fallback_distinct_id,+ run_identity_policy=self.run_identity_policy,+ )++ def _capture_event(self, event: PostHogEvent, *, properties: dict[str, Any], identity: TelemetryIdentity) -> None:"""Capture an event to PostHog, handling anonymous vs identified users.PostHog requires a valid distinct_id - passing None will cause the event to be rejected.For anonymous tracking, we omit distinct_id and set $process_person_profile=False.++ The identity lands in the fields PostHog reserves for it — `distinct_id`+ and `groups` — and in no property: a property cannot produce a group-level+ count after the fact, and a second copy of the user id is a second thing+ to keep in step."""- if self.distinct_id:- # Identified user: pass distinct_id+ if identity.distinct_id:+ # Identified user: pass distinct_id, and the entities the run belongs toself.posthog_client.capture(- distinct_id=self.distinct_id,+ distinct_id=identity.distinct_id,event=event,properties=properties,+ groups=identity.groups or None,)else:# Anonymous user: don't pass distinct_id, mark as anonymous@@ -283,7 +318,11 @@ class PostHogSpanExporter(SpanExporter):f" model={properties.get(PostHogAttr.MODEL)}")- self._capture_event(event=PostHogEvent.GENERATION, properties=properties)+ self._capture_event(+ event=PostHogEvent.GENERATION,+ properties=properties,+ identity=self._resolve_identity(attributes=attributes),+ )def _export_pipe_span(self, span: ReadableSpan, *, attributes: Mapping[str, AttributeValue]) -> None:"""Export a pipe execution span."""@@ -323,7 +362,11 @@ class PostHogSpanExporter(SpanExporter):f" parent_id={properties.get(PostHogAttr.PARENT_ID)}")- self._capture_event(event=PostHogEvent.SPAN, properties=properties)+ self._capture_event(+ event=PostHogEvent.SPAN,+ properties=properties,+ identity=self._resolve_identity(attributes=attributes),+ )@overridedef export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:diff --git a/pipelex/system/telemetry/telemetry_identity.py b/pipelex/system/telemetry/telemetry_identity.pynew file mode 100644index 000000000..b54214079--- /dev/null+++ b/pipelex/system/telemetry/telemetry_identity.py@@ -0,0 +1,342 @@+"""`TelemetryIdentity` — who a span or an event is attributed to, on one stream.++**Identity is a fact of the run, not of the process.** A `TelemetryManager` binds+one `distinct_id` when it is built — the operator's configured `user_id` on their+own stream, the gateway-key hash on Pipelex's — and before this module every span+and every event carried that one id. On a host that serves many callers from one+process that is exactly wrong: every tenant's generations land under one identity+per deployment. The runtime already knows who each run belongs to, on+`RunMetadata.user_id`, and what it belongs to, on `RunMetadata.analytics_groups`;+this module is the single place that turns those two facts into what a capture+needs.++**One resolution, applied everywhere.** The exporter resolves per span, the+tracker per event, and both go through `TelemetryIdentity` so they cannot drift.+What a stream may do with a run's identity is its `RunIdentityPolicy`, and there+are three — the enum below says which stream carries which, and why:++1. `NONE` — the capture is anonymous whatever the run says, and the caller marks+ it so. No user, no fallback, no groups.+2. `DIRECT` — the run's own `user_id` is the `distinct_id`, spelled as the+ operator's own backend already knows it, and the run's groups ride the+ capture.+3. `NAMESPACED` — the run's `user_id` is folded one way into the stream's own+ id. The capture stays per-run, but it is never a raw caller value and can+ never collide with another deployment's. The run's own groups do not travel;+ the stream's own id travels in their place, as the deployment group, so a+ capture that names a caller is still counted under the deployment that+ produced it.++**A run that names nobody** reports under the stream's configured fallback, and+so does an event that belongs to no run at all — a CLI command, a dry-run sweep.+`RunMetadata.user_id` is required, but some of its values name a caller without+distinguishing one; see `_NON_DISTINGUISHING_RUN_USER_IDS` below. This is why a+local run keeps reporting exactly as it did before per-run attribution existed.+Under `DIRECT` the run's groups still ride such a capture: a host may know which+entities a run belongs to without naming its caller, and the two facts are+independent.++**Where the identity is NOT.** The user id goes on the capture as `distinct_id`+and the groups go through the capture's `groups` argument. Neither is ever copied+into an event property or a span name: one field, one meaning, and a second copy+is a second thing to keep in step.+"""++import json+from enum import StrEnum+from typing import TYPE_CHECKING, Any, Mapping++from pydantic import BaseModel, ConfigDict, Field++from pipelex.system.storage_scope import DRY_RUN_USER_ID, LOCAL_USER_ID, SINGLE_TENANT_USER_ID+from pipelex.system.telemetry.otel_constants import LangfuseSpanAttr, PipelexSpanAttr+from pipelex.system.telemetry.telemetry_config import PostHogMode+from pipelex.tools.log.log import log+from pipelex.tools.misc.hash_utils import hash_sha256+from pipelex.tools.misc.json_utils import pure_json_str++if TYPE_CHECKING:+ from pipelex.system.job_metadata import RunMetadata+++# The `user_id` values that name a caller without distinguishing one, and must+# therefore never become a `distinct_id`.+#+# `RunMetadata.user_id` is required precisely so that a host must state who is+# running — but the two constructor defaults in+# :mod:`pipelex.system.storage_scope`, and the single-tenant id a host without a+# user model states, are true statements rather than identities a backend can+# tell apart. `LOCAL_USER_ID` is the literal "local", the same+# string on every machine on earth: honoured as a `distinct_id` it would collapse+# every Pipelex user's local runs onto one person, and on the Pipelex stream it+# would replace a per-deployment identity with a single global one. `DRY_RUN_USER_ID`+# says outright that there is no caller. `SINGLE_TENANT_USER_ID` is what+# pipelex-api states for every run on a deployment configured with no users, and+# is as global as `"local"`.+#+# So for telemetry these mean "this run names nobody", and the stream's own+# fallback answers instead: the operator's configured `user_id` on their stream,+# the gateway-key hash on Pipelex's — which is exactly what a CLI run reported+# under before per-run attribution existed, and still does.+_NON_DISTINGUISHING_RUN_USER_IDS = frozenset({LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID})+++# The group type Pipelex's own stream carries its deployment under.+#+# `NAMESPACED` folds the gateway-key hash into the `distinct_id`, which is what+# makes that id per-caller and unable to collide across deployments — and which+# consumes the hash, because the fold is one-way. Nothing else on the capture+# would carry it back: the identity is deliberately kept out of every property,+# and the Pipelex project is one shared project that cannot partition itself by+# deployment. So the hash travels as a group, the one facet PostHog can still+# aggregate on after the fact, and the stream stays what it is documented to be+# — tied to the Gateway API key that produced it.+PIPELEX_DEPLOYMENT_GROUP_TYPE = "deployment"+++class RunIdentityPolicy(StrEnum):+ """What one stream may do with a run's identity.++ Two streams, three policies, because they do not report to the same kind of+ audience.++ The operator's own stream reports into the operator's own PostHog project. A+ caller's `user_id` there is a value they already hold and a group key is+ their own vocabulary, so `DIRECT` sends both as they are — and `anonymous`+ mode turns the whole thing off with `NONE`, because an operator who chose to+ identify nobody chose it for their users too.++ Pipelex's stream is a SHARED project: every deployment of this version+ reports into the one key the remote config names. A raw `user_id` there is+ wrong twice over — two customers whose callers are both `user-42` would+ merge into a single person, and a caller id that is frequently an e-mail+ address would leave the deployment that holds it. The host's group keys are+ its own business vocabulary and PostHog's group types are project-global, so+ they do not belong there either. Hence `NAMESPACED`: Pipelex still tells two+ callers of one deployment apart, but only through a one-way digest taken+ inside that deployment's gateway-key hash, and the host's own groups stay+ home. The deployment itself does travel, as the one group this stream sets:+ the fold consumes the gateway-key hash, and without a facet carrying it back+ Pipelex could no longer count a deployment's usage under the key that paid+ for it.+ """++ NONE = "none"+ DIRECT = "direct"+ NAMESPACED = "namespaced"++ @classmethod+ def make_for_operator_stream(cls, *, posthog_mode: PostHogMode) -> "RunIdentityPolicy":+ """The policy an operator's own stream carries, read from the mode they set.++ Stated once, because the operator's stream resolves an identity from+ three places — the event tracker, the trace-start capture and the span+ exporter — and a mode that meant one thing in one of them and something+ else in another is exactly the drift this module exists to prevent.+ """+ return cls.DIRECT if posthog_mode.is_identified else cls.NONE+++def _make_namespaced_distinct_id(*, namespace: str, run_user_id: str) -> str:+ """Fold a run's user into one deployment's namespace, one way.++ The result tells two callers of the same deployment apart, and can never+ equal another deployment's id for a caller of the same name, because the+ namespace is inside the digest rather than beside it. It is not reversible,+ so a caller id that happens to be an e-mail address never leaves the+ deployment that holds it as a value anyone can read.++ It is a pseudonym and not a secret, and the difference is worth stating: the+ namespace is itself sent in the clear, as the `distinct_id` of every capture+ that names nobody, so a party who already holds a candidate caller id can+ fold it the same way and see whether the result is present. Making a guess+ useless would take a key that never reaches the stream — the raw gateway key+ rather than its hash — which is a wider change than this function.++ Args:+ namespace: The stream's own id for this deployment — the gateway-key hash.+ run_user_id: The run's user, already known to distinguish a caller.++ Returns:+ A 16-character digest, the same shape as the namespace itself.+ """+ return hash_sha256(data=f"{namespace}:{run_user_id}", length=16)+++def make_run_identity_span_attributes(*, run_metadata: "RunMetadata", is_langfuse_enabled: bool) -> dict[str, str]:+ """Build the span attributes that carry a run's identity to every exporter.++ Called at each span site from the `JobMetadata` already in hand, so the+ identity travels on the span object through the batch processor: no exporter+ has to correlate a span back to a run through shared state, and every+ exporter sees the same fact. PostHog is the one that reads the groups as+ group identity; Langfuse gets the user id in the field it reserves for it,+ which it has defined and never received until now.++ Args:+ run_metadata: The run half of the job metadata the span site holds.+ is_langfuse_enabled: Whether the Langfuse exporter is configured, which+ is the only reason to spend an attribute on Langfuse's spelling of+ the same value.++ Returns:+ The attributes to merge into the span's attribute dict. The groups entry+ is absent — not empty — when the run carries no groups. `pipelex.run.user_id`+ is written as the run states it, placeholders included: a span attribute+ records what the run IS, and whether that value may be attributed to a+ person is a separate decision, taken once in `_resolve` below. Langfuse's+ own field is the opposite case — it IS that decision, with no `_resolve`+ behind it to take it later — so a placeholder is withheld from it rather+ than becoming a person named `local` in the operator's project.+ """+ attributes: dict[str, str] = {PipelexSpanAttr.RUN_USER_ID: run_metadata.user_id}+ if run_metadata.analytics_groups:+ attributes[PipelexSpanAttr.RUN_ANALYTICS_GROUPS] = pure_json_str(data=run_metadata.analytics_groups)+ if is_langfuse_enabled and run_metadata.user_id not in _NON_DISTINGUISHING_RUN_USER_IDS:+ attributes[LangfuseSpanAttr.USER_ID] = run_metadata.user_id+ return attributes+++class TelemetryIdentity(BaseModel):+ """The resolved attribution of one capture on one stream.++ Frozen, because it is a decision already taken: a capture path reads it, it+ never edits it.+ """++ model_config = ConfigDict(frozen=True)++ # The identity the capture is sent under, or None for an anonymous capture.+ distinct_id: str | None = None++ # The entities the capture belongs to, forwarded through the backend's own+ # groups facet. Empty on an anonymous capture, which has no person for a+ # group to qualify. On a `NAMESPACED` stream it holds exactly one entry — the+ # deployment — and never a host's own vocabulary.+ groups: dict[str, str] = Field(default_factory=dict)++ @property+ def is_anonymous(self) -> bool:+ """Whether this capture has no person to attribute itself to."""+ return not self.distinct_id++ @classmethod+ def make_anonymous(cls) -> "TelemetryIdentity":+ """The identity of a stream that identifies nobody, whatever the run says."""+ return cls(distinct_id=None, groups={})++ @classmethod+ def _resolve(+ cls,+ *,+ run_user_id: str | None,+ run_groups: dict[str, str] | None,+ fallback_distinct_id: str | None,+ run_identity_policy: RunIdentityPolicy,+ ) -> "TelemetryIdentity":+ """The one resolution, stated once — see this module's docstring for the three cases."""+ if run_identity_policy is RunIdentityPolicy.NONE:+ return cls.make_anonymous()++ distinguishing_user_id = run_user_id if run_user_id and run_user_id not in _NON_DISTINGUISHING_RUN_USER_IDS else None++ if run_identity_policy is RunIdentityPolicy.NAMESPACED:+ if not fallback_distinct_id:+ # No namespace to fold into means no id that is safe to send on a+ # shared project, so the capture goes out with none rather than+ # under a raw caller value.+ return cls.make_anonymous()+ # The deployment travels as a group whether or not the run names a+ # caller, so one query counts a deployment's whole usage — the runs+ # that name somebody, under their digests, and the runs that name+ # nobody, under the hash itself.+ deployment_groups = {PIPELEX_DEPLOYMENT_GROUP_TYPE: fallback_distinct_id}+ if distinguishing_user_id:+ return cls(+ distinct_id=_make_namespaced_distinct_id(namespace=fallback_distinct_id, run_user_id=distinguishing_user_id),+ groups=deployment_groups,+ )+ return cls(distinct_id=fallback_distinct_id, groups=deployment_groups)++ resolved_distinct_id = distinguishing_user_id or fallback_distinct_id+ if not resolved_distinct_id:+ return cls.make_anonymous()+ return cls(distinct_id=resolved_distinct_id, groups=dict(run_groups or {}))++ @classmethod+ def make_from_run_metadata(+ cls,+ *,+ run_metadata: "RunMetadata | None",+ fallback_distinct_id: str | None,+ run_identity_policy: RunIdentityPolicy,+ ) -> "TelemetryIdentity":+ """Resolve the attribution of an event, from the run that produced it.++ Args:+ run_metadata: The run half of the metadata the emitter holds, or None+ for an event that belongs to no run — a CLI command, a dry-run+ sweep, the "telemetry just enabled" notice.+ fallback_distinct_id: The identity this stream falls back to.+ run_identity_policy: What this stream may do with the run's identity.+ """+ return cls._resolve(+ run_user_id=run_metadata.user_id if run_metadata is not None else None,+ run_groups=run_metadata.analytics_groups if run_metadata is not None else None,+ fallback_distinct_id=fallback_distinct_id,+ run_identity_policy=run_identity_policy,+ )++ @classmethod+ def make_from_span_attributes(+ cls,+ *,+ attributes: Mapping[str, Any],+ fallback_distinct_id: str | None,+ run_identity_policy: RunIdentityPolicy,+ ) -> "TelemetryIdentity":+ """Resolve the attribution of a span, from the attributes it carries.++ The counterpart of `make_run_identity_span_attributes`: a span produced+ before this release, or by a code path that sets no identity, simply+ carries neither attribute and resolves to the stream's fallback.+ """+ run_user_id = attributes.get(PipelexSpanAttr.RUN_USER_ID)+ return cls._resolve(+ run_user_id=run_user_id if isinstance(run_user_id, str) else None,+ run_groups=cls._read_groups_attribute(attributes=attributes),+ fallback_distinct_id=fallback_distinct_id,+ run_identity_policy=run_identity_policy,+ )++ @classmethod+ def _read_groups_attribute(cls, *, attributes: Mapping[str, Any]) -> dict[str, str]:+ """Read back the groups a span site serialized, refusing anything else.++ The value was written by `make_run_identity_span_attributes` from an+ already-validated mapping, so a shape other than a JSON object of strings+ means the span was built by something else. Telemetry never breaks the+ app, so that is a debug line and an empty mapping — the span still+ exports, attributed to the run's user without its groups.+ """+ raw = attributes.get(PipelexSpanAttr.RUN_ANALYTICS_GROUPS)+ if raw is None:+ return {}+ if not isinstance(raw, str):+ log.debug(f"Ignoring span attribute '{PipelexSpanAttr.RUN_ANALYTICS_GROUPS}': expected a JSON string, got {type(raw).__name__}")+ return {}+ try:+ decoded = json.loads(raw)+ except ValueError as exc:+ log.debug(f"Ignoring span attribute '{PipelexSpanAttr.RUN_ANALYTICS_GROUPS}': not decodable JSON ({exc})")+ return {}+ if not isinstance(decoded, dict):+ log.debug(f"Ignoring span attribute '{PipelexSpanAttr.RUN_ANALYTICS_GROUPS}': expected a JSON object, got {type(decoded).__name__}")+ return {}+ groups: dict[str, str] = {}+ for group_type, group_key in decoded.items(): # type: ignore[union-attr]+ if isinstance(group_type, str) and isinstance(group_key, str):+ groups[group_type] = group_key+ else:+ log.debug(f"Ignoring a non-string entry in span attribute '{PipelexSpanAttr.RUN_ANALYTICS_GROUPS}'")+ return groupsdiff --git a/pipelex/system/telemetry/telemetry_manager.py b/pipelex/system/telemetry/telemetry_manager.pyindex 8daa5670a..857d3465a 100644--- a/pipelex/system/telemetry/telemetry_manager.py+++ b/pipelex/system/telemetry/telemetry_manager.py@@ -1,5 +1,6 @@+import sysfrom contextlib import contextmanager-from typing import TYPE_CHECKING, Any, Generator+from typing import TYPE_CHECKING, Any, Generator, castimport posthogfrom opentelemetry.trace import Tracer as OTelTracer@@ -19,11 +20,14 @@ from pipelex.system.telemetry.exception_capture import DualClientExceptionCapturfrom pipelex.system.telemetry.otel_constants import OTelConstants, PostHogAttr, PostHogEventfrom pipelex.system.telemetry.otel_factory import OtelFactoryfrom pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig, TelemetryRedactionConfig+from pipelex.system.telemetry.telemetry_identity import RunIdentityPolicy, TelemetryIdentityfrom pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstractif TYPE_CHECKING:from opentelemetry.sdk.trace import TracerProvider as OTelTracerProvider+ from pipelex.system.job_metadata import RunMetadata+class TelemetryManager(TelemetryManagerAbstract):PRIVACY_NOTICE = "[Privacy: exception message redacted]"@@ -94,7 +98,8 @@ class TelemetryManager(TelemetryManagerAbstract):else:pipelex_gateway_redaction_config = TelemetryRedactionConfig.make_from_posthog_config(posthog_config=None)self._otel_tracer, self._tracer_provider = OtelFactory.make_ai_tracer(- user_id=telemetry_config.custom_posthog.user_id,+ custom_fallback_distinct_id=telemetry_config.custom_posthog.user_id,+ custom_run_identity_policy=RunIdentityPolicy.make_for_operator_stream(posthog_mode=telemetry_config.custom_posthog.mode),custom_posthog_client=self.custom_posthog_client if telemetry_config.custom_posthog.tracing.enabled else None,custom_redaction_config=custom_redaction_config,pipelex_posthog_client=self.pipelex_posthog_client,@@ -121,12 +126,36 @@ class TelemetryManager(TelemetryManagerAbstract):# Set up dual-client exception autocapture if any client is enabledif self.custom_posthog_client or self.pipelex_posthog_client:- self._exception_capture = DualClientExceptionCapture(- custom_posthog_client=self.custom_posthog_client,- custom_distinct_id=self.telemetry_config.custom_posthog.user_id,- pipelex_posthog_client=self.pipelex_posthog_client,- pipelex_distinct_id=self._pipelex_distinct_id,- )+ self._exception_capture = self._make_exception_capture()++ def _make_exception_capture(self) -> DualClientExceptionCapture:+ """Build the exception autocapture, with each stream's runless identity resolved.++ An unhandled exception belongs to no run that the interpreter hooks can+ see — they are handed `(type, value, traceback)` and there is no+ ContextVar layer to read a run out of — so what each stream sends is the+ identity it uses for a capture that names nobody, resolved through the+ same `TelemetryIdentity` as every other path. That is what carries+ `anonymous` mode onto this path: a stream that identifies nobody must not+ identify somebody when the process crashes.++ Stated as its own method so the resolution can be exercised without the+ constructor, which builds live clients and registers a singleton.+ """+ return DualClientExceptionCapture(+ custom_posthog_client=self.custom_posthog_client,+ custom_identity=TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id=self.telemetry_config.custom_posthog.user_id,+ run_identity_policy=RunIdentityPolicy.make_for_operator_stream(posthog_mode=self.telemetry_config.custom_posthog.mode),+ ),+ pipelex_posthog_client=self.pipelex_posthog_client,+ pipelex_identity=TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id=self._pipelex_distinct_id,+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ ),+ )def _handle_transmission_error( # kw-only: ignore — PostHog on_error callback, invoked positionally as (error, items)self, error: Exception | None, _items: list[dict[str, Any]]@@ -164,32 +193,189 @@ class TelemetryManager(TelemetryManagerAbstract):exception: ExceptionArg | None = None,**kwargs: Unpack[OptionalCaptureArgs],) -> Any:- """Capture exception with message sanitization for PipelexError subclasses."""- if exception and isinstance(exception, PipelexError):- # Create a new exception with sanitized message while preserving the class type- # Use __new__ to create an instance without calling __init__, which may require extra args- # This creates a "shell" instance with NO custom attributes- exception_type = type(exception)- sanitized_exception = exception_type.__new__(exception_type)-- # Set the exception args to our privacy notice- # This is what str(exception) will return- sanitized_exception.args = (self.PRIVACY_NOTICE,)-- # Preserve the traceback so we still get stack trace information- if hasattr(exception, "__traceback__"):- sanitized_exception.__traceback__ = exception.__traceback__-- # Note: No custom attributes (tested_concept, wanted_concept, etc.) are present- # because we used __new__() without calling __init__(). The __dict__ is already empty.-- return original_capture_exception(sanitized_exception, **kwargs)- else:- # For non-PipelexError, capture as-is (or auto-detect current exception)- return original_capture_exception(exception, **kwargs)+ """Capture exception with message sanitization for PipelexError subclasses.++ PostHog marks what it sent so a second capture of the same error is+ dropped, and what it sent may be a redacted copy rather than the+ error the caller holds. The mark is carried back onto the original,+ so the guard still works for a caller that captures an error and+ then lets it escape. Given nothing, PostHog would read the error+ being handled for itself, so that error is resolved first: both the+ check and the mark need it, and `None` can carry neither.++ PostHog's own `capture_exception` never raises into its caller, and+ the redaction in front of it must not either. An argument the SDK+ would decline, or a chain too deep to copy, drops the capture with a+ debug line: sending the original in its place is the one fallback+ that is not open, since it is what the redaction exists to stop.+ """+ try:+ resolved = self._resolved_exception_arg(exception=exception)+ if resolved is None or DualClientExceptionCapture.is_marked_as_captured(exception=resolved):+ return None+ sanitized = self._sanitized_exception_arg(exception=resolved)+ except Exception as sanitize_exc: # ruff: ignore[blind-except]+ log.debug(f"Dropped an exception capture the privacy redaction could not complete: {sanitize_exc!r}")+ return None+ result = original_capture_exception(sanitized, **kwargs)+ DualClientExceptionCapture.carry_capture_marks(source=sanitized, target=resolved)+ return resultclient.capture_exception = sanitized_capture_exception # type: ignore[method-assign]+ @classmethod+ def _resolved_exception_arg(cls, *, exception: ExceptionArg | None) -> ExceptionArg | None:+ """Return `exception`, or when it is `None` the error being handled, which is what PostHog would read for itself.++ Resolved here rather than left to the SDK so the error can still be+ redacted, and so the capture mark lands on the error the caller holds.+ """+ if exception is not None:+ return exception+ current_type, current_value, current_traceback = sys.exc_info()+ if current_type is None or current_value is None:+ return None+ return (current_type, current_value, current_traceback)++ @classmethod+ def _sanitized_exception_arg(cls, *, exception: ExceptionArg) -> ExceptionArg:+ """Return what may go out in place of `exception`, in whichever form PostHog was handed.++ PostHog takes an error either bare or as the `(type, value, traceback)`+ triple an interpreter excepthook receives. And it does not send one+ exception: it walks `__cause__`, `__context__` and the members of an+ exception group, and sends the message of every error it reaches. A+ `PipelexError` message may repeat whatever the caller passed in — the+ path it was reading, a slice of the document that would not parse — so+ the message of every `PipelexError` in that graph is replaced by the+ privacy notice, wherever it sits, and not only the one at the top.++ The redaction goes by class. An error of another class keeps its own+ message, so one that copied a `PipelexError`'s text into it —+ `raise RuntimeError(f"failed: {error}") from error` — still sends that+ text; only the linked `PipelexError` itself is replaced.+ """+ if isinstance(exception, BaseException):+ return cls._redacted_exception_graph(exception=exception)+ _, exception_value, exception_traceback = exception+ if exception_value is None:+ return exception+ redacted_value = cls._redacted_exception_graph(exception=exception_value)+ if redacted_value is exception_value:+ return exception+ return (type(redacted_value), redacted_value, exception_traceback)++ @classmethod+ def _redacted_exception_graph(cls, *, exception: BaseException) -> BaseException:+ """Return `exception` itself when nothing it reaches is a `PipelexError`, else a redacted copy of all of it.++ The copy is never written back: the live exception is the one the+ interpreter goes on to print and a host may still be holding, so only a+ copy of the graph has its links pointed at the stand-ins.+ """+ if not cls._reaches_pipelex_error(exception=exception):+ return exception+ redacted = cls._redacted_link(exception=exception, copies={})+ # The root is never cut: only a link back to an exception still being copied is.+ assert redacted is not None+ return redacted++ @classmethod+ def _reaches_pipelex_error(cls, *, exception: BaseException) -> bool:+ """Whether a `PipelexError` sits anywhere PostHog would walk from `exception`.++ A loop and not a recursion, as PostHog's own walk is: the SDK sends a+ chain as long as the recursion limit, and the check deciding whether it+ needs redacting must not be what fails on it.+ """+ pending: list[BaseException | None] = [exception]+ visited: set[int] = set()+ while pending:+ current = pending.pop()+ if current is None or id(current) in visited:+ continue+ visited.add(id(current))+ if isinstance(current, PipelexError):+ return True+ pending.extend((current.__cause__, current.__context__, *cls._group_members(exception=current)))+ return False++ @classmethod+ def _group_members(cls, *, exception: BaseException) -> tuple[BaseException, ...]:+ """The members PostHog expands out of an exception group, or nothing for any other error."""+ if isinstance(exception, BaseExceptionGroup):+ return cast("BaseExceptionGroup[BaseException]", exception).exceptions+ return ()++ @classmethod+ def _redacted_link(cls, *, exception: BaseException | None, copies: dict[int, BaseException | None]) -> BaseException | None:+ """Copy one exception of the graph, with its own links redacted in turn.++ `copies` maps an original to its copy so a shared or cyclic link is+ copied once. An entry of `None` means that exception is still being+ copied, and a link back to it is cut rather than followed forever.+ """+ if exception is None:+ return None+ exception_id = id(exception)+ if exception_id in copies:+ return copies[exception_id]+ copies[exception_id] = None+ copied: BaseException+ if isinstance(exception, PipelexError):+ copied = cls._redacted_stand_in(exception=exception)+ elif isinstance(exception, BaseExceptionGroup):+ group = cast("BaseExceptionGroup[BaseException]", exception)+ members = [cls._redacted_link(exception=member, copies=copies) for member in cls._group_members(exception=group)]+ kept_members = [member for member in members if member is not None]+ copied = cast("BaseException", group.derive(kept_members)) if kept_members else Exception(type(group).__name__)+ if hasattr(group, "__notes__"):+ copied.__notes__ = list(group.__notes__)+ else:+ copied = cls._plain_copy(exception=exception)+ copied.__traceback__ = exception.__traceback__+ copies[exception_id] = copied+ copied.__cause__ = cls._redacted_link(exception=exception.__cause__, copies=copies)+ copied.__context__ = cls._redacted_link(exception=exception.__context__, copies=copies)+ copied.__suppress_context__ = exception.__suppress_context__+ return copied++ @classmethod+ def _redacted_stand_in(cls, *, exception: PipelexError) -> PipelexError:+ """Build a same-class stand-in for `exception` whose message is the privacy notice.++ `__new__` without `__init__` is the point: the instance comes out with an+ empty `__dict__`, so none of the subclass's own attributes — the+ concepts it was comparing, the file it was reading — travel with it. What+ goes out is the class name, the notice and the traceback.+ """+ exception_type = type(exception)+ stand_in = exception_type.__new__(exception_type)+ stand_in.args = (cls.PRIVACY_NOTICE,)+ stand_in.__traceback__ = exception.__traceback__+ return stand_in++ @classmethod+ def _plain_copy(cls, *, exception: BaseException) -> BaseException:+ """Copy an error that is not a `PipelexError`, message and attributes included, without running its `__init__`.++ Its own message is sent as it is, because the redaction goes by class:+ a message that copied a `PipelexError`'s text travels with that text.+ Its attributes stay because a subclass's `__str__` may read them. Only+ the links are rewritten, by `_redacted_link`.+ """+ exception_type = type(exception)+ try:+ copied = exception_type.__new__(exception_type, *exception.args)+ copied.args = exception.args+ copied.__dict__.update(+ {name: value for name, value in exception.__dict__.items() if name not in DualClientExceptionCapture.POSTHOG_CAPTURE_MARKS}+ )+ except Exception: # ruff: ignore[blind-except]+ # A class whose construction cannot be reproduced keeps its name and message, and nothing else.+ return Exception(f"{exception_type.__name__}: {exception}")+ return copied+@overridedef setup(self, *, integration_mode: IntegrationMode):pass@@ -235,7 +421,13 @@ class TelemetryManager(TelemetryManagerAbstract):TelemetryManagerAbstract.clear_instance()@override- def track_event(self, event_name: EventName, *, properties: dict[EventProperty, Any] | None = None):+ def track_event(+ self,+ event_name: EventName,+ *,+ properties: dict[EventProperty, Any] | None = None,+ run_metadata: "RunMetadata | None" = None,+ ):# We copy the incoming properties to avoid modifying the original dictionary# and to convert the keys to str# and to remove the properties that are in the redact list@@ -248,45 +440,78 @@ class TelemetryManager(TelemetryManagerAbstract):# Track to custom PostHog based on user's posthog.modematch self.telemetry_config.custom_posthog.mode:case PostHogMode.ANONYMOUS:- self._track_anonymous_event(event_name=event_name, properties=tracked_properties)+ # The operator chose not to identify people, and that choice covers+ # their users too: no run identity is applied and no groups are sent.+ self._capture_custom_event(event_name, properties=tracked_properties, identity=TelemetryIdentity.make_anonymous())case PostHogMode.IDENTIFIED:if not self.telemetry_config.custom_posthog.user_id:log.error(f"Could not track event '{event_name}' as identified because user_id is not set, tracking as anonymous")- self._track_anonymous_event(event_name=event_name, properties=tracked_properties)+ self._capture_custom_event(event_name, properties=tracked_properties, identity=TelemetryIdentity.make_anonymous())else:- self._track_identified_event(- event_name=event_name,+ self._capture_custom_event(+ event_name,properties=tracked_properties,- user_id=self.telemetry_config.custom_posthog.user_id,+ identity=TelemetryIdentity.make_from_run_metadata(+ run_metadata=run_metadata,+ fallback_distinct_id=self.telemetry_config.custom_posthog.user_id,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ ),)case PostHogMode.OFF:log.verbose(f"Custom telemetry is off, skipping event '{event_name}' for custom client")# Always track to Pipelex PostHog if enabled (independent of posthog.mode)if self._pipelex_telemetry_enabled:- self._track_to_pipelex(event_name=event_name, properties=tracked_properties)+ self._track_to_pipelex(event_name, properties=tracked_properties, run_metadata=run_metadata)- def _track_anonymous_event(self, event_name: str, *, properties: dict[str, Any]):- if not self.custom_posthog_client:- log.error("Could not track event to custom telemetry because custom_posthog_client is not set")- return- properties[PostHogAttr.PROCESS_PERSON_PROFILE] = False- self.custom_posthog_client.capture(event_name, properties=properties)- log.verbose(f"Tracked anonymous event '{event_name}' with properties: {properties}")+ def _capture_custom_event(self, event_name: str, *, properties: dict[str, Any], identity: TelemetryIdentity):+ """Capture one event on the operator's stream, under the identity resolved for it.- def _track_identified_event(self, event_name: str, *, properties: dict[str, Any], user_id: str):+ Captures a COPY of the properties, because the same dict is offered to+ both streams and the anonymous branch below stamps+ `$process_person_profile` on the one it sends. Without the copy that mark+ travelled on to the Pipelex capture, labelling an event sent WITH a+ distinct_id as having no person profile.+ """if not self.custom_posthog_client:log.error("Could not track event to custom telemetry because custom_posthog_client is not set")return- self.custom_posthog_client.capture(event_name, distinct_id=user_id, properties=properties)- log.verbose(f"Tracked identified event '{event_name}' with properties: {properties}")+ capture_properties = dict(properties)+ if identity.distinct_id:+ self.custom_posthog_client.capture(+ event_name,+ distinct_id=identity.distinct_id,+ properties=capture_properties,+ groups=identity.groups or None,+ )+ log.verbose(f"Tracked identified event '{event_name}' with properties: {capture_properties}")+ else:+ capture_properties[PostHogAttr.PROCESS_PERSON_PROFILE] = False+ self.custom_posthog_client.capture(event_name, properties=capture_properties)+ log.verbose(f"Tracked anonymous event '{event_name}' with properties: {capture_properties}")- def _track_to_pipelex(self, event_name: str, *, properties: dict[str, Any]):- """Track event to Pipelex's PostHog (always identified)."""+ def _track_to_pipelex(self, event_name: str, *, properties: dict[str, Any], run_metadata: "RunMetadata | None" = None):+ """Track event to Pipelex's PostHog (always identified, always namespaced).++ The stream has no mode to turn identification off, but it is a shared+ project: a run's own user reaches it only as a digest taken inside the+ gateway-key hash, and the hash itself is what an event with no run+ reports under. The run's groups do not travel here at all.+ """if not self.pipelex_posthog_client or not self._pipelex_distinct_id:log.error("Could not track event to Pipelex telemetry because pipelex_posthog_client or _pipelex_distinct_id is not set")return- self.pipelex_posthog_client.capture(event_name, distinct_id=self._pipelex_distinct_id, properties=properties)+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=run_metadata,+ fallback_distinct_id=self._pipelex_distinct_id,+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ self.pipelex_posthog_client.capture(+ event_name,+ distinct_id=identity.distinct_id,+ properties=dict(properties),+ groups=identity.groups or None,+ )log.verbose(f"Tracked event '{event_name}' to Pipelex telemetry")@override@@ -390,7 +615,7 @@ class TelemetryManager(TelemetryManagerAbstract):return self._pipelex_telemetry_enabled@override- def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int) -> None:+ def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int, run_metadata: "RunMetadata | None" = None) -> None:"""Hook to do something when a trace starts.Emits a trace start event to establish the trace name in PostHog.@@ -402,6 +627,10 @@ class TelemetryManager(TelemetryManagerAbstract):trace_name: Full trace name with pipe code (for custom telemetry).trace_name_redacted: Redacted trace name without pipe code (for Pipelex telemetry).trace_id: The trace ID.+ run_metadata: The run this trace belongs to. Carries the identity the+ event is attributed to, resolved exactly as every pipe span under+ the same trace will be, so the trace's first event and its spans+ land on the same person. None leaves the stream's fallback."""log.verbose(f"[Telemetry] Emitting trace start event:\n"@@ -419,11 +648,17 @@ class TelemetryManager(TelemetryManagerAbstract):PostHogAttr.SPAN_NAME: custom_trace_name,PostHogAttr.TRACE_NAME: custom_trace_name,}- if self.telemetry_config.custom_posthog.user_id:+ custom_identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=run_metadata,+ fallback_distinct_id=self.telemetry_config.custom_posthog.user_id,+ run_identity_policy=RunIdentityPolicy.make_for_operator_stream(posthog_mode=self.telemetry_config.custom_posthog.mode),+ )+ if custom_identity.distinct_id:self.custom_posthog_client.capture(- distinct_id=self.telemetry_config.custom_posthog.user_id,+ distinct_id=custom_identity.distinct_id,event=PostHogEvent.SPAN,properties=custom_properties,+ groups=custom_identity.groups or None,)else:custom_properties[PostHogAttr.PROCESS_PERSON_PROFILE] = False@@ -439,11 +674,17 @@ class TelemetryManager(TelemetryManagerAbstract):PostHogAttr.SPAN_NAME: trace_name_redacted,PostHogAttr.TRACE_NAME: trace_name_redacted,}- if self._pipelex_distinct_id:+ pipelex_identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=run_metadata,+ fallback_distinct_id=self._pipelex_distinct_id,+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ if pipelex_identity.distinct_id:self.pipelex_posthog_client.capture(- distinct_id=self._pipelex_distinct_id,+ distinct_id=pipelex_identity.distinct_id,event=PostHogEvent.SPAN,properties=pipelex_properties,+ groups=pipelex_identity.groups or None,)else:pipelex_properties[PostHogAttr.PROCESS_PERSON_PROFILE] = Falsediff --git a/pipelex/system/telemetry/telemetry_manager_abstract.py b/pipelex/system/telemetry/telemetry_manager_abstract.pyindex 1c1ffbc59..2b18aae8f 100644--- a/pipelex/system/telemetry/telemetry_manager_abstract.py+++ b/pipelex/system/telemetry/telemetry_manager_abstract.py@@ -12,6 +12,8 @@ if TYPE_CHECKING:# Deferred import: avoid pulling heavy SDK at module-load timefrom opentelemetry.trace import Tracer as OTelTracer+ from pipelex.system.job_metadata import RunMetadata+class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):@classmethod@@ -94,8 +96,24 @@ class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):pass@abstractmethod- def track_event(self, event_name: EventName, *, properties: dict[EventProperty, Any] | None = None):- pass+ def track_event(+ self,+ event_name: EventName,+ *,+ properties: dict[EventProperty, Any] | None = None,+ run_metadata: "RunMetadata | None" = None,+ ):+ """Track one product event.++ Args:+ event_name: The event.+ properties: Free scalars on the event, minus whatever the operator redacts.+ run_metadata: The run that produced the event, when there is one. It carries+ the identity the event is attributed to — see+ :mod:`pipelex.system.telemetry.telemetry_identity`. An emitter that holds+ a ``PipeJob`` passes ``pipe_job.job_metadata.run_metadata``; a CLI command+ or a dry-run sweep passes nothing and reports under the configured id.+ """@abstractmethod@contextmanager@@ -153,13 +171,15 @@ class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):"""Whether Pipelex internal telemetry is enabled (for gateway usage)."""@abstractmethod- def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int) -> None:+ def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int, run_metadata: "RunMetadata | None" = None) -> None:"""Hook to do something when a trace starts.Args:trace_name: Full trace name with pipe code (for custom telemetry).trace_name_redacted: Redacted trace name without pipe code (for Pipelex telemetry).trace_id: The trace ID.+ run_metadata: The run this trace belongs to, carrying the identity the+ event is attributed to. None leaves the stream's fallback."""@@ -174,7 +194,13 @@ class TelemetryManagerNoOp(TelemetryManagerAbstract):TelemetryManagerAbstract.clear_instance()@override- def track_event(self, event_name: EventName, *, properties: dict[EventProperty, Any] | None = None):+ def track_event(+ self,+ event_name: EventName,+ *,+ properties: dict[EventProperty, Any] | None = None,+ run_metadata: "RunMetadata | None" = None,+ ):pass@override@@ -234,5 +260,5 @@ class TelemetryManagerNoOp(TelemetryManagerAbstract):return False@override- def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int) -> None:+ def handle_trace_start(self, *, trace_name: str, trace_name_redacted: str, trace_id: int, run_metadata: "RunMetadata | None" = None) -> None:passdiff --git a/subject_grants.toml b/subject_grants.tomlindex 87e03afa1..b7dd51238 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -5872,13 +5872,9 @@ rationale = "Predicate on the integration mode; single operand."param = "posthog_config"rationale = "Factory from source: builds from the posthog config; single operand."-["pipelex/system/telemetry/telemetry_manager.py::TelemetryManager._track_anonymous_event"]+["pipelex/system/telemetry/telemetry_manager.py::TelemetryManager._capture_custom_event"]param = "event_name"-rationale = "Verb-object: tracks the event named; properties stay keyword."--["pipelex/system/telemetry/telemetry_manager.py::TelemetryManager._track_identified_event"]-param = "event_name"-rationale = "Verb-object: tracks the event named; properties stay keyword."+rationale = "Verb-object: captures the event named; properties and identity stay keyword."["pipelex/system/telemetry/telemetry_manager.py::TelemetryManager._track_to_pipelex"]param = "event_name"diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_analytics_groups.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_analytics_groups.pyindex 68841455e..351f8ee69 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_analytics_groups.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_analytics_groups.py@@ -4,15 +4,18 @@ onto :class:`JobMetadata`.The groups ride the ``PipeJob``(``arg.pipe_job.job_metadata.run_metadata.analytics_groups``) — the same payload-firstroute ``request_id`` takes, and for the same reason: there is no ContextVar layer-and the worker on the far side of a Temporal hop rehydrates the whole job. No-exporter reads them yet; this test pins the dispatcher-side contract, which is-what the hosted runner will call once it accepts the field on the wire.+and the worker on the far side of a Temporal hop rehydrates the whole job. The+PostHog exporters read them back off the span attributes; this test pins the+dispatcher-side contract, which is what the hosted runner will call once it+accepts the field on the wire.It also pins WHERE the refusal happens. Validation sits above-``add_new_pipeline`` and above ``handle_trace_start`` deliberately: a malformed-mapping must not register a run or emit a trace-start event that cannot be-unsent. Asserting only that a ``ValueError`` escapes would pass with the-validation at the bottom of ``prepare_pipe_job``, which is where it used to be.+``add_new_pipeline`` deliberately: a malformed mapping must not register a run.+Asserting only that a ``ValueError`` escapes would pass with the validation at+the bottom of ``prepare_pipe_job``, which is where it used to be. It sits above+``handle_trace_start`` too, but that clause no longer distinguishes anything —+the trace-start has since moved below ``prepare_pipe_job``, so every gate in+this function precedes it."""import pytest@@ -91,9 +94,12 @@ class TestPipelineRunSetupAnalyticsGroups:"""A rejected mapping must leave NOTHING behind — not a pipeline entry, not a telemetry event.The teardown can release a registered pipeline and an opened tracer; it cannot- unsend a ``handle_trace_start`` that already reached the backend. So the gate has- to sit above both, and this test fails if it is moved back down into- ``prepare_pipe_job`` where the mapping was first validated.+ unsend a ``handle_trace_start`` that already reached the backend. The pipeline+ half is what bites today: ``add_new_pipeline`` runs above ``prepare_pipe_job``,+ so this test fails if the gate is moved back down there, where the mapping was+ first validated. The telemetry half is now satisfied by the ordering whatever+ the gate does — the trace-start sits below ``prepare_pipe_job`` — and stays as a+ guard against that ordering moving back."""execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False,diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.pyindex ba8515067..18d28a5eb 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_storage_scope_gate.py@@ -3,10 +3,12 @@The scope decides where a tenant's bytes land, so a traversal in it is thehighest-stakes value this function takes. It was validated only inside`prepare_pipe_job`, which runs below the pipeline registration, the library-acquire, the tracer open and `handle_trace_start` — so a scope carrying `..`-registered a run and emitted a trace event that cannot be unsent before-anything looked at it. These pin the two gates that close that window, and both-fail if either gate is removed.+acquire and the tracer open — so a scope carrying `..` registered a run before+anything looked at it. It ran below `handle_trace_start` as well, and emitted a+trace event that cannot be unsent; the trace-start has since moved beneath+`prepare_pipe_job`, so that half of the window is now closed by the ordering+itself. These pin the two gates that close the rest of it, and both fail if+either gate is removed.The lower gate in `prepare_pipe_job` is NOT replaced by these and must stay: thesentinel path rebinds the scope to the caller's `pipeline_run_id`, and only the@@ -71,7 +73,9 @@ class TestPipelineRunSetupStorageScopeGate:The first gate sees only `LOCAL_STORAGE_SCOPE`, which is a valid scope, sothis is the first look at what the scope actually became — and it still- has to happen before the library, the tracer and the trace-start.+ has to happen before the library and the tracer. Before the trace-start+ too, though that now follows `prepare_pipe_job` and so asks nothing of+ this gate."""execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)telemetry_manager = mocker.spy(pipeline_run_setup_module, "get_telemetry_manager")diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.pynew file mode 100644index 000000000..ee4bd873b--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_telemetry_identity.py@@ -0,0 +1,82 @@+"""``pipeline_run_setup`` hands the run's identity to the two telemetry emissions it makes.++The trace-start event and ``pipeline_execute`` are the only telemetry this seam+emits, and both used to report under the process identity — the trace-start under+the configured id or the gateway hash, and the event under whichever the stream+resolved. Both now name the run, which is what makes a trace and its spans land+on one person instead of two.++It also pins WHERE the trace-start is emitted. It sits below ``prepare_pipe_job``+because that seam builds the run's ``RunMetadata``, and the event has to be+attributed to the same object its spans will be rather than a second one+assembled from the same parameters. Asserting only that the call happened would+pass with it back above, unattributed.+"""++from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.pipeline import pipeline_run_setup as pipeline_run_setup_module+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup+from pipelex.system.telemetry.events import EventName++_MINIMAL_MTHDS = """+domain = "prs_telemetry_identity"+description = "Minimal bundle for the pipeline_run_setup telemetry identity test"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = { type = "text", description = "Topic name" }++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used only to set up a PipeJob"+inputs = { subject = "Text" }+output = "Topic"+prompt = "Echo the $subject as a topic"+"""+++async def _run_setup(*, mocker: MockerFixture) -> Any:+ """Run the seam with a stubbed telemetry manager and a tracer present, and return it."""+ telemetry_manager = mocker.MagicMock()+ mocker.patch.object(pipeline_run_setup_module, "get_telemetry_manager", return_value=telemetry_manager)+ # The trace-start is gated on a tracer existing; the test environment has none.+ mocker.patch.object(pipeline_run_setup_module, "get_otel_tracer", return_value=mocker.MagicMock())++ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)+ await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="user-42",+ execution_config=execution_config,+ mthds_contents=[_MINIMAL_MTHDS],+ pipe_code="echo_topic",+ inputs={"subject": "a subject"},+ analytics_groups={"organization": "org_acme"},+ )+ return telemetry_manager++[email protected](loop_scope="class")+class TestPipelineRunSetupTelemetryIdentity:+ async def test_the_trace_start_is_handed_the_runs_metadata(self, mocker: MockerFixture) -> None:+ telemetry_manager = await _run_setup(mocker=mocker)++ telemetry_manager.handle_trace_start.assert_called_once()+ run_metadata = telemetry_manager.handle_trace_start.call_args.kwargs["run_metadata"]+ assert run_metadata.user_id == "user-42"+ assert run_metadata.analytics_groups == {"organization": "org_acme"}++ async def test_the_pipeline_execute_event_is_handed_the_runs_metadata(self, mocker: MockerFixture) -> None:+ telemetry_manager = await _run_setup(mocker=mocker)++ execute_calls = [call for call in telemetry_manager.track_event.call_args_list if call.kwargs.get("event_name") == EventName.PIPELINE_EXECUTE]+ assert len(execute_calls) == 1+ run_metadata = execute_calls[0].kwargs["run_metadata"]+ assert run_metadata.user_id == "user-42"+ assert run_metadata.analytics_groups == {"organization": "org_acme"}diff --git a/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py b/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.pynew file mode 100644index 000000000..6a15f7e75--- /dev/null+++ b/tests/integration/pipelex/system/telemetry/test_pipe_span_carries_run_identity.py@@ -0,0 +1,100 @@+"""The pipe execution span carries the run's identity.++The companion of the LLM-span unit test, but driven through a real library: the+pipe comes out of ``pipeline_run_setup``, so this pins the whole dispatcher-side+route — the groups the caller supplied reach ``RunMetadata``, and the span site+reads them off it — rather than a hand-built metadata object.+"""++from typing import Any++import pytest+from opentelemetry.sdk.trace import Span as SdkSpan+from opentelemetry.sdk.trace import TracerProvider+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup+from pipelex.system.job_metadata import OtelContext+from pipelex.system.telemetry.otel_constants import LangfuseSpanAttr, OTelConstants, PipelexSpanAttr+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract++_MINIMAL_MTHDS = """+domain = "pipe_span_identity_test"+description = "Minimal bundle for pipe-span identity test"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = { type = "text", description = "Topic name" }++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used only to build a span"+inputs = { subject = "Text" }+output = "Topic"+prompt = "Echo the $subject as a topic"+"""++_PARENT_OTEL_CONTEXT = OtelContext(+ trace_id=1234,+ trace_name="echo_topic_abc12345",+ trace_name_redacted="abc12345",+ span_id=OTelConstants.OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID,+)+++async def _make_pipe_job(*, analytics_groups: dict[str, str] | None = None) -> PipeJob:+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)+ pipe_job, _, _ = await pipeline_run_setup(+ storage_scope="test/scope",+ user_id="user-42",+ execution_config=execution_config,+ mthds_contents=[_MINIMAL_MTHDS],+ pipe_code="echo_topic",+ analytics_groups=analytics_groups,+ inputs={"subject": "a subject"},+ )+ return pipe_job+++def _start_span(*, mocker: MockerFixture, pipe_job: PipeJob) -> SdkSpan:+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=TracerProvider().get_tracer("test"))+ span, _ = pipe_job.pipe._start_pipe_span( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ parent_otel_context=_PARENT_OTEL_CONTEXT,+ run_metadata=pipe_job.job_metadata.run_metadata,+ working_memory=pipe_job.get_working_memory(),+ )+ assert isinstance(span, SdkSpan)+ return span+++def _attributes(span: SdkSpan) -> dict[str, Any]:+ return dict(span.attributes or {})++[email protected](loop_scope="class")+class TestPipeSpanIdentity:+ async def test_the_span_names_the_runs_user(self, mocker: MockerFixture) -> None:+ pipe_job = await _make_pipe_job()++ assert _attributes(_start_span(mocker=mocker, pipe_job=pipe_job))[PipelexSpanAttr.RUN_USER_ID] == "user-42"++ async def test_the_groups_the_caller_supplied_reach_the_span(self, mocker: MockerFixture) -> None:+ pipe_job = await _make_pipe_job(analytics_groups={"organization": "org_acme"})++ attributes = _attributes(_start_span(mocker=mocker, pipe_job=pipe_job))+ assert attributes[PipelexSpanAttr.RUN_ANALYTICS_GROUPS] == '{"organization": "org_acme"}'++ async def test_a_run_with_no_groups_writes_no_groups_attribute(self, mocker: MockerFixture) -> None:+ pipe_job = await _make_pipe_job()++ assert PipelexSpanAttr.RUN_ANALYTICS_GROUPS not in _attributes(_start_span(mocker=mocker, pipe_job=pipe_job))++ async def test_langfuse_gets_the_user_id_when_it_is_enabled(self, mocker: MockerFixture) -> None:+ pipe_job = await _make_pipe_job()+ mocker.patch.object(TelemetryManagerAbstract, "get_langfuse_enabled", return_value=True)++ assert _attributes(_start_span(mocker=mocker, pipe_job=pipe_job))[LangfuseSpanAttr.USER_ID] == "user-42"diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run.py b/tests/unit/pipelex/pipe_run/test_pipe_run.pyindex 85f2eb246..89a2b8c7f 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_run.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_run.py@@ -57,8 +57,14 @@ class TestPipeRun:# A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it# onto the output, and `assemble_tracing_on_output` then feeds its# `run_metadata` into `TracingAssembly`, which is a typed field.- mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-req", storage_scope="test/scope"))- mock_job.job_metadata.run_metadata.request_id = "req-direct-mode"+ mock_job.job_metadata = JobMetadata(+ run_metadata=RunMetadata(+ user_id="pytest",+ pipeline_run_id="plr-req",+ storage_scope="test/scope",+ request_id="req-direct-mode",+ )+ )pipe_run = PipeRun(pipe_router=mock_router)assignment = DeliveryAssignment(storage=StorageTarget())diff --git a/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py b/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.pyindex 4f4fc92dd..2af52fa0a 100644--- a/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py+++ b/tests/unit/pipelex/pipeline/test_runner_execute_error_paths.py@@ -111,6 +111,8 @@ class TestRunnerExecuteErrorPaths:assert wrapped.run_mode == PipeRunMode.DRYassert wrapped.pipe_stack == ["root_pipe", "echo_pipe"]assert wrapped.__cause__ is original+ # The failure event is attributed to the run, not to the process: `run_metadata`+ # is the job's own, which is what carries the caller's user id and groups.env.telemetry_manager.track_event.assert_called_once_with(event_name=EventName.PIPELINE_COMPLETE,properties={@@ -118,6 +120,7 @@ class TestRunnerExecuteErrorPaths:EventProperty.PIPE_TYPE: "PipeLLM",EventProperty.PIPELINE_OUTCOME: Outcome.FAILURE,},+ run_metadata=env.pipe_job.job_metadata.run_metadata,)async def test_pipelex_error_during_setup_propagates_unwrapped(self, mocker: MockerFixture) -> None:diff --git a/tests/unit/pipelex/system/telemetry/test_exception_capture_identity.py b/tests/unit/pipelex/system/telemetry/test_exception_capture_identity.pynew file mode 100644index 000000000..c5b49da72--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_exception_capture_identity.py@@ -0,0 +1,152 @@+"""Unit tests for the identity an unhandled exception is captured under.++An `$exception` is the one capture no run can hand its metadata to: the two+interpreter hooks receive `(type, value, traceback)` and nothing else, and there+is no ContextVar layer for them to read a run out of. So what each stream sends+is its runless identity, resolved once when the capture is built — and these+tests are about that resolution reaching this path, because for a while it did+not: the operator's configured `user_id` went out on every crash, `anonymous`+mode included.++The manager is built WITHOUT its constructor, as in the tracker tests beside+these, because `TelemetryManager.__init__` creates live PostHog clients, writes+`posthog.default_client` and registers a process-wide singleton. The wiring is+still what is under test: `_make_exception_capture` is the constructor's own+call, exercised here against stub clients. It installs the interpreter hooks, so+every test closes the capture again.+"""++import sys+import threading+from typing import Any, Generator++import pytest+from posthog import Posthog+from pytest_mock import MockerFixture++from pipelex.system.telemetry.exception_capture import DualClientExceptionCapture+from pipelex.system.telemetry.otel_constants import PostHogAttr+from pipelex.system.telemetry.telemetry_config import PostHogConfig, PostHogMode, TelemetryConfig+from pipelex.system.telemetry.telemetry_identity import PIPELEX_DEPLOYMENT_GROUP_TYPE+from pipelex.system.telemetry.telemetry_manager import TelemetryManager+++def restore_excepthooks() -> Generator[None, None, None]:+ """Give the interpreter back the hooks a capture installs when it is built."""+ original_excepthook = sys.excepthook+ original_threading_excepthook = threading.excepthook+ yield+ sys.excepthook = original_excepthook+ threading.excepthook = original_threading_excepthook+++def _make_capture(+ *,+ mocker: MockerFixture,+ mode: PostHogMode,+ configured_user_id: str | None,+ pipelex_distinct_id: str | None = None,+) -> tuple[DualClientExceptionCapture, Any, Any]:+ """Resolve the capture the constructor would have built, over stub clients."""+ manager = TelemetryManager.__new__(TelemetryManager)+ manager.telemetry_config = TelemetryConfig(+ custom_posthog=PostHogConfig(mode=mode, user_id=configured_user_id, api_key="phc_test"),+ )+ custom_client = mocker.MagicMock(spec=Posthog)+ pipelex_client = mocker.MagicMock(spec=Posthog) if pipelex_distinct_id else None+ manager.custom_posthog_client = custom_client+ manager.pipelex_posthog_client = pipelex_client+ manager._pipelex_distinct_id = pipelex_distinct_id # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ capture = manager._make_exception_capture() # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ return capture, custom_client, pipelex_client+++def _crash(*, capture: DualClientExceptionCapture) -> None:+ capture._capture_exception((ValueError, ValueError("boom"), None)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++[email protected]("restore_excepthooks")+class TestExceptionCaptureIdentity:+ def test_an_anonymous_stream_captures_an_exception_under_nobody(self, mocker: MockerFixture) -> None:+ """A mode that identifies nobody must not identify somebody when the process crashes.++ The configured `user_id` is deliberately left in place: the config+ validator requires one only for `identified` and never forbids one under+ `anonymous`, so a stale id after a mode switch is the reachable state.+ """+ capture, custom_client, _ = _make_capture(+ mocker=mocker,+ mode=PostHogMode.ANONYMOUS,+ configured_user_id="configured-id",+ )++ _crash(capture=capture)++ capture_kwargs = custom_client.capture_exception.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_an_identified_stream_captures_an_exception_under_the_configured_id(self, mocker: MockerFixture) -> None:+ """An exception belongs to no run, so the operator's stream reports it under the fallback."""+ capture, custom_client, _ = _make_capture(+ mocker=mocker,+ mode=PostHogMode.IDENTIFIED,+ configured_user_id="configured-id",+ )++ _crash(capture=capture)++ assert custom_client.capture_exception.call_args.kwargs["distinct_id"] == "configured-id"++ def test_the_pipelex_stream_captures_an_exception_under_its_deployment(self, mocker: MockerFixture) -> None:+ """No run means no fold, so the gateway hash is both the person and the group."""+ capture, _, pipelex_client = _make_capture(+ mocker=mocker,+ mode=PostHogMode.IDENTIFIED,+ configured_user_id="configured-id",+ pipelex_distinct_id="gateway-hash",+ )++ _crash(capture=capture)++ capture_kwargs = pipelex_client.capture_exception.call_args.kwargs+ assert capture_kwargs["distinct_id"] == "gateway-hash"+ assert capture_kwargs["groups"] == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_the_pipelex_stream_is_unaffected_by_the_operators_anonymous_mode(self, mocker: MockerFixture) -> None:+ """The two streams answer to different settings, and anonymity on one is not anonymity on the other."""+ capture, _, pipelex_client = _make_capture(+ mocker=mocker,+ mode=PostHogMode.ANONYMOUS,+ configured_user_id="configured-id",+ pipelex_distinct_id="gateway-hash",+ )++ _crash(capture=capture)++ assert pipelex_client.capture_exception.call_args.kwargs["distinct_id"] == "gateway-hash"++ def test_an_exception_carrying_no_value_is_not_captured(self, mocker: MockerFixture) -> None:+ """`threading.excepthook` can hand over a null value, and a null exception is not an event."""+ capture, custom_client, _ = _make_capture(+ mocker=mocker,+ mode=PostHogMode.IDENTIFIED,+ configured_user_id="configured-id",+ )++ capture._capture_exception((ValueError, None, None)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ custom_client.capture_exception.assert_not_called()++ def test_a_failing_capture_never_reaches_the_app(self, mocker: MockerFixture) -> None:+ """Telemetry must not turn one crash into two."""+ capture, custom_client, _ = _make_capture(+ mocker=mocker,+ mode=PostHogMode.IDENTIFIED,+ configured_user_id="configured-id",+ )+ custom_client.capture_exception.side_effect = RuntimeError("posthog is down")++ _crash(capture=capture)diff --git a/tests/unit/pipelex/system/telemetry/test_exception_capture_sdk_contract.py b/tests/unit/pipelex/system/telemetry/test_exception_capture_sdk_contract.pynew file mode 100644index 000000000..3ec231c4b--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_exception_capture_sdk_contract.py@@ -0,0 +1,342 @@+"""Unit tests for what the real PostHog SDK does with the crash this class hands it.++The module beside this one pins the identity each stream resolves, and it does+so against `MagicMock(spec=Posthog)` — which is the right stub for that question+and a blind one for this. Two defects lived underneath those mocks: the SDK+drops a capture of an error it has already sent, so whichever stream went second+received nothing; and the privacy wrapper redacted a `PipelexError` handed over+bare but not the `(type, value, traceback)` triple an excepthook is given, which+is the only form this path has. A mock answers neither, because neither happens+until the SDK runs.++So these tests use real `Posthog` clients with only `capture` stubbed — the one+call that would put an event on the wire. Everything above it is the production+article: the wrapper the manager installs, the SDK's own deduplication, and the+event the SDK builds out of the triple. `sync_mode=True` keeps the client from+starting its consumer threads, and the stub returns an event id because the SDK+marks an error as captured only when the send reported one.+"""++import sys+import threading+from typing import Any, Generator++import pytest+from posthog import Posthog+from pytest_mock import MockerFixture++from pipelex.system.exceptions import ToolError+from pipelex.system.telemetry.exception_capture import DualClientExceptionCapture+from pipelex.system.telemetry.telemetry_config import PostHogConfig, PostHogMode, TelemetryConfig+from pipelex.system.telemetry.telemetry_manager import TelemetryManager++_CONFIDENTIAL_MESSAGE = "could not read patient record 123-45-6789 at /home/acme/confidential.pdf"+++def restore_excepthooks() -> Generator[None, None, None]:+ """Give the interpreter back the hooks a capture installs when it is built."""+ original_excepthook = sys.excepthook+ original_threading_excepthook = threading.excepthook+ yield+ sys.excepthook = original_excepthook+ threading.excepthook = original_threading_excepthook+++def _make_stubbed_client(*, mocker: MockerFixture, api_key: str) -> tuple[Posthog, Any]:+ """A real client whose only stubbed method is the one that would reach the network."""+ client = Posthog(project_api_key=api_key, host="http://localhost:1", sync_mode=True, disabled=False)+ capture = mocker.patch.object(client, "capture", return_value=f"{api_key}-event-id")+ return client, capture+++def _make_dual_capture(*, custom_client: Posthog, pipelex_client: Posthog) -> DualClientExceptionCapture:+ """Assemble the capture the constructor would have built, over the two given clients.++ The manager is built without its constructor, as in the module beside this+ one, because `TelemetryManager.__init__` creates live clients, writes+ `posthog.default_client` and registers a process-wide singleton. The two+ steps under test are its own: wrapping each client for privacy, then+ resolving each stream's runless identity.+ """+ manager = TelemetryManager.__new__(TelemetryManager)+ manager.telemetry_config = TelemetryConfig(+ custom_posthog=PostHogConfig(mode=PostHogMode.IDENTIFIED, user_id="operator-user", api_key="phc_custom"),+ )+ manager.custom_posthog_client = custom_client+ manager.pipelex_posthog_client = pipelex_client+ manager._pipelex_distinct_id = "gateway-hash" # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ manager._wrap_capture_exception(custom_client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ manager._wrap_capture_exception(pipelex_client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ return manager._make_exception_capture() # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _crash(*, capture: DualClientExceptionCapture, error: BaseException) -> None:+ """Hand the capture the triple an excepthook would, which is all it ever receives."""+ capture._capture_exception((type(error), error, None)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _sent_exception_values(*, capture_mock: Any) -> str:+ """Render the `$exception_list` of the one event this client was asked to send."""+ return repr(capture_mock.call_args.kwargs["properties"]["$exception_list"])++[email protected]("restore_excepthooks")+class TestExceptionCaptureSdkContract:+ def test_both_streams_record_one_crash(self, mocker: MockerFixture) -> None:+ """Both streams record the crash, which is the whole point of capturing to two.++ A `PipelexError` is handed to each client as its own redacted stand-in,+ so the SDK never sees one object twice on this path and its+ deduplication has nothing to bite on. The crash with no stand-in is the+ test below, and that is the one the clearing is there for.+ """+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ToolError(_CONFIDENTIAL_MESSAGE))++ assert custom_capture.call_count == 1+ assert pipelex_capture.call_count == 1++ def test_both_streams_record_a_crash_that_is_not_a_pipelex_error(self, mocker: MockerFixture) -> None:+ """The SDK drops an error it has already sent, and both streams are handed the same object.++ A `ValueError` is never redacted and so never copied: it is the object+ PostHog stamps, and the stamp is still on it when the second stream+ asks. Left in place, the stream that goes second — always Pipelex's, on+ a machine that also reports to an operator's own project — returns+ before building an event, and the deployment group it resolved is+ recorded nowhere.+ """+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ValueError("boom"))++ assert custom_capture.call_count == 1+ assert pipelex_capture.call_count == 1++ def test_no_stream_is_sent_the_message_as_it_was_written(self, mocker: MockerFixture) -> None:+ """A `PipelexError` message repeats what the caller passed in, so it does not travel.++ The redaction was written for an error handed over bare and was never+ reached by the triple, which is the only form an excepthook produces —+ so the one path that captures without anybody asking was the one path+ sending the message verbatim.+ """+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ToolError(_CONFIDENTIAL_MESSAGE))++ for capture_mock in (custom_capture, pipelex_capture):+ sent = _sent_exception_values(capture_mock=capture_mock)+ assert _CONFIDENTIAL_MESSAGE not in sent+ assert TelemetryManager.PRIVACY_NOTICE in sent++ def test_the_stand_in_keeps_the_class_that_failed(self, mocker: MockerFixture) -> None:+ """Redaction costs the message, not the diagnosis: the error's own class still travels."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, _ = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ToolError(_CONFIDENTIAL_MESSAGE))++ assert ToolError.__name__ in _sent_exception_values(capture_mock=custom_capture)++ def test_each_stream_is_recorded_under_its_own_identity(self, mocker: MockerFixture) -> None:+ """Both streams emitting is only worth having if each still emits as itself."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ToolError(_CONFIDENTIAL_MESSAGE))++ assert custom_capture.call_args.kwargs["distinct_id"] == "operator-user"+ assert pipelex_capture.call_args.kwargs["distinct_id"] == "gateway-hash"++ def test_a_pipelex_error_raised_as_the_cause_is_redacted(self, mocker: MockerFixture) -> None:+ """PostHog sends every error it reaches through `__cause__`, not only the one at the top."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ try:+ try:+ raise ToolError(_CONFIDENTIAL_MESSAGE)+ except ToolError as pipelex_error:+ wrapper_message = "wrapper"+ raise RuntimeError(wrapper_message) from pipelex_error+ except RuntimeError as crash:+ _crash(capture=capture, error=crash)++ for capture_mock in (custom_capture, pipelex_capture):+ sent = _sent_exception_values(capture_mock=capture_mock)+ assert _CONFIDENTIAL_MESSAGE not in sent+ assert TelemetryManager.PRIVACY_NOTICE in sent+ assert "wrapper" in sent++ def test_a_pipelex_error_being_handled_when_another_is_raised_is_redacted(self, mocker: MockerFixture) -> None:+ """An error raised inside `except PipelexError` carries it as `__context__`, which PostHog sends too."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ try:+ try:+ raise ToolError(_CONFIDENTIAL_MESSAGE)+ except ToolError:+ handling_message = "while handling"+ raise RuntimeError(handling_message) # ruff: ignore[raise-without-from-inside-except] — the implicit __context__ is what this test exercises+ except RuntimeError as crash:+ _crash(capture=capture, error=crash)++ for capture_mock in (custom_capture, pipelex_capture):+ assert _CONFIDENTIAL_MESSAGE not in _sent_exception_values(capture_mock=capture_mock)++ def test_a_pipelex_error_inside_an_exception_group_is_redacted(self, mocker: MockerFixture) -> None:+ """PostHog expands an exception group's members, and a `TaskGroup` wraps whatever its tasks raised."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)++ _crash(capture=capture, error=ExceptionGroup("tasks failed", [ToolError(_CONFIDENTIAL_MESSAGE), ValueError("other")]))++ for capture_mock in (custom_capture, pipelex_capture):+ sent = _sent_exception_values(capture_mock=capture_mock)+ assert _CONFIDENTIAL_MESSAGE not in sent+ assert TelemetryManager.PRIVACY_NOTICE in sent++ def test_a_capture_with_no_argument_is_redacted(self, mocker: MockerFixture) -> None:+ """Given `None`, the SDK reads `sys.exc_info()` itself, past any wrapper that did not resolve it first.++ `None` is what the module-level `posthog.capture_exception()` passes.+ """+ client, capture_mock = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ manager = TelemetryManager.__new__(TelemetryManager)+ manager._wrap_capture_exception(client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ try:+ raise ToolError(_CONFIDENTIAL_MESSAGE)+ except ToolError:+ client.capture_exception(None)++ assert _CONFIDENTIAL_MESSAGE not in _sent_exception_values(capture_mock=capture_mock)++ def test_the_live_exception_is_left_as_it_was(self, mocker: MockerFixture) -> None:+ """Only a copy of the chain is redacted: the interpreter prints the original after the hook returns."""+ custom_client, _ = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, _ = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)+ pipelex_error = ToolError(_CONFIDENTIAL_MESSAGE)+ crash = RuntimeError("wrapper")+ crash.__cause__ = pipelex_error++ _crash(capture=capture, error=crash)++ assert crash.__cause__ is pipelex_error+ assert str(pipelex_error) == _CONFIDENTIAL_MESSAGE++ def test_an_error_the_host_already_captured_is_not_sent_again(self, mocker: MockerFixture) -> None:+ """A host that captures an error and re-raises it has sent it once, and the hook must not add a second."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)+ error = ValueError("boom")++ custom_client.capture_exception(error)+ _crash(capture=capture, error=error)++ assert custom_capture.call_count == 1+ assert pipelex_capture.call_count == 0++ def test_a_redacted_error_the_host_already_captured_is_not_sent_again(self, mocker: MockerFixture) -> None:+ """The SDK marks the redacted copy it sent, so the mark has to reach the error the host holds."""+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)+ error = ToolError(_CONFIDENTIAL_MESSAGE)++ custom_client.capture_exception(error)+ custom_client.capture_exception(error)+ _crash(capture=capture, error=error)++ assert custom_capture.call_count == 1+ assert pipelex_capture.call_count == 0++ def test_an_argument_less_capture_of_a_redacted_error_is_not_sent_again(self, mocker: MockerFixture) -> None:+ """`capture_exception()` inside `except`, which passes `None`, is the SDK's documented form, and the mark must reach the error it resolved.++ Given nothing, the wrapper reads the error being handled out of+ `sys.exc_info()` and sends a redacted copy, which is what PostHog marks.+ Carrying that mark back onto the argument it was handed, `None`, left the+ live error unmarked, so a second capture and then the excepthook each+ sent it again.+ """+ custom_client, custom_capture = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ pipelex_client, pipelex_capture = _make_stubbed_client(mocker=mocker, api_key="phc_pipelex")+ capture = _make_dual_capture(custom_client=custom_client, pipelex_client=pipelex_client)+ error = ToolError(_CONFIDENTIAL_MESSAGE)++ try:+ raise error+ except ToolError:+ custom_client.capture_exception(None)+ custom_client.capture_exception(None)+ _crash(capture=capture, error=error)++ assert custom_capture.call_count == 1+ assert pipelex_capture.call_count == 0++ def test_a_chain_longer_than_the_recursion_limit_is_still_sent(self, mocker: MockerFixture) -> None:+ """PostHog walks a chain with a loop, so the check deciding whether it needs redacting must not recurse either.++ Nothing in this chain is a `PipelexError`, so nothing in it is copied,+ and the SDK sends it as it would have without the wrapper.+ """+ client, capture_mock = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ manager = TelemetryManager.__new__(TelemetryManager)+ manager._wrap_capture_exception(client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ error: BaseException = ValueError("root")+ for link_index in range(sys.getrecursionlimit()):+ wrapper = ValueError(f"link {link_index}")+ wrapper.__context__ = error+ error = wrapper++ client.capture_exception(error)++ assert capture_mock.call_count == 1++ def test_a_capture_the_redaction_cannot_complete_is_dropped_rather_than_raised_or_sent_raw(self, mocker: MockerFixture) -> None:+ """PostHog's `capture_exception` never raises into its caller, and the redaction in front of it must not either.++ A `PipelexError` at the bottom of a chain longer than the recursion+ limit cannot be copied. Sending the original instead is the one fallback+ that is not open, so nothing goes out.+ """+ client, capture_mock = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ manager = TelemetryManager.__new__(TelemetryManager)+ manager._wrap_capture_exception(client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ error: BaseException = ToolError(_CONFIDENTIAL_MESSAGE)+ for link_index in range(sys.getrecursionlimit()):+ wrapper = ValueError(f"link {link_index}")+ wrapper.__context__ = error+ error = wrapper++ assert client.capture_exception(error) is None+ assert capture_mock.call_count == 0++ @pytest.mark.parametrize("malformed", ["not an exception", (ValueError, ValueError("two")), 42])+ def test_an_argument_the_sdk_would_decline_does_not_raise_into_the_caller(self, mocker: MockerFixture, malformed: Any) -> None:+ """The SDK answers `None` to an argument it cannot use, and the wrapper answers the same instead of raising."""+ client, capture_mock = _make_stubbed_client(mocker=mocker, api_key="phc_custom")+ manager = TelemetryManager.__new__(TelemetryManager)+ manager._wrap_capture_exception(client) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert client.capture_exception(malformed) is None+ assert capture_mock.call_count == 0diff --git a/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py b/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.pynew file mode 100644index 000000000..76fac84b9--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_llm_span_carries_run_identity.py@@ -0,0 +1,123 @@+"""The LLM generation span carries the run's identity.++`$ai_generation` is the artifact the whole campaign exists for: on the hosted+plane every one of them used to be attributed to the deployment. The exporter+resolves per span, which only helps if the span site actually writes the run's+identity onto the span — so this drives the real `_start_otel_span_llm` on a real+`LLMJob` and reads the attributes back off the span OTel produced.+"""++from typing import Any++from opentelemetry.sdk.trace import Span as SdkSpan+from opentelemetry.sdk.trace import TracerProvider+from pytest_mock import MockerFixture+from typing_extensions import override++from pipelex.cogt.inference.inference_constants import InferenceOutputType+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract+from pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata, UnitJobId+from pipelex.system.telemetry.otel_constants import LangfuseSpanAttr, OTelConstants, PipelexSpanAttr+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar+++class _StubLLMWorker(LLMWorkerAbstract):+ """A worker that is never asked to generate — only to build a span."""++ @override+ async def _gen_text(self, llm_job: LLMJob) -> str:+ raise NotImplementedError++ @override+ async def _gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar:+ raise NotImplementedError+++def _make_worker() -> _StubLLMWorker:+ inference_model = InferenceModelSpec(+ backend_name="openai",+ name="gpt-test",+ sdk="openai",+ model_type=ModelType.LLM,+ model_id="gpt-test-id",+ inputs=["text"],+ outputs=["text"],+ costs={CostCategory.INPUT: 1, CostCategory.OUTPUT: 2},+ thinking_mode=ThinkingMode.NONE,+ max_tokens=None,+ max_prompt_images=None,+ )+ return _StubLLMWorker(inference_model=inference_model)+++def _make_llm_job(*, analytics_groups: dict[str, str] | None = None) -> LLMJob:+ job_metadata = JobMetadata(+ run_metadata=RunMetadata(+ user_id="user-42",+ pipeline_run_id="run-1",+ storage_scope="tenant/run-1",+ analytics_groups=analytics_groups or {},+ ),+ pipe_code="some_pipe",+ unit_job_id=UnitJobId.LLM_GEN_TEXT,+ otel_context=OtelContext(+ trace_id=1234,+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ span_id=OTelConstants.OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID,+ ),+ )+ return LLMJob(+ job_metadata=job_metadata,+ llm_prompt=LLMPrompt(user_text="hello"),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ )+++def _start_span(*, mocker: MockerFixture, llm_job: LLMJob) -> SdkSpan:+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=TracerProvider().get_tracer("test"))+ span = _make_worker()._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.TEXT) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ assert isinstance(span, SdkSpan)+ return span+++def _attributes(span: SdkSpan) -> dict[str, Any]:+ return dict(span.attributes or {})+++class TestLLMSpanIdentity:+ def test_the_span_names_the_runs_user(self, mocker: MockerFixture) -> None:+ span = _start_span(mocker=mocker, llm_job=_make_llm_job())++ assert _attributes(span)[PipelexSpanAttr.RUN_USER_ID] == "user-42"++ def test_the_span_carries_the_runs_groups_serialized(self, mocker: MockerFixture) -> None:+ span = _start_span(mocker=mocker, llm_job=_make_llm_job(analytics_groups={"organization": "org_acme"}))++ assert _attributes(span)[PipelexSpanAttr.RUN_ANALYTICS_GROUPS] == '{"organization": "org_acme"}'++ def test_a_run_with_no_groups_writes_no_groups_attribute(self, mocker: MockerFixture) -> None:+ span = _start_span(mocker=mocker, llm_job=_make_llm_job())++ assert PipelexSpanAttr.RUN_ANALYTICS_GROUPS not in _attributes(span)++ def test_langfuse_gets_the_user_id_only_when_it_is_enabled(self, mocker: MockerFixture) -> None:+ mocker.patch.object(TelemetryManagerAbstract, "get_langfuse_enabled", return_value=True)+ span = _start_span(mocker=mocker, llm_job=_make_llm_job())++ assert _attributes(span)[LangfuseSpanAttr.USER_ID] == "user-42"++ def test_langfuse_user_id_is_absent_when_langfuse_is_off(self, mocker: MockerFixture) -> None:+ mocker.patch.object(TelemetryManagerAbstract, "get_langfuse_enabled", return_value=False)+ span = _start_span(mocker=mocker, llm_job=_make_llm_job())++ assert LangfuseSpanAttr.USER_ID not in _attributes(span)diff --git a/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py b/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.pynew file mode 100644index 000000000..c3de3cad8--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_observer_telemetry_identity.py@@ -0,0 +1,61 @@+"""The observer's per-pipe events name the run they came from.++`pipe_run` and `pipe_complete` are the runtime's per-pipe drill-down beside the+platform's one-number-per-run `run_started`, so they have to land on the same+person and the same groups as the run's spans. The observer holds the whole+`PipeJob`, so the forward is the one thing that can be missing — and a missing+forward is silent: the event still fires, just under the process identity.+"""++from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.observer.observer_protocol import PayloadKey+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.telemetry.events import EventName+from pipelex.system.telemetry.observer_telemetry import ObserverTelemetry++_RUN_METADATA = RunMetadata(+ user_id="user-42",+ pipeline_run_id="run-1",+ storage_scope="tenant/run-1",+ analytics_groups={"organization": "org_acme"},+)+++def _payload(*, mocker: MockerFixture) -> dict[str, Any]:+ pipe_job = mocker.MagicMock(name="pipe_job")+ pipe_job.pipe_type = "PipeLLM"+ pipe_job.job_metadata = JobMetadata(run_metadata=_RUN_METADATA)+ return {PayloadKey.PIPELINE_RUN_ID: "run-1", PayloadKey.PIPE_JOB: pipe_job}+++class TestObserverTelemetryIdentity:+ async def test_pipe_run_names_the_run(self, mocker: MockerFixture) -> None:+ telemetry_manager = mocker.MagicMock()+ observer = ObserverTelemetry(telemetry_manager=telemetry_manager)++ await observer.observe_before_run(_payload(mocker=mocker))++ call = telemetry_manager.track_event.call_args+ assert call.kwargs["event_name"] == EventName.PIPE_RUN+ assert call.kwargs["run_metadata"] is _RUN_METADATA++ async def test_a_successful_pipe_complete_names_the_run(self, mocker: MockerFixture) -> None:+ telemetry_manager = mocker.MagicMock()+ observer = ObserverTelemetry(telemetry_manager=telemetry_manager)++ await observer.observe_after_successful_run(_payload(mocker=mocker))++ assert telemetry_manager.track_event.call_args.kwargs["run_metadata"] is _RUN_METADATA++ async def test_a_failing_pipe_complete_names_the_run(self, mocker: MockerFixture) -> None:+ telemetry_manager = mocker.MagicMock()+ observer = ObserverTelemetry(telemetry_manager=telemetry_manager)++ await observer.observe_after_failing_run(_payload(mocker=mocker))++ assert telemetry_manager.track_event.call_args.kwargs["run_metadata"] is _RUN_METADATAdiff --git a/tests/unit/pipelex/system/telemetry/test_posthog_span_exporter.py b/tests/unit/pipelex/system/telemetry/test_posthog_span_exporter.pynew file mode 100644index 000000000..8a57ad7fa--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_posthog_span_exporter.py@@ -0,0 +1,225 @@+"""Unit tests for `PostHogSpanExporter`, which had none.++The exporter is where a run's identity becomes a PostHog capture, and the whole+point of the change these tests pin is that it resolves the identity PER SPAN+rather than binding one for the life of the process. Each test drives the real+exporter with a real OTel span and a stubbed PostHog client, and asserts on the+capture call: the `distinct_id`, the `groups`, and — in every case — that the+user id did not also leak into the properties, where it would be a second copy+nobody keeps in step and a value no group-level count can use.+"""++from typing import Any++from opentelemetry.sdk.trace import Span as SdkSpan+from opentelemetry.sdk.trace import TracerProvider+from posthog import Posthog+from pytest_mock import MockerFixture++from pipelex.system.telemetry.otel_constants import PipelexSpanAttr, PostHogAttr, PostHogEvent, SpanCategory+from pipelex.system.telemetry.posthog_span_exporter import PostHogSpanExporter+from pipelex.system.telemetry.telemetry_config import TelemetryRedactionConfig+from pipelex.system.telemetry.telemetry_identity import PIPELEX_DEPLOYMENT_GROUP_TYPE, RunIdentityPolicy+from pipelex.tools.misc.hash_utils import hash_sha256++_FULL_CAPTURE = TelemetryRedactionConfig(+ redact_content=False,+ redact_pipe_codes=False,+ redact_output_class_names=False,+ content_max_length=None,+)+++def _make_span(*, span_category: SpanCategory, attributes: dict[str, Any] | None = None) -> SdkSpan:+ """Produce a real, ended `ReadableSpan` the exporter can be handed."""+ tracer = TracerProvider().get_tracer("test")+ span_attributes: dict[str, Any] = {+ PipelexSpanAttr.SPAN_CATEGORY: span_category.value,+ PipelexSpanAttr.PIPELINE_RUN_ID: "run-1",+ PipelexSpanAttr.PIPE_CODE: "some_pipe",+ PipelexSpanAttr.PIPE_TYPE: "PipeLLM",+ }+ span_attributes.update(attributes or {})+ span = tracer.start_span(name="PipeLLM: some_pipe", attributes=span_attributes)+ span.end()+ assert isinstance(span, SdkSpan)+ return span+++def _make_exporter(+ *,+ client: Posthog,+ fallback_distinct_id: str | None,+ run_identity_policy: RunIdentityPolicy,+) -> PostHogSpanExporter:+ return PostHogSpanExporter(+ posthog_client=client,+ fallback_distinct_id=fallback_distinct_id,+ run_identity_policy=run_identity_policy,+ redaction_config=_FULL_CAPTURE,+ )+++class TestPostHogSpanExporter:+ def test_a_span_naming_a_user_is_captured_under_that_user(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.PIPE, attributes={PipelexSpanAttr.RUN_USER_ID: "user-42"})])++ client.capture.assert_called_once()+ assert client.capture.call_args.kwargs["distinct_id"] == "user-42"++ def test_two_spans_of_two_runs_are_captured_under_two_users(self, mocker: MockerFixture) -> None:+ """The defect this whole change exists to fix: one process, one identity, every tenant."""+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export(+ [+ _make_span(span_category=SpanCategory.PIPE, attributes={PipelexSpanAttr.RUN_USER_ID: "user-42"}),+ _make_span(span_category=SpanCategory.PIPE, attributes={PipelexSpanAttr.RUN_USER_ID: "user-99"}),+ ]+ )++ captured_ids = [call.kwargs["distinct_id"] for call in client.capture.call_args_list]+ assert captured_ids == ["user-42", "user-99"]++ def test_a_generation_span_resolves_the_same_way(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.INFERENCE, attributes={PipelexSpanAttr.RUN_USER_ID: "user-42"})])++ client.capture.assert_called_once()+ assert client.capture.call_args.kwargs["event"] == PostHogEvent.GENERATION+ assert client.capture.call_args.kwargs["distinct_id"] == "user-42"++ def test_a_span_naming_no_user_falls_back_to_the_configured_id(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.PIPE)])++ assert client.capture.call_args.kwargs["distinct_id"] == "deployment-hash"++ def test_no_user_and_no_fallback_is_an_anonymous_capture(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id=None, run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.PIPE)])++ capture_kwargs = client.capture.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_the_runs_groups_ride_the_capture(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id=None, run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export(+ [+ _make_span(+ span_category=SpanCategory.PIPE,+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"organization": "org_acme"}',+ },+ )+ ]+ )++ assert client.capture.call_args.kwargs["groups"] == {"organization": "org_acme"}++ def test_a_run_without_groups_sends_none_rather_than_an_empty_mapping(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id=None, run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.PIPE, attributes={PipelexSpanAttr.RUN_USER_ID: "user-42"})])++ assert client.capture.call_args.kwargs["groups"] is None++ def test_an_anonymous_stream_sends_no_identity_at_all(self, mocker: MockerFixture) -> None:+ """`NONE` is the operator's `posthog.mode = "anonymous"`, and it covers the operator too.++ The exporter is built with the configured `user_id` as its fallback —+ `anonymous` mode does not forbid one, so an operator may set one, switch+ modes and leave it behind. Sending it would put a person profile on every+ span of a stream whose mode promises the runtime identifies nobody.+ """+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="configured-id", run_identity_policy=RunIdentityPolicy.NONE)++ exporter.export(+ [+ _make_span(+ span_category=SpanCategory.PIPE,+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"organization": "org_acme"}',+ },+ )+ ]+ )++ capture_kwargs = client.capture.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert "groups" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_a_namespaced_stream_digests_the_user_and_keeps_only_its_own_group(self, mocker: MockerFixture) -> None:+ """Pipelex's own exporter writes into one shared project across every deployment.++ The host's group keys are its own business vocabulary and PostHog group+ types are project-global, so none of them travel. The deployment does:+ the digest consumes the gateway hash, and the group facet is what carries+ it back.+ """+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.NAMESPACED)++ exporter.export(+ [+ _make_span(+ span_category=SpanCategory.PIPE,+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"tenant": "acme-corp"}',+ },+ )+ ]+ )++ capture_kwargs = client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == hash_sha256(data="deployment-hash:user-42", length=16)+ assert capture_kwargs["groups"] == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "deployment-hash"}+ assert "acme-corp" not in repr(capture_kwargs)++ def test_neither_the_user_nor_the_groups_appear_in_the_properties(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id=None, run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export(+ [+ _make_span(+ span_category=SpanCategory.INFERENCE,+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"organization": "org_acme"}',+ },+ )+ ]+ )++ properties = client.capture.call_args.kwargs["properties"]+ rendered = repr(properties)+ assert "user-42" not in rendered+ assert "org_acme" not in rendered++ def test_the_fallback_identity_is_not_a_property_either(self, mocker: MockerFixture) -> None:+ client = mocker.MagicMock(spec=Posthog)+ exporter = _make_exporter(client=client, fallback_distinct_id="deployment-hash", run_identity_policy=RunIdentityPolicy.DIRECT)++ exporter.export([_make_span(span_category=SpanCategory.PIPE)])++ assert "deployment-hash" not in repr(client.capture.call_args.kwargs["properties"])diff --git a/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py b/tests/unit/pipelex/system/telemetry/test_telemetry_identity.pynew file mode 100644index 000000000..6ec34fee0--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_telemetry_identity.py@@ -0,0 +1,479 @@+"""Unit tests for the one resolution every capture path shares.++`TelemetryIdentity` is where "who is this capture for" is decided, for both+PostHog streams and for events as well as spans. The matrix below is the whole+contract, and it is a matrix of `RunIdentityPolicy` against what the run names:++- `NONE` captures anonymously whatever the run says, which is what an operator+ asking to identify nobody asked for.+- `DIRECT` takes the run's own user, or the stream's fallback when the run names+ nobody, and carries the run's groups either way.+- `NAMESPACED` takes a one-way digest of the run's user inside the stream's own+ id, so a shared project can tell two callers of one deployment apart without+ ever receiving a raw caller value. It carries none of the host's groups, and+ exactly one of its own: the deployment, which the digest would otherwise+ consume, and without which a capture could no longer be counted under the+ gateway key that produced it.++The span-attribute half is the round trip: what a span site writes is what the+exporter reads back, and a span that carries nothing resolves to the fallback —+which is what every span produced before this release does.+"""++import pytest++from pipelex.system.job_metadata import RunMetadata+from pipelex.system.storage_scope import DRY_RUN_USER_ID, LOCAL_USER_ID, SINGLE_TENANT_USER_ID+from pipelex.system.telemetry.otel_constants import LangfuseSpanAttr, PipelexSpanAttr+from pipelex.system.telemetry.telemetry_config import PostHogMode+from pipelex.system.telemetry.telemetry_identity import (+ PIPELEX_DEPLOYMENT_GROUP_TYPE,+ RunIdentityPolicy,+ TelemetryIdentity,+ make_run_identity_span_attributes,+)+from pipelex.tools.misc.hash_utils import hash_sha256+++def _run_metadata(*, user_id: str = "user-42", analytics_groups: dict[str, str] | None = None) -> RunMetadata:+ return RunMetadata(+ user_id=user_id,+ pipeline_run_id="run-1",+ storage_scope="tenant/run-1",+ analytics_groups=analytics_groups or {},+ )+++class TestTelemetryIdentity:+ # ----------------------------------------------------------------- DIRECT++ def test_a_direct_stream_takes_the_runs_user_and_groups(self) -> None:+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "user-42"+ assert identity.groups == {"organization": "org_acme"}++ def test_an_event_with_no_run_falls_back_to_the_configured_id(self) -> None:+ """A CLI command or a dry-run sweep has no run — the configured id is what it reports under."""+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "configured-id"+ assert identity.groups == {}+ assert identity.is_anonymous is False++ def test_no_run_and_no_fallback_is_anonymous(self) -> None:+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id is None+ assert identity.groups == {}+ assert identity.is_anonymous is True++ def test_a_run_without_groups_is_identified_with_an_empty_facet(self) -> None:+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "user-42"+ assert identity.groups == {}++ def test_the_groups_are_copied_not_shared_with_the_run(self) -> None:+ """The identity is a decision already taken; editing it must not reach back into the run."""+ run_metadata = _run_metadata(analytics_groups={"organization": "org_acme"})+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=run_metadata,+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ identity.groups["tenant"] = "t-1"++ assert run_metadata.analytics_groups == {"organization": "org_acme"}++ # ------------------------------------------------------------------- NONE++ def test_a_none_stream_sends_no_identity_at_all_even_with_a_fallback_configured(self) -> None:+ """An operator who chose `anonymous` chose it for their users AND for themselves.++ The fallback is the configured `user_id`, which `anonymous` mode does not+ forbid — an operator may set one, switch modes, and leave it behind. If+ the fallback still reached the capture, a mode whose documented promise is+ that the runtime identifies nobody would create a person profile on every+ span and on the trace-start event.+ """+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.NONE,+ )++ assert identity.distinct_id is None+ assert identity.is_anonymous is True+ assert identity.groups == {}++ def test_a_none_stream_ignores_a_spans_identity_and_its_fallback(self) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"organization": "org_acme"}',+ },+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.NONE,+ )++ assert identity.is_anonymous is True+ assert identity.groups == {}++ def test_an_event_with_no_run_is_anonymous_on_a_none_stream(self) -> None:+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.NONE,+ )++ assert identity.is_anonymous is True++ @pytest.mark.parametrize(+ ("posthog_mode", "expected_policy"),+ [+ (PostHogMode.IDENTIFIED, RunIdentityPolicy.DIRECT),+ (PostHogMode.ANONYMOUS, RunIdentityPolicy.NONE),+ (PostHogMode.OFF, RunIdentityPolicy.NONE),+ ],+ )+ def test_an_operator_stream_reads_its_policy_off_the_mode(self, posthog_mode: PostHogMode, expected_policy: RunIdentityPolicy) -> None:+ assert RunIdentityPolicy.make_for_operator_stream(posthog_mode=posthog_mode) is expected_policy++ # ------------------------------------------------------------- NAMESPACED++ def test_a_namespaced_stream_sends_a_digest_and_never_the_raw_user(self) -> None:+ """The Pipelex stream is one shared project, so a raw caller value must never reach it."""+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id="[email protected]"),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert identity.distinct_id == hash_sha256(data="gateway-hash:[email protected]", length=16)+ assert identity.distinct_id != "[email protected]"+ assert identity.distinct_id != "gateway-hash"+ assert "alice" not in str(identity.distinct_id)++ def test_two_deployments_never_collide_on_the_same_caller_name(self) -> None:+ """The whole point: `user-42` at one customer is not `user-42` at another."""+ one = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash-one",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ other = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash-other",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert one.distinct_id != other.distinct_id++ def test_two_callers_of_one_deployment_stay_apart(self) -> None:+ """Namespacing must not cost the per-run attribution it exists to make safe."""+ one = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id="user-42"),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ other = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id="user-43"),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert one.distinct_id != other.distinct_id++ def test_the_same_caller_resolves_to_the_same_digest_every_time(self) -> None:+ """A digest that moved between two captures would make every run a new person."""+ first = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ second = TelemetryIdentity.make_from_span_attributes(+ attributes={PipelexSpanAttr.RUN_USER_ID: "user-42"},+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert first.distinct_id == second.distinct_id++ def test_a_namespaced_stream_never_forwards_the_runs_groups(self) -> None:+ """Group keys are the host's own business vocabulary and PostHog group types are project-global."""+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(analytics_groups={"tenant": "acme-corp", "plan_tier": "enterprise"}),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert "tenant" not in identity.groups+ assert "plan_tier" not in identity.groups+ assert identity.groups == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_a_namespaced_stream_keeps_the_deployment_countable_under_a_caller_digest(self) -> None:+ """The fold consumes the gateway hash, so the group facet is the only thing that carries it back.++ Without this the shared project can tell two callers apart and can no+ longer say whose key paid for either, which is what the stream is for.+ """+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert identity.distinct_id != "gateway-hash"+ assert identity.distinct_id != "user-42"+ assert identity.groups == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_one_deployment_counts_the_same_whether_a_run_names_a_caller_or_not(self) -> None:+ """Two runs, two different persons, one group — which is what makes the rollup a single query."""+ named = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ nameless = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id=LOCAL_USER_ID),+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert named.distinct_id != nameless.distinct_id+ assert named.groups == nameless.groups == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_two_deployments_never_share_a_deployment_group(self) -> None:+ first = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash-a",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )+ second = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id="gateway-hash-b",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert first.groups != second.groups++ def test_a_namespaced_span_never_forwards_the_spans_groups(self) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"tenant": "acme-corp"}',+ },+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert "tenant" not in identity.groups+ assert identity.groups == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_a_namespaced_stream_reports_the_namespace_itself_when_the_run_names_nobody(self) -> None:+ """Which is the deployment-level identity everything reported under before per-run attribution."""+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=None,+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert identity.distinct_id == "gateway-hash"+ assert identity.groups == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_a_namespaced_stream_with_no_namespace_is_anonymous_not_raw(self) -> None:+ """No namespace to fold into means no id that is safe to send on a shared project."""+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(),+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert identity.is_anonymous is True+ assert identity.distinct_id != "user-42"++ # ------------------------------------------ user ids that distinguish nobody++ @pytest.mark.parametrize("placeholder", [LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID])+ def test_a_placeholder_user_id_falls_back_to_the_stream(self, placeholder: str) -> None:+ """`RunMetadata.user_id` is required, so a caller with no real identity says so with a constant.++ Both constants are the SAME STRING on every machine. Honouring `"local"`+ as a `distinct_id` would merge every Pipelex user's local runs into one+ person, and would silently replace an operator's configured id with a+ literal on theirs.+ """+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id=placeholder),+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "configured-id"++ @pytest.mark.parametrize("placeholder", [LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID])+ def test_a_placeholder_user_id_keeps_the_runs_groups(self, placeholder: str) -> None:+ """A host may know which entities a run belongs to without naming its caller.++ The two facts are independent: groups carry no person identity, so tying+ them to the distinguishing-user test discarded them from every capture of+ a run that simply left `user_id` at its default — which is what+ `PipelexRunner(analytics_groups=...)` does by default.+ """+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id=placeholder, analytics_groups={"organization": "org_acme"}),+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "configured-id"+ assert identity.groups == {"organization": "org_acme"}++ @pytest.mark.parametrize("placeholder", [LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID])+ def test_a_span_carrying_a_placeholder_falls_back_too(self, placeholder: str) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={PipelexSpanAttr.RUN_USER_ID: placeholder},+ fallback_distinct_id="gateway-hash",+ run_identity_policy=RunIdentityPolicy.NAMESPACED,+ )++ assert identity.distinct_id == "gateway-hash"++ @pytest.mark.parametrize("placeholder", [LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID])+ def test_a_placeholder_with_no_fallback_is_anonymous_not_the_placeholder(self, placeholder: str) -> None:+ identity = TelemetryIdentity.make_from_run_metadata(+ run_metadata=_run_metadata(user_id=placeholder),+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.is_anonymous is True++ # -------------------------------------------------------- span attributes++ def test_a_run_with_groups_writes_both_attributes(self) -> None:+ attributes = make_run_identity_span_attributes(+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ is_langfuse_enabled=False,+ )++ assert attributes[PipelexSpanAttr.RUN_USER_ID] == "user-42"+ assert attributes[PipelexSpanAttr.RUN_ANALYTICS_GROUPS] == '{"organization": "org_acme"}'+ assert LangfuseSpanAttr.USER_ID not in attributes++ def test_a_run_without_groups_omits_the_groups_attribute(self) -> None:+ """Absent, not `{}`: an empty attribute on every span of every run is noise."""+ attributes = make_run_identity_span_attributes(run_metadata=_run_metadata(), is_langfuse_enabled=False)++ assert attributes[PipelexSpanAttr.RUN_USER_ID] == "user-42"+ assert PipelexSpanAttr.RUN_ANALYTICS_GROUPS not in attributes++ def test_langfuse_gets_the_user_id_in_the_field_it_reserves_for_it(self) -> None:+ attributes = make_run_identity_span_attributes(run_metadata=_run_metadata(), is_langfuse_enabled=True)++ assert attributes[LangfuseSpanAttr.USER_ID] == "user-42"++ @pytest.mark.parametrize("placeholder", [LOCAL_USER_ID, DRY_RUN_USER_ID, SINGLE_TENANT_USER_ID])+ def test_langfuse_is_never_handed_a_placeholder_as_a_person(self, placeholder: str) -> None:+ """`langfuse.user.id` IS the attribution decision, with no `_resolve` behind it to take it later.++ `pipelex.run.user_id` may carry a placeholder because something downstream+ still gets to decide what it means; Langfuse's field does not, so a+ placeholder there becomes a person literally named `local`.+ """+ attributes = make_run_identity_span_attributes(+ run_metadata=_run_metadata(user_id=placeholder),+ is_langfuse_enabled=True,+ )++ assert attributes[PipelexSpanAttr.RUN_USER_ID] == placeholder+ assert LangfuseSpanAttr.USER_ID not in attributes++ def test_what_a_span_site_writes_is_what_the_exporter_reads_back(self) -> None:+ run_metadata = _run_metadata(analytics_groups={"organization": "org_acme", "tenant": "t-1"})+ attributes = make_run_identity_span_attributes(run_metadata=run_metadata, is_langfuse_enabled=False)++ identity = TelemetryIdentity.make_from_span_attributes(+ attributes=attributes,+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "user-42"+ assert identity.groups == {"organization": "org_acme", "tenant": "t-1"}++ def test_a_span_carrying_no_identity_resolves_to_the_fallback(self) -> None:+ """Every span produced before this release, and every span from a site that sets nothing."""+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={PipelexSpanAttr.PIPE_CODE: "some_pipe"},+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "configured-id"+ assert identity.groups == {}++ def test_undecodable_groups_keep_the_user_and_drop_the_groups(self) -> None:+ """Telemetry never breaks the app: a malformed attribute costs the group facet, not the span."""+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: "not json at all",+ },+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "user-42"+ assert identity.groups == {}++ def test_a_json_value_that_is_not_an_object_drops_the_groups(self) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '["organization"]',+ },+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.groups == {}++ def test_non_string_entries_are_dropped_and_the_rest_kept(self) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={+ PipelexSpanAttr.RUN_USER_ID: "user-42",+ PipelexSpanAttr.RUN_ANALYTICS_GROUPS: '{"organization": "org_acme", "count": 3}',+ },+ fallback_distinct_id=None,+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.groups == {"organization": "org_acme"}++ def test_a_non_string_user_id_attribute_is_not_used_as_an_identity(self) -> None:+ identity = TelemetryIdentity.make_from_span_attributes(+ attributes={PipelexSpanAttr.RUN_USER_ID: 42},+ fallback_distinct_id="configured-id",+ run_identity_policy=RunIdentityPolicy.DIRECT,+ )++ assert identity.distinct_id == "configured-id"diff --git a/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py b/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.pynew file mode 100644index 000000000..0867555d2--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_telemetry_manager_identity.py@@ -0,0 +1,267 @@+"""Unit tests for the tracker half: `track_event` and `handle_trace_start`.++Events resolve their identity exactly as spans do, from the run in hand, so a+run's `pipe_run` and its `$ai_span` land on the same person. The emitters that+hold a `PipeJob` pass its `RunMetadata`; a CLI command passes nothing and reports+under the configured id.++The manager is built WITHOUT its constructor. `TelemetryManager.__init__` creates+live PostHog clients, installs an `sys.excepthook`, writes `posthog.default_client`+and registers a process-wide singleton — none of which these tests are about, and+all of which would leak into the rest of the suite. What they are about is the+resolution the two methods perform, so the instance is assembled with exactly the+attributes those methods read.+"""++from typing import Any++from posthog import Posthog+from pytest_mock import MockerFixture++from pipelex.system.job_metadata import RunMetadata+from pipelex.system.telemetry.events import EventName+from pipelex.system.telemetry.otel_constants import PostHogAttr+from pipelex.system.telemetry.telemetry_config import PostHogConfig, PostHogMode, PostHogTracingConfig, TelemetryConfig+from pipelex.system.telemetry.telemetry_identity import PIPELEX_DEPLOYMENT_GROUP_TYPE+from pipelex.system.telemetry.telemetry_manager import TelemetryManager+from pipelex.tools.misc.hash_utils import hash_sha256+++def _run_metadata(*, user_id: str = "user-42", analytics_groups: dict[str, str] | None = None) -> RunMetadata:+ return RunMetadata(+ user_id=user_id,+ pipeline_run_id="run-1",+ storage_scope="tenant/run-1",+ analytics_groups=analytics_groups or {},+ )+++def _make_manager(+ *,+ mocker: MockerFixture,+ mode: PostHogMode,+ configured_user_id: str | None,+ pipelex_distinct_id: str | None = None,+ tracing_enabled: bool = True,+) -> tuple[TelemetryManager, Any, Any]:+ """Assemble a manager with stub clients, bypassing the constructor's global side effects."""+ telemetry_config = TelemetryConfig(+ custom_posthog=PostHogConfig(+ mode=mode,+ user_id=configured_user_id,+ api_key="phc_test",+ tracing=PostHogTracingConfig(enabled=tracing_enabled),+ )+ )+ manager = TelemetryManager.__new__(TelemetryManager)+ manager.telemetry_config = telemetry_config+ custom_client = mocker.MagicMock(spec=Posthog)+ pipelex_client = mocker.MagicMock(spec=Posthog) if pipelex_distinct_id else None+ manager.custom_posthog_client = custom_client+ manager.pipelex_posthog_client = pipelex_client+ manager._pipelex_telemetry_enabled = pipelex_distinct_id is not None # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ manager._pipelex_distinct_id = pipelex_distinct_id # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ return manager, custom_client, pipelex_client+++class TestTelemetryManagerIdentity:+ def test_an_event_from_a_run_is_attributed_to_the_runs_user(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.IDENTIFIED, configured_user_id="configured-id")++ manager.track_event(+ EventName.PIPE_RUN,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == "user-42"+ assert capture_kwargs["groups"] == {"organization": "org_acme"}++ def test_an_event_with_no_run_reports_under_the_configured_id(self, mocker: MockerFixture) -> None:+ """A CLI command, a dry-run sweep — the configured id is what "no run" means."""+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.IDENTIFIED, configured_user_id="configured-id")++ manager.track_event(EventName.PIPES_LIST)++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == "configured-id"+ assert capture_kwargs["groups"] is None++ def test_an_anonymous_mode_does_not_apply_the_runs_identity(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.ANONYMOUS, configured_user_id=None)++ manager.track_event(+ EventName.PIPE_RUN,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert "groups" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_an_off_mode_captures_nothing(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.OFF, configured_user_id=None)++ manager.track_event(EventName.PIPE_RUN, run_metadata=_run_metadata())++ custom_client.capture.assert_not_called()++ def test_the_pipelex_stream_digests_the_run_inside_the_gateway_hash(self, mocker: MockerFixture) -> None:+ """One shared project serves every deployment, so a raw caller value must never reach it.++ Two customers whose callers are both `user-42` would otherwise merge into+ a single person there, and a caller id that is an e-mail address would+ leave the deployment holding it.+ """+ manager, _, pipelex_client = _make_manager(+ mocker=mocker,+ mode=PostHogMode.OFF,+ configured_user_id=None,+ pipelex_distinct_id="gateway-hash",+ )+ assert pipelex_client is not None++ manager.track_event(+ EventName.PIPE_RUN,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = pipelex_client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == hash_sha256(data="gateway-hash:user-42", length=16)+ assert capture_kwargs["groups"] == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}+ assert "org_acme" not in repr(capture_kwargs)++ def test_two_deployments_never_merge_on_the_same_caller_name(self, mocker: MockerFixture) -> None:+ captured_ids: list[str] = []+ for gateway_hash in ("gateway-hash-one", "gateway-hash-other"):+ manager, _, pipelex_client = _make_manager(+ mocker=mocker,+ mode=PostHogMode.OFF,+ configured_user_id=None,+ pipelex_distinct_id=gateway_hash,+ )+ assert pipelex_client is not None+ manager.track_event(EventName.PIPE_RUN, run_metadata=_run_metadata())+ captured_ids.append(pipelex_client.capture.call_args.kwargs["distinct_id"])++ assert captured_ids[0] != captured_ids[1]++ def test_an_event_with_no_run_falls_back_to_the_gateway_hash(self, mocker: MockerFixture) -> None:+ manager, _, pipelex_client = _make_manager(+ mocker=mocker,+ mode=PostHogMode.OFF,+ configured_user_id=None,+ pipelex_distinct_id="gateway-hash",+ )+ assert pipelex_client is not None++ manager.track_event(EventName.PIPES_LIST)++ assert pipelex_client.capture.call_args.kwargs["distinct_id"] == "gateway-hash"++ def test_an_anonymous_custom_capture_does_not_mark_the_pipelex_capture(self, mocker: MockerFixture) -> None:+ """Both streams are offered the same properties dict; one branch used to stamp the other's."""+ manager, _, pipelex_client = _make_manager(+ mocker=mocker,+ mode=PostHogMode.ANONYMOUS,+ configured_user_id=None,+ pipelex_distinct_id="gateway-hash",+ )+ assert pipelex_client is not None++ manager.track_event(EventName.PIPE_RUN, run_metadata=_run_metadata())++ pipelex_properties = pipelex_client.capture.call_args.kwargs["properties"]+ assert PostHogAttr.PROCESS_PERSON_PROFILE not in pipelex_properties++ def test_the_trace_start_is_attributed_to_the_run(self, mocker: MockerFixture) -> None:+ """It is the first event of the trace, so it has to land on the same person its spans will."""+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.IDENTIFIED, configured_user_id="configured-id")++ manager.handle_trace_start(+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ trace_id=1234,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == "user-42"+ assert capture_kwargs["groups"] == {"organization": "org_acme"}++ def test_without_a_run_it_keeps_the_configured_identity(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.IDENTIFIED, configured_user_id="configured-id")++ manager.handle_trace_start(trace_name="some_pipe_abc12345", trace_name_redacted="abc12345", trace_id=1234)++ assert custom_client.capture.call_args.kwargs["distinct_id"] == "configured-id"++ def test_an_anonymous_operator_stream_still_identifies_nobody(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.ANONYMOUS, configured_user_id=None)++ manager.handle_trace_start(+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ trace_id=1234,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_an_anonymous_stream_ignores_a_user_id_left_behind_in_the_config(self, mocker: MockerFixture) -> None:+ """`anonymous` mode does not forbid a `user_id`, so one survives a mode switch.++ While the trace-start and the span exporter read that leftover as their+ fallback, a stream documented as identifying nobody created a person+ profile on every capture but its ordinary events.+ """+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.ANONYMOUS, configured_user_id="left-behind-id")++ manager.handle_trace_start(+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ trace_id=1234,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = custom_client.capture.call_args.kwargs+ assert "distinct_id" not in capture_kwargs+ assert capture_kwargs["properties"][PostHogAttr.PROCESS_PERSON_PROFILE] is False++ def test_the_pipelex_stream_trace_start_is_namespaced_like_its_spans(self, mocker: MockerFixture) -> None:+ """The trace's first event has to land on the same person its spans will."""+ manager, _, pipelex_client = _make_manager(+ mocker=mocker,+ mode=PostHogMode.OFF,+ configured_user_id=None,+ pipelex_distinct_id="gateway-hash",+ )+ assert pipelex_client is not None++ manager.handle_trace_start(+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ trace_id=1234,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ capture_kwargs = pipelex_client.capture.call_args.kwargs+ assert capture_kwargs["distinct_id"] == hash_sha256(data="gateway-hash:user-42", length=16)+ assert capture_kwargs["groups"] == {PIPELEX_DEPLOYMENT_GROUP_TYPE: "gateway-hash"}++ def test_the_identity_never_becomes_a_property(self, mocker: MockerFixture) -> None:+ manager, custom_client, _ = _make_manager(mocker=mocker, mode=PostHogMode.IDENTIFIED, configured_user_id="configured-id")++ manager.handle_trace_start(+ trace_name="some_pipe_abc12345",+ trace_name_redacted="abc12345",+ trace_id=1234,+ run_metadata=_run_metadata(analytics_groups={"organization": "org_acme"}),+ )++ rendered = repr(custom_client.capture.call_args.kwargs["properties"])+ assert "user-42" not in rendered+ assert "org_acme" not in rendereddiff --git a/wip/run-scoped-telemetry-identity/review-deferred.md b/wip/run-scoped-telemetry-identity/review-deferred.mdnew file mode 100644index 000000000..3032808b4--- /dev/null+++ b/wip/run-scoped-telemetry-identity/review-deferred.md@@ -0,0 +1,96 @@+---+status: active+item: L-260922-75299c+---++# Deferred review findings — run-scoped telemetry identity++Findings from the `/rev` passes on this branch (profile 4) that were real but did not meet the round's bar, kept here so nothing is dropped without somewhere to chase it. Round 1 ran at bar `open`, round 2 at bar `defects`, rounds 3 to 5 at bar `necessity`; round 5 ran past the ladder's cap of four on Louis's call, focused on the exception path.++## One capture helper instead of the hand-rolled branches++Raised by cubic in round 1. Still **unverified** — it sorted to deferral before either round's verifier ran, so it rests on cubic's reading alone.++The identified-versus-anonymous capture shape is written out by hand at several sites: `_capture_custom_event` and `_track_to_pipelex` in `pipelex/system/telemetry/telemetry_manager.py`, both branches of `handle_trace_start` in the same file, and `_capture_event` in `pipelex/system/telemetry/posthog_span_exporter.py`. Each one reads "if the identity has a `distinct_id`, capture with it and with `groups or None`; otherwise stamp `$process_person_profile = False`".++cubic's argument for folding them into one `capture_under_identity(client, event, properties, identity)` is that the two streams cannot then drift apart again, and that the anonymous-mode defect round 1 fixed would have been a one-line change. That argument held twice: round 1's fix had to be made once in `_resolve` and then checked at every one of those sites, and round 2 found the one site nobody had checked — the exception autocapture, which was not on cubic's list because it did not resolve an identity at all.++Round 2's fix added a site rather than removing one: `DualClientExceptionCapture._capture_to_client` now writes the same branch a sixth time, though it at least collapses what used to be two copies inside that class into one. The case for the helper is therefore stronger than it was, not weaker.++cubic raised it again in round 3, unprompted and from a clean context, making it the only finding to appear in all three passes.++It stays deferred because it is a structural refactor rather than a defect, and `RunIdentityPolicy` concentrates the part that was actually drifting — which branch a stream takes — in one place. Worth doing when that file is next opened for another reason.++## A run's exception and its spans are still two different people++Raised by `codex:adversarial` and by `code-review` in round 2, and **verified**: `exception_capture.py` captures under the stream's runless identity, while the run's `$ai_span`, `$ai_generation` and product events capture under the run-scoped one. In PostHog those are two distinct persons, so "what was this person doing before the crash" returns nothing joinable on either stream.++Round 2 fixed the half that was a privacy defect — `anonymous` mode no longer names the operator on a crash — and left this half, which is not a patch. `sys.excepthook` and `threading.excepthook` are handed `(type, value, traceback)` and nothing else, and this codebase has no ContextVar layer to read a run out of: run-scoped state rides the payload here, deliberately, because no process spans a run under Temporal. Attributing a crash to the run that caused it therefore needs a design answer — a last-known-run identity the hook may read, with its own concurrency story — rather than a change at the capture site.++Its blast radius bounds the urgency: `grep` shows the two excepthooks are the only callers of `_capture_exception`, so this fires for exceptions that escape to the top of the interpreter or of a thread. A server that handles its own exceptions never reaches it; a CLI crash does.++## The Gateway digest is a pseudonym, not a secret++Raised by `codex:adversarial` and, as a documentation finding, by cubic. The mechanism half was **verified and classified an improvement**, and the prose half was fixed in round 2.++`_make_namespaced_distinct_id` is `sha256(f"{namespace}:{run_user_id}")` truncated, with no key and no salt beyond the namespace — and the namespace, the deployment's Gateway-key hash, is itself sent in the clear as the `distinct_id` of every capture that names nobody. A party with read access to Pipelex's PostHog project can therefore fold a candidate caller id the same way and see whether the result is present, which is cheap against low-entropy ids such as e-mail addresses. That party is in practice Pipelex staff, which is what keeps this an improvement rather than a defect.++Round 2 corrected the claims instead: `docs/setup/telemetry.md` and the function's own docstring no longer say Pipelex can tell callers apart "without learning who either is".++The fix, if it is taken: fold the **raw** Gateway API key rather than its hash, since the raw key never reaches the stream. Nothing else about the construction changes, and the sentences that were weakened could then be restored. It needs the raw key carried to the resolution site, which is the reason it was not done inside a review pass.++## `user_id` has no validation and no provenance++Raised by `codex:adversarial` in round 2, **verified as a mechanism and ruled a theoretical edge case**.++`RunMetadata.user_id` is a bare `str` with no field validator — unlike `storage_scope` and `analytics_groups`, which both have one — and `pipeline_run_setup` does not validate it either. Telemetry decides a run names nobody by comparing that string against the literals in `_NON_DISTINGUISHING_RUN_USER_IDS`, keeping no record of whether the value came from the local default or from an authenticated host. A hosted caller genuinely named `local` is therefore collapsed onto the deployment fallback.++Deferred because no deployment shape was found that reaches it: the default is set in exactly one place, `PipelexMTHDSProtocol.__init__`, and a multi-tenant host must pass its own value explicitly. The cure — a nullable principal, or an identity-kind field, instead of inferring the answer from string contents — is a model change, not a review fix.++## pipelex-api's `single-tenant` is a placeholder the set does not know++Raised by cubic in round 4, and **not verified** — it sorted to deferral before the verifier ran. **The stopgap was taken after round 4**, once `pipelex-api/api/routes/pipelex/pipeline.py` confirmed the default: `SINGLE_TENANT_USER_ID` now lives in `pipelex/system/storage_scope.py` beside `LOCAL_USER_ID` and is in the set, so pipelex-api can import the one string rather than spell its own. The model change below remains the cure.++`_NON_DISTINGUISHING_RUN_USER_IDS` in `pipelex/system/telemetry/telemetry_identity.py` holds `local` and `dry-run-no-user`. cubic reports that pipelex-api returns `SINGLE_TENANT_USER_ID = "single-tenant"` for every run on a single-tenant deployment, which is the same kind of shared placeholder: once pipelex-api pins this version, every such deployment's runs would land on one PostHog person named `single-tenant`, and Langfuse would receive it as `langfuse.user.id`. It is one more instance of the section above — identity absence inferred from string contents — and the cure is the same: a nullable principal carried from the host, rather than a longer list of reserved strings. Adding the string to the set is the cheap stopgap if pipelex-api pins this version before that model change lands.++## `track_event` names `DIRECT` instead of deriving it++Raised by cubic in round 2, and correct today by its own account. `RunIdentityPolicy.make_for_operator_stream` documents itself as the single place the operator stream's policy is derived, and `track_event` is the one of those sites that writes `RunIdentityPolicy.DIRECT` literally instead of calling it. It is inside `case PostHogMode.IDENTIFIED`, so it cannot currently be wrong; a fourth mode, or a change to that mapping, would diverge there silently. An improvement, and cheap whenever that method is next edited.++## Anonymous mode is a claim about PostHog, and the prose does not always say so++Raised by `code-review` in round 3. Real but narrower than reported, and **not verified** — it sorted to deferral before the verifier ran.++`make_run_identity_span_attributes` writes `pipelex.run.user_id` and `pipelex.run.analytics_groups` onto every span unconditionally, and sets `langfuse.user.id` on `is_langfuse_enabled` alone. The operator's `custom_posthog.mode` is read only by the PostHog exporters, through `RunIdentityPolicy.NONE`. So an operator running `mode = "anonymous"` with Langfuse or an OTLP exporter enabled does see per-caller ids at those destinations.++Whether that is a defect turns on what the mode is a claim about. `custom_posthog.mode` governs the operator's PostHog stream, and the docs site scopes it correctly — "the runtime identifies nobody **on your stream**". Langfuse is a separate opt-in with its own configuration, and prompts and completions already reach it unredacted, so an operator who enabled it has already consented to far more than a user id. `code-review`'s own reading is that the prose overreaches rather than the code misbehaving.++What is genuinely loose is the CHANGELOG sentence, which sits in a paragraph that also states the span behaviour plainly and so can be read either way. The cure is either to scope the claim to PostHog in that entry, or to gate the Langfuse attribute on the same decision the PostHog exporter takes — a design question about whether one operator setting should reach a second vendor's destination, which is why it was not settled inside a review pass.++## `TelemetryIdentity.is_anonymous` is a second way of asking one question++Raised by `code-review` in round 3 as a cosmetic sibling of its freeze finding, and **not verified**.++All the capture sites branch on `identity.distinct_id` being truthy; `is_anonymous` is referenced only from tests. Two ways of asking the same question, with nothing keeping them in step — either make it the one the capture sites use, or drop it. Cheap either way, and a judgement about which reads better at the call sites rather than a defect.++## Crash redaction goes by class, so a wrapper's message still carries the text++Raised by cubic and `codex:adversarial` in round 5, and **verified**; filed as `L-260923-65619d`, which carries the reproductions. An error of another class that copied a `PipelexError`'s text into its own message — `raise RuntimeError(f"failed: {error}") from error`, or `pipe_spec.py`'s `ValueError(str(exc)) from exc` — sends that text on both streams, and `PromptContentError`, `PipeValidationError` and `ConceptValueError` are not `PipelexError` subclasses, so their messages are never redacted. It predates the branch, which narrowed it: on `origin/dev` the linked `PipelexError` went out verbatim too. Round 5 corrected the prose that implied otherwise — the `_sanitized_exception_arg` and `_plain_copy` docstrings and the CHANGELOG headline now state the class limit.++## A CLI failure may never reach the Pipelex stream when both streams are on++Noted by round 5's verifier from reading the code, and **not verified**. `telemetry_context()` is PostHog's `new_context()`, which captures on `posthog.default_client` only — the operator's client whenever one exists — and the CLI turns every failure into `typer.Exit`, so the excepthook never runs. With both streams enabled a CLI failure would then reach the operator's project and not Pipelex's. The same holds on `origin/dev`; the CHANGELOG's "both telemetry streams record a crash" is about unhandled exceptions, which this is not.++## Settled in rounds 2 to 5, kept for the record++**The stale docstrings in the two integration-test modules** — `test_pipeline_run_setup_analytics_groups.py` and `test_pipeline_run_setup_storage_scope_gate.py` — were deferred in round 1 as unverified. Round 2 verified them: the docstrings were stale (the exporters do read the groups, and `handle_trace_start` did move below `prepare_pipe_job`), but cubic's stronger claim, that the tests would no longer fail if the gate moved back, was **refuted** — both modules also spy `get_pipeline_manager`, and `add_new_pipeline` still runs above `prepare_pipe_job`, so the pipeline half of each assertion still bites. Only the telemetry half went vacuous. Both docstrings were corrected in round 2 and say so.++**Round 3 found two defects on the exception path, both predating the branch and both fixed.** The privacy redaction replaced a `PipelexError`'s message only when the error arrived bare, never in the `(type, value, traceback)` form an excepthook produces — which is the only form the autocapture has — so a message repeating the caller's own input went out verbatim, reachable on the shipped default because `custom_posthog.mode = "off"` still builds the Gateway client that installs the hooks. And PostHog stamps an error once a client has captured it and drops any later capture of that object, so of the two streams only the first ever recorded a crash, which meant round 2's deployment group was never recorded anywhere. `git` dates both to v0.18.0; the branch changed their shape without introducing either, and made the second consequential by adding the group that never shipped. `tests/unit/pipelex/system/telemetry/test_exception_capture_sdk_contract.py` now exercises both against real PostHog clients, because round 2's `MagicMock(spec=Posthog)` could see neither.++**The `analytics_groups` module docstring** said "no exporter reads the mapping yet, so do not take the field's presence as evidence that a span or an event is already grouped by it" — false as of this branch, and the file is outside the diff, which is why rounds 1 and 2 swept past it. Corrected in round 3.++**The `RunMetadata` freeze** was raised by cubic and `code-review` as an unrecorded breaking change, and `code-review` additionally warned that a dependent repo might reassign a field, `RunMetadata` being on the import surface `pipelex-server/transport/` consumes. That half was **refuted**: `pipelex-server/transport/` and `pipelex-api` contain no `RunMetadata` field assignment, `pipelex-server`'s four sites are all on `mocker.MagicMock()` stubs in `pipelex_temporal` tests, and there are no `setattr` or `object.__setattr__` escape hatches anywhere. The documentation half was taken — the freeze now has a `Changed` entry.++**Round 4 found round 3's crash redaction incomplete, and one regression in its mark clearing; both fixed.** The redaction replaced a `PipelexError` only at the top of what it was handed, while PostHog sends every error it reaches through `__cause__`, `__context__` and the members of an exception group, and reads `sys.exc_info()` itself when handed nothing — so `raise RuntimeError(...) from pipelex_error`, an error raised while handling one, or a `TaskGroup` failure still sent the message verbatim to both streams. The verifier reproduced each against the real SDK. `_sanitized_exception_arg` now redacts a copy of the whole graph and resolves the argument-less form first. Separately, round 3 cleared PostHog's capture marks before the first stream as well as the second, which erased the mark a host's own `posthog.capture_exception()` had left, so an error the host captured and re-raised was recorded twice. The hook now skips an error already marked when it runs and clears marks only between the streams, and the privacy wrapper carries the mark from the redacted copy it sent back onto the caller's error — which also closes the older duplicate where a host's capture of a `PipelexError` marked only the stand-in. The Codex adversarial reviewer re-raised the pseudonym, the reserved-string and the crash-attribution findings above, which stay deferred as recorded.++**Round 5 found two defects in round 4's own changes on the exception path; both fixed.** The privacy wrapper resolved `capture_exception()`'s argument-less form inside the sanitizer only, so the "already captured" check and the mark carried back both saw `None`: a host doing `except PipelexError: posthog.capture_exception(); raise` had the redacted copy marked and the live error not, and the excepthook sent it again on both streams. It now resolves `sys.exc_info()` first. And the graph walk round 4 added ran unguarded before PostHog's own `try`, so a chain of about five hundred links or an argument the SDK would decline raised into the host's `except` block — and could replace the original error inside the CLI's `new_context()`. The reach check is a loop now, as PostHog's walk is, and anything the redaction cannot complete drops the capture with a debug line rather than raising or sending the original. The Codex adversarial reviewer re-raised the crash-attribution finding above, which stays deferred.