Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Fix/tracing across workers (#860)
Louis Choquel258e324130d2
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.mdindex ac5bdb835..fd1e69242 100644--- a/.claude/skills/temporal-e2e-validate/SKILL.md+++ b/.claude/skills/temporal-e2e-validate/SKILL.md@@ -25,6 +25,9 @@ allowed-tools:- Bash(open *)- Bash(cat *)- Bash(.venv/bin/python *)+ - Bash(grep *)+ - Bash(sort *)+ - Bash(head *)---# Temporal E2E Validation Suite@@ -532,6 +535,69 @@ wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"After this, tell the user both results.+### Step 5b: Tier 8 — Cross-worker usage emission (writer_id fallback)++This validates that `UsageReportEvent`s emitted from inference activities running on+the runner process land in the same NDJSON partition as the router-side trace events,+stamped with a runner-process `writer_id` (`act_{pid}_{uuid8}`). It's the e2e+counterpart to the Phase 4 integration test+`tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`.++**Important — dry-run does NOT exercise this path.** In dry-run mode, `PipeLLM`+(and friends) instantiate `ContentGeneratorDry()` directly inside the workflow+body — on the router process — and never dispatch `act_llm_gen_text` to the+runner. So a vanilla `pipelex run bundle --temporal --dry-run` against+router+runner workers will emit ALL `usage_report` events with+`writer_id="primary"`, never `act_*`. To deterministically observe runner-side+fallback in dry-run, run the Phase 4 integration test, which substitutes the+inference activity with a wrapper that synthesizes a real `LLMJob` server-side:++```bash+.venv/bin/pytest -x -v \+ tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \+ -m temporal --temporal-server local 2>&1+```++Expected: both test cases pass —+`test_runner_usage_event_lands_in_same_ndjson_dir` (lands in same dir, with an+`__w_act_*` file) and `test_no_double_emit_in_split_worker_pool` (no+double-counting between fast path and fallback).++**To observe `act_*` writer files via the CLI**, run live mode (real LLM call —+costs money) so `act_llm_gen_text` is actually dispatched to the runner:++```bash+.venv/bin/pipelex run bundle \+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \+ --pipe native_text_sequence \+ --temporal --no-logo --graph+```++After completion, list the trace files for the run:++```bash+RUN_ID=$(ls -t .pipelex/traces/ | head -1)+ls -la .pipelex/traces/$RUN_ID/+grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson+grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u+```++Expect:++- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file alongside the+ router-side `wf_*.ndjson` files.+- The runner-side file contains `usage_report` events with+ `writer_id` starting `act_`.+- Router-side files contain `pipe_start` / `pipe_end_success` events with+ `writer_id="primary"`.++If only `writer_id="primary"` appears, the inference activity ran in the+router process — check `worker_config.inference_task_queue` and confirm both+workers are running with the latest code (restart them if they predate the+Phase 2 runner-side fallback commit).++---+### Step 6: StoragePayloadCodec tests — does the codec work end-to-end?These tests validate Phase 5: large payloads are transparently offloaded to external@@ -658,6 +724,7 @@ ls results/*/reactflow.html| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no || Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — || Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |+| Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — || Concept isolation (alpha) | Conflicting `Result` concepts don't cross-contaminate | PASS/FAIL | path | — || Concept isolation (beta) | (same test, other side) | PASS/FAIL | path | — || Pipe isolation (alpha) | Conflicting `shared_step` pipes use correct version | PASS/FAIL | path | — |diff --git a/.gitignore b/.gitignoreindex 36d6030fe..c9fe49d82 100644--- a/.gitignore+++ b/.gitignore@@ -85,4 +85,5 @@ package-lock.json# Claude plans.claude/plans/+.claude/scheduled_tasks.lock.gstack/diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 230acb044..177ba5ec6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,6 +19,7 @@- **`StoragePayloadCodec`**: offloads Temporal payloads larger than a configurable threshold (default 1 MiB) to external storage (S3 or local) and replaces them with lightweight URI references. Configurable via `[temporal.payload_codec_config]`. New `pipelex codec serve` CLI launches the codec server consumed by the Temporal Web UI for payload inspection.- **Schema-based dynamic model construction (`pipelex/cogt/content_generation/schema_to_model_factory.py`)**: rebuilds dynamic Pydantic models on a Temporal worker from the JSON schema embedded in `ObjectAssignment`, instead of relying on cross-process class identity. Cache bounded by an LRU (`_SCHEMA_CACHE_MAX_SIZE = 1024`); codegen serialized through a file lock to kill thundering-herd duplication on long-running workers receiving many distinct schemas. Adds `datamodel-code-generator>=0.55.0` as a runtime dependency.- **Worker scopes**: `pipelex worker --scope ...` lets a worker subscribe to a subset of activities/workflows so deployments can specialize workers (e.g. inference workers vs orchestration workers).+- **`WorkerConfig.inference_task_queue`**: optional task-queue override for `act_llm_gen_text` so router and inference activities can be deployed on separate worker pools.- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run-record creation.- **`workflow_id` in start response**: `PipelexPipelineStartResponse` includes `workflow_id`.- **Temporal `callbacks` passthrough**: `WorkflowExecutor.start_workflow()` accepts optional Temporal `Callback` objects.@@ -27,7 +28,12 @@- **`EventLogProtocol` with pluggable backends**: `NdjsonEventLog`, `DynamoDBEventLog`, `InMemoryEventLog`, `BufferingEventLog`. Emits `TraceEvent`s as pipes start/finish across processes. Configured via `[pipelex.tracing_config]` (`backend = "ndjson" | "dynamodb"`). Tracing is **enabled by default** and decoupled from Temporal — single-process runs also produce a trace.- **`pipelex[dynamodb]` extra**: `pip install "pipelex[dynamodb]"` for the DynamoDB event log backend.-- **`writer_id` on `TraceEvent`**: every event records which writer (worker / process) produced it, propagated through every backend. Enables proper cross-worker dedup and per-writer aggregation.+- **`writer_id` on `TraceEvent`** (default `"primary"`): every event records which writer (worker / process) produced it, propagated through every backend. Standalone activity worker processes use a distinct `writer_id` so they can emit into the same backend partition as workflow workers without colliding on `(workflow_id, sequence)`. Stamped at construction time (no copy-on-emit). Enables cross-worker dedup and per-writer aggregation.+- **NDJSON multi-writer key scheme**: dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence is primary so the runner's `UsageReportEvent` does not sort before the router's `PipeStartEvent`. The file-handle cache key includes `writer_id`, so two writers for the same `(pipeline_run_id, workflow_id)` cannot share a stale handle.+- **DynamoDB sort key format**: `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.+- **Cross-worker `UsageReportEvent` emission from runner processes**: runner processes that never execute `WfPipeRouter.run()` (so `_event_log_contexts` has no entry for the workflow's `lookup_key`) emit usage events through a process-local event log built from `tracing_config`. The fallback log is cached per process via a `threading.Lock`-guarded factory in `pipelex.tracing.activity_event_log` and keyed by a stable per-process `writer_id = f"act_{pid}_{uuid8}"`. A single one-shot warning per process records that the fallback engaged. Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught and logged at WARNING — never silently dropped, never re-raised into the inference path.+- **`ReportingManager` registry lookup split into `_get_registry_strict` and `_get_or_create_registry`**: strict raises on miss and is used by `_report_*_job`, so runner processes silently skip the registry add when `open_registry` was never called; or-create is used by `inject_tokens_usages`, the console cost-report path, and `generate_report` for never-opened runs. Activity workers do not accumulate orphan registries.+- **`ContentGeneratorDry` emits `UsageReportEvent`s**: invokes `report_inference_job` with a synthetic `LLMTokensUsage` so dry-run mode produces real events observable through the same backend path as live runs.- **`GraphSpecAssembler`**: rebuilds a complete `GraphSpec` offline from a stream of `TraceEvent`s — equivalence with the live `GraphTracer` is pinned by tests.- **`UsageAggregator`**: aggregates per-pipeline LLM/extract/img-gen usage across workers from the event log.- **Hub DI**: `PipelexHub.set_event_log()` / `get_event_log()` for dependency injection of custom event log backends. `make_event_log()` factory selects the backend from config or hub injection; the 3 prior call sites that hardcoded `NdjsonEventLog` now go through it.@@ -63,7 +69,7 @@- **Cold start trimmed**: heavy SDK imports (`boto3`, `huggingface_hub`, etc.) deferred behind `TYPE_CHECKING` or function-local guards across multiple plugins.- **Graph viewer updated to mthds-ui v0.3.4**: bumped from v0.3.0 — additional polish atop the resizable detail panel, escape-to-close, sticky header, prompt expand/collapse with copy button, concept refinement display.- **Default models**: small-vision and creative defaults now point to `gemini-3.0-flash-preview`. Added `claude-4.6-sonnet` and Bedrock token-auth support.-- **`kajson` upgraded** from `0.3.1` to `0.4.2`.+- **`kajson` upgraded** from `0.3.1` to `0.5.0`.- **README install instructions**: replaced step-by-step Claude Code setup with single copy-paste messages for Claude Code and Codex, added manual install section.### Fixeddiff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex 19d94ff51..f9db0d907 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -1,3 +1,4 @@+from datetime import datetimefrom typing import Anyfrom typing_extensions import override@@ -12,8 +13,11 @@ from pipelex.cogt.image.generated_image import GeneratedImageRawDetailsfrom pipelex.cogt.image.image_size import ImageSizefrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReportfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract+from pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TemplatingStyle@@ -22,11 +26,56 @@ from pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContentfrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.hub import get_report_delegatefrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsingfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVarfrom pipelex.urls import URLs+_DRY_RUN_INFERENCE_MODEL_NAME = "dry_run"+_DRY_RUN_INFERENCE_MODEL_ID = "dry_run"+++def _report_dry_llm_job(job_metadata: JobMetadata, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None:+ """Build a synthetic LLMJob with zero-token usage and report it.++ This makes the runner-side cross-worker emission path observable via the+ existing dry-run e2e harness without forcing assertions to run live LLM.+ The synthetic LLMTokensUsage carries zero counts and a "dry_run" model id+ so consumers can distinguish dry-run usage from real-inference usage.++ The synthetic job_metadata copy gets completed_at set so report_inference_job+ can format the duration without crashing on None — matching what+ LLMJob.llm_job_after_complete does in the live path. JobMetadata.started_at+ can arrive either offset-naive (from the default factory) or offset-aware+ (e.g. callers in pipe_abstract.py use datetime.now(timezone.utc)), so we+ build the synthetic ``now`` with the same tzinfo as the incoming started_at+ — mixing aware/naive raises TypeError when JobMetadata.duration subtracts.+ """+ tz = job_metadata.started_at.tzinfo if job_metadata.started_at is not None else None+ now = datetime.now(tz)+ synthetic_metadata = job_metadata.model_copy(+ update={+ "started_at": job_metadata.started_at or now,+ "completed_at": now,+ },+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=synthetic_metadata,+ inference_model_name=_DRY_RUN_INFERENCE_MODEL_NAME,+ inference_model_id=_DRY_RUN_INFERENCE_MODEL_ID,+ unit_costs={},+ nb_tokens_by_category={},+ )+ synthetic_job = LLMJob(+ job_metadata=synthetic_metadata,+ llm_prompt=llm_prompt,+ job_params=llm_setting.make_llm_job_params(),+ job_config=LLMJobConfig(max_retries=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+ get_report_delegate().report_inference_job(inference_job=synthetic_job)+class ContentGeneratorDry(ContentGeneratorProtocol):"""This class is used to generate mock content for testing purposes.@@ -60,6 +109,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):func_name = "make_llm_text"log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")prompt_truncated = llm_prompt_for_text.desc(truncate_text_length=self._text_gen_truncate_length)+ _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_main, llm_prompt=llm_prompt_for_text)return f"DRY RUN: {func_name} • llm_setting={llm_setting_main.desc()} • prompt={prompt_truncated}"@override@@ -76,6 +126,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):# We run validators to ensure mock data is valid. Fields with format constraints# (snake_case, PascalCase, etc.) should have `examples` defined in their Field()# so polyfactory uses those instead of random strings.+ _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_for_object, llm_prompt=llm_prompt_for_object)return object_factory.build()@overridediff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.pyindex bcca1e633..7471f59e7 100644--- a/pipelex/graph/graph_tracer.py+++ b/pipelex/graph/graph_tracer.py@@ -171,6 +171,17 @@ class GraphTracer(GraphTracerProtocol):self._event_sequence += 1return seq+ def _event_writer_id(self) -> str:+ """Writer-id to stamp on every constructed TraceEvent.++ Must only be called when self._event_log is not None. Reading from+ the event log instance keeps the writer_id invariant local to the+ emit site and avoids drift between the log's writer_id and the+ events it stores.+ """+ assert self._event_log is not None+ return self._event_log.writer_id+def _make_node_id(self) -> str:"""Generate a node ID, including workflow_id when in Temporal mode."""if self._workflow_id != "direct":@@ -453,6 +464,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(BatchItemEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),@@ -495,6 +507,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(BatchAggregateEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),@@ -536,6 +549,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(ParallelCombineEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),@@ -597,6 +611,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(PipeStartEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=started_at,sequence=self._next_event_sequence(),@@ -646,6 +661,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(ExecutionDataEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),@@ -704,6 +720,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(PipeEndSuccessEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=ended_at,sequence=self._next_event_sequence(),@@ -747,6 +764,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(ControllerOutputEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),@@ -786,6 +804,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(PipeEndErrorEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=ended_at,sequence=self._next_event_sequence(),@@ -827,6 +846,7 @@ class GraphTracer(GraphTracerProtocol):self._emit_event(EdgeEvent(pipeline_run_id=self._event_pipeline_run_id,+ writer_id=self._event_writer_id(),workflow_id=self._workflow_id,timestamp=datetime.now(timezone.utc),sequence=self._next_event_sequence(),diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 490227793..750a3cad8 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -7,6 +7,7 @@ from pydantic import Field, RootModelfrom typing_extensions import overridefrom pipelex import log+from pipelex.base_exceptions import PipelexConfigErrorfrom pipelex.cogt.exceptions import ReportingManagerErrorfrom pipelex.cogt.extract.extract_job import ExtractJobfrom pipelex.cogt.img_gen.img_gen_job import ImgGenJob@@ -16,14 +17,22 @@ from pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.search.search_job import SearchJobfrom pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.config import get_config+from pipelex.graph.graph_context import GraphContextfrom pipelex.pipeline.pipeline_models import SpecialPipelineIdfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.reporting.reporting_types import AnyTokensUsage, TokensUsage+from pipelex.system.exceptions import MissingDependencyErrorfrom pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_pathfrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of+from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.trace_events import UsageReportEvent+try:+ from botocore.exceptions import ClientError as _BotoClientError # type: ignore[import-untyped]+except ImportError:+ _BotoClientError = None # type: ignore[assignment, misc]+if TYPE_CHECKING:from pathlib import Path@@ -107,16 +116,43 @@ class ReportingManager(ReportingProtocol):# Private methods############################################################- def _get_registry(self, pipeline_run_id: str) -> UsageRegistry:- if pipeline_run_id not in self._usage_registries:- # Auto-create registry for unknown pipeline IDs. This happens when- # Activities report inference jobs on a Temporal worker where- # open_registry() was never called (it runs on the API process).+ def _get_registry_strict(self, pipeline_run_id: str) -> UsageRegistry:+ """Return the registry for ``pipeline_run_id`` or raise ``KeyError`` on miss.++ Used by ``_report_*_job``: on the runner, the registry is never opened+ because ``open_registry`` runs on the API process. Callers swallow+ ``KeyError`` and skip the local-add — the runner-side+ ``UsageReportEvent`` emission (Phase 2 fallback) is independent of+ the registry, so distributed reassembly still gets the data.+ """+ return self._usage_registries[pipeline_run_id]- # TODO: replace with proper distributed reporting system+ def _get_or_create_registry(self, pipeline_run_id: str) -> UsageRegistry:+ """Return the registry for ``pipeline_run_id``, creating it on miss.++ Used by ``inject_tokens_usages`` (the P1 cross-worker assembly path),+ the console cost-report path, and ``generate_report`` when called for+ a specific run that wasn't opened on this process.+ """+ if pipeline_run_id not in self._usage_registries:self._usage_registries[pipeline_run_id] = UsageRegistry()return self._usage_registries[pipeline_run_id]+ def _try_add_to_registry(self, pipeline_run_id: str, tokens_usage: AnyTokensUsage) -> None:+ """Add to the registry if it exists; silently skip if not.++ Skipping on miss is the runner-process behavior — the registry was+ never opened here. The local-skip is intentional: the+ ``UsageReportEvent`` emitted by ``_emit_usage_event`` carries the+ data into the shared backend partition, and the P1 readback path+ will assemble it on the process that owns ``inject_tokens_usages``.+ """+ try:+ registry = self._get_registry_strict(pipeline_run_id)+ except KeyError:+ return+ registry.add_tokens_usage(tokens_usage)+def _report_llm_job(self, llm_job: LLMJob):llm_tokens_usage = llm_job.job_report.llm_tokens_usage@@ -125,7 +161,7 @@ class ReportingManager(ReportingProtocol):returnpipeline_run_id = llm_job.job_metadata.pipeline_run_id- self._get_registry(pipeline_run_id).add_tokens_usage(llm_tokens_usage)+ self._try_add_to_registry(pipeline_run_id, llm_tokens_usage)self._emit_usage_event(llm_job, llm_tokens_usage)if self._reporting_config.is_log_costs_to_console:@@ -140,7 +176,7 @@ class ReportingManager(ReportingProtocol):returnpipeline_run_id = img_gen_job.job_metadata.pipeline_run_id- self._get_registry(pipeline_run_id).add_tokens_usage(img_gen_tokens_usage)+ self._try_add_to_registry(pipeline_run_id, img_gen_tokens_usage)self._emit_usage_event(img_gen_job, img_gen_tokens_usage)if self._reporting_config.is_log_costs_to_console:@@ -155,7 +191,7 @@ class ReportingManager(ReportingProtocol):returnpipeline_run_id = extract_job.job_metadata.pipeline_run_id- self._get_registry(pipeline_run_id).add_tokens_usage(extract_tokens_usage)+ self._try_add_to_registry(pipeline_run_id, extract_tokens_usage)self._emit_usage_event(extract_job, extract_tokens_usage)if self._reporting_config.is_log_costs_to_console:@@ -170,7 +206,7 @@ class ReportingManager(ReportingProtocol):returnpipeline_run_id = search_job.job_metadata.pipeline_run_id- self._get_registry(pipeline_run_id).add_tokens_usage(search_tokens_usage)+ self._try_add_to_registry(pipeline_run_id, search_tokens_usage)self._emit_usage_event(search_job, search_tokens_usage)if self._reporting_config.is_log_costs_to_console:@@ -198,30 +234,51 @@ class ReportingManager(ReportingProtocol):pipeline_run_id: The pipeline run to add usage data to.tokens_usages: Token usage records to inject."""- registry = self._get_registry(pipeline_run_id)+ registry = self._get_or_create_registry(pipeline_run_id)for tokens_usage in tokens_usages:registry.add_tokens_usage(tokens_usage)def _emit_usage_event(self, inference_job: InferenceJobAbstract, tokens_usage: AnyTokensUsage) -> None:- """Emit a UsageReportEvent if event log is configured for this job's context."""+ """Emit a UsageReportEvent for this job.++ Fast path: when set_event_log was registered for this graph context's+ lookup_key (router process or direct mode), emit through the cached+ per-context event log.++ Fallback: when context lookup misses (runner process — set_event_log+ was never called here), emit through the per-process activity event+ log so the event still lands in the same backend partition as the+ rest of the run. See _emit_usage_event_runner_fallback for details.+ """graph_context = inference_job.job_metadata.graph_contextif graph_context is None:returncontext = self._event_log_contexts.get(graph_context.lookup_key)- if context is None:+ if context is not None:+ self._emit_via_registered_context(context, graph_context, tokens_usage)return- # Determine the node_id from graph context (the pipe that dispatched this inference)- node_id: str = "unknown"- if graph_context.parent_node_id is not None:- node_id = graph_context.parent_node_id+ self._emit_usage_event_runner_fallback(+ inference_job=inference_job,+ tokens_usage=tokens_usage,+ graph_context=graph_context,+ )+ @staticmethod+ def _emit_via_registered_context(+ context: _EventLogContext,+ graph_context: GraphContext,+ tokens_usage: AnyTokensUsage,+ ) -> None:+ """Fast-path emit through a context registered via set_event_log."""+ node_id = graph_context.parent_node_id or "unknown"seq = context.event_log.next_sequence()event = UsageReportEvent(pipeline_run_id=context.pipeline_run_id,workflow_id=context.workflow_id,+ writer_id=context.event_log.writer_id,timestamp=datetime.now(timezone.utc),sequence=seq,node_id=node_id,@@ -229,6 +286,71 @@ class ReportingManager(ReportingProtocol):)context.event_log.emit(event)+ def _emit_usage_event_runner_fallback(+ self,+ inference_job: InferenceJobAbstract,+ tokens_usage: AnyTokensUsage,+ graph_context: GraphContext,+ ) -> None:+ """Emit through a per-process activity event log when no context was registered.++ On the runner, ``set_event_log`` was never called — the workflow only+ registered a context on the router process. We fall back to a+ process-local event log built from ``tracing_config``, stamped with a+ stable per-process writer_id of the form ``act_{pid}_{uuid8}``.++ Documented over-counting risk (R2): retried activities re-emit a fresh+ event at sequence N+1 instead of overwriting the original at N, so the+ same usage may be counted twice if the activity is retried by Temporal.+ Suppression of retried-emit duplicates is a separate, harder problem+ (deferred follow-up).++ Specific exceptions caught and dropped with WARNING:+ - ``OSError``: NDJSON dir unwritable, file system errors.+ - ``MissingDependencyError``: ``boto3`` missing for the DynamoDB backend.+ - ``PipelexConfigError``: factory misconfigured.+ - ``botocore.exceptions.ClientError`` (when boto3 is installed):+ DynamoDB throttle / auth fail at PutItem time.+ Other exceptions propagate.+ """+ tracing_config = get_config().pipelex.tracing_config+ if not tracing_config.is_enabled:+ return++ try:+ process_event_log = ActivityEventLogCache.get_or_create(tracing_config)+ except (OSError, MissingDependencyError, PipelexConfigError) as exc:+ log.warning(f"Runner-side activity event log construction failed; dropping usage event: {exc}")+ return++ if process_event_log is None:+ return++ workflow_id = graph_context.tracer_key or graph_context.graph_id+ node_id = graph_context.parent_node_id or "unknown"++ ActivityEventLogCache.warn_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)++ seq = process_event_log.next_sequence()+ event = UsageReportEvent(+ pipeline_run_id=inference_job.job_metadata.pipeline_run_id,+ workflow_id=workflow_id,+ writer_id=process_event_log.writer_id,+ timestamp=datetime.now(timezone.utc),+ sequence=seq,+ node_id=node_id,+ tokens_usage=tokens_usage,+ )++ emit_exceptions: tuple[type[BaseException], ...] = (OSError,)+ if _BotoClientError is not None:+ emit_exceptions = (*emit_exceptions, _BotoClientError)++ try:+ process_event_log.emit(event)+ except emit_exceptions as exc:+ log.warning(f"Runner-side usage event emit failed; dropping: {exc}")+@overridedef report_inference_job(self, inference_job: InferenceJobAbstract):log.verbose(f"Inference job '{inference_job.job_metadata.unit_job_id}' completed in {inference_job.job_metadata.duration:.2f} seconds")@@ -256,7 +378,7 @@ class ReportingManager(ReportingProtocol):registries_to_process: dict[str, UsageRegistry] = {}if pipeline_run_id:- registries_to_process = {pipeline_run_id: self._get_registry(pipeline_run_id)}+ registries_to_process = {pipeline_run_id: self._get_or_create_registry(pipeline_run_id)}else:registries_to_process = self._usage_registriesdiff --git a/pipelex/temporal/config_temporal.py b/pipelex/temporal/config_temporal.pyindex 4444decc4..059cd3ed9 100644--- a/pipelex/temporal/config_temporal.py+++ b/pipelex/temporal/config_temporal.py@@ -175,6 +175,7 @@ class WorkerConfig(ConfigModel):"""Configuration model for workflow execution settings."""task_queue: str+ inference_task_queue: str | None = Noneworkflow_execution_timeout: timedelta = Field(strict=False)run_timeout: timedelta | None = Field(default=None, strict=False)task_timeout: timedelta | None = Field(default=None, strict=False)diff --git a/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.pyindex f635e5bed..48f296c57 100644--- a/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py+++ b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py@@ -27,6 +27,7 @@ class WfMakeLLMText(WorkflowClass[LLMAssignment, str]):arg=workflow_arg,start_to_close_timeout=worker_config.workflow_execution_timeout,retry_policy=worker_config.retry_policy,+ task_queue=worker_config.inference_task_queue,)except ActivityError as exc:if isinstance(exc.cause, ApplicationError):diff --git a/pipelex/tracing/activity_event_log.py b/pipelex/tracing/activity_event_log.pynew file mode 100644index 000000000..262051c66--- /dev/null+++ b/pipelex/tracing/activity_event_log.py@@ -0,0 +1,128 @@+"""Process-local cached event log for runner-side activity emission.++When activities run on a separate worker pool from the workflow router,+the router's `set_event_log` never registers a context on the runner's+ReportingManager — the runner-side `_event_log_contexts` dict is permanently+empty for every workflow that runs there. This module provides the per-process+backstop: a lazily-constructed event log, stamped with a stable+`act_{pid}_{uuid8}` writer_id, that emits into the same backend partition as+the rest of the run.++The cache-and-writer-id construction is guarded by `threading.Lock` with+double-checked locking so concurrent first-emitters from N activity threads+agree on a single writer_id and reuse the same backend instance.+"""++import atexit+import os+import threading+from typing import ClassVar+from uuid import uuid4++from pipelex import log+from pipelex.system.configuration.configs import TracingConfig+from pipelex.tracing.event_log_factory import make_event_log+from pipelex.tracing.event_log_protocol import EventLogProtocol+++class ActivityEventLogCache:+ """Process-local cache holding the runner-side activity event log.++ All state is stored as ``ClassVar`` so the cache is naturally shared across+ every activity thread inside the worker process, with no module-level+ globals. ``_lock`` guards cache construction and the one-shot WARNING flag.+ """++ _lock: ClassVar[threading.Lock] = threading.Lock()+ _cached_event_log: ClassVar[EventLogProtocol | None] = None+ _writer_id: ClassVar[str | None] = None+ _warning_emitted: ClassVar[bool] = False+ _atexit_registered: ClassVar[bool] = False++ @classmethod+ def get_or_create(cls, tracing_config: TracingConfig) -> EventLogProtocol | None:+ """Return the per-process event log for runner-side emission.++ Returns None when tracing is disabled — callers must skip emit. On the+ first successful call, generates a stable per-process writer_id of the+ form ``act_{pid}_{uuid8}`` and constructs the backend via+ :func:`pipelex.tracing.event_log_factory.make_event_log`. Subsequent+ calls reuse the cached instance.++ Thread-safety: cache check + creation is wrapped in a class-level+ :class:`threading.Lock` (double-checked) so concurrent first-callers+ from N activity threads observe the same writer_id and the same+ backend instance.++ The caller is responsible for catching specific exceptions that+ ``make_event_log`` may raise (``OSError``, ``MissingDependencyError``,+ ``PipelexConfigError``); this method does not swallow them so the+ runner-side emit site can log a single WARNING with the right context.+ """+ if not tracing_config.is_enabled:+ return None+ if cls._cached_event_log is not None:+ return cls._cached_event_log+ with cls._lock:+ if cls._cached_event_log is not None:+ return cls._cached_event_log+ new_writer_id = f"act_{os.getpid()}_{uuid4().hex[:8]}"+ new_event_log = make_event_log(tracing_config, writer_id=new_writer_id)+ cls._writer_id = new_writer_id+ cls._cached_event_log = new_event_log+ if not cls._atexit_registered:+ atexit.register(cls._close_atexit)+ cls._atexit_registered = True+ return cls._cached_event_log++ @classmethod+ def warn_once_runner_fallback_engaged(cls, workflow_id: str, writer_id: str) -> None:+ """Log a single WARNING the first time the runner fallback engages.++ The warning is per-process; subsequent fallback emissions from the same+ process are silent. Operators see one unmistakable signal in the worker+ logs that runner-side emission is in use, without WARNING spam at high+ activity throughput.+ """+ if cls._warning_emitted:+ return+ with cls._lock:+ if cls._warning_emitted:+ return+ cls._warning_emitted = True+ log.warning(+ f"Runner-side usage event emission engaged "+ f"(workflow_id={workflow_id}, writer_id={writer_id}). "+ "Activity is emitting into the per-process activity event log because "+ "no _event_log_contexts entry was registered in this process. "+ "This is expected when activities run on a separate worker pool from the workflow router."+ )++ @classmethod+ def reset_for_tests(cls) -> None:+ """Reset cache state. For test fixtures only — never call from production code.++ Closes any cached event log, clears the writer_id, and resets the one-shot+ warning flag so subsequent tests observe a fresh process-like state. The+ atexit registration flag is intentionally left alone — the registration+ is process-global and the handler is a no-op once the cache is cleared.+ """+ if cls._cached_event_log is not None:+ try:+ cls._cached_event_log.close()+ except OSError:+ pass+ cls._cached_event_log = None+ cls._writer_id = None+ cls._warning_emitted = False++ @classmethod+ def _close_atexit(cls) -> None:+ """Best-effort close of the cached event log at interpreter shutdown."""+ if cls._cached_event_log is None:+ return+ try:+ cls._cached_event_log.close()+ except OSError:+ # Interpreter shutdown — file handles may already be torn down. Drop quietly.+ passdiff --git a/pipelex/tracing/buffering_event_log.py b/pipelex/tracing/buffering_event_log.pyindex 95fd0c642..ff55bd1d8 100644--- a/pipelex/tracing/buffering_event_log.py+++ b/pipelex/tracing/buffering_event_log.py@@ -4,6 +4,8 @@ Collects trace events in a list with zero I/O. Events are flushed to thereal backend (DynamoDB) via a Temporal activity after pipe execution."""+import threading+from typing_extensions import overridefrom pipelex.tracing.event_log_protocol import EventLogProtocol@@ -18,16 +20,30 @@ class BufferingEventLog(EventLogProtocol):flush them to the real backend via act_flush_trace_events."""- def __init__(self) -> None:+ def __init__(self, writer_id: str = "primary") -> None:self._buffer: list[TraceEvent] = []self._sequence: int = 0+ self._sequence_lock = threading.Lock()+ self._writer_id = writer_id++ @property+ @override+ def writer_id(self) -> str:+ return self._writer_id@overridedef next_sequence(self) -> int:- """Return the next sequence number. Shared by all emitters."""- seq = self._sequence- self._sequence += 1- return seq+ """Return the next sequence number. Shared by all emitters.++ The Temporal workflow sandbox is single-threaded so this lock is+ practically uncontended, but the contract — "implementations must be+ safe under concurrent emitters" — is the same for every backend, so+ we keep the guard for symmetry.+ """+ with self._sequence_lock:+ seq = self._sequence+ self._sequence += 1+ return seq@overridedef emit(self, event: TraceEvent) -> None:diff --git a/pipelex/tracing/dynamodb_event_log.py b/pipelex/tracing/dynamodb_event_log.pyindex f95d81440..368cd91f9 100644--- a/pipelex/tracing/dynamodb_event_log.py+++ b/pipelex/tracing/dynamodb_event_log.py@@ -2,13 +2,16 @@Stores trace events in a DynamoDB table with the schema:PK: PIPELINE_RUN#{pipeline_run_id}- SK: EVENT#{workflow_id}#{sequence:010d}+ SK: EVENT#{workflow_id}#{writer_id}#{sequence:010d}payload: dict (full event via model_dump)+Legacy emissions write writer_id="primary" so existing rows remain valid.+Compatible with the pipelex-api-infra TraceEventDynamoDBAdapter schema.Requires: pip install "pipelex[dynamodb]""""+import threadingfrom typing import Anyfrom pydantic import TypeAdapter, ValidationError@@ -43,7 +46,7 @@ class DynamoDBEventLog(EventLogProtocol):the deadlock detector. Use BufferingEventLog + act_flush_trace_events instead."""- def __init__(self, table_name: str, region: str) -> None:+ def __init__(self, table_name: str, region: str, writer_id: str = "primary") -> None:if boto3 is None:lib_name = "boto3"lib_extra_name = "dynamodb"@@ -52,24 +55,36 @@ class DynamoDBEventLog(EventLogProtocol):self._table_name = table_nameself._region = region+ self._writer_id = writer_idself._sequence: int = 0+ self._sequence_lock = threading.Lock()dynamodb: Any = boto3.resource("dynamodb", region_name=self._region) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]self._table: Any = dynamodb.Table(self._table_name) # pyright: ignore[reportUnknownMemberType]+ @property+ @override+ def writer_id(self) -> str:+ return self._writer_id+@overridedef next_sequence(self) -> int:- """Return the next sequence number. Shared by all emitters."""- seq = self._sequence- self._sequence += 1- return seq+ """Return the next sequence number. Shared by all emitters.++ Guarded by a per-instance lock so concurrent activity threads+ sharing this backend cannot collide on the dedup key.+ """+ with self._sequence_lock:+ seq = self._sequence+ self._sequence += 1+ return seq@staticmethoddef _make_pk(pipeline_run_id: str) -> str:return f"PIPELINE_RUN#{pipeline_run_id}"@staticmethod- def _make_sk(workflow_id: str, sequence: int) -> str:- return f"EVENT#{workflow_id}#{sequence:010d}"+ def _make_sk(workflow_id: str, writer_id: str, sequence: int) -> str:+ return f"EVENT#{workflow_id}#{writer_id}#{sequence:010d}"def _key_condition(self, pipeline_run_id: str) -> Any:"""Build a KeyConditionExpression for querying by pipeline_run_id."""@@ -80,9 +95,10 @@ class DynamoDBEventLog(EventLogProtocol):"""Write a single event to DynamoDB. Synchronous and idempotent."""item: dict[str, Any] = {"PK": self._make_pk(event.pipeline_run_id),- "SK": self._make_sk(event.workflow_id, event.sequence),+ "SK": self._make_sk(event.workflow_id, event.writer_id, event.sequence),"pipeline_run_id": event.pipeline_run_id,"workflow_id": event.workflow_id,+ "writer_id": event.writer_id,"sequence": event.sequence,"payload": event.model_dump_json(),}diff --git a/pipelex/tracing/event_log_factory.py b/pipelex/tracing/event_log_factory.pyindex b9f145a2b..d4715f03c 100644--- a/pipelex/tracing/event_log_factory.py+++ b/pipelex/tracing/event_log_factory.py@@ -7,14 +7,19 @@ from pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.ndjson_event_log import NdjsonEventLog-def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:- """Create an event log backend from tracing configuration."""+def make_event_log(tracing_config: TracingConfig, writer_id: str = "primary") -> EventLogProtocol:+ """Create an event log backend from tracing configuration.++ The writer_id is stamped on every TraceEvent emitted through the resulting+ log. Defaults to "primary" so existing call sites keep emitting into the+ legacy single-writer NDJSON file / DDB sort-key shape.+ """match tracing_config.backend:case TracingBackend.NDJSON:if tracing_config.ndjson is None:msg = "ndjson config is required when backend is 'ndjson'"raise PipelexConfigError(msg)- return NdjsonEventLog(traces_dir=tracing_config.ndjson.traces_dir)+ return NdjsonEventLog(traces_dir=tracing_config.ndjson.traces_dir, writer_id=writer_id)case TracingBackend.DYNAMODB:if tracing_config.dynamodb is None:msg = "dynamodb config is required when backend is 'dynamodb'"@@ -22,6 +27,7 @@ def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:return DynamoDBEventLog(table_name=tracing_config.dynamodb.table_name,region=tracing_config.dynamodb.region,+ writer_id=writer_id,)case TracingBackend.TEMPORAL_DYNAMODB:if tracing_config.temporal_dynamodb is None:@@ -30,4 +36,5 @@ def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:return DynamoDBEventLog(table_name=tracing_config.temporal_dynamodb.table_name,region=tracing_config.temporal_dynamodb.region,+ writer_id=writer_id,)diff --git a/pipelex/tracing/event_log_protocol.py b/pipelex/tracing/event_log_protocol.pyindex fa70c90a4..1006f3cfb 100644--- a/pipelex/tracing/event_log_protocol.py+++ b/pipelex/tracing/event_log_protocol.py@@ -16,14 +16,37 @@ class EventLogProtocol(Protocol):Implementations must guarantee that emit() is synchronous and durable:when emit() returns, the event must be persisted. This invariant ensuresall events from a child workflow are flushed before it returns to its parent.++ Each event log instance carries an immutable `writer_id` that emitters+ stamp on every TraceEvent they construct via this log. Two writers+ sharing the same (pipeline_run_id, workflow_id) partition must use+ distinct writer_ids so the read-side dedup key+ (workflow_id, writer_id, type, sequence) keeps their events apart."""+ @property+ def writer_id(self) -> str:+ """Stable identifier for this event log instance.++ Emitters read this to stamp every constructed TraceEvent. Defaults+ to "primary" for the legacy single-writer path so existing NDJSON+ files / DDB rows are not invalidated by code change.+ """+ ...+def next_sequence(self) -> int:"""Return the next monotonically increasing sequence number.All emitters sharing the same event log must use this method to- obtain sequence numbers, ensuring events never collide on the- (workflow_id, sequence) key regardless of storage backend.+ obtain sequence numbers. Combined with writer_id, this guarantees+ events never collide on the (workflow_id, writer_id, sequence) key+ regardless of storage backend.++ Implementations MUST be safe to call from multiple threads+ concurrently — the runner-side activity event log is cached+ per-process and shared by every activity thread on the worker, so+ unguarded reads/writes of the underlying counter would cause+ duplicate sequence numbers and silent dedup-key collisions."""...diff --git a/pipelex/tracing/in_memory_event_log.py b/pipelex/tracing/in_memory_event_log.pyindex 639104e16..2fcf3bfa9 100644--- a/pipelex/tracing/in_memory_event_log.py+++ b/pipelex/tracing/in_memory_event_log.py@@ -1,5 +1,7 @@"""In-memory event log implementation for unit tests and direct mode."""+import threading+from typing_extensions import overridefrom pipelex.tracing.event_log_protocol import EventLogProtocol@@ -13,16 +15,28 @@ class InMemoryEventLog(EventLogProtocol):Provides the same deduplication and ordering guarantees as NdjsonEventLog."""- def __init__(self) -> None:+ def __init__(self, writer_id: str = "primary") -> None:self._events: list[TraceEvent] = []self._sequence: int = 0+ self._sequence_lock = threading.Lock()+ self._writer_id = writer_id++ @property+ @override+ def writer_id(self) -> str:+ return self._writer_id@overridedef next_sequence(self) -> int:- """Return the next sequence number. Shared by all emitters."""- seq = self._sequence- self._sequence += 1- return seq+ """Return the next sequence number. Shared by all emitters.++ Guarded by a per-instance lock so concurrent emitters cannot+ collide on the dedup key.+ """+ with self._sequence_lock:+ seq = self._sequence+ self._sequence += 1+ return seq@overridedef emit(self, event: TraceEvent) -> None:@@ -33,20 +47,22 @@ class InMemoryEventLog(EventLogProtocol):def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:"""Return events for a pipeline run, deduplicated and sorted.- Deduplicates by (workflow_id, sequence), keeping the first occurrence.- Sorts by (workflow_id, sequence) for deterministic ordering.+ Deduplicates by (workflow_id, writer_id, type, sequence), keeping+ the first occurrence. Sorts by (workflow_id, sequence, writer_id) —+ sequence is primary so two writers emitting into the same workflow+ partition do not get reordered by writer-id lexicographic sort."""filtered = [evt for evt in self._events if evt.pipeline_run_id == pipeline_run_id]- seen: set[tuple[str, str, int]] = set()+ seen: set[tuple[str, str, str, int]] = set()deduped: list[TraceEvent] = []for event in filtered:- dedup_key = (event.workflow_id, type(event).__name__, event.sequence)+ dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)if dedup_key not in seen:seen.add(dedup_key)deduped.append(event)- deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence))+ deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))return deduped@overridediff --git a/pipelex/tracing/ndjson_event_log.py b/pipelex/tracing/ndjson_event_log.pyindex 00deb7e82..07155820b 100644--- a/pipelex/tracing/ndjson_event_log.py+++ b/pipelex/tracing/ndjson_event_log.py@@ -7,6 +7,7 @@ One JSON event per line, one file per workflow, organized by pipeline run:import jsonimport osimport shutil+import threadingfrom pathlib import Pathfrom typing import IO@@ -25,20 +26,59 @@ class NdjsonEventLog(EventLogProtocol):Write path: appends one JSON line per event, flushed immediately.Read path: globs all .ndjson files in the run directory, parses,- deduplicates by (workflow_id, sequence), and sorts deterministically.+ deduplicates by (workflow_id, writer_id, type, sequence), and sorts+ by (workflow_id, sequence, writer_id) — sequence primary so a+ runner-side writer's events do not sort before earlier router events.++ Multi-writer file naming: events from writer_id="primary" land in the+ legacy file `wf_{workflow_id}.ndjson`; events from any other writer+ land in `wf_{workflow_id}__w_{writer_id}.ndjson`. The file-handle cache+ key is `(pipeline_run_id, workflow_id, writer_id)` so two writers+ emitting concurrently never share a stale handle.++ For multi-process / multi-host deployments, traces_dir must be a+ filesystem visible to all writer processes (NFS/EFS); use the+ DynamoDB backend for fully separated hosts."""- def __init__(self, traces_dir: str) -> None:+ def __init__(self, traces_dir: str, writer_id: str = "primary") -> None:self._traces_dir = traces_dir- self._file_handles: dict[tuple[str, str], IO[str]] = {}+ self._file_handles: dict[tuple[str, str, str], IO[str]] = {}self._sequence: int = 0+ self._sequence_lock = threading.Lock()+ self._writer_id = writer_id++ @property+ @override+ def writer_id(self) -> str:+ return self._writer_id@overridedef next_sequence(self) -> int:- """Return the next sequence number. Shared by all emitters."""- seq = self._sequence- self._sequence += 1- return seq+ """Return the next sequence number. Shared by all emitters.++ The increment is guarded by a per-instance lock so concurrent activity+ threads sharing this backend (via ``get_or_create_activity_event_log``)+ cannot read the same value before either increments — duplicate+ sequence numbers would collide on the+ ``(workflow_id, writer_id, type, sequence)`` dedup key and silently+ drop one event.+ """+ with self._sequence_lock:+ seq = self._sequence+ self._sequence += 1+ return seq++ @staticmethod+ def _file_name_for(workflow_id: str, writer_id: str) -> str:+ """File name for a (workflow_id, writer_id) pair.++ The legacy single-writer name is preserved when writer_id="primary"+ so existing files continue to be written and read correctly.+ """+ if writer_id == "primary":+ return f"wf_{workflow_id}.ndjson"+ return f"wf_{workflow_id}__w_{writer_id}.ndjson"# ------------------------------------------------------------------# Write@@ -49,15 +89,16 @@ class NdjsonEventLog(EventLogProtocol):"""Append event as a JSON line, flush immediately.Creates the run directory on first write. Caches file handles- to avoid repeated open() calls for high-frequency events.+ keyed by (pipeline_run_id, workflow_id, writer_id) to avoid+ repeated open() calls for high-frequency events."""- cache_key = (event.pipeline_run_id, event.workflow_id)+ cache_key = (event.pipeline_run_id, event.workflow_id, event.writer_id)handle = self._file_handles.get(cache_key)if handle is None:run_dir = os.path.join(self._traces_dir, event.pipeline_run_id)os.makedirs(run_dir, exist_ok=True)- file_path = os.path.join(run_dir, f"wf_{event.workflow_id}.ndjson")+ file_path = os.path.join(run_dir, self._file_name_for(event.workflow_id, event.writer_id))handle = open(file_path, "a", encoding="utf-8") # noqa: SIM115self._file_handles[cache_key] = handle@@ -83,7 +124,7 @@ class NdjsonEventLog(EventLogProtocol):ndjson_files = sorted(Path(run_dir).glob("*.ndjson"))- seen: set[tuple[str, str, int]] = set()+ seen: set[tuple[str, str, str, int]] = set()events: list[TraceEvent] = []for ndjson_path in ndjson_files:@@ -99,7 +140,7 @@ class NdjsonEventLog(EventLogProtocol):log.warning(f"Skipping corrupt line in {file_path}:{line_number} — {exc}")continue- dedup_key = (event.workflow_id, type(event).__name__, event.sequence)+ dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)if dedup_key not in seen:seen.add(dedup_key)events.append(event)@@ -108,7 +149,7 @@ class NdjsonEventLog(EventLogProtocol):# workflow ID, not execution order. In parent/child workflow topologies this can cause# incorrect producer map overwrites in GraphSpecAssembler. Consider timestamp-based# or topology-aware ordering.- events.sort(key=lambda evt: (evt.workflow_id, evt.sequence))+ events.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))return events# ------------------------------------------------------------------@@ -119,9 +160,9 @@ class NdjsonEventLog(EventLogProtocol):def cleanup(self, pipeline_run_id: str) -> None:"""Close cached file handles and remove the run directory."""keys_to_remove = [key for key in self._file_handles if key[0] == pipeline_run_id]- for key in keys_to_remove:- self._file_handles[key].close()- del self._file_handles[key]+ for cache_key in keys_to_remove:+ self._file_handles[cache_key].close()+ del self._file_handles[cache_key]run_dir = os.path.join(self._traces_dir, pipeline_run_id)if os.path.isdir(run_dir):diff --git a/pipelex/tracing/trace_events.py b/pipelex/tracing/trace_events.pyindex 0445db9d3..00edb687b 100644--- a/pipelex/tracing/trace_events.py+++ b/pipelex/tracing/trace_events.py@@ -37,6 +37,12 @@ class TraceEvent(BaseModel):Shared fields:- pipeline_run_id: identifies the pipeline execution- workflow_id: Temporal workflow ID or "direct" for in-process mode+ - writer_id: identifies the emitting event-log instance, so two writers+ sharing the same (pipeline_run_id, workflow_id) partition can emit+ independent sequence streams without colliding. Defaults to "primary",+ which preserves the legacy single-writer NDJSON file naming and DDB+ sort-key shape; non-default values come from the per-process activity+ event log used by separate-process workers.- event_kind: discriminator for the event subclass (defined on each subclass as Literal)- timestamp: UTC, for display/debugging only- sequence: per-writer monotonic counter, for ordering and deduplication@@ -44,6 +50,7 @@ class TraceEvent(BaseModel):pipeline_run_id: strworkflow_id: str+ writer_id: str = "primary"timestamp: datetimesequence: intdiff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.pyindex b85bb33ec..b171c4a39 100644--- a/tests/integration/pipelex/temporal/tracing/helpers.py+++ b/tests/integration/pipelex/temporal/tracing/helpers.py@@ -2,16 +2,30 @@import uuidfrom collections import defaultdict+from collections.abc import AsyncIterator+from contextlib import asynccontextmanager+from datetime import datetimefrom pathlib import Path+from temporalio import activityfrom temporalio.client import Client as TemporalClient+from pipelex.cogt.content_generation.assignment_models import LLMAssignment+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReport+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_configfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.graph.graph_config import DataInclusionConfigfrom pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, PipelineRef+from pipelex.hub import get_report_delegatefrom pipelex.pipe_run.pipe_job import PipeJob+from pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_textfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouterfrom pipelex.tracing.graphspec_assembler import GraphSpecAssemblerfrom pipelex.tracing.ndjson_event_log import NdjsonEventLog@@ -133,3 +147,129 @@ def ndjson_files_for_run(traces_dir: str, pipeline_run_id: str) -> list[Path]:def node_pipe_codes(graph_spec: GraphSpec) -> set[str | None]:"""Extract the set of pipe_codes from all nodes in a GraphSpec."""return {node.pipe_code for node in graph_spec.nodes}+++_RUNNER_FAKE_INFERENCE_MODEL_NAME = "split_runner_fake"+_RUNNER_FAKE_INFERENCE_MODEL_ID = "split_runner_fake_id"+_RUNNER_FAKE_RESPONSE_TEXT = "split-worker fake response"++[email protected](name="act_llm_gen_text")+async def _runner_isolated_act_llm_gen_text(llm_assignment: LLMAssignment) -> str: # noqa: RUF029+ """Substitute for `act_llm_gen_text` that exercises the Phase-2 runner-side fallback.++ Within a single pytest process the router worker registers a context via+ `set_event_log` on the process-wide `ReportingManager` singleton; a real+ runner activity in the same process would otherwise hit that same dict and+ take the fast path, masking the cross-worker bug. Instead this activity:++ - Clears `_event_log_contexts` on entry, simulating a cold runner process.+ - Synthesizes an `LLMJob` with non-zero usage and reports it via+ `report_inference_job`. The reporting path runs synchronously in the+ activity, so `_emit_usage_event` sees the cleared contexts dict and+ takes the fallback (writes to the per-process activity event log).+ - Returns a fixed string so the calling workflow can finish without+ attempting a real LLM call.++ This avoids the LIVE/DRY-mode dilemma: the existing `pipe_job_from_bundle`+ fixture defaults to DRY mode, where the LLM activity is normally bypassed+ by `ContentGeneratorDry` reporting inline inside the workflow. Subbing+ `act_llm_gen_text` with this version forces the activity to fire from the+ workflow's `start_activity` call (because we set+ `inference_task_queue=q_runner`), exercising the cross-worker path even+ in DRY mode and keeping the test hermetic.+ """+ delegate = get_report_delegate()+ if isinstance(delegate, ReportingManager):+ delegate._event_log_contexts.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++ now = datetime.now()+ synthetic_metadata = llm_assignment.job_metadata.model_copy(+ update={+ "started_at": llm_assignment.job_metadata.started_at or now,+ "completed_at": now,+ },+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=synthetic_metadata,+ inference_model_name=_RUNNER_FAKE_INFERENCE_MODEL_NAME,+ inference_model_id=_RUNNER_FAKE_INFERENCE_MODEL_ID,+ unit_costs={CostCategory.INPUT: 0.0, CostCategory.OUTPUT: 0.0},+ nb_tokens_by_category={TokenCategory.INPUT: 1, TokenCategory.OUTPUT: 1},+ )+ synthetic_job = LLMJob(+ job_metadata=synthetic_metadata,+ llm_prompt=llm_assignment.llm_prompt,+ job_params=llm_assignment.llm_setting.make_llm_job_params(),+ job_config=LLMJobConfig(max_retries=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+ delegate.report_inference_job(inference_job=synthetic_job)+ return _RUNNER_FAKE_RESPONSE_TEXT+++@asynccontextmanager+async def make_split_workers(+ temporal_client: TemporalClient,+ q_router: str,+ q_runner: str,+) -> AsyncIterator[None]:+ """Open two scoped workers on two task queues in the current process.++ - `q_router`: workflows + `act_flush_trace_events`. We don't use the bare+ `router` scope (`disable_all_activities=True`) here because+ `WfPipeRouter.run` dispatches `act_flush_trace_events` without a+ `task_queue` argument — the activity lands on the workflow's own queue+ and would never be picked up if the router registered no activities.+ - `q_runner`: activity-only (runner scope, `disable_all_workflows=True`),+ with `act_llm_gen_text` substituted by the isolation wrapper that clears+ the in-process `_event_log_contexts` cache so the runner cannot+ accidentally use the router's registered context.++ Pair this with `worker_config.inference_task_queue = q_runner` so the+ workflow on `q_router` actually dispatches `act_llm_gen_text` to `q_runner`.+ """+ worker_scopes = get_config().temporal.worker_scopes+ base_router = worker_scopes.scopes["router"]+ runner_scope = worker_scopes.scopes["runner"]++ # Override the bare router scope to also register `act_flush_trace_events`+ # so the workflow's flush call does not deadlock waiting for an activity+ # worker on q_router. We keep `disable_all_activities=True` and add the+ # activity back via `required_activities` — but resolution clears+ # `disable_all_activities` last, so we instead use an empty exclude set+ # against the runner scope's activity surface.+ router_scope = base_router.model_copy(+ update={+ "disable_all_activities": False,+ "required_activities": ["act_flush_trace_events"],+ "excluded_activities": [+ "act_llm_gen_text",+ "act_llm_gen_object",+ "act_llm_gen_object_list",+ "act_img_gen_images",+ "act_jinja2_gen_text",+ "act_extract_gen_extract_pages",+ "act_render_page_views",+ "act_assemble_graph",+ "act_deliver",+ ],+ },+ )++ async with (+ get_task_manager().make_worker(+ temporal_client,+ task_queue=q_router,+ is_not_sandboxed=True,+ scope=router_scope,+ ),+ get_task_manager().make_worker(+ temporal_client,+ task_queue=q_runner,+ is_not_sandboxed=True,+ scope=runner_scope,+ substitute_activities={act_llm_gen_text: _runner_isolated_act_llm_gen_text},+ ),+ ):+ yielddiff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.pynew file mode 100644index 000000000..bf53616fb--- /dev/null+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py@@ -0,0 +1,190 @@+"""Integration tests for cross-worker UsageReportEvent emission via two task queues.++Validates the Phase 2 runner-side fallback: when `WfMakeLLMText` dispatches+`act_llm_gen_text` to a separate task queue (the production deployment topology+where router and runner are physically separated), the runner-side worker —+with no `_event_log_contexts` entry registered for the workflow — emits its+`UsageReportEvent` into the same `traces_dir` partition via the per-process+activity event log, stamped with a stable `act_*` `writer_id`.+"""++import uuid+from collections import defaultdict+from collections.abc import Generator+from pathlib import Path+from typing import TYPE_CHECKING++import pytest+from temporalio.client import Client as TemporalClient++from pipelex.config import get_config+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from pipelex.tracing.activity_event_log import ActivityEventLogCache+from pipelex.tracing.ndjson_event_log import NdjsonEventLog+from pipelex.tracing.trace_events import PipeStartEvent, UsageReportEvent+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output+from tests.integration.pipelex.temporal.tracing.helpers import (+ inject_graph_context,+ make_split_workers,+ ndjson_files_for_run,+)+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++[email protected](loop_scope="class")+class TestSplitWorkerUsageEmission:+ """Cross-process emission: router on q_router, runner on q_runner.++ Each test uses its own pair of task queues (UUID-named) so concurrent+ executions don't share queue state, and resets the per-process activity+ event log cache so the writer_id is regenerated and the warn-once flag+ re-arms for each test.+ """++ @pytest.fixture+ def split_queues(self) -> tuple[str, str]:+ """A fresh (q_router, q_runner) pair per test."""+ return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")++ @pytest.fixture+ def live_sequence_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:+ """Like the class-scoped `sequence_tracing_job` but in LIVE mode.++ DRY mode short-circuits the `act_llm_gen_text` activity dispatch+ (`ContentGeneratorDry` reports inline inside the workflow), so the+ cross-worker activity hop never fires. LIVE mode forces the workflow+ to dispatch through `WfMakeLLMText` → `act_llm_gen_text`, which is+ what the runner-side fallback pinned in this suite is supposed to+ exercise. The fake `act_llm_gen_text` substitute installed by+ `make_split_workers` returns a stub string and synthesizes the+ usage report, so no real LLM call is made.+ """+ yield from pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=SequenceTracingTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ isolated_registry=is_class_registry_isolated,+ )++ @pytest.fixture+ def configure_inference_task_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:+ """Route `act_llm_gen_text` to `q_runner` for the duration of the test.++ Without this override, `WfMakeLLMText` would dispatch the activity to+ whichever task queue the workflow is running on (the router queue),+ defeating the split-worker topology this suite is supposed to exercise.+ """+ _q_router, q_runner = split_queues+ worker_config = get_config().temporal.worker_config+ original = worker_config.inference_task_queue+ worker_config.inference_task_queue = q_runner+ yield+ worker_config.inference_task_queue = original++ @pytest.fixture(autouse=True)+ def reset_activity_event_log(self) -> Generator[None, None, None]:+ """Clear the per-process activity event log cache between tests.++ The cache (writer_id, event log instance, warn-once flag) is class-level+ on `ActivityEventLogCache`. Without this reset, two tests in the same+ module would share a writer_id and the second test would find a stale,+ possibly closed, event log handle.+ """+ ActivityEventLogCache.reset_for_tests()+ yield+ ActivityEventLogCache.reset_for_tests()++ async def _execute_split(+ self,+ live_sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ split_queues: tuple[str, str],+ ) -> str:+ """Run the workflow on q_router; activities dispatch to q_runner."""+ q_router, q_runner = split_queues+ execution_run_id = f"split_exec_{uuid.uuid4().hex[:12]}"+ execution_job = inject_graph_context(live_sequence_tracing_job, execution_run_id)+ workflow_id = f"wf_{uuid.uuid4().hex[:12]}"++ async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRouter.run,+ arg=execution_job,+ id=workflow_id,+ task_queue=q_router,+ )+ rehydrate_pipe_output(pipe_output)+ return execution_run_id++ async def test_runner_usage_event_lands_in_same_ndjson_dir(+ self,+ live_sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ tracing_tmp_dir: Path,+ split_queues: tuple[str, str],+ configure_inference_task_queue: None, # noqa: ARG002+ ) -> None:+ """Router on q_router dispatches inference activity to q_runner.++ The runner — with no `_event_log_contexts` entry — emits its+ `UsageReportEvent` into the same `traces_dir` partition via the+ Phase-2 fallback. Both the router-written `wf_{wf_id}.ndjson`+ (from `act_flush_trace_events`) and at least one runner-written+ `wf_{wf_id}__w_act_*.ndjson` are present.+ """+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)++ ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), run_id)+ primary_files = [path for path in ndjson_files if "__w_" not in path.name]+ runner_files = [path for path in ndjson_files if "__w_act_" in path.name]+ assert primary_files, f"Expected a primary wf_*.ndjson router file in {[path.name for path in ndjson_files]}"+ assert runner_files, f"Expected at least one wf_*__w_act_*.ndjson runner file in {[path.name for path in ndjson_files]}"++ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))+ try:+ events = reader.read_events(run_id)+ finally:+ reader.close()++ primary_pipe_starts = [evt for evt in events if isinstance(evt, PipeStartEvent) and evt.writer_id == "primary"]+ runner_usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent) and evt.writer_id.startswith("act_")]+ assert primary_pipe_starts, "Expected at least one PipeStartEvent with writer_id='primary' from the router"+ assert runner_usage_events, "Expected at least one UsageReportEvent with writer_id='act_*' from the runner"++ async def test_no_double_emit_in_split_worker_pool(+ self,+ live_sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ tracing_tmp_dir: Path,+ split_queues: tuple[str, str],+ configure_inference_task_queue: None, # noqa: ARG002+ ) -> None:+ """Exactly-once invariant: `_emit_usage_event` takes either the fast+ path OR the fallback path, never both. Submit one workflow; assert+ exactly one `UsageReportEvent` per `(node_id, workflow_id)` across+ all NDJSON files.+ """+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)++ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))+ try:+ events = reader.read_events(run_id)+ finally:+ reader.close()++ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]+ assert usage_events, "Expected at least one UsageReportEvent across the run"++ counts: dict[tuple[str, str], int] = defaultdict(int)+ for usage_event in usage_events:+ counts[usage_event.node_id, usage_event.workflow_id] += 1++ for key, count in counts.items():+ assert count == 1, f"Duplicate UsageReportEvent for (node_id, workflow_id)={key}: count={count}"diff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.pynew file mode 100644index 000000000..8796bb314--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py@@ -0,0 +1,55 @@+"""Pin that ContentGeneratorDry invokes report_inference_job with a synthetic LLMJob.++This is a Phase-2 prerequisite for the Phase-6 e2e cross-worker emission tier:+without the dry generator emitting a UsageReportEvent, the Tier 8 assertions in+.claude/skills/temporal-e2e-validate/SKILL.md cannot run dry — they would have+to require live LLM calls to observe the runner-side fallback.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.pipeline.job_metadata import JobMetadata+++class TestContentGeneratorDryReporting:+ """Pin the ContentGeneratorDry → ReportingManager.report_inference_job hook."""++ @pytest.mark.asyncio+ async def test_make_llm_text_invokes_report_inference_job(self, mocker: MockerFixture) -> None:+ """make_llm_text builds a synthetic LLMJob and reports it through get_report_delegate()."""+ captured_jobs: list[LLMJob] = []++ def _capture(inference_job: LLMJob) -> None:+ captured_jobs.append(inference_job)++ mock_delegate = mocker.MagicMock()+ mock_delegate.report_inference_job.side_effect = _capture+ mocker.patch(+ "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",+ return_value=mock_delegate,+ )++ generator = ContentGeneratorDry()+ job_metadata = JobMetadata(+ user_id="test_user",+ pipeline_run_id="run_dry_001",+ )+ llm_setting = LLMSetting(model="gpt-4o", temperature=0.5)+ llm_prompt = LLMPrompt()++ await generator.make_llm_text(+ job_metadata=job_metadata,+ llm_setting_main=llm_setting,+ llm_prompt_for_text=llm_prompt,+ )++ assert len(captured_jobs) == 1+ reported_job = captured_jobs[0]+ assert isinstance(reported_job, LLMJob)+ assert reported_job.job_metadata.pipeline_run_id == "run_dry_001"+ assert reported_job.job_report.llm_tokens_usage is not Nonediff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.pynew file mode 100644index 000000000..d919e22f7--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py@@ -0,0 +1,68 @@+"""Pin that the dry generator copes with both naive and aware ``started_at``.++The synthetic ``completed_at`` must share the tzinfo of the incoming+``started_at`` so ``JobMetadata.duration`` can subtract them without crossing+naive and aware datetimes (which raises ``TypeError``). Callers in the codebase+do construct aware datetimes (e.g. ``pipe_abstract.py``), so the dry path must+not crash when one of those flows in.+"""++from datetime import datetime, timezone++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.pipeline.job_metadata import JobMetadata+++class TestContentGeneratorDryTimezone:+ @pytest.mark.parametrize(+ "started_at",+ [+ pytest.param(datetime(2020, 1, 1, 10, 0, 0), id="naive"),+ pytest.param(datetime(2020, 1, 1, 10, 0, 0, tzinfo=timezone.utc), id="aware_utc"),+ pytest.param(None, id="none_uses_default_factory"),+ ],+ )+ @pytest.mark.asyncio+ async def test_duration_is_finite_for_any_started_at_tz(+ self,+ started_at: datetime | None,+ mocker: MockerFixture,+ ) -> None:+ captured_jobs: list[LLMJob] = []++ def _capture(inference_job: LLMJob) -> None:+ captured_jobs.append(inference_job)++ mock_delegate = mocker.MagicMock()+ mock_delegate.report_inference_job.side_effect = _capture+ mocker.patch(+ "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",+ return_value=mock_delegate,+ )++ generator = ContentGeneratorDry()+ job_metadata = JobMetadata(+ user_id="test_user",+ pipeline_run_id="run_dry_tz",+ started_at=started_at,+ )+ llm_setting = LLMSetting(model="gpt-4o", temperature=0.5)+ llm_prompt = LLMPrompt()++ await generator.make_llm_text(+ job_metadata=job_metadata,+ llm_setting_main=llm_setting,+ llm_prompt_for_text=llm_prompt,+ )++ assert len(captured_jobs) == 1+ reported_metadata = captured_jobs[0].job_metadata+ duration = reported_metadata.duration+ assert duration is not None+ assert duration >= 0.0diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.pynew file mode 100644index 000000000..87bdcfb75--- /dev/null+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py@@ -0,0 +1,388 @@+"""Tests for the runner-side activity event log fallback in ReportingManager._emit_usage_event.++Pins the behavior introduced in Phase 2 of the cross-worker tracing P0 plan:+when context lookup misses (runner process never had set_event_log called for+this workflow), the manager emits the usage event through a per-process+activity event log instead of dropping silently.+"""++import logging+import threading+from datetime import datetime, timedelta, timezone+from pathlib import Path+from typing import Any++import pytest+from botocore.exceptions import ClientError+from pytest_mock import MockerFixture++from pipelex.base_exceptions import PipelexConfigError+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_config+from pipelex.graph.graph_config import DataInclusionConfig+from pipelex.graph.graph_context import GraphContext+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId+from pipelex.reporting.reporting_manager import ReportingManager+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend+from pipelex.system.exceptions import MissingDependencyError+from pipelex.tracing import activity_event_log+from pipelex.tracing.activity_event_log import ActivityEventLogCache+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.ndjson_event_log import NdjsonEventLog+from pipelex.tracing.trace_events import UsageReportEvent++DATA_INCLUSION_OFF = DataInclusionConfig(+ pipe_and_concept_registry=False,+ stuff_json_content=False,+ stuff_text_content=False,+ stuff_html_content=False,+ error_stack_traces=False,+)+++def _make_llm_job(+ pipeline_run_id: str,+ graph_context: GraphContext | None,+ nb_input_tokens: int = 100,+ nb_output_tokens: int = 50,+) -> LLMJob:+ now = datetime.now(timezone.utc)+ job_metadata = JobMetadata(+ user_id="test_user",+ pipeline_run_id=pipeline_run_id,+ graph_context=graph_context,+ started_at=now,+ completed_at=now + timedelta(seconds=1),+ unit_job_id=UnitJobId.LLM_GEN_TEXT,+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},+ nb_tokens_by_category={TokenCategory.INPUT: nb_input_tokens, TokenCategory.OUTPUT: nb_output_tokens},+ )+ return LLMJob(+ job_metadata=job_metadata,+ llm_prompt=LLMPrompt(),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(max_retries=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+++def _make_graph_context(+ graph_id: str,+ parent_node_id: str | None = "g:node_0",+ tracer_key: str | None = None,+) -> GraphContext:+ return GraphContext(+ graph_id=graph_id,+ tracer_key=tracer_key,+ parent_node_id=parent_node_id,+ node_sequence=0,+ data_inclusion=DATA_INCLUSION_OFF,+ )++[email protected](autouse=True)+def _reset_activity_event_log_state() -> Any: # pyright: ignore[reportUnusedFunction]+ """Class-level state on ActivityEventLogCache persists across tests; reset it."""+ ActivityEventLogCache.reset_for_tests()+ yield+ ActivityEventLogCache.reset_for_tests()+++def _enable_ndjson_tracing(mocker: MockerFixture, traces_dir: Path) -> None:+ """Patch the live tracing config to enabled NDJSON pointing at traces_dir."""+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))+++class TestEmitRunnerFallback:+ """Pins the runner-side fallback emission contract."""++ def test_fallback_engages_when_context_missing_and_tracing_enabled(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """When set_event_log is never called and tracing is enabled, the fallback emits an NDJSON usage event."""+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz", parent_node_id="g:node_2")+ llm_job = _make_llm_job("run_abc", graph_context=graph_context)++ manager.report_inference_job(llm_job)++ run_dir = tmp_path / "run_abc"+ ndjson_files = list(run_dir.glob("*.ndjson"))+ assert len(ndjson_files) == 1+ assert ndjson_files[0].name.startswith("wf_wf_xyz__w_act_"), ndjson_files[0].name++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ events = reader.read_events("run_abc")+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]+ assert len(usage_events) == 1+ assert usage_events[0].workflow_id == "wf_xyz"+ assert usage_events[0].pipeline_run_id == "run_abc"+ assert usage_events[0].node_id == "g:node_2"+ assert usage_events[0].writer_id.startswith("act_")++ def test_runner_fallback_uses_tracer_key_when_set(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """workflow_id on the emitted event is graph_context.tracer_key when set, else graph_id."""+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()++ # Case 1: tracer_key set → workflow_id == tracer_key+ ctx_with_tracer = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=ctx_with_tracer))++ # Case 2: tracer_key None → workflow_id == graph_id+ ctx_no_tracer = _make_graph_context(graph_id="run_def", tracer_key=None)+ manager.report_inference_job(_make_llm_job("run_def", graph_context=ctx_no_tracer))++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ events_abc = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]+ events_def = [evt for evt in reader.read_events("run_def") if isinstance(evt, UsageReportEvent)]++ assert len(events_abc) == 1+ assert events_abc[0].workflow_id == "wf_xyz"+ assert len(events_def) == 1+ assert events_def[0].workflow_id == "run_def"++ def test_fallback_caches_event_log_per_process(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """make_event_log is called only once per process — the second emission reuses the cached log."""+ _enable_ndjson_tracing(mocker, tmp_path)+ spy = mocker.spy(activity_event_log, "make_event_log")++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ assert spy.call_count == 1++ def test_concurrent_first_call_yields_single_writer_id(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """Concurrent first-callers from N threads observe the same writer_id and event log instance.++ Pins the threading.Lock fix: without double-checked locking, two threads could each+ construct their own backend, generating two writer_ids and racing on the file.+ """+ _enable_ndjson_tracing(mocker, tmp_path)+ cfg = get_config().pipelex.tracing_config++ nb_threads = 16+ barrier = threading.Barrier(nb_threads)+ seen_logs: list[Any] = []+ seen_lock = threading.Lock()++ def call_fn() -> None:+ barrier.wait()+ log_instance = ActivityEventLogCache.get_or_create(cfg)+ with seen_lock:+ seen_logs.append(log_instance)++ threads = [threading.Thread(target=call_fn) for _ in range(nb_threads)]+ for thread in threads:+ thread.start()+ for thread in threads:+ thread.join(timeout=10)++ assert len(seen_logs) == nb_threads+ # All threads must observe the same backend instance and writer_id.+ for log_instance in seen_logs:+ assert log_instance is seen_logs[0]+ writer_ids = {log_instance.writer_id for log_instance in seen_logs}+ assert len(writer_ids) == 1++ def test_disabled_tracing_skips_emit(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """When tracing_config.is_enabled=False, the fallback returns without writing any file."""+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", False)++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent")++ manager.report_inference_job(_make_llm_job("run_silent", graph_context=graph_context))++ # No NDJSON files should have been created.+ assert list(tmp_path.glob("**/*.ndjson")) == []++ def test_no_graph_context_skips_emit(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """When job_metadata.graph_context is None, neither fast path nor fallback engages."""+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()++ manager.report_inference_job(_make_llm_job("run_abc", graph_context=None))++ assert list(tmp_path.glob("**/*.ndjson")) == []++ @pytest.mark.parametrize(+ ("exc_type", "exc_args"),+ [+ (OSError, ("disk full",)),+ (MissingDependencyError, ("boto3", "dynamodb")),+ (PipelexConfigError, ("misconfigured",)),+ ],+ ids=["oserror", "missing_dep", "config_error"],+ )+ def test_explicit_log_when_emit_path_unavailable(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ exc_type: type[BaseException],+ exc_args: tuple[Any, ...],+ ) -> None:+ """Specific exception types from make_event_log are caught, logged WARNING, and dropped.++ Implementation must NOT use except Exception; each type is named explicitly.+ """+ _enable_ndjson_tracing(mocker, tmp_path)+ mocker.patch(+ "pipelex.tracing.activity_event_log.make_event_log",+ side_effect=exc_type(*exc_args),+ )++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ with caplog.at_level(logging.WARNING):+ # Must not raise — the failure is logged and dropped.+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ assert any(record.levelno == logging.WARNING for record in caplog.records)++ def test_explicit_log_when_emit_raises_client_error(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """A botocore ClientError raised by event_log.emit() is caught, logged WARNING, and dropped."""+ _enable_ndjson_tracing(mocker, tmp_path)++ # Force the event log to raise ClientError on emit.+ stub_log = InMemoryEventLog(writer_id="act_test")+ client_error = ClientError(+ error_response={"Error": {"Code": "ProvisionedThroughputExceededException"}},+ operation_name="PutItem",+ )+ mocker.patch.object(stub_log, "emit", side_effect=client_error)+ mocker.patch(+ "pipelex.tracing.activity_event_log.make_event_log",+ return_value=stub_log,+ )++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ with caplog.at_level(logging.WARNING):+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ assert any(record.levelno == logging.WARNING for record in caplog.records)++ def test_warning_emitted_once_per_process_when_fallback_engages(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """The 'runner fallback engaged' warning fires only on the first emission per process."""+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ with caplog.at_level(logging.WARNING):+ for _ in range(100):+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]+ assert len(engaged_records) == 1++ def test_warning_emitted_once_even_with_multiple_managers(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """Module-level once-flag survives multiple ReportingManager instances."""+ _enable_ndjson_tracing(mocker, tmp_path)++ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ with caplog.at_level(logging.WARNING):+ manager_a = ReportingManager()+ manager_a.setup()+ manager_a.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ manager_b = ReportingManager()+ manager_b.setup()+ manager_b.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]+ assert len(engaged_records) == 1++ def test_retried_activity_emits_duplicate_usage_event_documenting_r2(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ ) -> None:+ """Pins R2: retried activities re-emit; today's behavior is two events at sequence N and N+1.++ Documents over-counting risk and prevents silent regression of this contract.+ """+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")++ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ events = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]+ assert len(events) == 2+ assert sorted(evt.sequence for evt in events) == [0, 1]diff --git a/tests/unit/pipelex/reporting/test_no_orphan_registries.py b/tests/unit/pipelex/reporting/test_no_orphan_registries.pynew file mode 100644index 000000000..65e24ad75--- /dev/null+++ b/tests/unit/pipelex/reporting/test_no_orphan_registries.py@@ -0,0 +1,120 @@+"""Pin the three-method registry split contract.++Phase 3 of the cross-worker tracing P0 plan replaces the single auto-creating+``_get_registry`` with two explicit methods:++- ``_get_registry_strict`` raises ``KeyError`` when the registry was never+ opened. Used by the ``_report_*_job`` paths so the runner-process case+ (no ``open_registry`` call) does not silently accumulate orphan registries.+- ``_get_or_create_registry`` creates on miss. Used by ``inject_tokens_usages``+ (the P1 cross-worker assembly path) and ``generate_report`` for runs not+ opened on this process.+"""++from datetime import datetime, timedelta, timezone++import pytest++from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId+from pipelex.reporting.reporting_manager import ReportingManager+++def _make_llm_job(pipeline_run_id: str) -> LLMJob:+ now = datetime.now(timezone.utc)+ job_metadata = JobMetadata(+ user_id="test_user",+ pipeline_run_id=pipeline_run_id,+ started_at=now,+ completed_at=now + timedelta(seconds=1),+ unit_job_id=UnitJobId.LLM_GEN_TEXT,+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ )+ return LLMJob(+ job_metadata=job_metadata,+ llm_prompt=LLMPrompt(),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(max_retries=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+++class TestNoOrphanRegistries:+ """Phase 3 contract: runner-process reporting must not silently auto-create registries."""++ def test_runner_does_not_accumulate_registries(self) -> None:+ """When open_registry was never called, _report_llm_job must not insert a registry.++ The runner process gets activities reported with pipeline_run_ids it has never seen.+ Pre-Phase-3 behavior auto-created an empty registry on every miss, accumulating+ memory for every distinct pipeline_run_id. Phase 3 makes _report_*_job silently skip+ the add (the usage event still emits via the runner fallback path).+ """+ manager = ReportingManager()+ manager.setup()++ manager.report_inference_job(_make_llm_job("never_opened"))++ assert "never_opened" not in manager._usage_registries # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++ def test_direct_mode_still_accumulates_registry(self) -> None:+ """When open_registry was called for this run, _report_llm_job must add to it."""+ manager = ReportingManager()+ manager.setup()+ manager.open_registry("run_x")++ manager.report_inference_job(_make_llm_job("run_x"))++ registry = manager._get_or_create_registry("run_x") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ usages = registry.get_current_tokens_usage()+ assert len(usages) == 1+ assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 100++ def test_inject_tokens_usages_still_creates_on_miss(self) -> None:+ """inject_tokens_usages preserves auto-create behavior via _get_or_create_registry.++ This is the P1 cross-worker assembly path: usage records collected from trace+ events get injected into a registry that may not have been opened on this+ process — the registry must be created on demand.+ """+ manager = ReportingManager()+ manager.setup()++ tokens_usage = LLMTokensUsage(+ job_metadata=_make_llm_job("new_run").job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},+ nb_tokens_by_category={TokenCategory.INPUT: 50, TokenCategory.OUTPUT: 25},+ )+ manager.inject_tokens_usages(pipeline_run_id="new_run", tokens_usages=[tokens_usage])++ registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ assert len(registry.get_current_tokens_usage()) == 1++ def test_get_registry_strict_raises_when_missing(self) -> None:+ """_get_registry_strict raises KeyError on miss — distinct from _get_or_create_registry."""+ manager = ReportingManager()+ manager.setup()++ with pytest.raises(KeyError):+ manager._get_registry_strict("never_opened") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]++ def test_generate_report_for_unopened_run_creates_empty_registry_and_renders(self) -> None:+ """generate_report on a never-opened run does not crash; renders empty cost report."""+ manager = ReportingManager()+ manager.setup()++ # Must not raise.+ manager.generate_report(pipeline_run_id="never_opened")diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.pyindex e8cdff7c7..51e9ac53b 100644--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py@@ -7,12 +7,15 @@ isolation for concurrent workflows.from datetime import datetime, timedelta, timezone+from pytest_mock import MockerFixture+from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReportfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_configfrom pipelex.graph.graph_config import DataInclusionConfigfrom pipelex.graph.graph_context import GraphContextfrom pipelex.pipeline.job_metadata import JobMetadata, UnitJobId@@ -159,7 +162,7 @@ class TestReportingEventEmission:tokens_usages=[tokens_usage],)- registry = manager._get_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ registry = manager._get_or_create_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]usages = registry.get_current_tokens_usage()assert len(usages) == 1assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 200@@ -182,7 +185,7 @@ class TestReportingEventEmission:tokens_usages=[tokens_usage],)- registry = manager._get_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]+ registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]usages = registry.get_current_tokens_usage()assert len(usages) == 1@@ -304,3 +307,25 @@ class TestReportingEventEmission:events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]assert [evt.sequence for evt in events_a] == [0, 1, 2]assert [evt.sequence for evt in events_b] == [0]++ def test_no_emit_when_no_event_log_set_and_no_fallback_yet(self, mocker: MockerFixture) -> None:+ """Pin the tracing-disabled silent baseline.++ When set_event_log is never called and tracing_config.is_enabled=False,+ report_inference_job must not write any event anywhere — even after the+ Phase 2 fallback lands. The fallback path is only allowed to engage when+ tracing is enabled.+ """+ # Force tracing to disabled regardless of process config.+ mocker.patch.object(get_config().pipelex.tracing_config, "is_enabled", False)++ manager = ReportingManager()+ manager.setup()+ manager.open_registry("run_silent")++ graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent", parent_node_id="g:node_0")+ llm_job = _make_test_llm_job("run_silent", graph_context=graph_context)+ manager.report_inference_job(llm_job)++ # No event log was ever set; no event should have been emitted anywhere.+ assert len(manager._event_log_contexts) == 0 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]diff --git a/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.py b/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.pynew file mode 100644index 000000000..a69abc2ce--- /dev/null+++ b/tests/unit/pipelex/tracing/test_event_log_sequence_threadsafe.py@@ -0,0 +1,85 @@+"""Concurrent sequence allocation must yield unique, contiguous values across threads."""++import sys+import threading+from collections.abc import Iterator+from typing import TYPE_CHECKING, Any++import pytest+from pytest_mock import MockerFixture++from pipelex.tracing.buffering_event_log import BufferingEventLog+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.ndjson_event_log import NdjsonEventLog++if TYPE_CHECKING:+ from pipelex.tracing.event_log_protocol import EventLogProtocol+++def aggressive_gil_switching() -> Iterator[None]:+ """Lower the GIL switch interval to force frequent thread interleaving.++ Without this, CPython batches many bytecodes between GIL releases and the+ `seq = self._sequence; self._sequence += 1` race effectively never triggers+ in short test runs. Restored after the test.+ """+ original_interval = sys.getswitchinterval()+ sys.setswitchinterval(1e-9)+ try:+ yield+ finally:+ sys.setswitchinterval(original_interval)++[email protected]("aggressive_gil_switching")+class TestEventLogSequenceThreadSafety:+ @pytest.mark.parametrize(+ "backend_name",+ ["ndjson", "in_memory", "buffering", "dynamodb"],+ )+ def test_concurrent_next_sequence_returns_unique_contiguous_values(+ self,+ backend_name: str,+ tmp_path: Any,+ mocker: MockerFixture,+ ) -> None:+ nb_threads = 16+ nb_calls = 2000+ total_calls = nb_threads * nb_calls++ event_log: EventLogProtocol+ match backend_name:+ case "ndjson":+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))+ case "in_memory":+ event_log = InMemoryEventLog()+ case "buffering":+ event_log = BufferingEventLog()+ case "dynamodb":+ mocker.patch("pipelex.tracing.dynamodb_event_log.boto3")+ event_log = DynamoDBEventLog(table_name="t", region="us-east-1")+ case _:+ msg = f"Unknown backend_name: {backend_name}"+ raise ValueError(msg)++ results: list[int] = []+ results_lock = threading.Lock()+ barrier = threading.Barrier(nb_threads)++ def worker() -> None:+ barrier.wait()+ local: list[int] = [event_log.next_sequence() for _ in range(nb_calls)]+ with results_lock:+ results.extend(local)++ threads = [threading.Thread(target=worker) for _ in range(nb_threads)]+ for thread in threads:+ thread.start()+ for thread in threads:+ thread.join()++ assert len(results) == total_calls+ assert len(set(results)) == total_calls, "next_sequence returned duplicate values under concurrency"+ assert set(results) == set(range(total_calls)), "next_sequence skipped or repeated values under concurrency"diff --git a/tests/unit/pipelex/tracing/test_writer_id_schema.py b/tests/unit/pipelex/tracing/test_writer_id_schema.pynew file mode 100644index 000000000..cbf6bacbc--- /dev/null+++ b/tests/unit/pipelex/tracing/test_writer_id_schema.py@@ -0,0 +1,220 @@+"""Writer-id schema tests.++Pins the contract that every TraceEvent carries a writer_id field, that backends+expose a writer_id property, and that the read-side dedup/sort/file-routing keys+all incorporate writer_id correctly.+"""++import json+from datetime import datetime, timezone+from pathlib import Path+from typing import Any++from pytest_mock import MockerFixture++from pipelex.graph.graphspec import NodeKind+from pipelex.tracing.buffering_event_log import BufferingEventLog+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.ndjson_event_log import NdjsonEventLog+from pipelex.tracing.trace_events import PipeStartEvent++_TS = datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc)+_PIPELINE_RUN_ID = "run_writer_id_001"+_WORKFLOW_ID = "wf_writer_id_xyz"+++def _make_pipe_start(sequence: int, writer_id: str = "primary", workflow_id: str = _WORKFLOW_ID) -> PipeStartEvent:+ return PipeStartEvent(+ pipeline_run_id=_PIPELINE_RUN_ID,+ workflow_id=workflow_id,+ writer_id=writer_id,+ timestamp=_TS,+ sequence=sequence,+ node_id=f"g:{workflow_id}:node_{sequence}",+ pipe_code="pipe_x",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ )+++class TestWriterIdSchema:+ """Tests pinning the writer_id contract across TraceEvent and event log backends."""++ def test_trace_event_has_writer_id_default_primary(self) -> None:+ """A TraceEvent built without writer_id round-trips with writer_id="primary"."""+ event = PipeStartEvent(+ pipeline_run_id=_PIPELINE_RUN_ID,+ workflow_id=_WORKFLOW_ID,+ timestamp=_TS,+ sequence=0,+ node_id=f"g:{_WORKFLOW_ID}:node_0",+ pipe_code="pipe_x",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ )++ json_str = event.model_dump_json()+ payload: dict[str, Any] = json.loads(json_str)++ assert payload["writer_id"] == "primary"+ restored = PipeStartEvent.model_validate_json(json_str)+ assert restored.writer_id == "primary"++ def test_legacy_ndjson_without_writer_id_field_reads_as_primary(self, tmp_path: Path) -> None:+ """An NDJSON line missing the writer_id field reads back with writer_id="primary".++ Pins backwards compatibility with files written before the writer_id field landed.+ """+ run_dir = tmp_path / _PIPELINE_RUN_ID+ run_dir.mkdir(parents=True)+ ndjson_file = run_dir / f"wf_{_WORKFLOW_ID}.ndjson"++ legacy_payload: dict[str, Any] = {+ "pipeline_run_id": _PIPELINE_RUN_ID,+ "workflow_id": _WORKFLOW_ID,+ "timestamp": _TS.isoformat(),+ "sequence": 0,+ "event_kind": "pipe_start",+ "node_id": f"g:{_WORKFLOW_ID}:node_0",+ "pipe_code": "pipe_x",+ "pipe_type": "PipeLLM",+ "node_kind": NodeKind.OPERATOR,+ }+ ndjson_file.write_text(json.dumps(legacy_payload) + "\n", encoding="utf-8")++ event_log = NdjsonEventLog(traces_dir=str(tmp_path))+ events = event_log.read_events(_PIPELINE_RUN_ID)++ assert len(events) == 1+ assert events[0].writer_id == "primary"++ def test_two_writers_same_workflow_dedup_keeps_both(self) -> None:+ """Read-side dedup distinguishes events by writer_id.++ Two events with same (workflow_id, type, sequence) but different writer_id+ must both survive dedup.+ """+ event_log = InMemoryEventLog()+ event_log.emit(_make_pipe_start(sequence=0, writer_id="a"))+ event_log.emit(_make_pipe_start(sequence=0, writer_id="b"))++ events = event_log.read_events(_PIPELINE_RUN_ID)+ writer_ids = {evt.writer_id for evt in events}++ assert len(events) == 2+ assert writer_ids == {"a", "b"}++ def test_ndjson_writer_id_in_filename_for_non_primary(self, tmp_path: Path) -> None:+ """NDJSON file naming includes writer_id when non-primary; legacy name when primary."""+ primary_log = NdjsonEventLog(traces_dir=str(tmp_path))+ runner_log = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_pid42")++ primary_log.emit(_make_pipe_start(sequence=0))+ runner_log.emit(_make_pipe_start(sequence=0, writer_id="act_pid42"))++ run_dir = tmp_path / _PIPELINE_RUN_ID+ ndjson_files = sorted(path.name for path in run_dir.glob("*.ndjson"))++ assert ndjson_files == [f"wf_{_WORKFLOW_ID}.ndjson", f"wf_{_WORKFLOW_ID}__w_act_pid42.ndjson"]++ def test_two_writers_same_workflow_get_separate_handles(self, tmp_path: Path) -> None:+ """Two NdjsonEventLogs with different writer_ids must not share the file-handle cache slot.++ Pins the (pipeline_run_id, workflow_id, writer_id) cache key. With a 2-tuple key,+ the second writer would steal the first writer's file handle and corrupt routing.+ """+ log_a = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_a")+ log_b = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_b")++ log_a.emit(_make_pipe_start(sequence=0, writer_id="writer_a"))+ log_b.emit(_make_pipe_start(sequence=0, writer_id="writer_b"))+ log_a.close()+ log_b.close()++ run_dir = tmp_path / _PIPELINE_RUN_ID+ a_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_a.ndjson").read_text(encoding="utf-8").splitlines()+ b_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_b.ndjson").read_text(encoding="utf-8").splitlines()++ assert len(a_lines) == 1+ assert len(b_lines) == 1+ assert json.loads(a_lines[0])["writer_id"] == "writer_a"+ assert json.loads(b_lines[0])["writer_id"] == "writer_b"++ def test_ndjson_dedup_uses_writer_id(self, tmp_path: Path) -> None:+ """Read-side dedup for NDJSON keeps events with same (workflow_id, sequence) but different writer_id."""+ log_a = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_a")+ log_b = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_b")++ log_a.emit(_make_pipe_start(sequence=0, writer_id="writer_a"))+ log_b.emit(_make_pipe_start(sequence=0, writer_id="writer_b"))+ log_a.close()+ log_b.close()++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ events = reader.read_events(_PIPELINE_RUN_ID)++ assert len(events) == 2+ assert {evt.writer_id for evt in events} == {"writer_a", "writer_b"}++ def test_ndjson_sort_order_is_sequence_primary_writer_id_secondary(self, tmp_path: Path) -> None:+ """Sort order is (workflow_id, sequence, writer_id) — sequence is primary.++ Without this fix, sorting by (workflow_id, writer_id, sequence) would put+ runner-side `act_*` events before router-side `primary` events for the same+ workflow even though the router's events came earlier in time.+ """+ primary_log = NdjsonEventLog(traces_dir=str(tmp_path))+ runner_log = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_x")++ # Emit out of order to exercise the sort.+ primary_log.emit(_make_pipe_start(sequence=2))+ runner_log.emit(_make_pipe_start(sequence=0, writer_id="act_x"))+ primary_log.emit(_make_pipe_start(sequence=1))+ primary_log.close()+ runner_log.close()++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ events = reader.read_events(_PIPELINE_RUN_ID)++ assert [evt.sequence for evt in events] == [0, 1, 2]+ assert events[0].writer_id == "act_x"+ assert events[1].writer_id == "primary"+ assert events[2].writer_id == "primary"++ def test_dynamodb_sk_includes_writer_id(self, mocker: MockerFixture) -> None:+ """DynamoDB SK encodes writer_id between workflow_id and sequence."""+ captured_items: list[dict[str, Any]] = []++ class _StubTable:+ def put_item(self, *, Item: dict[str, Any]) -> None: # noqa: N803 # boto3 kwarg name+ captured_items.append(Item)++ class _StubResource:+ @staticmethod+ def Table(_name: str) -> _StubTable: # noqa: N802 # boto3 method name+ return _StubTable()++ mocker.patch(+ "pipelex.tracing.dynamodb_event_log.boto3.resource",+ return_value=_StubResource(),+ )++ event_log = DynamoDBEventLog(table_name="my_table", region="us-east-1", writer_id="act_pid42")+ event_log.emit(_make_pipe_start(sequence=0, writer_id="act_pid42"))++ assert len(captured_items) == 1+ assert captured_items[0]["SK"] == f"EVENT#{_WORKFLOW_ID}#act_pid42#0000000000"++ def test_event_log_exposes_writer_id_property(self, tmp_path: Path) -> None:+ """Every event log backend exposes its writer_id via a read-only property.++ Emitters use this to stamp writer_id at event construction time.+ """+ in_mem = InMemoryEventLog()+ ndjson = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_x")+ buffering = BufferingEventLog()++ assert in_mem.writer_id == "primary"+ assert ndjson.writer_id == "act_x"+ assert buffering.writer_id == "primary"diff --git a/wip/02-master-plan.md b/wip/02-master-plan.mdindex 583337bd9..41672d5ef 100644--- a/wip/02-master-plan.md+++ b/wip/02-master-plan.md@@ -12,7 +12,9 @@ The big direction now is to actually run **activities on standalone Worker pools| # | Item | Status | Owner file ||---|---|---|---|-| **P0** | **Tracing & cost reporting across separate-process workers** | Not started — top priority | this file (problem statement) |+| **P0** | **Tracing & cost reporting across separate-process workers** | Done — shipped on `fix/Tracing-across-workers`; see [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md) (T1 marked fixed) | this file (problem statement) |+| **P0.1** | **Dry-run through activity dispatch (testing affordance)** | Not started — surfaced during P0 Phase 6 validation | this file (problem statement) |+| **P0.2** | **Deferred follow-ons from P0** | Not started — surfaced during P0 eng review + PR #860 SWE-agent review | this file (problem statement) || **P1** | **Cross-worker cost report assembly wiring** | Not started — depends on P0 design | this file (problem statement) || **P2** | Phase 6a — Local cross-package dependencies in crate | Not started | [phase6a-local-cross-package-deps.md](phase6a-local-cross-package-deps.md) || **P3** | Phase 6b — Remote dependencies from GitHub | Not started — needs P2 | [phase6b-remote-deps-from-github.md](phase6b-remote-deps-from-github.md) |@@ -23,7 +25,9 @@ Sibling tracks (separate branches, not in this plan): worker error-handling Phas----## P0 — Tracing & cost reporting across separate-process workers+## P0 — Tracing & cost reporting across separate-process workers — DONE++Shipped on `fix/Tracing-across-workers`. The full implementation history (six phases, eng-review notes, decisions, test inventory, atomic-commit list) is preserved in the branch's git log and summarized in [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md). The architectural notes below are kept for context. Follow-ons that were explicitly deferred during implementation are tracked under §P0.2.### Why this is top priority@@ -41,13 +45,56 @@ The whole point of the distributed-execution work is to allow **activities on stWe're not designing the fix yet — that comes when we pick up P0. The two natural starting points are (a) the original Phase 4.5 Step 6 `TracingActivityInboundInterceptor` design (now in `archive/00-master-plan.md` and `archive/01-master-plan.md`), and (b) plumbing tracing config + workflow id through `JobMetadata` so each activity can construct its own event log directly. Either way the activity needs request-scoped tracing data, not process-scoped state.-### Acceptance criteria (sketch — refine when planning)+### Acceptance criteria — all met-- [ ] Activities deployed on standalone Worker pools (separate process from workflow Workers) emit `UsageReportEvent`s that land in the same backend partition as the rest of the run.-- [ ] No reliance on `WfPipeRouter` having executed in the activity's process.-- [ ] No silent drops — if tracing is enabled, an activity that fails to emit raises or logs explicitly.-- [ ] Direct mode and the current single-bundle Worker mode keep working unchanged.-- [ ] Tests covering: (1) standalone activity Worker, (2) mixed worker pool, (3) tracing disabled, (4) backend = NDJSON, (5) backend = DynamoDB.+- [x] Activities deployed on standalone Worker pools (separate process from workflow Workers) emit `UsageReportEvent`s that land in the same backend partition as the rest of the run.+- [x] No reliance on `WfPipeRouter` having executed in the activity's process.+- [x] No silent drops — if tracing is enabled, an activity that fails to emit raises or logs explicitly.+- [x] Direct mode and the current single-bundle Worker mode keep working unchanged.+- [x] Tests covering: standalone activity Worker, mixed worker pool, tracing disabled, NDJSON backend, DynamoDB backend (unit-level via stubbed boto3; full DDB e2e gated on `pytest -m dynamodb`).++---++## P0.1 — Dry-run through activity dispatch (testing affordance)++Today, `--dry-run` instantiates `ContentGeneratorDry()` directly inside the workflow body (`pipe_llm.py:515`, and similar sites in `pipe_extract.py`, `pipe_compose.py`, `pipe_img_gen.py`). The activity (`act_llm_gen_text` etc.) is never dispatched. As a result, the cross-worker `UsageReportEvent` emission path that P0 ships cannot be exercised in dry-run mode against router+runner workers — the runner-side fallback only fires when the activity actually dispatches across processes.++This was discovered while validating P0 Phase 6.1: a vanilla `pipelex run bundle --temporal --dry-run --mock-inputs` against a 2-worker (router+runner) topology produces only `writer_id="primary"` events; no `wf_*__w_act_*.ndjson` files appear. The Phase 4 integration test (`TestSplitWorkerUsageEmission`) works around this by substituting `act_llm_gen_text` with a wrapper that synthesizes an `LLMJob` server-side, but that's test-internal scaffolding, not a general-purpose mode.++### What's needed++A run mode that combines "no real LLM call" with "still go through the activity dispatch path", so a `pipelex run bundle` against router+runner workers exercises the same code path as live mode, just with mocked inference inside the activity. Concretely:++- `act_llm_gen_text` is dispatched to the runner as in live mode.+- Inside the activity, the inference is mocked (use `ContentGeneratorDry` *inside* the activity body, not in place of dispatching it).+- All cross-process surfaces — `_event_log_contexts` lookup miss, runner-side fallback, `writer_id="act_*"` emission — fire normally.++### Why this matters beyond P0++Same affordance unlocks dry-run e2e for any future cross-process work (P1 cost report assembly, distributed graph assembly, payload codec stress testing, etc.) without needing real LLM spend.++### Acceptance criteria (sketch)++- [ ] A `--mock-inference` (or similar) flag — distinct from `--dry-run` — that keeps activity dispatch but mocks inside the activity. Or: redefine `--dry-run` to mean "dispatch activities, mock inside" and rename the current behavior.+- [ ] `temporal-e2e-validate` Tier 8 can run in this mode and surface `wf_*__w_act_*.ndjson` files deterministically.+- [ ] Equivalent mocking sites in `pipe_extract.py`, `pipe_compose.py`, `pipe_img_gen.py` updated.++---++## P0.2 — Deferred follow-ons from P0++Items surfaced during the P0 eng review and the post-merge SWE-agent review of PR #860. Each is independently scoped and not blocking for P0.1 / P1 / P2 / P3. Pick up individually as priorities and signals dictate.++### From eng review (logged at the time of P0 implementation)++- [ ] **`tracing_config.strict_mode: bool`** — opt-in flag that raises (instead of WARNING + drop) when the runner-side emit path fails. Useful for compliance/audit deployments where missing trace data is a hard error. (Eng review R6.)+- [ ] **DynamoDB `BatchWriteItem` for runner emission** — current path does one `PutItem` per usage event. At high-concurrency runners that's one boto3 call per emit; `BatchWriteItem` batches up to 25. Defer until actual throughput shows it matters. (Eng review R3.)+- [ ] **NDJSON shared-filesystem invariant enforcement** — the NDJSON backend assumes `traces_dir` is visible to all writer processes (router pool + runner pool + `act_flush_trace_events`). For multi-host deployments this needs NFS/EFS or equivalent. Follow-up: a startup check that warns if `traces_dir` looks like a local-only path on a multi-worker deployment. (Eng review TODO-2.)++### From PR #860 SWE-agent review (post-merge triage of `fix/Tracing-across-workers`)++- [ ] **Per-thread `writer_id` for activity event log** — the surgical lock fix from PR #860 closes the duplicate-sequence race on `next_sequence()`, but a per-thread writer_id would eliminate the contention entirely and align the design with how Temporal's worker pool already partitions activities. Each thread would emit into its own writer namespace; dedup naturally drops to per-thread, no lock needed in the hot path.+- [ ] **Project-wide migration of `JobMetadata.started_at` to timezone-aware datetimes** — the PR #860 patch builds the synthetic `now` with the same `tzinfo` as the incoming `started_at`, which removes the immediate `TypeError` crash but doesn't fix the underlying inconsistency: `JobMetadata.started_at` default factory is naive (`datetime.now`) while several callers (e.g. `pipe_abstract.py:428`) construct aware datetimes (`datetime.now(timezone.utc)`). A clean migration would standardize on `datetime.now(timezone.utc)` everywhere, document the tz contract on `JobMetadata`, and let `duration` subtract without defensive tzinfo matching at every site.---@@ -94,6 +141,9 @@ In one line: extend the blueprint collector from P2 with a remote (GitHub) resol```P0 (Tracing across separate workers)+ │+ ├─► P0.1 (Dry-run through activity dispatch — testing affordance)+ ├─► P0.2 (Deferred follow-ons — independent, pick up individually)│▼P1 (Cross-worker cost report assembly)@@ -104,4 +154,4 @@ P2 (Phase 6a — local cross-package deps)P3 (Phase 6b — remote deps from GitHub)```-P0/P1 and P2/P3 are independent tracks; P0 blocks P1, P2 blocks P3.+P0/P0.1/P0.2/P1 and P2/P3 are independent tracks; P0 unlocks P0.1, P0.2, and P1; P2 blocks P3. P0.1 is a parallel testing affordance that improves P0 / P1 / future cross-process work confidence. P0.2 items are independent of each other — pick them up individually as needs and signals dictate.diff --git a/wip/tracing-cost-reporting-as-built.md b/wip/tracing-cost-reporting-as-built.mdindex e4c471fac..caed9acb0 100644--- a/wip/tracing-cost-reporting-as-built.md+++ b/wip/tracing-cost-reporting-as-built.md@@ -34,21 +34,25 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set## What is broken or missing-### Issue T1 — Activities on standalone (separate-process) workers lose usage events+### Issue T1 — Activities on standalone (separate-process) workers lose usage events — **FIXED (P0)**-**Severity**: Blocks the whole point of distributed activity workers.+**Severity (was)**: Blocked the whole point of distributed activity workers.-`tasks.py` currently bundles workflows + activities into the same `TaskPack`, so this issue is latent. But the explicit design intent is to split activities onto standalone Worker pools (separate processes from the workflow Worker pool). When that split happens:+**Status**: Resolved on `fix/Tracing-across-workers` (P0 in [02-master-plan.md](02-master-plan.md)). The full implementation history (six phases, eng-review notes, decisions, test inventory, atomic-commit list) is preserved in the branch's git log.-- The activity's process never runs `WfPipeRouter.run()`, so its `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`.-- `ReportingManager._emit_usage_event` returns silently when the context lookup misses (`reporting_manager.py:211-213`).-- `ReportingManager._get_registry` auto-creates a worker-local `UsageRegistry` (with the explicit TODO at `reporting_manager.py:111-117`: *"Auto-create registry for unknown pipeline IDs. This happens when Activities report inference jobs on a Temporal worker where open_registry() was never called (it runs on the API process). TODO: replace with proper distributed reporting system"*).-- The worker-local registry is never read back by anyone — usage data is lost on the activity worker.+The fix has two independent parts that ship together:-**Note**: The `BufferingEventLog` + `act_flush_trace_events` pattern is structurally tied to a workflow lifecycle; it cannot serve standalone activity workers as-is. A different mechanism is needed (the original interceptor design, or per-activity event log construction from tracing config + JobMetadata).+- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot warning per process records that the fallback engaged.+- **Writer-id disambiguation in `TraceEvent`.** A new `writer_id: str = "primary"` field makes `(workflow_id, sequence)` collisions impossible across writers. NDJSON file naming becomes `wf_{workflow_id}__w_{writer_id}.ndjson` for non-primary writers; the `(pipeline_run_id, workflow_id, writer_id)` cache key prevents two writers from sharing a stale file handle. Read-side dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence primary so a runner-side `UsageReportEvent` does not sort before earlier router events. DynamoDB SK becomes `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.++The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is gone: the method was split into `_get_registry_strict` (used by `_report_*_job` — runner processes silently skip the registry add) and `_get_or_create_registry` (used by `inject_tokens_usages`, the console cost-report path, and `generate_report`).++The known retry-related over-counting case (R2) is documented and pinned by `test_retried_activity_emits_duplicate_usage_event_documenting_r2`. A `tracing_config.strict_mode` flag (raise instead of WARNING + drop) is deferred — see [02-master-plan.md §P0.2](02-master-plan.md).### Issue T2 — Cross-worker cost report assembly is not wired+**Status**: Still open. P0 (T1 above) ensures the events now land in the same backend partition from every worker; P1 is the remaining plumbing change to read them back into a cost report.+**Severity**: High — events are persisted but never turned into a report.The pieces all exist:@@ -63,6 +67,8 @@ Net effect: even when usage events ARE captured (i.e., the same-worker case in t### Issue T3 — `set_event_log` is on a process-singleton `ReportingManager`+**Status**: Partially mitigated by P0. The runner-side fallback now bypasses `_event_log_contexts` entirely when the lookup misses, so the singleton-+-context-dict pattern no longer blocks standalone activity workers. The deeper "request-scoped tracing data instead of process-scoped state" refactor remains open and is the natural follow-up alongside P1.+**Severity**: Design smell that magnifies T1.`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme works because the workflow process is the same process that emits events from activities (today). The moment activity execution moves off-process, the singleton + context-dict pattern stops being the right shape — the activity needs to pull its event log from request-scoped data (e.g., `JobMetadata` / `TracingContext`), not from a process-local dict that was never populated on its process.@@ -71,6 +77,6 @@ Net effect: even when usage events ARE captured (i.e., the same-worker case in t## How this connects to 02-master-plan-- T1 is the **top priority** in [02-master-plan.md](02-master-plan.md) — the whole point of distributed execution is to allow activities on standalone Worker pools.-- T2 is the **second priority** — once T1 is solved, the assembly wiring is the small remaining step that makes cost reporting actually report.-- T3 is the architectural shape that the T1 + T2 work will refactor.+- T1 was the **top priority** in [02-master-plan.md](02-master-plan.md) — now resolved on `fix/Tracing-across-workers`.+- T2 is the **next priority** — events now land in the same partition from every worker; the small remaining step is wiring `read events → UsageAggregator → inject_tokens_usages → generate_report`.+- T3 is the architectural shape; the T1 fix mitigates the immediate impact, the deeper refactor will land alongside T2 / further follow-ups.